Bibliometric Analysis of Interdisciplinary Research

Report to the

Higher Education Funding Council for England

Bibliometric analysis of interdisciplinary research

Jonathan Adams, Louise Jackson, Stuart Marshall

November 2007 Contact details The reporting organisation is: Evidence Ltd 103 Clarendon Road, Leeds LS2 9DF T/ 0113 384 5680 F/ 0113 384 5874 E/ [email protected] Evidence Ltd is registered in England, Company no 4036650, VAT registration 758467185 http://www.evidence.co.uk

Bibliometric analysis of interdisciplinary research

Contents

Executive summary ………………………………………………………………………………………...1

1 Introduction...... 3

2 Defining and indexing interdisciplinarity ...... 4

3 Differential citation impact ...... 15

4 Presumed interdisciplinarity...... 23

5 Implications for bibliometric assessment and suggestions for further work ...... 26

6 Subject category assignment ...... 28

Annex A Units of Assessment clustered on journal frequency...... 31

Bibliometric analysis of interdisciplinary research

Executive summary

The research community has from time to time argued for separate assessment of interdisciplinary research, but has not found an objective definition of such work. The notion is that interdisciplinary work is treated and valued differently from other work. It is hypothesised that, being marginal to core subjects, it would be systematically cited less often. If so, this might be a problem for a metrics-based system of assessment where peer judgments that distinguish work of this nature would be absent.

In this study, we have explored this notion further by: • developing an objective description of research output according to an index of its interdisciplinarity; • testing whether there is differential citation impact across the range of interdisciplinarity; • evaluating our findings against presumed interdisciplinary outputs submitted to the Research Assessment Exercise (RAE) and outputs assigned to Thomson Scientific® Inc’s ‘Multidisciplinary’ category; • considering options for handling interdisciplinary work within the proposed system of bibliometric- based indicators.

We collated article records supplied by Thomson Scientific to create databases of publications – source articles – for two large research-intensive UK universities. We analysed the work (other publication references) that cites and is cited by these source articles. To define whether a citation link to or from a source article was within or external to the article’s discipline, we used the Thomson journal categorisation.

We developed measures of interdisciplinarity by indexing the subject spread of the references cited by the source articles. We found that: • there was a continuum of interdisciplinarity at article level within journal categories; • the categories themselves fitted a broad but not categorical typology ranging from monodisciplinary to multidisciplinary.

Using the citing references, within each journal subject category we analysed the citation count (impact) of each source article. We then compared the interdisciplinarity of each article against its rebased citation impact (normalised for year of publication and subject category). We found: • no tendency for the most interdisciplinary articles to be less frequently cited on any consistent basis within or across categories; • a weak tendency for the articles with highest citation impact to be neither very monodisciplinary nor very multidisciplinary in terms of their cited references.

Our comparative analysis of presumptive interdisciplinary datasets provided only limited further information.

Our findings have implications for the assessment of interdisciplinary research: • There is no evidence to support an explicit intervention regarding articles that refer to work across a broader range of disciplines. They do not receive systematically fewer citations than more monodisciplinary publications. • If impact is to be normalised at any categorical level, there is evidence that careful attention must be paid to the choice of baseline category, especially for articles in journals that are assigned to multiple categories. The Multidisciplinary category used by Thomson does not, however, necessarily contain articles that cross more subject boundaries than research in other Thomson categories. • Since our index of interdisciplinarity may not satisfy all parties, the methodology should ideally be evaluated in due course through consultation with informed researchers.

In summary, there is no strong case on the basis of this analysis for research outputs to be treated differently for the purposes of bibliometric assessment on the grounds of interdisciplinarity. It is important, however, to exercise care in choosing the appropriate field against which to normalise the citation rates of more interdisciplinary outputs.

1 Introduction

This study has provided a preliminary exploration of the notion that interdisciplinary work might be at a disadvantage in research evaluation because it is systematically judged in different terms to core disciplinary activity. Specifically, the project considered whether published work which is more or less interdisciplinary might receive consistently different rates of citations from other publications. Our approach was, first, to develop an objective methodology for indexing interdisciplinarity. Secondly, we tested for differential citation impact among articles with higher and lower levels of indexed interdisciplinarity. The interdisciplinary nature of an article was indexed by analysing the diversity of the subject categories from which it drew its cited references. Within each subject category, we found that articles varied across a continuum from monodisciplinary to multidisciplinary, with no obvious discontinuities or clustering. Across subject categories, we found that some tended to be more generally multidisciplinary than others. Thus it was possible to characterise the subject categories as low, medium or high diversity. When we compared interdisciplinarity with citation impact, we found no evidence that interdisciplinary work had consistently lower citation impact. Indeed, there was a tendency for many highly cited articles to be mid-range in terms of interdisciplinarity, and in most subjects there was a weak or very weak positive correlation between interdisciplinarity and citation impact. That is to say, publications which referred to articles across a broad range of subject categories were (marginally) more likely to attract a higher number of citations.

While our approach to indexing interdisciplinarity has been based on widely accepted methodologies, we suggest that their validity should be further tested against the perceptions of researchers. The remainder of this report is split into four sections: Section 2 Defining and indexing interdisciplinarity Section 3 Differential citation impact Section 4 Presumed interdisciplinarity Section 5 Implications for bibliometric assessment and suggestions for further work.

2 Defining and indexing interdisciplinarity

This section describes how we indexed the interdisciplinarity of research outputs. There is no clear or generally accepted definition of interdisciplinarity; it is a subjective term. Many researchers believe that they are working in an interdisciplinary area. This seems possible, given that it is widely agreed that good and innovative research often draws on discoveries at the margins of more than one discipline.1 But there are no criteria for judging whether this is true. This is problematic because some researchers also believe that interdisciplinary research, working outside the core of established disciplines, is at risk of being marginalised in research evaluation and assessment. A desirable outcome might therefore be lost if the best researchers were to shun interdisciplinarity because of this perception.

How can we determine whether such a risk exists and take steps to avoid it? Rather than relying on the opinion of the researcher or an expert group, it is preferable for assessment purposes to identify an objective (measurable) characteristic that marks out interdisciplinary research. Our study began, therefore, by developing an index of interdisciplinarity.

2.1 Methodology Research outputs can be characterised at different levels of aggregation, from individual outputs to researchers and research groups or departments. For this study, we focused on the article level. Our aim was to characterise the (relative) interdisciplinarity of an individual article so that this could be compared against the article’s citation impact.

What characteristics of an article might define its interdisciplinary nature? One approach is to look at the previous work referred to by the article in question. Arguably, work claiming interdisciplinary influence would draw on, or cite, work from a greater than average range of fields. Similarly, it would be expected that work claiming interdisciplinary utility would be cited by journals in a greater than average range of fields.

1 See for example the Government’s Science and innovation investment framework, 2004-2014.

There are precedents for this approach. Investigations into interdisciplinarity via the analysis of cited references have been carried out by Leydesdorff2 at journal level and Porter et al3 at article level. Within the bibliometric databases of Thomson is another precedent, in the way that the individual articles from multidisciplinary journals (such as Nature and Science) are categorised. These articles are individually assigned to subject categories according to the subject categories of the articles they cite and are subsequently cited by. For example, an article with many references to physics journals is probably a physics article and will be assigned to the Physics category.

Following these established approaches, the cited papers for any article should provide an appropriate source of material to evaluate the association between article and subject category. Accordingly, we based our definition of an article’s interdisciplinarity on the range of subject categories of the cited (and citing) references of the article.

Subject categories Whether a researcher’s work can be described as interdisciplinary depends on how finely or coarsely the discipline categories are defined. Fewer research outputs cross boundaries where subject categories are broad than would be the case if the categories were narrower. So, the level of interdisciplinarity identified by the analysis of cited references is determined by the level of aggregation of the subject categories.

For the purposes of the current study, we used the subject categories from the Thomson Current Contents® range of products There are 106 Current Contents categories in total, and the categories are grouped into seven discipline-specific products. A list of Current Contents® subject categories and further details about subject categorisation are included in section 6 of this report. Full journal lists and scoping notes for each Thomson category are available at http://scientific.thomson.com/mjl/

It would be quite feasible to apply our methodology at other levels of granularity. For example, it could operate at the finer level of Thomson’s Science Citation Index Expanded™ categories, or at the coarser level of UK RAE Units of Assessment (UoAs). We have shown elsewhere that several Current Contents categories map closely onto each UoA, based on RAE publications.4 For the same study, we also

2 Leydesdorff L, "Betweenness centrality" as an indicator of the "interdisciplinarity" of scientific journals. Journal of the American Society for Information Science and Technology, 58(9), 1303-1309, 2007. 3 Porter A L, Cohen A S, Roessner J D and Perreault M, Measuring researcher interdisciplinarity. Scientometrics, 72(1) 117-147, 2007. 4 Adams J, Benchmarking international research. Nature, 396, 615-618, 1998. Adams J, Bailey T, Jackson L, Scott P, Pendlebury D and Small H, Benchmarking of the international standing of research in England: report of a consultancy study on bibliometric analysis. CPSE, University of Leeds. 108 pp, 1998. ISBN 1 901981 04 5

calculated the similarity between UoAs, based on the frequency with which common journals were submitted to the 1996 RAE. The tree diagram at Annex A shows the similarity between UoAs and illustrates how a coarser level of aggregation would lead to lower levels of interdisciplinarity.

If the analysis were to be repeated at a coarser level of subject aggregation, although fewer articles would be identified as interdisciplinary, each article’s impact value would then be normalised against a broad (less appropriate) subject baseline. Elsewhere,5 we have explored the effect of changing the normalisation baseline and established the importance of selecting the appropriate level of ‘zoom’ for each analysis.

At Evidence we routinely use Current Contents categories as the preferred level of granularity, based on our experience of providing bibliometric analyses for a range of clients. Our databases are structured to allow a wide variety of analyses to be performed at this level. We discussed the options for introducing an alternative (finer or coarser) category system in background work for this project, but ultimately concluded that the resource implications would outweigh the possible benefits. Similarly, we reviewed the possibility of building in a measure of 'distance' between categories, but it was felt that this level of complexity was outside the scope of the project.

Defining the dataset The dataset for this analysis contained articles produced by two research-intensive UK universities during the seven-year period 1997-2003. The analysis was restricted to those outputs (sources, cited and citing articles) which had been assigned by Thomson to a Current Contents® subject category and which had been indexed as ‘articles’, ‘notes’ or ‘reviews’. In total, the dataset contained 37,000 source articles in the Science, Technology, Engineering and Medicine (STEM) subject categories.6 There were upwards of 500,000 associated cited and citing articles. The discussion and presentation of results focuses on the 88 STEM subject categories. We have included Mathematics and some Social Science categories, however, since this makes for an informative comparison.

Of the 37,000 articles, 31,000 were from University A and 7,000 from University B. Some articles were collaborative between the two. In the text that follows, most of the detailed analyses relate to outputs from University A, the larger of the two. We have included comparative data from University B to substantiate

5 Adams J, Gurney K, Jackson L, Calibrating the zoom – a test of Zitt’s hypothesis. Scientometrics, 2007, forthcoming 6 The UK Government has announced that changes to research assessment after 2008 will initially affect only STEM disciplines (science, technology, engineering and medicine).

our claim that the findings are general, i.e. that they will apply to outputs from any research-intensive university. We have avoided repeating the same statements for B that we have already made for A.

We characterised the interdisciplinarity of the source articles by analysing the subject categories of their cited (and citing) articles, using only those citing articles that were published during the seven-year period 1997-2003. When calculating the citation impact for the source articles, we used citation counts to the end of 2006. We calculated the rebased impact (RBI) by normalising the number of citations a stated article had received in the context of the world average number of citations for articles published in the same subject category and year (see also section 3).

2.2 Defining interdisciplinarity indices We indexed the interdisciplinarity of each article by analysing the way in which its cited references were spread across subject categories. We prefer cited references to citing references as a first test of disciplinary spread because this best reflects authors’ perceptions of the diverse relevance of their work. It is also the case that new papers are likely to have few citations, and that articles with high citations are de facto ‘high impact’ irrespective of disciplinarity. At the outset, however, we compared both cited and citing information to evaluate consistency.

First, we defined each article’s subject category as its ‘source’ category. We chose three different approaches to measuring the reference category spread, resulting in three indices of interdisciplinarity. The first and simplest of the three indices is the proportion of cited references that lie outside the source category. The second index is a count of distinct cited categories. For the third index we used the Shannon diversity index.7 This combines information about the number of cited categories and the evenness of the spread of cites among categories; the index value increases with the number of cited categories and with greater evenness of cites among categories. Inevitably, it is the least transparent but also the most informative index. The indices and their definitions are summarised in Table 1. We calculated a further three interdisciplinarity indices for each article by analysing the subject categories of the citing references.

7 The Shannon information index originally arose out of work on information theory and has subsequently been widely used, e.g. as a measure of biodiversity.

Table 1 Definitions of interdisciplinarity indices using analysis of cited references

Index D escription Definition Minimum value

INDEX I Proportion of cited references Number of cited references in categories other than the source 0 outside source category category divided by number of cited references.

INDEX II Count of cited categories Number of distinct categories into which cited references fall. 1

INDEX III Shannon diversity index Sum, over all categories, of: 0 the proportion of cited references in that category multiplied by the (if all cited references are in (if reference natural logarithm of the proportion of cited references in that category. the same category)

2.3 Analysing the interdisciplinarity indices

Category-level interdisciplinarity We calculated a category-level version of index II (number of cited categories) by counting the number of distinct categories cited by each of the source categories. Then from among the same set of cited references, we counted the number of categories citing each source category. There was a range of category-level interdisciplinarity, but no obvious relationship between numbers of cited and citing categories. For roughly half of the source categories, the number of cited categories exceeded the number of citing categories; for the other half, the reverse was true. Figure 1 shows these results for the 88 STEM subject categories arranged by decreasing number of cited categories. The number of categories cited by the source category is shown as a dark square. The vertical bars show the number of categories that cite the source category.

Figure 1 Category-level analysis of cited categories and citing categories

100 Number of categories How many categories cite this category? How many categories does this category cite?

0 AS EA MT EN PH SOC PU PH EN ON DG INF PMC HE PS CG CMP MT PE CV RE PS AR RA UR CA I/M SU CML OR HL AP GE NU OP AIC BT F PL OR GA O/ DE EN VE EMA ME IN RH AQU CMA CS SP EL OTO PS CME EE RE CIV DE NC GN S/I MU BIL BE MB MC MG EX ME PS BI OG NE CE SIA IM PH GN O C A M T C L E H P O R D I P T Y H R V D S C M B S A R T N M P D R E U L O T U L O H S H R G E C C T G C D X X L B S

Source: data, Thomson Scientific® Inc; analysis, Evidence Ltd. Categories and their abbreviated codes are listed in section 6; highlighted categories are reviewed in more detail later.

Article-level interdisciplinarity We found no clustering of articles at any particular level of interdisciplinarity within a subject category. Instead, there was normally a continuum of values for each of the three indices. The exceptions were extreme cases of monodisciplinarity where all of an article’s cited references fell within the source category. Each of the three indices addresses a slightly different aspect of the interdisciplinarity of the cited articles: the values of indices II and III are connected (though not proportional) since low numbers of cited categories mean low diversity and vice versa. Index I (the proportion of cited references outside the source category) gives differing results from the other two because it is the only one of the three that differentiates citations to articles within the source category from citations to articles outside it. It is possible for an article with minimum values for indices II and III to have a maximum value for index I. This occurs when all the cited references are in the same category, but that category is not the same as the source category. As there is overlap between some of the Current Contents® categories, references outside the source category are not necessarily outside the source field. An example of this would be where a source article assigned to the Haematology category cites articles assigned to Cardiovascular and Haematology Research. Such an article would gain a high score for index I, while its Shannon diversity index would be low. As a first step to evaluating the advantages and disadvantages of using the three indices, we compared the values of each index for each article in two different subject categories. For this comparison we chose Space Science (788 articles, a relatively monodisciplinary category) and Biochemistry and Biophysics

(1,775 articles, a relatively multidisciplinary category). The articles were arranged in descending order of Shannon diversity with the respective values of all three indices plotted for each article. The results are illustrated in Figure 2.

Figure 2 Comparison of article-level interdisciplinarity using three indices

Space Science Biochemistry & Biophysics Index I and III Index II Index I and III Index II

3 20 3 40

15 30

0 10 0 20

5 10

-3 0 -3 0

Index I - Proportion of cited references outside source category Index I - Proportion of cited references outside source category Index II - Number of distinct cited catgories Index II - Number of distinct cited catgories

– Index III - Shannon diversity index for cited categories – Index III - Shannon diversity index for cited categories

Source: data, Thomson Scientific® Inc; analysis, Evidence Ltd.

The left-hand chart in Figure 2 shows the spread of values of the three indices for the articles in the Space Science category. The maximum number of cited categories was 16, and the most frequently occurring value – the mode - was 2. Articles where both the Shannon index and the proportion of cites outside the source category were at zero can be described as monodisciplinary. This applied to about one-third of the Space Science articles.

The right-hand chart in Figure 2 shows that Biochemistry and Biophysics had a much higher proportion of multidisciplinary articles. The number of cited categories was higher (maximum 35, mode 8), and there was a much lower percentage of monodisciplinary articles in this subject category.

From this presentation we concluded that where only one index is used, to avoid redundancy in the analysis, then our preferred index should be index III – the Shannon diversity index. Its variation is broadly in line with, and certainly not contradictory to, the other two indices (see Figure 3), but it encapsulates more information. The disadvantage is that its interpretation is less transparent and straightforward.

Comparison of diversity indices for selected We selected seven subject categories for closer scrutiny (see Table 2). These were categories with categories relatively high volume chosen from broader disciplinary areas across STEM, and also with varying levels of interdisciplinarity. As we shall be returning to these categories later to discuss differential citation impact, we also ensured that the selection contained categories with varying levels of rank correlation between interdisciplinarity and citation impact (see section 3).

Table 2 Comparison of interdisciplinarity indices for selected subject categories Figure 3 Average interdisciplinarity indices

Rebased 10 Subject category CODE Article count INDEX I INDEX II INDEX III DIVERSITY impact INDEX II Cited categorPropn outsideShannon index Space Science SP 7881.70.12.10.3Low Physics PHS 2208 1.8 0.2 2.4 0.5 Low Mech Eng MEC 264 1.2 0.4 2.7 0.7 Medium 5 Envt & Geo GEO 184 1.4 0.6 3.0 0.7 Medium Chemistry CMP 511 1.8 0.7 4.9 1.2 Medium Biochem & Biophys BIL 1775 1.4 0.6 7.0 1.4 High Microbiology MCB 7611.50.66.91.5High 0 SP PHS MEC GEO C

INDEX I - Average proportion of cited references outside source category INDEX II - Average number of cited categories INDEX III - Average Shannon diversity index for cited references

Source: data, Thomson Scientific® Inc; analysis, Evidence Ltd. Source: data, Thomson Scientific® Inc; analysis, Evidence Ltd.

In Figure 3, the subject categories have been arranged (left to right) in ascending order of Shannon diversity. By comparing the heights of the bars for each subject, we see that each index places the subject categories in almost the same rank order. The notable exception is Chemistry, which is ranked highest when interdisciplinarity is indexed by the proportion of references outside the source category. We believe this is mainly due to an artefact of the Current Contents system. There are effectively two Thomson chemistry categories: ‘Chemistry’ (used in Figure 3) forms part of the Physics, Chemistry and Earth Sciences Current Contents® product, whereas ‘Chemistry and Analysis’ forms part of the Life Sciences product. The two categories inevitably cross-refer.

Variation of interdisciplinarity with category There was wide variation in size among the Current Contents® categories for each of the samples of size articles from University A and University B. One reason for this could be that some distinct sub-disciplines had been aggregated into larger categories. If that were the case, then one might expect the articles in

those larger categories to cite fewer articles outside the source category – i.e. that the larger categories would appear more monodisciplinary. However, when we tested for a connection between the size of the category and the interdisciplinarity of the articles within it, we detected no variation of interdisciplinarity with category size. The results are illustrated in Figure 4. The subject categories have been arranged in order of descending size (frequency). The red triangle represents the average Shannon diversity index (cited references) for the articles in each subject category.

Figure 4 Diversity of articles’ cited references set against category size

Average Shannon diversity index 2500 Number of articles 2.0

2000 1.5 1500 1.0 1000 500 0.5 0 0.0 CVS PUB ME ARA CAR PE PL CGX SUR ORT RA O/ CSE GE I/M URO AI EL RHU PM OP DER NU EN PSY HL CM GA F VE CM CIV S/ BT EM OT AQ CM REH NCL ME DEN IS LIB A/ EEE AE GNE GP A P BI AP BEH MGN PHC MU MB IM SP MTH EAR MC NE PS MTR DGX CM ME OR PS OGS CEL ENV ONC GN HEM END SI AS EXP SO INF BI INC PHM REP PS N H T C I A L A O A M T C D T R P O L I D T U S O T C O H S M U G G C A T C D C L B L E A P

Subject categories in descending order of size

j Average Shannon diversity index for articles in this category (cited references)

Number of source articles in this category Source: data, Thomson Scientific® Inc; analysis, Evidence Ltd.

Using cited references rather than citing As noted earlier, although it is possible to index the interdisciplinary utility of a piece of work by analysing references the subject categories of the articles that subsequently cite it, analyses based on indices that use these citing references may give an unbalanced view. This is because articles which have received a very low number of cites cannot, logically, show a high level of interdisciplinarity.

We compared interdisciplinarity calculated using cited references against the corresponding indices using citing references, and found that they were broadly in agreement. Index values based on cited references generally exceeded those based on citing references as might be expected since most articles refer to previous work, whereas some remain uncited. The average Shannon indices for cited and citing references are compared in Figures 5 and 6.

Figure 5 Comparison between average diversity for cited and citing references by category Figure 6 Shannon (cited) vs Shannon (citing)

2 Average Shannon Average Shannon diversity index (citing) Average Shannon diversity index (c diversity index 2 Average Shannon diversity index (cited)

1 1

0 RH UR SO CA NE DG DE PH PE GN INC PS PS BE EN CM MUL EN RE SI PU AS PL RA HLT AQU SU AI PS MT EM PH MG A/ OR CME OR PS O/A DE OTO GP AP OP NC EE EL I/M AR GE ME MEC EA AE S/ CIV CS PH IS LIB MT SP G A CG EN CE ON ME MBG RE CMA BT BI PMC IM IN MC EX CML NU CV GA VE F OG HE BI N N T A C A I O L F 0 M C T O D P E R R P Y I H L D S A D M T V B R C E L H S M R U R P N L O C H S T U R M O H C A T D T E C X X P B S 01 Average Shannon div Subject categories in descending order Shannon diversity index for CITED references

Source: data, Thomson Scientific® Inc; analysis, Evidence Ltd.

Validation and further development of the Having reviewed the relationships explored in the above sections and figures, we elected to concentrate methodology our work for the remainder of the study on Shannon diversity indices based on cited references. We maintained comparisons across indices where informative. If indices of interdisciplinarity were to be put to further use for research assessment purposes: • it would be necessary to validate the results against the views of experts in the relevant fields; • some measure of the inter-relatedness of fields might be developed so that citations to ‘distant’ fields would be weighted more heavily than citations to ‘neighbouring’ fields.8

8 Porter A L, Cohen A S, Roessner J D and Perreault M, Measuring researcher interdisciplinarity. Scientometrics, 72(1) 117-147, 2007.

3 Differential citation impact

The aim of this study was to test the notion that interdisciplinary research is cited relatively less frequently than other research. So, having developed and applied a methodology for indexing articles’ interdisciplinarity, it remained to establish whether there is a relationship between interdisciplinarity and citation impact.

Citation counts vary by year, because more recent articles have less time to accumulate citations, and by discipline, because of cultural differences among fields. To take account of this, it is customary to normalise the citation count for a given article by reference to the average citation count for any article published in the same year and in the same field. This is also called rebasing, so the impact (citation count) is referred to as rebased impact or RBI.

The RBI was calculated for each source article by dividing the number of citations it had received by the world average number of citations for articles published in the same subject category and year. We then correlated the RBI against each of our three interdisciplinarity indices for the source articles in each subject category.

The data in these analyses were highly skewed. Spearman rank correlation was therefore used, rather than a parametric correlation.

3.1 Rank correlation between citation We found that rank correlation between RBI and: impact and interdisciplinarity • Shannon diversity was positive or insignificant for most subject categories; • number of cited categories was positive or insignificant for most subject categories; • proportion of cited references outside the source category was negative for about half of the subject categories, and suspect that this was largely due to overlapping subject categories (see 2.3). A positive rank correlation between RBI and interdisciplinarity means that the more interdisciplinary papers tended to be more highly cited. Conversely, a negative correlation would mean that more interdisciplinary papers received relatively fewer citations. Figure 7 shows the values for the correlation coefficient between RBI and each of the three indices for each STEM subject for University A.

Figure 7 Spearman rank correlation between citation impact and three interdisciplinarity indices for each subject category (University A)

Spearman rank correlation coefficient 1

Rank correlation between RBI and proportion of cited references outside source category Rank correlation between RBI and count of cited categories Subject categories in discipline order Rank correlation between RBI and Shannon diversity index for cited references -1 CGX HEM IMM END GNC OPH PMC REP SUR DEN NEU PSO HLT SOC MCB VET AQU ENT AS CMP PHC APP EAR NCL MET CSE IST CME GEO Clinical medicine Life sciences Physical sciences Engineering

Source: data, Thomson Scientific® Inc; analysis, Evidence Ltd.

Rank correlation I: RBI against proportion of In about half the categories, there was a negative rank correlation between index I and RBI, and for the cited references that are outside the source other half the correlation was positive. This is illustrated by the position of the grey squares in Figure 7, category half of which are below the horizontal axis and half of which are above. As noted above (in section 2.3), the category system used has a number of overlapping categories, which may lead to some spurious results for this index. If a high proportion of the cited references are in a closely related category, then the article will have a correspondingly high interdisciplinary index.

Rank correlation II: RBI against number of In more than 90% of the subject categories, there was a positive rank correlation between the number of distinct cited categories categories cited and the article’s RBI. This is illustrated by the position of the pink diamonds in Figure 7, nearly all of which are above the horizontal axis.

Rank correlation III: RBI against Shannon In more than 90% of the subject categories, there was a positive rank correlation between diversity index for cited articles interdisciplinarity indexed by the Shannon diversity index for the distribution of cited references across subject categories and the article’s RBI. This is illustrated by the position of the red triangles in Figure 7, nearly all of which are above the horizontal axis.

3.2 Did the rank correlation vary Figure 7 shows the rank correlation for University A. Figure 8 adds the corresponding values for between universities? University B, shown as outlined shapes. It is clear that the results for the two institutions are very similar. We infer that the consistency of outcomes for these two universities could be generalised to other research-intensive universities. In the rest of this report we focus on the larger dataset (University A).

Figure 8 Spearman rank correlation between citation impact and three interdisciplinarity indices for each subject category (University A and University B)

Spearman rank correlation coefficient Solid shapes: University A. Outline shapes: U 1.0

0.0

Rank correlation between RBI and proportion of cited references outside source category Rank correlation between RBI and count of cited categories Rank correlation between RBI and Shannon diversity index for cited references Subject categories in discipline orde -1.0 CGX HEM IMM END GNC OPH PMC REP SUR DEN NEU PSO HLT SOC MCB VET AQU ENT AS CMP PHC APP EAR NCL MET CSE IST CME GEO Clinical medicine Life sciences Physical sciences Engineering

Source: data, Thomson Scientific® Inc; analysis, Evidence Ltd.

3.3 Did the rank correlation vary with We interrogated the data to determine whether source category size (the numbers of articles in our category size? sample in that category) affected the rank correlation between RBI and interdisciplinarity. We found no apparent variation of correlation with category size (Figure 9).

Figure 9 Spearman rank correlation between RBI and diversity of articles’ cited references set against category size

2500 Number of articles Spearman rank correlation between RBI and Shannon diversity index 1.0 2000 0.5 1500 1000 0.0 500 0 -0.5 CV PUB ME ARA CAR PE PL CGX SUR ORT RAD O/A CS GE I/M URO AI EL RH PM OP DER NUT EN PS HL CM GA F VE CM CIV S/ BT EM OT AQ CM REH NCL ME DE IS LIB A/ EEE AER GN GP A P BI AP BEH MGN PHC MU MB IM SP MTH EAR MC NE PS MTR DGX CM ME OR PS OGS CEL ENV ONC GN HE END SI AS EXP SO IN BI INC PHM REP PS N H T C I A L A O F M T C Y P O L D T I S E M T N U S O U U C O H S M G E G C C T D C A E L B L A P

Subject categories in descending order of size

Spearman rank correlation between RBI and average Shannon diversity index for articles in this category (cited references)

Number of source articles in this category Source: data, Thomson Scientific® Inc; analysis, Evidence Ltd.

3.4 Closer inspection of rank The rank correlation coefficient summarises, in one value, the relationship between two values for each correlation article in a subject category: the article’s interdisciplinarity and its rebased impact. The actual spread of values can best be visualised using a scatter-plot where each point represents an article and the position of each point is plotted according to its interdisciplinarity index (on the horizontal axis) and its RBI (vertical axis).

Figure 10 shows examples of scatter plots which illustrate the relationship between RBI and interdisciplinarity. They show the three indices for each of two subject categories: Space Science (relatively monodisciplinary) and Biochemistry and Biophysics (more multidisciplinary).

Figure 10 Sample scatter plots comparing articles’ RBI against three interdisciplinarity indices for Space Science and Biochemistry and Biophysics

Space Science Biochemistry & Biophysics In the scatter plots in Figure 10, each point represents RBI INDEX I RBI INDEX I an article. The position of each point represents the rebased impact and interdisciplinarity of each article. 40 40 For each subject there are three plots, to illustrate each of the interdisciplinarity indices. 20 20 The two topmost plots show, for each article, the 0 proportion of cited references that lie outside the subject 0 0.0 0.5 1.0 category (index I). For Space Science, most of the cited 0.0 0.5 1.0 references for most of the articles are also in Space Proportion of cited references outside source Proportion of cited references outside source category category Science. By contrast, this is less the case for Biochemistry and Biophysics, where the articles are RBI INDEX II RBI INDEX II spread towards the right-hand side of the plot. 40 40 A similar contrast between the subjects is apparent from the two central plots for index II. 20 20 The bottom two plots illustrate the spread of values for the Shannon diversity index of cited references. The 0 0 Space Science articles are to the left-hand, low-diversity 0102001020 end of the axis whereas most of the Biochemistry and Number of cited categories Number of cited categories Biophysics articles are in the middle. The scatter plots allow us to visualise the patterns RBI INDEX III RBI INDEX III behind the correlation coefficients reported in Table 3 40 40 below. For Space Science, the rank correlation between RBI 20 20 and interdisciplinarity is very low on each index. For Biochemistry and Biophysics the correlation is positive

0 0 and significant at the 1% level for indices II and III. 0123 0123 There is no rank correlation with RBI for index I. Shannon diversity index Shannon diversity index

Source: data, Thomson Scientific® Inc; analysis, Evidence Ltd

We selected a further five of the larger subject categories, across a range of disciplines and with a range of values for interdisciplinarity and rank correlation with RBI. The rank correlation for each of the categories is shown in Table 3, and Figure 11 illustrates how the correlation coefficients varied between the three interdisciplinarity indices for each subject.

Table 3 Rank correlation of RBI with interdisciplinarity indices for selected subject categories Figure 11 Rank correlation with RBI

Rebased RCORREL 0.5 Correlation Subject category CODE Article count RCORREL I RCORREL II impact III with RBI Cited categori Propn outside Shannon index Space Science SP 788 1.7 -0.05 0.06 -0.02 0.3 Physics PHS 2208 1.8 0.00 0.14 0.06 Mech Eng MEC 264 1.2 0.20 0.27 0.26 Envt & Geo GEO 184 1.4 -0.01 0.29 0.25 0.0 Chemistry CMP 511 1.8 -0.21 0.29 0.19 Biochem & Biophys BIL 1775 1.4 -0.02 0.32 0.26 Microbiology MCB 761 1.5 0.03 0.32 0.22 -0.3 SP PHS MEC GEO C

RCORREL I - Spearman rank correlation of RBI against average proportion of cited references outside source category RCORREL II - Spearman rank correlation of RBI against average number of cited categories RCORREL III - Spearman rank correlation of RBI against average Shannon diversity index

Source: data, Thomson Scientific® Inc; analysis, Evidence Ltd. Source: data, Thomson Scientific® Inc; analysis, Evidence Ltd.

We found that citation impact was positively correlated with rebased impact in most subjects for indices II and III. The correlation was significant at the 0.1% level for five subjects, excluding Space Science and Physics. For index I there was a weak negative correlation between interdisciplinarity and RBI for Space Science, Environmental Studies, Chemistry, and Biochemistry and Biophysics. To investigate the spread of articles behind the correlation coefficients, we then compared the index III scatter plots for each of the seven subject categories. We also compared the results for University A with those of the smaller University B (Figure 12).

Figure 12 Scatter plots comparing RBI against Shannon diversity index for selected subject categories (University A and University B)

UNIVERSITY A UNIVERSITY B There is a marked similarity between the scatter

20 RBI RBI 20 plots for University B, on the right, and those for 20 20 University A. The distribution of the articles in

10 Space Science 10 relation to the axes follows a similar pattern in both institutions for each of the seven subjects.

0 Diversity 0 Diversity 0 1 2 3 0 1 2 3

20 RBI RBI 20 The subject categories themselves have 20 20 characteristic distributions according to whether

10 Physics 10 they are more or less interdisciplinary. Those like Space Science and Physics are relatively 0 Diversity 0 Diversity 0 1 2 3 0 1 2 3

20 RBI 20 RBI monodisciplinary, with more articles towards the 20 20 left-hand (low diversity) side. Mechanical 10 10 Engineering In the lower diagrams, interdisciplinarity

0 0 Diversity Diversity 0 1 2 3 increases through Mechanical Engineering, 0 1 2 3

20 RBI RBI 20 Environmental Studies and Chemistry, where 20 20 most articles are in the mid-range of diversity.

10 Environment 10 Biochemistry and Biophysics and Microbiology

0 Diversity 0 Diversity 0 1 2 3 0 1 2 3 are the most multidisciplinary, with a heavier RBI 20 RBI 20 20 20 concentration of articles to the right-hand (high

Chemistry 10 10 diversity) side of the plot. Visual inspection suggests that the highest 0 Diversity 0 Diversity 0 1 2 3 0 1 2 3

RBI 20 RBI 20 impact articles fall in the mid-range – neither 20 20 monodisciplinary nor very multidisciplinary. This Biochemistry & 10 10 Biophysics can be tested by analysing articles in decile bands, based on their Shannon diversity, and

0 Diversity 0 Diversity 0 1 2 3 0 1 2 3 calculating average impact for each band (see RBI 20 RBI 20 20 20 below).

Microbiology 10 10

0 Diversity 0 Diversity 0 1 2 3 0 1 2 3 0 3 0 3 Source: data, Thomson Scientific® Inc; analysis, Evidence Ltd.

Figure 13 shows the outcome of analysing the data in Figure 12 by deciles, across the diversity range. The graph suggests that articles with the highest average rebased citation impact often occur in the mid- range of subject diversity of cited references, but this is not statistically significant. It is clear, however, that there is no fall in impact with increasing diversity or interdisciplinarity.

Figure 13 Average RBI by diversity decile band

4 Average RBI increasing diversity 3

0 Space Science Physics Mechanical Environmental Chemistry Biochemistry & Microbiology Engineering Studies, Biophysics Geography & Development

The articles in each subject category were split into 10 bands of increasing diversity. The average RBI is shown by a grey bar for the articles in each band in each subject. The shapes suggest a very weak tendency towards higher RBI for articles with higher diversity within a subject category.

Source: data, Thomson Scientific® Inc; analysis, Evidence Ltd.

This analysis supports the observations from the correlations and scatter plots that some papers in the mid-diversity range have high impact. There is also no evidence that papers with a high diversity of cited material – which we believe have the characteristics of interdisciplinary papers – have unusually low impact.

4 Presumed interdisciplinarity

In the present study, we have used the spread of a paper’s cited references across subject categories to identify interdisciplinary research. In the original specification, we intended to test our methodology and findings against two presumptive datasets. One was the set of submissions to the 2001 UK Research Assessment Exercise (RAE2001) that was cross-referred between panels. The other was that set of articles which fell into the ‘Multidisciplinary’ category on Thomson Scientific’s databases. On closer examination it became apparent that these datasets were of limited utility for our purposes.

4.1 Multidisciplinary RAE submissions The ‘Refers’ table for RAE2001 contains information about (partial or whole) submissions that were referred, at the request of staff in an institution, to other UoA panels. Some of the referrals applied to one or a few members of staff, or even individual outputs, while others applied to whole research groups. When we analysed the STEM referrals more closely, it became apparent that the majority of them were not to do with interdisciplinarity as such, but were more likely to be because of differences between the organisational structure of institutions and the way in which this related to the category structure of UoAs.

Once these and other referrals not relevant to purpose had been discounted, we were left with a small number of cross-referred items. These represented less than 0.05% of the submitted outputs. We felt that, with so little data to work on, it was not feasible to proceed with our original plan. This had been (i) to evaluate interdisciplinary indices for these outputs using the typology outlined above, and then (ii) to compare the citation impact of cross-referred outputs with the average impact for other outputs with the same UoA and grade.

We note that this very small volume of flagged interdisciplinary cross-referrals is some evidence that most STEM researchers do not, in practice, have a great deal of conflict with the assessment of discipline- based peer review panels. In terms of cost-benefit, it might be deemed that the penalty suffered for submitting interdisciplinary outputs would have to be very significant to justify making an adjustment to an evaluation system for such a small proportion of activity.

4.2 Thomson’s ‘Multidisciplinary’ Thomson Scientific has three different definitions for its ‘Multidisciplinary’ category, one for each of three ® category different Current Contents products. Some interdisciplinary journals are always assigned to the Multidisciplinary category, regardless of which product they are in, but some have different assignments for different products. Within a product, it is the whole journal – rather than individual articles – that gets assigned to a particular category.

It is important to be aware that an article’s category assignment may change from year to year, depending on the citations it has received.

There has been an increase in Multidisciplinary articles per year in the 2006 version of the UK National Citation Report (UKNCR) compared with the 2005 version. We believe this is because: • articles are assigned to categories according to the citations they have received to date; • journal coverage has changed.

This has an effect on rebased impact at the article level. As an example, one article published in Nature (2000) was assigned to the Multidisciplinary category in UKNCR 2005, but to Animal and Plant Science in UKNCR 2006. As a result, the RBI of this particular article decreased from 2.8 to 1.6 because of the difference in baselines for the different subject categories.

To improve our understanding of the effects of Thomson’s categorisation on interdisciplinary articles, we analysed one year’s worth of UK Nature articles. Figure 14 illustrates the spread of articles across categories, the differential citation rates and subject baselines, and the resulting RBI for UK articles published in Nature in 2000.

Figure 14 Average citation impact and frequency of UK Nature articles (2000) by subject category Figure 15 Scatter plot for Multidisciplinary category

500 A verage impact Number of papers 40 category baseline rebased impact 400 30 RBI 100 300

200

10 0 012 100

0 0 C GAS EN CM IMM CEL OG BIL ENV PH MC PH PH AP MB BE AQ EA PL MG AS EX CM EN BIO MU ME SP VE CAR AR GNC P S G P R P H T O C D M S T C U X T G L P B L S

Average impact Average rebased impact Category baseline impact Number of papers

Source: data, Thomson Scientific® Inc; analysis, Evidence Ltd Source: data, Thomson Scientific® Inc; analysis, Evidence Ltd The Multidisciplinary category is mid-range in interdisciplinarity on all three indices when compared with other subject categories. The plot in Figure 15 shows that the correlation between citation diversity and RBI is similar to mid-range subjects such as Chemistry or Mechanical Engineering (Figure 12).

This initial exploration of Thomson’s Multidisciplinary category suggests that its contents do not differ significantly from any other categories in the balance of interdisciplinary articles. This means that the assignment of an article to the Multidisciplinary category does not tell us much about the interdisciplinarity of its cited references. Interdisciplinary articles are to be found in all Thomson categories.

5 Implications for bibliometric assessment and suggestions for further work

If the premise that the relative interdisciplinarity of research should be reflected in the diversity of fields on which it draws is accepted, then there is no indication that interdisciplinary work suffers from a lack of esteem. We found no evidence that articles classified as interdisciplinary by the subject spread of their cited references receive systematically fewer citations than other comparable articles.

For this study, we looked at data drawn from two research-active but rather different universities: different in their volume of research, their diversity of capacity and the strengths of their departments. We found no substantive difference in comparable analyses made between the two institutions, so we feel that the outcomes of this study have some general applicability.

We used three different indices of diversity for the citations made by the source articles from the two institutions. We found no conflicting difference between these indices, which generally showed similar relativities among subject categories. We therefore preferred the Shannon diversity index as our guide because this was more information-rich, albeit less transparent.

Non-parametric correlation analysis revealed that articles with a more diverse range of cited material had a relatively higher rebased citation impact than articles for which a greater share of cited material was within the same journal category as the source article.

We found some limited but consistent indications that the most highly cited articles tended to be those in the middle range of interdisciplinary diversity. Tentatively, we would suggest that these are articles which remain attached to their core discipline but establish relevance in cognate areas. However, further examination is required to validate this.

In conclusion, if articles are indexed on their normalised citation impact, there is no reason to suppose that those which appear to be more interdisciplinary will be in any way systematically disadvantaged.

Before developing policy based on this finding, however, we suggest that our method of indexing interdisciplinarity should be tested against researchers’ perceptions and expert opinion. If this definition of interdisciplinarity proves acceptable, we would conclude that there is no reason to single out interdisciplinary research for differential treatment under a metrics-based system of assessment.

We also draw attention to our exploration of presumptive interdisciplinarity in associated datasets. Our analysis of Thomson’s Multidisciplinary category alerted us to the critical issue of using an appropriate base for normalisation. This was not the focus of the current study, but is an important finding in the context of the general use of bibliometrics for research assessment.

What therefore remains crucial to the successful use of bibliometric data for assessment purposes is the selection of an appropriate field-level baseline against which citation counts are normalised. This applies to all research outputs, but especially to those that fall into one or more subject category. It is particularly important in cases where the baseline citation rates of the possible categories are significantly different from each other.

6 Subject category assignment

There are 106 Thomson Scientific Current Contents® subject categories in total. Each article is assigned to one or more subject category according to the journal in which it is published. Exceptions to this are the journals Nature, Science and Proceedings of the National Academy of Sciences (PNAS), where articles are individually assigned to subject categories according to the subject categories of the articles they cite and are subsequently cited by. This means that an article’s category assignment can change over time. Within a Current Contents® product, a journal is mapped to one or more of the subject categories, and every article within that journal is assigned to those categories. Journals can appear in more than one product. For example, the journal Plant Physiology is in Current Contents®/Agriculture, Biology and Environmental Sciences – assigned to the category Plant Sciences – and also in the Current Contents®/ Life Sciences product where it is assigned to the category Animal and Plant Science. For the current study, we needed a method of dealing with multi-category journals that avoided double- counting. We drew up a one-to-one mapping between journals and subject categories. This ensured that cited and citing references could be assigned unambiguously to single subject categories, even where information on subsequent citations was lacking. Journals that had previously been assigned by Thomson to more than one subject category were re-assigned to the most frequently occurring category for articles in that journal, based on the UK National Citation Report for 2006. This new mapping was then used to re-assign source articles so that all the articles from each journal were assigned to the same category. (Under this methodology, all articles from Nature and Science were assigned to the Multidisciplinary category, and all articles from PNAS were assigned to Biochemistry and Biophysics.) The 88 STEM (plus Mathematics) subject category titles and their abbreviated codes are listed below.

Thomson Scientific® Inc Current Contents® 2006 subject categories, alphabetically by category code

Code Curr ent Contents(R) Subject Category Code Current Contents(R) Subject Category Code Current Contents(R) Subject

A/A Agriculture/Agronomy ENT Entomology/Pest Control OGS Medical Research, Organs & S AER Aerospace Engineering ENV Environment/Ecology ONC Oncology AIC Anesthesia & Intensive Care EXP Experimental Biology OPH Ophthalmology AN Animal & Plant Science F Food Science/Nutrition ORG Organic Chemistry/Polymer S APP Applied Physics/Condensed Matter/Materials Science GAS Gastroenterology & Hepatology ORT Orthopedics, Rehabilitation & AQU Aquatic Sciences GEO Environmental Studies, Geography & Development OTO Otolaryngology ARA AI, Robotics & Automatic Control GNC General & Internal Medicine PED Pediatrics AS Animal Sciences GNE Engineering Management/General PHC Physical Chemistry/Chemical BEH Neurosciences & Behavior GPM Geological, Petroleum & Mining Engineering PHM Pharmacology & Toxicology BIL Biochemistry & Biophysics HEM Hematology PHS Physics BIO Biology HLT Health Care Sciences & Services PL Plant Sciences BTC Biotechnology & Applied Microbiology I/M Instrumentation & Measurement PMC Pharmacology/Toxicology CAR Cardiovascular & Respiratory Systems IMM Immunology PSI Psychiatry CEL Cell & Developmental Biology INC Inorganic & Nuclear Chemistry PSL Physiology CGX Oncogenesis & Cancer Research INF Clinical Immunology & Infectious Disease PSO Psychology CIV Civil Engineering IST Information Technology & Communications Systems PSY Clinical Psychology & Psychia CMA Agricultural Chemistry LIB Library & Information Sciences PUB Public Health & Health Care S CME Chemical Engineering MBG Molecular Biology & Genetics RAD Radiology, Nuclear Medicine & CML Chemistry & Analysis MCB Microbiology REH Rehabilitation CMP Chemistry MEC Mechanical Engineering REP Reproductive Medicine CSE Computer Science & Engineering MED Research/Laboratory Medicine & Medical Technology RHU Rheumatology CVS Cardiovascular & Hematology Research MET Metallurgy S/I Social Work & Social Policy DEN Dentistry/Oral Surgery & Medicine MGN Medical Research, General Topics SIA Spectroscopy/Instrumentation DER Dermatology MTH Mathematics SOC Environmental Medicine & Pub DGX Medical Research, Diagnosis & Treatment MTR Materials Science & Engineering SP Space Science EAR Earth Sciences MUL Multidisciplinary SUR Surgery EEE Environmental Engineering & Energy NCL Nuclear Engineering URO Urology & Nephrology EL Electrical and Electronics Engineering NEU Neurology VET Veterinary Medicine/Animal He EMA Engineering Mathematics NUT Endocrinology, Metabolism & Nutrition END Endocrinology, Nutrition & Metabolism O/A Optics & Acoustics

Annex A Units of Assessment clustered on journal frequency

Czekanowski similarity index 0.00 0.16 0.32 0.48 0.64 1996 UoA Clinical Lab Sci... Hosp. based... Medical Com. based... Other stud. Pharmacy Biochemistry Biol. sciences Pre-clin. stud. Physiology Pharmacology Bio-Med Anatomy Veterinary sci. Clin. Dentistry Food sci... Agriculture Earth sci. Environ. sci. Environment Geography Archeology Mineral/mining... Chemistry Metallurgy... Physics Physical Chem. eng. Computer sci. Gen. Eng. Mechanical eng... Engineering Electrical eng... Civil eng. Pure maths. Applied maths. Maths Statistical res... Nursing` Sports related... Psychology Education Politics... Social policy... Sociology Social work Communication... Built environ. Town/country... Social Economics... Business... stud. Accountancy Law Library and info... Anthropology Asian stud. This tree diagram illustrates similarity Middle east... Theology... in the frequency with which journals American stud. Iberian... were submitted to RAE1996 European stud. French German, Dutch... Arts & English History Italian Humanities Russian... Linguistics Classics... Philosophy History of Art... Art and Design Drama, Dance... Music Source: Adams et al 1998 (see footnote 4) Celtic stud.