Report to the Higher Education Funding Council for England Bibliometric analysis of interdisciplinary research Jonathan Adams, Louise Jackson, Stuart Marshall November 2007 Contact details The reporting organisation is: Evidence Ltd 103 Clarendon Road, Leeds LS2 9DF T/ 0113 384 5680 F/ 0113 384 5874 E/ [email protected] Evidence Ltd is registered in England, Company no 4036650, VAT registration 758467185 http://www.evidence.co.uk Bibliometric analysis of interdisciplinary research Contents Executive summary ………………………………………………………………………………………...1 1 Introduction..........................................................................................................................3 2 Defining and indexing interdisciplinarity ..............................................................................4 3 Differential citation impact .................................................................................................15 4 Presumed interdisciplinarity...............................................................................................23 5 Implications for bibliometric assessment and suggestions for further work ......................26 6 Subject category assignment ............................................................................................28 Annex A Units of Assessment clustered on journal frequency.......................................................31 Bibliometric analysis of interdisciplinary research Executive summary The research community has from time to time argued for separate assessment of interdisciplinary research, but has not found an objective definition of such work. The notion is that interdisciplinary work is treated and valued differently from other work. It is hypothesised that, being marginal to core subjects, it would be systematically cited less often. If so, this might be a problem for a metrics-based system of assessment where peer judgments that distinguish work of this nature would be absent. In this study, we have explored this notion further by: • developing an objective description of research output according to an index of its interdisciplinarity; • testing whether there is differential citation impact across the range of interdisciplinarity; • evaluating our findings against presumed interdisciplinary outputs submitted to the Research Assessment Exercise (RAE) and outputs assigned to Thomson Scientific® Inc’s ‘Multidisciplinary’ category; • considering options for handling interdisciplinary work within the proposed system of bibliometric- based indicators. We collated article records supplied by Thomson Scientific to create databases of publications – source articles – for two large research-intensive UK universities. We analysed the work (other publication references) that cites and is cited by these source articles. To define whether a citation link to or from a source article was within or external to the article’s discipline, we used the Thomson journal categorisation. We developed measures of interdisciplinarity by indexing the subject spread of the references cited by the source articles. We found that: • there was a continuum of interdisciplinarity at article level within journal categories; • the categories themselves fitted a broad but not categorical typology ranging from mono- disciplinary to multidisciplinary. 1 Using the citing references, within each journal subject category we analysed the citation count (impact) of each source article. We then compared the interdisciplinarity of each article against its rebased citation impact (normalised for year of publication and subject category). We found: • no tendency for the most interdisciplinary articles to be less frequently cited on any consistent basis within or across categories; • a weak tendency for the articles with highest citation impact to be neither very monodisciplinary nor very multidisciplinary in terms of their cited references. Our comparative analysis of presumptive interdisciplinary datasets provided only limited further information. Our findings have implications for the assessment of interdisciplinary research: • There is no evidence to support an explicit intervention regarding articles that refer to work across a broader range of disciplines. They do not receive systematically fewer citations than more monodisciplinary publications. • If impact is to be normalised at any categorical level, there is evidence that careful attention must be paid to the choice of baseline category, especially for articles in journals that are assigned to multiple categories. The Multidisciplinary category used by Thomson does not, however, necessarily contain articles that cross more subject boundaries than research in other Thomson categories. • Since our index of interdisciplinarity may not satisfy all parties, the methodology should ideally be evaluated in due course through consultation with informed researchers. In summary, there is no strong case on the basis of this analysis for research outputs to be treated differently for the purposes of bibliometric assessment on the grounds of interdisciplinarity. It is important, however, to exercise care in choosing the appropriate field against which to normalise the citation rates of more interdisciplinary outputs. 2 1 Introduction This study has provided a preliminary exploration of the notion that interdisciplinary work might be at a disadvantage in research evaluation because it is systematically judged in different terms to core disciplinary activity. Specifically, the project considered whether published work which is more or less interdisciplinary might receive consistently different rates of citations from other publications. Our approach was, first, to develop an objective methodology for indexing interdisciplinarity. Secondly, we tested for differential citation impact among articles with higher and lower levels of indexed interdisciplinarity. The interdisciplinary nature of an article was indexed by analysing the diversity of the subject categories from which it drew its cited references. Within each subject category, we found that articles varied across a continuum from monodisciplinary to multidisciplinary, with no obvious discontinuities or clustering. Across subject categories, we found that some tended to be more generally multidisciplinary than others. Thus it was possible to characterise the subject categories as low, medium or high diversity. When we compared interdisciplinarity with citation impact, we found no evidence that interdisciplinary work had consistently lower citation impact. Indeed, there was a tendency for many highly cited articles to be mid-range in terms of interdisciplinarity, and in most subjects there was a weak or very weak positive correlation between interdisciplinarity and citation impact. That is to say, publications which referred to articles across a broad range of subject categories were (marginally) more likely to attract a higher number of citations. While our approach to indexing interdisciplinarity has been based on widely accepted methodologies, we suggest that their validity should be further tested against the perceptions of researchers. The remainder of this report is split into four sections: Section 2 Defining and indexing interdisciplinarity Section 3 Differential citation impact Section 4 Presumed interdisciplinarity Section 5 Implications for bibliometric assessment and suggestions for further work. 3 2 Defining and indexing interdisciplinarity This section describes how we indexed the interdisciplinarity of research outputs. There is no clear or generally accepted definition of interdisciplinarity; it is a subjective term. Many researchers believe that they are working in an interdisciplinary area. This seems possible, given that it is widely agreed that good and innovative research often draws on discoveries at the margins of more than one discipline.1 But there are no criteria for judging whether this is true. This is problematic because some researchers also believe that interdisciplinary research, working outside the core of established disciplines, is at risk of being marginalised in research evaluation and assessment. A desirable outcome might therefore be lost if the best researchers were to shun interdisciplinarity because of this perception. How can we determine whether such a risk exists and take steps to avoid it? Rather than relying on the opinion of the researcher or an expert group, it is preferable for assessment purposes to identify an objective (measurable) characteristic that marks out interdisciplinary research. Our study began, therefore, by developing an index of interdisciplinarity. 2.1 Methodology Research outputs can be characterised at different levels of aggregation, from individual outputs to researchers and research groups or departments. For this study, we focused on the article level. Our aim was to characterise the (relative) interdisciplinarity of an individual article so that this could be compared against the article’s citation impact. What characteristics of an article might define its interdisciplinary nature? One approach is to look at the previous work referred to by the article in question. Arguably, work claiming interdisciplinary influence would draw on, or cite, work from a greater than average range of fields. Similarly, it would be expected that work claiming interdisciplinary utility would be cited by journals in a greater than average range of fields. 1 See for example the Government’s Science and innovation investment framework,