Interdisciplinary Research Has Consistently Lower Funding Success Lindell Bromham1, Russell Dinnage1 & Xia Hua1

LETTER doi:10.1038/nature18315

Interdisciplinary research has consistently lower funding success Lindell Bromham1, Russell Dinnage1 & Xia Hua1

Interdisciplinary research is widely considered a hothouse for and ‘transdisciplinary’ vary widely11, and researchers will differ innovation, and the only plausible approach to complex problems in their inclination to label their research as ‘interdisciplinary’2, such as climate change1,2. One barrier to interdisciplinary research particularly if they perceive that identification as interdisciplinary is the widespread perception that interdisciplinary projects influences funding outcomes15. Because bibliometric analyses are pri- are less likely to be funded than those with a narrower focus3,4. marily applied to publications11, they may be of limited applicability However, this commonly held belief has been difficult to evaluate in assessing funding proposals, where the outputs have not yet been objectively, partly because of lack of a comparable, quantitative published, and citations may not be in an analysable format. The lack of measure of degree of interdisciplinarity that can be applied to clear definitions and objective analyses is an impediment to evaluating funding application data1. Here we compare the degree to which the relative success of interdisciplinary proposals1. Conflicting findings research proposals span disparate fields by using a biodiversity have been reported using a range of approaches4, and most studies metric that captures the relative representation of different fields of funding success of interdisciplinary research have selected only a (balance) and their degree of difference (disparity). The Australian sample of proposals for evaluation15,16. What is needed is a measure Research Council’s Discovery Programme provides an ideal test of the degree to which a proposal spans many different disciplines, case, because a single annual nationwide competitive grants independent of use of words such as ‘interdisciplinarity’ and without scheme covers fundamental research in all disciplines, including relying on cited publication data. arts, humanities and sciences. Using data on all 18,476 proposals Although no single metric will capture all salient aspects of interdis- submitted to the scheme over 5 consecutive years, including ciplinarity, developing a simple measure of the disciplinary spread of successful and unsuccessful applications, we show that the greater research proposals does provide a tractable way to compare the relative the degree of interdisciplinarity, the lower the probability of being success of proposals having a narrow disciplinary focus with those with funded. The negative impact of interdisciplinarity is significant a broader research programme17. To this end, we use information sup- even when number of collaborators, primary research field and plied on funding applications to score each proposal on the disparity type of institution are taken into account. This is the first broad- and balance of the component disciplines12. We base our analysis on scale quantitative assessment of success rates of interdisciplinary methods established in evolutionary biology to account for relatedness research proposals. The interdisciplinary distance metric allows between biological lineages, but instead of using an evolutionary tree efficient evaluation of trends in research funding, and could be used (phylogeny), we use a hierarchical classification of research fields. This to identify proposals that require assessment strategies appropriate metric can be applied to any funding scheme where multiple discipline to interdisciplinary research5. categories can be selected by applicants or identified from proposal The 5th Annual Meeting of the Global Research Council in New documents7. Delhi in May 2016 focused on interdisciplinarity as one of its main We calculated this interdisciplinary distance (IDD) metric for all topics of concern, reflecting increasing interest in research that breaks proposals submitted to the Australian Research Council Discovery free of traditional discipline boundaries, and the growing concern that Programme between 2010 and 2014 (Supplementary Table 1). This interdisciplinary research is not adequately supported under current national competitive grants scheme funds fundamental research in funding structures. Funding agencies play a key role in shaping inter- all academic fields, receiving approximately 3,500 proposals in each disciplinary research3, with both positive influence, such as dedicated annual funding call, with success rates being around 15–20% of pro- programmes for interdisciplinary projects, and negative impacts, as posals (Extended Data Table 1). Our analysis is unique in including all perceived biases can discourage submission of interdisciplinary pro- submitted proposals, both successful and unsuccessful, whereas most posals to open funding calls. This leads to the ‘paradox of interdisci- analyses are restricted to the published lists of funded proposals7,18 or plinarity’: interdisciplinary research is often encouraged at policy level to samples of case studies3. but poorly rewarded by funding instruments4. There is a clear need to Every application must nominate at least one of a defined set of test the widely held belief that interdisciplinary proposals fare poorly in 1,238 Field of Research codes, assigning a percentage weighting to competitive funding rounds: confirmation could prompt examination each code selected. Field of Research codes are grouped into related of evaluation strategies for interdisciplinary projects, while rejection of disciplines: for example, the Division ‘06 Biological Sciences’ contains this claim might encourage more interdisciplinary proposals. nine groups, including ‘0603 Evolutionary Biology’, which contains Critical to evaluation of current practice is the ability to compare 12 Fields including ‘060309 Phylogeny and Comparative Analysis’ levels of interdisciplinarity of research projects to track trends, evaluate (Supplementary Fig. 1). Because we wish to capture the disciplinary outputs and compare success rates6,7. Measures of interdisciplinarity breadth of proposals, we need to measure not only the number and rel- have typically relied on textual references, detecting use of words such as ative representation of disciplines selected, but also how disparate those ‘interdisciplinarity’8, or bibliometric analysis, tracking patterns of author research fields are. For example, we want to score a project that involves affiliation6 or citations within publications9–14. But these approaches collaboration between biologists and artists as more interdisciplinary are limited in use for evaluating funding applications. Interpretation than one between biochemists and geneticists. Just as many biodiver- of the terms ‘multidisciplinary’, ‘cross-disciplinary’, ‘interdisciplinary’ sity metrics use a phylogeny to measure disparity of species, research

1Research School of Biology, Australian National University, 116 Daley Road, Canberra 0200, Australia.

Successful proposals Extended Data Table 2). The response variable was a binary vector with two states: recommended for funding (1) or not recommended for funding (0). We provide details of the analysis and results in the Supplementary Information. We find that IDD is consistently negatively correlated with funding success (slope = −0.40, P = 1.1 × 10−11; Fig. 1), independent of year of application, number of research codes selected and primary research 0.25 field. Nearly all research fields have reduced funding success with increasing interdisciplinarity (Fig. 2 and Extended Data Table 3). If the association between IDD and funding success was largely a matter of averaging success rates of the component disciplines—so that less successful fields benefit from collaboration with more successful fields 0.2 but more successful fields have their success rates reduced through collaboration—then we would expect many points both above and below y = 0 in Fig. 2. However, most fields have negative values of y, suggesting that proposals with high IDD are expected to have lower

oportion successfully funded success rates than those with low IDD in most research divisions. Pr 0.15 We conducted additional analyses using metrics that reflected only Unsuccessful proposals variety (number of codes) or balance (evenness) of disciplines (details in Supplementary Information), which demonstrated that it is both disparity and balance between disciplines that influence chance of funding success, justifying the use of the IDD metric which captures both of these aspects of interdisciplinarity. We also searched for relevant 0.00 0.25 0.50 0.75 1.00 keywords in proposal titles and summaries, such as ‘interdisciplinary’, IDD ‘multidisciplinary’, ‘transdisciplinary’ and ‘cross-disciplinary’. We found Figure 1 | Relationship between funding success and IDD score. that proposals with these keywords also had higher IDD measures and The central black line is the regression line (success = logit−1[− 1.35506 + lower success rates, demonstrating that the metric-based approach ech- − 0.40268 × IDD]); grey area represents the confidence intervals oes text-based analysis, yet with greater power to detect differences in (see Supplementary Information for details). The horizontal lines around funding rates (see Supplementary Information). the regression line represent the mean success rate for each of 10 ‘bins’ of IDD reflects the interdisciplinarity of the project, not the inclusion of IDD values of width 0.1. Each horizontal line’s y value represents the mean practitioners from different disciplines, because a single researcher can success rate of proposals whose IDD values fall within a ‘bin’ defined by 11 the ends of the horizontal line. The bars at the top and bottom of the devise a project that spans different disciplinary traditions . To test the figure indicate the number of proposals for each IDD score, with darker influence of collaboration, we added the number of named chief inves- lines corresponding to more proposals. Fitted line based on a generalized tigators to the analysis. We find that proposals with more chief investi- linear mixed model as described in the Supplementary Information gators have slightly higher success rates (slope = 0.03, P = 0.003), across (z = − 6.789; P = 1.13 × 10−11; n = 18,476). all research fields, independently of any link between number of collaborators and interdisciplinarity (Supplementary Table 2). Although the fields can be arranged as a dendrogram, where fields of enquiry that relationship between number of chief investigators and IDD is positive, are more similar are more closely connected to each other than more the effect is small (Spearman’s ρ = 0.09), suggesting that the number distant fields. Our approach is not restricted to funding programmes of participants is not strongly associated with the interdisciplinarity of that use hierarchically structured research codes, but can be applied the project proposal. to any research field identifiers by using patterns of co-occurrence to Because of the perceived negative association between funding define clusters of similar fields7. success and interdisciplinarity, interdisciplinary projects are often We use the phylogenetic species evenness 19 metric to measure IDD. regarded as high-risk proposals4. Do institutions with higher rates This metric was designed to compare biodiversity between samples (for of funding success submit more interdisciplinary proposals (using example, between conservation areas), incorporating both evenness of higher success rates to support risky proposals) or fewer (because species representation in the biota and relatedness between species20. higher success rates arise from narrowly focused research)? We find IDD reflects both the relative contribution of disciplines within a pro- that overall funding success rates varied between institutions, with posal (balance) and scores collaborations between distantly related significantly higher funding success rates in leading research-intensive fields more highly than those between more closely related disciplines universities (Extended Data Table 2)18. Differences in IDD between (disparity)21. The metric is standardized so it falls between 0 (single institutions were very small (R2 = 0.001) and the negative relationship disciplinary) and 1 (maximum disparity with even representation), between IDD and success rate was significant when institution was allowing direct comparison between proposals (Extended Data Fig. 1). taken into account (slope = − 0.39, P = 7.6 × 10−11; Supplementary Patterns of co-occurrence of field identifiers (for example, research Table 2). This suggests that the negative relationship between inter- codes or key words) can be used to generate a hierarchy of discipline disciplinarity and funding success is not due to institutions with relationships (see Supplementary Information). Basing this hier- high funding success submitting more narrowly focused proposals archy on observed patterns of collaboration would rank proposals (Extended Data Fig. 2). with respect to the relative novelty of the disciplinary combinations Why do interdisciplinary proposals have lower funding success rates? proposed22. It is widely believed that grant evaluation processes are biased against The Australian Research Council provided de-identified data on interdisciplinary projects, because proposals may be assigned to a panel all applications to the Discovery Programme for five annual funding or reviewers who are ill-equipped to evaluate all parts of the project23, rounds. We used a generalized linear mixed model to ask whether the while more narrowly focused proposals may be better matched to IDD score of grant proposals is associated with funding success. We assessor expertise4. Proposals that fit within a well-defined discipline included as variables in the analysis the year of application, number of may be more easily explained and justified, whereas the novelty of Field of Research codes selected per proposal, number of named chief combinations of different perspectives may be more difficult to investigators and institution (grouped into higher education networks: explain22 or result in less-focused proposals24.

1.5 Number of grant

riting submissions

200

1,000 1.0 2,500 eative Arts and W 1 Education

0.5 Medical and Health Sciences

Agricultural and Veterinary Sciences Studies in Cr

Earth Sciences

Built Environment and Design Law and Legal Studies 0.0 Commerce, Management, Language, Communication and Culture Tourism & Services Chemical Sciences Biological Sciences Engineering Philosophy and Religious Studies Psychology and Cognitive Sciences –0.5 Technology Information and Computing Sciences Mathematical Sciences Change in proportion successfully funded

due to interdisciplinarity from IDD = 0 History and Archaeology

–1.0 Physical Sciences nmental Sciences Economics ro –1.5 Envi Studies in Human Society

0.0 0.1 0.2 0.30.4 Proportion successfully funded Figure 2 | Relationship between interdisciplinarity and funding success with a single primary Field of Research code (IDD = 0) for each division, by research division. The x axis gives the success rate of a research so that y = − 0.5 indicates that the logit success rates of proposals with division as the proportion of successfully funded proposals in that research IDD = 1 is 0.5 lower than the logit success rate of proposals with IDD = 0. division. The error bars along the x axis show the confidence interval of The standard error of the predicted difference (the error bar along the success rate approximated by Wilson interval. The number of proposals in y axis) is the square root of the sum of the squared standard error of each research division is given in Extended Data Table 1. The y axis gives the IDD coefficient and that of the interaction coefficient. The average the average predicted difference in logit success rates between proposals difference of each research division and its error bar are predicted by the with maximum interdisciplinarity (IDD = 1) compared with proposals generalized linear mixed model in Extended Data Table 3.

While interdisciplinary research can have considerable benefits, of previous applications (see, for example, Extended Data Table 1), it can also incur substantial costs, owing to the need to invest significant subjectively (based on experience) or by any other relevant means7. time in building collaborative relationships, developing a shared lan- Such analyses will bring much needed clarity to determining whether guage and honing a common perspective from disparate viewpoints25. interdisciplinary research programmes are being adequately supported The outputs of interdisciplinary projects may be fewer and of differ- under current funding models. In addition to enabling assessment of ent kinds to projects with a narrower disciplinary focus26,27. Research biases in success rates, the IDD metric could provide a way of iden- evaluation systems with a narrow range of measures of success—for tifying highly interdisciplinary proposals that might require special example, number of primary research publications in peer-reviewed evaluation strategies, such as seeking reviewers who have experience journals—may disadvantage interdisciplinary proposals where some in research spanning multiple fields. key outputs are less easily measured, such as the establishment of 23 Online Content Methods, along with any additional Extended Data display items and collaborative networks or data-sharing agreements . While some inter- Source Data, are available in the online version of the paper; references unique to disciplinary studies produce significant advances, the average quality these sections appear only in the online paper. of interdisciplinary proposals may not be the same as more narrowly focused research. Studies of the long-term scholarly impact of interdis- Received 10 December 2015; accepted 11 May 2016. ciplinary research have had mixed results: whereas some suggest greater 1. Rylance, R. Grant giving: global funders to focus on interdisciplinarity. benefits, others find no support for higher impact of interdisciplinary Nature 525, 313–315 (2015). 12–14 research . 2. Ledford, H. How to solve the world’s biggest problems. Nature 525, 308–311 Whatever the cause of the correlation, our result confirms the long- (2015). 3. Lyall, C., Bruce, A., Marsden, W. & Meagher, L. The role of funding agencies in held belief that interdisciplinary proposals have lower funding success creating interdisciplinary knowledge. Sci. Public Policy 40, 62–71 (2013). rates, providing the basis for further investigation into the development 4. Woelert, P. & Millar, V. The “paradox of interdisciplinarity” in Australian and evaluation of interdisciplinary research. Although IDD does not research governance. High. Educ. 66, 755–767 (2013). 5. Committee on Facilitating Interdisciplinary Research. Facilitating capture all key aspects of interdisciplinary research, it does provide a Interdisciplinary Research (National Academy of Sciences, National Academy of tractable and adaptable way of comparing interdisciplinarity between Engineering, Institute of Medicine, 2004). proposals and tracking trends in application rates and funding success. 6. Langfeldt, L. The policy challenges of peer review: managing bias, confict of interests and interdisciplinary assessments. Res. Eval. 15, 31–41 IDD can be applied to any funding programme where research fields (2006). are identified. Relatedness between disciplines can be defined a priori 7. Nichols, L. G. A topic model approach to measuring interdisciplinarity at the (for example, Field of Research codes), through clustering analysis National Science Foundation. Scientometrics 100, 741–754 (2014).

8. Van Noorden, R. Interdisciplinary research by the numbers. Nature 525, 23. Porter, A. L., Garner, J. & Crowl, T. Research coordination networks: evidence of 306–307 (2015). the relationship between funded interdisciplinary networking and scholarly 9. Porter, A. & Rafols, I. Is science becoming more interdisciplinary? Measuring impact. Bioscience 62, 282–288 (2012). and mapping six research felds over time. Scientometrics 81, 719–745 24. Boix Mansilla, V., Feller, I. & Gardner, H. Quality assessment in interdisciplinary (2009). research and education. Res. Eval. 15, 69–74 (2006). 10. Porter, A. L., Roessner, J. D., Cohen, A. S. & Perreault, M. Interdisciplinary 25. Haythornthwaite, C., Lunsford, K. J., Bowker, G. C. & Bruce, B. C. in New research: meaning, metrics and nurture. Res. Eval. 15, 187–195 (2006). Infrastructures for Science Knowledge Production (ed. Hine, C.) 143–166 11. Wagner, C. S. et al. Approaches to understanding and measuring (Idea Group, 2006). interdisciplinary scientifc research (IDR): a review of the literature. 26. Laudel, G. Conclave in the Tower of Babel: how peers review interdisciplinary J. Informetrics 5, 14–26 (2011). research proposals. Res. Eval. 15, 57–68 (2006). 12. Yegros-Yegros, A., Rafols, I. & D’Este, P. Does interdisciplinary research lead to 27. Goring, S. J. et al. Improving the culture of interdisciplinary collaboration in higher citation impact? The diferent efect of proximal and distal ecology by expanding measures of success. Front. Ecol. Environ 12, 39–47 (2014). interdisciplinarity. PLoS ONE 10, e0135095 (2015). 13. Wang, J., Thijs, B. & Glänzel, W. Interdisciplinarity and impact: distinct efects Supplementary Information is available in the online version of the paper. of variety, balance, and disparity. PLoS ONE 10, e0127298 (2015). 14. Shi, X., Adamic, L. A., Tseng, B. L. & Clarkson, G. S. The impact of boundary Acknowledgements We thank the Australian Research Council for providing spanning scholarly publications and patents. PLoS ONE 4, e6547 (2009). de-identified application data for analysis, and for their commitment to 15. Huutoniemi, K., Klein, J. T., Bruun, H. & Hukkinen, J. Analyzing transparency and improvement of research proposal assessment. We are interdisciplinarity: typology and indicators. Res. Policy 39, 79–88 (2010). grateful to A. Byrne for his feedback and encouragement. We also thank 16. Bruun, H., Hukkinen, J., Huutoniemi, K. & Klein, J. T. Promoting Interdisciplinary M. Jennions for feedback, and G. Bammer, J. Bennett and the participants of the Research: The Case of the Academy of Finland (The Academy of Finland, 2005). workshop on Interdisciplinary Research: Evaluating and Rewarding High-Quality 17. Bammer, G. Strengthening Interdisciplinary Research: What It Is, What It Does, Projects held at the University of New South Wales in August 2015. How It Does It and How It Is Supported (Australian Council of Learned Academies, 2012). Author Contributions All authors contributed equally to this work. L.B. 18. Ma, A., Mondragón, R. J. & Latora, V. Anatomy of funded research in science. conceived the project and wrote the paper; R.D. and X.H. designed, conducted Proc. Natl Acad. Sci. USA 112, 14760–14765 (2015). and interpreted the analyses. 19. Helmus, M. R., Bland, T. J., Williams, C. K. & Ives, A. R. Phylogenetic measures of biodiversity. Am. Nat. 169, E68–E83 (2007). Author Information Reprints and permissions information is available 20. Cadotte, M. W. et al. Phylogenetic diversity metrics for ecological communities: at www.nature.com/reprints. The authors declare no competing financial integrating species richness, abundance and evolutionary history. Ecol. Lett. interests. Readers are welcome to comment on the online version of the 13, 96–105 (2010). paper. Correspondence and requests for materials should be addressed to 21. Stirling, A. A general framework for analysing diversity in science, technology L.B. ([email protected]). and society. J. R. Soc. Interface 4, 707–719 (2007). 22. Uzzi, B., Mukherjee, S., Stringer, M. & Jones, B. Atypical combinations and Reviewer Information Nature thanks L. Amaral, M. Helmus and the other scientifc impact. Science 342, 468–472 (2013). anonymous reviewer(s) for their contribution to the peer review of this work.

Extended Data Figure 1 | Comparison of observed distribution of IDD generated by random sampling of Field of Research codes conditional scores to a null distribution. a, Distribution of IDD scores for 18,476 on the observed frequencies of number of selected codes and percentage proposals to the Australian Research Council Discovery Programme, allocations. pooled over 5 years (2010–2014). b, Null distribution of IDD scores

Extended Data Figure 2 | Distribution of IDD scores by institutional more proposals to the Australian Research Council Discovery Programme networks. See Extended Data Table 2 for the membership of research and have higher funding success rates, but the overall patterns of networks. The research-intensive Group of Eight (Go8) universities submit interdisciplinarity scores and success rates are similar across institutions.

Extended Data Table 1 | Summary of proposals submitted to Australian Research Council Discovery Programme between 2010 and 2014

The number of proposals with a primary identifcation to each Division in 5 years of pooled applications to the Australian Research Council Discovery Programme, with the percentage recommended for funding (% success), median interdisciplinary distance (IDD) and median number of six-digit FOR codes selected per application (Median codes). Divisions (frst two digits of the FOR codes) have been clustered into Domains, as described in the Supplementary Information. Note that medical research is predominantly funded through a diferent scheme, as is research in collaboration with industry partners.

Extended Data Table 2 | Institutional networks

Australian higher education institutions grouped by network, with the number proposals submitted to the Australian Research Council Discovery Programme over 5 years (2010–2014), the average percentage success rate (percentages that were recommended for funding) and the median IDD score of all proposals.

Extended Data Table 3 | Effect size of interdisciplinarity on funding success in each division

The generalized linear mixed model used to predict success rate has IDD, division and their interaction as fxed variables and year as a random variable. ‘Mathematical Sciences’ is used as the reference category for other divisions, so its coefcient and interaction are zeros. Coefcients show diferences in success rates of other divisions compared with Mathematical Sciences. Interaction shows diferences in the efect of interdisciplinarity on success rates of other divisions compared with Mathematical Sciences. The coefcient of IDD = − 0.68** . A likelihood ratio test suggests that 2 −5 + including IDD as a fxed variable signifcantly increases model ft to the data (χ22 = 57.94, P = 4.5 × 10 ). ** * P = 0; ** P = 0.001; *P = 0.01; P = 0.05. Coef, coefcient; Int’n, interaction; Y, year; Wtd avg, weighted average.