Lyall, Catherine, Ann Bruce, Joyce Tait, and Laura Meagher. "Assessing the Route: Evaluating interdisciplinary proposals, programmes and publications." Interdisciplinary Research Journeys: Practical Strategies for Capturing Creativity. London: Bloomsbury Academic, 2011. 137–163. Bloomsbury Collections. Web. 1 Oct. 2021. <http://dx.doi.org/10.5040/9781849661782.ch-007>. Downloaded from Bloomsbury Collections, www.bloomsburycollections.com, 1 October 2021, 20:36 UTC. Copyright © Catherine Lyall, Ann Bruce, Joyce Tait and Laura Meagher 2011. You may share this work for non-commercial purposes only, provided you give attribution to the copyright holder and the publisher, and provide a link to the Creative Commons licence. 7 Assessing the Route Evaluating interdisciplinary proposals, programmes and publications Introduction Improved evaluation criteria and processes are the key to achieving a more sta- ble and consistent role for interdisciplinary initiatives of various kinds within academic and research-based organizations, and for improving the intellectual status of interdisciplinary research. As noted in chapter 2, sensitive and appro- priate evaluation of interdisciplinary research can also play a role in delivering improved value for money for the investments being made in this area. Individuals, evaluation panels, funding agency staff and university lead- ers are increasingly placed in positions where they are expected to judge interdisciplinary work or plans and they often struggle to fi nd defensible ways of differentiating levels of quality in interdisciplinary projects and their out- comes. Since the fl ipside of interdisciplinarity’s ambitions and vision is an increase in risk, evaluators will be aware of the need to behave accountably but perhaps also uncertain about how to achieve impartial balance and fairness in evaluations. Likewise, for researchers developing interdisciplinary proposals or attempting to publish interdisciplinary work, uncertainty about evaluation processes can leave them feeling disadvantaged. Evaluation of interdisciplinarity occurs in a variety of situations. The crite- ria appropriate to evaluation of academically oriented interdisciplinary research may often be different from problem-focused projects and programmes. Within each of these categories, the range of issues to be considered will differ according to the context – the type and scale of a project or programme, the organizational setting, the kinds of actors involved or the kind of publication being reviewed (Feller 2006). This chapter is written from the perspective of the evaluator, which should resonate with research funders and university managers, but we also consider how interdisciplinary researchers and research managers themselves can benefi t from a better understanding of evaluation processes and use these insights to develop proposals that are more likely to succeed, or to write papers that are more likely to be published, and thus advance interdisciplinary careers. This chapter discusses challenges inherent in the evaluation of interdisci- plinarity, as perceived by numerous researchers who have published on this 137 BOOK.indb 137 02/06/11 7:24 PM 138 INTERDISCIPLINARY RESEARCH JOURNEYS subject, supplemented by some of the authors’ own experiences. In particular, we highlight two key aspects of evaluation: 1 the quality of interdisciplinary proposals and research outcomes; and 2 the quality of the evaluation processes themselves. We make some suggestions as to how different types of evaluation of inter- disciplinarity might be approached: ● evaluation of interdisciplinary research proposals from individuals or small teams, including postgraduate or postdoctoral research; ● evaluation of major interdisciplinary programmes, centres and funding schemes; ● evaluation within institutions; ● evaluation of interdisciplinary submissions for publication; ● self-evaluation. Some of these evaluation occasions come before the fact (ex ante) so that doors are opened, or not, for interdisciplinary initiatives. Some arise after interdis- ciplinary work is done (ex post), to check on the effectiveness or impact of ventures such as interdisciplinary programmes or entire funding schemes pro- moting interdisciplinarity. Some occasions may occur sporadically, as when, for example, a university develops a research strategy and surveys its own work when placing relative weighting on interdisciplinarity for its future. None of these evaluations takes place in a vacuum and approaches to evalu- ation of one type will have implications for others. However, Feller suggests that, even though all interdisciplinary evaluations will share some core val- ues such as commitment to new knowledge, different contexts will respond at different rates to interdisciplinarity: The set of assessment questions about quality and the metrics used to produce answers are not always the same … across subsystems different, contradictory, and divergent answers may be given to the same questions. These fractionated defi nitions and metrics of what constitutes quality account in part for the checkered and uneven pace of commitments to and acceptance of interdisciplinary research within federal science agencies and research universities. (Feller 2006) Inevitably, as evaluation is intimately bound up with funding, career advancement and strategy development, issues arising here may draw upon, and have implications for, foci of the other chapters. We hope to shed some light on practicalities of a) improving evaluation of interdisciplinarity, and b) enhancing recognition that interdisciplinarity can in fact be subject to respectable rigour, and thus to contribute to tackling the barriers that evalua- tion at various points can pose to interdisciplinary ventures. BOOK.indb 138 02/06/11 7:24 PM ASSESSING THE ROUTE 139 We have argued elsewhere that, if programmes and funding schemes are viewed as experiments, their evaluation can in a sense be viewed as formative evaluation, capturing lessons learned along the way so as to enhance future effectiveness (e.g. Meagher and Lyall 2005a). Given the relatively short history of interdisciplinary research as a recognized phenomenon, and the even shorter trail of self-aware, explicit evaluations of interdisciplinarity, this chapter can be seen as contributing to refl ection, informed experimentation and long-term capacity-building in the evaluation of interdisciplinarity. Whatever the evaluation situation, interdisciplinary work overall is done no favours if evaluation is not rigorous. However, achieving shared defi nitions of rigour and quality across a range of settings takes extra effort. The distinc- tiveness of the challenges posed by interdisciplinarity should be recognized, planned for and accommodated. Judging quality in interdisciplinary proposals and outcomes Peer review is an essential component of evaluation of discipline-based projects and must also be the cornerstone of evaluation of the quality of interdisciplinary proposals and outcomes. However, the criteria adopted by disciplines do not translate well across to interdisciplinary initiatives. Indeed, because interdisciplinary research does not yet have its own widely recog- nized criteria for identifi cation of quality this can leave its quality open to debate (Feller 2006). Lamont has studied peer review in many forms and suggests that evaluating interdisciplinary research well and fairly, and achieving consensus, is a signifi - cant challenge, as seen in several of her observations on criteria and multiple layers of evaluation: ● ‘The standards used to evaluate interdisciplinary research are not a simple combination of the standards of single disciplines. They are a hybrid, and an emergent hybrid at that – one that has developed through practice and deliberation.’ ● ‘Interdisciplinarity often brings about a broadening and multiplication of evaluation criteria, which makes both individual judgment and group agreement much more diffi cult.’ ● ‘The lack of canonized agreement about how to evaluate interdisciplinarity gives researchers more leeway concerning how to go about their work, but it also creates greater uncertainty about how to establish the resulting project’s quality.’ ● ‘Combining traditional standards of disciplinary excellence with interdisciplinarity presents a potential for double jeopardy.’ BOOK.indb 139 02/06/11 7:24 PM 140 INTERDISCIPLINARY RESEARCH JOURNEYS ● ‘Not surprisingly, given the emergent quality of the standards of evaluation for interdisciplinary genres, panelists readily fall back on existing disciplinary standards to determine what should and should not be funded. This may mean that at the end of the day, interdisciplinary scholarship is evaluated through several disciplinary lenses.’ (Lamont 2009: 208–211) In directing attention to the challenges of evaluation regarding interdiscipli- narity, the journal Research Evaluation covered a range of insights from a meet- ing that convened experts including research administrators, editors of journals and social scientists to discuss quality assessment in interdisciplinary research and education (Boix Mansilla et al. 2006). An underlying premise was the need to fi nd the most helpful and least damaging ways to evaluate interdisciplinary research. The group concluded that ‘there is no single quantifi able formula to measure quality in interdisciplinary research’ but offered instead a set of four ‘hot spots’ that merit attention in evaluation of quality by asking, does the research: ● focus on ‘the