Measuring Excellence in Research and Research Training
Total Page:16
File Type:pdf, Size:1020Kb
Measuring excellence in research and research training Proceedings of a conference held at the Shine Dome, Canberra, on 22 June 2004 The conference, Measuring excellence in research and research training, was organised by the National Academies Forum and was held at the Shine Dome in Canberra on 22 June 2004. The published proceedings, sponsored by the Australian Government Department of Education, Science and Training, are also available online at www.science.org.au/researchexcellence. © Australian Academy of Science, 2005 GPO Box 783 Canberra ACT 2601 www.science.org.au ISBN 0 85847 223 6 Minister’s Foreword The pursuit of excellence has long characterised the Australian research community—excellence in research itself, and excellence in the training of the researchers of the future. But how do we measure excellence or gauge the success of the pursuit? The National Academies Forum (NAF) has contributed greatly to our understanding of the ways in which we might assess the quality and impact of research and research training. These Proceedings, from the NAF symposium on the subject held in June 2004, will continue the debate. In May 2004, the Australian Government announced that a Quality Framework for publicly funded research would be developed. This would measure the quality and social impact of research conducted in universities and publicly funded research agencies. The aim is to create a Framework that will help the Government build on the already outstanding achievements of our world-class innovation system. I have established an Expert Advisory Group to advise me on the development of the Research Quality Framework. Over the coming months, the Department of Education, Science and Training will also consult with stakeholders. We are in the fortunate position of being able to learn from assessments that have already taken place in Australia and abroad. There is much we can learn from this work and much of the existing knowledge is brought together in these Proceedings. Reaching a consensus on the way forward is the next step. These Proceedings will be an invaluable resource as we take that step and I commend them to everyone interested in the future of Australian research and research training. The Hon Dr Brendan Nelson MP Minister for Education, Science and Training Contents Conference welcome Dr Jim Peacock 1 Measuring excellence: A Chief Scientist perspective Dr Robin Batterham 3 Excellence in investigator-driven research Professor Robert Graham 10 Excellence in the humanities, creative arts and media Professor Iain McCalman 20 Excellence in mission-directed research Dr Michael Barber 26 Excellence in industry-funded research Dr Bob Watts 36 Excellence in social sciences in the context of impact on public policy Dr Valerie Braithwaite 44 Assessing excellence in research in terms of private and public benefits Dr Geoffrey Vaughan 51 Changing research practices in the digital information and communication environment Mr Colin Steele 61 Assessment criteria for excellence in research and research training Address to breakout groups by Dr Phil McFadden 74 Reports from breakout groups 76 Open forum discussion 85 Conference welcome Dr Jim Peacock, FAA, FRS, FTSE President, Australian Academy of Science The National Academies Forum is a gathering of the four learned academies in Australia – the Australian Academy of Science, the Australian Academy of Technological Sciences and Engineering, the Australian Academy of the Humanities and the Academy of the Social Sciences in Australia. The National Academies Forum held a small workshop in March of this year to scope this public symposium on the topic of measuring excellence in research and research training. We are delighted that the symposium has attracted some 150 participants including scientists, scholars, administrators, research managers and policy makers from universities, research organisations and government agencies. A number of countries, as you know, have put various schemes to measure excellence in research in place, and in some of them at least the institutions being measured, as well as collaborating with the measurement process, have also used some strategies to present the best phenotype for the measurement. We don’t particularly want games-playing to affect what happens in Australia; if we have got a good, open, light-touch system it gives us the best opportunity to get things right. Importantly, today we want to talk about research and research training, and it is also important to think about the recognition of emerging excellence as well as extant excellence. So I think we need to think of ways of looking at the vitality, the flexibility, of institutions and groups in research and research training. So how do you do this? Do you make an assessment on an institution, like a university, say? Do you assess departments and then have some form of aggregation to give an institutional score? Do you assess disciplinary teams? Or do you use different assessments for multidisciplinary teams? Where does peer assessment feature? I am sure we all want to see that, and peer assessment will be very important. But we don’t have that many peers in Australia, and that presents a problem in itself – or a challenge, anyway. I think the main thing is that today we, the stakeholders, if you like, in this process, have a chance to discuss these issues together. We are not likely to come up with a single solution today, and there probably shouldn’t be – nor could there be – a single solution. I think, though, we do need to recognise that ultimately the assessment of research training and research in regards to excellence is going to affect investment. So that is something that we need to hold in our minds. Welcome | 1 This process will be a continuing process for us. We will have the opportunity, as I have said, either individually or through other arrangements that the National Academies Forum might try and put in place, to have further interactions with the people who are battling to come up with a scheme, so we have a continuing dialogue. So what do we want out of today? We would like to know your views on what the assessment framework must achieve, for both training and research, and what it must avoid doing. I think we all have some quite strong opinions about both those things, so let’s hope we are going to have a creative day. 2 | Measuring excellence in research and research training Measuring excellence A Chief Scientist perspective Dr Robin Batterham, FAA, FTSE Chief Scientist Commonwealth of Australia Thanks for the opportunity to kick off today’s discussions. It really is an important topic. By the way, this is a commercial which you can read whilst I am getting into the opening. I have to give it these days so that people are quite clear what my role is. I had the benefi t of quite some time last week with Dave King, the UK Chief Scientifi c Advisor. Dave King is quite adamant that the measuring of the excellence of research in the United Kingdom is one of the key reasons that research is now better funded in that country and taken seriously by government and wider afi eld – quite adamant. Indeed, coming out in Nature in June is a paper by David King, ‘The scientifi c impact of nations’. It continues Bob May’s earlier work showing the scientifi c footprint, as it is now quite developed, of different countries. I have got the data here for Australia versus many other countries, including most of the OECD list, and it makes some interesting reading. I am not going to get into it, other than saying there are starting to be some quite well developed systems for measuring the footprint, and the excellence that sits behind that footprint, of research in countries as a whole. And I would refer you to Dave King’s paper. Robin Batterham | 3 Let me run through a few things, because there are three points that I would like to make. They are the changing nature of science, the overall funding levels and a few comments about measuring performance. But fi rstly, we don’t really have a robust or a consistent way to measure the quality of research conducted in universities and publicly funded research agencies. That is what is regarded as the state of play now, and is of course the background and context for why you are here. YouYou wouldwould also all be wellwell awareaware that a small amount of moneymoney has been setset aside toto help develop an appropriate framework. I wanted to emphasise, particularly on the science side, the changing nature of science. I have used here an illustration from Calum Drummond, a Federation Fellow, from a presentation that he was to give to PMSEIC last week. I thought of it when I was sitting in the Institution of Electrical Engineers, in London, in this marvellous wood-panelled room with a picture of Faraday looking down at you, and you sit there and listen to whoever is presenting – it happened to be Sir David King – and doing the mental comparison, ‘Is this person up to Faraday’s standard or not?’ One of the intriguing things about Faraday, and ultrahigh capacity capacitors, which was the subject of Calum Drummond’s talk, is that the scientifi c direction for how to get really high capacity capacitors was actually set by Faraday and it has taken all this time before it has been realised. And it has been realised because some great science out of CSIRO in the nanotechnology area has allowed the ultrafi ne carbon particles, which are at the core of these capacitors, to be produced. 4 | Measuring excellence in research and research training The point that I am making here is that the nature of science – and wider endeavours – is changing, in that there is much more intersection across disciplines and even bridging, of course, between science, social sciences and the humanities.