Download The

OUTLOOK

ASSESSING SCIENCE

Lessons from Australia and New Zealand

OUTLOOK ASSESSING SCIENCE

24 July 2014 / Vol 511 / Issue No 7510

OUTLOOK ociety benefits enormously from scientific research. We

ASSESSING CONTENTS SCIENCE get new technologies, live longer and healthier lives, and gain deeper knowledge of our planet and the Universe. SThe issue of how to evaluate the fruits of research confronts S52 SCIENCE BUDGET scientists and policy-makers all over the world. Every country Funding by numbers has its own set of circumstances surrounding its research Australia and New Zealand’s different infrastructure, wealth, and economic, environmental and approaches to financing research Lessons from Australia and New Zealand developmental objectives — so there is no universal solution. S57 RESEARCH METRICS Cover art: Dale Edwin Murray Earlier this year, at a symposium organized by Nature in Calling science to account Melbourne, Australia, a group of leading academics, funders A Nature symposium in Melbourne Editorial initiated debate on research outcomes Herb Brody, Michelle and government advisers discussed how research outcomes Grayson, Stephen are measured (see page S57). This Outlook supplement was S64 RESEARCH ASSESSMENT Pincock, Kathryn influenced by these debates, although we at Nature take sole The limits of excellence Miller, Rebecca responsibility for its content. Measuring academic output has Dargie, Afsaneh Gray As discussed at the symposium, both Australia and New benefits and drawbacks Art & Design — Wesley Fernandes, Zealand have research assessment programmes that place S67 PERSPECTIVE Mohamed Ashour, heavy emphasis on research excellence (S52) — a qualitative On the verge of a new ERA Alisdair Macdonald, determination that is heavily informed by quantitative Margaret Sheil helped devise and Andrea Duffy metrics concerning, for instance, how often a paper is cited implement Australia’s research Production evaluation framework Karl Smart, Susan (S64). Both Australia (S67) and New Zealand (S82) have Gray, Ian Pope, seen their global scientific standings rise in recent years — S72 RESEARCH IMPACT Robert Sullivan, attributable at least in part to their assessment systems, even Income for outcome Chris Gilloch though Australia’s system offers little financial reward (S81). Governments want value for money Advertising Measuring research using academic yardsticks largely when funding science research Kylie Ahern, Kate Aylett ignores the wider impacts of research such as new policies S77 PERSPECTIVE Powering up citations Marketing or improved technologies. Academics and policymakers in Alan Finkel offers a way to make Hannah Phipps both countries are considering the benefits and difficulties patents and PhD supervision equivalent Project Manager of trying to measure such impact (S72). Could the creation to citations for research appraisal Anastasia Panoutsou of ‘citation equivalents’ enable comparison of non-academic Art Director work against peer-reviewed literature (S77)? S81 PERSPECTIVE If not funding then teaching Kelly Buckheit Krause We hope that the intense focus on these issues in Australia Publisher Brian Schmidt unpicks the financial Richard Hughes and New Zealand will inform and stimulate this crucial questions surrounding the Excellence Magazine Editor debate throughout the scientific world. in Research for Australia system Rosie Mestel S82 Q&A Editor-in-Chief Philip Campbell, Editor-in-Chief, Nature Individual approach Philip Campbell Michelle Grayson, Senior Editor, Supplements Jane Harding balances the pros and cons of New Zealand’s system and its focus on individual researchers Nature Outlooks are sponsored supplements that aim to stimulate All featured articles will be freely available for 6 months. interest and debate around a subject of interest to the sponsor, SUBSCRIPTIONS AND CUSTOMER SERVICES while satisfying the editorial values of Nature and our readers’ For UK/Europe (excluding Japan): Nature Publishing Group, expectations. The boundaries of sponsor involvement are clearly Subscriptions, Brunel Road, Basingstoke, Hants, RG21 6XS, UK. delineated in the Nature Outlook Editorial guidelines available at Tel: +44 (0) 1256 329242. Subscriptions and customer services for go.nature.com/e4dwzw Americas — including Canada, Latin America and the Caribbean: CITING THE OUTLOOK Nature Publishing Group, 75 Varick St, 9th floor, New York, NY Cite as a supplement to Nature, for example, Nature Vol. XXX, 10013-1917, USA. Tel: +1 866 363 7860 (US/Canada) or +1 212 726 No. XXXX Suppl., Sxx–Sxx (2014). 9223 (outside US/Canada). Japan/China/Korea: Nature Publishing VISIT THE OUTLOOK ONLINE Group — Asia-Pacific, Chiyoda Building 5-6th Floor, 2-37 Ichigaya The Nature Outlook Assessing Science supplement can be found at Tamachi, Shinjuku-ku, Tokyo, 162-0843, Japan. Tel: +81 3 3267 8751. http://www.nature.com/nature/outlook/assessingscience CUSTOMER SERVICES It features all newly commissioned content as well as a selection of [email protected] relevant previously published material. Copyright © 2014 Nature Publishing Group

THE VIEW FROM INSIDE A MICROSCOPE

onash University has built the microscope of the 21st century, Ma cinematic virtual reality environment that gives researchers a new spatial awareness of their subjects. Dubbed CAVE2™, it comprises a curved room with 80 high-definition LCD screens projecting 3D images that allow a viewer, when wearing special “tracked” control glasses, to not only see their subject but also to manipulate and walk through and around it. “This opens up new possibilities for insight and discoveries across a range of sciences,” says Professor Paul Bonnington, director of the Monash eResearch Centre CAVE2 at Monash University displaying brain white matter data courtesy of UIC. (MeRC), who has overseen the develop- Photo: Paul Jones ment of CAVE2 at Monash University’s Clayton campus in Melbourne. advanced imaging facilities, that has The “focusing” components of the tradi- For example, the brain’s white matter made it into the modern microscope. tional microscope are replaced by com- viewed up close in CAVE2 can provide bio- putational tools that transform and filter medical researchers with new insights into MORE THAN THE VIEWFINDER the data sample to extract features. These disease—especially when they are able to A microscope viewfinder makes a are provided by a purpose-built interac- compare diseased and healthy samples. good analogy for CAVE2, according to tive supercomputer installed at Monash Senior research fellow and CAVE2 Professor Bonnington. At the centre University called the Multi-modal Australian platform manager Dr David Barnes of scientific discovery for hundreds of Sciences Imaging and Visualisation explains: “If you looked at this on a years, a microscope has three key com- Environment, also known as MASSIVE. desktop display it would basically look ponents: at the bottom, a light source to The Centre also developed their own like a bundle of wool. You don’t ap- illuminate a sample; in the middle, the data transport, management and storage preciate the space and gaps between focusing dials; and at the top, an eyepiece software called MyTardis, which takes the the circuitry and its 3D structure, but for viewing. huge amount of data from the bottom CAVE2 lets researchers see the actual “Five years ago we set out saying, ‘sci- instrument layer of the “microscope” and structural differences.” entific discovery is still going to depend places it into MASSIVE. The technology behind CAVE2 was on this concept, but we need a modern Finally, the modern-day “viewfinder” is developed by the Electronic Visualization equivalent’,” he says. a powerful viewing lens—CAVE2 itself— Laboratory at the University of Illinois In his 21st century version, the “light which can reveal features, details and a at Chicago in the US, and their experts source” is the imaging technology that perspective never before possible. helped install it in Melbourne, where it’s provides the sample. It could be instru- been operating since November 2013. ments like the Australian Synchrotron, a SEEING WHAT WE’RE MADE OF But it’s the additional work by Monash magnetic resonance imaging scanner or These possibilities include building

IMAGES: UCSD CALIT2, UIC EVL & PAUL JONES PAUL & EVL UIC CALIT2, UCSD IMAGES: University, integrating CAVE2 with other a next-generation DNA sequencer. further on the Human Genome Project to

Advertiser retains sole responsibility for content ADVERTISEMENT FEATURE

examine the finer details of the human Mars that has been reconstructed from a virtual finished building and experi- proteome, the complete set of proteins images originally taken by the NASA ence what it will look, feel and even expressed by our genes. rover Curiosity. sound like. “Mapping the human genome was As with the biomedical images, see- More than just a “wow factor”, this really just prep for where the research is ing the planet’s features at scale and lets engineers test the impact of design going, and that is to understand, not just looking real enough to touch gives choices on noise level and identify the building blocks, but how they func- new insight for researchers. “You can conflicts with plumbing or electricity tion,” explains Professor Bonnington. deduce a lot more from this perspective,” conduits, avoiding costly revisions later. The Human Proteome Project is Dr Barnes says. Mr Bowtell says “virtual construction” like providing this understanding by map- Dr Barnes also says CAVE2 can assist this is driving anticipated savings of 15 to ping proteins to shed light on protein research and industry across many other 20 per cent in the US and the UK. function, and to advance the treatment fields. Archaeologists, for example, can “You can see new ways in which CAVE2 of disease. study fragile historical ruins in detail with- could really be incredibly valuable for A protein’s function is often deter- out disturbing them. And engineers can proving building and project concepts mined by its shape, and how this shape safely observe how a severe storm would before and during their production cycle,” can change. And there is no better batter infrastructure. he says. place to analyse protein shape than in “We have had more than 1500 visitors Professor Bonnington says the poten- CAVE2, where, for example, electron from research and industry,” Dr Barnes tial for application and collaboration with microscopy data converted and mod- says. “People walk away feeling inspired research and industry—and as a teaching elled through MASSIVE can be studied to imagine new uses for the cave in their tool—is boundless. up close in a super-sized format in own fields.” “The technology itself inspires people,” three dimensions. he says. “They see technology used in a Similarly, CAVE2 could be used to VIRTUAL ENGINEERING way they have never seen before, and fast-track drug design. “When you know One imaginative new use was model- they can begin to imagine what the the structure of a target molecule you ling crowd flow and bottlenecks at the future could be like.” could come in here with a bunch of redeveloped Second Avenue Subway For further information on CAVE2 drug candidates and literally carry them precinct at Fulton Street, the site of the please email Professor Paul Bonnington: over to the molecule and see if they fit,” former World Trade Center towers in [email protected] Dr Barnes says. New York. To be kept informed of our latest He says the human brain is still the “You can actually see people moving research developments, subscribe to best pattern-recognition tool we have, around you in their avatar form, and you Monash Magazine online. Visit: which is why viewing data in high- get a greater understanding of how you www.monash.edu/monashmag resolution, virtual-reality detail in CAVE2 can improve the physical operations is such an extraordinary opportunity of a building when people are there,” for discovery. says Peter Bowtell, buildings practice leader of global design and engineering TRANSCENDING TIME AND SPACE firm Arup Australia. “It is all about being Medicine is not the only field to able to visualise big data sets in a mean- benefit from CAVE2, and the instru- ingful way.” ment can display more than just the Arup is also using the 3D format of Monash University microscopic world. Planetary scientists CAVE2 to give clients and practitioners www.monash.edu are able to step inside a panorama of the ability to walk through and around

CAVE2 is a trademark of the University of Illinois Board of Trustees. Advertiser retains sole responsibility for content OUTLOOK ASSESSING SCIENCE

FUNDING BY NUMBERS All data are from 2010 unless otherwise stated, because 2010 provides the most recent, reliable, comparable and complete data. Australia and New Zealand both rely on assessment schemes to improve research quality, 200 All dollar amounts are in US dollars, converted using a 2010 exchange rate. yet the money associated with each is very diﬀerent. By Julie Gould.

The Excellence in Research for The Performance-Based Research Fund Darwin Australia de nes research as: AUSTALIA NEW ZEALAND de nes research as: “e creation of new knowledge ESEACH ASSESSMENT ESEACH ASSESSMENT “Original investigation and/or the use of existing FUNDING BY STATE FUNDING BY UNIVESITY $6M Auckland University undertaken in order to contribute Australia’s budget for higher-education The majority of New Zealand’s higher of Technology knowledge in a new and creative Cairns to knowledge and understanding research and development is an order education funding is allocated by the way so as to generate new $1.3M University and, in the case of some of magnitude larger than that of New Performance-Based Research Fund of Auckland Auckland concepts, methodologies and Northern disciplines, cultural innovation Townsville Zealand, yet the amount of money (PBRF), which spent a total of $56M understanding.” Territory determined by Excellence in Research for $168 million on tertiary education or aesthetic reﬁnement.” 2 universities Australia (ERA) is signi cantly smaller. organizations in 2010. University $12M of Waikato Queensland $14.2M 8 universities THE GEAT DIVIDE Western The system used by the PBRF to allocate $8.5M Australia 4 University of funds to universities . 5 universities Massey South Brisbane Canterbury Wellington University Australia $7.7M $21M $27M Funding for 3 universities Australian 100% universities as Capital New $6.9M Lincoln determined by South Wales Territory $7M 60% Quality evaluation University Christchurch Victoria ERA assessment1. Perth 10 universities 2 universities Population: Population: $18M University $28.6M Sydney of Wellington Adelaide Other Canberra (Australian $0.3M University Dunedin 25% Research degree completions Catholic University, Melbourne of Otago across 4 states) Victoria $41M Funding for 15% External research income Tasmania 9 universities each university $24.1M 1 university allocated by *The system used to allocate funds by Excellence in Research 4 $2.3M the PBRF . for Australia is more complicated and relies on a more Hobart 22M 4.4M peer-review structure than New Zealand’s system.

MONEY MATTES SCHOLALY SPENDING PINTED MATEIAL MONEY MATTES The Excellence in Research for Australia (ERA) assessment comprises only 0.08% of all government New Zealand continues to increase spending on Any published articles* that include at least one Almost 90% of all funding for higher-education research and development (HERD) in New Zealand is funding for higher-education research and development (HERD)1. HERD, but Australia is starting to pull back2. author from Australia or New Zealand5. provided by the government and universities6.

2.5 8,000 Overseas Other Experimental $0.2B Experimental development Overseas $21M development Donations $0.7M Business Australia $19M $100M $0.2B 6,000 Business $0.4B Strategic basic research 2.0 $32M General (new knowledge in Australia University speci c areas) 4,000 Government Basic $1.8B Government funds Applied (inc. PBRF) research (state, local, (e.g. student fees) research 1.5 $0.6B $399M $0.6B $298M federal and $5.1B $4.2B Applied $9.B $9.B New Zealand* 2,000 Sources of HERD competitive research grants, inc. ERA) Sources of Pure basic HERD funding expenditure funding expenditure $219M $4.2B Number of published articles research New Zealand Universities (new knowledge (inc. student fees)

Percentage of GDP spent on HERD Percentage 1.0 0 with no agenda) $147M $2.3B 2000 2002 2004 2006 2008 2010 2000 2002 2004 2006 2008 2010 Year Year *Data for Australia are produced on even years; *Journals, conference proceedings, book series, books or trade publications. data for New Zealand are produced on odd years.

EXCELLENCE IN ESEACH FO AUSTALIA: TIMELINE PEFOMANCE-BASED ESEACH FUND: TIMELINE Policy-makers and government ocials have changed funding allocations from formula favouring quantity to assessment criteria favouring quality3. Before 2000, funding was provided to tertiary education organizations based on the number of researchers. Now funding is dependent upon individual performance reviews7.

1990 1995 2000 2005 2010 2015 1990 1995 2000 2005 2010 2015

Australian Research block Minister for ERA pilot ERA Next Education Act White Paper to First Performance- PBRF Full PBRF Research Council funding became Education, study. 2012 ERA establishes assure research Based Research Fund reviewed. assessment. established. Introduction of Uni ed formula based, Brendan Nelson, report. evaluation. the Tertiary Funding now quality and (PBRF) assessment. All funding National System, which measuring calls for new Excellence in Education based on number accountability. now allocated was a framework to number of papers Research Quality Research for Australia Commission. of equivalent through the reorganize and improve published. Framework (ERA) announced. full-time students, Tertiary Education Advisory PBRF. Next PBRF higher education in (RQF). weighted by Commission (TEAC) established to TEAC introduces assessment Australia. RQF abandoned. course costs. provide advice on strategic direction. the PBRF. is in 2018.

References: 1. Research and Experimental Development, Higher Education Organisations (Australian Bureau of Statistics, 2010), available at go.nature.com/1obhsa; 2. United Nations Educational Scienti c and Cultural 4. Performance-Based Research Fund Annual Report 2010 (Tertiary Education Commission), available at go.nature.com/s5t6p7; 5. Scopus; 6. Research and Development in New Zealand 2010 Organization Institute for Statistics; 3. Kwok, J. T. Impact of ERA Research Assessment on University Behaviour and their Sta (NTEU National Policy and Research Unit, 2013) available at go.nature.com/c2ypxa; (Statistics New Zealand), available at go.nature.com/c7bbp5; 7. A History and Overview of the PRBF (New Zealand Ministry of Education), available at go.nature.com/vwXjbu.

1990 1995 2000 2005 2010 2015 1990 1995 2000 2005 2010 2015

A CULTURE OF INDEPENDENT THINKERS

ome universities might be content “We have a very strong interest in conduct real-time tests on single cells with fostering Prime Ministers and tackling intergenerational transmission of within living organisms. SRhodes Scholars, Nobel Prize win- obesity; preventing the obesity epidemic In collaboration with Macquarie ners and astronauts. But the University of by taking the new strategy of stopping University, the institute’s researchers have Adelaide has its sights set even higher—to it at its root rather than trying to reverse developed optical fibers laced with nano- be a world-leading research institution, the situation when people are already crystals, which are small enough to interact meeting the challenges that will reshape overweight,” says the Institute’s Director, with targets at a cellular level. The aim is to lives and ecosystems. Professor Sarah Robertson. develop a miniaturized laboratory that can As one of Australia’s most research-inten- Bringing together laboratory studies, operate directly with cells in vivo. sive institutions, the University has a proud population-based cohort studies and The next step will be to use this ap- history that goes back to the youngest ever clinical trials, the Institute’s researchers have proach to make meaningful measurements Nobel laureate, Sir William Lawrence Bragg, found that obesity is programmed into in the brain, the cardiovascular system and who attended the University of Adelaide a child as early as conception, with both even developing embryos, says Institute in the early 1900s and who, when aged 25, parents contributing equally to a child’s Director Professor Tanya Monro. shared the Nobel Prize in Physics with his metabolic destiny, through their genes “Any assay that would normally be father in 1915. and the conditions of the reproductive carried out in the lab could be done at the “We have a tradition of exceptional environment. level of a single cell, without taking it out of research and a research culture generated “It means that parents have a major the body,” Professor Monro says. by a long history of independent think- effect on the life course of their offspring ers,” says Pro Vice-Chancellor (Research right from the time of conception, when PROTECTING A FRAGILE ENVIRONMENT Strategy) Professor Robert Saint. “It is a most people don’t even realize they’re The global timber industry is worth an broad-based institution and we do a wide getting pregnant,” Professor Robertson says. estimated $180 billion each year. Yet ap- range of research, from basic through to The discoveries offer potent new targets proximately one-third is illegal. An effective the applied.” for obesity prevention. Already, principles strategy for cracking down on this illegal Consistently ranked in the top 1% of learned in animal studies are being borne trade, and thereby reducing associated the world’s universities, Adelaide has 60 out in large-scale human clinical studies and problems of tropical deforestation and research areas rated at or above world trials of interventions aimed at breaking this greenhouse-gas emissions, is to empower standard. It also ranks among Australia’s pathway to obesity. Results of some trials consumer choice by identifying the source top three universities for contract have already been published in high-impact of a piece of timber. research and commercialization activity, journals, while other studies are ongoing. Researchers at the Australian Centre for demonstrating strong connections to Evolutionary Biology and Biodiversity have external partners. NEW HORIZONS IN SENSING developed a DNA-based tracking method The 1966 science fiction filmFantastic that uses genetic analysis to pinpoint the A HEALTHIER START TO LIFE Voyage envisaged miniaturization tech- species and geographic origin of a piece of At the Robinson Research Institute, nology that could send a medical team timber with remarkable accuracy. This tech- researchers are taking a ‘bench to bedside’ traveling around inside the body. nique can reveal whether a piece of merbau approach to providing children with While it might be impossible to send timber (Intsia palembanica), for example, the healthiest possible start to life. The tiny people into the body, scientists at has been harvested from a certified planta- Institute’s transdisciplinary focus is deliver- the University of Adelaide’s Institute for tion or whether it has been taken illegally ing results, particularly in the vexed area of Photonics and Advanced Sensing are from an uncertified source, even within the obesity prevention. developing sensors at the nanoscale to same country.

Advertiser retains sole responsibility for content ADVERTISEMENT FEATURE

As large-scale DNA screening becomes “The University has become an impor- “The challenges with the production of easier and cheaper, and can be conducted tant regional centre for its holdings of biofuels from biomass are sustainability and at any stage along the supply chain, a population and migration data from the high productivity,” says Associate Professor reference database of DNA samples from region, and its staff have high levels of skill David Lewis, one of the project’s lead timber is being compiled to ensure easier in analyzing them,” Professor Hugo says. researchers and CEO of Muradel Pty Ltd. and more accurate matching of timber Using a strain of microalgae isolated by products to their source. Murdoch University, which grows rapidly “THESE ACHIEVEMENTS EXEMPLIFY This DNA-based tracking system—the in saline water, the team has been able THE CAPACITY OF THE UNIVERSITY TO only one of its kind available in the world— to upscale an initial pilot plant, while the TURN ITS HIGH-QUALITY FUNDAMENTAL has now been commercialized in partner- University of Adelaide’s chemical engineers RESEARCH INTO OUTCOMES THAT ship with the Thünen Institute in Germany have developed the vital technologies that CREATE A BETTER FUTURE” and Double Helix Tracking Technologies allow concentration and extraction of the in Singapore, and it is proving invaluable dilute algae into a commodity product. for verification of sustainably harvested As well as examining demographic A demonstration plant has been built timber products. changes at the national and regional levels, and commissioned in Whyalla, South “There are a range of certification meth- researchers are also working with global Australia, with plans to build a larger ods being used, but certification papers can organizations such as the Nansen Initiative thousand-hectare plant that will be able to be falsified,” says Centre Director Professor to study the impact of climate change produce around 450,000 barrels of green Andrew Lowe. “The role of DNA analysis is to on migration. crude per annum—enough to supply a verify those certification claims.” “Climate change is going to create medium-sized industry such as mining, massive disruptions and displacements of trucking or a regional airline. CHANGING HUMAN POPULATIONS people,” Professor Hugo says. “But in fact This project, and the many others like We live in an era of unprecedented demo- there is a much more complex relationship it, are proof of the University of Adelaide’s graphic changes, with migration altering in which the effects of environment and capabilities, says Professor Robert Saint. the face of nations, and changing climates climate change are going to be part of a “These achievements exemplify the triggering the relocation of populations. wider pattern of economic change.” capacity of the University to turn its high- The University of Adelaide’s Professor quality fundamental research into out- Graeme Hugo is one of the world’s pre- GROWING NEW FUELS comes that create a better future for local eminent demographers. As director of With the twin challenges of climate change communities and the wider world.” the Australian Population and Migration and peak oil looming, the race to find Research Centre, he is at the forefront of viable, sustainable alternatives to fossil fuels exploration into how our global population has never been more urgent. is changing, evolving and moving, and the In response to this need, the University of economic and social impacts of these shifts. Adelaide’s School of Chemical Engineering The University’s long established linkages partnered with Murdoch University to with Asia have helped make it the largest establish Muradel Pty Ltd, which has research and training Centre in migration developed Australia’s first commercial-scale research in the Asia-Pacific region, with demonstration plant for the production of more than 50 scholars from countries green crude from microalgae. It’s another il- in the region gaining PhDs in migration lustration of how the University encourages The University of Adelaide and population issues, and more than 20 connections that reach from basic research www.adelaide.edu.au currently studying. to commercialization.

Advertiser retains sole responsibility for content ADVERTISEMENT FEATURE

PURSUING PLASTIC POLLUTION AT THE ATOMIC LEVEL

lastic pollution is finding its way pattern of elements in each feather— detected by ANSTO’s instruments and into the tissues of marine wildlife. much like the rings of trees. It also con- determine the litter’s original source. P Although many environmental risks firmed the researchers’ initial suspicions: “Plastics will always have a place in of plastic have long been known, analysis trace elements from plastic were found our world,” says Richard. “But we need to by the Australian Nuclear Science and in the feathers of birds that had eaten it. be mindful that a seemingly ‘single-use’ Technology Organisation (ANSTO) has The health effects of these elements throw away item will change form many detected chemicals from plastic pollution aren’t well understood, but their pres- times and stay in the system at the in the feathers of seabirds. However, this ence in tissues shows the complexity atomic level for eternity.” research is also giving clues to how to of the pollution problem. Contrary to ANSTO’s Institute for Environmental manage the problem. expectations, plastics that degrade in Research already uses the ability to The work is a collaboration between the environment are not necessarily distinguish naturally occurring isotopes Richard Banati, a biomedical scientist safer for wildlife, as the increased surface to follow the movement of groundwater at ANSTO LifeSciences, and Monash area as they break down can exacerbate and rainwater into the hydrologic cycle, University conservation biologist the release of toxins like cadmium and to examine other human impacts on Jennifer Lavers. and mercury. the climate and environment. They analysed the elemental composi- “A traditional approach to envi- These activities sit alongside ANSTO’s tion of plastic items collected from the ronmental management has been long-standing roles in medical research— stomachs of flesh-footed shearwaters. ‘the solution to pollution is dilution’,” contributing to new ways of treating These were then compared with the says Richard Banati. “However, we are conditions like Alzheimer’s and Parkinson’s atomic elements in the feathers from finding that mass plastic consumption, diseases, as well as producing 85 per cent birds of the same species—some together with increased degradability of the nuclear medicines used in Australian that had eaten plastics and some that of plastics, may actually lead to a steady hospitals—and materials engineering, such had not. increase of hazardous contaminants in as the development of Synroc—tailored The plastic and feather samples the environment which would be dif- ceramic forms for locking up high-level were analysed first at the Australian ficult to reverse.” radioactive waste. Synchrotron light source in Melbourne However, the researchers stress that and then at the country’s only operating their aim is not to demonise plastic, nuclear reactor, the ANSTO Open-Pool but rather to better understand it at an Australian Lightwater (OPAL) research atomic level. reactor in Sydney. There they were Richard Banati, who has used radioac- subjected to neutron activation, in which tive elements in pharmaceutical research atoms are bombarded with neutrons to to track a drug’s progress through the make them slightly radioactive. body, points to the intriguing possibility Each element is then identified by that manufacturers, too, could add an its distinctive gamma-ray spectrum, isotopic signature of non-radioactive giving measurements of composition so trace elements to plastics to trace accurate they’ve been compared with their lifecycle. determining the vintage of a single glass A combination of non-abundant of red wine spilled into Sydney Harbour. elements together with a small amount This produced some interesting of precious metal like gold—as little findings, such as a regular distribution as 10 milligrams per tonne—could be

Advertiser retains sole responsibility for content ASSESSING SCIENCE OUTLOOK SDP MEDIA

Multidisciplinary funding and the benefits of research evaluation programmes were hotly debated at an event organized by Nature.

RESEARCH METRICS Calling science to account Systematic evaluation of scientific research might strengthen public support, but could it also stifle innovation? The issues were debated at a symposium in Melbourne.

BY TIM THWAITES In his introduction to the symposium, even before research starts; and Australia’s Nature editor-in-chief Phil Campbell out- quantitative evaluation of its research strengths ince the United Kingdom’s first Research lined several of the issues and views that were and weaknesses. Two things were clear: there Assessment Exercise in 1986, the later discussed. “There is a need for research are many reasons for evaluating research, concept of a national evaluation of evaluators to be explicit about the methods and there are lots of approaches to get results. Spublicly funded research has expanded to they use to measure impact,” he said. “Open- Perhaps the first hurdle to overcome is deciding other countries, including Belgium, France, ness is an essential part of earning trust. Nature what you want to achieve. Italy, Australia and New Zealand. Some assess- welcomes a diversity of indicators.” Rely- ments are performed specifically to determine ing solely on citations, Campbell added, VALUING RESEARCH allocation of research funds, whereas others “absolutely can’t be sustained”. The symposium’s keynote speaker was David are benchmarking exercises of the perfor- The United Kingdom has recently re- Sweeney, the director for research, innova- mance of local research in a global context. oriented its research-assessment programme tion and skills at the Higher Education Fund- Although the overall goals of these assessment to bring peer review, case histories and metrics ing Council for England (HEFCE) in Bristol. systems are well understood, there is doubt as into a system called the Research Excellence Sweeney, who managed development of the to how well each is working. Framework (REF), which runs for the first time REF, told delegates there was “no right to Their relative effectiveness was the focus of this year (see ‘How research benefits the United research funding”. He said, “If, as happened a symposium in February 2014 in Melbourne, Kingdom’). The Melbourne symposium exam- in previous budget proposals in the UK, Australia. Nature brought together experts ined this approach against various schemes in senior scientists say to government ‘Give us from institutes and universities in Australia, the Asia-Pacific region, including Singapore’s the money, and we will deliver the goods’, the New Zealand and Singapore to examine issues carefully programmed development of knowl- treasury has a right to say, ‘Prove it!’.” surrounding the outcomes and impact of how edge-based industry; New Zealand’s proposi- Sweeney said that scientists cannot assume research is measured. tion that criteria for assessment be laid down that the general public understands the value

24 JULY 2014 | VOL 511 | NATURE | S57 © 2014 Macmillan Publishers Limited. All rights reserved OUTLOOK ASSESSING SCIENCE of their research, so evaluation has become an Real-world issues, such as water and Australia’s largest national research body, essential tool for convincing UK government, energy usage, are complex and inter- the Commonwealth Scientific and Industrial business and society why they should invest in connected, and research addressing these Research Organisation (CSIRO), based in universities and research. In fact, he said, the matters needs to draw on expertise from Canberra, is no stranger to multidisciplinary UK government wanted to enlist companies physical and biological sciences, as well as research, said its chief executive Megan Clark. to help fund university research — unlocking social sciences including economics, behav- CSIRO, she noted, specializes in large-scale, some of the capital that businesses had put ioural psychology and law. Yet, according to broad, “pan-disciplinary” research groups. away during the global financial crisis to pro- participants in a panel discussion on multi- “There is an understanding from the minute tect against hard times. The outcomes of the disciplinary research, such crucial work has you walk in that this is not a place to work REF, teamed with matched-funding schemes, rarely been valued appropriately in research on personal research,” she said. “We work in could help the government release previously assessment exercises. multidisciplinary teams on mission-directed hidden private pots of money, he argued. The intrinsic value of multidisciplinary research.” As a result, CSIRO’s evaluation of its Sweeney outlined the REF’s methodology. teams, and the difficulties of their coordina- own research includes “Academic excellence is still the number one tion, were well illustrated by the story of the “The traditional outputs, objective of public funding,” he said. But con- Murray-Darling Basin Plan, set up to man- methodology such as patents and ventional gauges of merit, such as peer review age water resources in Australia’s largest and does the job journal publications, and citations, should not comprise the whole most agriculturally productive area. “It was a that needs to be and quality assess- assessment; it’s also important, he said, to wonderful document that told us exactly what done now, even ments by independent reward research that has a positive impact on we should do,” said Robert Saint, pro vice- though it’s not peer review panels, society. He asserted that the REF did not open chancellor of research strategy at the Univer- perfect.” but crucially also takes the way for government to dictate research sity of Adelaide. The plan was unpopular as it into account the direction. Nor did it mean a bias towards fund- proposed swingeing cuts to water allocation impact of its work on end users — including ing applied research. Instead, said Sweeney, for many farmers. “Its release was closely fol- the public, government departments, private REF provided a means of validating the contri- lowed by farmers burning it, and the whole companies and environmental organizations bution of all research: “It’s not about favouring business had to go back to the drawing board.” (see page S72). one discipline over another.” He presented REF The problem was that the Murray-Darling CSIRO runs large-scale multidisciplinary not as a perfect measure of impact, but as a first Basin Authority, which compiled the report, research partnerships known as National step. “The methodology does the job that needs lacked the specific capabilities for incorporat- Research Flagships (see ‘Launching flagships’). to be done now, even though it’s not perfect,” ing legal, political and social issues alongside These focus on issues of national and global he explained. the science. importance such as biosecurity, preventative health, manufacturing and sustainable agriculture. In a little more than a decade, the HOW RESEARCH BENEFITS THE UNITED KINGDOM Flagships programme has grown to encompass more than half of all CSIRO research activity. Evaluating research can increase its public support Many stakeholders, Clark recalled, feared that the Flagships programme would lead to The Research Excellence Framework a decline in the quality of the organization’s (REF) is the new system for assessing UK SCOECAD science. But CSIRO’s experience has been the the quality of research at UK universities. How a Research Excellence Framework (REF) score is determined. reverse, she said. “Last year, we hit a record It aims to demonstrate the benefits of in the quality of our science and our stand- public investment in research; to show ing globally.” For instance, the citation rate 15% accountability for government research Research for CSIRO research publications is now 56% environment funding; and to rate the quality of the United 65% more than the global average, according to Kingdoms’s research efforts on a global Research the organization’s latest Science Health and quality scale across all academic disciplines. Excellence report. The REF — successor to the Research Total EF CSIRO’s approach differs from multi- Assessment Exercise — will produce its first 20% Impact score disciplinary work undertaken at universities, report in December 2014. It will be used to beyond which are the primary training grounds for academia assist the four UK higher-education funding researchers, said Kim Langfield-Smith, vice- bodies — the Higher Education Funding provost for academic performance at Monash Research quality (up to four papers per Council for England, the Scottish Funding researcher, published between 2008 and 2013). University in Melbourne. The academic Council, the Higher Education Funding Impact beyond academia (case studies written environment tends to have discipline-focused Council for Wales, and the Department by researchers). organization underpinning promotion Research environment (research strategy, of Education and Learning in Northern facilities, sta ng, etc.). tracks. This silo structure is not conducive to Ireland — to allocate government funding researchers thinking outside their speciality. for research, at present about £1.6 billion Langfield-Smith spoke of the difficul- (US$2.7 billion) a year. internationally excellent, internationally ties in recruiting university researchers for It assesses the efforts of higher education recognized, nationally recognized or multidisciplinary projects. In particular, institutions across 36 subject areas, unclassified. mid-career and older researchers found it dif- determined by the quality and impact The assessment is made by expert ficult to justify interrupting their research to of research as well as what the research panels, which include representation from join projects that might not yield publications environment is like (see ‘UK scorecard’). people in business or government who in the top journals of their own fields. What’s The results will be published in the form of use research outputs in their professional more, multidisciplinary research is difficult a quality profile in which each submission activities or who commission or collaborate to get underway: it routinely lacks common is ranked as either world-leading, with academic researchers. T.T. language, modes of analysis, conceptual frameworks and dedicated journals (many outcomes

S58 | NATURE | VOL 511 | 24 JULY 2014 © 2014 Macmillan Publishers Limited. All rights reserved ASSESSING SCIENCE OUTLOOK are instead published as government reports). impact on the economy, the environment, entrepreneur and chancellor of Monash Saint observed that peer review could be defence and public health. Such priorities University, Alan Finkel, suggested that fund- disadvantageous to multidisciplinary projects were greater in small economies that cannot so ing bodies needed a framework where activi- at both the funding and publication stage. “I easily spare money for blue-sky research. ties such as working in industry, contributions remember the early days of bioinformatics: One of the purposes of the Excellence in to government reports or communication statisticians would argue that all the theory had Research for Australia (ERA) programme at its of research outcomes to audiences other been done 40 years ago, and biologists couldn’t inception was to determine in which research than a researcher’s peers could be converted see anything interesting in statistics.” The fields Australia had world standing, said into a “citation equivalent” for the purpose panellists suggested several ways to promote Margaret Sheil, provost of the University of of improving the measurement of research multidisciplinary work, including setting up Melbourne and a former head of the Austral- impact (see page S77). dedicated funding streams for such research, ian Research Council (ARC). Sheil, who was and altering the criteria of assessment so that heavily involved in the design and operation METRICS ARE NOT THE ANSWER work published in government reports is of the ERA (see page S67), pointed out that Assessing a country’s research enterprise is not eligible for consideration. although Australia had a small population, it an end in itself. And when it comes to acting Hugh Durrant-Whyte, chief executive was competing globally in many disciplines. upon the outcomes of research assessment, of NICTA, Australia’s largest information Representing these different viewpoints funders have vastly differing viewpoints. and communications technology research in one assessment tool is not easy. Science The one issue on which they tend to agree is organization, suggested that the solution lay in removing disincentives. Funding agencies, he said, should foster a research culture that LAUNCHING FLAGSHIPS encourages scientists to undertake projects because they were “cool and exciting, not Restructuring CSIRO to tackle national problems because there is a paper at the end”. Young scientists, he said, should be urged to “find The National Research Flagships are account for more than half of CSIRO’s budget something interesting and get on with it”. This large-scale, multidisciplinary research of about AUS$1.5 billion (US$1.4 billion), would naturally stimulate collaborations and partnerships between the Commonweath expected to increase to 65% by July 2015. multidisciplinary work, he added. Scientific and Industrial Research Setting up the Flagships required a Organisation (CSIRO; Australia’s largest substantial organizational shake-up. Initially, WHEN OPTIONS ARE LIMITED national research body), universities, other CSIRO retained its 11 traditional discipline- The value of research to government can be publicly funded research institutions, based divisions across the country, with very different from its value to business, or to the private sector and international the Flagships able to draw staff from any academia or the public. That’s why it’s critical organizations. There are 11 current division or partner organization. From July to set the criteria for evaluation from the very Flagships tackling significant national 2014 they will merge into nine Flagships. beginning, said Peter Gluckman, chief science challenges (see table, below). Since 2005, a Flagship Collaboration adviser to the prime minister of New Zealand. The initiative began with three Flagships Fund has committed more than This approach “changes the way research is in 2003 and has grown into one of largest AUS$130 million in grants to encourage done”, said Gluckman. “It influences how scientific research programmes ever partnerships between universities, CSIRO scientists work and think.” undertaken in Australia. Together, they and other research agencies. T.T. Gluckman was mainly referring to government-directed projects that account for a large portion of the science budget of small FLAGSHIP TITLE YEAR LAUNCHED DESCRIPTION countries such as New Zealand. Perhaps the Energy 2003 Investigating low carbon alternative energy sources and the most compelling argument for this principle future of transport. can be seen in Singapore, which has taken little Food Futures 2003 Transforming the global reach of the Australian agrifood sector. more than a decade to generate a biomedical Preventative 2003 Improving well-being through research into prevention, industry from a low starting point (see ‘How Health detection and health interventions. to grow an industry’). Wealth from 2003 Investigating the network of resources, capacity and David Lane is chief scientist of Singa- Oceans sustainability of the oceans and developing appropriate pore’s Agency for Science, Technology and technologies. Research (A*STAR) which, with the country’s Water for a 2004 Developing technologies to improve the social, economic and Economic Development Board, was respon- Healthy Country environmental outcomes around water access and use. sible for implementing the Biomedical Future 2007 Developing cleaner advanced materials and technologies. Sciences Initiative to develop the industry. Manufacturing He said that determining impact was a major Minerals Down 2007 Growing Australia’s resource base, increasing productivity part of the government’s strategy. “Our budget Under of the minerals industry and reducing its environmental was increased,” he said, “but 30 to 40% was set footprint. aside and would only be released if we could Climate 2008 Supporting Australia's efforts to adapt to climate change. show we were doing work aligned with indus- Adaptation try.” The yardstick by which the effort was Sustainable 2010 Addressing productivity and food security in a carbon- measured was the level of corporate invest- Agriculture constrained world. ment in Singapore’s biomedical industry. Digital 2012 Developing and delivering improved online services and Such a utilitarian view of science by govern- Productivity & changing the way people engage with technology. Services ments, said Gluckman, differs enormously from the academic perspective, which focuses Biosecurity 2013 Helping to protect Australia from biological risks posed by exotic and endemic pests and diseases. on accumulation of knowledge. Governments, he said, were typically concerned with research

Research Council (NHMRC) in Canberra, HOW TO GROW AN INDUSTRY which dispenses more than AUS$750 million (US$700 million) of government money in Singapore becomes a biomedical powerhouse research grants each year (see page S52). “It’s not only the research that’s important, but also how it is used,” he said. Health researchers typically wish to influence decision-makers and medical practitioners as well as other scientists, which means they need to publish in areas outside academic literature. To properly evaluate their BLOOMBERG/GETTY work, he said, you needed to consult sources other than scientific journals, such as government reports and health-care experts. Government has a huge interest in health care because of its enormous cost. Australia’s AUS$140 billion health-care industry — comprising vaccine manufacturers and medical device developers, among others — is also the nation’s second largest exporter of manufactured goods, Anderson said. Australia’s other major research funding body is the ARC, responsible for disbursing more than AUS$900 million a year. It also administers the ERA, which aims to determine areas of Australia’s research strengths. Singapore’s impressive bioscience research hub, Biopolis, which opened in 2003. ERA assessments are made by internationally recognized researchers, organized by In little more than a decade, Singapore has hub, the Biopolis — seven buildings of discipline and clustered into eight Research established a thriving biomedical industry 185,000 square metres at a cost of SG$500 Evaluation Committees. They use traditional from scratch. The country, with few natural million — next to the National University of measures of quality, such as citation analysis resources, set itself a goal in the early 1990s Singapore. or peer review, but also incorporate a broader to become a knowledge-based, innovation- The second phase (2006–10) bolstered view, considering income from commerciali- driven economy. The government identified Singapore’s capacity to undertake zation and measures of esteem — for example biomedical sciences as an area with translational and clinical research and turn it being admitted to a learned society such as tremendous growth potential and decided into health-care products and applications. becoming a fellow of the Australian Academy to try to grow its own industry. In 2000, it Large biomedical corporations, such as of Science. started to invest in a Biomedical Sciences Lonza and Genentech, were attracted by ERA ranks research quality against a global initiative and, by 2012, this industry favourable business and investment terms scale and is “a rigorous and robust measure had grown to more than SG$30 billion including generous tax rates and access to a across all discipline domains”, ARC’s chief (US$24 billion), comprising more than 50 flexible and skilled labour market. executive Aidan Byrne told the symposium. It manufacturing plants, 50 R&D centres and The current phase (2011–15) focuses on aims, he said, to get researchers to change their 30 regional headquarters of biotechnology encouraging international investment and focus from quantity of work to quality. “In that, and pharmaceutical companies. links with industry. An Industry Partnership the ERA exercise has been spectacularly suc- The first phase of the initiative (2000–05) Office has been set up to facilitate cessful. And it did it without tying the exercise concentrated on rapidly expanding collaborations with private enterprise. The to financial rewards.” Singapore’s basic research capabilities and five-year budget was increased by 16% Furthermore, despite its reliance on metrics, infrastructure as well as attracting industry to nearly SG$16 billion, with a substantial ERA results for academic excellence correlate R&D laboratories from other countries. portion channelled into industry-oriented with other real-world outputs, Byrne said. The government also constructed an R&D research. T.T. For instance, 95% of industry investment in research in Australia is in the same areas in which researchers performed at world-class that any worthwhile evaluation of research they will perform in the future, said Tony Pea- or better. And the same is true for 98% of the — whether it be for disbursing grants or cock, chief executive of the Australian Coop- research that was commercialized and for 97% encouraging excellence — needs to be based erative Research Centre (CRC) Association in of the work that was patented. on a range of measures, not just the quantity Canberra, which runs the nation’s 40 CRCs HEFCE’s Sweeney’s take on various of publications and how often they are cited — collaborative partnerships between pub- methods of assessment was straightforward. by others. In the final panel of Melbourne licly funded researchers and industry. In fact, No system will be perfect, he said, but you symposium, representatives of four significant Peacock argued, rewarding only those strat- have to start somewhere: “You can propose funding organizations discussed how best to egies that were successful in the past would alternatives, and spend five years discuss- incorporate the information gained from tend to discourage new approaches and stifle ing them, but that’s not going to solve today’s assessments. innovation, the essence of successful science. problems.” ■ Traditionally, research assessment evaluates Relying solely on citation and peer review completed projects. But, in an ever-changing metrics was opposed for similar reasons by Tim Thwaites is chief science writer for research environment, a scientist’s past suc- Warwick Anderson, chief executive of Science in Public, a science communication cesses might not be a predictor of how well Australia’s National Health and Medical agency based in Melbourne.

SAVING SPECIES WITH BIG DATA AND APPLIED STATISTICS

errie Mengersen, Professor of solutions, but also raise questions that statistics and machine learning to create Statistics at the Queensland can motivate discussion and gener- innovative models and make vital contri- K University of Technology (QUT), ate new insights. butions to society, business and govern- knows she has the best job in the For Arun Sharma, QUT’s Deputy ment. Its work is organised into three world. Studying everything from the Vice-Chancellor, Research and research programs—Big Data Analytics effect of chemotherapy on the brain to Commercialisation, Mengersen’s work will focus on innovative methods for the efficient management of airports, epitomises the university’s focus on extracting and communicating informa- Mengersen applies the tools of Bayesian supporting high-impact research tion from big and complex data; Big statistics to some of the world’s most with a view to achieving significant Models will study new theoretical and important problems. public and practical benefit. “We’re methodological approaches to model- A Bayesian statistician views the known as ‘a university for the real world’ ling large, complex systems; and New world as an opportunity for adap- because of our close links with end Insights will consider new ways of using tive learning, Mengersen explains. users,” he says. the modelling and analytic approaches “We create new mathematical and In 2013, the success of that strat- to inform our understanding of issues statistical methods and compu- egy saw QUT ranked by The Times in the areas of health, the environment tational solutions,” she says. “Then Higher Education as Australia’s top and societies. we apply these new approaches university under 50 years in its 100 “All our work involves engaging with to help solve important problems under 50 ranking. In recent years, QUT great researchers and the expert clini- across diverse areas including has built critical mass in the fields of cians, conservationists, managers health, environment, conservation, robotics and automation, big data and others who are doing amazing business and industry.” and personalisation—technological things in their own fields,” Mengersen “As statisticians we find the stories trends that are advancing disciplines, says. “It is humbling and very reward- in the data, and then convey these transforming professions and disrupting ing to be able to help them do their insights to people who make decisions business models. These technological jobs better.” to address some of the world’s grand capabilities, coupled with an interdis- “One of the great things about work- challenges.” ciplinary culture and a desire to solve ing at QUT for me is its very strong Recently, Mengersen and co-authors real-world problems, differentiate support for maths and statistics,” she caused a stir when they used this ap- QUT’s research. adds. “The University encourages proach to study the number of living The diversity and importance of cross-disciplinary collaboration, which species that exist on coral reefs around Mengersen’s work have led her to be means that I actively work with the world. They found that despite named one of QUT’s most successful people from social science, engineer- decades of study, scientists appear no researchers. In the past year alone, ing, environmental science, health closer to understanding how many spe- she was an integral leader in the con- and business.” cies are alive today. Existing estimates sortium securing $20 million in funding On top of all that, Mengersen is vary wildly, seemingly with little refer- over the next seven years to establish particularly proud of her group of ence to previous findings. a new Australian Research Council researchers and students, known as the They called for a more sys- Centre of Excellence for Mathematical Bayesian Research and Applications tematic and statistically based and Statistical Frontiers of Big Data, Big Group, or BRAG. “What more could I ask approach to this important topic. Models, New Insights. of a workplace?” It’s work that illustrates how statisti- The aim of the Centre is to bring cal analysis can not only provide together researchers in mathematics, www.qut.edu.au

Advertiser retains sole responsibility for content ADVERTISEMENT FEATURE

FROM HEAVY INDUSTRY TO NOVEL MANUFACTURING AND SUSTAINABLE SEAS

e live in an age of uncertainty, coastal environments, and the changing a world with a population that nature of manufacturing and industry. W is both ageing and growing. “We have taken an innovative approach It is a world of climate change, of rapid to our research, concentrating on three economic transformation and of immense challenges and then marshalling multidisci- technological advances. plinary resources and expertise from across Most cities and regions have developed the University to work on research projects slowly—over many decades, if not cen- under each challenge,” Professor Gibson says. turies. But in order to survive in this fast- changing world, communities must now Professor Chris Gibson, Global Challenges MANUFACTURING INNOVATION be able to adapt quickly. Program Leader, University of Wollongong Taking advantage of Wollongong’s history So says Professor Chris Gibson, leader as a region of industrial creativity, we’re of the Global Challenges Program at the Developed at the Australian Research already tackling the puzzle of what we University of Wollongong, one of a small Council Centre of Excellence for should be making in Australia and how we number of universities around the world Electromaterials Science (ACES), which is should be making it. This means exploring that is harnessing research strengths headquartered at the University, the BioPen the potential of the broadband-powered to target the global challenges of the uses techniques similar to 3D printing to ex- digital economy and the possibilities of 21st century. trude cellular material between layers of gel. smart materials, robotics and automation, as By delivering this material directly to the site well as innovative medical devices. TRANSFORMING LIVES AND REGIONS of an injury, it can accelerate the regenera- It’s also where we use our strengths in Ranking in the top 2% of universities world- tion of functional bone and cartilage. 3D printing and additive manufacturing wide, we are especially qualified to tackle The University of Wollongong was the to create results that were previously not these challenges having played a leading first in Australia—and remains one of feasible—such as custom-printed flutes role in the transformation of our home only five labs in the world—to fabricate that can play microtonal scales, or the town of Wollongong, one hour south of silicene, a new two-dimensional material. notes between notes. Sydney, to a city of the future. Structurally similar to graphene, but made After bringing together economists, Throughout the last century, of silicon instead of carbon, silicene has planners, social marketers, creative artists Wollongong was Australia’s heavy in- intrinsic semiconductor properties that and designers to work on projects, we have dustry heartland, best known as the graphene lacks. initiated a global benchmarking exercise to home of the nation’s largest steel mill. These properties could help it revolutio- ensure we are operating at best practice in The University was established to train nise materials science, leading to smaller, these novel manufacturing techniques in the industrial chemists, metallurgists faster computer chips and more practical our local region. and engineers needed to keep these and efficient solar cells, as well as improve- factories working. ments in medical technologies and vehicle SUSTAINING COASTAL AND But, although these traditions con- and aircraft parts. MARINE ZONES tinue, our research is now helping turn Having demonstrated our expertise Through our international collaborations Wollongong into a 21st century centre for through achievements like these, we are we’re seeking to improve the sustainable technology. This includes innovations like now focusing our efforts on three Global management of fish stocks, like bigeye the BioPen, which lets surgeons “draw” on Challenges—areas in which we can make and yellowfin tuna, that are under pres- damaged bone with an “ink” containing the greatest impact: the ageing popula- sure from fishing fleets. This project, run live cells. tion, the growing pressure on marine and through the Australian National Centre for

Advertiser retains sole responsibility for content ADVERTISEMENT FEATURE

Ocean Resources and Security, will help to “Regions all over the world are facing KEY NUMBERS deliver the food security needed for social common challenges, and it is important ♦♦ 31,500 students stability in Pacific island nations that face that from an empirical basis we develop ♦♦ 12,800 international students declining fisheries. tailored outcomes that have a global ♦♦ 1,520 higher degree research We’re also harnessing social media impact,” he says. students (893 domestic; technology to build resilience to climate The Global Challenges Program shows 627 international) change-induced extreme weather events how we’re playing a significant role in trans- ♦♦ 138 nationalities represented in in South-East Asian megacities. PetaJakarta forming lives and regions for the better. student body uses real-time data from 150,000 Twitter To learn more, visit globalchallenges. ♦♦ 2,350 staff users in Indonesia’s capital to track and uow.edu.au. ♦♦ 476 degree courses. analyse flooding and inform emergency services and residents. UNIVERSITY OF WOLLONGONG RESEARCH AND COLLABORATION Our SMART Infrastructure Facility, which A dynamic Australian university with a OPPORTUNITIES runs the project, was one of only six strong research focus, located just an hour ♦♦ Record $49.1 million in Australian research institutions in the world to receive from Sydney’s International Airport. Research Council (ARC) funding an inaugural Twitter Data Grant and gain announced in 2013 access to this vital information. CURRENT RANKINGS ♦♦ Postgraduate scholarships ♦♦ Top 2% of world universities (The ♦♦ Multidisciplinary PhDs LIVING WELL, LONGER Times Higher Education, QS, Leiden ♦♦ Vice-Chancellor’s Postdoctoral By taking a holistic approach to the prob- and Academic Ranking of World Fellowships lem of an ageing population—considering Universities) ♦♦ Visiting academic appointments physical and mental health as well as ♦♦ Top 100 for global graduates (QS ♦♦ Study leave and sabbaticals access to services—we’re discovering the Graduate Employers Survey) ♦♦ Financial assistance for international physical and mental requirements for a ♦♦ Five stars (QS World University exchanges long, healthy and high-quality life. Rankings) ♦♦ Joint PhDs with overseas universities. Our research is making a real impact on ♦♦ 1st in Australia for Educational dementia, which is expected to affect one Experience and Graduate Outcomes LEARN MORE million Australians by the middle of the (Good Universities Guide) Visit: www.uow.edu.au century. Two dementia-friendly communi- ♦♦ 22nd and 33rd in The QS and www.uow.edu.au/research ties that we’re designing and piloting will Times Higher Education rankings Facebook.com/UOW not only accommodate but also actually respectively for the world’s Top 100 Twitter.com/UOW welcome people who suffer from this Universities under 50 years of age. debilitating illness. ♦♦ 2nd (relative to size) and 8th Phone: +61 2 4221 3555 And we’re bringing together doctors, on total research funding psychologists, geneticists, paediatricians, awarded nationally. obstetricians and social scientists to follow three generations of local residents to CAMPUSES understand their patterns of mental health ♦♦ Two campuses each in Wollongong and wellbeing as they age. and Sydney, and regional campuses in Professor Gibson says these collabora- the Shoalhaven, Batemans Bay, Bega tions capture the power of the University’s and Moss Vale University of Wollongong multidisciplinary research capacity. ♦♦ University of Wollongong in Dubai.

Advertiser retains sole responsibility for content OUTLOOK ASSESSING SCIENCE DALE EDWIN MURRAY EDWIN DALE

RESEARCH ASSESSMENT The limits of excellence Young researchers and interdisciplinary science might be getting short-changed by research assessment in Australia and New Zealand.

BY ANNABEL MCGILVRAY high-impact international journals rather had been linked to postgraduate student than working on a book with a domestic numbers. The flaws of that arrangement had he day before he and I speak, Jonathan publisher; pressure to change research focus to become evident in the mid-1990s when non- Boston receives an email on a familiar better align with mainstream or more highly university higher education providers — such theme. It is from a colleague concerned esteemed fields; or encouragement to accept as polytechnics and institutes of technology T about a junior researcher whose career a position as a non-PBRF-eligible teaching — began offering postgraduate degrees. The decisions are “being twisted in an uncomfort- fellow and move away from active research. research pot was suddenly being split between able way” by the demands of New Zealand’s Each of these outcomes can be traced to the ever more institutions, many of which had Performance-Based Research Fund (PBRF). way in which the PBRF measures research limited research capacity. When Helen Clark’s “I have had many such messages over the years excellence, and so its influence on the country’s Labour government came to power in 1999 — which reflect the good and the bad of the research funding environment. with five former academics — including Clark, PBRF,” says Boston. At its conception in 1999, explains Bos- a lecturer in political studies — among its sen- Boston holds a personal chair in public ton, the PBRF was an ambitious under- ior ranks, it vowed to strengthen the process of policy at Victoria University of Wellington, taking to measure research excellence research funding and increase accountability. and was one of the architects of the New andraise standards at institutions across New “The only option was some sort of perfor- Zealand system. He says that research paths Zealand. It was intended to remedy years of mance-based regime,” says Boston. can become conflicted by one of three neglect of the research sector through which The PBRF is based on the individual, scenarios: compulsion to publish articles in the bulk of funds, dubbed research top-ups, making it unique among measures of national

S64 | NATURE | VOL 511 | 24 JULY 2014 © 2014 Macmillan Publishers Limited. All rights reserved ASSESSING SCIENCE OUTLOOK research excellence. Every six years, it gauges But these measurements of excellence researchers quite compromised by this drive and reports the standard of research of each are creating obstacles for young scientists, to produce the one paper that will get into of New Zealand’s approximately 6,000 says Attila Brungs, deputy vice-chancellor Nature,” says Gluckman, “when their career researchers in universities and colleges (so- for research at the University of Technol- would have been much more developed had called tertiary educational establishments). ogy, Sydney (UTS). “Narrow metrics can they focused on getting solid, excellent papers These rankings — A, B, C and R — are drive some bizarre behaviours. People don’t in the appropriate journals”. provided to the institutions; a researcher can publish as much with PhD students because apply to receive his or her own rating. The out- PhD students are often published in lower- TOGETHER YET APART comes are then weighted by quality and sub- ranked journals.” Researchers undertaking interdisciplinary ject area, in line with the resources required More broadly, the strengths of early-career work are also feeling compromised. Campuses for different fields. The individual results are researchers aren’t readily demonstrated by across New Zealand and Australia are bringing aggregated by institution and are the major reference to an objective publication review, together researchers from multiple disciplines, determinant, alongside external income and a particular flaw of the individual-centred from the hard sciences to the humanities, to research degree completions, of the distribu- PBRF. Assessment encourages institutions to look at societal problems in a holistic way. tion of research funding. The PBRF is now the employ staff with These fields include environmental sustain- largest single source of tertiary research fund- established research ability and medical research and, in many ing in New Zealand, worth NZ$262.5 million records rather than There are fears cases, work is carried out under the auspices (US$224.2 million) in 2013 (see page S52). emerging research- that the way of a centre or an institute within a university. Across the Tasman Sea, the Australian ers who are doing excellence is Despite this big-picture approach, assess- Research Council (ARC) is gearing up for excellent science measured may ments such as the PBRF and the ERA continue the third round of its own national measure but who are yet to hinder the to view research through a mono-disciplinary of research quality, the Excellence in Research amass publications. careers of young lens. The final report from the 2012 PBRF con- for Australia (ERA) evaluation, which takes There is early evi- researchers. ceded that the 42 subject areas under which all place for every three years for the country’s 41 dence of this reluc- research is assessed “do not accurately reflect universities. The forthcoming ERA2015 will tance to engage young researchers, with one the way research activity is organized and con- categorize and evaluate the nation’s entire study showing a 14% drop in research staff ducted”. Despite this acknowledgment, there higher education research output, compris- aged 35 and younger between the first and are no plans for a review, says Marny Dickson, ing more than 400,000 publications. The ERA second rounds of the PBRF3. chief policy analyst for tertiary education at the assesses work by discipline, and the initiative Indeed, a 2008 independent review of PBRF New Zealand Ministry of Education. directly influences only a small portion of found that morale of otherwise high-achieving The story isn’t much more encouraging university research funding. young researchers was being hurt by low rat- in Australia. The Australian Council of the Results suggest that the overall quality of ings. The review, commissioned by the body Learned Academies (ACOLA) — represent- research has increased in both countries since that oversees PBRF, the Tertiary Education ing the four Australian academic societies: the the introduction of national assessments. Commission (TEC), stated that “the assign- Australian Academy of Science, the Academy More New Zealand researchers are achieving ment of a ‘C’ grade was seen by rising stars of Social Sciences in Australia, the Australian an A rating, and more Australian disciplines to undermine morale and to stigmatize their Academy of the Humanities and the Austral- are classed at ‘above world-standard’1,2. But the position”. Boston says that when the PBRF ian Academy of Technological Sciences and PBRF and ERA prompt passionate reaction in scheme was designed there was no intention Engineering — seeks to inform policy specifi- their respective research communities. With to reveal individuals’ ratings. Not until after the cally related to multidisciplinary research. In increasing awareness of the need to assess the system was established did Boston and his col- a 2012 report, ACOLA found that the ERA societal impact of research, merely weighing leagues realize they were compelled to impart “has difficulty in evaluating and reporting academic excellence makes less sense. There is that information to researchers. “We simply interdisciplinary research”. And the situation concern that subject-focused assessment pro- failed to fully realize the implications of the is likely to be exacerbated as universities base grammes don’t adequately recognize the value Privacy Act and Official Information Act,” he their internal benchmarks around the ERA, of interdisciplinary research. And, as Boston’s says. “If I had known we would end up with a which, like the PBRF, focuses researchers on recent email correspondence implied, there are regime in which individuals had their scores higher impact journals — few of which are fears that the way excellence is measured — in reported to them, and that other people could interdisciplinary. particular the focus on high impact publica- potentially know what they were, I would not For instance, the Centre for Cosmopoli- tions — may be hindering the careers of young have supported it.” tan and Civil Societies at UTS, does a lot of researchers. The TEC has created a specific ‘new and applied research related to policy, and fre- emerging researcher’ category to counter dis- quently produces work for the local, state THE FOLLY OF YOUTH incentives to employ early career researchers and federal governments. When the centre’s PBRF and ERA both use metrics of quality when evaluation rounds loom. Researchers in researchers publish, they have to do so in the and peer review to determine ratings. The this category can qualify for C(NE) rating and journals of their individual expertise, whether same indicators are used in different forms contribute to their institution’s funding allo- marketing, business and economics, or social in research assessment schemes around the cation. Their evidence portfolio assessment science and the humanities. The univer- world. However, for the PBRF, all nominated is weighted against their time as a researcher, sity is then rated separately for each of these publications and other research outputs are with a minimum of two research outputs fields, rather than for the centre’s projects as rated by selected reviewers — a qualitative pro- generally expected. a whole. “In an exercise like ERA, their work cess that depends on individual judgements. But many people, including Boston and disappears, it doesn’t exist,” says Brungs. The ERA, by contrast, places more emphasis Peter Gluckman, chief science advisor to the Brungs says that the distortion doesn’t on citation analysis. “The number of people New Zealand prime minister, believe that the yet affect the university’s research priorities, who are citing and making reference to work is individual judgements inherent in the PBRF because the funding linked to ERA is very a pretty good indicator of the significance and reviewing process continue to place undue small. In 2014 it was just AUS$69 million, or impact it has had in the academic community,” pressure on emerging researchers to publish in 4% of the research block grants made by the says Aidan Byrne, ARC chief executive officer. high-impact journals. “I’ve seen several young Department of Education. But, an increase in

INTERNALIZING TARGETS Measurements used by the Excellence in Research for Australia system are finding their way into internal targets set by research institutions. Here is one example of 2011 targets from a leading Australian university.

Laboratory-based sciences: minimum (Min) and aspirational (Asp) targets Research outputs Lecturer Senior lecturer Associate professor Professor Min Asp Min Asp Min Asp Min Asp No publications 1.5 4 3 6 4 8 7.5 15 Impact factor 4.5 16 9 24 12 32 22.5 60 Proportion in A/A* journals † 35% 55% 45% 65% 55% 80% 55% 80% Research income (AUS$) $5,000 $40,000 $30,000 $125,000 $150,000 $500,000 $500,000 $1,000,000

† Ranking of journals as a proxy for quality (A/A* being the two highest) was abandoned by the ERA in 2011. the proportion of ERA funding would make strategic decisions about what research they measurement of impact. The Australian gov- future collaborations harder to justify. “Uni- wish to pursue.” The importance placed ernment’s current aversion to any increase in versities are not allowing the drivers to distort on the ERA rankings by Australian uni- red tape does not help. their behaviour too much,” says Brungs. “But if versity management has been evident, not In New Zealand, the 2008 review of PBRF we continue to go down that path it does have only in the tendency for researcher churn- cautioned against diluting its focus on excel- real danger for the interdisciplinary sector.” ing, but also in the tailoring of internal lence by aligning it with government innova- research benchmarks to better meet the tion policy. However, a re-evaluation has seen a PLAYING GAMES terms of the ARC system (see ‘Internalizing number of alterations, including an increase in If PBRF and ERA become more significant, targets’). Consequently, universities have the significance of investment from industry in recruitment policies at universities will inevi- set departmental and school-wide targets determining overall funding awards, coupled tably be coloured by how a candidate might regarding quantity and quality of publications. with a moderate reduction on the emphasis affect a pending evaluation. Some fear this will Despite this, Byrne does not accept that ERA of the research quality assessment. According lead to a widespread gaming of both systems as is forcing institutional change. “It does get used to the Ministry of Education, these changes institutions try to improve their scores. by institutions in various ways, but we are pro- reflect the fact that external research income Australia’s National Tertiary Education viding an evaluation against the best possible is a “strong proxy indicator” for the transfer Union has found that ERA gaming already standards we can come up with”, he says. of knowledge between academia and indus- occurs, as individuals, departments and insti- Perhaps the most egregious example of try and the change will encourage “research tutions strive for results needed to influence an attempt to game the system occurred in of relevance to end-users”. It is a tangible shift funding decisions that last for three years. In New Zealand in 2006. One leading univer- towards reward for research impact. the case of the PBRF, the equivalent decision sity reclassified dozens of staff members, But Boston is not satisfied that, even with influences six years of funding, a magnitude notably those who were PBRF-eligible but such changes, assessments like the PBRF and which further incentivises manipulation. The performed little active research. By reclas- ERA will continue to be relevant. He refers to problem is likely to persist, says Frank Larkins, sifying inactive researchers away from Goodhart’s law, which states that once a meas- former deputy vice-chancellor for research at subjects such as economics and biology to ure becomes a target it ceases to be a good the University of Melbourne. “Universities fields such as philosophy and religious studies, measure. “I don’t see the logic of running the have a lot of smart people and they can learn the university would improve its standing in same assessment process every six years ad pretty fast how to optimize their performance,” the former fields. The surge in the number of infinitum, with only minor tweaks,” he says. says Larkins. New Zealand philosophers piqued the curios- “It sets up a particular set of incentives and a The ERA peer-review panels are asked to ity of PBRF reviewers who eventually reversed particular kind of process within institutions, look closely for idiosyncrasies in the research the classifications. some of which is undesirable.” performance of institutions, and the ARC To keep improving research excellence, is now able to cross-reference dubious sub- MEASURING A MOVING TARGET Boston says, the government needs to increase missions against previous rounds. However, The objectives and the structure of ERA and funding and other resources, or make big- there is nothing to stop universities taking on PBRF have changed little, but the status quo ger changes to the assessments, for instance researchers, and sometimes whole research may be threatened by demand for the explicit by introducing new criteria. Continual departments, in order to boost output prior inclusion of research impact as a quality indi- re-allocation of finite resources can only do to an ERA round. The Australian newspa- cator within the assessment exercises (see page so much. “I don’t know how you’re going to per described the “churning” of researchers S81). At UTS, Brungs says that more focus on squeeze more drips out of the orange.” ■ this year in the lead-up to the 31 March 2014 impact might bring much-needed formal census deadline. Any staff hired after this date recognition of interdisciplinary work within Annabel McGilvray is a freelance science and are not eligible for assessment in ERA 2015. the system. For example, work that has valu- medical writer based in Sydney. But, during the preceding Australian summer, able outputs concerned more with policy than research groups and even whole departments scholarship. “Publishing in Nature is one way 1. Performance-Based Research Fund: Evaluating were poached by the Australian Catholic of demonstrating excellence in research,” he Research Excellence — the 2012 Assessment, Final Report (Tertiary Education Commission, 2013); University, Central Queensland University and says. “Changing the way that a nation drinks available at go.nature.com/gprpet. Charles Sturt University, among others. water is another way.” 2. Excellence in Research for Australia 2012, ARC leader, Byrne, doesn’t endorse such The ARC is considering the inclusion of National Report (Australian Research Council, activities, but says that calculated realloca- impact measurements, but Byrne says the Commonwealth of Australia, 2012); available at go.nature.com/q8sxiw. tion of resources for the purposes of ERA organization does not want to just graft these 3. Çinlar, N. & Dowse, J. Staffing and Performance ranking is not necessarily a bad thing. “We on to the existing system and does not have Trends in Research Subject Areas (Tertiary Education don’t want to stop institutions from making the resources to develop an independent Commission, 2008).

S66 | NATURE | VOL 511 | 24 JULY 2014 © 2014 Macmillan Publishers Limited. All rights reserved ASSESSING SCIENCE OUTLOOK PERSPECTIVE On the verge of a new ERA Despite its limitations, Excellence in Research for Australia was the right assessment tool at the right time, says Margaret Sheil. CASAMENTO PHOTOGRAPHY CASAMENTO

n the early part of this century, the Australian government made a suggested that metrics alone could be used as indicators for many decision to improve the country’s research effort. It made targeted scientific disciplines, although less so for others, such as the humani- investments in research agencies, built infrastructure and created ties and mathematics where citation data were less reliable or books Ia number of world-class centres of excellence. Yet some argued our were more important than journal articles. research efforts were spread too thinly and performance was uneven. In coming up with our own solution, the ARC consulted widely By 2007, only two of Australia’s 40 universities were in the top 100 of with institutions, learned academies, research leaders and bibliomet- the Academic Ranking of World Universities (ARWU). That same ric experts. We settled on ‘discipline’ as the best unit for evaluation year, an Australian Productivity Commission report on public support because it avoided focusing on either university structures or individu- for science and innovation identified several areas of the innovation als (thereby minimizing the value and prospect of poaching individual system requiring improvement, which included a lack of effective sup- stars). Where citation analyses were not appropriate, peer review of port for industry-based research and development, deficiencies in the selected publications or outputs was used instead. This approach also scientific workforce and inadequate methods of evaluation. had the benefit of enabling evaluation of the creative arts — important The government determined that Australia’s dual funding system, for Australia where most conservatoria and visual and performing arts comprising individual competitive grants and block grants to insti- schools had been incorporated into universities. tutions, was not providing the right incentives. In Excellence in Research for Australia (ERA) was particular, block funding was distributed according born in 2010. By including all outputs within a dis- to formulae that took into account only the num- cipline, rather than just the premier efforts of a select ber of scholarly publications, with no consideration group of researchers, ERA ensured that the attention for their quality. The results were, in hindsight, pre- CITATION of every researcher was on quality rather than quan- dictable: the number of publications increased but ANALYSIS tity. And because its discipline-specific measures of not their overall quality. The government needed a achievement accorded with established academic new framework: one that would encourage univer- CAN BE USED AS A practice, the results were largely accepted with only sities to focus their endeavours and build on their a small number of disputed outcomes. strengths; to address weaknesses in research perfor- PRINCIPLE There have been difficulties. ERA initially ranked mance; and to provide researchers with an incentive around 22,000 journals in four bands (A*, A, B and to target quality. All of that had to happen within a INDICATOR OF C) to provide another set of indicators for the com- system that was streamlined and cost effective. mittees to use. However, some institutions used the QUALITY. journal ranks out of context, potentially harming CORE VALUES the careers of young academics and those working From 2007 until 2012, I was chief executive of the in cross-disciplinary areas, since both groups are Australian Research Council (ARC), the body tasked with developing unlikely to publish in top tier journals. Bandings were removed in 2012. the new framework. We weren’t starting from scratch; the previous Overall, ERA has been of considerable value — despite the fact that government spent nearly four years working on a new design, and we it drives only a small proportion of the block funding allocations. could draw inspiration from other assessment systems, including the Crucially, it has demonstrated that citation analysis can be used as a United Kingdom’s Research Assessment Exercise (RAE), and New principle indicator of quality in many disciplines, producing finely Zealand’s Performance-Based Research Fund (PBRF). grained information about research strengths and weaknesses. None of the approaches, however, were entirely appropriate for Governments, universities, industry and the academic disciplines Australia. The proposed design from the last government relied on themselves have welcomed this information and make regular use of case studies to assess research impact. Case studies are expensive and it. This success shows in our international standing: Australia now has time-consuming to prepare and assess, yet were not, the new minis- 5 universities in the ARWU top 100, and 19 in the top 500. ter felt, sufficiently robust to inform funding allocations. Similarly Any comprehensive evaluation system is bound to have its critics. both PBRF and RAE — to different degrees — are selective exercises, But widespread consultation and the use of discipline-based solu- focusing on just the best output, allowing universities to hide poor tions has helped minimize opposition to the ERA. Though there is performance. RAE used panels of experts to assess the quality of publi- widespread acceptance that there are benefits in competitive processes cations — a costly and lengthy process. And the units of assessment for for teams and individuals that have winners and losers, the political both schemes were unsuited to Australia: PBRF evaluates individual dimensions associated with ‘losing’ loom larger when applied to insti- portfolios, making it difficult to scale up; RAE evaluates departments. tutions, for example, negatively affecting newer institutions in areas But our universities have varied and complex organizations of depart- serving growing and diverse populations. These considerations are ments, schools, faculties and/or research centres which do not lend beyond the control of the ARC and are unrelated to the ERA itself, themselves to a simple comparison. which, while not perfect, has nonetheless achieved its goals. ■ These schemes did, however, provide useful leads. In particular, the 21 years of the RAE saw a growing correlation between the quality Margaret Sheil is the provost at the University of Melbourne. ratings assigned to departments and their citation performance. This e-mail: [email protected]

RMIT UNIVERSITY: SMARTER TECHNOLOGY SOLUTIONS

ith an international reputation can make a significant contribution photovoltaic cells, allowing the remain- for excellence in applied and to meeting everyday power require- ing light to reach a solar thermal collec- W outcome-oriented research, ments. An even simpler idea is to take tor, generating temperatures of up to RMIT University in Melbourne, Australia, sunlight and convert it directly into heat. 150 degrees Celsius. is a global university of technology and Gary Rosengarten (pictured, top left), For Rosengarten, RMIT’s mission of design and one of Australia’s leading professor of sustainable systems engi- tackling key issues that affect communi- educational institutions. neering at RMIT’s School of Aerospace, ties and the environment is important. Founded in 1887, RMIT has a proud Mechanical and Manufacturing “I like to know that I am contributing to history of strong links with industry. Engineering, is doing just that — by society by helping industry develop new Under the leadership of RMIT Vice- applying smart technology solutions to products for market and that I am doing Chancellor and President, Professor the field of solar thermal energy. it in an environmentally sustainable way.” Margaret Gardner AO, the university “Solar thermal energy is fundamen- He is also impressed by the atmosphere has continued to foster connections tally more efficient than photovoltaics,” that the university has created for its between research and the private sector. explains Rosengarten. While photovolta- researchers. “RMIT has invested consider- At present, RMIT has over 200 active ics struggle to reach 20 per cent sunlight ably in people and infrastructure in research collaborations with industry conversion efficiency, solar thermal targeted areas to ensure there is critical and overseas partners. energy can easily achieve 70 per cent mass to achieve world-class outcomes,” With a student body that includes efficiency. Plus, heat is also easier and he says. more than 10,000 postgraduate cheaper to store than electricity. students, 5,000 staff comprising 2,500 Rosengarten leads the MUSIC (Micro CONNECTING MEDICINE, PHYSICS AND academic and teaching staff, and cam- Urban Solar Integrated Concentrators) ENGINEERING puses in Australia and Vietnam — plus project, which aims to revolutionize the RMIT’s location in the city of Melbourne a recently opened center in Barcelona, use of solar collectors in urban environ- places the university within easy reach Spain — RMIT boasts a rich portfolio of ments. The thin and lightweight con- of 19 major research hospitals and teaching and research partnerships that centrators being developed will share facilities, meaning researchers are ideally cover every continent. the look and placement of conventional positioned for immersion in the medical As part of its strategic plan for 2015, photovoltaic panels. However, by using a community. Consequently, smart tech- RMIT has set itself three goals: to be clever combination of vacuum insulation nology solutions to meet biomedical global, urban and connected. Entwined to keep the heat in and mirrors or lenses needs are constantly evolving at RMIT, with this is the university’s desire to to concentrate the light, the concentra- as are the facilities in which they are develop innovative solutions to tackle tors will be able to reach temperatures being developed. the complex, technical challenges of of up to 400 degrees Celsius, without the “At RMIT, world-class facilities are today. In particular, researchers at RMIT need for expensive systems that track being set up and a growing group are developing smart-technology the Sun. of leading researchers are working solutions in the areas of energy, health Another of Rosengarten’s projects together to publish their research in and computing. combines solar thermal and photovol- leading journals,” says James Friend taic technologies to efficiently create (pictured, top middle), director of the SMART SOLAR SOLUTIONS both electricity and hot water. This flagship AUS$30 million MicroNano Solar, or photovoltaic, cells are a com- system employs a collector to appro- Research Facility (MNRF) at RMIT and a mon feature in today’s cities and the priately siphon off solar wavelengths professor at the RMIT School of Electrical

LEFT & CENTRE IMAGES: CARLA GOTTGENS;.RIGHT IMAGE: JOHN GOLLINGS JOHN IMAGE: GOTTGENS;.RIGHT CARLA IMAGES: CENTRE & LEFT electricity they generate from sunlight that can be efficiently absorbed by and Computer Engineering.

Advertiser retains sole responsibility for content ADVERTISEMENT FEATURE

2014 will see the completion of the and microfluidics,” explains Friend. “This At present, quantum computing MNRF, a significant upgrade of RMIT’s is a huge motivation for conducting devices that employ a small number of existing fabrication and metrology my research.” qubits already exist. But scaling the tech- capabilities. The new facility will meet nology up remains a challenge — albeit the needs of research at the nanoscale COMPUTING WITH DIAMONDS one that Greentree is willing to embrace. and integrate biomedical research activi- RMIT is a university focused on both “I want to ensure that my research is ties. Research at the facility will span the design and technology. For Andrew important and relevant,” he says, “and to traditional disciplines of physics, chemis- Greentree (pictured, top left), an as- transition my research into real world try, engineering and medicine. sociate professor at the RMIT School of applications and devices.” Much of Friend’s own research lies at Applied Sciences, “it is the technology Greentree is also an expert in the use the boundaries of medicine, physics and part that is key”. of diamond for quantum purposes. In engineering, and his development of Greentree is guiding RMIT’s efforts to particular, he hopes to use diamond miniaturized motors, or microactuators, bring computing into the quantum age. to create hybrid quantum–classical for the retrieval of blood clots from deep A quantum computer stores information computers that boast solid-state within the brain is a prominent example as qubits — the equivalent of bits in quantum memories. Beyond comput- of this union. The use of tiny actua- classical computing. But unlike classical ing, Greentree’s interests in diamond tors capable of navigating weakened bits, qubits can exist simultaneously in are shaping part of the research blood vessels will enable minimally multiple states, a feature that can be programme at the upcoming Australian invasive neurological intervention in exploited to allow new and more power- Research Council Centre for Nanoscale people affected by strokes or aneurysms, ful forms of computing. And while a BioPhotonics, in which RMIT is a partner who might otherwise be unsuited classical computer uses only one aspect institution. to treatment. of quantum mechanics — barrier tun- RMIT is a university that is excited Friend’s background in microfluidics neling — a quantum computer also uses by change and isn’t afraid to back key and micro- and nanofabrication is also the quantum phenomena of superposi- ideas from its researchers, he notes. “If contributing to substantial improve- tion and entanglement to harness the quantum technology is as important ments in the delivery of drugs via the power of quantum states. “This provides as we think, soon we will need to be lungs. By employing acoustic waves greater control, allowing computation educating quantum engineers who that travel along a material’s surface, his to be carried out in new ways,” observes are ready to design and build practical group can atomize large biomolecules Greentree. “Simply put, a quantum quantum devices,” he says. “It’s my aim — including drugs, DNA and antibod- computer is the ultimate computer.” that RMIT will be at the forefront of ies — and even cells into suitably sized For information to be transmitted, this revolution.” droplets, while avoiding the damage qubits must travel from one location that conventional methods of nebuliza- to another. Although wires serve this tion can cause. purpose in classical computing, they are The technology has already incapable of preserving quantum infor- proven effective for the delivery of a mation. Before joining RMIT, Greentree DNA vaccine in sheep, and has the conceived an elegant solution to this potential to be used to convey gene problem, called coherent tunneling adia- therapy or stem cells to inaccessible batic passage (CTAP), which continues sites, deep within the lungs. “Our work to inform his current research. CTAP will RMIT University promises to overcome fundamental likely play a key role in facilitating the on- www.rmit.edu.au/research

LEFT IMAGE: CARLA GOTTGENS; CENTRE IMAGES: JOHN GOLLINGS; RIGHT IMAGE: EARL CARTER EARL IMAGE: RIGHT GOLLINGS; JOHN IMAGES: CENTRE GOTTGENS; CARLA IMAGE: LEFT problems neurosurgery, drug delivery chip transport of quantum information.

Advertiser retains sole responsibility for content The University of Auckland MAKING OUR MARK IN LASER SCIENCE AND OPTICS

Information processing and advanced telecommunications technology are just some of the disciplines that laser science and optics have the potential to impact in the future. Temporal cavity solitons – as persistent light pulses – demonstrate the beauty of laser science and constitute the most fundamental example of self-organisation in optics. They are an ideal information carrier to store data and create an all-optical buffer, a critical function for high-speed routing technology. University of Auckland Associate Professor Stéphane Coen has contributed to the first-ever capture of these pulses of light using nothing more than a continuous-wave laser and an unamplified loop of standard optical fibre. Now his team at Auckland can retain pulses for more than one hour, equivalent to a propagation distance of close to one billion kilometres, and has evidenced their ultra-weak long range interactions. These pioneering achievements have implications for many disciplines, from life sciences to ocean physics. Published in Nature Photonics, the research has revealed the rich physics of these little-known objects. It also has deep practical significance, as temporal cavity solitons provide key insights into new microscopic light sources that have the potential to provide ultra-accurate measurements on a chip. Associate Professor Stéphane Coen is one of a team of award-winning scientists achieving international recognition for their work in our world-class laser laboratory.

To find out more, visit LEADINGRESEARCH.ac.nz KingSt12580/NAT/A ADVERTISEMENT FEATURE

UNIVERSITY OF WESTERN SYDNEY: TACKLING BIGGER, UNANSWERED QUESTIONS

xperimental psychologists, neu- of the university’s emphasis on research Australia’s native forests and intensively roscientists, engineers, computer excellence. “Our big goal is to make sure managed landscapes in the future.” E scientists, linguists, composers and all of our research is high impact, not One of the largest projects con- dancers — few people might guess what only in terms of citations but also the ducted at the Hawkesbury Institute for

kind of research projects would require effects on professions and our communi- the Environment is EucFACE, a free-air CO2 such a diverse team of professionals, but ties,” he says. enrichment field experiment that exam- Kate Stevens from the MARCS Institute Indeed, UWS has come a long way ines the response of a native Australian

of the University of Western Sydney since its founding in 1989. Over the past eucalypt forest to an atmospheric CO2 (UWS) knows. She is the leader of Music 25 years, the institution has evolved into concentration of 550 ppm, which is Cognition and Action, a research program a vibrant, modern, outer-metropolitan expected later this century. “We really that uses behavioral and neurophysi- university where students receive profes- stand out internationally in terms of our ological approaches to study the temporal sional training in science, health and research programs and our extensive dynamics of perception and creativity, as medicine, technology, engineering, array of world-class research infrastructure well as individual and group performance. humanities, education, business and law. to study climate change impacts,” says “We are trying to understand how UWS is now home to over 3,000 staff Anderson. “We run a lot of large-scale, innovation and knowledge emerge from and 42,000 students, most of whom are comprehensive experiments and the a group,” says Stevens, “However, these residents of Western Sydney — one of university has been very supportive.” are difficult and complex questions the fastest growing and most culturally While EucFACE is still in its infancy, that require an interdisciplinary team diverse regions in Australia. preliminary findings indicate that mature to answer.” UWS places a strong emphasis on eucalyptus trees may not respond to

Stevens has joined forces with a profes- research and development. It has invested future concentrations of CO2 as predicted sional dance company and built a re- heavily in areas including neuroscience, by findings from experiments in other search team of internationally recognized infrastructure engineering, complemen- parts of the world. “A significant differ- scientists, choreographers and dancers. tary medicine, education and humanities. ence is the poor nutrient status of most They analyze behavioural and brain 70% of UWS research reviewed by the Australian soils and the fact we have very processes as people interact and create. Excellence in Research for Australia long dry spells where there is not a lot of Every week, Stevens holds seminars and assessment in 2012 was rated as “world water. So the ability of these tree species

specialist group meetings where other standard” or above. to respond to increasing CO2 level is research groups at the institute join and limited,” says Anderson. contribute to the discussion. “Here at the BIG INFRASTRUCTURE, BIG ANSWERS MARCS Institute we have experts in, for Environmental sustainability is a core example, perception, language develop- area for UWS. The university has invested ment, neuromorphic engineering and heavily in the Hawkesbury Institute for the human-machine interaction,” says Stevens. Environment, which focuses on the im- “We all come together for one purpose: to pacts of environmental change on terres- answer those big and risky questions.” trial ecosystems. “Our research spans from the molecular and microbe level through FOCUSSING ON IMPACT to plants, animals and entire ecosystems,” Scott Holmes, the Deputy Vice- says Ian Anderson, Director of the institute. University of Western Sydney Chancellor of Research and Development “The findings from our research are aimed www.uws.edu.au

IMAGES 1-3 SALLY TSOUTAS SALLY 1-3 IMAGES at UWS, says Stevens’ story is reflective towards helping us better manage both

Advertiser retains sole responsibility for content OUTLOOK ASSESSING SCIENCE

And whereas large economies have the capacity to invest in a wide range of scientific endeavours, from nanotechnology to cosmol- ogy, smaller countries such as Australia and New Zealand do not have that luxury. “The smaller the country, the more limited the MURRAY EDWIN DALE choice,” says Gluckman. “So when looking at science and innovation systems, you start to be more conscious in the prioritization process; it becomes a much more strategic issue.” Determining the impact of research on wider society has the potential to assist decision-makers within organizations and institutions. But what is troubling people like Gluckman are the definitions. “You have to be really clear about the word: there are many different kinds of impact and perceptions differ,” he says. “Governments have to decide what impacts they are looking for.” Questions sur- round what constitutes impact and at what point during or after the research process it should be evaluated. Can something that is subjective and qualitative ever be appropriately measured? VALUE JUDGEMENT Gluckman’s office in Auckland serves as the administrative headquarters for the science, technology and innovation stream of the Small Advanced Economies Initiative (SAEI) — a network for the discussion of challenges, opportunities and policies that are of particular relevance to small developed nations. For these countries, prioritizing the areas of science and innovation in which they invest is crucial to economic prosperity. The SAEI has begun to develop an ‘impact taxonomy’ to help categorize the range of impacts that can arise from research. These include not only the direct economic effects but also intangible factors — for example, a Nobel prizewinner’s role in enhancing a country’s scientific reputation. Gluckman says that a holistic science system has to consider all the different kinds of impact that matter to people; a taxonomy, he contends, will facilitate that discussion. “What is the ‘value’ of doing the RESEARCH IMPACT kind of biosecurity research that makes foot and mouth disease less likely or a country more resilient to an earthquake?” he asks. “This type of research could easily get forgotten if you Income for outcome focus on only one form of impact.” But measuring the breadth of impacts aris- ing from research that often takes place over Australia and New Zealand are experimenting with ways many years poses tough challenges. Assessors of assessing the impact of publicly funded research. need to identify proxies and intermediate outcomes to gauge the direction a study is taking. That is where Adam Jaffe, director of New BY BRANWEN MORGAN New Zealand. “It is what the whole of a nation’s Zealand’s Motu Economic and Public Policy science policy process is about: how much to Research think tank in Wellington, comes in. hen it comes to research, govern- allocate to public sector support; how much to Jaffe is working on an ments the world over are asking industry sector support; how much to invest evaluation framework NATURE.COM more questions about whether in heath versus relative amounts for environ- covering five categories More on measures Wthey are getting value for money — and there is ment, for instance,” he says. “Whether it is of impact: financial, quantifying research nothing wrong with that, says Peter Gluckman, done implicitly or explicitly, everyone in that social, environmen- available at: chief science adviser to the prime minister of process is thinking about impact.” tal, public policy and go.nature.com/kpjgf7

S72 | NATURE | VOL 511 | 24 JULY 2014 © 2014 Macmillan Publishers Limited. All rights reserved ASSESSING SCIENCE OUTLOOK capability. The purpose of this framework is to the real benefits of the National Science been developed, it was up to state and federal give decision-makers an idea of the full range Challenges, an economist would need to know governments to adopt it — and then for of potential outcomes and help them decide what we chose not to fund as well as what we many other actors to implement it. It will take which to track — and how. did,” he explains. “These sorts of exercises are another decade at least for the final impacts Many scientific discoveries are much more about bureaucratic box-ticking (environmental, agricultural, social and eco- serendipitous, and critics suggest that such than any real attempt to measure the value of nomic, for example) to be realized; in the a goal-oriented approach could lead to less science to society”. meantime, the researchers have moved on to blue-sky research funding. Gluckman disa- other projects. Monitoring progress towards grees, saying that knowledge advancement AUSSIE RULES impact recognizes where the researcher’s is in itself a goal. “To say impact assessment The Commonwealth Scientific and Research role is important (for example, in conduct- moves you away from basic science is to nar- Organisation (CSIRO), whose headquarters ing the work and helping communicate the rowly use the term ‘impact’.” Jaffe, an Ameri- is in Canberra, is the largest of the Australian results) and avoids any negative implications can, uses a baseball analogy to defend impact government’s portfolio-funded research agen- of assigning responsibility for behaviours assessments. “The fact that sometimes you cies. CSIRO is one of the few public research that influence the final impact yet are beyond strike out and some- and development entities in Australia — and the researchers’ control (in this example, the times you do well No single possibly the world — that formally and trans- implementation of the policies). doesn’t stop us from parently plans, monitors and evaluates the group of MAPWORK PROJECT thinking about who measures may impact of its research, according to CSIRO’s is better on aver- Mark Johnson. He is project manager of Over its lifetime, CSIRO has conducted tens age,” he says. “We indicate both Impact 2020, launched four years ago with the of thousands of projects. Johnson is working can look at which excellence aim of developing a framework to assess the on ways to show these in an easy to appreci- models on average and impact. economic, environmental and social impacts ate, visual manner. They have come up with generate the great- of CSIRO’s Flagship programmes for use across an impact map, shown on page S74, designed est outcomes and impacts across a number of the organization. For CSIRO, impact is used to be a conversation starter rather than a pre- different measures.” The inevitable random- to influence its “4As”: allocation (of resources), cise diagnostic tool. It shows 286 projects — ness in the process, he adds, “doesn’t invalidate advocacy, accountability (to government and only those that meet a minimum realized or this approach”. Using a framework such as his, other key stakeholders) and analysis (for projected financial return criterion and where which incorporates multiple measures across performance improvement). data are available to evaluate the delivered many dimensions, will minimize the chance of CSIRO uses an impact pathway model that or intended impact. Impacts can be social, missing important effects. describes a project’s inputs, activities, outputs, environmental and/or economic. Projects are Such a framework will probably be attractive expected outcomes and eventually impact — assigned a primary impact category based on to New Zealand’s recently launched National for example, the adoption of new research the same 17 socioeconomic objectives used by Science Challenges (NSCs), which provides protocols that improve productivity (See the Organisation for Economic Co-operation designated pots of money to research areas ‘Pathway to impact’). Within the Flagship and Development. deemed to be of national significance. The programme, project leaders can modify The size of the bubbles is based on a four- NSCs have a funding horizon of ten years these pathway stages depending on changes point scale that allows projects with different — part of the government’s move to support in resources and goals of either CSIRO or its types of outcome to be compared with each science over a longer term than most other clients and partners. other. The placement of the bubbles are indic- funding bodies, which typically give three- to CSIRO’s research impact planning is a ative of when projects have delivered, or will five-year grants. In their proposals, applicants dynamic process; the organization is continu- likely deliver, a significant milestone. From are required to describe their ten-year vision, ally evaluating changes beyond the bounds of 2004, the timescale changes from five- to two- anticipated outcomes and impact. a study to see whether the focus of that study yearly, as there are more data are available for Auckland University physicist Shaun Hendy is still appropriate. This eye to the wider world recent projects. Here are some examples, high- and his team have recently submitted an also helps with internal engagement. “A lot of lighted on the image. application for one NSC, called Science for people get hung-up on the long-term nature Wireless LAN: CSIRO scientists solved Technological Innovation, which covers work of most impacts, which they see as so far the main problem impeding fast wireless that “enhances the capacity of New Zealand removed from their actions today that they networking of electronic devices — that of to use physical and engineering sciences for are reticent to engage,” says Mark Bazzacco, reverberation within rooms. The organiza- economic growth”. Hendy, who is also the CSIRO’s executive manager of performance tion applied for several patents and, in 1996, director of the Te Pūnaha Matatini Centre for and evaluation. He says that “monitoring was granted a US patent for wireless local-area Complex Systems and Networks, hosted by progress towards impact” as opposed to network (WLAN, or wifi). This outcome has Auckland University, found the requirement “monitoring impact” helps scientists to led to major social and economic impacts and to discuss impact hampered his application. keep an eye on their project’s goals and get has revolutionized communication. The tech- “Our proposal covers a very broad range of a greater sense of the role they play in eff- nology is now used in an estimated five billion disciplines and industry stakeholders,” he says. ecting change. devices worldwide. CSIRO has licence agree- “Manufacturing technologies are changing so Bazzacco gives an example. Some research ments, worth more than AUS$430 million rapidly that we’ve struggled to design a research impacts are the culmination of decades of (US$400 million) with more than 20 interna- programme that will deliver short-term impact work. The Murray-Darling Basin manage- tional companies. but will also be relevant in a decade.” ment plan was a large collaborative research Murray-Darling basin plan: The Murray- Research teams that are awarded NSC fund- project that aimed to measure and model Darling is the largest river system in Australia. ing will be required to develop a ‘pathway water flows within this vital river basin. In It supplies water to approximately 10% of the to impact’ plan, complete with monitoring addition to collecting data, the researchers population and produces 40% of national and evaluation procedures. Hendy does not involved also helped policy-makers under- agricultural output. Growing demands on believe this sort of assessment is particularly stand the modelling and the plan’s findings to its increasingly variable flows have caused helpful in the long run, because it does not enable creation of new regulations concern- widespread concern among communities measure the opportunity costs. “To determine ing water usage. But once the new policy had and industry. CSIRO scientists measured the

24 JULY 2014 | VOL 511 | NATURE | S73 © 2014 Macmillan Publishers Limited. All rights reserved OUTLOOK ASSESSING SCIENCE available water within the basin and created The Cooperative Research Centres (CRCs) the value of research — they want to hear our a model. In 2011 they produced a report on programme in Canberra, set up in 1990 by the stories,” he says. “These retrospective studies resource planning, management and invest- federal government, also looks at the broader were directed solely at those who were likely ment, as well as modelling scenarios for future impact derived from its applied research. to determine whether the CRC programme catchment development, groundwater extrac- CRCs support multidisciplinary teams, which continues to get money.” tion and climatic conditions out to 2030. The often include groups from CSIRO, and are report and models directly informed govern- carefully managed to deliver impacts, says UNIVERSITY CHALLENGE ment investment; the project’s impact has been Tony Peacock, chief executive of the CRC Australia’s higher, or tertiary, education an estimated saving of at least AUS$2.8 billion Association, the CRC umbrella advocacy body. sector, which includes all the country’s univer- (US$2.6 billion) through better use of funds All CRCs include participation of end-users sities and almost two-thirds of its scientists, and water infrastructure efficiencies. from the outset, ensuring that projects are does not routinely conduct research impact Tiger prawn and Aquaculture feeds: In always addressing real-world situations. Any assessments. These institutions are, however, 2010, CSIRO announced that its ten-year proposal for CRC funding involves complet- compelled to monitor research quality through collaboration with Australian prawn farm- ing an impact tool similar to CSIRO’s impact the Excellence in Research for Australia (ERA) ers had led to successful selective breeding pathway approach. initiative (see page S64). of Black Tiger prawns with improved growth Over the past decade, there have been three In 2003, the then Liberal government started and survival rates that could be sustain- independent retrospective analyses of the developing an assessment system — called the ably farmed in salt-water ponds. The esti- CRC programme. The methodologies were Research Quality Framework (RQF) — that mated economic impact on the industry is agreed up front with government officials and looked at the impact of research as well as AUS$120 million (US$112 million) per annum. use a counterfactual point of view — that is, its quality. But in December 2007, just weeks Simultaneously, another CSIRO research they compare the impact of the CRCs with a before the RQF was due to be implemented, team developed a new aquafeed ingredient scenario in which each project had not taken an incoming Labor government scrapped it (Novacq), derived from marine microbes, place. All three studies showed that CRCs (see page S52). Matt Brown, senior policy ana- which increases Black Tiger prawn growth have had a positive impact, including on lyst in research at the Australian Technology rates by 30%. The combined value of the economic indicators such as GDP. Peacock Network of Universities (ATN), in Adelaide, increase in prawn yield is estimated to be is open about the intended target audience. believes that an opportunity to provide a use- AUS$430 million (US$400 million). “There is no problem convincing the public of ful decision-making tool was lost. “We’ve seen

Impact Categories Defence Information and Communication Services CSIRO Plant Production and Primary Products Commercial Services and Tourism Delivering positive impact Animal Production and Primary Products Economic Framework CSIRO’s National Research Flagships are taking on the biggest challenges and opportunities in manufacturing, minerals, energy, Mineral Resources Health digital services, water, agriculture, food and nutrition, oceans and atmosphere, and biosecurity. This map represent the key impacts Energy Law, Politics and Community Services CSIRO has delivered and intends to deliver. Some impacts will not happen, new discoveries will make others possible, and Australia Manufacturing Environment will ask CSIRO to respond to new challenges. CSIRO does not deliver impact alone – we work with more than 2000 Industry, Construction Expanding Knowledge Transport Government and Research partners each year to create a better future for Australia and humanity.

Tiger prawn and Aquaculture Prawns & Novel Feeds Aquaculture feeds Tiger Prawn Social Wireless local- Media Marine PAPT Analysis Incidents Hendra Secure Tools Plans Unlocking Virus Mine Food area network Pilot Coal Resources Projections Supply Vaccine Methane QEMSCAN® Test for Carbon of Ocean Chains Drainage GHG Hornless Capture Conditions Disaster Management Abatement Cattle Atlas of Program Information Systems Living CSP Technology Telehealth System RAFT Census of Australia Flinders & Zebedee Green Services National Materials House Marine Life Gilbert Square Models Quanti- Reserve from Conservation Energy APSIM Bovilis™ Agricultural Kilometre FERON® SiroLock® System Feed Shipping Policy & Rating MH Vaccine eWater Resource Array O¢shore -TBTest Assessment Stocks & Ports Adaptation Assessment Ocean EIAs Nematode Scheme SOURCE Fisheries Breath Strategies Wireless Solar Mineral Forecasting Actions Sirex Maia Management For Food Electric P@noptic Disease Systems System ECOGRAZE Bioba® Australian X-ray Strategies Footprints Producers Wasp LAN Motor Bowel Diagnosis Consumer Novel Landguard™ Hydrological Geoscience Toolkit Control Zirconia Gene Shears Cancer Strategies Crustacean Lifestyle Wireless Geospatial Data for Powders CASTvac™ Screening & Fish Feeds Programs Fabric for CRC Minerals Personalised Dung Process Positioning Diabetes & Casting for Healthy Cape Diet & Exploration Retirement Beetle System Technology Aging Water Program Grim Reditus™ Lifestyle Benet Coalscan Solospun OptiCOOL Coastal Regional Sector Baseline Barrier Grains Plan Food Cyber Future- Improved Products System Beenyup Environmental Nitrogen Assessments Image Polymer Rust Sportwool™ Screening Security proong Wheat Water Station NovoSorb™ Get Set Structure Groundwater Impact Improved & Outlooks Analysis Banknotes RHDV Paintbond Breeding Flexible Ecient Plans Resistance 4 Life of B-amyloid Technologies Technologies Assessments Colonoscopy Australia's System Rabbit Replenishment Wheat Climate Gold Industry Electronics Australia Control Indian Stormwater Trial Training Change in Telescope Electron Ocean Pest Ant Harvesting Iodine Multiplier Climate Australia Chile Novel Government & Sustainable National Eradication in the ACT Digitally Deciency Laterite Initiative Risk Report Centre of Cattle Titanium Commercial Mineral & Gas Phoslock™ Management Great Excellence Enabled Health Facility Exelgram MicroMax Feeds Green Metal Resources Treatment Geochemistry for Financial Barrier Marine System Services Smart Climate Materials Production Technology Sector Reef Asia-Pacic Innovations Agricultural Fisheries Bioregional Adaptation from Feed Urban Water Spectacle Managed National Imaging Climate Production Management Assessments Plans Services Stocks Services SIROFLOC Hydraulic Aquifer Water Use ASKAP Software for In Situ Adaptation Increases Water Potyvirus Lens Tools MIEX Frameworks Framework Atomic SIRATAC & Fracturing Recharge Eciency ASTER Alzheimer Renewable Ore Analysis Health Pollution water Absorption CottonLOGIC Taxonomy MineStar Initiative Mineral Fuels from System Sirolan-BAP Tanning Guidelines Management purication Maps Waste Planning Spectroscopy Waste Low-energy Oral Alzheimer's Tools Mining Oil-free Minimisation Longwall Emerging Aluminium Rehydration New Water Sector BCAider Australian Diagnostics & Management Vacuum Automation Flexible Infectious Production Formulation Mining GHG Climate Hairpin TEMPEST O¢shore HyLogging High Therapeutics of Extreme Operations Interscan System Marine Solar Diseases Prognostic Minerals Emissions Adaptation Biosecurity Pump AUSSAT-B RNAi Mineral Systems Ceramic Performance Climate Options Microwave RoadCrack Air Cargo Report Cells AIBL Mitigation Platform for Processing Mitigation Decision Australian Antenna Technology Locations Fuel Cells Light-weight Optical Scanner Card Study Alzheimer Optimisation Tools Tropical Landing Design Map Materials Predictive Systems Aeroguard Fibre Night and HydroDec Sustainable Water Plant ID System Nephel- Simulations Ngara Human Pre-border Day™ Contact Hydrocarbon One Second More Management in Cycloprothrin ometer of Marine ACCESS Aquaculture Wireless Coastal Settlements Decision Lens Sensor Array Digital Elevation Productive Developing Entomo- Ecosystems Breeding Technology Unlocking Observation & Tools Sirovision™ Model Wheat pathogenic Oil & Gas Information Economies SIROFLO® Smart Test Supercapacitors Healthy Great Eco- National Land Managers Nematodes Resources Systems Better New Leucaena Super- Analyser Elast-Eon™ Heart Artesian Next-G ecient Mine Marine New Participate in Management Softly® Total Advanced Environmental Cotton Toxicity conductivity OPTIM™ Program Basin Armour Management Bio-terrorism Bowel Cancer Carbon Markets of Emerging Detergent AMIRA Wellbeing HVCCC Combat Competitive Carbon Policy Solution & The Assessment Manufacture Models Test Diseases Varieties Mineral PARMS Apparel Low-emission Bauxite & Models Methodologies Absolute Volt Diet Thickener SNOMED CT Coal Alumina Past Ocean & Sustainable & GrazFeed Discovery SilviScan™ Environmental Productive Zero-waste Technologies Production Sea Level Bushre of Relenza™ Digital Management High Urban Water Salvinia Coating Building High Change Chytridio- Brain Wellbeing Toolkit Clinical Fibre Carbon Protection Biocontrol Solarhart WindScape™ Energy mycosis Remote-i Imaging Diet Decision Tonnage Capture Land Use Resource Water Linola™ Reduction Grains SIRO2 BarleyMax™ PIPEASSURE™ Drug Support Ore Sorting Technologies Knowledge Ecient Heater IBDV Omega-3 Alsecure® Global Tools Systems Australian Oxygen Poultry Air Sentinel Adaptation Seeds Pollution Cables Soil Map Grain Farms Sensor SiroSmelt Vaccine Titanium Hotspot Map Model Medical & Magnetic Broadband India-Australia Strategic Technologies Powder Adaptive Software Dairy Healthcare Resonance Applications Water S&T Metals Consumer Improving Water Use SiroCLEAR™ Flood Renewable Primary Processing Fabric Sustainable Sensor Production Partnership Industry Iron Ore & Strategies Food Security Eciency Prediction & Services Energy Industries Technologies Technology Yields Expansion Coke Options in Asia Wellbeing Tool Integration Projects & Africa LANDTEM™ Plan for South Cybernose® Low-emission Ultrabattery Kids Eastern National Platform Steelmaking Environment Smart Ocean Australian Intelligent Fisheries Bio-control of Process Climate Protection Carbon Flight Path Manufacturing Clean Assessment Weeds & Pests to Sustainable Initiative Measure Systems CRC Coatings Methods Budget Assessments Streamow Aviation Increased Protective Forecasting Gas Well Foods Telehealth Low Cost Service Browneld Productivity Monitoring & Risk GHG Colonoscopy Water Exploration Simulator Extreme Systemst National Abatement ML2.0 IBD Minerals Climate Monitoring Improved Events Tools Strategy Colonoscopy Celiac Test Friendly Training Aquatic Food Test for Ecosystems Products Alzheimer Protection & Murray-Darling HSDRA Unlocking Disease Murray-Darling Basin Plan Restoration Nickel & Copper Nanobre Lightweight Resources Component Manufacturing basin plan for Transport Australian-Korean Telescope Link-up

Integrated Catchment to Coast Models

Before 1979 1979 1984 1989 1994 1999 2004 2006 2008 2010 2012 2014 2016 2018 2020 2022 2024 2026 2028 2030 2032 2034

MAY 2014 Organizations worldwide wrestle with the issue of how to show and compare the impact of their projects. Australian national science agency CSIRO is working on an impact map. Each of the 286 bubbles represents one project with one primary category. The four sizes of circle relate to size of impact (economic, social or environmental), centred on when projects met, or will meet, a significant milestone. Larger version: go.nature.com/bduk6p. Website: csiro.au/impact.

car manufacturing suddenly disappear from

CSIRO Australia and now the country is asking what PATHWAY TO IMPACT its future industries will be,” he says. “Research Each of CSIRO’s Flagship projects is guided by this framework, which gives project leaders a way to think impact assessment would have helped guide about their work so they can plan and monitor for impact. today’s decisions.” Feedback loops Another attempt to push the research agenda came in 2012, when 12 Australian universities took part in the Excellence in Innovation for Australia (EIA) trial. The EIA INPUTS ACTIVITIES OUTPUTS OUTCOMES IMPACT e.g. e.g. e.g. e.g. change in e.g. lives saved, demonstrated how a case-study approach resources, methodology, publications, working practices, economic could be used to systematically assess the sta work reports technology benets impact of university research across a wide licences range of disciplines and areas. “The time was right to have another go at putting impact on PLANNED WORK the agenda,” says Brown. “And when trying to INTENDED RESULTS convince policy-makers, you need proof by Can be Under direct Under indirect demonstration.” In the EIA trial report, the controlled in uence in uence authors note that they met their objective to “measure the innovation dividend of research generated by Australian universities” and advanced the methodology to do so. change in government. The incumbent Liberal with fundamental research in particle physics. Australia’s chief scientist, Ian Chubb, seems party has not endorsed the plan. Assessing both types of research by their excel- undecided as to the benefits of an impact In the absence of government leadership, lence and impact “would level the playing field”, assessment system for the tertiary sector and one Australian university has decided to go he adds. University promotion committees and has concerns around the retrospective case- it alone. “We are in a different political envi- grant application reviewers could also take this study approach, as recently implemented in ronment now,” says Warren Payne, pro vice- information into consideration. the United Kingdom’s new Research Excel- chancellor in research and research training at Unfortunately, no single group of lence Framework Victoria University, Melbourne, “and we need measures may be able to indicate both the (which, ironically, “If the purpose to be ready for a measure of impact that might excellence and the impact of research. Pub- was influenced by of measuring come on top of the current ERA framework.” lishing a paper, even an ‘excellent’ one, may the aborted RQF). impact is to Individual research programmes at Victoria not have an impact outside academia without This approach is University are assessed for retrospective and additional effort — often undertaken by oth- also highly selec- tell politicians prospective impact; economic, social and envi- ers — to translate that knowledge into practice. tive, with universi- the value of ronmental outcomes are combined with quali- This raises the issue of timeframe. The wider ties submitting only research, let’s tative surveys that gather the views of people benefits of research might not appear for years those studies that own up to it.” who are socially or financially invested in the or decades, by which point many individu- demonstrate the best project on potential future impacts. The uni- als and organizations may have contributed. results. Chubb echoes Gluckman in the need versity is currently trialling the system and has How can an impact measure tease out separate to articulate the goal of any assessment system. brought in an independent assessor to evaluate contributions? “If the purpose of measuring impact is to show progress. So far, 11 projects from Victoria Uni- Indeed, coming up with a suitable meth- politicians just how much scientific research versity’s social science and science technology odology is a big sticking point. Payne has underpins sectors of the economy, then we departments have been successfully assessed discussed Victoria University’s approach with should own up to it,” he says. “But if impact via a rating scale that considers the significance the ARC, which administers the ERA assess- evaluation is to be tied to funding, then you and reach of each of the impact claims. ment exercise and is the larger of Australia’s have to be careful that you’re not just being Payne sees Victoria University’s impact tool two research-funding councils. “ARC feels fashionable. You’re not going to give a univer- as a useful way to guide its research decisions. that although our method can probably be sity funding because 15 years ago someone But he says that perhaps the greatest attribute scaled quite nicely within the university, it had an idea that turned out to be profitable.” of the system is to focus the minds of research- isn’t sure it can be done systemically,” he In an effort to ensure better coordination ers as to what expectations they are raising and says. The other funding body, the NHMRC, of activity and investment, without succumb- whether they are delivering on them. And, as requires some mention of impact in its grant ing to the whims of fashion, the Australian an added benefit, they are more motivated proposals, but the guidelines are not explicit Research Committee (ARCom), based at because they understand how their work feeds and vary by scheme. “Consensual views as the Department of Industry in Canberra and future impacts. to what can be claimed as impact and what chaired by Chubb, put together a national evidence should be provided would be research investment plan to guide govern- GROUP EFFORT very helpful,” says NHMRC chief executive ment. The plan was released in November One of the strongest arguments for assessing Warwick Anderson. 2012. As part of its plan, ARCom, which impact is the inclusion of research that falls out- So far, it is a discussion that has been dis- includes the Canberra-based chief executives side the traditional criteria for academic excel- parate — and one that, given its membership, of the Australian Research Council (ARC), the lence. “Work that previously might have been ARCom seems well placed to continue. “I’m National Health and Medical Research Coun- seen as highly applied — almost in a pejorative not saying it shouldn’t be done — or that a cil (NHMRC) and CSIRO, recommended the sense — might then be recognized as being whole bunch of smart people can’t find a way,” development of a university-focused impact important,” says Tim Wess, executive dean of says Chubb. “We just can’t afford to be simplis- assessment mechanism as a companion to the science at Charles Sturt University in Wagga tic about it.” ■ ERA. ARCom released a discussion paper in Wagga. Wess gives the example of research June 2013 and the submissions window was into changing a nursing procedure that Branwen Morgan is a freelance science writer closed in August — just in time for another reduces post-operative mortality, compared based in Sydney, Australia.

24 JULY 2014 | VOL 511 | NATURE | S75 © 2014 Macmillan Publishers Limited. All rights reserved ASSESSING SCIENCE OUTLOOK PERSPECTIVE Powering up citations BRETT D’ELTON BRETT Changing the way we measure and reward research could enrich academia and improve outcomes for society, says Alan Finkel.

ustralian science suffers from a fundamental misalignment. than peer-reviewed publication of academic papers. In STEM dis- Publicly funded researchers at universities face considerable ciplines, citation equivalents could be calculated for issued patents, pressure to generate academic papers. Taxpayers, however, commercial contracts and licence fees. More broadly, citation equiva- Awould prefer to see more significant commercial and social benefits lents could be awarded for activities including writing books, opin- from their research investment. ion pieces and government submissions, PhD student supervision, At the national level, the Excellence in Research for Australia (ERA) and development of new approaches to teaching practices or novel process, through which the federal government evaluates universi- training courses. ties, is driven by assessment of the quality of research publications. Citation equivalents earned could be counted in the same way as In the science, technology, engineering and mathematics (STEM) normal citations, even contributing to higher order measures such disciplines this assessment is based on citations — a score measur- as the H-index (a measure of a researcher’s impact and productivity) ing a published paper’s influence by the number of other papers in or institutional-level evaluations such as the ERA. Each contributing which it is cited. In the humanities and social sciences (HASS), the activity would count as equivalent to a paper with an agreed num- assessment is based on academic peers reading selected papers to ber of citations. For example, an Australian patent might be rated as determine quality. The ERA does credit other accomplishments1, equivalent to a paper with a small number of citations, say five. On the including fellowships of learned academies, patents and registered other hand, a triadic patent (which covers the US, Europe and Japan) designs, plant-breeders’ rights and research might be rated as equivalent to a paper with 50 commercialization income, but there is little citations; patents taken up and used would be evidence that assessments have given much rated more highly than ones that lie dormant. weight to such achievements. TAXPAYERS A system such as this, aiming to provide Because faculty members, departments and WANT TO SEE impact measures for individual effort, would universities want to be judged as being world be cheaper and faster than the labour-intensive class or better in the ERA assessment, they pur- methods that are needed to gauge institution- sue research for academic publications that are COMMERCIAL level impact. Case studies and expert evalua- likely to be well cited — almost to the exclusion AND SOCIAL tion panels take a long-term view, in some cases of other activities. And since evaluation pro- considering outcomes a decade or more after grammes such as the ERA affect funding and BENEFITS publication. The proposed citation equivalents, student demand, they drive academic behaviour. by contrast, would measure near-term achieve- Individual researchers realize that the path to FROM THEIR ments as soon as the impact activity is definitive, promotion is paved with academic papers, and RESEARCH for example, the issue of a patent. so rarely spend time working on anything else. Because traditional citations are global in For example, academics have little incentive to INVESTMENT. extent, citation equivalents could be consid- join in industry programmes such as the suc- ered for adoption not only in Australia, but cessful Cooperative Research Centres (CRCs) worldwide. To be successful, citation equiva- if participation limits their ability to publish extensively. Univer- lents would have to be embraced by research institutes, universities, sities and research institutes considering the appointment of an national granting agencies and, ideally, international evaluation academic who has spent years in industry often worry that the appli- programmes and databases such as Scopus, Google Scholar and Web cant’s grant-winning ability might be compromised by his or her time of Science. away from academia. Similar issues dog researchers within govern- Citation equivalents could be tested on a small scale and rolled out ment research institutes such as the Commonwealth Scientific and as experience is gained. With a little funding and some determination Industrial Research Organisation (CSIRO). we could broaden the existing publications-focused metrics to achieve These problems thwart engagement with industry, thus depriv- a better balance — acknowledging the best basic research while also ing researchers of useful commercial skills. In Australia, more than promoting the STEM research that delivers the greatest impact for the twice as many PhDs are employed in universities than in industry, society that is paying for it. ■ whereas in Germany the ratio is the other way round2. As a result, Australia was rated last in the recent ranking from the Organisation Alan Finkel is president of the Australian Academy of Technological for Economic Co-operation and Development (OECD) of university– Sciences and Engineering and chancellor of Monash University. industry collaborations3. Without such collaborations, industry e-mail: [email protected] does not fully benefit from the fundamental research undertaken in 1. Excellence in Research for Australia 2012, National Report (Australian Research academic settings, and academic researchers are not as aware as they Council/Commonwealth of Australia, 2012); available at go.nature.com/fcunry should be about market and societal needs and trends. 2. Pettigrew, A. G. Australia’s Position in the World of Science, Technology & One way to encourage academics to collaborate more with indus- Innovation. Occasional Paper Series, Issue 2 (Australian Government, 2012); available at go.nature.com/x4yp7g try would be to award ‘citation equivalents’ to various activities 3. OECD Science, Technology and Industry Scoreboard 2011 (OECD, 2011); that advance the practical impact of science through means other available at go.nature.com/n914ae.

AN ISLAND LABORATORY HELPING TO SAVE LIVES AND SPECIES ACROSS THE WORLD

he University of Tasmania has makes it invisible to the devil’s immune more than 10 years old, relatively large a growing reputation as one of system, meaning that cancer cells pass to and some distance from fished areas. T Australia’s foremost teaching and a new devil without triggering a protec- “It’s these kinds of areas that we need research institutions, ranking in the top tive response. to create and at the same time retrofit two per cent of universities worldwide and This discovery raises hope for a future the existing ones that are unlikely to ever in the Australian top ten, and leading the vaccine, but in the meantime we are reach their conservation goals,” said the world in our distinctive research themes. protecting populations of healthy devils study’s lead author, Graham Edgar. While maintaining our island identity, on isolated islands and peninsulas, as well we undertake research of international as in captive breeding facilities through- ENABLING TECHNOLOGIES AND scope, thanks to collaborative partner- out Australia. PLATFORMS: BACKWARDS IN TIME ships we’ve built across the globe, For a billion years of Earth’s history, life strengthened by a network of more CREATIVITY, CULTURE AND SOCIETY: stopped evolving and remained little than 90,000 alumni spanning more than UNDERSTANDING CONVICT LIFE more than a layer of slime. University 120 countries. University historians gained new perspec- geologists used a technique developed tives on Australia’s convict history and other for studying mineral ores to determine BETTER HEALTH: FOR BABIES AND FOR aspects of colonial life from records of the that the pause was due to a shortage GROWN-UPS 73,000 people transported to Van Diemen’s of oxygen and other elements essential Beginning in the late 1980s, University Land, gathered with the help of Oxford, for life. researchers investigated why Tasmanian Sussex and Liverpool universities. The team at the Centre for Ore Deposit babies had a relatively high risk of sud- “It’s about using Tasmania as an and Exploration Science vaporised ancient den infant death syndrome (SIDS). Their island laboratory to look at national and mineral samples and measured the discovery that sleeping on the stomach international problems,” says Hamish levels of trace elements present when is a major risk factor saves the lives of Maxwell-Stewart, leader of the Founders they formed. hundreds of babies every year. and Survivors project. “The fluctuations in these levels may This was one of the first of many These insights have led curators of help explain events like the emergence successes of what is now the Menzies heritage sites like Port Arthur to replace of life, mass extinctions, and the develop- Research Institute Tasmania. Recently, stereotypical blood-soaked interpreta- ment of gold and other ore deposits,” says its scientists have discovered genetic tions with a more complete picture of geologist Ross Large. markers for prostate cancer risk, and how colonial life. Our publication Research to Reality has vitamin D helps prevent multiple more stories of research from University of sclerosis relapse. MARINE, ANTARCTIC AND MARITIME: Tasmania, at: www.utas.edu.au/research- PRESERVING FUTURE FISH to-reality. ENVIRONMENT, RESOURCES AND Marine parks must be more than just SUSTAINABILITY: DEFEATING boundaries on a map if they’re to con- DEVIL’S DISEASE serve biodiversity, our researchers found Medical scientists from the Menzies with the help of recreational divers. Their Research Institute Tasmania are helping survey of 87 marine protected areas in 40 our zoologists study the facial tumour countries found that many were no more disease that’s devastating Tasmanian devil diverse than non-protected areas nearby. University of Tasmania populations. They’ve found a genetic Most successful were parks with a www.utas.edu.au mutation in the contagious cancer that well-enforced ban on fishing; also those

Advertiser retains sole responsibility for content ADVERTISEMENT FEATURE

SMART IDEAS FOR CANCER RECOVERY, SAVING ENERGY AND FIGHTING CRIME

dith Cowan University in Perth, of electricity, by using technology that drives without damaging their integrity Western Australia, is recognised for turns them into transparent solar panels. as forensic evidence. E research that tackles real problems, TropiGlas, being developed commer- Called SImPLE—short for Simple concentrating our efforts in areas where cially by a Perth company using research Image Preview in Live Environment—the we have a proven track record. from the University’s Electron Science software was designed to be both Our research institutes—including Research Institute, is an “intelligent glass” effective and easy for officers to use with Health and Wellness, Electron Science technology that converts the ultraviolet only minimal training. This contrasts Research and Security Research—and and infrared components of sunlight with the currently used sophisticated, our close links with nine other Australian into electricity while allowing visible multi-purpose software, which can universities are part of our commitment to light to pass through. be operated only by a central unit of a vibrant research culture and strong sup- A hair-breadth, nano-engineered expert technicians. port for our staff and student researchers. film sandwiched between two sheets Doing the analysis on-site also of glass selectively diverts the energy enables a quicker response for identify- WORKING OUT BETTER LIFE FOR of this radiation to photovoltaic cells ing and rescuing victims of abuse, as CANCER PATIENTS at the edge of the pane. Although well as providing evidence needed to Toxicity in cancer treatment leads to a not yet as efficient as traditional solar bring a suspect into custody or elicit number of well-established musculo- panels, TropiGlas is able to cover a a confession. skeletal deficits, as well as osteoporosis much larger area than a rooftop instal- In addition to harnessing the exper- and bone fractures that can substantially lation, effectively turning buildings into tise of the Institute’s researchers, the reduce quality of life, physical function power plants. development of SImPLE used the skills and independence. But research by the “We believe that within a couple of of University students, giving them the University’s Health and Wellness Institute years we should be able to double the chance to contribute to a project with has shown that exercise can act as medi- efficiency, which is very good consider- substantial community benefits. As the cine to significantly reduce these risks and ing we’re not using visible light,” says project moves into its beta stage, the help cancer survivors recover. Institute director Kamal Alameh. students will continue to support the Their studies on prostate cancer patients The first commercial trials of the software and keep it up to date in a found clinical benefits of resistance training product are taking place in South Africa, rapidly changing field. for reversing muscle loss and improving where TropiGlas is being installed in Find out more about our work in all physical function and quality of life. a new government office building fields atwww.ecu.edu.au/research . This approach is a big change from in Pretoria. historical practice, in which clinicians advised cancer patients to rest and avoid SIMPLE SOFTWARE FOR CYBER activity. However, exercise guidelines SLEUTHING co-authored by the Institute have now Some of the most serious modern been adopted as the prescribed model crimes are also the most technically for cancer management in North America challenging to police, requiring the and much of the world. analysis of suspects’ computers for illegal images such as child pornography. WINDOWS FOR A CLEAN ENERGY FUTURE But the University’s Security Research The vast area taken up by windows on Institute has produced software that lets buildings could soon become a source police view photos and videos on hard

Advertiser retains sole responsibility for content ADVERTISEMENT FEATURE

KNOWLEDGE LEADERSHIP FOR A BETTER WORLD

anked in the world’s top 100, The world’s largest mining companies as University of Queensland (UQ) is best practice for active slope monitoring Rone of Australia’s leading teaching and research universities, and has ♦♦conservation planning software educated more than 210,000 alumni— (Marxan) used in more than 100 coun- including over 11,000 PhDs—who have tries to support the design of marine made and continue to make positive and terrestrial reserves impacts throughout the world. UQ’s excellence in research has been ♦♦innovative treatments for diseases translated into positive impact for people linked to autoimmune, neuropathic and communities worldwide, with many and inflammatory conditions, which economic, environmental, health and are being translated by UQ biotech- social benefits. Leading outcomes include: nology start-ups including Spinifex Research conducted at UQ by Professor and Dendright. Ian Frazer AC (pictured) and the late ♦♦the world’s first vaccine against cervi- Dr Jian Zhou on virus-like particles led cal cancer, which has benefited tens To bring these and many other innova- to the development of the world’s first of millions of women worldwide, tions to the global market, UQ’s excel- cervical cancer vaccine. and has the capacity to save an esti- lence in research has been enhanced mated quarter of a million lives annually by the University’s commercialisation companies: UniQuest, which ranks in ♦♦the Triple P – Positive Parenting the world’s top 10 per cent of such assessment. In this benchmarking Program, which has reached more companies, and JKTech, which specialises exercise, all of UQ’s research fields were than seven million children and their in commercialising resource industry- ranked as world standard or above. families in approximately 25 countries, related innovations. With a focus on excellence in and has been translated into more than This translation of research excellence discovery, learning and engagement, 20 languages to real impact for society defines the UQ’s record of success has far-reaching “excellence-plus” culture embedded at impacts. We are committed to “excel- ♦♦signal correction technology used UQ. We particularly seek to partner with lence-plus” to enhance UQ’s positive in approximately two-thirds of the industry, government and non-govern- worldwide impact. world’s magnetic resonance imaging ment organisations to enhance these (MRI) machines outcomes. In 2012, UQ received more research funding from non-government ♦♦genome mapping studies that have sources than any other Australian univer- greatly increased our understanding of sity, and UQ consistently features among the genetic causes of chronic, debilitat- the top few Australian universities for ing conditions such as ankylosing research funding from government. spondylitis, osteoporosis, tuberculosis, UQ’s success is underpinned by leukaemia and rheumatoid arthritis the quality of its people and research excellence, which is evidenced by the ♦♦mine safety technology (GroundProbe) Australian Government’s Excellence

TOP RIGHT IMAGE COURTESY OF GROUNDPROBE ® GROUNDPROBE OF COURTESY IMAGE RIGHT TOP used to preserve lives in some of the in Research for Australia (ERA) 2012 www.uq.edu.au

Advertiser retains sole responsibility for content ASSESSING SCIENCE OUTLOOK PERSPECTIVE If not funding then teaching The lack of financial reward from Australia’s national system of research

BELINDA PRATTEN/ANU PRATTEN/ANU BELINDA assessment is obscuring the real issue, says Brian Schmidt.

hen he announced the formation of the Excellence in can provide a benchmark to gauge the effect on research quality. The Research for Australia (ERA) initiative in 2008, Kim results, positive or negative, can be used to inform policy decisions Carr, then minister for innovation, industry, science and around the impact of the decision to deregulate fees as well as other Wresearch, said that it would provide a “transparent, workable system reforms proposed in the future. to assess the quality of home-grown research”. Carr strongly hinted But how much influence can the ERA continue to wield once the that future funding decisions affecting higher education institutions HERD sector realizes that the total pool of money on the table is tiny? would be informed by outcomes of the ERA, which would collect data Some commentators have said that, without significant funding flow- and rate the quality of their research output. ing from ERA rankings, the programme is not worthwhile — but this For a comparable model, Australian universities looked to the is not where the problem lies with funding for HERD. equivalent system in the United Kingdom, the Research Assessment A far more sensible system is one that contributes towards the full Exercise (RAE), which allocated significant fractions of the available cost of research as part of the granting process, as happens in the research money to universities on the basis of their RAE scores. The United Kingdom, United States and Canada, for example. An assess- implication for Australia was that a poor score would lead to financial ment system like ERA would then give additional strategic money disadvantage. to help institutions do even better research, at a level in line with Six years later the sector is gearing up for the third instalment of current funding. Unfortunately, Australian grants provide nowhere the assessment process. But during this time, near the full cost of research; significant cross- the money that was supposed to be tied to ERA subsidization is required from student fees. This outcomes has all but vanished. The incen- undesirable method of research funding is unfair tive structure to award the money is in place: A FAR MORE to students who believe they are paying for their on a scale of one to five the two lowest ratings education but are in fact paying for the country’s attract nothing, whereas the top rating (five, SENSIBLE research. or ‘well-above world-standard’) earns seven SYSTEM IS ONE THAT As a fraction of GDP, Australia spends more times that of the middle rating (three, or sim- on research within higher education than most of ply ‘world-standard’). But the total amount of PROVIDES THE the countries in the Organisation for Economic money available is trivial compared to the over- Co-operation and Development (OECD), but all budgets of the participating universities. In FULL COST its overall rate of R&D investment is well below fact, ERA financial reward accounts for only OF RESEARCH the OECD average. It is therefore important that 1.2% of Australia’s investment in higher educa- Australia maximizes its returns from research tion research and development (HERD), or just IN THE GRANTING within the higher education sector. ERA has over AUS$116 million (US$109 million). successfully emphasized research quality, but Given that the government has spent PROCESS. this is against a dearth of assessment on how our AUS$43.5 million on the ERA, and universi- universities interact with industry (see page S77). ties themselves have outlaid substantial sums to Given this set of incentives, it is perhaps unsur- undertake the ERA evaluations since 2008, one might question the prising that although our universities’ research outputs are ranked value of this exercise that awards so little money. eleventh in the world, Australia was ranked last for business collaboration with higher education and public research agencies within the NON-FINANCIAL INCENTIVES OECD. Australia needs to invest more in R&D, but without a strategic Nonetheless, it is clear that ERA has helped influence the AUS$9.6 bil- plan to achieve educational and business outcomes in tandem with lion invested annually in HERD. By focusing its assessment on excellence in research as captured by ERA, our country will not fully research quality, rather than quantity, the ERA has helped elevate benefit from its investment. the research at many of Australia’s universities (see page S67). The The next round of ERA evaluations is scheduled for 2015. And sector is now strategizing about research quality — and these plans although the ERA has been worthwhile, it is unclear how much is to are manifested in new initiatives across various universities. There is be gained by undertaking this formidable exercise again so soon. Not evidence that excellence is being recognized and rewarded as one of much has changed in the past three years in the Australian HERD a series of outcomes. sector, so this triennial exercise — as it stands — seems too frequent. ERA’s impact has gone beyond universities and helped to measure On the other hand, if ERA can spur the government to strategically Australian capabilities against benchmarks across the breadth of the plan its research agenda, then supporters and naysayers alike would HERD sector. This is a useful exercise that should help Australia invest rejoice in being assessed as often as is deemed necessary. ■ more strategically in research in the future — a necessity thrown into sharp relief by the 2014 budget, which introduced big changes for the Brian Schmidt is professor of astronomy at the Research School of higher education sector. Among other items, the budget removed the Astronomy & Astrophysics, Australian National University, Weston cap for university tuition fees bringing potentially profound implica- Creek, Australian Capital Territory. tions for the higher education sector. In this instance the ERA process e-mail: [email protected]

a rough approximation after the last PBRF round on the costs of the process to universi- Q&A Jane Harding ties; our best estimate was less than 3% of the total PBRF income over the six-year period, which is not a huge overhead.

Individual approach What is the grading experience like for researchers? Jane Harding is deputy vice-chancellor for research and professor of neonatology at the University From an institutional point of view, the of Auckland, which is New Zealand’s most well-funded university under the Performance-Based distribution of individual grades — A, B, C or Research Fund. She discusses the country’s approach to assessing science and measuring impact, R (for research inactive) — has little overall and describes why she prefers a model that grades the individual not the research group. effect on us in terms of dollars. If one of our researchers is awarded a B instead of an A, that is usually balanced somewhere else in the What do researchers in New Zealand think stages are mostly about preparation and edu- institution by somebody getting an A instead about the Performance-Based Research cation — making sure that everybody knows of a B. Such variations don’t make much of Fund (PBRF)? what they have to do. Then we get into the a difference to the profile of the institution, Most researchers see the PBRF as an inevi- phase of assembling individual portfolios. We but they make a huge difference to the table chore. Still, although they find it time- run an internal review of the draft portfolios individuals. Getting an assessment of a B consuming and distracting to prepare before they get polished for final submission, when you thought you might have been a portfolio, they see some advantages in and when the assessment is completed, we eligible for an A is a huge disappointment. having something that provides an exter- manage the results and make sure that they People will inevitably interpret the grades as nal validation of research performance on a get to the right people. defining something about themselves; you regular basis — in our case, every six years. It is a very big, complicated and continuous can’t stop them from taking it personally. Although many researchers believe that the process involving a lot of human resources PBRF is a rather imperfect measure of research work: we have close to 2,000 portfolios to sub- Have any categories of researchers been quality, it is, by and large, a useful thing. It mit by a specific date. At its peak it can occupy disadvantaged by the assessment? would be difficult to find an alternative way about half of my time for two to three weeks. The assessment is experience-dependent, to distribute the money to the most research- which makes it difficult for a junior researcher intensive groups. And there is evidence that This seems like a significant investment for to get an A grade. A brand-new postdoc- the PBRF has improved the quality of research. the university. toral researcher is not going to have a strong Yes — but it is also very important to us. research portfolio. The PBRF has a category What is your involvement with the PBRF? The PBRF contributes about 8% of our for new and emerging researchers, in which I am responsible for running the whole process total budget, which is a significant chunk of the threshold for getting a C is much lower, within the University of Auckland. The early funding. Universities in New Zealand did and this does mitigate some of the disparity.

You could even argue that researchers with less experience benefit more from the PBRF because there is assistance at an institutional level: the PBRF provides institutions with funds to specifically support supervision for research degrees. In our university, the PBRF UNIVERSITY OF AUCKLAND has contributed to an increase in support for less-experienced researchers.

Does the PBRF have a bias against specific research fields or types of output? That is what everybody worries about, but I don’t think it is a reality. You have to trust that the reviewers can assess the different kinds of research in an appropriate way. A few concerns have been raised about the criteria for ‘world-class’ research, which could disadvantage disciplines focused on The University of Auckland was allocated more than NZ$72 million from the PBRF in 2010. indigenous research in local communities. But any bias would result from a miscon- be interested in coming to New Zealand and been written. The system is also expensive, ceived equating of world-class research with accepts our salary levels, he or she is often because each individual portfolio in New international research. You can do world-class discouraged by the lack of research funding Zealand needs to be prepared and assessed. research on New Zealand topics. and so might choose not to come here. That is a A metrics-based system is much cheaper There is also a minor concern that major disadvantage. More funding through the and simpler to run and would come out some types of research, for example those PBRF would help to support more research, with almost the same outcomes if one was involving commercially relevant work with but would not make up for the serious under- simply talking about allocation of the money private companies, might be discouraged funding across the sector. and alignment of because it doesn’t necessarily result in a report “Researchers institutional goals that can go into a portfolio. How does New Zealand’s approach compare with less to research quality But overall, I don’t think that the PBRF with other peer-review-based models such as experience objectives. But New has changed the nature of scientific inquiry. in the United Kingdom? benefit because Zealand’s approach of It does create pressure to produce outputs, The difference in New Zealand is that the the PBRF submitting individual which means that people who would like to assessment is done at the level of the individual provides funds portfolios brings the sit and spend 15 years writing a book are not as opposed to the research group. Arguably, an to support incentives back to going to do well in the PBRF, but they won’t individual-based system could lead to selfish each individual staff do well in any environment that is focused on behaviour because there is no direct incentive supervision.” member in a much research quality. to work collaboratively and to support a team more direct way than or more junior researchers. does submission of metrics at the institutional Overall, more than 10% of funding to In New Zealand, this is counterbalanced level. There is also the issue that the metrics universities in New Zealand comes from by requiring that portfolios include not only themselves, rather than research quality, can the PBRF. This is much lower than the 25% research outputs, but also evidence for the become the target — and the lack of peer allocated by a similar programme in the section called Contribution to the Research review means that different disciplines might United Kingdom, but higher than the 2% Environment — a category that covers activi- be differentially treated. allocated in Norway. Do you think that enough ties such as engaging in peer review activities, money is distributed through the PBRF? leading collaborative groups, supervising post- Should the PBRF include measurements of The amount distributed through the PBRF — graduate students and mentoring early career impact, similar to the United Kingdom’s new NZ$262.5 million (US$224.2 million) in 2013 researchers. It would be very difficult to apply Research Excellence Framework? — is enough to provide a significant incen- the individual portfolio model to a larger sys- It is challenging to get a single system to tive but not enough to cover the costs of the tem because of the scale of the assessment. measure two different things. If you consider research that it is designed to support or to I prefer the individual model. The UK research quality and research impact to be ensure the highest quality research. However, process requires gathering all of the individ- separate things, then you need separate the international comparison comes down ual material and then assembling that into an processes. It is difficult to measure impact — to how other components of the system are aggregate submission, so it seems to be a lot and expensive in terms of the effort required funded. Universities in New Zealand are seri- of additional work for not a lot of additional to assemble the evidence. ously underfunded by any measure. We have gain. My colleagues in the UK talk about their Introducing new measurements would one of the lowest funding rates per student universities employing people full-time just to be useful if they created incentives for aca- compared to other OECD [Organisation for write the submissions. demics to increase the impact of their Economic Co-operation and Development] research, but impact is so closely related to countries. And our expenditure on research And how does it compare with indicator-based research quality that I am unconvinced as and development as a proportion of GDP is models like those in Denmark and Australia? yet that a separate assessment is worth the about half the OECD average. Any peer-review process is vulnerable to the enormous cost. We will learn from the UK’s As a result of this shortfall, researchers in vagaries of individuals. Assessments can vary attempt to assess impact on a national scale. ■ New Zealand spend a substantial amount of based on who is on the panel, how well they time seeking funding. We also face difficulties know the subject, their own personal preju- Interview by Smriti Mallapaty, assistant in recruiting and retaining good researchers. dices, as well as many other unquantifiable editor for Macmillan Science Communication Even though a world-leading professor may factors such as how well the portfolios have in Tokyo.

Achieving positive impact together

At CSIRO we shape the future. We do this by using science to solve real issues. Our research makes a difference to industry, people and the planet.

Connect with us: FOOD & NUTRITION | MANUFACTURING | LAND & WATER | MINERAL RESOURCES | OCEANS & ATMOSPHERE

An open letter from Dr Megan Clark, Chief Executive CSIRO

Dear valued members of the international research community,

Responding to the major scientific challenges of our time takes an agile and considered approach, as well as the ability to change and evolve.

Indeed, embracing change is essential for CSIRO to continue to be one of the top ten applied research agencies in the world and Australia’s most trusted name in science and technology.

In order to make it easier to achieve common goals for long-lasting benefit we have streamlined CSIRO into nine National Research Flagships.

These National Research Flagships allow us to focus on the biggest challenges that face our nation and the globe. By taking a multidisciplinary approach, we can bring to bear the expertise we need to undertake groundbreaking scientific research.

We’re creating an operating environment where we can work with you, the international research community, to create positive impact within and across our areas of specialty, and on the world around us.

This is an exciting time for our research. Our doors remain open for researcher exchanges and visits, joint ventures, collaboration agreements, joint publications, joint forums for knowledge exchange and co-investment. We want the best minds in Australia working with the best minds in the world to solve the greatest challenges of our times.

We look forward to continuing to make a positive impact together.

Dr Megan Clark

[email protected] | csiro.au http://www.nature.com/nature/outlook/assessingscience