OFFICE OF THE PRIME MINISTER’S SCIENCE ADVISORY COMMITTEE

The role of evidence in policy formation and implementation

A report from the Prime Minister’s Chief Science Advisor

September 2013 “[An] important role that public servants play is to help Ministers and the community in general to understand the options and choices they have. It is too easy, perhaps even negligent; to leave Ministers to make decisions with insufficient information, without the best possible evidence, and without learning from what has gone before. And the point here is that there is rarely something where the issues are clear-cut, or where choices don’t have to be made.” “The policy advice that informs these decisions must be built on a strong foundation. We have to make sure what looks like a good policy idea is backed up by solid evidence and quality analysis.”

Excerpts from a speech by Gabriel Makhlouf, Secretary to the Treasury, April 2013 Part of the Better Public Services Initiative of the State Services Commission

Office of the Prime Minister’s Science Advisory Committee PO Box 108-117, Symonds Street, Auckland 1150, New Zealand Telephone: +64 9 923 6318 Website: www.pmcsa.org.nz Email: [email protected]

ISBN 978-0-477-10404-3 (paperback) ISBN 978-0-477-10405-0 (PDF) The role of evidence in policy formation and implementation

Letter to the Prime Minister

September 2013

The Prime Minister Rt Hon John Key Parliament Buildings WELLINGTON 6160

Dear Prime Minister

Re: Evidence in the formation and evaluation of policy

You have asked me to advise on how New Zealand’s ministries and agencies might improve their use of evi- dence in both the formation and evaluation of policy. I now present my report The role of evidence in policy formation and implementation which contains recommendations on I how I consider this might be achieved. This report builds on my previous discussion paper from 2011, entitled Towards better use of evidence in policy formation, which summarised the complex relationship between evidence and its application to policy. Since that time, and at your request, I have continued to reflect and consult both nationally and internation- ally on these issues. To further inform my thinking, and with your agreement and the assistance and support of the Department of Prime Minister and Cabinet, my Office undertook a survey of a number of government ministries, depart- ments and agencies to assess how they perceived the role of research-informed evidence in their work. This survey and its conclusions have confirmed my earlier thinking and have allowed greater focus on identifying solutions. A summary of the major findings of that survey is appended to my report. These findings demonstrate a wide and rather inconsistent range of practices and attitudes toward evidence across government agencies. Not all are bad. Indeed, there are examples of exemplary practice. Some of the promising initiatives that your Government has made are highlighted. Nevertheless, the variability suggests that a more systematic and whole-of-government approach would be desirable. Similarly, the quality of assessment and evaluation of policy implementation is quite variable. The required scrutiny can be devalued by agencies that assume their primary mandate is to implement political decisions. As a result, funding for evaluation is frequently trimmed or diverted. For instance, the concept of controlled trials in public policy implementation is well accepted in other jurisdictions but this has not been generally promoted by ministries in New Zealand. Internationally there is a consensus that research, science (very broadly defined) and technology are playing a much greater role in identifying, mitigating and communicating risks, and maximising opportunities for governments to advance their nation’s interests. It is noteworthy that within the global conversation, there is growing recognition of the critical need to be more rigorous both in the employment of evidence for the development of policy, and in the assessment of its implementation. Yet there are also dangers if clear pro- tocols are not in place to manage the processes of incorporating research-informed evidence into the policy process. My report highlights these as well. Page 3 The role of evidence in policy formation and implementation All of this occurs within a complex and uncertain environment where human responses and decision-making are influenced by many factors other than well-informed knowledge. Indeed, as I have stated previously, my view is that quality evidence should be seen as base knowledge on which, in a democracy, multiple values and associated perspectives must be overlaid. However, where evidence is conflated with values, its power is diminished. Where evidence is not considered properly, the risk of less than desirable policy outcomes is inevitable. For instance, it is possible for the research process to be corrupted by inputs that are not objective, or by the failure to recognise personal biases in bringing forward evidence. Researchers can become impassioned ad- vocates for a cause that their expertise could meaningfully inform dispassionately. Similarly, where evidence for policy is generated externally, the science community can fail to understand how the policy (or indeed the political) process works. Scientists can be naïve in assuming that policy follows directly from evidence. There are standard processes (supported by relevant skill-sets) that must be in place to ensure that such subjectivity and bias are minimised. My analysis suggests that these skills and processes can be overlooked in political and policy domains. Lobbyists can easily misconstrue science and exploit such a situation. The role of appropriately appointed science advisors within departments can ensure appropriate processes and mitigate against these risks. It is also concerning that in New Zealand, there has been insufficient attention paid to proactive investment in research needed to support policy formation. For at least the last 20 years, our public research funding bodies have not prioritised policy-relevant research, resulting in a disconnect between central agency needs and funded research priorities (to the extent that these are set). In turn, this has led to a growing gap be- tween the research community and the policy community in identifying the needs of the policy community that research could address. My report makes a number of low-cost suggestions for your consideration, some of which were heralded in my earlier paper. In particular, I recommend that protocols be established for how scientific advice is sought and incorporated into the policy process. Secondly, I recommend the appointment of science advisors to major departments to address multiple functions associated with enhancing departmental use of evidence in policy formation and evaluation. In turn, this community of science advisors could play important roles in technology assessment, research need identification, risk identification and assessment across government. It is important to note that strengthening and assuring the use of research-informed evidence in policy ad- vice in no way weakens the authority of the political process. On the contrary, it must strengthen it. After all, political processes are about making difficult choices based on a range of complex options where there are inevitably trade-offs and spill- over effects – both good and bad. I thank you for your encouragement to consider this important matter.

Yours sincerely

Sir Peter Gluckman KNZM FRS FRSNZ

Page 4 The role of evidence in policy formation and implementation Contents EXECUTIVE SUMMARY...... 6 INTRODUCTION...... 6 PART 1 – Evidence and its application within the New Zealand policy environment...... 8 1.1. Context...... 8 Situating this report...... 8 Re-aligning the Government’s policy process toward the more systematic use of robust evidence.....9 1.2. The nature of knowledge and evidence...... 10 How evidence is generated...... 10 Epistemology: ways of knowing...... 11 1.3. Science and values...... 12 1.4. Social science...... 12 Evaluating social programmes: applying the science of what works...... 13 Promising practices in New Zealand...... 14 1.5. Typical avenues by which evidence is incorporated into policy formation...... 14 1.6. (Un)informed public discourse...... 16 Understanding and communicating risk...... 16 1.7. A stock-take of evidence-informed policy practices in New Zealand...... 17 Promising initiatives on the road to improvement...... 18 Grounds for concern...... 18 PART 2 – Suggested steps to enhance the use of evidence in New Zealand’s policy framework .....19 2.1. Setting standards...... 19 2.2. Science leadership...... 20 2.3. Long term planning, risk assessment and evaluation...... 22 2.4. Government funds towards policy-relevant research...... 23 2.5. Transparent and accessible information for public consultation...... 23 CONCLUDING COMMENTS...... 24 LIST OF APPENDICES...... 24

Page 5 The role of evidence in policy formation and implementation

EXECUTIVE SUMMARY Key Recommendations At the request of the Prime Minister, this report See full description of recommendations in Part 2. has been designed to explore in greater detail the 1. Develop a standard set of protocols across issues that were brought to light in an earlier dis- government regarding obtaining expert sci- cussion paper, Towards better use of evidence in entific advice; policy formation (2011). This paper extends that 2. Extend the use of Departmental Science discussion and makes some specific suggestions as Advisors (DSAs) more broadly across gov- to how to improve the use of robust evidence in ernment; policy formation and evaluation. 3. Use the community of DSAs and the Chief The report is partially informed by a survey that my Science Advisor to assist central agencies Office undertook in 2012 to assess the knowledge, with longer-term planning, risk assessment attitudes and practices of public servants toward and evaluation; the use of research-informed evidence in policy 4. Improve and make more explicit the use formation. A number of government agencies were of government funds for research to assist selected to take part in the study, which comprised policy formation; staff surveys, key informant interviews and docu- 5. Provide greater transparency regarding the ment analysis. use of research-informed data (or its ab- sence) with respect to complex and contro- Findings of the study pointed to a high degree of versial areas of decision-making where the variability across the New Zealand public service public is directly or indirectly consulted. with respect to the understanding and application of robust evidence for policy formation and the evaluation of policy implementation. While there INTRODUCTION were examples of promising attitudes and practic- Effective decision-making requires good advice, es, my report suggests approaches that would help and that depends on informed use of evidence to bring the New Zealand policy environment in both in developing policy and in evaluating its effect once implemented. In this way the value of gov- line with current international practice. ernment’s performance to the benefit of citizens Recommendations are discussed in full in Part 2 of is maximized. However, the relationship between this report (page 19). They include: evidence and policy formation is neither linear nor 1. Develop a standard set of protocols across unidirectional. Policy formation is a complex pro- government regarding obtaining expert scien- cess, in which many factors other than evidence tific advice; need to be brought to bear. The complexity of policy formation stems from at 2. Extend the use of Departmental Science Advi- least two issues. The first is the need to balance the sors (DSAs) more broadly across government; many inputs into the process (such as rigorous anal- 3. Use the community of DSAs and the Chief ysis of a problem, analysis of social values, analysis Science Advisor to assist central agencies with of political context, and analysis of economic im- longer-term planning, risk assessment and pacts). Secondly, the process is further complicated evaluation; by the variability of uptake capacity and the appe- tite for such information by policy makers. There 4. Improve and make more explicit the use of is not always the culture and capability within the government funds for research to assist policy public service to seek out appropriate evidence formation; and to critically appraise and apply it to a policy 5. Provide greater transparency regarding the question. use of research-informed data (or its absence) This complexity has been at the heart of a major with respect to complex and controversial global shift in policy making within democratic so- areas of decision-making where the public is cieties over the past 15 years. The concept of ‘evi- directly or indirectly consulted. dence-based policy making’ began to gain currency Page 6 The role of evidence in policy formation and implementation throughout the 1990s, with the UK Cabinet Office clearly adopting it as a motivating philosophy in its Note On Terminology 1999 white paper Modernising government.1 In this report, I variously refer to ‘science’, ‘re- search’ and, in their more applied form, ‘evi- Since then, scholars have added nuance, remind- dence’. I am aware that these terms can have ing us that policy must take into account both ro- different interpretations and that knowledge bust evidence derived from research, as well as an production in science should not and cannot understanding of social values. Hence, in its more claim to be the production of objective ‘truth.’ contemporary iteration, evidence-based policy However, the role of science is to provide pro- has been more accurately re-cast as ‘evidence-in- cesses that significantly reduce subjectivity, bias formed policy’, which is not to diminish the role of and uncertainty in our understanding of our nat- research-informed evidence. To the contrary, it has ural, built, and social environments. helped to fully establish a particular place for ro- Thus, when I use the terms ‘science’ or ‘re- bust evidence among the multiple inputs into pol- search,’ I am referring to formal processes that icy formation. use standardised, systematic and internationally The present report is concerned primarily with bet- recognised methodologies to collect and ana- ter ensuring that the highest quality research-de- lyse data and draw conclusions. rived evidence informs policy development in When I refer to science I am including the biolog- New Zealand. In doing this, it acknowledges that ical, natural, mathematical, engineering and so- such evidence is not context-free. Rather, from the cial sciences; the methodological commentary is methods used to generate it to the way it is applied equally applicable across all of these domains. in practice, the use of evidence for policy formation I use ‘evidence’ to mean robust and verifiable must be viewed both as a product of, and in rela- knowledge, derived from the processes de- tionship to, its cultural milieu. scribed above and used to establish the case for The challenge, as outlined in this paper, is to build a a specific policy response. public service culture that has the attitudes, capa- These definitions apply throughout the report. bilities and internal processes to support the gener- ation and use of quality evidence derived through • Enabling practices and administrative infra- the formal processes of research. The interface be- structures that can accommodate robust pro- tween science and policy is, after all, an interface; cesses of data collection and analysis, whether it demands as much capability from knowledge intra- or extramurally. providers as it does from knowledge users. That is, The discussion that follows is divided into three there must be a culture of policy making that, as a matter of course, recognises the need for rigorous sections: evidence to justify policy directions and decisions. Part 1 is a general overview of issues involved in ap- As with any organisational cultural shift, this pro- plying evidence to the policy process. This section cess requires: highlights both the practical and philosophical chal- • Skilled leadership of experts embedded within lenges inherent to the process, and relates these to government departments; the New Zealand context. • Access to well-developed scientific and re- Part 2 makes specific recommendations aimed at search expertise outside of government; improving the quality of, and mechanisms for the • Expertise in translation (‘brokering’) between use of, evidence in policy formation in New Zealand. researchers and policy experts; The Appendices offer specific detail on: (1) results • Concerted efforts to lift capabilities within pub- of an assessment undertaken by the Office of the lic service communities of practice so that there Chief Science Advisor regarding attitudes to and is capacity to evaluate such evidence without use of research-derived evidence in policy forma- bias and with rigour; tion; and (2) examples of how recommendations

1 Cabinet Office (1999) Modernising government, Stationery similar to those herein have been applied in other Office: London, Ch. 2, paragraph 6. jurisdictions. Page 7 The role of evidence in policy formation and implementation PART 1 – Evidence and its application reliability and the utility of what they know, while within the New Zealand policy policy makers may underestimate what they do not environment know, as well as the value of objective evidence.4, 5 As a result, democratic institutions are increasingly 1.1. Context seeking ways to improve the use of evidence-based As observed in my earlier discussion paper, To- inputs in the processes of policy formation and im- wards better use of evidence in policy formation2, plementation. The United Kingdom has made the it is generally recognised that the relationship be- most extensive use of formal processes to increase tween evidence and policy formation must be in- the use of evidence in policy formation – these teractive. It is rare indeed that any single piece of have been developed through the office of their data is sufficient for a policy shift. Instead, it can Chief Scientific Advisor and assisted by the estab- be argued that the role of evidence is to ‘nudge’ lishment of the positions of Departmental Science the complex and iterative policy formation process Advisors. In the United States, the Department of in certain directions. As others have argued, this Health and Human Services Translating evidence ‘nudge’ can be further broken down into five key into practice (TRIP) Initiative is another example tasks for promoting better use of evidence in pol- of government effort in this regard. Similarly, the icy making: 1) identifying problems; 2) measuring Campbell Collaboration6 demonstrates the global their magnitude and seriousness; 3) reviewing pol- interest in evidence-based policy making. This in- icy options; 4) systematically assessing likely con- ternational organisation conducts and maintains sequences of options; and 5) evaluating results of a database of systematic reviews of the effects of policy interventions.3 social interventions. It is accessible to researchers Thus, while it is understood that policy decisions and policy makers alike. in a democracy are inevitably largely based on val- What these programmes cannot do, however, is ues domains such as fiscal priorities, affordability, to provide any assurance that they will be used by public opinion, political ideology and electoral con- policy makers. Thus, while the evidence may be siderations, effective policy formation must also available, its application is a matter of preference, incorporate the use of evidence as a fundamen- personal judgement and organisational culture. For tal underpinning. Without objective evidence, the this reason, internal scientific leadership is impor- options and the implications of various policy ini- tant within government agencies in order to assist tiatives cannot be measured. When this happens, knowledge users to appreciate the value of and op- judgment can only be on the basis of dogma, belief timally use research-informed and robust evidence or opinion. This cannot provide useful estimates in the development of policies and programmes. of the magnitude of any desired policy effect or of any spill-over benefits and risks. Increasingly it Situating this report is acknowledged that, in areas where complexity makes policy formation particularly challenging, This report builds on the discussion paper entitled programmes that are implemented in response to Towards better use of evidence in policy formation government decisions require ab initio and planned (2011). It also provides some follow-up (albeit fo- evaluation to ensure that the desired effects of the cused explicitly on science advice) to the report of policy are being realised. the government-appointed committee chaired by Dr Graham Scott entitled Improving the quality and Where evidence for policy formation is generated value of policy advice (2010). outside of government, there is inevitably the po- tential for some tension between those engaged in 4 Cullen, P. (1990) The turbulent boundary between water sci- policy making and researchers engaged in knowl- ence and water management. In Freshwater Biology Vol 24, pp. 201-209. edge production. There can be cultural divides and 5 Haynes, A. et al. (2011) From “our world” to the “real world”: quite different understandings of the policy devel- Exploring the views and behaviour of policy-influential Austral- opment process. Researchers can overestimate the ian public health researchers. In Social Science and Medicine, Vol. 72, pp. 1047-1055. 2 Gluckman, P. (2011) Towards better use of evidence in policy 6 The Campbell Collaboration is an international research net- formation: a discussion paper. Office of the Prime Minister’s work that produces systematic reviews of the effects of social Science Advisory Committee. interventions for use by policy makers and programme manag- 3 Prewitt, K. et al. (2012) Using science as evidence in public ers in education, justice, social welfare and international devel- policy, National Academies Press, p. 4. opment sectors (www.campbellcollaboration.org).

Page 8 The role of evidence in policy formation and implementation While the Scott report was concerned primarily It is against this backdrop, and with the support with government expenditure regarding policy ad- of the Prime Minister and of the Chief Executive vice, it made useful recommendations regarding of the Department of Prime Minister and Cabinet, “improvements to the leadership and management that my Office undertook a survey in 2012 of how of the policy advice function,”7 for which, it noted, government departments and agencies currently “there is significant room for improvement.” approach the use of evidence in their work. The The report pointed to the general need for a com- results of this survey have led me to formulate a mitment to evidence-informed policy advice and for number of specific recommendations that follow in more effective systems for accessing evidence and Part 2 of this report. applying this to policy advice challenges.8 Specifical- In short, results were very variable; there were ex- ly, to help to “improve the management and dissem- amples of good practice but also evidence of some ination of data and information” for policy formation disappointing attitudes and, in some cases, igno- and monitoring, the Scott report called for: rance of the principles by which evidence could • A shared approach by agencies for knowledge and should inform policy formation. The quality of management and developing capability in re- government data and access to it for analysis is var- search and policy functions (recommendation iable, despite explicit programmes seeking to im- 28); prove Open Government. This in turn means that evaluation and assessment cannot always ensure • A proactive approach to accessing knowledge value for programmes that government wishes to and expertise held outside the public sector see implemented. (recommendation 29); It can be argued that these issues are particular- • The routine (where appropriate) publication ly acute in a small country such as New Zealand. of agency data, analysis, research findings and Inevitably we have a less complex system of con- models, particularly on cross-portfolio and/or nectivity between elected officials, policy makers, long term issues and ‘big questions’ (recom- the public and the media. This, combined with the mendation 30). pressures created by a very short electoral cycle, In addition to the issue of access to policy-relevant results in greater potential for evidence to be ig- knowledge, the report also touched on the need to nored. For these reasons, the issues raised in the develop processes to ensure that this knowledge present report are particularly cogent. is of the highest quality. It suggested that agencies institute a quality management process for policy Re-aligning the Government’s policy process to- analysis and advice. It suggested that “advice on sig- ward the more systematic use of robust evidence nificant issues should be developed using accepted A key theme in the survey’s findings is that the pub- standards […] to assemble evidence within a culture lic service in general requires a better appreciation of analysis, open debate and peer review”.9 and understanding of how high-quality and scien- Since publication of the Scott report in 2010, a tifically-derived evidence fits into the processes of number of central programmes have been re- formation and evaluation of policy. As a start, pol- formed or strengthened to meet these aims. For in- icy makers need to consider such evidence as base stance, Treasury has taken on a central leadership information upon which to build a policy position. role in overseeing the quality of advice through On this foundation can then be overlain the various routine Regulatory Impact Statements (RIS) that values dimensions that properly form part of pol- are attached to Cabinet Papers. However, this im- icy formation. These include: fiscal and diplomat- provement is almost exclusively aimed at the reg- ic considerations, public opinion, and the political ulatory domain and has less impact on shifting the theory and ideology that distinguishes one politi- policy-making culture of the public service toward cal party’s policy position from that of another. If greater use of research-informed evidence. evidence is treated in the same way as these val- 7 Scott, G. (2010). Improving the quality and value of policy ad- ues-based components and conflated with them, vice: Finding of the Committee Appointed by the Government then it loses its critical informative value. This is the to Review Expenditure on Policy Advice. p. 6. 8 ibid p. 46. very consideration that has led other democratic 9 ibid p. 40. jurisdictions to develop protocols and processes Page 9 The role of evidence in policy formation and implementation that define how objective evidence is to be used as and epistemology of science.11 But while the phil- a key foundation to the policy process. osophical underpinnings of evidence generation Such a process must be sought for New Zealand. can and should be debated, international scientific There have been too many examples where ap- standards do provide a framework for a common pealing to apparently confused science masks what understanding and acceptability of evidence. Clear- is in fact a policy or ideological debate (for exam- ly the most robust evidence comes from formal ob- ple, exploiting scientific uncertainty to justify inac- servation and scientific study. tion on climate change). This has been termed the ‘misuse of science as a proxy for a values debate’.10 How evidence is generated Such misalignment can only undermine confidence When ‘data’ are systematically collected and ana- in both science and policy formation. lysed, they become ‘information’; When ‘informa- At the same time there has been an increasing tion’ is applied to specific problems or questions, trend towards seeking public consultation on com- it can be used as ‘evidence’ to establish a position plex issues, but such consultations are not mean- or course of action. However, the value of data in ingful if the public is questioned on issues in the ab- the first instance depends on the robustness of the sence of an unbiased presentation of the evidence. approaches used to collect and then analyse and interpret them. At times the dominance of ideological rhetoric has inhibited the ability of the public to obtain such in- Modern science can be defined by the processes formation. Put simply, majority public opinion does underpinning it.12 That is, science uses a range of not create reliable evidence and must not be taken relatively standardized procedures of systematic as such. This is not to diminish the key role of public data collection and analysis. To such data are ap- consultation, but simply reiterates that the policy plied recognisable and replicable methodologies of process needs to recognise the inherently different analysis that are used to produce the increasingly domains of objective evidence and opinion. As will reliable information about the universe, our envi- be discussed, this distinction is not always clear and ronment and our society that represents the re- sometimes never can be, which makes the role of sults of the modern scientific enterprise. intermediaries to assist the policy process all the Depending on how research is conducted and how more important. Further, while the political pro- the data are interpreted, the robustness and ap- cess is heavily influenced by anecdote, it must be plicability of the information produced can be af- noted that the plural of anecdote is not data. fected greatly. This issue of the conduct of research It is generally understood that governments, within is critical, but it is frequently misunderstood even the parameters of their electoral contracts, wish to within the scientific community. In my previous pa- make effective decisions that do the best for New per entitled Interpreting science – implications for Zealand and New Zealanders. It is my argument public understanding, advocacy and policy forma- that the better use of quality research-informed ev- tion (2013), I highlighted how misinterpretation can idence is key to more effective policy development arise from the inappropriate or unsystematic colla- irrespective of ideology. Such improvements in the tion and analysis of data. There is also the danger policy process may well imply some small compo- of bias creeping into the collection or the analysis nent of additional government expenditure in a 11 A considerable body of scholarship including from Michel time of fiscal restraint, but this is surely the time Foucault, Nancy Cartwright, Ian Hacking and Peter Galison, to when good policy advice becomes even more valu- name a few, has helped define a field of enquiry concerned with analysing the mechanisms of science and the processes able than ever. by which it can make claims of evidence. This ‘post-positivist’ analysis of science has been highly influential in helping to 1.2. The nature of knowledge and evidence uncover hidden biases and unintended effects of context and positioning within the scientific process. The results have the What is considered ‘evidence’ has been the sub- potential to help make scientists more aware of and account- ject of scholarly enquiry in the last several decades, able for the procedural choices they make and how these may creating a considerable literature in the philosophy affect the conclusions they can draw, ultimately strengthening the scientific output. 10 Pielke, R. (2007) The honest broker: making sense of science 12 Marks, J. (2009) Why I am not a scientist. University of Cali- in policy and politics. Cambridge University Press. fornia Press.

Page 10 The role of evidence in policy formation and implementation Further, researchers in general need to be more Key Features Of Evidence-Informed explicit about the limits of knowledge – it is as im- Policy Making portant to define what they know as well as what 1. Quality and accessible data; they do not know. The Popperian view of science 2. Robust and accessible data collection and may be debatable when applied to grand scientif- analytical instruments; ic theories, but it does make the crucial point that 3. Critical awareness of analytical assump- it is much easier to disprove something and, ulti- tions and choices, and of theoretical mately, it is virtually impossible to ever absolute- perspectives that underpin the research ly prove something in science. But this conceptual methodology; issue can be misunderstood and misused. This has particular implications when exaggerated in the 4. Understanding the limitations of even the inappropriate use of the Precautionary Principle – most robust evidence; for instance, when it is demanded that something 5. Adjusting expectations of certainty and be- cannot be done unless it is proved to be absolutely ing able to manage uncertainty. safe. Of course such a demand is impossible to sat- isfy to an absolute degree and thus, when misused of data; the latter is more likely where analysts are in this way, becomes an excuse for policy inaction. not formally trained in research methodologies. From the policy perspective, such inferential gaps in knowledge are especially challenging because: Several key features of scientific research can help to make policy making more transparent. These • The lack of certainty can be used as an argu- include the importance of (1) quality and accessi- ment by positional advocates to avoid action ble data; (2) robust and accessible data collection and this can create extreme positions where 15 instruments; and (3) critical awareness of analyt- the default becomes inaction; ical assumptions and choices, and of theoretical • Most policy decisions have to be made in a fast- perspectives that underpin the research meth- paced environment where multiple considera- odology.13 Incorporating a better understanding tions compete for attention and where there is of these features within the public service policy often little time (and in many cases little capac- toolbox will help to improve the critical appraisal ity) for properly collecting and analysing data, of evidence and thus the quality of policy advice. let alone for building a nuanced understanding 16 Evidence must be critically appraised and its limits of uncertainty of evidence. understood. Understanding and clearly describing these gaps In addition to the issue of research quality and in- must become an essential part of the policy-mak- tegrity, it is important to understand the limitations ing process, both at the outset and through on-go- ing evaluation of policy implementation. of even the most robust evidence. Much research output, particularly that most relevant to the policy Epistemology: ways of knowing maker, does not create certainty but rather, it re- duces levels of uncertainty within complex systems. There are other sources of information, more sub- Because of this and many other factors related to jective forms of knowledge, and other pathways the inherent variability of social, biological, and to obtaining input that interplay in the policy for- environmental systems and to statistical considera- mation process. These other sources of knowledge tions14, a single scientific finding can be at variance need to be made explicit and distinguished from the role that scientifically based research methods with others. This does not mean that a scientific should play in policy development. consensus does not exist or cannot be reached, but expert skills are needed to navigate this knowledge Indeed, it is important to acknowledge the multi- space and critically appraise the material. ple societal considerations that go into knowledge production for policy work. Certainly, social, cul- 13 Argyous, G. (2012) Evidence based policy: principles of transparency and accountability. Australian Journal of Public 15 Oreskes, N. and E. Conway (2010) Merchants of doubt: how a Administration. Vol 71, no. 4, pp 457-468. handful of scientists obscured the truth on issues from tobacco 14 Gluckman, P. (2013) Interpreting science – implications for smoke to global warming. Bloomsbury Press. public understanding, advocacy and policy formation. Office of 16 Douglas, H. (2009) Science, policy and the values-free ideal. the Prime Minister’s Science Advisory Committee. University of Pittsburgh Press.

Page 11 The role of evidence in policy formation and implementation tural and indeed spiritual belief systems that give However, the key to objective and robust science is rise to subjective knowledge have a major impact for researchers to adhere to internationally accept- on the political – and subsequently policy – process able practices and to limit the role of any personally through the values domains that shape actions. held values to areas such as decisions about their This is especially important in the New Zealand research interests and operational principles (for context where Māori culture and ways of knowing instance a researcher may make a personal choice can have a positive influence on policy. not to engage in work that requires animal models for instance, or to only accept funds from certain Such perspectives, however, must be seen as com- sources and not others). plementary to knowledge derived from rigorous research methods rather than equated with it. This Where ever personally held values and biases may is not to say that social and cultural perspectives come into play, it is necessary to ensure that these cannot be reflected in the research undertaken – are clearly identified and do not interfere with the indeed they should (in helping to define the ques- collection and analysis of data as part of a robust tions asked, the analytical approach and the target and transparent research design. Such transparen- population for instance). But the knowledge result- cy ensures that scientific interpretation is based on ing from formal scientific processes must strive for the greatest possible objectivity and standardised a high degree of objectivity and should be used as a processes. Much of the scientific method (experi- relatively neutral base on which the policy and po- mental design, peer review, replicability, statistical litical process must weigh all other inputs into the approaches, publication, etc.) is designed to pro- decisions that a democracy requires. tect the objective nature of science. Professional researchers are trained to recognise and mitigate Thus, from the scientific advisor’s perspective, ad- any biases that might arise throughout the process. vice needs to be proffered in a way that accepts both the presence of gaps in knowledge and the Despite all the safeguards, however, there will al- role of other more values-based and socio-cultural ways be individual researchers who choose to elements in the policy decision process. For their advocate for particular positions based on their part, policy advisors need to recognise that anec- expertise. To some extent this is inevitable and un- dotes do not generate data despite the fact that, in derstood. In doing so, however, these researchers highly political contexts, it is often the most com- risk losing their objectivity in the course of advo- pelling story that carries the most influence. cacy. It is therefore advantageous for the policy process to have access to intermediary knowledge 1.3. Science and values brokers (i.e. science advisors) who can mitigate this I am not suggesting that science itself is totally free risk. Building on Pielke (2007)18, I have argued else- from values and potential biases. Indeed, research- where19,20 that the preferred position for the pro- ers make very real value decisions in the course of fessional researcher embedded within the policy their work. For instance, such things as choosing process is as an ‘honest broker’ explaining what is baselines against which to measure, and which known, what is not known, and thus the implica- indicators and metrics to adopt, are choices (with tions of the options that emerge. trade-offs) that will affect how research results can 1.4. Social science be interpreted and applied. But science also pro- This paper is concerned with research-informed ev- vides accepted processes for making these choices idence for policy making in general, and does not transparently, so that the end product passes the attempt to analyse the specific situation by depart- replicability test. These issues are discussed in de- ment or by scientific field of expertise. Indeed, my tail in my recent paper Interpreting science – im- commentary applies equally across the physical, plications for public understanding, advocacy and biological, natural and social sciences. However, policy formation.17 18 Pielke, R. (2007) The honest broker: making sense of science Of course researchers do have spiritual and cultural in policy and politics. Cambridge University Press. values too. Science is a human endeavour after all 19 Gluckman, P. (2011) Towards better use of evidence in policy and, naturally, human values can come into play. formation: a discussion paper. Office of the Prime Minister’s Science Advisory Committee. 17 Gluckman, P. (2013). Interpreting science – implications for 20 Gluckman, P. (2013) Scientists, media and socie- public understanding, advocacy and policy formation. Office of ty: where are we now? http://www.pmcsa.org.nz/blog/ the Prime Minister’s Science Advisory Committee. scientists-the-media-and-society-where-are-we-now/.

Page 12 The role of evidence in policy formation and implementation given the large government responsibility for social benefit of research training to recognise and miti- programme expenditure, it is worth understanding gate such bias. the use of social science for policy making. In fact, rigorous social science practice is necessar- Social policy has not historically benefited from ily based on a set of methodologies that are sub- the kind of investment in public research capabil- ject to the same rules and professional standards ities that areas such as conservation and primary as in the natural sciences. Indeed, just as in other production have had over the years. Indeed, the forms of science, the same caveats apply: is the infrastructure and capabilities for policy-relevant underpinning research design and the approach to science advice is quite established in these areas. interpretation adequate? Simply looking at appar- The same cannot be said for the social sciences. ent associations in datasets of variable quality does It is perhaps for this reason that incorporation of not meet that standard. But when done well, social robust social science research into policy making scientific analysis makes a powerful contribution to remains a challenge (and one that is not unique to policy development. New Zealand). The social sciences have developed systematic and Evaluating social programmes: applying the science empirical methods to study and draw general con- of what works clusions about social phenomena. Yet, in applying Given the large fraction of the public purse that is social science research to policy, there are still at expended in the social policy domains, quality ev- least three widely-held misconceptions that are idence to support appropriate policy development worth noting.21 and formal evaluation of desired impacts is critical. 22 First, there can be an assumption that, because the Evaluative science and intervention research is very nature of the social sciences is a focus on soci- particularly important in the implementation of so- ety, this allows for values-based interpretations of cial policy because the reality is that the nature of the research, which can become more prominent human systems is such that it is not possible to pre- in the mind of the policy maker who is not trained dict with certainty the direct effect and spill-over to recognise their own hidden biases. In fact, the consequences of any one intervention. social sciences can indeed provide robust social Too often, social programmes that result from pol- data for policy purposes. As with all sciences, how- icy initiatives are rolled out based on an idea or a ever, this requires training and skill on the part of successful pilot that may not, for a variety of rea- the scientist and the interpreter. sons, be successful when they are scaled up na- Secondly, empirical techniques used in the social tionwide or applied in a new geographic location. sciences are broad and can range from quantitative This points strongly to the need to consider formal to qualitative (for instance, statistical analysis of so- evaluation when substantive programmes are ini- cial metrics, case studies, surveys, or interviews). tiated. There is also a compelling case for formal Quite often, there is still a perception that the more evaluative processes to be incorporated into any qualitative techniques are more susceptible to bias substantive new programme. The UK Cabinet Of- and personal judgement by the researchers them- fice recently released a paper entitled Test, learn, selves. However, robust social science methods adapt: developing public policy with randomised used by trained researchers minimise this risk. controlled trials23 which pointed out that controlled trials should be more readily incorporated into the Related to this is a third misconception that under- roll-out of social programmes than is currently the taking social research for policy does not require case. particular expertise and can be done by policy prac- titioners with little or no formal research training. 22 Intervention research is about examining proven policy or This is both a symptom and a cause of potential programme interventions to determine what works best, for whom and in what circumstances. This type of research helps devaluation of academic social science for policy policy makers who want to introduce proven policy or pro- purposes. It can also make the research more vul- gramme interventions into new jurisdictions and adapt them nerable to unintended personal biases, without the to the local context. 23 Haynes L, et al. (2012) Test, learn, adapt: developing public 21 Prewitt, K. et al. (2012) Using science as evidence in public policy with randomised controlled trials. Cabinet Office Be- policy: report on the use of social science knowledge in public havioural Insights Team. Available from: www.gov.uk/govern- policy. National Research Council of the (US) National Acade- ment/uploads/system/uploads/attachment_data/file/62529/ mies of Science. TLA-1906126.pdf.

Page 13 The role of evidence in policy formation and implementation Promising practices in New Zealand • Internal knowledge: Policy analysts are the With respect to evaluating the implementation of so- backbone of policy making. Though a signifi- cial policy, New Zealand has started to make moves cant number have postgraduate or professional in a positive direction. Two examples stand out: research experience, the bulk of the policy staff cadre in the New Zealand public service is still • Prime Minister’s Youth Mental Health Project unlikely to have the research experience and/or (2012): These programmes represented a mile- competency to critically scan the scholarly liter- stone in social science and policy interaction. ature and fully interpret the science. Depend- It was acknowledged from the outset, both ing on the domain, there is the risk that non-ex- by the contributing researchers and policy ad- pert assessment of complex literature can lead visors, that it was not known which of the 22 to cherry picking – that is, finding something programmes in the initiative would in fact be to bolster an established opinion or relying effective. This was simply because of the gener- only on what has been made accessible to the al lack of understanding of many of the factors non-scientist via the popular or semi-popular associated with modern adolescent morbidity. literature. Neither newspapers nor Wikipedia Such acknowledgement by the political process are robust sources of scientific information. is in itself refreshing, but importantly the launch coincided with the allocation of funding specifi- • Expert advice: Policy makers may identify a cally for on-going programme evaluation. knowledge need and go to a known expert for advice. The quality of that interaction will de- • Social Policy and Evaluation Research Unit pend on the nature and framing of the question (SuPERU): The reconfiguration of the Families being asked and the understanding of both the Commission and the establishment within it of agency and the knowledge broker. Protocols that SuPERU and its independent Scientific Adviso- have been developed elsewhere to clarify these ry Board should now provide an autonomous expectations could be of benefit in New Zealand. unit with expertise in social science research There is also the problem of identifying an ap- and evaluation that could provide support and propriate expert in the first place. Many people best practices across multiple ministries. While may claim ‘expertise’, but without sufficient this unit is in its early days, it will be critical that knowledge of the scientific literature, policy the Commissioners24 ensure that the standards makers may have difficulty discerning the quality and mode of operation of this entity are of the of that expertise and in distinguishing advocacy highest quality. It must develop specific skills in from unbiased knowledge transmission. programme evaluation such that Ministers will wish to encourage agencies to take advantage • Advocacy efforts: Policy makers and elected of its expertise. officials may be lobbied by scientists either for issues related to science policy or where 1.5. Typical avenues by which evidence is researchers engage as advocates for a particu- incorporated into policy formation lar cause. As discussed above, there is a risk The policy formation process is iterative and deci- of a loss of objectivity. The role of Academies, sions along the development chain should be con- such as the Royal Society of New Zealand, in tinually checked against the best available evidence, moderating such dialogue should be important which can change over time. This means that, and should be reinforced. However, the ability throughout the policy process (development/imple- of the Royal Society to provide robust advice mentation/evaluation), there is an on-going role for requires resources and an independence that ensuring that the evidence is appropriately applied. may be limited by its dependency on current Obtaining quality evidence for policy formation can arrangements for Crown funding. The more occur in a number of ways. The list below provides recent evolution into a much broader Acad- an annotated summary of the most common ways emy that includes the humanities (where it is that the results of scientific research can be inject- inherent that values dimensions are part of ed into the policy-making system. scholarship) will require the development of processes to ensure that, when advice from the 24 Professor Sir Peter Gluckman, Chief Science Advisor to the Prime Minister and author of the present report, is one of five Academy is proffered, it is clear on what basis it Commissioners of the Families Commission. is being provided, in part so that the Academy Page 14 The role of evidence in policy formation and implementation is not perceived as usurping the proper role of to scientific journals, conference travel, or re- the policy maker. lease time to write and publish papers. Conse- • Commissioned research: Policy makers may quently, such staff can become isolated from contract a piece of research externally. How- their scientific peers. A solution encouraged in ever, unless staff members are appropriately other jurisdictions is to second academic staff skilled in research methods and critical apprais- into policy positions for periods of 1 to 3 years, al, then the overall project may be at risk – ei- and to allow for policy staff to spend time in ther by not recognising if the deliverable is of academia. This has the mutual benefit of ensur- poor quality or by not knowing what to do with ing that more academics understand the policy high quality but highly technical information. formation process, and vice versa. These issues were expanded in my previous • Departmental Science Advisors: Departments paper Towards better use of evidence in policy may establish Departmental Science Advisors formation.25 with defined terms of reference. This system • Scientific advisory bodies: Policy makers may is fairly well-developed in the UK and could invite experts onto particular advisory commit- provide a suitable model for broad adoption tees or panels. As I have argued before26, this is here, and as such is discussed at length in Part to be applauded. However, if the required out- 2. In New Zealand, the Department of Conser- put is scientific advice, then only committees vation has an embedded Chief Scientist, and of acknowledged researchers can give such the Ministry of Primary Industries now has an advice, which should then come to the policy independent Departmental Science Advisor. maker in an unfettered form. To ensure this, These different approaches are discussed be- scientific advisory committees require formal low. However, these Departments stand out governance structures (protocols of member- in their recognition of the importance of re- ship and operation) to protect their objectivity. search-informed evidence in the processes of The UK has developed very specific criteria in policy formation, which is far from the norm this regard. This does not mean that scientists across government. do not have valuable contributions to make to • Parliamentary structures: In some countries other forms of advisory committees, but these Parliament itself can foster interaction between have to be viewed as something other than science and policy development and implemen- scientific advisory committees. Scientific advi- tation. The Parliamentary Office for Science sory bodies also need full access to the relevant and Technology (POST) offers non-partisan data, as the quality of their advice rests on the advice on scientific matters to both UK Houses quality of data they have at their disposal. of Parliament. Parliamentary Select Commit- • Intramural research: Departments may retain tees (particularly where there are bicameral internal research units or staff as the primary systems) focus on the quality and nature of sources of broad evidence-informed advice. scientific advice or conduct expert enquiries. Staff members may have varying qualifications However, such a function is very distinct from and training, but most important is that they the role that science advisors play in advising have proven skills in applying rigorous and rep- the Executive and engaging in the policy forma- 27 licable methods of data collection and/or anal- tion process, and the two cannot be linked. ysis. More desirable still is in-house researchers Novel approaches are required to enhance being active in handling the scholarly literature parliamentarians’ understandings of science and regularly exchanging informed opinions and the scientific method. This is important with research peers, thereby helping to ensure given that relatively few parliamentarians come that they are providing state-of-the-art and from backgrounds where such understanding is balanced advice. However, too often, public widespread. One approach is a pairing scheme service budget constraints do not allow access operated by the Royal Society (London) that pairs scientists and parliamentarians/senior 25 Gluckman, P. (2011) Towards better use of evidence in policy formation: a discussion paper. Office of the Prime Minister’s 27 Tyler, C. (2013) Scientific advice in parliament. In Future direc- Science Advisory Committee. tions for scientific advice in Whitehall, Doubleday and Wilsdon, 26 Ibid. eds. Cambridge Centre for Science and Policy.

Page 15 The role of evidence in policy formation and implementation civil servants for a reciprocal week of exposure those who generated the data, and prior to any pol- to the other’s domain.28 icy decisions, could have contributed to the depth The above summary reflects the usual methods and quality of the public consultation. employed to ensure the supply and quality of science advice for policy, but the extent to which Understanding and communicating risk that advice is implemented depends on the inter- An on-going challenge for all nations is the adop- nal capabilities within the public service to bro- tion of new technologies. Those technologies – ker the knowledge and apply it appropriately. The such as the internet and social media – that show need for leadership in lifting staff capabilities and immediately perceptible benefit or convenience create a culture of evidence use is addressed in have been incorporated and embedded into soci- Part 2 of this paper. etal structures without much critical reflection, yet they are now having profound effects. This con- 1.6. (Un)informed public discourse trasts with other technologies that have had more There are public debates which have influenced contested introductions to the world; examples policy decisions in recent years where the New include in vitro fertilisation, genetic modification, Zealand public would have been better served nanotechnology, synthetic biology and hydrological had an evidence-based process been used within fracking. Here it has become apparent that explicit the policy formation process and within the public rather than tacit social license is needed for such dialogue that informs it. We have had examples technologies to be accepted. where public consultation has occurred without the benefit of transparent peer-reviewed scientif- The difference between this class of technologies ic advice. This has limited the quality of the pub- and the former is that, at least superficially, they lic and media discourse and may have well led to more directly confront individual and social values decisions that in other circumstances might have and beliefs. This sometimes leads to the debate be- been very different. ing solely about values without any ability for the public to gain a clear appreciation of the technology One example is that of folic acid supplementation itself. Indeed, the level of contention and strongly of bread. A decade ago there may have been no held values by sectors of the public may completely scientific consensus in this area, but there is now inhibit the provision of evidence to inform societal a strong consensus that it is safe and effective in opinion. reducing the incidence of neural tube defects in newborns. Documents from the Ministry of Prima- Better application of research-derived evidence ry Industries made that clear, yet consultation went encourages more informed and mature public de- ahead in the absence of these reports being wide- bate in policy questions. Importantly, it can help to ly available. There may well be valid values-based distinguish the risks and benefits of the application reasons why a country may decide not to supple- of a technology from the fundamentals of the tech- ment bread with folic acid, for example perceptions nology itself. of medicalising a common food staple (although we Nuclear technology is a case in point. New Zealand- have long supplemented salt, and more recently ers are proud of being ‘nuclear free’, but of course bread, with iodine), or because a cost-benefit anal- we are not – what we have done is limit the appli- ysis may not seem to justify it. But irrespective, in cation of the technology to medical and biosecurity this case decisions were made with a component purposes, while restricting its use in areas such as of public consultation in the absence of up-to-date power generation, transport and defence. By con- scientific data being made freely available. trast, where we limit the technology itself rather Another recent debate is that of school class size. than control its application, there is the danger of Clearly there are very diverse views of the trade- embedding rules in the national mind-set that could offs, costs and benefits of changing class size. The be difficult to change. Yet technologies change very use of an expert educational scientific advisory quickly, and well-communicated quality evidence panel to review the empirical evidence about en- contributes to the development of appropriate hancing educational outcomes independently of regulatory frameworks that are responsive as tech- 28 The Royal Society. Pairing scheme. http://royalsociety.org/ nologies change and the risks and benefits become training/pairing-scheme/. better understood. Page 16 The role of evidence in policy formation and implementation In such areas, better use of quality evidence Better Risk Analysis For Better Policy will make that debate more productive and less Development polemic. • Understanding risk is a matter of both sci- ence and human values. Mature public dis- Indeed, such issues in the practice of risk assess- course must be informed by both; ment and management are themselves complex because there are very different meanings of risk. • A more informed understanding of risk will These may range from the technical calculation become increasingly important in the com- of probabilities to the perceptions of likely ‘harm’ ing years as the pace of new technologies based on personal or societal values. Different kinds increases; of risk are accepted differently. For instance, in the • Quality evidence that is conveyed to the area where technologies and values can be in con- public in an accessible way can position so- flict, the issue of how risks are addressed depends ciety better for decisions on whether to give in no small part on whether trust exists or does not social license for the introduction of new exist between the public and the regulator.29 The technologies. assessment of risk is therefore intimately linked both to the provision to the public of high quality All use of technologies requires a social license. information by a trustworthy source and to appro- Whether a social license is gained or not depends priate and timely engagement of stakeholders.30 on when and how the public is engaged with the technology. If the science is presented in a non-par- 1.7. A stock-take of evidence-informed tisan manner, rather than becoming embroiled in policy practices in New Zealand partisan debate, then a different type of construc- My survey of 17 government agencies and minis- tive public conversation can emerge. Such a conver- tries showed that there are departments where sation can and should include a full examination of the need for a quality database is acknowledged; public values about the use of the technology, but conversely, there are others where staff attitudes values must be distinguished from (and informed toward the use and analysis of data to develop a by) robust evidence. The recent preliminary report policy case were disappointing. of the Parliamentary Commissioner for the Envi- Some policy practitioners held the view that their ronment on fracking highlights the value of good primary role was to fulfil ministerial directives, evidence-informed public discourse early in the rather than to provide an evidence-informed range consideration of many technologies. of policy options on which Ministers could develop Given that some of the most intense debates in a position. Surprisingly, this view was held in some New Zealand, and indeed across all western socie- departments that most need to use objective evi- ties, in coming years are going to be about the in- dence in their day-to-day operation. corporation of new technologies, the science-poli- Worryingly, some officials had limited understanding cy nexus will be even more essential in assisting the of the scientific process of knowledge production, public and political leaders to understand the risks, or were uncertain about it. In addition, they were the opportunities, and the trade-offs involved. In- not clear on how research-based evidence could be deed, we can expect significant debate on issues used to support policy processes. Rather, it seemed such as: that some preferred to work from their own beliefs • Next generation internet technologies; or rely on their own experience. At its extreme, I find • Genetic modification related technologies this deficiency to be unacceptable. In part, I think where no new genes are introduced (instead these deficits reflect the dire need to build some there is altered regulation of the native genes); basic competencies in research methodologies and critical appraisal skills across the public service, and • Synthetic biology, neural implants and regener- to bolster the leadership ranks with people formally ative medicine; trained in the relevant disciplines. • The use of technologies to manage the balance 29 between environmental protection and the Lofstedt, R. (2005) Risk management in post-trust societies. Earthscan Press. need to enhance social and economic prosperi- 30 These matters will be expanded upon in a future discussion ty in a sustainable way. paper. Page 17 The role of evidence in policy formation and implementation Promising initiatives on the road to improvement quality of its policy formation processes. For This is not the first time that the quality of policy example, the Fisheries group has developed an advice has been scrutinised across government. extensive protocol for seeking scientific advice In the years since the Public Finance Act was en- on the management of the national fisheries acted (1989), there have been other initiatives resource, which is a particularly sensitive pro- aimed at improving the quality of policy advice, cess. It is significant that this protocol is very including most recently the government appoint- similar to the more general guidelines for ad- ed committee chaired by Dr Graham Scott (2010) vice released by the Chief Scientific Advisor to mentioned earlier. the UK government in 2011. Yet, within MPI there is also recognition of the need to improve As a result, some concrete steps have already matters further. For example, the report on the been taken to lift and monitor the quality of pol- Psa outbreak suggests that there was over-de- icy advice. For instance, under the State Services pendency on a passing comment within a single Commission’s recent Better Public Services Initia- academic paper without the benefit of experi- tive, systemic mechanisms are being put in place enced scientific review. The on-going issues to provide evidence-informed critical analysis. As regarding importation of pig meat can be inter- mentioned earlier, the Treasury’s Regulatory Im- preted in part as a result of conflating scientific pact Statements (RISs) are intended to provide due and end-user advice in a single process. Overall, diligence in critically analysing the various options however, it is encouraging that the recent Chief and their consequences – something that requires Executive of MPI had given priority to improv- good data and analytical methodologies. An exter- ing the use of evidence-informed advice in nal consultancy is contracted by Treasury to regu- departmental processes and this culminated in larly review the quality of a random sample of RISs. late 2012 in the appointment of a senior and Similarly, the New Zealand Institute for Economic internationally respected scientist as MPI’s first Research (NZIER) is annually contracted by a num- independent Departmental Science Advisor. ber of agencies to assess a random sample of policy • The Department of Conservation (DOC) also papers for quality and effectiveness. has such a Chief Scientist position, though This is a laudable level of self-monitoring by agen- without the advisory independence. DOC also cies and, across government, by Treasury. By all ac- benefits from good internal capabilities and counts the exercise is already revealing some seri- processes for knowledge production, analysis ous knowledge and capability gaps, which is a good and the ability to judiciously commission ex- start on the road to improvement. While promis- ternal scientific input and expertise. However, ing, it is unclear whether these efforts will be useful there are systemic obstacles that can prevent in helping to build the internal capabilities of staff the application of objective evidence in policy to generate and/or make better use of research-de- and practice. DOC has previously undertaken a rived evidence for policy making. review of its science transfer process31, which In addition to these efforts, some specific agency tracked the mechanisms and the roadblocks of examples bear mentioning: getting science into policy. Some of these find- ings are being implemented. • PHARMAC is perhaps the best known agency with a rigorous approach to evidence evalua- Grounds for concern tion, and it is interesting to note the high pub- lic and professional acceptance of the model While these examples of promising practices are despite the fact that it must deal with highly encouraging, our survey also revealed grounds for contentious issues. PHARMAC’s obvious use of concern arising in departments relying primarily on science in decision making has fostered its pub- internal research of questionable quality and/or lic credibility and, in turn, the acceptance of the commissioning external advice that was not scien- difficult decisions it has to make about which tifically peer reviewed. medicines are funded by the public health sys- Given that many of the most important decisions tem. that must be taken by any government will relate • The Ministry of Primary Industries (MPI) re- to matters of resource allocation and risk assess- cently took important steps to improve the 31 Geoff Hicks, DOC Chief Scientist, personal communication. Page 18 The role of evidence in policy formation and implementation ment, the current lack of protocols for commission- gests, among the key factors that determine the ing or generating evidence-informed advice across use of objective evidence and science in support government is of concern and runs contrary to best of policy making are reliable mechanisms such as practices internationally. cross-government work groups and the establish- Another concern common to most departments ment of expert bodies that specialise in the sub- (with the exception of Treasury and the defence/ stance of a policy domain. intelligence sectors), was the minimal horizon Organisational culture shifts occur when ena- scanning, technology forecasting and broader risk bling mechanisms and leadership are in place. In assessment capabilities with a view to the longer this regard, my recommendations below should term. While other jurisdictions put significant re- prove useful. sources into these areas, it may well be that our electoral cycle leads to a focus on short-term pri- 2.1. Setting standards orities. However, the country also needs considera- Key recommendation: Develop a standard set of tion of the longer term. protocols across government regarding obtaining expert scientific advice. Policy units that are by definition non-partisan have an obligation to advise Ministers on short-term is- Both the UK Office of the Government Chief Sci- sues but also must consider the longer-term and entific Adviser and the US Office of Science and multi-generational implications of policy actions. Technology Policy have now established protocols Indeed, internationally, science plays a very impor- for obtaining independent scientific advice, either tant role in such longer-term forecasting – partici- from individuals or from scientific advisory commit- pants in the Small Advanced Nations meeting that tees (see Appendix 2). Importantly, principles have New Zealand hosted in Auckland in November 2012 been established with the goal of ensuring that such regarded this scientific function as critical. advice is free from bias or filtration. Given the var- iable state of the use of evidence-informed advice PART 2 – Suggested steps to enhance and the limited and uneven appreciation of both the use of evidence in New Zealand’s its value and its limitations, such protocols need to policy framework be developed and applied across the New Zealand policy framework in a whole-of-government man- In my view there is an overwhelming case to en- ner. This is made difficult because of New Zealand’s hance the ability to produce and use quality re- search and scientifically based evidence within small corpus of expertise within any given domain, New Zealand’s policy development system. Indeed, and thus the need for expert oversight becomes I have argued for the critical position of evidence more critical. derived from rigorous research as the base on Key elements in seeking advice either from individu- which the values perspectives, required in policy al researchers or expert committees were laid out in development, should then be overlain.32 This sec- my previous report Towards better use of evidence in tion identifies five areas that stand out for potential policy formation and are reiterated here. Procedures action to help to improve the system. must be in place to ensure that the advice is: These recommendations are aimed at ensuring • Politically neutral; that sound evidence gets incorporated into the pol- • Focused on the data and its appropriate inter- icy system and that there is the capability internally pretation; to critically understand how best to employ it. • Unbiased with respect to its use of data; Of course, the internal ‘culture’ of departments and • Explicit about what is known and unknown and agencies will influence uptake and implementation the quality of the available data; of science-based inputs and this puts the onus for improvement on Chief Executives and their leader- • Clear in communicating probabilities and mag- ship team. However, as the work of Nutley33 sug- nitude of effect;

32 Gluckman, P. (2011) Towards better use of evidence in policy • Free from real and perceived conflicts of interest. formation: a discussion paper. Office of the Prime Minister’s I believe there would be value in enshrining these Science Advisory Committee. 33 Nutley, S. and I. Davies (2007). Using evidence: how research principles within a set of formal guidelines for the can inform public services. Bristol, UK. The Policy Press. New Zealand public service. Page 19 The role of evidence in policy formation and implementation 2.2. Science leadership benefit from more formalised access to scientific Key recommendation: Extend the use of Depart- advisors. mental Science Advisors (DSAs) more broadly In my 2011 discussion paper Towards better use of across government. evidence in policy formation I pointed out the value The Prime Minister established the Office of the of Departmental Science Advisors to the UK policy Chief Science Advisor in 2009, modifying and de- framework. In the UK, each department is required veloping the model from that used in other juris- to have a scientific advisor – the details of the post dictions. Internationally, central science advisory vary across agencies, depending on their character- roles fall into two categories: those that involve istics. Some for example administer departmental the direction, management and operation of the research budgets. Recently the UK House of Lords science and technology system, and those that are Select Committee reviewed the function and oper- independent of the management component, but ation of these positions and concluded that they advise the Executive. are valuable and critical roles that deserve to be New Zealand, in my view wisely, has distinguished strengthened and made more consistent across between those types of role and focused on the government.34 The Select Committee suggested a second, even if this arrangement can potentially framework for appointment and terms of reference create tensions in the provision of advice on poli- for these positions (Appendix 3) which, in modified cy development regarding science and innovation. form, would work well in New Zealand. Australia Moreover, its independence strengthens the neu- has also committed to progress in this direction. trality of the role and gives weight to the broader functions of science communication, advice on ev- There is thus an evidential base for New Zealand idence in policy formation, science diplomacy and to follow in strengthening the positioning of sci- specific policy issues. Together with a role in struc- ence advisors across agencies. However, we have turing and enabling risk assessment and foresight- neither the scale of public service nor the scientif- ing (which is yet to be actioned in New Zealand), ic capacity to simply follow the UK system. In our these are the key functions that other jurisdictions case, scientific advice across ministry groupings be- increasingly perceive as central to the role of a gov- comes logical, and it is also in keeping with the gov- ernmental science advisor. ernment’s desire to encourage a more coordinated Of these responsibilities, assisting government to public service. ensure the quality and application of research-in- The beginnings of such a system are already emerg- formed input in the policy development process ing in New Zealand. For instance: across government is of paramount importance. As • As mentioned above, DOC has a Chief Scientist noted above, New Zealand does not have a robust and MPI has recently created the position of framework across government in this regard (al- Departmental Science Advisor. The terms of though Treasury has made significant progress to- reference for the latter position are largely in ward advancing the quality of evidence for making line with UK practice. A senior and experienced regulatory – if not policy – decisions). scientist has been appointed to that role and Clearly departments require access to expertise, within a few months has demonstrated within yet in some cases they do not recognise that they the Ministry the value of the appointment. are lacking it. There is a cultural divide across parts of the New Zealand public service when it comes • The Ministry of Defence has similarly appointed to the use of scientifically generated evidence for a Defence Technology Advisor, in a somewhat policy formation. As my survey indicated, this is narrower role that could be expanded. likely the result of differences in modes of think- • MFAT counts on the Prime Minister’s Chief Sci- ing, including in some places how to look at data ence Advisor in the area of science/diplomacy, objectively, how to identify appropriate sources of and on MPI’s Science Advisor to provide coun- advice, how to recognise scientific bias, and how sel on more technical matters. to identify knowledge gaps. There is also variabili- 34 ty in the extent of appreciation of the implications House of Lords. Select Committee on Science and Technology (2012) The role and functions of Departmental Chief Scientific of technological developments. Overall, it is clear Advisers. www.publications.parliament.uk/pa/ld201012/ldselect/ that departments or clusters of departments would ldsctech/264/264.pdf. Page 20 The role of evidence in policy formation and implementation

Table 1. Possible sectoral groupings of Departmental Science Advisors Departments/agencies State of science advice Other observations DPMC The position of Chief Science Advisor is in place. MPI A Departmental Science Advisor was appointed in 2012. MFAT Arrangements in place. This might need expansion in due course. Defence and intelligence agencies The Defence Technology Advisor The role might need to be was appointed in 2012. expanded from its current description. Health An appointment is needed. Education and Tertiary Education An appointment is needed. Social Development and other The CE of SuPERU is a partial A formal appointment within the social ministries solution. ministry is desirable given that SuPERU cannot influence the op- erations of MSD. MBIE (including Building and An appointment is needed for this Given the different tasks and Housing, Immigration, Labour), cluster. scope of work this is likely a Transport, Internal Affairs separate need from the recently appointed Chief Science Advisor for science and innovation. Environment (including climate An appointment is needed. change) Conservation An appointment is needed. It might be combined with Environment. Treasury This function is likely largely un- dertaken by the Chief Economist, but some access to formal exter- nal advice may be logical. Statistics This is already the role of the Gov- ernment Statistician. Police, Corrections, Justice An appointment would seem to be desirable.

• The establishment of SuPERU, with a degree of The Ministry of Health already has a number of autonomy via the Families Commission, could advisory committees. However, there remain con- provide a route for that unit to provide some cerns about the capability of analysts to identify advisory function across the social ministries. appropriate sources of advice and perform critical appraisal. This is in no way a criticism of the general With a relatively small number of additional ap- competencies of the staff, but it does throw into pointments, I believe that a valuable community question the criteria used to appoint staff mem- of Departmental Science Advisors together with bers ultimately responsible for handling scientific the Prime Minister’s Chief Science Advisor could be information. Thus a key role of a Departmental Sci- created. This grouping should work across govern- ence Advisor would be to assist in the up-skilling of ment to provide leadership in lifting internal capa- the analyst community. For instance, within such a bilities and bringing consistency to all avenues of ministry, a Science Advisor could play an essential evidence-informed advice provision. role in coordinating the development and adoption Page 21 The role of evidence in policy formation and implementation of protocols and standards. The role of a Science Overall the actual number of appointments need- Advisor would be quite distinct from that of the ed to create the optimal group of Departmental Chief Medical and Nursing Officers. Science Advisors across government is small – in Education policy is an area where it is easy for re- the order of four to six FTE with minimal support ceived wisdom to determine policy. Values are of- requirements. Yet these appointments would have ten conflated with evidence, again making obvious major impact by: the need for independent scientific advice. • Up-skilling staff and lifting departmental capa- 35 Thus I would recommend the creation of a num- bilities in the use of evidence; ber of Departmental Science Advisor positions, • Ensuring the appropriate incorporation of evi- each with specific terms of reference appropriate dence into policy advice; to their ministry or multi-ministry sector (Table • Ensuring the rigour of scientific input into pol- 1). Such a group of Advisors has the potential to icy advice; add greatly to the quality of policy formation, be- • Ensuring the quality of external and internal cause policy practitioners in those departments commissioned research; would be assured of starting from a sound scien- tific footing before undertaking more values-based • Ensuring the protocols for external scientific considerations. advice and ensuring that appropriate advice is sought; It may also be worth considering how a second set of ministries not mentioned above might operate in • Ensuring appropriate standards of policy and having their own advisors or by arrangements with programme evaluation are put in place; the clusters already suggested above. For example, • Collectively identifying the government’s short- Te Puni Kōkiri may see value in its own Advisor to and longer-term needs for research to support liaise with the several Advisors suggested above. policy; and Each appointment would call for a broadly experi- • Assisting the Executive when called upon to enced scientist with significant national and inter- liaise with Select Committees to ensure that national mana within the relevant sector. However, evidence submitted is of the highest quality. it is important to note that, while such roles must be filled by people with a solid professional track 2.3. Long term planning, risk assessment and record in research, providing evidence-informed evaluation advice is not a matter of being expert in every tech- Key recommendation: Use the community of DSAs nical field in which advice is sought. Rather it is hav- and the Chief Science Advisor to assist central agen- ing the science literacy, knowledge, networks and cies with longer-term planning, risk assessment and wisdom to know where to go for information and evaluation. then interpreting it appropriately (including assess- There is inevitably a tension between policy mak- ing limitations and possible biases) for use by the ing focused on the short term and the planning policy community. It is also about having sufficient needed for longer-term readiness. Indeed, policy leadership skills and authority to help a depart- makers not infrequently need to consider the bal- ment’s cadre of analysts to develop their own skills ance between the two and thereby make temporal in this regard. trade-offs. However, the political process and the In short, the Departmental Science Advisor must be approach of the media make a focus on the shorter able to: term the more likely default position. It is therefore important for policy makers to look over the longer • Curate expertise; term using advice informed by up-to-date model- • Engage as a peer with other scientists; ling and projection methodologies. This is especial- • Critically appraise scientific inputs from raw ly significant for advice relating to new and emerg- data to selected methodologies to evidential ing technologies.

claims; 35 For example: by designing learning and knowledge exchange • Be able to interpret, synthesise and deliver opportunities between active researchers and end users (e.g. staff rotation, sabbaticals, joint appointments); by liaising with the results in a manner that is relevant to policy New Zealand School of Government to help develop the required development and/or its evaluation. analytical capabilities in trainees). Page 22 The role of evidence in policy formation and implementation For instance, a particular set of issues relates to priorities for such research are identified and then consideration of the scientific approach to risk, risk provided to the funding agencies is haphazard and assessment and risk management. Here, the formal uneven. Similarly, there is little formal approach by and objective analysis of the likelihood of certain policy makers to harvesting the results of the re- events occurring and their management implica- search, once it is complete. tions can often diverge significantly from wider perceptions of risk, which are inextricably linked Clearly there is a need to coordinate a process of to public values. It is important that decisions are what has been called elsewhere Integrated Knowl- made with research-informed appreciation of the edge Translation (IKT).36 A concept borrowed from ‘actual risk’ as well as taking account of the public’s health research, IKT proposes that knowledge us- understanding. In some areas such as risk manage- ers (in this case policy practitioners) work close- ment for natural disasters (for example, building ly with knowledge producers from the outset to codes or insurance issues), there is a primary role identify the most relevant research priorities and for the formal engagement of the risk research methodologies that could respond to particular community. Objective consideration of new and policy needs. Again, the group of Departmental emerging technologies requires similar analysis. A Science Advisors would be an effective way of en- skilled Departmental Science Advisor can expertly suring such links between the policy and research broker that engagement. communities. In general, the community of Departmental Science Another way to improve the use of government Advisors would become a resource to use in build- funds is to improve the focus and commitment to ing greater capabilities in longer-term planning, risk assessment and evaluation – a role that is indeed programme evaluation. Ministers should expect explicit in some jurisdictions elsewhere. and demand: (1) that more programmes resulting from particular policy decisions are subject to effi- 2.4. Government funds towards policy- cacy evaluation; (2) that funds are allocated for this relevant research purpose; and (3) that reviews consider not only Key recommendation: Improve and make more ex- new programmes, but also existing programmes. plicit the use of government funds for research to assist policy formation. There is no embarrassment in acknowledging that the impact of a new programme is not known at Taxpayer funds are used in three ways to support re- the outset and must be evaluated. On the contrary, search for policy formation: (1) agencies undertake a culture of on-going enquiry and evaluation should research intramurally; (2) agencies contract research be fostered in all policy-related activities (following extramurally; and (3) some of the contestable re- search undertaken by the Crown Research Institutes the promising example of the youth mental health (CRIs), universities and research institutes/associa- programmes). For instance, ‘intervention research’ tions relates directly to the policy agenda. and the use of well-constructed pilot programmes are extremely valuable in assessing a programme’s The first two of these types of activity do not nec- essarily meet the standards that research funded success factors before deciding whether and how through contestable processes is required to meet. to take it to scale. A role of the Departmental Science Advisor would In fact, it is worth considering whether SuPERU be to ensure that such activity is of a sufficient should be developed as a government-wide re- standard to be worthy of the investment made. source to assist in social policy and programme eval- MBIE, MoH, MoE and MPI all directly or indirect- uation. SuPERU is still in its establishment phase, but ly operate contestable programmes of funding its terms of reference require it to build particular research (e.g. Marsden Fund, Primary Growth expertise in programme evaluation. SuPERU expects Partnership, Health Research Council, Centres of to make its services available across government and Research Excellence). The objectives of each of it should be encouraged in this role. these schemes vary, but internationally it is accept- 36 ed that governments increasingly want and need Bowen, S. and I. Graham (2013) Chapter 1.2: Integrated knowledge translation. In Knowledge translation in healthcare: academia to engage in research which has policy moving from evidence to practice, S. Strauss et al (eds). John implications. However, the manner in which the Wiley and Sons Ltd. Page 23 The role of evidence in policy formation and implementation 2.5. Transparent and accessible information My survey results suggest that there is a uneven for public consultation understanding of the role of research-informed ev- Key recommendation: Provide greater transparen- idence across the New Zealand public service. Deal- cy regarding the use of research-informed data (or ing effectively with this challenge is key to providing its absence) with respect to complex and controver- an evidentially defensible foundation on which to sial areas of decision-making where the public is build policy. Once this is in place, values-based con- directly or indirectly consulted. siderations can be addressed and integrated – but with the full knowledge of what is at stake from a The public, either by sector or more generally, is con- scientific perspective. To summarise, good science sulted frequently by ministries on a range of mat- cannot make policy, but bad science or the absence ters. However, there have been examples where no of science will almost certainly lead to poor policy significant research-based information was made decisions. widely available prior to such consultation. Of my recommendations made in this report, two This lack of disclosure limits the value of the con- priorities that would make a substantive difference sultation, and indeed can change the nature of its but at low cost would be: outcome. Such information should be released in advance, and with this, one further role of the De- • The establishment of government-wide formal partmental Science Advisor (and group of advisors protocols to guide policy makers in sourcing collectively) would be to ensure the validity and ac- quality research-based advice. cessibility of any report released. This would mean • The appointment of Departmental Science Ad- providing commentary on knowledge gaps and the visors to major ministries. limit of what is known. This should be particularly Given that governments want to make decisions so when such consultation occurs in a climate of that meet their agendas but also are effective and strongly vested interest, or when the public is being have both short-term and long-term benefit for asked to consider contentious issues. A satisfactory New Zealand, well-informed policy advice is more resolution is more likely when the available knowl- likely to meet a government’s goals, and is cer- edge is presented in a logical, objective and acces- tainly more likely to meet its desired goals and be sible manner. This work would require some effort cost-effective. and resources toward public outreach and knowl- edge translation to enable more meaningful public LIST OF APPENDICES engagement. Appendix 1 CONCLUDING COMMENTS Review of government agencies and ministries This report has been restricted to a discussion of about their understanding and use of research-in- the issues related to the development of an evi- formed evidence in the policy process: dence base for policy formation, and to providing • Appendix 1.1 Agencies surveyed. some options for consideration. • Appendix 1.2 Survey questionnaire. In making my recommendations, I am conscious Appendix 2 that they cannot be undertaken in isolation. They link closely with work already underway under the US and UK protocols for obtaining independent sci- banner of Better Public Services and ideally would entific advice. be part of a more comprehensive ‘knowledge Appendix 3 strategy’ for the public service, as there are impli- Suggested framework for appointment and terms cations for how new policy staff are recruited and of reference for Departmental Science Advisors. trained – even involving the content of academic programmes.

Page 24 The role of evidence in policy formation and implementation Appendix 1

Review of government agencies and ministries about their understanding and use of research-informed evidence in the policy process

Contents

1. Overview...... 26 2. Background and purpose of the survey...... 26 3. Methods...... 26 4. General findings...... 27 4.1. Knowledge of, and attitudes toward, the science-policy nexus...... 27 4.2. Practices employed at the science-policy nexus...... 28 4.2.1. The practice of identifying and filling knowledge gaps...... 28 4.2.2. The practice of commissioning intramural and extramural research...... 29 4.2.3. The practice of peer review...... 29 4.3. Specific knowledge gap areas where a robust evidence base is lacking...... 30 4.3.1. Use of evidence for risk assessment...... 30 4.3.2. Use of evidence for foresighting...... 30 4.3.3. Use of scientific expertise and research design for evaluation...... 30 4.4. Use of scientific expertise ...... 31 5. Discussion and conclusions...... 31 Acknowledgements...... 32 Appendix 1.1: Agencies surveyed...... 32 Appendix 1.2: Survey questionnaire...... 32

Page 25 The role of evidence in policy formation and implementation 1. Overview 2. Background and purpose of the In 2011 the Office of the Prime Minister’s Science survey Advisory Committee published a report on the use Citizens have a right to expect that policies will be of knowledge-based evidence in policy formation. based on sound reasoning and good judgement. That report posed the challenge of “how to do better To ensure this, it is essential that all stages of pol- in two related domains: the generation and applica- icy formation and evaluation use research-derived tion of knowledge to inform policy making, and the knowledge optimally to achieve the best outcomes. use of scientific approaches to the monitoring and It is clear that there are deficits in how government evaluation of policy initiatives.”1 The report argued obtains and uses knowledge and evidence, and this that sound evidence provides for potential options necessarily affects the quality of policy formation. and solutions which government can subsequently The Chief Science Advisor published his discussion weigh up against other, often competing, factors. paper on the science-evidence-policy nexus in mid- That position has been expanded upon by others.2 2011 and, in the course of the rest of that year, With the support of the Department of Prime Min- had a number of discussions with Chief Executives, management groups and those involved in pro- ister and Cabinet, the Office of the Prime Minister’s viding science advice or commissioning research Science Advisory Committee undertook a survey across government agencies. From these initial dis- of a range of government agencies during 2012 as cussions, the impression was garnered that there to their attitudes and practices relevant to utilising were very different practices across government, strong research-informed evidence in policy devel- some of which were very good while others raised opment and evaluation. The present report sum- concerns. marises the results of that survey and some general conclusions that may be drawn. After discussion with the Prime Minister and the Chief Executive of DPMC, the Chief Science The key finding of the survey was that attitudes to Advisor decided to undertake an assessment of the use of research-informed evidence in policy knowledge, attitudes and practices (KAP) related formation are highly variable. This variability re- to the science-evidence-policy nexus across gov- flects the absence of whole-of-government proce- ernment. The purpose was to inform a further re- dures on how and when science should be used in port to the Prime Minister, to which this review is policy formation and evaluation. The survey results now appended. showed a disappointing approach to integrating science into policy formation in many agencies. 3. Methods Indeed, there is a general lack of formal protocols The research design for this assessment was based or guidelines for seeking expert advice and input, on KAP methodology, which is known for its analyt- and for ensuring that robust evidence informs pol- ical power in psycho-social research that looks at icy advice. There are also inconsistencies around behaviours and behaviour change. Its applicability methods of seeking such knowledge and commis- to the policy-development context is appropriate sioning science to support policy development. In because appreciation and use of quality evidence addition, there was marked diversity across the by policy makers is, arguably, a matter of socialisa- agencies surveyed in their understandings of risk, tion and competencies (i.e.: knowledge, attitudes and little focus on foresighting – both in the medi- and practices). um term and, more significantly, in the long term. The data collection comprised a structured ques- This results in policy formation that answers imme- tionnaire, which was supplemented by key-inform- diate needs but does not necessarily provide well ant interviews. The aim was to determine how for the future. agencies obtain and use evidence for the develop- 1 Gluckman, P. (2011) Towards better use of evidence in policy ment and evaluation of policy. For the purposes of formation: a discussion paper. Office of the Prime Minister’s this assessment, we define ‘evidence’ as knowledge Science Advisory Committee. in any area obtained by a process of systematic en- 2 Bromell, D. (2012) Evidence, values and public policy. The Australia and New Zealand School of Government, ANZSOG quiry undertaken in accordance with standard and Occasional Paper, Wellington. internationally recognised research methodologies Page 26 The role of evidence in policy formation and implementation (whether qualitative or quantitative) and applied to In general, most saw the need for better provision the policy development context. of options to ministers rather than presenting con- 3 A number of key public service departments, inde- clusions (that is, brokering rather than advocating ) pendent Crown entities and offices of Parliament but there were a lot of different viewpoints. These were chosen for this exercise. The Chief Executive ranged from insistence by agencies that their pol- of each agency was requested by the Chief Execu- icy development process is ‘pure’ to those which tive of the Department of Prime Minister and Cab- acknowledge that policy analysis, development inet to nominate a spokesperson to take part in an and evaluation involve a values judgement, while extensive structured interview with a staff member stressing that values should not be seen as the only from the Office of the Prime Minister’s Science criterion. Advisory Committee. The questionnaire was sent There was rather broad ignorance about how sci- to the nominated spokesperson prior to the inter- ence could inform policy, and indeed some scep- view taking place, so that they could solicit infor- ticism about the concept of the ‘honest broker’. mation from staff and colleagues. Each interview Some could not see the role of the scientist as a was recorded and transcribed. In addition, some broker of unbiased information, but rather as an respondents also supplied written answers to the advocate. To quote one respondent: questions, and many supplied additional written “If you’re passionate about science, it [pol- material in support of the position of their agency. icy development] is possibly not where you The responding agencies are listed in Appendix should be working because actually, it clouds 1.1, and the questionnaire outline is described in your judgement to be passionate about any Appendix 1.2. Typically, respondents tended to be area and then to write policy on it. It can principal policy advisors with particular respon- insert itself everywhere, it can insert itself sibility for their agencies’ knowledge acquisition by your choice of the evidence, it can insert and utilisation. The same interviewer conducted all itself in the framework you use to select to interviews. analyse your evidence, it can insert itself into the models you need to analyse it, then in 4. General findings those options that you might consider you’ll consciously weigh some options more than 4.1. Knowledge of, and attitudes toward, the others.” science-policy nexus On the other hand there was a broad acknowledge- There was a variable understanding of how re- ment that peer review (as it was understood inter- search-informed evidence plays into the policy nally by staff members) in policy formation has to formation process, along with diverse attitudes be rigorous so that it is used as a mechanism for toward the use of evidence. Some departmental ensuring that the advice is transparent, unbiased staff interviewed had well-informed and developed and based on the evidence rather than from a po- concepts of the relationship between evidence sition of advocacy. However, respondents noted a and policy formation but others had quite naïve tension, particularly in the social policy area, be- concepts of the interaction between science, evi- cause people get passionate about what they are dence and the policy development process and this working on. appeared to be reflected in departmental practice. Most respondents saw scientific knowledge as one There was a broad recognition, however, that if a of the inputs to policy development, but the grow- Cabinet paper, for example, is highly contentious ing international consensus about how science fits sometimes “you have just got to, on the balance into policy was not well understood. of judgements, present the department’s view and expect it is going to be escalated up.” Respondents For example, some staff members appeared to recognised that the more contentious the issue, see their role as primarily about identifying policy the more it is about values judgements: needs or responding to the political context, and therefore saw evidence solely in terms of inform- “Policy is a social and political economy equa- ing those needs, rather than identifying options or tion. The clear thing is that science is a criti- solutions. They did not give priority to seeking or 3 Pielke, R. (2007) The honest broker: making sense of science commissioning research to inform policy options. in policy and politics. Cambridge University Press. Page 27 The role of evidence in policy formation and implementation cal input to policy but you do not go straight situations. One agency uses a preferred provider from ‘this is the science, therefore this is the list which includes Crown Research Institutes, uni- policy’. It’s taken into account alongside the versities and private consultants – but researchers other pieces of information that are critical have to be on the list to be considered for a con- in making that decision. There is also a need tract. There was no information as to how this list to recognise that, for Ministers to make in- was developed or the quality criteria for selection. formed decisions, they need all of the infor- Some of the larger agencies with their own research mation however palatable it may be. They teams tend to do most of the work in-house, but make judgements as to the weighting they utilise external expertise for peer review of some give to that information, and the role of pol- of the work. ‘Social’ agencies that require large- icy advisors is to test science advice, to sep- scale population sampling and data collection will arate the knowledge component from any generally contract that out. The Ministry of Justice advocacy aspect and instead apply standard uses technical advisory groups for major evaluation policy approaches such as systems thinking, or research projects where there is representation intervention logic and NRS principles to the from a range of different agencies. The processes use of the information.” used to identify the appropriate sources (both in- 4.2. Practices employed at the science-policy ternal/external) of the research-informed knowl- edge depends on the issue at hand, and whether nexus the expertise exists in-house or not. Where there Many agencies do not appear to have a formal- are in-house experts in particular fields, they will ised protocol for conducting research or obtaining do the work, particularly with smaller agencies evidence for policy and programme development where the experts are issue/area specific. and evaluation – or at least did not report such a protocol was used. Those that did, however, tend- Agencies dealing with issues in the natural scienc- ed towards a formalised research programme and es almost exclusively sought advice from relevant a formalised evaluation programme as well. Some public institutions, especially CRIs, whereas those agencies have formalised processes and criteria for dealing with issues in the social or economic prioritising and delivering research programmes. areas tended to favour private research organisa- Evaluation of research programmes is generally ad tions and consultancies. Some respondents com- hoc, although some agencies have a more formal mented that the CRI reforms initiated in 2010 have system of routinely assessing three to five research helped with the integration of science into policy projects, areas, or programmes each year (usually development. Overseas providers of evidence are involving international experts). in the minority and their involvement tends to be driven by the specific nature of the research knowl- In general there appears to be an ad hoc appreci- edge required. Australia is the largest overseas pro- ation of the role of science and robust evidence vider (e.g. in the petroleum and minerals sector, within many departments. When questioned how geology, and water quality guidelines). they went about procuring evidence-informed ad- vice, whether it was via internal departmental ad- 4.2.1. The practice of identifying and filling knowl- visors, independent science advisory committees, edge gaps part-time scientific advisors from outside the de- partment (on a contracted or standing basis), or It is important that research-informed knowledge utilising a mixed model (such as technical advisory is gathered in a structured manner, rather than in groups containing both independent scientists and an ad hoc fashion. To that end, therefore, we were representatives of end-users), it was apparent that interested in whether agencies had a set of formal there seem to be a variety of models utilised. A principles or guidelines governing the procure- number of agencies establish panels which oversee ment of scientific advice or identifying and filling the procurement and delivery of advice. On occa- research needs. sion the methods are combined, e.g. the Ministry The survey revealed that procurement policies for the Environment in developing the National and guidelines commonly exist. However, these Environmental Standards generally requires both are limited to setting out a standard purchase contracted advice and technical advisory groups. process, and to supporting decisions about size/ Different approaches are appropriate for different scale, rather than providing guidelines for how to Page 28 The role of evidence in policy formation and implementation procure evidence and scientific advice, or on how • The topic area; to identify and fill research needs specifically. The • Internal expertise and capability (or lack of); survey identified that there is a lack of simple and • Funding availability for external contracts; whole-of-government guidelines or protocols for obtaining expert scientific advice, which depart- • Desire to promote internal capability externally ments can tailor to what is needed in their sector, (e.g. in a CRI); such as those based on the UK model. • Desire to build capacity internally; However there are a number of examples of prom- • Deadlines and expected turn-around times; ising practice in individual agencies that should be • Large-scale fieldwork needs (e.g. customer sat- highlighted. For instance, the former Ministry of isfaction monitoring for the labour function has Fisheries (now part of the Ministry for Primary Indus- to be out-sourced); tries) has developed a range of principles and guide- lines on the procurement of all services, including • Requirements for highly specialised facilities scientific advice and research needs. This ministry is such as PC3 labs (e.g. the Ministry for Primary also currently developing and implementing specif- Industries contracts out the majority of their ic procurement strategies regarding the purchasing research, but keeping it in-house may be due to of research services to ensure transparency and to the need for specialised facilities). identify key providers for partnering. One agency was quite explicit that all research work Two other agencies also stand out in that they now is done in-house unless: apply considerable focus to their knowledge needs. • They do not have the specialist technical exper- The Ministry for Primary Industries has specifically tise in-house; moved from a previous ad hoc approach to setting • They do not have the capacity to deliver the up a more structured process due to influences work within required time frames; from the recent appointment of a departmental Science Advisor. For its part, the Ministry of Civil • There is a particular reason to contract out (e.g. Defence and Emergency Management has a formal some evaluations need to be done outside the process by which guidance to local government is organisation in order for the results to be ac- reviewed every 3 to 5 years and this can drive new cepted as impartial and unbiased); research needs. • There is an advantage to the organisation in en- Furthermore, a small number of agencies have a tering into a strategic partnership with another system whereby departmental Chief Executives organisation to deliver a piece of research; and/or general managers meet with the research • There are budget constraints. manager on a regular basis (usually monthly) re- garding their knowledge needs on the policy front. 4.2.3. The practice of peer review There is a framework of strategic priorities for the One of the most important aspects of sound scien- respective organisations and the sector, which tific knowledge generation is the quality assurance leads them to identify the research needs and how procedures that are in place to assure the validity to meet the priorities. They have a policy work pro- of research done within, or contracted out by, the gramme with ministerial endorsement and there- agency. In general, respondents stated that mate- fore the policy development process can be sup- rial that is published goes through an independent ported by whatever research is deemed necessary. peer review process (often both internal and exter- nal, as required by the domestic or international 4.2.2. The practice of commissioning intramural journal). For non-published material, peer review and extramural research is triggered on a case-by-case basis. For the big- We identified a range of internal processes for ger, more significant or riskier projects, the review undertaking and contracting of policy-relevant re- shifts from internal to external. In one agency, two search, especially in the areas around peer review external peer reviewers are required for everything and quality assurance (QA). There was a range of published or going to the minister. One agency factors that determine whether research is under- stated they had no established procedures, while taken within an agency or contracted out. These in another the statement of intent is the baseline factors included: for establishing QA procedures. Page 29 The role of evidence in policy formation and implementation Agencies that undertake all projects in teams may ment stands out for best practice in this regard for have in-house QA and peer review procedures, the development of their Cost Opportunity Benefit with formal QA checks prior to publication. Other Risk Analysis (COBRA) guide, which underlines that agencies ensure formal incorporation of QA re- evidence, particularly scientific evidence, is a key quirements in research contracts, and peer review input to policy advice, and provides guidance on requirements during and on completion of re- using evidence in policy development. search, or may ensure ISO accreditation of internal laboratories and external science providers. The 4.3.2. Use of evidence for foresighting recent QA standard for fisheries research could be One of the most significant issues that was uncov- 4 considered a model of good practice. A focus of ered by the survey was the sparse consideration the recent MPI redesign was to improve procure- given to foresighting: looking ahead 10 to 30 years ment processes, with the aim of ensuring quality to ascertain how the world will change, and what processes for securing, monitoring and managing policies should be put in place now, in order to contract performance. For research done through, reach that point or to adapt to foreseeable changes. or as an input to, international forums there is usu- When questioned about what degree of foresight- ally international independent review. ing and systematic planning was undertaken to 4.3. Specific knowledge gap areas where a address the potential of significant and long-term robust evidence base is lacking events before they occur, and how this is informed Along with the findings regarding knowledge, by research-derived knowledge, the survey uncov- attitudes and practices toward incorporating quali- ered a tendency by many of the agencies to think ty evidence into the policy process, the survey also of foresighting in the lifespan of their Statement of considered specific areas within ministries’ scope Intent/Minister (i.e. only 3 years). Some may fore- of responsibility that require a more systematic see as far as 5 years, but almost none are envision- approach to evidence gathering and use. These ing the 20-plus year horizon. Many of the agencies included risk assessment, foresighting and technol- surveyed are ad hoc in their approach, and see it as ogy assessment. simply modelling rather than scientific foresighting that could reveal emerging issues such as poten- 4.3.1. Use of evidence for risk assessment tial risks and benefits in the long term of emerging technologies. In terms of risk assessment undertaken in the development of policy, many stated that it was a 4.3.3. Use of scientific expertise and research de- standard part of the policy process. In other words, sign for evaluation it was implicit and therefore they had no explicit procedures for identifying and dealing with it. There The survey revealed major gaps in approaches to was an uneven understanding of the importance of programme monitoring and evaluation of policy in- risk assessment beyond political and financial risk, itiatives. There are two issues here. First, there is and risk related to immediate policy in formation. evaluation of whether a policy is concordant with There is a lack of formal protocols and training in currently available information at the time of its risk assessment. By and large the main aspects of development (which should be part of the Regu- risk were seen to be fiscal, legislative issues and latory Impact Statements, now required by Treas- Treaty of Waitangi, all of which tended to be very ury). The second issue is evaluation of whether the explicitly identified by respondents. The Ministry policy is producing its intended outcomes after its for Primary Industries has risk assessment teams introduction. A number of agencies admitted they because “a fundamental part of a large amount do not systematically undertake post-implemen- of what this organisation does is as a risk manage- tation evaluations of effectiveness, with a spec- ment organisation”. The key aspect of this is where trum of implementation monitoring ranging from science and risk assessment supports risk manage- informal checking through to reviews and formal ment decision-making (e.g. biosecurity, food safe- evaluations. ty, fisheries by-catch). The Ministry for the Environ- The Ministry of Civil Defence and Emergency Man- agement tends to focus on the effect of implemen- 4 Ministry of Fisheries (2011) Research and science informa- tion standard for New Zealand fisheries. Ministry of Fisheries, tation into the civil defence emergency manage- Wellington. ment framework, rather than the science and policy Page 30 The role of evidence in policy formation and implementation process. In agencies where the research team sits these opportunities were sometimes constrained within a wider policy group (e.g. Te Puni Kōkiri) by resource and capacity availability, some people there is greater ability to see the degree of influ- did move in and out of agencies. In addition, the ence and integration of the research into the policy temporary use of interns and rotation of staff to/ being generated. The Parliamentary Commissioner from other government agencies was common. for the Environment will often issue update reports Another issue raised was the relationship between at a period of time after a report is issued, where policy managers and the ‘senior science advisor’ they will comment on matters such as what chang- function. Many agencies felt that the Departmen- es were identified, what has been implemented, tal Science Advisor/Chief Science Advisor role was what has not, and what has changed. not relevant to their operations, especially those Our survey revealed that evaluation of policy up- with a high degree of analysis and modelling. The take and implementation over an extended time question that was often asked was “Who would period is not being undertaken in a systematic the person be reporting to?” It was generally felt manner across the public service. The Ministry of that the key person driving much of the research Economic Development (now part of MBIE) com- and evaluation output is a team manager, but not mented that legislation may have a requirement for a science advisor. This attitude, however, serves to a review every 3 to 5 years, which is the primary confirm the conclusion that understanding of the driver for undertaking the analysis. Within the Min- role of research-derived evidence in policy forma- istry for the Environment, on-going monitoring and tion, along with the potential role of Departmental evaluation of policies is part of the COBRA process Science Advisors, is still very much in development. and is overseen by the ministry’s evaluation func- tion, which is proactive in analysing how their pol- 5. Discussion and conclusions icy is performing and has been built into the latest There appears to be significant unevenness across iteration of COBRA. government regarding departmental use of and respect for research-derived evidence and under- 4.4. Use of scientific expertise standing of how such evidence should fit into the Perhaps one of the most significant findings of the policy process. There is little consistency: govern- survey had to do with the variability in capacity and ment agencies use a range of mechanisms, some capability across departments to engage with sci- more explicit than others, for identifying what re- ence and critically assess and employ scientific input. search and scientific knowledge is needed in their While there was evidence of strong science back- policy formation. There is variable capacity for grounds for policy advisors in some agencies, there using these mechanisms optimally in seeking ad- was more generally a lack of current scientific ex- vice, contracting and evaluating research inputs. pertise. In order to address the need for those sen- In many departments, procedures and protocols ior staff charged with providing science advice to for scientific advice were uncertain. On the oth- keep up to date with developments in their field, er hand, there were examples of good practice in the use of professional development plans allows some departments which provide a template for some staff to identify a number of mechanisms in- more widespread improvement. cluding: conferences; workshops; memberships of Government is clear in wishing to improve the per- relevant societies or professional bodies; regular formance of the public service. One component of access to and communication with peer networks; this might be a general approach across the public participation in science workshops and conferenc- service to strengthening the processes that ensure es; on-line activities such as chat rooms and group high quality scientific assessment, which is one circulations; guest lecture series. However, such important input into policy-making, alongside so- professional development plans are more difficult cial values-based, economic, cultural and political for staff at the more senior levels. The expectation perspectives. Currently, there are no simple and is that they will invest in their own learning and whole-of-government guidelines or protocols for on-going development. expert scientific advice. The availability of agreed Another mechanism for supporting technical staff is generic guidelines would allow departments to through secondments/rotations to/from academ- fine-tune protocols and policies to what is appro- ia, CRIs or industry. The survey showed that while priate for their sector. The relative lack of direct sci- Page 31 The role of evidence in policy formation and implementation ence expertise and the variable nature of scientific this study. Dr Alan Beedle, formerly Chief of Staff of inputs suggest there would be considerable value my office and Dr Rick McGovern-Wilson conducted in following the UK model in the appointment of the survey. Ms Kristiann Allen consulted on report Departmental Science Advisors, whose principal preparation. responsibility would lie in procuring and translating technical advice. Collectively with the Chief Science Appendix 1.1: Agencies surveyed Advisor, they would create a valuable across-gov- The following public service departments, inde- ernment resource. pendent Crown entities and offices of Parliament There appears to be little systematic liaison across were surveyed for the purposes of this report: government to define national research needs for • Department of Building and Housing* the benefit of policy formation and public sector utility. Nevertheless, most of the agencies inter- • Department of Conservation viewed identified some degree of liaison with oth- • Ministry of Civil Defence and Emergency Man- er agencies to identify mutual science/information agement needs, but some cited the social sector as being • Ministry of Economic Development* particularly able to benefit from progress in this direction. Although some agencies identified their • Ministry of Education relationship with Statistics New Zealand as being • Ministry for the Environment sufficient to meet their information requirements, the demise of the Social Policy Evaluation and Re- • Ministry of Health search Committee (SPEAR) drew strong comment • Ministry of Justice from some agencies. • Department of Labour* In summary, the primary outcome of the survey was • Ministry of Māori Development (Te Puni Kōkiri) the highly variable manner in which government agencies integrate evidence-informed knowledge • Ministry for Primary Industries into the development of policy. In some aspects • Ministry of Science and Innovation* of the survey, many agencies showed promising approaches, but none demonstrated a consist- • Ministry for Social Development ently high standard of scientific rigour in policy • Statistics New Zealand development. • The Treasury This variability is concerning, and demonstrates • Children’s Commissioner the potential for establishing general guidelines for incorporation into departmental and agency pro- • Office of the Parliamentary Commissioner for tocols. In addition, the survey results point to the the Environment. merit in considering expanded use of Departmen- (NB: * = now part of the Ministry of Business, Inno- tal Science Advisors to lead agencies in making use vation and Employment) of quality evidence for policy formation. The value of scientific advice in providing options Appendix 1.2: Survey for policy, in assisting risk assessment, technology questionnaire assessment and long-term planning, and in evalu- EVIDENCE IN POLICY: SURVEY OF ating policy implementation is increasingly under- stood in democratic systems. New Zealand policy GOVERNMENT AGENCIES formation and evaluation would be significantly INTRODUCTION enhanced if the issues identified were addressed. This survey is being undertaken to help to under- Acknowledgements stand the role that science-based evidence plays I acknowledge the support of the Prime Minister in the development of policy. It is recognised that and Sir Maarten Wevers, former Chief Executive the response to the science need will vary between of the Department of the Prime Minister and Cab- government agencies, depending on the circum- inet. I thank the Chief Executives of the participat- stances, and therefore not all of the following ques- ing agencies and their staff for their cooperation in tions will necessarily apply equally. Page 32 The role of evidence in policy formation and implementation 1. AUDIT 4. RESEARCH PROCUREMENT a) How many scientifically trained Advisors do This section assumes that new research is needed you have in-house (science = natural, social, to inform a particular policy. engineering and statistics), who are working in a) Do you have standing committee(s) for the a role providing scientific advice? management and maintenance of research b) Do you have anyone in the role of a Depart- needs? mental Science Advisor (DSA) – even if that b) What individuals (definitions and roles) have name is not used explicitly – whose principal scientific responsibility in commissioning re- responsibility is for procuring and translating search, for example in issuing and reviewing technical advice? requests for proposals? c) Does that person consider themselves as ‘head c) What determines whether you do research of profession’ across the Ministry/ Department within the Ministry or contract it out? (M/D) or even whole-of-government? d) Is there a clearly defined balance between internal and external research providers, and 2. IDENTIFYING RESEARCH NEEDS what are the trigger points in making that decision? a) How, at the agency level, do you identify what research and scientific knowledge is needed in e) What scientific quality assurance procedures do you have in place for assuring the quality policy formation? of research done within or contracted out by b) What degree of foresighting is undertaken by the Ministry? the M/D? In other words, (a) what systematic f) If research is undertaken internally: planning is undertaken to address the potential i) Who undertakes the research, what is their of significant and long-term events before they background, and what level are they in the have occurred, and (b) how is this informed by hierarchy? scientific knowledge? ii) What capacity do they have to undertake c) What degree of risk assessment is undertaken the research – do they have a dedicated in the development of policy? research role, are funds made available to d) Do you have a formalised protocol for research support the research? programme development and evaluation? iii) What degree of experience do they have in e) Do you liaise with any other agency to identify undertaking strategic research? mutual science/information needs? iv) What degree of independence are they afforded? 3. PROCESS g) If you commission external research, what pro- portion of that is contracted to: a) How do you procure scientific advice? i) Public research institutions in New Zealand i) Internal Departmental Advisors. (universities and CRIs); ii) Independent Science Advisory committees. ii) Private research organisations or consul- iii) Part-time scientific advisors from outside tancies in New Zealand; the Department (on a contracted or stand- iii) Overseas public or private research ing basis). providers? iv) A mixed model (such as technical advisory h) If you commission external research, please groups containing both independent scien- comment on: tists and representatives of end-users). i) Who sets research aims and protocols? b) What processes do you have to identify the ii) Who monitors that protocols are being appropriate sources (both internal/external) followed? of that knowledge? iii) Who provides research ‘stewardship’ and is c) What arrangements do you have to obtain sci- able to make stop/start decisions? entific advice in crisis situations, such as the iv) Whether a formal report is required for all recent Canterbury earthquakes? commissioned research? Page 33 The role of evidence in policy formation and implementation v) Whether peer review of findings is under- inhibit external researchers and agencies hav- taken to ensure validity, quality and rele- ing access to it? vance before research findings are incor- d) Are any such developments in prospect? porated into the policy process? If so, who performs that peer review? 8. MAINTAINING SCIENTIFIC CAPABILITY vi) Whether the Department maintains a a) How do senior staff, charged with providing knowledge register or report archiving science advice (such as the DSA), keep up to system? date with developments in their field? 5. PRINCIPLES b) What level of support is provided to ensure they are current with available knowledge – a) Do you have a set of formal principles or guide- either through such activities as attendance lines governing the procurement of scientific at conferences, or access to professional advice or research needs? If so, please provide journals? a copy. c) Do any of your technical staff take part in se- b) Please describe how the Department deals condments/rotations to/from academia, CRIs with: or industry? i) Conflicts of interest among science advisors and research providers. 9. ONGOING MONITORING ii) Relatedly, the involvement of end-users in a) Is there a process of monitoring post-policy providing scientific advice. development to ensure that the science advice iii) The independence of external science advi- has been fully and properly integrated into the sors, for example confidentiality and their policy? right to publicly challenge policy decisions. b) What evaluation is there in the out-years as c) How do you ascertain the validity, quality and to how successfully the science informed the relevance of the knowledge obtained? policy?

6. DEVELOPMENT OF POLICY ADVICE c) When necessary, is there any follow-up re- search to explore deviations from the expect- a) Describe how scientific knowledge informs po- ed results derived from the policies that were tential policy options? implemented? b) What procedures are in place to ensure that science advice is developed and reported 10. TRENDS “purely” to inform policy options – the concept a) What dollars go into research – and how is this of the ‘honest broker’ – versus science advi- broken down in absolute value and as a per- sors acting as ‘issues advocates’ who conflate centage of the total M/D budget? advice with values judgements? b) What is the ratio between internal and exter- c) What is the sequence of events for integration nal expenditure on the acquisition of scientific of science with policy development? knowledge and advice?

7. USE OF DATA c) How many science advisors are employed in this M/D now versus 3 years ago [trying to es- a) Is research data obtained by the Department tablish a trend]? made available to other users (e.g. other agen- cies, the public)? 11. KNOWLEDGE OF SCIENCE POLICY INTERFACE b) What issues around interoperability, data a) Do you have any prior knowledge of work in searchability and data warehousing are consid- the science/policy interface, such as that of ered when selecting a research provider? Roger Pielke? c) Are there any legal issues around archiving or availability of your Department’s data, that will

Page 34 The role of evidence in policy formation and implementation Appendix 2 Protocols for seeking scientific advice • Agencies have articulate and knowledgea- Both the US and the UK have introduced general ble spokespeople who can respond in a clear guidelines for obtaining independent scientific ad- non-partisan fashion to media requests regard- vice for governments. ing scientific and technological questions; • Federal scientists can speak to the media about Appendix 2(a): United States their official work, with appropriate coordina- In December 2010, Dr John Holdren, the Assistant tion with supervisors and public affairs offices. to the President for Science and Technology and Public affairs offices may not ask/direct federal Director of the Office of Science and Technology scientists to alter their findings; Policy, issued a memo1 to heads of executive de- • Mechanisms are in place to resolve disputes partments and agencies providing them with guid- that may arise from decisions on whether to ance on implementing the government’s policies proceed with interviews and public activities. on scientific integrity. This guidance deals with four key concepts that have been summarised below for 3. Use of Federal advisory committees consideration in the New Zealand context. Agencies should develop policies for convening sci- entific advisory committees that ensure that: 1. Foundations of scientific integrity in • There is a transparent recruitment process to Government technical committees; Successful application of science in public policy de- • Professional biographical information is made pends on the integrity of the science process both publicly available; to ensure the validity of the information itself and to engender public trust in Government. Agencies • The selection of committee members is based should therefore develop policies that: on expertise, knowledge and relevant scientific contribution in the subject area. Membership • Ensure a culture of scientific integrity by shield- should reflect diversity and balanced points of ing scientific data and analysis from inappropri- view; ate political influence; • Conflict of interest waivers are made publicly • Strengthen the actual and perceived credibili- available except where prohibited by law; ty of government research by: filling scientific positions based on credentials, experience and • The products of such committees should be integrity; ensuring expert peer review of data treated as their own findings rather than gov- as appropriate; setting standards to govern ernment’s, and thus are not subject to internal conflicts of interest; adopting whistle-blower revisions. protections; 4. Professional development of Government • Establish principles for conveying scientific in- scientists and engineers formation to the public; Agencies should establish policies that promote • Ensure that communication includes the un- and facilitate the professional development of gov- derlying assumptions, the uncertainties and ernment scientists and engineers, subject to all ap- the probabilities associated with both best and plicable laws and policies. These policies should: worst case scenarios. • Encourage publication of findings in peer-re- viewed journals; 2. Public communications • Encourage presentation of findings at profes- Agencies should develop communication policies sional meetings; that promote and maximise openness and trans- parency while ensuring compliance around appro- • Allow government scientists to become editors priate disclosure of classified information. Policies or editorial board members of professional or should ensure that: scholarly journals;

1 http://www.whitehouse.gov/sites/default/files/microsites/ • Allow full participation in professional or schol- ostp/scientific-integrity-memo-12172010.pdf. arly societies;

Page 35 The role of evidence in policy formation and implementation • Allow government scientists to receive honours and awards for their contributions to the scien- Key Messages • Identify early the issues which need scientific tific record, where there is no financial gain. and engineering advice and where public en- gagement is appropriate; Appendix 2(b): United Kingdom • Draw on a wide range of expert advice, par- In June 2010, the UK’s Government Office for ticularly when there is uncertainty; Science issued the 4th edition of its guidance • Adopt an open and transparent approach to on the use of scientific evidence within the the scientific advisory process and publish the policy process. The Government Chief Scien- evidence and analysis as soon as possible; tific Adviser’s Guidelines on the use of scien- tific and engineering advice in policy making2 • Explain publicly the reasons for policy deci- sions, particularly when the decision appears supersedes the 2005 document. A summary of to be inconsistent with scientific advice; the most pertinent elements of this guidance is provided below for consideration in the New Zea- • Work collectively to ensure a joined-up ap- land context. proach throughout government to integrat- ing scientific and engineering evidence and 1. The advisory process advice into policy making.

1.1. Triggers for seeking scientific and/or engi- • Departments’ own experts and analysts and neering advice programmes of internally and externally com- Government departments should, in the first in- missioned research; stance, ensure that they have sufficient in-house • Departments’ existing expert advisory systems expertise to deal with the majority of issues requir- such as science advisory committees; ing scientific input and advice. However, there will • Other departments’ research programmes; be occasions when seeking external advice is nec- • Research from non-departmental sources in- essary. In particular, external science advice must cluding international bodies; be sought when: • The issues raise questions that are beyond the • National Academies and learned societies; expertise of in-house staff; • The broad science and engineering communi- • Responsibility for a particular issue cuts across ty; for example universities, private and charity government departments; sector research and development funders. • A wide range of expert opinion exists and/or Consideration should be given to understanding there is considerable uncertainty; the broader international perspectives on issues, through the work undertaken in embassies and • New findings are emerging rapidly; consulates. • There are potentially significant implications for areas of public policy; 1.3. Expectations • Public confidence in science advice from gov- The roles, responsibilities, expectations and bound- ernment could be strengthened. aries of external advisors should be clarified at the outset so that there is no misunderstanding in the 1.2. Sources of scientific advice advisory process. These include: Departments should encourage and enable staff • Review of existing data; members responsible for individual issues to estab- • Collection and analysis of new scientific data; lish new networks continually in order to capture the full diversity of good evidence-based advice. • Interpretation of research from different Sources for research and science advice include: sources;

2 http://www.bis.gov.uk/assets/goscience/docs/g/10-669-gc- • Application of expert judgement where data is sa-guidelines-scientific-engineering-advice-policy-making.pdf. lacking or inconclusive; Page 36 The role of evidence in policy formation and implementation • Identification of policy options based on data dence and its use. Scientific advisory committees, and research evidence; learned societies, academics and other experts • Providing expert scientific advice on policy op- can assist in this peer review process. The level of tions; quality assurance and peer review to which the ev- idence was subject should be clearly stated when • Risk and uncertainties. departments are responding to public concerns When assessing the levels of risk or establishing over emerging issues. risk management strategies in relation to a specif- ic policy, all known sources of uncertainty must be 2. Capacity and capability taken into account, along with differing perspec- Departments and policy makers should work collec- tives of risk. Early public engagement is often vital tively to ensure a whole-of-government approach to this process. to integrating scientific and engineering evidence and advice into policy making. Varying levels of uncertainty are inevitable and de- partments must recognise, communicate and man- Government departments and agencies need suf- age this. Government officials or staff should not ficient capacity to recognize the full spectrum of press experts to come to conclusions that cannot relevant evidence and to know how to access it. be justified by the evidence. They may be assisted in this by those working in a ‘knowledge-brokering’ capacity. 1.4. Quality assurance and peer review Benchmarking reviews of how departments use All evidence should be subject to critical evaluation and manage scientific and engineering evidence which is proportionate to the nature of the evi- have been conducted in the UK3, with on-going de- partmental scrutiny and self-assessment.

3 http://www.bis.gov.uk/go-science/science-in-government/ reviewing-science-and-engineering.

Page 37 The role of evidence in policy formation and implementation Appendix 3 Suggested framework for the Suggested characteristics of New appointment of Departmental Science Zealand Departmental Science Advisors and their terms of reference: Advisors Extrapolation from the UK model While the UK experience suggests that a ‘one-size- The framework suggested in the main report draws fits-all’ DSA model may not be desirable or achiev- on a recent UK House of Lords Science Committee able across all government departments, there are report on the topic.1 The following presents key ele- nonetheless a number of essential characteristics ments of that report. common to the position. These would apply equal- ly in the New Zealand context. They include: Suggested core functions of Departmental Science Advisors (DSAs) • Good standing within the scientific community as result of his or her scholarly experience or In the UK, all ministerial government departments achievements. This first and most important now include a DSA post, though not all are currently characteristic allows the DSA to access and filled. These posts are always given to senior scien- benefit from the expertise of the national and tists who are in a recognised position to influence global community of scientists and also ensures departmental decision making. Their primary role that the DSA can exercise sufficient influence is to ensure that a robust and integrated evidence within the department. base underpins policy formulation, delivery and evaluation. This role can be further broken down • Good communication and management skills. as follows: These are practical skills that are essential to any senior post. It includes building effective • Manage the development, implementation and relationships and, in this case, often bridging monitoring of a science and innovation strategy quite different communities and cultures. for the department; • Public engagement skills. While the country’s • Lead departmental engagement on relevant Chief Science Advisor would necessarily take on national and international science and engi- the high profile work of addressing the public neering issues; when asked by the Prime Minister, the DSAs • Network and share good practices across gov- nonetheless need similar skills to engage in ernment. content-area consultations etc. These skills are Accordingly, the DSA functions usually include the important in helping to build and maintain pub- following: lic trust in the science-policy nexus. • Provide an ‘advise and challenge’ function di- • Understanding the policy environment and rectly to the department’s most senior officials risk assessment of policy. To be effective, a including the minister; DSA must understand the policy landscape, structures and timeframes. DSAs must have the • Independently challenge the evidence base for critical analytical skills to assess risks associated departmental policies; with various policy options. This means draw- • Oversee departmental systems for ensuring ing on skills to evaluate evidence and weigh up that policy makers consider relevant evidence; sometimes conflicting evidence from a wide • Oversee any departmental scientific advisory range of relevant disciplines. committees; • Project delivery skills. Experience with project • Manage the departmental research budget; delivery in government is a desirable charac- • Ensure exemplary departmental science and teristic in a DSA to help reduce the likelihood engineering quality and capability; of policy programme failures. As the UK Select Committee report suggests, it’s important that • Guide analytical staff in conducting assessments. DSAs have experience in ‘delivering the science, 1 House of Lords Select Committee on Science and Technology rather than the theory of the science.’ (2012) Report: The role and functions of Departmental Chief Scientific Advisers. http://www.publications.parliament.uk/ The departmental recruitment process for a DSA pa/ld201012/ldselect/ldsctech/264/264.pdf. should be designed to ensure that the chosen can-

Page 38 The role of evidence in policy formation and implementation didate possesses, at minimum, the above charac- merits consideration. This could help to support teristics. This does not preclude consideration of the DSA’s active and on-going engagement with internally recruited staff members for the post; scientific communities outside of government. however the ability to engage with the scientific • Access to ministers: the DSA should have ac- community as a respected peer is a key considera- cess to their relevant minister(s) to ensure that tion in recruitment. the evidence they present can influence deci- Suggested terms of employment of sions at the highest level. New Zealand Departmental Science • Guidelines in the event of disagreement: the Advisors UK model recommends the development of Based on UK experience, and adapted for the New clear guidelines to be invoked in the event of Zealand context, the following are some considera- disagreement between the DSA and ultimate tions in regards to structuring the terms of employ- decision makers regarding the implication of ment of DSAs. science in policy decisions. • Part-time employment: it is worth considering • Position assessment: the DSA should be po- a part-time/cross-appointment of a DSA so that sitioned such that they have the authority to the incumbent can maintain links with academ- work directly with ministers and across the ia or industry. In exceptional cases where the whole department. DSA takes on major management or project • Relationship with CSA: the DSA should have a delivery functions, full-time employment may ‘dotted-line’ relationship to the Prime Minis- be necessary. ter’s CSA so that each can assist the other as • Length of tenure: a seconded position of at appropriate. This will also encourage the de- least 3 years (with the possibility of renewal) velopment of a cohesive community of DSAs.

Page 39

ISBN 978-0-477-10404-3