Better Indigenous Policies: the Role of Evaluation
Total Page:16
File Type:pdf, Size:1020Kb
Better Indigenous Policies: The Role of Evaluation Roundtable Proceedings Canberra, 22–23 October 2012 Commonwealth of Australia 2013 ISBN 978-1-74037-433-0 This work is copyright. Apart from any use as permitted under the Copyright Act 1968, the work may be reproduced in whole or in part for study or training purposes, subject to the inclusion of an acknowledgment of the source. Reproduction for commercial use or sale requires prior written permission from the Productivity Commission. Requests and inquiries concerning reproduction and rights should be addressed to Media and Publications (see below). This publication is available from the Productivity Commission website at www.pc.gov.au. If you require part or all of this publication in a different format, please contact Media and Publications. Publications Inquiries: Media and Publications Productivity Commission Locked Bag 2 Collins Street East Melbourne VIC 8003 Tel: (03) 9653 2244 Fax: (03) 9653 2303 Email: [email protected] General Inquiries: Tel: (03) 9653 2100 or (02) 6240 3200 An appropriate citation for this paper is: Productivity Commission 2013, Better Indigenous Policies: The Role of Evaluation, Roundtable Proceedings, Productivity Commission, Canberra. The Productivity Commission The Productivity Commission is the Australian Government’s independent research and advisory body on a range of economic, social and environmental issues affecting the welfare of Australians. Its role, expressed most simply, is to help governments make better policies, in the long term interest of the Australian community. The Commission’s independence is underpinned by an Act of Parliament. Its processes and outputs are open to public scrutiny and are driven by concern for the wellbeing of the community as a whole. Further information on the Productivity Commission can be obtained from the Commission’s website (www.pc.gov.au) or by contacting Media and Publications on (03) 9653 2244 or email: [email protected] Foreword The Productivity Commission’s 2012 policy roundtable, held at Old Parliament House in Canberra on 22–23 October, examined the topic ‘Better Indigenous policies: the role of evaluation’. Participants included representatives of Indigenous and non-government organisations, government officials, academics and consultants. As secretariat for the Review of Government Services, under the Council of Australian Governments (COAG), the Productivity Commission is responsible for the development and publication of the regular Overcoming Indigenous Disadvantage report. The report has demonstrated that, despite several decades of good policy intentions and effort, there remain unacceptable disparities between outcomes for Indigenous and other Australians. As noted at the roundtable by Gary Banks, then chairman of the Productivity Commission, ‘it is said that the greatest tragedy of failure is failing to learn from it. But that seems to be the predominant history of Indigenous policies and programs’. Yet the situation is not hopeless; the roundtable discussed several examples of good practice in evaluation of Indigenous programs, from both Australia and overseas. Our aspiration for this roundtable was to identify principles and practices to guide the better use of evaluation in Indigenous policy in the future — so that more high- quality evaluations will be undertaken and the evidence gained from them will drive policy improvements that benefit Indigenous people and the wider community. While we found genuine, broadly based support for better processes and outcomes, and confidence that we do know at least some of the key problems and potential responses, there was general agreement that an overarching policy review was required to address fundamental issues of ‘government governance’: the way governments work with, and in, Indigenous communities. Mike Woods Deputy Chairman April 2013 FOREWORD iii Contents Foreword iii Summary of roundtable discussions 1 Session 1 Introduction 1 Introduction— why this roundtable? 17 Gary Banks 2 Outcomes for Indigenous Australians — the current situation 21 Robert Fitzgerald 3 The Indigenous policy experience 1960 to 2012 51 Fred Chaney Session 2 The challenges in evaluating social and Indigenous policy 4 Challenges in evaluating Indigenous policy 69 Les Malezer 5 The case for making public policy evaluations public 81 Deborah A Cobb-Clark Dinner address 6 Holding governments to account 93 Jody Broun Session 3 Key themes in evaluation of Indigenous policy and programs 7 Designing evaluation strategies 107 Matthew James 8 Data for better Indigenous policy evaluation: achievements, constraints and opportunities 119 John Taylor 9 Evaluating Indigenous programs and policies: communicating the outcomes 131 David Kalisch and Fadwa Al-Yaman CONTENTS v Session 4 International experience 10 Better Indigenous policies: an Aotearoa New Zealand perspective on the role of evaluation 155 Helen Moewaka Barnes 11 Is evaluation a tool for social justice? Reconciliation? Control? — Reflections on the Canadian experience in Indigenous affairs 183 Frances Abele Session 5 Evaluation and policy 12 The use of evaluation in Indigenous policy and program development: the case of Stronger Futures in the Northern Territory 227 Michael Dillon 13 Learning from experience? Getting governments to listen to what evaluations are telling them 245 Prue Phillips-Brown, Tim Reddel and Brian Gleeson Appendixes A Roundtable program 267 B Roundtable participants 271 vi CONTENTS Summary of roundtable discussions ‘We will be equally harshly judged in the future if we don’t act to drive change.’ On 22–23 October 2012, the Productivity Commission held a policy roundtable on the role of evaluation in improving outcomes for Australia’s Indigenous peoples. It brought together key thinkers in Indigenous policy to discuss the particular challenges in Indigenous policy evaluation and the actions needed to ensure that evidence gained from evaluations is used in policy-making and program implementation. At five sessions of the roundtable, the formal presentation of papers was followed by wide-ranging discussion. This summary of the discussions is structured to address the themes of: • the mechanics of Indigenous evaluation • institutionalising better evaluation practices and use of evidence • international experiences • evaluation and broader Indigenous policy • where to from here? While the summary refers to the presentations relevant to each theme, it does not identify the sources of comments made during discussions, as the roundtable was conducted under the Chatham House rule. The mechanics of Indigenous evaluation ‘Incomplete, non-systematic reporting of results undermines reliability.’ Relevant chapters • Les Malezer (Chapter 4) argued that conventional evaluation methodologies used by government fail to include Indigenous people’s expectations, perspectives and participation in the delivery of services. Any evaluation of Indigenous social policy should recognise the right of Aboriginal and Torres Strait Islander peoples to self-determination and empowerment (including the SUMMARY OF 1 DISCUSSIONS provision of necessary technical and financial resources to assist their development). • Deborah Cobb-Clark (Chapter 5) addressed the particular challenges associated with impact evaluation of Indigenous policy, including data limitations, lack of appetite for randomised control trials, difficulty establishing causation among multiple programs, and limitations imposed by time, budget and ‘political’ constraints. • Matthew James (Chapter 7) provided some practical lessons for those undertaking Indigenous evaluations. He drew on work of the Department of Families, Housing, Community Services and Indigenous Affairs, including the evaluation of the Northern Territory Emergency Response (NTER) and the evaluation of Cape York Welfare Reform. • John Taylor (Chapter 8) noted that the range and volume of data comparing Indigenous and non-Indigenous Australians have grown substantially and can inform the high-level evaluation of the Closing the Gap agenda. However, he argued that the available statistics do not meet the needs of Indigenous people, who are increasingly seeking community-level information to inform local decision-making. He also argued that Indigenous groups need support to build capacity to compile and use customised data to meet their needs. • David Kalisch (Chapter 9) noted that, where there are gaps in available information, the Closing the Gap Clearinghouse is considering using ‘realist synthesis’, which involves assessing types of evidence that may otherwise be discarded as they do not meet medical research standards, including observational studies, case studies, field visits, expert advice, lay knowledge and reports on interventions. • Michael Dillon (Chapter 12) (paper presented by Matthew James) highlighted the importance of developing the evidence base through the use of administrative data sets, robust survey tools and community consultation. He noted that, as it can take a long time to build evidence, the program logic should be clear during policy development and open to change as monitoring and evaluation inform continual learning. Discussion Participants noted that the term ‘evaluation’ can have different meanings in different contexts, ranging from mechanisms designed to provide accountability (for example, for spending government money), through measures encompassing process evaluation and/or impact evaluation of either single programs or groups of programs, to broad reviews of system