Zenda Ofir EES Conference, Prague, Oct 2010
Total Page:16
File Type:pdf, Size:1020Kb
Zenda Ofir EES Conference, Prague, Oct 2010 1 “Evaluation was conceived as an undertaking useful in …. an open society…. an experimenting society …. in which we ask serious and important questions about what kind of society we should have, and directions we should take.” Tom Schwandt, “Educating for Intelligent Belief in Evaluation”, AEA Keynote published in AJE 29(2), 2008 2 “We …. have to be wary of the latest fads in the development field. They are frequently transformed into simplistic and extremist ideologies which often cruelly mark the life of nations…. ” Jacques Lesourne 25th Anniversary of the OECD Development Centre “The field of development is a veritable junkyard of abandoned models, each focused on a particular aspect while ignoring the rest.” Brian Walker, former Executive Director, Oxfam 3 “The substitution of reasoned assessment for ‘spin’ - (is) the act of …. relating a story in such a way as to influence public opinion…. (it) often uses strategies such as cherry-picking and euphemisms or doublespeak….” Tom Schwandt, “Educating for Intelligent Belief in Evaluation” AEA Keynote published in AJE 29(2), 2008 4 ` ‘Rigorous’ Delegitimisation of other ` ‘Scientific’ Notion of a designs/ hierarchy of ` ‘Hard data’ methodologies designs, as ‘unscientific’, ` ‘Credible evidence’ methodologies ‘not rigorous’, ` ‘Evidence-based’ ‘not credible’ 5 Especially prominent in Impact Evaluation - the design ‘hierarchy’ 1. Experimental 1. True experimental design 2. Regression-discontinuity 2. Quasi-experimental design 3. ‘Non-experimental’ - e.g. 3. Time-series design • Single case study design 4. Constructed matched • Comparative case studies comparison group design design 5. Exhaustive alternative causal identification and elimination • Statistical correlation (used design with these designs) 6. Expert opinion summary (using General Elimination judgment design Methodology, Multiple Lines & 7. Key informant summary Levels of Evidence, judgment design contribution analysis, process Paul Duignan (2008) tracing, etc.) 6 “We know now that one of the central principles of development theory and practice is that those programs that are most precisely and easily measured are the least transformative, and those that are most transformative are the least measureable.” From: The Clash of the Counter-bureaucracy and Development Andrew Natsios, July 2010, CGD Essay 7 Key development intervention types stand to be neglected : 1. Interventions with insufficient numbers (n) 2. Complicated interventions with multiple partners, objectives, expected outcomes - e.g. capacity building 3. Complex interventions – risky, highly adaptive 4. Interventions with heterogeneous outcomes – ◦ cannot assess disparities ◦ track important evolving outcomes/impacts over time. 8 Poor practice in, and under-funding of those development interventions most likely to lead to transformation and sustained development 1. Towards standardized, simple projects - away from risk and innovation towards 2. Away from integrated, transformative solutions and critical development priorities (institution building, empowerment, governance, livelihoods, ecosystems) 3. Away from a focus on understanding systems and complexity 9 5. Away from evaluating ‘mechanisms’ and understanding the role of context 6. Away from other issues in need of serious attention in evaluation - in particular how to evaluate for sustained development, up-scaling, unintended consequences 7. Away from comprehensive synthesis of all information about development - synthetic reviews 10 1. Increasing potential for progression from ‘spin’ to policy 2. Power of ‘converts’, ‘missionaries’ and various lobby groups 3. NONIE experiences 11 “Creating a culture in which randomized evaluations are promoted, encouraged and financed has the potential to revolutionize social policy during the 21st Century, just as randomized trials revolutionized medicine during the 20th. Esther Duflo in editorial, “The World Bank is finally embracing Science” (2004) 12 Factors that might accelerate decision-makers’ susceptibility to ‘spin’ Confluence of factors clustered around • Increasing competition for resources – national, regional, global level • Political movements to the right • Decreasing attention spans, increasing information flood - need for soundbytes, ‘easy think’, numbers • Not enough understanding, capacities • Powerful actors with agendas, profile - and a hold over (resources for) knowledge • Orthodoxy spin mechanisms to influence policy 13 ` “Set in place rigorous procedures to evaluate the impact of policies and programs, report on results and reallocate resources accordingly, incorporate relevant evidence and analysis from other institutions, and inform the policy and budget process.... ` ....undertake a more substantial investment of resources in monitoring and evaluation, including with a focus on rigorous and high-quality impact evaluations.” 14 Link to site “One of the principles motivating the President’s Budget is that, as a nation, of Coalition we haven’t been making the right investments to build a new foundationfor for economic prosperity ..... Evidence- based Policy But, in making new investments, the emphasis has to be on "smarter.” ` ….Wherever possible, we should design new initiatives to build rigorous data about what works and then act on evidence that emerges ..... ` ..... and shutting down those that are failing. ` …. Over time, we hope that some of those programs will move into the top tier — but, if not, we’ll redirect their funds to other, more promising efforts. 15 “An agreed methodology would allow comparing the results reaped by very different actions… Each actor in development aid…. would have the possibility to declare the results of his development action measured according to this methodology. QuODA? An international effort would urgently need to fill the evaluation gap…. Harmonizing methodologies between institutions…. Undertaking a global rating and ranking of public and private institutions along the lines of agreed methodologies …. This would create emulation, and encourage opting out of weak players.” 16 Key actors who can shape (Impact) Evaluation US BASED OR LINKED KEY ACTORS AND COALITIONS NONIE ECG (group of •IPA / CGD / 3ie / JPAL / DIME etc. OECD DAC IFI / Dev. Banks •Promising Practices Network Evaluation Evaluation Network Offices) •Campbell / Cochrane Collaborations •What Works Clearinghouse Evaluation •Coalition for Evidence-based Policy UNEG (UN Associations and •Etc. Networks under Evaluation Group) IOCE auspices EMERGING ACTORS ALNAP •China / BRICs (Humanitarian/emergency •Venezuela work coalition) •Gulf States Private Sector GOVERNMENTS Institutions NGOs •Corporate •Within the Executive •UNIVERSITIES •Local •Serving the •Churches •National •FOUNDATIONS & legislative & THEIR COALITIONS •Sectoral •INGOs judicial •Topical •ALLIANCES •Embracing all •RESEARCH CENTRES levels of state, •Financial para-statal and meta-statal units 17 “Quality impact evaluations are those which use the appropriate methods and deliver policy-relevant conclusions.” “…. The agenda should not be driven by amenability to certain impact evaluation methodologies.” “NONIE advocates an eclectic and open approach to finding the best methods for the task of impact evaluation…” “NONIE seeks to …use the best available methods from whatever source, and to develop new approaches that are can tackle unanswered questions about development impact, particularly for more complex interventions …that have so far been under-evaluated.” 18 ` “A tighter focus does have some costs and risks, including that developing countries may feel that their needs have not been fully considered or that a narrow view develops and gains currency. ` “...accept that advocacy is less urgent now that the profile of IE has increased and commissioning more impact evaluations internationally are being actively pursued by 3ie. ` “3ie is also providing services in synthesising the existing literature and identifying enduring questions.” 19 ` Impact: “….net changes for a particular group of people …. that can be attributed to a specific program…. using the best methodology available ….” ` Quality standards for inclusion in its IE database: “Quality evaluations employ either experimental or quasi- experimental approaches.” ` From the 3ie Glossary: “Impact evaluations have either an experimental or quasi-experimental design.” 20 Cordaid’s experiences with and lessons learned on Participatory Impact Assessment Cordaid 9 Almost 1000 partner organisations in 40 countries. 9 Programmes: Participation Emergency aid and reconstruction Health & Wellbeing Entrepreneurship & microfinance 13-10-2010 2 Why Participatory Impact Assessment? 1 Partner organisations in general have well‐developed informal feedback mechanisms But: –Reports and project evaluations don’t give information on impact and little on outcome –No use of comparison: no baseline data or triangulation – Programme evaluations commissioned by Cordaid for accountability face the problem of lack of data, and therefore their quality is often substandard 13-10-2010 3 Why Participatory Impact Assessment? 2 ‐ Growing public and political pressure on development organisations to show results/outcome ‐ CIDIN (Dutch research institute) proposed Cordaid to pilot methodology for Impact Assessment. CIDIN is responsible for data gathering and processing, Cordaid finances pilot and connects with partner organisations ‐ Methodology is based on with/without + before/after as a principle for evidence