DISCUSSION PAPER SERIES January 2015 Tilman Brück Yashodhan Ghorpade Vegard Iversen Aladysheva Anastasia Puri Jyotsna Humanitarian Assistance? What Methods May Be Used in Impact Evaluations of IZA DP No. 8755 of Labor Institute for the Study zur Zukunft der Arbeit Forschungsinstitut
What Methods May Be Used in Impact Evaluations of Humanitarian Assistance?
Jyotsna Puri Yashodhan Ghorpade International Initiative for Institute of Development Studies Impact Evaluation (3ie)
Anastasia Aladysheva Tilman Brück Stockholm International Peace SIPRI and IZA Research Institute (SIPRI)
Vegard Iversen University of Manchester
Discussion Paper No. 8755 January 2015
IZA
P.O. Box 7240 53072 Bonn Germany
Phone: +49-228-3894-0 Fax: +49-228-3894-180 E-mail: [email protected]
Any opinions expressed here are those of the author(s) and not those of IZA. Research published in this series may include views on policy, but the institute itself takes no institutional policy positions. The IZA research network is committed to the IZA Guiding Principles of Research Integrity.
The Institute for the Study of Labor (IZA) in Bonn is a local and virtual international research center and a place of communication between science, politics and business. IZA is an independent nonprofit organization supported by Deutsche Post Foundation. The center is associated with the University of Bonn and offers a stimulating research environment through its international network, workshops and conferences, data service, project support, research visits and doctoral program. IZA engages in (i) original and internationally competitive research in all fields of labor economics, (ii) development of policy concepts, and (iii) dissemination of research results and concepts to the interested public.
IZA Discussion Papers often represent preliminary work and are circulated to encourage discussion. Citation of such a paper should account for its provisional character. A revised version may be available directly from the author. IZA Discussion Paper No. 8755 January 2015
ABSTRACT
What Methods May Be Used in Impact Evaluations of Humanitarian Assistance?
Despite the widespread occurrence of humanitarian emergencies such as epidemics, earthquakes, droughts, floods and violent conflict and despite the significant financial resources devoted to humanitarian assistance, systematic learning from such interventions using rigorous theory-based impact evaluations is very rare. The objective of this paper is to examine the extent to which scientific impact evaluation methods can provide evidence to improve the effectiveness and efficiency in humanitarian action. This paper explores the methodological options and challenges associated with generating high quality evidence needed to answer key questions about the performance of humanitarian assistance, including whether assistance is reaching the right people, at the right time, is bringing about the desired changes in their lives (effectiveness) and is being delivered in the right doses, ways and with manageable costs (efficiency). With the help of six case studies and drawing on real-life examples from the small but growing academic literature, we demonstrate how impact evaluation methods can be used successfully and in an ethical manner to improve humanitarian assistance. A key lesson from our review is that it pays to be prepared. Much information is being collected these days about the risks of various emergencies unfolding, be they sudden onset or slow onset emergencies. Hence national actors and international donors can prepare for these events and for conducting meaningful impact evaluations. Given the overwhelming needs and the lack of funds, doing more with limited resources is a key challenge for humanitarian assistance and impact evaluation is one way of achieving this.
JEL Classification: H84, C93, O12, Q54
Keywords: impact evaluation, methodology, research design, statistics, humanitarian emergency, humanitarian assistance, disaster, violent conflict, reconstruction, aid, development
Corresponding author:
Jyotsna Puri International Initiative for Impact Evaluation (3ie) 202-203, 2nd floor, Rectangle One D-4, Saket District Center New Delhi - 110017 India E-mail: [email protected] Acknowledgements
This paper is part of a larger study supported by UKaid through the Department for International Development and USAID. The scoping paper examines the scope of evidence and need for evidence in humanitarian assistance. The scoping paper has been prepared by Evidence Aid with support from 3ie. 3ie, along with assistance from the SIPRI and humanitarian assistance experts, led the work on the methods paper.
Jyotsna Puri provided overall leadership and management of these papers with support from Deo-Gracias Houndolo and Peter Giesen. Bharat Dhody provided research assistance.
The team that worked on the scoping and methods paper included: Anastasia Aladysheva, Claire Allen, Frank Archer, Tilman Brück, Mike Clarke, Anneli Eriksson, Yashodhan Ghorpade, Peter Giesen, Vegard Iversen, Jyotsna Puri and Diana Wong. The authors are also grateful to Howard White, Jeannie Annan, John Mitchell, Francesca Bonino, Joanna Macrae, Christine Kolbe, Alison Girdwood, Jonathan Patrick and Joanna Macrae.
We are also grateful to members of the steering committee which included: Caroline Andreson, Jeannie Annan, Alison Girdwood, Langdon Greenhalgh, Penny Hawkins, Christine Kolbe, Joanna Macrae, John Mitchell, John Murray, Jennie Richmond and Howard White.
iii
Executive summary
Humanitarian crises are complex situations where the demand for aid has traditionally far exceeded its supply. The humanitarian assistance community has long asked for better evidence on how each dollar should be effectively spent. Impact evaluations of humanitarian assistance can help answer these questions and also respond to the increasing call to estimate the impact of humanitarian assistance and supplement the rich tradition for undertaking real-time and process evaluations in the sector. This working paper gives an overview of the methodological techniques that can be used to address some of the important questions in this area, while simultaneously considering the special circumstances and constraints associated with humanitarian assistance.
Key findings for the scope of future study
This working paper is part of a larger study undertaken to assess the scope and methods for impact evaluation in the humanitarian sector. Findings from the scoping paper show that:
• Insufficient high-quality evidence: High-quality evidence that can causally relate changes in the conditions of people and their outcomes to specific programmes and interventions undertaken in humanitarian assistance are clearly scarce. In an investigation of studies conducted since 2005, we found 39 studies that could be described as impact evaluations that used (implicit or explicit) comparison groups to measure attributable change. However, these too were deficient in many ways: 29 had a theory of change but 23 did not show whether the choice of comparison groups was valid (i.e. did not have balance tests); 29 did not discuss the confidence with which their results were measured (i.e. did not undertake power analyses or show sample size calculations) and only five discussed ethical issues.
• Sectors: Using gap-maps of evidence, the study finds that most high-quality studies of humanitarian assistance are in the area of health (and particularly mental health), nutrition and peace building.
• Timing: Most existing impact evaluation studies examine changes in conditions and resilience once the affected area is in the recovery phase (there are approximately 27 studies that examine the results of peace building and conflict prevention). There are few studies of unanticipated disasters (four) —all of which examine recovery and resilience and few studies (six) of efforts of immediate relief.
• A needs map: A needs map drawn from interviews and strategy documents helped us visualise main areas in which practitioners require additional evidence and research. In particular, more than 20 per cent cited accountability, food security, protection, water and sanitation, and health and said that it was important to assess their impact not just on food security but also on nutrition, income and, in the longer term, on recovery and resilience. Education, humanitarian assistance as a whole, nutrition and logistics were said to be important for study by 10–20 per cent and
iv less than 10 per cent suggested emergency telecommunications and camp management as areas that require additional evidence and research.
Key findings for methods used in impact evaluations of humanitarian assistance
There are many constraints that impact evaluations need to overcome in humanitarian situations, in addition to those that are faced in studies that are undertaken in less complex and challenging situations. The robustness of studies can be especially compromised in the absence of baseline data and inability to plan for and construct counterfactuals. The need for speed of action and low predictability of such situations also means that little advance preparation is possible. Furthermore, most humanitarian situations have a multiplicity of actors and it is usually difficult to de-couple actions and outcomes. High-covariability or the fact that conflict and disasters don‘t usually have clean boundaries means that it is also difficult to find or establish comparable groups that can serve as counterfactuals in a scientifically robust and ethically sound way. Last but not least, there is a lack of impact evaluation experts in the humanitarian sector and a lack of humanitarian experts in the impact evaluation sector.
However, traditional evaluations that either monitor processes or assess if targets have been achieved are clearly insufficient for robust evidence by themselves. They are unable to examine unintended consequences; or to deal with a variety of biases, such as selection bias (i.e. areas targeted by humanitarian assistance are likely to have attributes that make them more or less likely to recover, compared to the average), non-random attrition (are unlikely to count people who either migrated as a result of the intervention or those who perished as a result of the disaster), and contamination bias (areas targeted by one actor are also likely to have other sources of assistance that may make it difficult to separate the different sources of changes); and are unable to measure the change that has occurred as a consequence of their action.
Clearly, new ways must be forged to combine the strengths of methods and traditions that exist in the sector so that these can be used to enhance evidence, while responding to increased demand for accountability in the context of rapid- onset and protracted crises. Impact evaluations can help answer other questions, such as: How much of the change in conditions was a result of the programme? Was the affected population able to recover to their pre-disaster levels? Are we ‗building back better‘? How much of the programme or intervention should be delivered, at what time, and with what frequency? What is the best way to deliver an assistance package? What difference did it make? Can it be delivered in a more cost-effective manner? How much difference did an agency make? Were some groups better off as a result of the programme compared to others?
v