Research Evidence in the Humanitarian Sector A PRACTICE GUIDE ACKNOWLEDGEMENTS
We are very grateful to those who participated in the Evidence Lounge in Geneva, Switzerland in September 2017 at which this guide was discussed, provided comments on earlier drafts of the Guide or did both: Prisca Benelli (Save the Children), Kate Bingley (Christian Aid), Jonathan Breckon (Alliance for Useful Evidence), Alyoscia D’Onofrio (International Rescue Committee - IRC), Tarah Friend (UK Department for International Development - DfID), Tara Karul (International Initiative for Impact Evaluation - 3ie), Paul Knox-Clarke (Active Learning Network for Accountability and Performance - ALNAP), Flurina Kuhn (UK Science & Innovation Network), David Loquercio (Core Humanitarian Standard on Quality and Accountability - CHS), Tim Martineau (Liverpool School of Tropical Medicine - LSTM), Virginia Murray (Public Health England - PHE), Alice Obrecht (ALNAP), Aninia Nadig (Sphere Project), Adina Rom (ETH Zurich), Rodolfo Rossi (International Committee of the Red Cross - ICRC), Eva Svoboda (Overseas Development Institute – ODI, and Humanitarian Policy Group - HPG), Pierluigi Testa (Terre des hommes), Erik Von Elm (Cochrane Switzerland), Karin Wendt (Humanitarian Exchange and Research Centre - HERE). Furthermore, the following organisations contributed examples (including case studies) and other content: 3ie, ALNAP, Christian Aid, Evidence Aid, International Committee of the Red Cross, International Rescue Committee, Liverpool School of Tropical Medicine, London School of Hygiene and Tropical Medicine, Overseas Development Institute, Public Health England, Save the Children, and ShelterBox.
CITATION FUNDING This document should be cited as: Funding for this document was provided Blanchet K, Allen C, Breckon J, Davies P, by the UK Science & Innovation Network Duclos D, Jansen J, Mthiyane H, Clarke M. (represented by SIN Switzerland, British (2018) Using Research Evidence in the Embassy Berne). Humanitarian Sector: A practice guide. London, UK: Evidence Aid, London School CORRESPONDENCE of Hygiene and Tropical Medicine and Correspondence about this document Nesta (Alliance for Useful Evidence). should be sent to Karl Blanchet, Director of the Health in Humanitarian Crises AUTHORS Centre, London School of Hygiene This document was written by Karl & Tropical Medicine, Tavistock Place, Blancheta, Claire Allenb, Jonathan London, UK; and Mike Clarke, Research Breckonc, Phil Daviesb, Diane Duclosb, Director of Evidence Aid, Centre for Public Jeroen Jansenb, Helen Mthiyanec and Health, Queen's University Belfast, ICS Mike Clarkeb (a Health in Humanitarian Block A, Royal Hospitals, Belfast, UK. Crisis Centre, London School of Hygiene Karl: [email protected] and Tropical Medicine; Evidence Aid; b Mike: [email protected] c Nesta, Alliance for Useful Evidence).
Cover Photo © Patrick Brown/UNICEF/Panos Pictures
1 | RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR THE WORK ON THE GUIDE WAS CO-ORDINATED BY THREE ORGANISATIONS:
Evidence Aid (www.evidenceaid.org) A four-year research and capacity- champions an evidence-based approach building programme, RECAP, was in the humanitarian sector. Its mission is launched in 2018 and has been focusing to alleviate suffering and save lives by on decision making and accountability providing the best available evidence in response to humanitarian crises on the effectiveness of humanitarian and epidemics. The RECAP project is action and enabling its use. Evidence supported by UK Research and Innovation Aid has developed a range of resources as part of the Global Challenges Research which are available free of charge from Fund, grant number ES/P010873/1. The its website. These include systematic Centre also offers courses for practitioners reviews of interventions for disasters, through modules on conflict and health, humanitarian crises and other major short courses and a free online MOOC on emergencies. Evidence Aid organises Health in Humanitarian Crises. events to promote the use of evidence in the humanitarian sector, including Nesta (Alliance for Useful Evidence) a yearly Humanitarian Evidence Week (www.alliance4usefulevidence.org) is a in November in collaboration with the global innovation foundation, based in the Centre for Evidence-Based Medicine at UK. It backs new ideas to tackle the big the University of Oxford, UK alongside challenges of our time, from the pressures Evidence Lounges. Evidence Lounges of an ageing population to stretched bring practitioners and members of the public services and a fast-changing jobs academic research community together market. It has produced several guides to enhance collaborations and develop relevant to the use of evidence to meet practical ways of using evidence in the this challenge and this publication draws humanitarian sector, including this guide. upon one of these: 'Using Research Evidence: A Practice Guide'. That guide Health in Humanitarian Crisis Centre was produced by Nesta's Innovation (http://crises.lshtm.ac.uk) at the London Skills Team and the Alliance for Useful School of Hygiene and Tropical Medicine Evidence and the present guide for the generates primary research and training humanitarian sector was developed in on public health in humanitarian crises, consultation with them and with grateful working closely with international acknowledgment to Nesta for allowing us humanitarian agencies and research to use some of their content. centres in affected countries to address critical health challenges.
RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR | 2 BACK TO CONTENTS CONTENTS
INTRODUCTION 4 Beyond experiments: new approaches to evaluation 32 Who might use this guide? 5 How to use this guide 5 Was it worth it? The value of cost-benefit and cost- effectiveness analyses 32 SECTION A: What is evidence- informed decision making, and Key messages from part 1 why focus on research? 6 of Section C 34 What is ‘evidence’ and why Part 2: How do you judge do we focus on research? 7 the quality of research? 35 The challenge of relying solely Peer-review: how does it on professional opinion 9 help decision makers? 35 Key messages from Section A 10 Defining ‘high-quality’ research 35 High-quality qualitative research 37 SECTION B: When can Avoid cherry picking by using evidence help you? 12 systematic reviews 37 Humanitarian Response Cycle 12 Meta-analysis 39 Developing and implementing Forest plots 39 a new intervention 14 Rapid evidence assessments 40 Adopting ideas from other The importance of repetition regions or sectors 18 and corroboration 40 Creating a theory of change 19 Review of reviews: how to Key messages from Section B 19 judge high–quality bodies of evidence 41 Key messages from part 2 SECTION C: What evidence of Section C 42 should you choose? 21 Part 1: Different types of research methods, designs and approaches 23 SECTION D: Where should 43 Experimental research – you look for evidence? why all the fuss? 28 Searching for research evidence 43 Experimental research designs 28 Sources of systematic reviews and other research for the Practical and ethical issues humanitarian sector 44 with randomised trials 30 Key messages from Section D 45 When randomisation is not possible or acceptable 30 Quasi–experimental designs 30 ENDNOTES References and endnotes 46
3 | RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR BACK TO INTRODUCTION CONTENTS
Introduction
Evidence coming from research and evaluation can help you understand what works, where, why and for whom. It can also tell you what does not work, and help you avoid repeating the failures of others by learning from evaluations of unsuccessful humanitarian programmes. Evidence can also guide the design of the most effective ways to deliver specific interventions.
Evidence challenges what we might investing money to solve a problem is think is common sense, perceived or supposed to do some good, we need received knowledge. to ask ourselves if that money could be For instance, it was long believed that spent more effectively elsewhere, for severe acute malnutrition required example on a different humanitarian specialised treatment in inpatient intervention. Research evidence can facilities with therapeutic products, help you make informed choices on even though this model posed many the most effective interventions to challenges to effective treatment for deliver in an emergency context. both health systems and patients. In the Evidence is a contested field, with mid-1990s, ready-to-use therapeutic differing opinions on what should be food was developed. In 2000, initial pilot most valued or deemed most relevant projects began to test the Community to decision makers.3 However, for the Management of Acute Malnutrition purpose of this practice guide, we (CMAM) approach during humanitarian emphasise research that is underpinned emergencies.1 It was found to be so by scientific notions of proof, validity, effective that it was endorsed by reliability, and has minimised bias. Such United Nations agencies in 2007,2 and research has the advantage of rigour, is now considered the standard of care relevance and independence. We focus on for managing acute malnutrition in population studies that aim to generate emergency and development contexts. average answers, which then need to As financial and other resources are be contextualised, for example taking limited and often insufficient in the into account cultural factors (such as humanitarian sector, we cannot afford the use of male doctors to examine and to waste such resources on policies treat female patients), and issues of and programmes that do not work. feasibility (such as the ability to deliver Interventions in any sector can have an intervention in the aftermath of a both positive and negative outcomes, disaster when access to populations or and both intended and unintended resources might be especially restricted). consequences. Even in cases when
RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR | 4 BACK TO INTRODUCTION CONTENTS
We have created this guide to help you HOW TO USE THIS GUIDE make best use of research evidence when you are in a humanitarian emergency The guide is divided into four main or when you are planning for the next sections: emergency. Our intention is to help you find and use evidence on interventions, actions and strategies that might SECTION A help you make informed choices and What is evidence-informed decisions. This guide is not about how decision making, and why focus to generate more research evidence. It on research? is about using and understanding what evidence exists and recognising when This section discusses what we mean by evidence-informed good evidence is lacking. It should help decision making, and why research you build your confidence in compiling, is an essential element of it. assimilating, distilling, and interpreting a strong evidence base of existing research, and think about how you might go on to evaluate your own projects and SECTION B commission research or evaluation. When can evidence help you? This section explores different scenarios in which using evidence WHO MIGHT USE THIS can help you, as well as the types GUIDE? of evidence you might need at different stages of developing or This practice guide is primarily implementing a new intervention aimed at humanitarian decision or policy. makers and practitioners working in the field or in the headquarters of donor, international, national, or non-governmental organisations. SECTION C It will help with decisions about What evidence should the financing, supervision, delivery you choose? or evaluation of humanitarian This section looks at different types interventions. It is not aimed at of evidence and examines how to trained evaluators and researchers, choose the most appropriate for but instead seeks to foster demand your case. It also discusses how to for research evidence from wider judge the quality of evidence. audiences in the humanitarian sector. SECTION D Where should you look for evidence? This section offers advice and resources to help you find the right evidence to support your case.
5 | RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR | WHAT IS EVIDENCE-INFORMED DECISION-MAKING, BACK TO SECTION A CONTENTS AND WHY FOCUS ON RESEARCH?
What is evidence-informed decision making, and why focus on research?
This section discusses what we mean by evidence- informed decision making, and why research is an essential element of it.
To begin, let us be clear about what of information and recognises the we do not mean. We are not talking importance of evidence on issues such about making decisions and choices as feasibility, preference and culture. by slavishly following rigid research In a field such as the humanitarian conclusions. Professional judgement sector where more and better evidence and other sources of information – such is required,6 we could assume that any as feedback from your stakeholders model of good decision making should – will always be important. This be wary of relying solely on professional practice guide is not about replacing judgement that is not supported by professional judgement but increasing scientific evidence. Later in this section, evidence use in humanitarian action. you will read about how we can all be A good start in defining what we mean ‘predictably irrational’ and – consciously or is borrowed from medicine. More than unconsciously – make errors in important two decades ago, David Sackett and judgements. We explore how to mitigate his colleagues proposed the following these errors of judgement in subsequent definition that has stood the test of time: sections. However, other decision making models have also stressed the importance “Evidence-based medicine is the of blending knowledge of evidence with conscientious, explicit and judicious use of judgement. The humanitarian sector is a current best evidence in making decisions sensitive area where we need to be aware about the care of individual patients. The of international and local politics and the practice of evidence-based medicine means dynamics between the various actors integrating individual clinical expertise involved in the delivery of humanitarian with the best available external clinical aid. This will, sometimes, determine evidence from systematic research”.4 access to evidence and information, but also how humanitarian aid is delivered. This attempt to define evidence- 5 However, the importance of evidence based medicine was not the first, but remains and, as noted in an ALNAP it has been influential and is just as report in 2014, “the failure to generate relevant to the humanitarian sector and use evidence in policy and response as it is to other sectors. It stresses makes humanitarian action less effective, how research can complement less ethical and less accountable”.7 professional judgement or other sources
RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR | 6 | WHAT IS EVIDENCE-INFORMED DECISION-MAKING, BACK TO SECTION A CONTENTS AND WHY FOCUS ON RESEARCH?
WHAT IS ‘EVIDENCE’ AND WHY DO Figure A.1 shows the different elements WE FOCUS ON RESEARCH? that should be part of evidence-informed decision making. Our focus in this practice The Oxford English Dictionary defines guide is on the top circle of the diagram: ‘evidence’ as “the available body of facts research and evaluation. or information indicating whether a belief or proposition is true or valid”,8 and, As the authors of the Alliance for Useful similarly, in their ALNAP report on the Evidence’s ‘What Counts as Good state of the evidence in the humanitarian Evidence?’ report state “The conduct sector, Paul Knox Clarke and James and publication of research involves the Darcy defined it as “information that explicit documentation of methods, peer helps to substantiate or prove/disprove review and external scrutiny, resulting the truth of a specific proposition”.7 in rigour and openness. These features We follow these definitions because contribute to its systematic nature many other definitions tend to be rather and help provide a means to judge the unhelpful by being overly inclusive trustworthiness of findings. They also (sometimes including almost all types offer the potential to assess the validity 9 of information) or by being too abstract of one claim compared to another”. and vague.
Figure A.1: The four elements of evidence-informed RESEARCH AND decision making EVALUATION
PRACTITIONER EXPERIENCE DECISION STAKEHOLDERS AND (e.g. employees, JUDGEMENTS preferences or values)
CONTEXT, ORGANISATION, ACTORS, CIRCUMSTANCES
Based on: Barends E, Rousseau DM, Briner RB. (2014) Evidence-based Management: The Basic Principles. Amsterdam: Center for Evidence-Based Management [www.cebma.org/wp-content/ uploads/Evidence-Based-Practice-The-Basic-Principles.pdf]
7 | RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR | WHAT IS EVIDENCE-INFORMED DECISION-MAKING, BACK TO SECTION A CONTENTS AND WHY FOCUS ON RESEARCH?
This practice guide focuses on research, time. So, decision makers require evidence but there are many overlaps with the that can be taken ‘off the shelf’ and field of evaluation and we discuss some combined with information on the local approaches to evaluating impact and context to inform their choice. Fortunately, process in Section C. We also give most it is possible to find such evidence and attention to research that deals with we cover some of the ways to do so in impact – whether something has had Section D. positive or negative results – because Research is a process engaged in for questions on impact are vital to those learning purposes. It seeks to answer involved in humanitarian action. These questions such as ‘What was the actors are concerned about showing their commonest type of injury after an ‘impact’ on populations, their ‘results’ in earthquake?’, ‘What are the effects on international terms or ‘what works’ for gender-based violence of different ways governments and local and international to protect women and children?’ or ‘How providers. The language may change, waterproof is a particular material when but the idea for their research stays the used for shelter?’ same: to see if they have really made a difference. Therefore, our aim with Evaluation is a process involving the this guide is to help you decide how assessment of findings and observations that research might help you choose against standards, for the purpose interventions, actions and strategies and of making decisions. Evaluations ask adopt policies that are most likely to questions such as ‘Which types of first make a positive difference. We provide aid should first responders be trained illustrative examples throughout the in?’, ‘Which is the best way to protect guide, and further examples of the use women and children from gender-based of evidence in the humanitarian sectors violence?’ or ‘What material should be are available in other collections of case used for making tents in a setting with 10 heavy rainfall?’ studies. We give prominence to research and Research does not necessarily require evaluation that is ready-made, with no evaluation. However, doing evaluation need to run a brand-new study. Decision always requires doing research. An makers have limited time and resources evaluation relates to an intervention and many simply cannot afford to that was actually implemented, while commission such a study and to wait for research is more comprehensive and, as its results to become available, which may well as including evaluations, it can also take years. Someone needing to make a seek to answer conceptual questions, decision now, needs the evidence now, if such as when planning for the needs that not yesterday, not in a year or more years’ are likely after a disaster or developing
We give prominence to research and evaluation that is ready-made, with no need to run a brand-new study.
RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR | 8 | WHAT IS EVIDENCE-INFORMED DECISION-MAKING, BACK TO SECTION A CONTENTS AND WHY FOCUS ON RESEARCH?
a novel intervention. However, whether picking and retro-fitting the evidence to we are thinking about research or the conclusions we have already fixed evaluation, we need to be aware that in our minds. If we are generous, this bias can distort the findings. In Section can be an unconscious mistake, but at C, we look at some of the biases that worst it can be a deliberate ploy to back can affect the results of a study, but it is up prejudice by finding and promoting also important to be aware of the biases the evidence that fits that prejudice, in how people interpret the results of a while ignoring the stuff that does not. study or which influence their thinking A similar but different phenomenon about the need for research or its impact. is optimism bias, which refers to the We discuss some of these biases next. “proven tendency for appraisers [of projects, programmes and policies] THE CHALLENGE OF RELYING to be too optimistic about key project SOLELY ON PROFESSIONAL OPINION parameters, including capital costs, One reason we privilege good quality operating costs, project duration and research over relying solely on benefits delivery”.12 This can result in professional judgement is that the biases setting targets for outputs, outcomes involved in professional judgement and impacts that are unrealistic and can mean that it is wrong. As humans, undeliverable. Christoplos noted that, we are ‘predictably irrational’11 and following the Indian Ocean Tsunami in may experience cognitive biases that 2004 “Methods and guidelines were filled distort our thinking. Cognitive bias with normative optimism. Terms such refers to our inability to be entirely as the ‘seamless web’ were applied to objective, which may manifest itself in describe the states of affairs that would multiples ways – such as perceptual emerge if the right methods were applied. distortion, inaccurate judgements and Individuals, households, communities illogical or irrational interpretations. and nations were expected to follow a Even with the best intentions, continuum from acute human suffering to professionals can get it wrong. Take ‘normal’ development. Over the years this for instance one type of cognitive optimism has faded. LRRD [Links between Relief, Rehabilitation and Development] error: confirmation bias. This is the tendency to concentrate on the evidence has come to refer more to a conundrum that fits with what we believe, and than to a gilded path. Intractable political to ignore or discount what does not. tensions and human insecurity have Even highly experienced professionals come to be associated with LRRD, first can fall into this cognitive trap. in post-conflict situations and then increasingly in natural disasters as well”.13 Being aware of how we can jump to conclusions is important for making us wary of experts. But confirmation bias also highlights how anybody – not just experts – can be highly selective in their use of research evidence. People tend to look for the evidence that fits their beliefs, intentionally or not. This might be thought of as ‘policy-based evidence’ rather than ‘evidence-based policy’: cherry
9 | RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR | WHAT IS EVIDENCE-INFORMED DECISION-MAKING, BACK TO SECTION A CONTENTS AND WHY FOCUS ON RESEARCH?
Optimism bias about both old and new This is not to say that professional interventions is often due to a lack of judgement is always wrong. Researchers evidence about their true effects. such as Gary Klein have sung the praises Research and evidence from evaluations of intuitive expert judgement, for instance of these interventions, or similar ones, in his work on ‘naturalistic decision can help to reduce this uncertainty. making’.14 Professional views and gut- instincts can be highly valuable, but we Just as in other sectors, there are also must be aware of their downsides. As many other biases relating to how people Daniel Kahneman asserted in a joint think that can afflict those working in the article with Professor Klein in American humanitarian sector. These include: Psychologist, “professional intuition is sometimes marvellous, and sometimes Hindsight bias: Tendency to see past flawed”.15 events as being more predictable than they were before the event occurred. Loss aversion: Tendency to prefer avoiding losses than to acquiring gains. Framing effect: Drawing different conclusions from the same information presented in different ways (e.g. would you prefer that ‘95% returned to work’ or that ‘5% did not return to work’?). The ‘availability heuristic’: When people relate the size, frequency or probability of a problem to how easy it is to remember or imagine. The ‘representativeness heuristic’: When people overestimate the probability of vivid events. The ‘need for coherence’: The urge to establish patterns and causal relationships when they may not exist. Meta-cognitive bias: The belief that we are immune from biases!
RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR | 10 | WHAT IS EVIDENCE-INFORMED DECISION-MAKING, BACK TO SECTION A CONTENTS AND WHY FOCUS ON RESEARCH?
CASE STUDY
How research has promoted the growth of Cash Transfer Programming
In recent years, Cash Transfer It also helped to tackle concerns Programming (CTP) has emerged as around cash transfers, such as one of the most significant innovations corruption and insecurity,18 and in international humanitarian increased awareness that cash assistance. generates different, but not necessarily greater, risks than in-kind assistance.19 In 2016, $2.8 billion of humanitarian The benefits of CTP have been shown assistance was disbursed through to cut across multiple sectors to cash and vouchers, a rise of 40% since 16 address women’s empowerment, food 2015 and a doubling from 2014. Cash security, education and health care. As disbursement has not only grown in a result, CTP is now an accepted tool in total terms, but also as a relative share almost every emergency response. of total international humanitarian One of the key successes of CTP assistance, from 7.8% in 2015 to 10.3% 17 research is that the generated in 2016. evidence has been widely used by This expansion has been supported champion institutions to promote the by a growing number of evaluations, use of cash in humanitarian crises. resulting in a body of evidence on More research is nevertheless the effects of different programmes needed to fine-tune the programmes on individual and household-level to maximise their benefits. outcomes. This body of research and evaluations has been instrumental in the growth of CTP.
Key messages from Section A
We do not advocate a form of Results from research are only one decision making that slavishly follows type of evidence, but have the the conclusions of research at the advantages of greater rigour, expense of professional judgement. relevance and independence when compared to other types of evidence. However, you should not put professional expertise on a pedestal. Creating new evidence can be Experts sometimes get it horribly costly and time-consuming, but there wrong, and they are not immune from is good quality evidence that can be the whole range of social and taken ‘off the shelf’ (see Section D). cognitive biases that affect both Research and evaluations of current experts and non-experts. emergencies can inform future interventions.
11 | RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR BACK TO SECTION B | WHEN CAN EVIDENCE HELP YOU? CONTENTS
When can evidence help you?
This section explores different scenarios in which using evidence can help you, as well as the types of evidence you might need at different stages of developing or implementing a new intervention or policy.
Evidence can make organisations HUMANITARIAN more effective. From more persuasive RESPONSE CYCLE campaigning to securing grant funding The United Nations Office for the and from developing an organisation’s Coordination of Humanitarian Affairs decision making abilities to making (UNOCHA) developed the humanitarian sure that programmes deliver results, response cycle to show the coordinated evidence can bolster your work. It does series of actions needed to help prepare not matter if you are a small voluntary for, manage and deliver humanitarian organisation or a large international action. It consists of five elements, with organisation. Whatever the scale, there each step building on the previous and may be existing research that can suit leading to the next.20 Evidence can help your needs. with decision making as you move around this cycle. Needs assessment and analysis: EVIDENCE CAN HELP YOU IN Looking at existing research will help MANY WAYS, INCLUDING: provide a basis for assessing needs, and Starting doing something. understanding the nature, magnitude and dynamics of specific problems. Stopping doing something. Along with the gathering of new data from the setting and circumstances Generating options or making the case. you are confronted with (or expect Identifying priorities. to be confronted with), it can help to identify opportunities for responding Assessing needs. appropriately. In some circumstances, this might be especially challenging if there Aligning services with needs. have been few opportunities for relevant 21 Designing and implementing more research. effective programmes. Strategic response planning: In planning Developing funding bids. your strategic response, you should search for evidence about what has been tried Creating effective advocacy and assessed previously that might help campaigns. you to make a well-informed decision. Increasing accountability to This might also include making use of stakeholders. evidence on how best to communicate the potential risks of a situation to those who might be involved.22
RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR | 12 BACK TO SECTION B | WHEN CAN EVIDENCE HELP YOU? CONTENTS
Figure B.1: Humanitarian Response Cycle
NEEDS ASSESSMENT AND ANALYSIS
OPERATIONAL STRATEGIC PEER REVIEW & PLANNING EVALUATION COORDINATION
INFORMATION MANAGEMENT
IMPLEMENTATION & RESOURCE MONITORING MOBILISATION
Resource mobilisation: The Operational peer review and evaluation: implementation of a new intervention Operational peer review can be used or policy will require the mobilisation as a course corrector and to identify of resources. These might be financial areas for immediate corrective action. to pay for activities or the personnel Along with an evaluation of your own needed to deliver them. Having adopted strategic response, and comparisons with an evidence-informed approach to evidence from similar interventions, it can planning the strategic response and help to determine whether adjustments choosing the interventions and policies or improvements are necessary, in to adopt, should make these more likely particular for leadership arrangements, to be effective and, as a consequence, implementation of other phases of the more attractive to donors.23 cycle, coordination and mechanisms for accountability to affected people. Implementation and monitoring: As you implement an intervention or policy you need to think about how you can monitor and report on your impact. This will help you to show if you are making a difference.
13 | RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR BACK TO SECTION B | WHEN CAN EVIDENCE HELP YOU? CONTENTS
DEVELOPING AND IMPLEMENTING The spiral below was developed by A NEW INTERVENTION Nesta to capture the different stages of the innovation process and can also When developing or implementing a new be used to plot the progress of a new humanitarian intervention or policy, it is approach to a social issue.24 Different a good idea to begin by thinking about types of evidence will be needed at the timing. Different evidence will be helpful different stages and the process should at different times in the development or be continuous, such that when you have implementation and you need to think reached the end of the spiral, you should about the appropriate research to suit start again and being a new cycle. We different stages of the lifecycle of the new discuss the various stages of this spiral intervention or policy. In the early days below, while some specific examples of of a new initiative, research can identify successful innovation in humanitarian emerging challenges and the scale of contexts are available elsewhere from the problem. It can be used to plot the a study of 15 projects funded by the development and implementation of a Humanitarian Innovation Fund.25 new intervention or policy.
Figure B.2: The Nesta Innovation Spiral
7 1
2
6 3
5 4
1 Exploring opportunities and 4 Making the case challenges 5 Delivering and implementing Generating ideas 2 6 Growing and scaling Developing and testing 3 7 Changing systems
RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR | 14 BACK TO SECTION B | WHEN CAN EVIDENCE HELP YOU? CONTENTS
1. EXPLORING OPPORTUNITIES AND CHALLENGES Working through current research will help you to understand the nature, magnitude and dynamics of the problems in front of you and the opportunities for responding appropriately. (See case study below.)
CASE STUDY
Bringing evidence on long-term health systems outcomes to protracted crisis settings With the principle of ‘leaving no-one on longer-term workforce distribution, behind’ embedded in the SDGs, there motivation and skills, or the value of is agreement across most development working with even weak local actors partners that as well as addressing during emergencies, to support longer- humanitarian health needs in crisis term capacity for coordination, planning settings, this should be done in a way and predictable resource fl ows. that supports longer-term equitable ReBUILD is therefore also engaging with health system development. those working on healthcare in current The ReBUILD Research Programme confl ict or protracted crisis settings. Consortium is an international research Successes include informing decision- partnership funded by UK Aid, working making and implementation in Sierra on improved access of the poor to Leone during the Ebola outbreak, and e ective health care in post-confl ict supporting aspects of a donor’s support settings through robust evidence plan for Syria. Bringing evidence on on health systems strengthening. long-term outcomes to those working ReBUILD’s research has been conducted in the humanitarian-development in relatively stable, if still fragile, post- interface, where humanitarian needs confl ict settings, using a historical lens are understandably the main focus, is to understand immediate, medium- challenging, with often limited space for and long-term impacts on the health long-term evidence to be considered system of policies implemented during within humanitarian organisations’ and after confl ict and crisis. Themes objectives and approaches, refl ecting include health worker incentives and the wider divide between humanitarian deployment, access to healthcare for the and development approaches. poorest, most vulnerable communities, ReBUILD is collaborating with a number aid e ectiveness and gender equity. of organisations considering evidence- The research has produced a rich based humanitarian approaches, understanding of the experience of including Evidence Aid, and is now communities and health workers working closely with working groups through confl ict and post-confl ict with specifi c focus on the humanitarian- periods, and recommendations for post- development interface for health, like confl ict health systems strengthening. the UHC2030 Working Group on But ReBUILD’s evidence is also relevant Fragile Settings and the Health Systems for humanitarian settings – e.g. the Global Thematic Working Group on e ects of approaches to engaging local Health Systems in Fragile and Confl ict health workers in emergency responses A ected States.
15 | RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR BACK TO SECTION B | WHEN CAN EVIDENCE HELP YOU? CONTENTS
2. GENERATING IDEAS In Section C, we discuss Standards of Evidence that show the journey new After you have identified your focus, it programmes should go on in terms of is time to search for interventions and providing evidence of impact. policies that have been studied in the past. Can you borrow successful ideas from 4. MAKING THE CASE others about what worked previously and avoid interventions that failed? Having evidence of the likely impact of your intervention will help you with 3. DEVELOPING AND TESTING making the case to those who might fund it and to those who will have to implement As new ideas are initiated, it will be it in the field. This will put you in a stronger time to start thinking about testing and position to persuade them to fund the experimenting with different approaches intervention and to support its delivery and about evaluating the impact of these. and implementation. (See case study below.)
CASE STUDY
Presenting evidence to country offices leaders
In June 2017, Christine Fernandes, She then proceeded to correct popular Save the Children’s global Humanitarian misperceptions with hard data. For Nutrition Adviser for Infant and Young example, by citing a Lancet article26 to Children Feeding in Emergencies show that appropriate breastfeeding (IYCF-E) delivered a presentation prevents the highest proportion of about IYCF-E to Save the Children Iraq these deaths in early childhood (13%) country director and senior leadership and that complementary feeding has team, who were reviewing their 2017- the second highest benefit. Asking 2018 strategy. Up to that point, nutrition people to make a guess, and then had not been one of the sectors of correcting their views is also an implementation in Iraq. Christine framed effective way for influencing people’s her 20-minute presentation as a quiz: opinion, according to evidence.27 each slide started with a question Christine managed to persuade the such as: whole Country Office leadership that supporting IYCF-E was the right thing Which do you think is the most to focus on, leading to a concrete effective means of preventing deaths change in the Iraq country office with in children under five? an IYCF-E programme start-up in Insecticide treated materials. Baghdad and support to the Federal Ministry of Health on the national Hib (meningitis) vaccine. nutrition strategy focusing on breast Appropriate breastfeeding. milk substitute monitoring and capacity building on a national level in the early Appropriate complementary feeding. 2018 investment of resources. Vitamin A and Zinc supplementation.
RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR | 16 BACK TO SECTION B | WHEN CAN EVIDENCE HELP YOU? CONTENTS
5. DELIVERY AND IMPLEMENTATION When you have implemented an intervention or policy you need to think about how you can document your impact. This will help take you from a position of rhetoric and saying ‘trust me, this project is working’ to one based on more trustworthy evidence from monitoring and evaluation. It will also help with your accountability by showing that you are making a difference, that your project is value for money, and that you have opened up your work to evaluation. (See case study below.)
CASE STUDY
Partnering for impact evaluations
The International Initiative for Impact insights from various stakeholders with Evaluation (3ie) seeks to improve experience working on nutrition and the lives of poor people in low- and disaster management. The findings middle-income countries by providing have also been disseminated to an and summarising evidence of which in-country working group comprising development interventions work, national and international NGOs, when, why; and for how much. In academia, and sector specialists. 2018, 3ie’s Humanitarian Assistance Contribution to improved data Thematic Window is funding seven quality: There have been significant impact evaluations in Chad, Democratic improvements in the Monitoring and Republic of the Congo (DRC), Mali, Evaluation (M&E) system. Best practices Niger, Pakistan, Sudan and Uganda. on data collection such as spot checks, The impact evaluation in Pakistan is debriefing sessions with field teams and being conducted by the University of regular feedback on discrepancies in Mannheim In Germany, in collaboration data collection, which were instituted with the Agency for Technical during the impact evaluation by Cooperation and Development the research team have now been (ACTED). ACTED supports replicated in another project being humanitarian assistance programmes in implemented by ACTED. disaster-prone areas and the evaluation assesses the effectiveness of these ACTED hopes to include these practices interventions in responding to these in their regular M&E protocol when they hazards and reducing community have been further tested in the field. vulnerability to emergency shocks. Building implementing agency The collaboration between ACTED capacity on evaluation: Staff at the and the University of Mannheim has implementing agency were sponsored included several instances of successful to attend a workshop on measuring capacity building and dissemination: the effect of poverty reduction Multi-layered engagement: The study programmes and help shape better and baseline have been presented to policies through the use of impact the Pakistan government with the aim evaluations. This will enable them to of informing Pakistan’s integrated better assimilate the findings from the nutrition strategy by drawing on evaluation, as well as provide a base for future in-house evaluations.
17 | RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR BACK TO SECTION B | WHEN CAN EVIDENCE HELP YOU? CONTENTS
6) GROWING, SCALING AND do the shopping. And for women in joint SPREADING households – meaning they live with their mother-in-law – as a sizeable minority do, The ultimate goal for many innovations is then the mother-in-law heads the women’s to replicate and grow so that they change domain. Indeed, project participation things on a bigger scale in everyday rates are significantly lower for women practice and can be successfully copied in living with their mother-in-law in more other settings. This will increase the scope conservative parts of the country.”28 for positive impact. He also noted that the right children have to be identified for the programme 7) CHANGING SYSTEMS to be effective. In the BINP areas, the Achieving system-wide changes is community nutrition practitioners who extremely complex. A good way to reach implemented the programme “could not this scale is to have multiple independent correctly identify from the [growth] charts studies showing that something does not which children should be admitted to the just work in the specific circumstances of program”, which led to the mis-targeting the early testing but is effective in many of children.28 These two examples of places. inappropriate programme implementation illustrate the importance of understanding local contextualized factors in the success ADOPTING IDEAS FROM OTHER or failure of a programme, and the need to REGIONS OR SECTORS use qualitative and ethnographic evidence (see Section C), in order to achieve Even if you find evidence of success of desired outcomes in different contexts. a policy or project elsewhere, would When designing new programmes aiming adoption of those ideas work in your at changing how people act – for example, area? Howard White gives an example to adhere to International Humanitarian of the failure of a supplementary Law or to use latrines instead of feeding programme in Bangladesh (the defecating in open areas – research from Bangladesh Integrated Nutrition Project social psychology, anthropology and (BINP)) to achieve the outcomes of behavioural change can help. It might “the acclaimed Tamil Nadu Integrated provide ideas based on robust research Nutrition Project (TINP) in India”.28 The about which of the range of techniques Bangladesh programme had a 30% lower to ‘nudge’ people towards the desired participation rate than that in India, and a actions would be worth adopting. counterfactual impact evaluation “found no significant impact of the program on Hugh Waddington and colleagues,29 for nutritional status, although there was a instance, noted that improving sanitation positive impact on the most malnourished by getting people to use latrines rather children”. Professor White noted that: than defecating openly in public spaces requires more than knowledge of germ “the people targeted have to be the theory. Evaluation evidence of Community right ones. The program targeted the Led Total Sanitation (CLTS) suggests that mothers of young children. But mothers improved technical knowledge of latrines, are frequently not the decision makers, the use of subsidies to reduce the costs and rarely the sole decision makers, with to households and changing social norms respect to the health and nutrition of their and collective action at the village level are children. For a start, women do not go to key factors in bringing about behavioural market in rural Bangladesh; it is men who change and increasing latrine adoption.
RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR | 18 BACK TO SECTION B | WHEN CAN EVIDENCE HELP YOU? CONTENTS
CREATING A THEORY OF CHANGE handwashing facilities be located? What should these facilities comprise of? What In the early stages of developing or information needs to be provided? What planning any intervention, it is important incentives need to be introduced to to logically describe what you do and why influence household decisions? it matters. You need to be coherent, clear and convincing. This description is often A Theory of Change (example on page referred to as a Theory of Change, and aims 20) helps you be explicit about your goals to give a “clear, concise and convincing and how you will achieve those goals. It explanation of what you do, what impact helps to avoid just simply hoping that your you aim to have, and how you believe you innovative approach will work and that will have it. It is a vital foundation of any your assumptions are correct. Another programme, and a prerequisite for effective benefit of doing a Theory of Change is that evaluation”.30 it is a first step in designing an effective evaluation, because it tries to identify A Theory of Change is a useful way to be all the outcomes that will need to be more explicit about the evidence that you measured. are using from others and to be clearer about how you are going to get results. For The most important message here is to instance, if you want to run a programme think about appropriateness. You want to to increase uptake of handwashing in a find research that fits your needs and the refugee camp, you may want evidence stage of development that you are at. The that answers a series of questions. evidence you will require in planning a new What techniques are likely to lead to programme or during the first few months effective behavioural change? Will the of its implementation will be very different new handwashing facilities be acceptable to what you will need when the programme to potential users? Where should the is more established. We revisit this crucial issue of appropriateness in Section C.
Key messages from Section B
There is a wide range of situations in Early stages of innovation are the which evidence can help you. This is not time to create a Theory of Change. This limited to just the obvious ones around is a useful way to be clearer about what capturing the results of interventions evidence you are using and how you and policies. There may be other are going to get results. It will also help benefits that you have not thought you to design an effective evaluation. about, such as creating more persuasive campaigning, obtaining Interventions or policies that are funding or stopping doing something more established can set up their own that is not working. experiments or evaluations of impact. As an innovation grows, you should also You need to think about the timing consider multiple replications to check and appropriateness of evidence. that the benefits you saw were not just Where are you on the innovation spiral an isolated success story, but that it can and what evidence may best suit your work in other places and contexts. needs? For instance, in the early days of developing an innovative programme, it It does not matter what size of may be more valuable to learn from organisation you are in. There is a lot of what others have already found, research that might be able to help you. through examining past evaluations or wider social and scientific research, rather than doing a large, costly evaluation of your own work.
19 | RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR BACK TO SECTION B | WHEN CAN EVIDENCE HELP YOU? CONTENTS assessment Rapid needs ACTIVITIES assessment In-depth needs Safety Information & security Logistics Contribution to Contribution LEVEL CHANGES IMPACT health & wellbeing psychological • Improved in education & retention to access • Improved morbidity & mortality • Reduced • Security of tenure basic services to access • Improved social structures • Strengthened sustained • Livelihoods disasters future to resilient more • Community Financial Preconditions to participate • Beneficiary willing in place leadership structure • Local support intervention • Stakeholders Community engagement Human OUTCOME communities) LONGTERM & reporting Coordination Material or capacity for self- or capacity for mproved resilience &/ resilience mproved
I recovery (households/ recovery Monitoring & reflection INPUTS OUTPUT shelter provision shelter Effective & timely Effective
Needs assessment relevant & needs relevant Needs assessment of vulnerable groups accounted for accounted groups of vulnerable coordination Functional logistics, & communication Protection from weather & environmental extremes & environmental weather from Protection & security of possessions personal safety Increased borne diseases & vector water from protection Increased household displacement Reduced intact are Households and communities materials utilise the provided & skills to Knowledge & anxiety stress Reduced dignity & retained privacy Improved ShelterBox response is accountable response ShelterBox • Aid selection appropriate to context to • Aid selection appropriate • 'Do no harm' principle followed of non-beneficiary groups • Sensitization purpose intended used for items • Shelter • Assumptions • SHORT/MEDIUM TERM OUTCOMES SHORT/MEDIUM
THEORY OF CHANGE THEORY
well being well
Protection
Example of Theory of Change on resilience of communities of Change on resilience of Theory of Example Psychological Physical Physical 'Outcomes and impacts - measuring the difference we make': and impacts - measuring the difference 'Outcomes https://cwarham.wordpress.com/2017/02/02/shelterbox-theory-of-change
RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR | 20 BACK TO SECTION C | WHAT EVIDENCE SHOULD YOU CHOOSE? CONTENTS
What evidence should you choose?
This section looks at different types of evidence and examines how to choose the most appropriate for the issue you are interested in. It also discusses how to judge the quality of evidence.
Not all evidence is equal. For instance, experimental research is A key message of this practice guide is to more suited to evaluating impact and think about usefulness. Too many people ‘what works’ (i.e. effectiveness), but other types of research can help give other get into trouble by not thinking clearly 32 about what sort of research design, insights. They may not look so much at impact, but can reveal why and how methodology or methods of collecting 33 data are really going to meet their needs. things are working. They tend to be ‘methods led’.31 In other Also, it might be important to establish words, we all have a tendency to pick our which sections of the population benefit pet approach, such as questionnaires, most from an intervention, and this randomised trials or interviews, because can often be masked by relying on the we are most comfortable with it, rather average effect for the population as a than being more open to the breadth of whole. methods and thinking which ones are best suited to answer our challenge and resolve our uncertainties.
21 | RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR BACK TO SECTION C | WHAT EVIDENCE SHOULD YOU CHOOSE? CONTENTS
CASE STUDY
Lessons learnt from using ethnography in the Ebola response
The Ebola epidemic in West Africa shed responders as a traditional resistance35 light on the need to foster ‘community to modern medicine, and as a threat to engagement’ when responding to biosecurity. Involving anthropologists outbreaks, and on the role played by who had extensive experience living in networks of social scientists to share the countries affected by the epidemic material and inform the response. was key to identify acceptable and safe This was exemplified by the Ebola practices (such as showing a picture Response Anthropology Platform, of the corpse to families to address whose members “work proactively with rumours around bodies disappearing36), health and humanitarian organisations as well as to overcome tensions to design, deliver and monitor more arising from burial practices through locally responsive and socially informed collaborative approaches. A key lesson interventions and research on the that emerged was to identify rationales ground”.34 Lessons can be learnt from explaining communities’ needs to bury how anthropologists used ethnography their dead as opposed to labelling to bring socio-cultural and political these practices as essentially ‘cultural’.37 dimensions in a biomedical response The Ebola response showed “the need in order to address the issue of so- for new global mechanisms to be called ‘unsafe’ burial practices. In a established that can rapidly mobilise context where corpses transmit the all experts who can bring relevant local Ebola virus, exceptional measures contextual, medical, epidemiological, were adopted where only certain and political information on global aid workers were authorised to bury health emergencies”.38 the corpses of the persons who had died from Ebola. Gatherings of local communities attending funerals of loved ones who had died from Ebola were perceived by the Ebola outbreak
The type of research you chose as the source of evidence to help in your decision making needs to fit the needs of the challenge that you face.39
Experimental research is more suited to evaluating impact and ‘what works’.
RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR | 22 BACK TO SECTION C | WHAT EVIDENCE SHOULD YOU CHOOSE? CONTENTS
PART 1: DIFFERENT TYPES OF RESEARCH METHODS, DESIGNS AND APPROACHES
So how do you go about Some types of research (such as understanding which research method experimental and quasi-experimental is appropriate for your case? designs) are better suited for identifying the presence of a causal relationship. If you are not a researcher, it can be daunting to come face-to-face with Other types of research are more the jargon and endless lists of different appropriate for explaining such causal relationships – see Beyond experiments: approaches.40 These issues are captured in new approaches to evaluation on page 32. a How to note from the UK’s Department for International Development about Some designs (such as observational assessing evidence41 and are just as studies) are more useful for understanding relevant to the humanitarian sector as political, social and environmental they are to international development: contexts. Some of the pros and cons of various research designs are summarised in Table C.1.42
Table C.1 Different designs, methods and approaches to research evidence – a brief overview
TYPES OF RESEARCH AND WHAT IS IT? PROS CONS EVALUATION
Ethnography An account by Powerful and Difficult to verify and someone with immediate; may give may lead to inflation direct experience vivid detailed insights of prevalence. of, or affected by, into events concealed Emotive first-person a particular issue. from much of the stories may inhibit The objective is to population. critical appraisal collect in-depth and individual information on anecdotes may not people’s experience be representative. and perceptions.
Case study Detailed and Easy and practical; Bad at inferring intensive can be used by that an intervention examination of practitioners and non- causes an outcome; a single case or specialists; good for small size means multiple cases such interventions that have hard to generalise as a community, already happened; to national or family, organisation, might identify adverse population level. sites, event or consequences from individual. intervention; helps to describe innovations; generates hypotheses for other studies.
23 | RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR BACK TO SECTION C | WHAT EVIDENCE SHOULD YOU CHOOSE? CONTENTS
TYPES OF RESEARCH AND WHAT IS IT? PROS CONS EVALUATION
Case control Compares a group Require fewer Rare in social studies who have, say, a health resources to policy (see closely condition with a group carry out than related ‘case- of people who do not intervention studies; based’ evaluation have it, and looks back useful when below for approach in time to see how randomised trials more common characteristics of the or prospective in social policy), two groups differ. cohort studies more frequent in are not practical epidemiology and (e.g. studies health; provide of cot death); less evidence for may generate causal inference hypotheses that can than a randomised be tested in more trial; high risk of robust studies. various biases (e.g. recall bias, selection bias and interviewer bias).
Process An approach to Strong on Lack of agreed evaluation evaluation that looks at explanation of methods; what happens compared causes; can be used opportunities with existing theories in messier areas of for bias; weak of change or causal social and complex on estimating pathways identified interventions where quantities or extent during an evaluation. Can there may be many of impact; relatively be associated with realist causes and context little evaluation evaluation; qualitative is important. of the techniques comparative analysis Process evaluation used in contrast to (QCA); contribution will investigate the large body of analysis; process tracing. mechanisms of literature criticising change, contextual randomised trials. factors and implementation evaluation.
Economic Economic evaluations Economic Economic evaluation evaluation determine whether evaluation provides involves several an intervention is an a good sense of the assumptions and efficient use of resources level of resources sometimes proxy and how it compares to needed and measures of other interventions both costs per benefit/ cost, which relies in terms of their costs effectiveness on judgements. and consequences. In produced, which The measure of the humanitarian sector, is a great indicator effectiveness needs the most common for most policy to be provided methods are cost/ makers and can be by a robust benefit analysis and cost- used for advocacy. comparative trial. effectiveness analysis.
RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR | 24 BACK TO SECTION C | WHAT EVIDENCE SHOULD YOU CHOOSE? CONTENTS
TYPES OF RESEARCH AND WHAT IS IT? PROS CONS EVALUATION
Cross-sectional A representative Quantitative data Establishes sample of people can be examined association at surveyed at to detect patterns best, not causality; one point in or association; rarely possible time. Although relatively cheap to attribute any surveys such as and ethical; survey measured change questionnaires can be repeated at to the intervention, and structured intervals, illustrating or to understand interviews are changing trends what would have commonly used over time (see happened in in cross-sectional Cohort/Longitudinal the absence of design, they are studies below); the intervention not the only way. may generate because any Other methods hypotheses that can change could include content be tested in more have been due analysis or analysis robust studies. to broader issues of official statistics. such as economic conditions, weather, or media campaigns, rather than the intervention. Other disadvantages are risk of recall bias, social desirability bias, researcher bias; unequal group sizes and unequal distribution of confounders.
Cohort/ The same sample Best source Data often emerges Longitudinal of people surveyed of evidence too late for effective studies over several on association policy–making; points over between childhood study members may time, sometimes experience and drop out over time; from childhood adult outcomes; expensive approach to old age. can give powerful when maintained support for over decades. certain early interventions; can be used to evaluate programmes that are implemented following randomised trials.
25 | RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR BACK TO SECTION C | WHAT EVIDENCE SHOULD YOU CHOOSE? CONTENTS
TYPES OF RESEARCH AND WHAT IS IT? PROS CONS EVALUATION
Quasi- Compares a group Can provide Inability to ensure experimental who have, say, a reasonably strong equivalence of design health condition evidence of the groups and to with a group of relationship prevent change people who do not between the over time can result have it, and looks intervention and in less reliable back in time to see the measured findings; matching how characteristics outcomes; powerful techniques tend to of the two method of exploring require a lot of data groups differ. the impact of an in both intervention intervention when and comparison randomisation is groups which can impossible; can be be time-consuming applied to large and expensive to communities as well collect; a good as groups; no need understanding is for randomisation required of the from the start factors that need (ex-ante), which to be matched avoids some of (without this, it the challenges remains possible and difficulties of that there are randomisation. systematic differences between the groups that are not being controlled for); these designs require complex analytical work and specialist knowledge.
RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR | 26 BACK TO SECTION C | WHAT EVIDENCE SHOULD YOU CHOOSE? CONTENTS
TYPES OF RESEARCH AND WHAT IS IT? PROS CONS EVALUATION
Randomised One group Offers the most Poor on taking context trial receives an robust, reliable into account (e.g. cultural, intervention findings, which institutional, historical while another give confidence and economic settings); does not receive that any measured difficult to do at a national that intervention difference between population level; when or receives groups are due to used at small pilot level, a different the intervention; this might not be relevant intervention; random allocation to national/ population groups are should prevent level (although this is a formed using a systematic risk for all designs); can random process, differences be hard to manipulate usually with the between groups; variables to experiment same chance of greater confidence in social policy (e.g. class, being allocated in the effect size race or where you live); to either group. and the relationship mistakes in randomisation between the can invalidate results; intervention can be hard to persuade and outcome; decision makers of internationally benefits of this design; recognised potential political and approach. ethical issues over randomisation (e.g. some groups randomly getting potential beneficial intervention, and not others); can take more management time and more time to set up than quasi-experiments.
Systematic Aggregation Best source of Requires a sufficient reviews, meta- of results from reassurance that an number of robust analysis eligible studies, intervention works studies in a given area; with the eligibility (or does not); methodology less well criteria defined meta-analysis pools developed for synthesising in advance and statistical results; qualitative data and ‘grey’ methodologies large reviews literature; might produce reviewed carry considerable misleading results if statistical power; is selective reporting biases replicable by other affect the research that is researchers; can be available for the review. applied to any kind of data or study.
27 | RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR BACK TO SECTION C | WHAT EVIDENCE SHOULD YOU CHOOSE? CONTENTS
EXPERIMENTAL RESEARCH – Instead, to mitigate this bias, we can WHY ALL THE FUSS? use a process of randomisation: after the population for the study has been Experimental research such as identified and they have agreed to join the randomised trials have received a lot of study, the individuals to be compared with attention – some might say too much. So each other are allocated randomly to the why have they attracted so much interest? study groups that will be compared, for Frequently, practitioners and policy example by drawing lots.44 makers want to know whether an intervention has had an impact. Has there EXPERIMENTAL RESEARCH been a causal link between their new DESIGNS programme and the ‘impact’, ‘effect’ and ‘result’ at the end? They want to be able to Choosing the appropriate experimental attribute the effects that they have seen design, and conducting the study properly to the policy or intervention that is being establishes the causal link between an tested. Experimental designs, even though intervention and an outcome. When they are sometimes difficult to implement random allocation is used to assign the in practice, have a better chance of participants to the intervention or the showing this cause and effect link. comparator group, this provides the most robust means for determining whether An experimental study could simply any difference in outcomes between test and compare two groups of people the groups is due to the intervention before and after the intervention is or chance, and chance is minimised by given to one of the groups. But if these doing a large enough study. The random groups are not formed by randomisation, allocation minimises the possibility of there is a risk that the groups might systematic differences between the differ because of selection bias and that groups, regardless of whether these are differences in their outcomes would due to known or unknown factors. then be due to these differences in their characteristics and not to the effects The introduction of a control group of the intervention being tested. For against which to compare the intervention example, somebody who asks for the minimises several biases that normally intervention may be more predisposed complicate the evaluation process. For to positive outcomes. Perhaps they are example, if you introduce a cash transfer more motivated, healthier, confident, and scheme to increase immunisation rates thus more likely to do well regardless and the use of facility-based deliveries, of any intervention. On the other hand, how would you know whether those someone wanting the intervention may receiving the extra cash would not have be the most vulnerable and most likely to used the services anyway? You need to do badly regardless of any intervention. compare them to a group of people who These biases will distort the results of do not receive the extra cash. the experiment and may mean that it As with any research design, the strength provides a false estimate for the effect of a randomised trial also requires that the of intervention. Furthermore, the biases study is conducted properly. This includes might be unpredictable and if we do not ensuring that the randomisation of know the direction of the bias, we cannot individuals or units (e.g. schools, hospitals, simply adjust the results of the experiment neighbourhoods etc.) is done properly to try to take account of them.43 (‘allocation fidelity’), and that only the experimental group has been exposed to
RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR | 28 BACK TO SECTION C | WHAT EVIDENCE SHOULD YOU CHOOSE? CONTENTS
the intervention being tested (‘treatment Further information on how to establish fidelity’). A well-conducted randomised whether a randomised trial has been trial also requires sufficient number of conducted and reported properly can be participants and statistical power to found in guidance on the conduct and ensure that other factors (‘confounders’) reporting of trials.45 that could have caused the outcome The International Initiative for Impact are balanced between the experimental Evaluation (3ie) identified evaluations of and control groups. A low drop-out rate humanitarian interventions in conflict and (‘attrition’) is also necessary to minimise reconciliation situations that have used the risk of bias. It is important, therefore, randomised trials and quasi-experimental to check the reporting of a randomised designs (see below).46 trial to ensure that these criteria of validity have been met.
CASE STUDY
Establishing the effectiveness of humanitarian interventions using experimental methods
Communicable diseases are of particular disinfectant water treatment and concern in conflict and disaster- improved water storage (intervention affected populations that reside in group) to households with only camp settings. In the acute emergency improved storage (control group). phase, diarrhoeal diseases can account The allocation of households to the for more than 40% of deaths among experimental and control groups was by camp residents. Diarrhoeal diseases randomisation. result from a variety of factors but, In intervention households, point-of-use in the emergency context, adequate water treatment with the flocculant– sanitation and water supply (in terms disinfectant plus improved storage of both quantity and quality) is the reduced diarrhoea incidence by 90% primary means for diarrhoea reduction. and prevalence by 83%, when compared Clear limitations exist in current water with control households with improved treatment technologies, and few water storage alone. Among the products are capable of treating turbid intervention, residual chlorine levels met water. or exceeded Sphere standards in 85% Shannon Doocy and Gilbert Burnham (95% CI: 83.1–86.8) of observations with described the findings of a 12-week a 95% compliance rate. effectiveness study of point-of-use Drs Doocy and Burnham concluded that water treatment with a flocculant– a point-of-use flocculant–disinfectant disinfectant among 400 households significantly reduced the incidence and in camps for displaced populations in 47 prevalence of diarrhoeal disease among Monrovia, Liberia. households residing in camp settings The study compared diarrhoea rates in the emergency context. among households with flocculant
29 | RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR BACK TO SECTION C | WHAT EVIDENCE SHOULD YOU CHOOSE? CONTENTS
PRACTICAL AND ETHICAL ISSUES WHEN RANDOMISATION IS NOT WITH RANDOMISED TRIALS POSSIBLE OR ACCEPTABLE Having a control group leads to a common In some circumstances, it might not be criticism of randomised trials, which is possible or acceptable to randomly assign that they are unethical because only some people to the groups to be compared. For of the population are given access to an example, it might be impossible to use a experimental intervention that might be randomised trial to compare the relative considered superior. This ‘rationing’ of the effects of different ways to co-ordinate intervention is justifiable if we do not know the response to a major emergency or the whether it will do more good than harm and impact of a national policy intended to we are doing the trial to find out. However, improve the social inclusion of refugees.50 if there is already convincing evidence In such cases, researchers might use a that the intervention will be beneficial, we quasi-experimental design to investigate might still need to do a randomised trial to the link between the interventions and find out how beneficial it is and for whom. the outcomes. Kayvan Bozorgmehr and If so, there are ways round the criticism Oliver Razum did this to compare personal and the ethical challenge. For example, spending on health care among asylum- in a waiting list control randomised trial, seekers and refugees in Germany who those allocated to the control group will be had either restricted or regular access to offered the intervention either at the end health care.51 Similarly, Rodolfo Rossi and of the trial or when their personal follow- colleagues used surveys before and after up has been completed. Another option is a vaccination campaign to investigate the ‘stepped wedge’ or ‘pipeline’ design,48 its effect on vaccination coverage in in which the order in which participants young children in a crisis-affected area of receive the intervention is determined Lebanon in 2015.52 at random, allowing it to be gradually phased in over time so that all participants QUASI–EXPERIMENTAL DESIGNS have been given access to it by the end There is a wide variety of quasi– of the trial. This design was used for the experimental designs for comparing the Progresa/Oportunidades study, one of the effects of interventions and these are largest randomised trials in international often used when randomised trials are development, which tested conditional 49 not feasible. Such studies are similar to cash transfers for the rural poor in Mexico. randomised trials to the extent that they Stepped wedge trials can be highly practical can be used to compare the outcomes in a world of austerity, where there are for individuals or groups who receive an not enough resources to do a full roll-out intervention or programme and those of a new programme all in one go. They who do not receive it. However, the allow a gradual roll-out to be done while, receipt or non-receipt of the intervention at the same time, using a randomised trial or programme would not be based on to evaluate its effectiveness. However, random allocation. Instead, other types of stepped wedge trials are complex and comparative design, such as interrupted can be difficult to implement, and if the time series or controlled before-and-after intervention turns out to be ineffective studies, or analysis, such as regression or harmful, it will have been rolled out discontinuity or propensity score to everyone in the trial. In the case of a matching, are used.53 It should be noted behavioural or educational intervention or that before-and-after designs without a the restructuring of service delivery, it might comparison group cannot establish the then be difficult or impossible to remove or impact of an intervention or programme ‘switch off’ the intervention after the trial. because in the absence of a counterfactual
RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR | 30 BACK TO SECTION C | WHAT EVIDENCE SHOULD YOU CHOOSE? CONTENTS
(i.e. a comparison with what would have greater risk of bias than well-conducted happened without the intervention), any randomised trials, but they might still observed impact on outcomes may be allow strong causal inferences to be made due to factors other than the intervention. in circumstances where a randomised trial Quasi-experimental designs have a would not be possible or acceptable.54
CASE STUDY
Establishing the effectiveness of humanitarian interventions using a quasi-experimental design. The Contribution of Food Assistance to Durable Solutions in Protracted Refugee Situations This evaluation was part of a series of unregistered Rohingya who did not. The impact evaluations jointly commissioned primary evaluation question was: What by the World Food Programme (WFP) are the differential impacts of long- and the Office of the United Nations term food assistance on the different High Commissioner for Refugees Rohingya refugee and refugee- (UNHCR) in 2011 and 2012 to assess affected populations in Bangladesh? the role of food assistance in, and The evaluation concluded that along its contribution to, self-reliance and with other forms of external assistance, durable solutions for the refugee and 55 food assistance was a contributing the refugee-affected populations. factor in households’ choice of The evaluation covered the protracted economic activity and adoption of refugee situation in southeast specific coping strategies. Compared Bangladesh, where approximately with their unregistered Rohingya 30,000 Rohingya refugees have been counterparts, registered refugees assisted in two official camps for more engaged in significantly different than two decades. In addition to the economic activities, including higher- registered refugees, approximately skilled and less risky employment for 45,000 unregistered Rohingya reside overall higher wage rates. They also in makeshift sites and more than had significantly better wealth status 150,000 reside in host communities based on asset accumulation. Food in Cox’s Bazar district. To evaluate assistance was an integral component impact in the most methodologically of their livelihoods, used mainly for rigorous manner for the context, consumption and as collateral and a a quasi-experimental design was value transfer for loans and mortgages. used. The evaluation design focused The value transfer of all external primarily on three population groups: assistance in the camps enabled refugees to work less and to rely on this 1. Refugees living in two external assistance in times of crisis. official refugee camps; Despite these differences, all refugee 2. Unregistered Rohingya living in and unregistered Rohingya groups two unofficial sites, the official relied on economic activity to camps or host communities; support their livelihoods. Unregistered Rohingya employed a wider range 3. Host communities. of coping mechanisms, both positive and negative, and were a significant The key quantitative comparison part of the region’s labour market. was between registered refugees who received food assistance and
31 | RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR BACK TO SECTION C | WHAT EVIDENCE SHOULD YOU CHOOSE? CONTENTS
BEYOND EXPERIMENTS: NEW unlocked or triggered”.61 For instance, in APPROACHES TO EVALUATION contribution analysis, the research does not attempt to prove that one factor – When thinking about impact, we such as a specific policy – ‘caused’ the also need to think beyond a sole desired outcome, but rather it sets out dependence on experiments such as to explore the contribution a policy is randomised trials. Although trials and making to observed results. It recognises quasi-experimental designs may work that effects are produced by several well when there is a simple intervention causes at the same time, not just one. that can be tested, interventions are often not simple. NGOs working in civil These approaches are not all new.62 society, for instance, rarely work alone Indeed, having a good theory has or have the chance to manipulate a clear arguably always been at the heart of experimental ‘treatment’ for a randomised good science. Therefore, using a ‘theory- trial. Evaluators are looking at other based’ approach is not novel, but ways of doing things, approaches that these ways of evaluating impact have involve moving beyond a simple one- grown in popularity and they do help to-one causality and require careful evaluators address multiple causality. thinking about your ‘contribution’. In the complex world of humanitarian WAS IT WORTH IT? THE VALUE action, it is unlikely that your programme OF COST-BENEFIT AND COST- alone is the necessary or sufficient EFFECTIVENESS ANALYSES condition for success. It might be just one Having determined and measured the factor among many - part of a ‘causal effects of an intervention, it is important package’. Programme success depends for the humanitarian sector, as with on what else is going on or has gone on any other sector, to know whether its around you. In the past, a simple causal programmes and interventions provide explanation of ‘what works’ may have value for money. When budgets are sufficed but nowadays it is more common constrained, we need to make difficult for evaluation researchers to ask ‘did our financial decisions about whether one intervention make a difference? What thing is more valuable than another. Policy would have happened without it?’ makers, commissioners of services, grant- There is a range of approaches that making bodies and charitable funders are becoming more popular with alike are asking for more and better evaluators, such as theory-based56 impact measures for every pound, dollar and case-based designs, complexity or euro they spend. Economic appraisal theory, realist evaluation,57 contribution techniques help create such insights on analysis,58 process tracing59 or qualitative value. comparative analysis.60 A common feature There is a range of techniques for doing of these approaches is a recognition of economic appraisal. A cost effectiveness the complexity of causality requiring the analysis (CEA) identifies which are the identification of the causal mechanisms better ways to achieve an outcome. across a range of specific cases or events, Whilst, cost benefit analysis (CBA) tells as well as elements in the causal chain us if there are better uses for the available that vary. This involves “identifying under resources. Cost-utility analysis (CUA) what conditions, and in what combinations calculates which intervention produces with other causes, the observed effect of the greatest sense of subjective well- an intervention is likely to be produced, being for the people affected by it.12
RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR | 32 BACK TO SECTION C | WHAT EVIDENCE SHOULD YOU CHOOSE? CONTENTS
Wherever feasible, economic appraisal been defined as “the maximum amount tries to attribute monetary values to all of money an individual is willing to give the inputs and activities that go into up in order to receive a good”.12 Similarly, producing the outputs which, in turn, lead willingness to accept (WTA) “is the to the desired outcomes and their longer minimum amount of money they would term economic, social and environmental need to be compensated to forego or impacts. This constitutes the cost analysis. give up a good”.12 The results of a WTP The same procedures are then used or WTA survey provide what economists to attribute monetary values to all the call ‘stated preferences’, that is, what outcomes and impacts that the policy people say they would be prepared to is hoping to achieve. Where there are pay or accept for some good or service. negative outcomes and impacts these Alternatively, economists will attempt to are deducted from the monetary value observe how people will actually behave of any positive outcomes/impacts that in a real market situation. That is, they have been achieved. This constitutes the will set up a situation in which people benefits analysis. are asked to pay different amounts of Monetary values are derived as much as money for a good or service or will be possible from where there is some market offered different amounts of money to activity. The labour market, for instance, give up a good or service, and then record provides monetary values for different the actual behaviour of people. This types and units of labour. The building provides what economists call ‘revealed trade market provides the monetary value preferences’. (the cost) for goods that are required to In addition to the procedures mentioned build a hospital or school – bricks, steel, above, economic appraisal also identifies glass, cement etc. The costs of providing where, or on whom, the costs and medical care can be derived from the benefits will fall and over what time fixed and variable costs of running span. The economic value of benefits hospitals, health centres and community that will be realised in the medium to clinics, medicines and medical supplies, long-term future will be discounted by and the like. This is what is meant by what is known at the discount rate. Other ‘monetising’ costs and benefits. adjustments that are made to costs Some costs and benefits (especially) are and benefits include changes in relative not traded in a market and are therefore prices, material differences in tax options, more difficult to monetise. In such cases optimism bias and contingencies such as economists will estimate the benefits floods, droughts and the effects of climate (and sometimes the costs) by simulating change. Economic appraisal is a fairly market activity with ‘willingness to pay’ technical enterprise, usually requiring the and ‘willingness to accept’ surveys of expertise of an economist. people. Willingness to pay (WTP) has Quasi-experimental designs might still allow strong causal inferences to be made in circumstances where a randomised trial would not be possible or acceptable.
33 | RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR BACK TO SECTION C | WHAT EVIDENCE SHOULD YOU CHOOSE? CONTENTS
Key messages for part 1 of Section C
Not all evidence is equal. Some is Theory-based evaluation and better quality and will be more techniques such as contribution appropriate to your challenge. analysis are increasingly popular with evaluators. These techniques can be The type of research needs to fit the helpful when it is difficult to do an needs of the challenge and whichever experiment or impossible to attribute method is used it needs to have been your single policy or programme to any carried out and analysed appropriately. single clear result.
Do not be discouraged by the long It is not just about whether your list of research methods, designs and intervention worked or not, but whether approaches. The key thing is to it was value for money. understand the assumptions that underpin these methods. Techniques such as cost-benefit analysis can help you to understand the Some research designs (such as financial value of an intervention’s experimental and quasi-experimental impact. designs) are better suited for demonstrating the presence of It might be important to establish a causal relationship. which sections of the population benefit the most from an intervention. Other research approaches are more This can often be masked by relying on appropriate for explaining how such the average effect for the population as causal relationships come about or a whole. answering other types of research question.
It is important for the humanitarian sector to know whether its programmes and interventions provide value for money.
RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR | 34 BACK TO SECTION C | WHAT EVIDENCE SHOULD YOU CHOOSE? CONTENTS
PART 2: HOW DO YOU JUDGE THE QUALITY OF RESEARCH?
Another way to help you choose hidden away. As an example, a survey which sort of research you need is to by researchers at Stanford University ask questions such as: what research found that most ‘null studies’ in the social can I trust? What is good enough sciences are never published: just 20% evidence to fit my needs? Might it be had appeared in a journal, and 65% had an article in a high-impact scientific not even been written up.65 This could journal? What about the in-house have serious consequences for decision evaluations conducted by my own makers reviewing evidence. If you never organisation? Should they also have see the negative studies on a new a place on the evidence table? intervention, you may wrongly conclude Being published in a peer-reviewed that all is well and that it is effective. research journal is one way to help you Nevertheless, despite these problems, feel confident about a piece of research, peer review remains, for most, the ‘gold but it is no guarantee of high quality. standard’ for deciding what makes it into In a famous paper, John Ioannidis from the scientific literature: providing a check Stanford University in the US caused a stir against bad work.66 Using a journal article by arguing that ‘most published findings or a report that has been independently are probably false’. He examined the most peer-reviewed by other experts is one cited papers (those with more than 1000 way of helping you be more confident citations) in some of the best regarded that you can trust the research. Although medical journals in the world – largely it can take years before research gets drawn from The Lancet, the New England published (and that can be too long if Journal of Medicine and the Journal of you have to make a quick decision), peer the American Medical Association.63 Of review brings some quality controls to those with claims of efficacy whose the literature to help you feel confident. results had been tested in future studies, 41% were either found to be wrong, DEFINING ‘HIGH-QUALITY’ or the impact was much smaller than RESEARCH the original study had suggested. Peer review may give us some modicum of comfort. But what do you do if you HOW DOES PEER REVIEW are going to include evidence that has HELP DECISION MAKERS? not been checked by other experts? We must also be mindful that peer How do you decide if it provides a good review, which is a cornerstone of enough basis for your decisions? academic journals is far from being It would clearly be a mistake to ignore perfect. There can be unconscious important research evidence just biases such as ‘herding’, where the because it had not been published in a behaviour of reviewers is influenced by peer-reviewed journal. Or to miss the the behaviour of their peers. And the rich seam of donor or NGO evaluations whole system of scientific journals can that never get published in academic be skewed by publication bias: positive outlets but, instead, are available only results have a better chance of being in the ‘grey literature’. There are a wide published,64 while negative data gets variety of definitions for grey literature,
35 | RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR BACK TO SECTION C | WHAT EVIDENCE SHOULD YOU CHOOSE? CONTENTS
but it usually refers to documents that quality, such as the GRADE68 or Maryland are unpublished or have been published Scientific Methods Scale69 systems. These without peer review. It can also refer to approaches to quality assessment for research that is still underway or being experimental evaluations are usually prepared for publication. Government based on the studies’ internal validity, reports, policy statements and briefs, quality of reporting and external validity. and conference proceedings are also Quality refers to how well studies have types of grey literature. Grey literature been conducted, reported and analysed,70 is important because it may contain as well as the researchers’ integrity in not evidence of negative outcomes and distorting or falsifying their data.71 Some unsuccessful interventions, which is people also link quality to how relevant important for the balance of evidence- the study is to policy and practice.72 informed decision making. Grey literature can be searched using electronic When trying to answer a causal question, databases such as Open Grey (www. you need to consider whether the opengrey.eu), conference proceedings research design used for a study is and the procurement records of research appropriate for determining causality and funders. Websites of organisations that whether the design was implemented have an interest or expertise in a topic properly in the study. High-quality impact are another source of grey literature. evaluations will answer questions of These organisations can be contacted to attribution: showing that the intervention identify researchers and decision makers caused the outcomes. This requires a who have expertise in a topic or issue. comparison or control group which is as similar as possible to the intervention A good start in trying to appraise the group in all regards expect the actual quality of evidence is defining it. One of intervention. If this is true and the study the problems, however, is that phrases has been well conducted, you can be such as ‘quality’, ‘standards’, ‘robustness’, more confident that, for example, the ‘bias’ and ‘strength’ are often used as if effects on the prevention of violence, they were interchangeable, and without reduction in family stress or faster return clearly defining what they mean. This to work are due to the intervention. It makes for a lot of misunderstanding. For 67 is also important to consider whether instance, in some guidance, research the effects found in the study will be ‘quality’ means using particular types replicated in other places. This drives the of design and method – such as a demand for mixed methods of research randomised trial. This focus on minimising and evaluation and might also require bias as a means of ensuring quality arises information from qualitative research. from some of the formal clinical and health approaches to assessing evidence
It would clearly be a mistake to ignore important research evidence just because it had not been published in a peer-reviewed journal.
RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR | 36 BACK TO SECTION C | WHAT EVIDENCE SHOULD YOU CHOOSE? CONTENTS
HIGH-QUALITY QUALITATIVE research and the accurate synthesis of RESEARCH existing information has been described as “the most important single offering by Many have highlighted the value of good academics to the policy process”.75 Much quality qualitative research73 for policy of the thinking on quality set out above makers. Qualitative evidence can be just has focused on single studies in ‘primary as scientifically credible as quantitative, research’; in other words, individual studies and a combination of the two is likely to such as experiments, surveys or a series of help decision makers combine evidence interviews. But, it’s important to think about of the effects of an intervention with the ‘research syntheses’, and the summarising knowledge to decide on the applicability and pooling together of a series of of that evidence to their situation. The key individual studies. This idea is not new, and thing is that research evidence that you the concept of making decisions on the use to deal with any challenge that you basis of accumulated evidence has been face is fit for purpose. It needs to have stressed for centuries.76 used the appropriate method, to have collected and analysed its data (whether Our earlier discussion of cognitive biases qualitative or quantitative) using well- showed how people can unconsciously fall defined and replicable methods and to into the trap of looking for evidence that report the findings in a transparent way fits our beliefs. So, we need to be careful that minimises bias. when collecting together the existing research and need to be willing to include As with quantitative research, there are research even if we do not like its findings. standards and guidelines for assessing One way to avoid ‘cherry picking’ is to the quality of qualitative research and use what are called systematic reviews.77 evaluation.74 Whereas the quality of These aim to be exhaustive, and to find as experimental studies is usually based on much as possible of the research relevant internal validity, adequacy of reporting and to answering the question that will help external validity, the quality of qualitative with the challenge we are faced with. research is usually built around the They use explicit methods to identify dimensions of contribution, defensibility, what can reliably be said on the basis rigour and credibility. Contribution refers of the available studies and assess the to whether the study advances wider research design and methods of these knowledge or understanding about a studies to determine their quality. policy, practice, theory or a particular substantive field. Defensibility refers to Some of the key characteristics78 of a whether the study provides an appropriate systematic review are: research strategy to address the evaluative Clearly stated research question, objectives questions posed. Rigour refers to the and eligibility criteria for studies. systematic and transparent collection, analysis and interpretation of qualitative Explicit and reproducible methodology data. Credibility refers to how well-founded to minimise bias. and plausible are the arguments about the Systematic search to identify as many evidence generated. studies as possible that meet the eligibility criteria. AVOID ‘CHERRY PICKING’ BY USING SYSTEMATIC REVIEWS Formal assessment of the validity of the findings of the included studies. Decision makers need to think about the quality of the whole body of evidence, Systematic presentation, and synthesis, not just single pieces of evidence. They of the characteristics and findings of the need to use aggregated collections of included studies.
37 | RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR BACK TO SECTION C | WHAT EVIDENCE SHOULD YOU CHOOSE? CONTENTS
CASE STUDY Systematic approaches also have the value of being explicit about how Using systematic reviews they searched for research studies to develop guidance for and decided on their eligibility. So, in theory at least,79 other people could staffs replicate the systematic review. The International Rescue Committee In 2018, Christl Donnelly and colleagues (IRC) place a great reliance on suggested four principles to help evidence in their development of researchers, policymakers and others programme guidance documents and to commission, do, share, appraise tools for field staff. They do this by and use evidence syntheses, including conducting evidence reviews across systematic reviews.80 They proposed that many high-quality sources around evidence synthesis should be inclusive, specific interventions or approaches, rigorous, transparent and accessible and and use what they learn about impact, elaborated on these four areas as follows: contexts, populations and conditions to inform whether and how to adapt Inclusive those interventions to specific Involves policy makers and is relevant contexts. and useful to them. This is not without its challenges Considers many types and sources because most of the available studies of evidence. do not provide critical information such as fidelity of implementation, causal Uses a range of skills and people. mechanisms and gender sensitivity. Rigorous. The IRC’s agency-wide effort to ensure Uses the most comprehensive feasible that evidence is readily available to body of evidence. their staff is through the development Recognises and minimises bias. of the Outcomes and Evidence Is independently reviewed as part Framework (oef.rescue.org). In this of a quality-assurance process. electronic publicly available platform, the IRC have defined the outcomes Transparent and sub-outcomes that IRC focuses Clearly describes the research question, on in its work, the general theories of methods, sources of evidence and change (or pathways) through which quality-assurance process. they can achieve those outcomes and Communicates complexities and areas indicators for measuring them. of contention. For each sub-outcome and outcome, Acknowledges assumptions, limitations the IRC have summarised the best and uncertainties, including any available quantitative evidence on the evidence gaps. effectiveness of interventions that aim Declares personal, political and to change/improve the relevant sub- organisational interests and manages outcome or outcome, with a primary any conflicts. focus on evidence from systematic reviews. For topics where systematic Accessible reviews do not yet exist, the IRC has Is written in plain language. identified and summarised individual Is available in a suitable time frame. impact evaluations. Is freely available online.
RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR | 38 BACK TO SECTION C | WHAT EVIDENCE SHOULD YOU CHOOSE? CONTENTS
META-ANALYSIS FOREST PLOTS Meta-analysis is an important part of Figure C.1 presents the findings of 28 systematic reviews where all or some impact evaluations that compared of the included studies are sufficiently people in humanitarian emergencies similar (‘homogeneous’) in terms of who had received mental health and population, intervention, comparators and psychosocial support programmes with outcomes for their statistical findings to those who had not received this type of be pooled and aggregated. This results intervention.82 In this forest plot, the data in a cumulative estimate of effect which presents the effects of the programmes generally has a lower risk of bias than the on the severity of post-traumatic effect estimate for individual studies in Figure C.1: Forest plot of the effect of mental health and psychosocial isolation support programmes on post-traumatic stress disorder82 and greater Measure: Continuous: d(Hedges g) precision. Heterogeneity: Q=206; df=27; p=0; I2=86.9%; tau-squared=0.29 Meta-analysis Random effects model: -0.463 (-0.689, -0.237) is usually represented by Berger (2009) -1.269 (-1.607 | -0.932) W:5.1 a forest plot81 Catani (2009) 0.251 (-0.47 | 0.971) W:1.1 Chen (2014) -1.277 (-2.41 | -0.144) W:0.5 such as that Chen (2014) -0.138 (-1.152 | 0.875) W:0.6 in Figure C.1, Cluver (2015) 0.59 (-0.257 | 1.436) W:0.8 which is taken Dybdahl (2001) -0.137 (-0.558 | 0.284) W:3.3 from a recent Ertl (2011) -0.457 (-1.12 | 0.206) W:1.3
systematic Ertl (2011) -0.035 (-0.7 | 0.629) W:1.3
review on Gordon (2008) -1.116 (-1.595 | -0.637) W:2.5
the impact Jordans (2010) -0.18 (-0.858 | 0.498) W:1.3
of support Khamis (2004) 0.205 (0.006 | 0.404) W:14.6
programmes Khamis (2004) 0.066 (-0.178 | 0.311) W:9.7
for populations Lange-Neilsen (2012) -0.11 (-0.462 | 0.243) W:4.6
affected by Layne (2008) -0.109 (-0.599 | 0.38) W:2.4
humanitarian McMullen (2013) -2.727 (-3.539 | 1.915) W:0.9
emergencies. O'Callaghan (2013) -1.944 (-2.66 | -1.228) W:1.1 Meta-analysis O'Callaghan (2014) -0.405 (-0.72 | -0.09) W:5.8 is perhaps O'Callaghan (2015) -1.994 (-2.848 | 1.141) W:0.8 best known O'Callaghan (2015) -2.595 (-3.558 | 1.633) W:0.6 for combining Pityaratstian (2015) -0.473 (-1.137 | 0.19) W:1.3 the results of Qouta (2012) 0.01 (-0.367 | 0.387) W:4.1 randomised Qouta (2012) -0.401 (-0.813 | 0.011) W:3.4 trials, but it can Schauer (2008) -0.062 (-0.7 | 0.576) W:1.4 also be done Tol (2008) -0.656 (-0.98 | -0.332) W:5.5 with data from Tol (2012) 0.329 (0.006 | 0.652) W:5.6 other types Tol (2012) -0.074 (-0.329 | 0.18) W:9 of study, such Tol (2014) -0.166 (-0.479 | 0.148) W:5.9 as those that Tol (2014) -0.102 (-0.426 | 0.222) W:5.5 have used case- TOTAL: -0.463 (-0.689 | -0.237) control, cross- -3.5 -3 -2.5 -2 -1.5 -1 0 0.5 1 sectional or Favours intervention Favours control cohort designs.
39 | RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR BACK TO SECTION C | WHAT EVIDENCE SHOULD YOU CHOOSE? CONTENTS
stress disorders (PTSD) as a continuous systematic review. They use the same basic variable. For each study in this forest structure and stages of full systematic plot, the red dot represents the average review, but are not as intensive, exhaustive treatment effect of the intervention, and or comprehensive. They will take more the parallel lines either side of the red short cuts with the searching, critical dots represent the confidence interval appraisal, data extraction and gathering, for that study. The solid black vertical and statistical analysis of included line running from 0 on the horizontal axis studies. The findings are also presented indicates no difference between using in a shorter and less detailed form than and not using the programme, and the a full systematic review, and might be results of all the studies are pooled to no longer than 25 pages, with a three- provide the overall estimate of the effects page executive summary and a one-page of the programmes. This new, summary briefing document for decision makers. statistic is the black diamond (circled The limitations of rapid evidence in red) at the bottom of the forest plot. assessments are that they are not as This represents the cumulative estimate comprehensive or exhaustive as systematic of effect of pooling and aggregating the reviews and are more likely to be subject average effects sizes and the variances to bias than a full systematic review. of all 28 impact evaluations included Consequently, greater caution is needed in the review. It allows us to conclude when basing a decision on evidence from that, on average, the mental health and a rapid evidence assessment than from psychosocial support programmes have a a full systematic review. Notwithstanding small, positive effect on PTSD compared these limitations, they are frequently with not using these interventions. commissioned and used by policy makers and programme implementers, RAPID EVIDENCE ASSESSMENTS especially where time is of the essence The preparation of systematic reviews and and no systematic reviews are available. meta-analyses can be time-consuming. This means that if an up-to-date A particular type of rapid review, called systematic review is not available, people a Rapid Research Needs Assessment, needing to make an urgent decision might can also be used to quickly identify need to conduct their own searches for evidence gaps. The UK’s Public Health the relevant pieces of evidence and then Rapid Support Team for disease appraise and synthesise this faster than outbreaks includes a plan to conduct would happen in a formal systematic these assessments with Evidence Aid, review. Fortunately, the already large to identify important uncertainties that number of systematic reviews is continuing could be tackled by research in the early to grow rapidly, and we describe how to stages of a humanitarian emergency find them and several collations that are associated with a disease outbreak. available in Section D. However, if you are unable to find what you are looking for THE IMPORTANCE OF REPETITION amongst the existing systematic reviews, AND CORROBORATION or the reviews you find are out of date, you Thinking about the concept of evidence might need to think about commissioning gaps, brings us to one of the other a ‘pared-down systematic review’, such as things that needs to be considered when a rapid evidence assessment.83 These rapid assessing the quality of a summary reviews normally take 1-3 months and are of research studies: the number of timed to meet the needs of policy makers studies that need to be included for and practitioners who cannot wait for a full you to be comfortable that the body
RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR | 40 BACK TO SECTION C | WHAT EVIDENCE SHOULD YOU CHOOSE? CONTENTS
of evidence is strong enough. Would REVIEW OF REVIEWS: HOW TO only a couple of studies be sufficient JUDGE HIGH–QUALITY BODIES OF if they are really good? Or might you EVIDENCE require dozens, or even hundreds? We need to remember that the formal The reality is that there is no magic ways of bringing research together, for number of studies. Yet, we cannot ignore example in systematic reviews, do not the fact that the size of the body of always lead to a high-quality answer. evidence is important: there is strength You still need to judge the quality of in numbers, and we must have repetition the design and implementation of the and corroboration. Even studies that evidence review, and the quality of any have won many accolades need to be synthesis can only be as good as the repeated. Amgen, a Californian drug quality of the studies it is based on. company, tried to replicate 53 landmark Even the best-done review cannot turn cancer studies. The work, published in low-quality research into a high-quality 84 Nature, only confirmed the findings answer. of six (11%) of these important studies. This is deeply troubling because the There are also formal appraisal tools studies have influenced drug companies for assessing the quality of systematic and cancer treatments globally and the reviews, such as AMSTAR.87 You might inability to replicate them might indicate also use one of the original checklists that their findings are unreliable. On the for assessing the quality of systematic other hand, we do not want to see studies reviews,78 which posed a series of being done again and again long after the questions: evidence base is robust.85 Unnecessary studies represent research waste86 and, Is the question clearly focused? in the context of randomised trials, Is the search for relevant studies may be unethical if some participants thorough? continue to be randomised to sub-optimal interventions. Are the inclusion criteria appropriate? Is the validity of the included studies A How to note on judging the adequately addressed? strength of evidence produced by the UK’s Department for International Is missing information obtained from the original researchers? Development lists four things to consider when checking bodies of How sensitive are the results to changes evidence:41 in the way the review is done? Are subgroup analyses interpreted The (technical) quality of the studies cautiously? constituting the body of evidence (or the degree to which risk of bias Do the conclusions flow from the has been addressed). evidence that is reviewed? The size of the body of evidence. Are recommendations linked to the strength of the evidence? The context in which the evidence is set. Are judgements about preferences (values) valid? The consistency of the findings produced by studies constituting Is ‘evidence of no effect’ confused with the body of evidence. ‘no evidence of effect’?
41 | RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR BACK TO SECTION C | WHAT EVIDENCE SHOULD YOU CHOOSE? CONTENTS
Judging all these criteria is always existing systematic reviews – and in going to be rather subjective and some sectors there are few. However, needs to take account of the context these reviews of reviews can provide an of the policy question. It is difficult to appraisal and summary of evidence from give blanket advice for what would multiple systematic reviews of the same constitute the right body of evidence intervention; or provide an overview of for any policy. We are also beginning the evidence from a collection of reviews to see the combination of the findings of different interventions in the same from multiple systematic reviews in topic area.88 ‘reviews of reviews’. Of course, this cannot work if there is not a body of
Key messages for part 2 of Section C numbers, and we need repetition and To find evidence that you can trust, corroboration. Even studies that have look for peer-reviewed research. But won many accolades need to be note that peer review is far from repeated, but we need to avoid doing perfect. There can be unconscious studies again and again long after the biases such as ‘herding’, or publication evidence base is strong enough. bias towards positive results, or even the deliberate distortion or falsification Use systematic reviews, which aim to of data. be exhaustive and screen studies for quality – usually based on the research When looking at questions of impact design and methods. and ‘what works’, use the frameworks and formal standards of evidence such If you do not have time to do a as those used by Nesta and others. systematic review or cannot find a completed one that answers your One study is never enough. Avoid question, you might wish to conduct or making decisions based on single commission a rapid evidence studies and look for multiple assessment.89 replications. There is strength in
Even the best done review cannot turn low-quality research into a high-quality answer.
RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR | 42 BACK TO SECTION D | WHERE SHOULD YOU LOOK FOR EVIDENCE? CONTENTS
Where should you look for evidence?
This section looks at different types of evidence and examines how to choose the most appropriate for the issue you are interested in. It also discusses how to judge the quality of evidence.
SEARCHING FOR RESEARCH journals, books and websites; many of EVIDENCE which are not free to use or are difficult to search. When thinking about where to With the rapid growth in the number of search, it is important to consider where people and organisations working on the evidence you are interested in is likely systematic reviews, the literature now to have been published and indexed, contains a large and ever-increasing and how comprehensive you wish to be. number of systematic reviews, so you For example, if they were published in a might be able to find one that meets your scientific journal, the articles might be needs. A recent estimate is that there are available through one of the thousands more than 200,000 systematic reviews of electronic bibliographic databases. across all topic areas. Of course, only These include, for instance, PubMed a small proportion of these would be for health care, LILACS for articles with relevant to the humanitarian sector but (as particular relevance to South America noted below) bundles of such reviews are or ERIC for educational literature. It freely available in online collections such may also be necessary to identify as those curated by Evidence Aid. There grey literature, such as government or are also large international organisations, NGO reports, and research presented such as the Cochrane (www.cochrane. at conferences, which will require org) and Campbell Collaborations (www. searches of electronic databases such campbellcollaboration.org) dedicated as Open Grey ( ) and to the preparation and maintenance www.opengrey.eu the websites of organisations that may of systematic reviews who make these have conducted relevant research. available online. In planning a search, it can be helpful However, if you need to do your own to divide the search elements into the searches for studies of the effects of types of intervention or policy you are humanitarian action, the starting point interested in, the relevant population is to recognise that these are not always or settings, the outcome measures that easy to find and you might need to would be most helpful to your decision get help from a librarian or information making and the types of study you specialist who will be to advise and, wish to find. You can then decide which perhaps, assist with designing and one or more of these domains is most running the search. The evidence is important and helpful for identifying the scattered across tens of thousands of material that is most relevant to you. reports spread across thousands of
43 | RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR BACK TO SECTION D | WHERE SHOULD YOU LOOK FOR EVIDENCE? CONTENTS
You should consider the types of SOURCES OF SYSTEMATIC source to search, which might include: REVIEWS AND OTHER RESEARCH FOR THE HUMANITARIAN SECTOR Bibliographic databases (e.g. PubMed, LILACS, ERIC). Because of these challenges in seeking evidence, you might find it more efficient Journal and conference websites and easier to use trusted repositories of (e.g. those with a particular focus on research and systematic reviews, rather disasters or humanitarian action). than relying on haphazard searches Registries of research (such as of the internet. Fortunately, several through the WHO portal for trial, organisations are now working to bring www.researchregistry.com and relevant evidence together into online PROSPERO for systematic reviews). resources. These include the following: Online collections of research evidence The Active Learning Network for (see below, e.g. ALNAP, 3ie). Accountability and Performance Website of relevant organisations (ALNAP) is hosted by the Overseas (e.g. for government and NGO reports). Development Institute (ODI) in London, UK, with a website containing more References in articles. than 15,000 resources including ALNAP publications relevant to evidence and Correspondence with researchers details of several thousand evaluations. and evaluators. The foundations of Evidence Aid were laid down within Cochrane following the When choosing the terms to include Indian Ocean tsunami of 2004, when in your search, you should consider: the full text of several dozen Cochrane Reviews relevant to disaster response Synonyms from different times were made freely available online. It was and places. established as a charity in 2015 and has Other words and phrases that now collated several hundred systematic are related to what you’re reviews that are all free to view from interested in. its website (www.evidenceaid.org), including, but not limited to, bundles of Words that are broader. Special Collections of reviews relevant Words that are more narrow or to windstorms, earthquakes, Ebola, the focused. health of refugees and asylum seekers, post-traumatic stress disorder, and Index terms or keywords assigned to articles by the original authors or the prevention of acute malnutrition in bibliographic database. emergencies and humanitarian crises. And, finally, you need to decide The International Initiative for Impact on whether you will apply any Evaluation (3ie) was established in 2008 restrictions based on language and and now offers four searchable databases the time period in which the research online (www.3ieimpact.org). Two of these, was conducted or published. the 3ie Database of Systematic Reviews and the Database of Impact Evaluations catalogue evidence of the effectiveness of interventions in the humanitarian sector. These databases also include
RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR | 44 BACK TO SECTION D | WHERE SHOULD YOU LOOK FOR EVIDENCE? CONTENTS
systematic reviews and impact evaluations on the broader landscape of international development, many of which have relevance to interventions in emergency situations.
We are maintaining an up-to-date fuller list of these types of resources online, at: www.evidenceaid.org/online-collections-of-research-for-the-humanitarian-sector The list provides a wide range of online research resources, many of which are free and easy to access. These should be useful to any policy maker, NGO or frontline professional in the humanitarian sector, providing easy access to reliable, high quality evidence on the effectiveness of interventions. If you would like to suggest additional resources for this list, please contact Evidence Aid: [email protected]
CONCLUSION In conclusion to this guide on the use of evidence in the humanitarian sector, we encourage you to take advantage of the freely available, accessible and actionable summaries of research, such as the systematic reviews contained on the websites we have listed. This will help you to move quickly to sources of evidence that should help inform your policy and practice.
Key messages for Section D
Research evidence of relevance to Systematic reviews provide the humanitarian sector is scattered summaries and synthesis of research across tens of thousands of reports evidence. spread across thousands of journals, books and websites. Several online repositories of research are available, improving access Searching for this evidence may to systematic reviews and other types require help from an information of research evidence. specialist and designing the search needs to consider carefully what is being looked for, the sources to search, to terms to use in the search and any restrictions relating to language or time period.
You might find it more efficient and easier to use trusted repositories of research and systematic reviews.
45 | RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR BACK TO ENDNOTES CONTENTS
8 www.oxforddictionaries.com/ Endnotes definition/english/evidence. 9 Nutley S, et al. (2013) What Counts 1 Collins S, Sadler K. (2002) Outpatient care for as Good Evidence? London, UK: severely malnourished children in emergency Alliance for Useful Evidence.
relief programmes: a retrospective 10 cohort study. Lancet 360:1824-30. Hallam A, Bonino F. (2013) Using Evaluation for a Change: Insights from humanitarian 2 WHO, World Food Programme, UN practitioners. ALNAP Study. London: System Standing Committee on Nutrition, ALNAP/ODI. See www.alnap.org/system/ and UNICEF. (2007) Community-based files/content/resource/files/main/alnap- management of severe acute malnutrition: study-using-evaluation-for-a-change.pdf.
a joint statement by the WHO, World Food 11 Programme, UN System Standing Committee Ariely D. (2009) Predictably Irrational: on Nutrition and UNICEF. New York: UNICEF. The Hidden Forces that Shape Our Decisions. London, UK: HarperCollins. 3 Price AI, Djulbegovic B. (2017) What 12 does evidence mean? Most languages HM Treasury. (2011) The Green Book: translate “evidence” into “proof”. Journal of Appraisal and Evaluation in Central Evaluation in Clinical Practice 23(5):971-3. Government. London, UK: HM Treasury.
13 4 Sackett D, et al. (1996) Evidence based Christoplos I. (2006) Links between medicine: what it is and what it isn’t. Relief, Rehabilitation and Development in the Tsunami Response: A Synthesis British Medical Journal 312:71. See www. of Initial Findings. Stockholm, bmj.com/content/312/7023/71. Sweden: Swedish International 5 The definition of evidence-based Development Cooperation Agency.
medicine came at a time when most 14 medical decision making was based on Zsambok CE, Klein G, (editors). (2014) experience, authority and eminence. Naturalistic decision making. New Medical practice was not informed by the York, USA: Psychology Press. best available scientific evidence. Some 15 Kahneman D, et al. (2009) Conditions for commentators and researchers have intuitive expertise: a failure to disagree. argued that social policy is in the same American Psychology 64:515-26. See place as medicine was 20 or 30 years ago, . namely that authority, rather than research www.ncbi.nlm.nih.gov/pubmed/19739881 evidence, dominates decision making. 16 These estimates come from Cash Learning Partnership (CaLP). (2018) The State of 6 Allen C, et al. (2016) Evidence Aid. Oxford the World’s Cash Report. Oxford: CaLP; Public Health August: 51-54; Blanchet K, Doing Cash Differently: Report of the High et al. (2017) Evidence on public health Level Panel on Cash Transfers. (2015); and interventions in humanitarian crises. Lancet Development Initiatives. (2017) Global 390:2287-96; and Christoplos I, et al. (2017) Humanitarian Assistance Report. Strengthening the quality of evidence in humanitarian evaluations. ALNAP Method 17 ODI, Development Initiatives. (2016) Note. London: ALNAP/ODI (see www.alnap. Counting Cash: Tracking Humanitarian org/system/files/content/resource/files/ Expenditure on Cash-Based Programming. main/alnap-eha-method-note-5-2017.pdf). 18 Oxfam. (2006) Good Practice Review 7 Knox Clarke P, Darcy J. (2014) Insufficient 11: Cash Transfer Programming in evidence? The quality and use of evidence Emergencies. Oxford: Oxfam. in humanitarian action. ALNAP Study. London: ALNAP/ODI. See www.alnap. org/system/files/content/resource/ files/main/alnap-study-evidence.pdf.
RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR | 46 BACK TO ENDNOTES CONTENTS
19 Jackson R (Save the Children UK), Kukrety 29 Waddington H, et al. (2009) Water, N (Oxfam GB) (2012) Institutionalising Sanitation and hygiene interventions to cash transfer programming. See https:// combat childhood diarrhoea in developing odihpn.org/magazine/institutionalising- countries: a systematic review. Delhi, India: cash-transfer-programming/. International Initiative for Impact Evaluation.
20 See www.humanitarianresponse. 30 Nesta and TSIP. (2014) Guidance for info/en/programme-cycle/space. Developing a Theory of Change for Your Programme. See www.nesta.org.uk/ 21 Discussion of the challenges of making sites/default/files/theory_of_change_ decisions following the rare circumstances guidance_for_applicants_.pdf. of a major radition emergency is available in Carr Z, et al. (2016) Using the GRADE 31 Stern E. (2015) Impact Evaluation: A approach to support the development Design Guide for Commissioners and of recommendations for public health Managers of International Development interventions in radiation emergencies. Evaluations in the Voluntary and Community Radiation Protection Dosimetry 171:144-55; Sector. London, UK: Big Lottery Fund, and Ohtsuru A, et al. (2015) Nuclear disasters Bond, Comic Relief and the Department and health: lessons learned, challenges, for International Development. and proposals. Lancet 386: 489-97. 32 Gerdin M, et al. (2014) Optimal evidence 22 Bradley DT, et al. (2014) The effectiveness in difficult settings: improving health of disaster risk communication: a systematic interventions and decision making in review of intervention studies. PLOS disasters. PLoS Medicine 11(4): e1001632. Currents Disasters August 22; Edition 1. 33 Mulgan G. (2015) The six Ws: a formula for 23 A discussion of the importance of paying what works. London: Nesta. See www.nesta. attention to how donors gather, use and org.uk/blog/six-ws-formula-what-works. share evidence and information is available in Obrecht A. (2017) Using Evidence to 34 www.ebola-anthropology. Allocate Humanitarian Resources: Challenges net/about-the-network. and Opportunities. ALNAP Working Paper. London: ALNAP/ODI. See https://reliefweb. 35 Faye SL. (2015) L“exceptionnalité” d’Ebola int/sites/reliefweb.int/files/resources/ et les “réticences” populaires en Guinée- alnap-eaar-resource-allocation-2017.pdf. Conakry. Réflexions à partir d’une approche d’anthropologie symétrique. Anthropologie 24 Nesta. (2013) Understand how & Santé. See https://journals.openedition. innovation works. Video available org/anthropologiesante/1796. at www.nesta.org.uk/resources/ understand-how-innovation-works. 36 http://pubman.mpdl.mpg.de/pubman/ item/escidoc:2096578/component/ 25 Obrecht A, Warner AT. (2016) More than escidoc:2103624/AAA-Ebola-Report-1.pdf. just luck: Innovation in humanitarian action. HIF/ALNAP Study. London: ALNAP/ODI. 37 Fairhead J. (2016) Understanding Social Resistance to the Ebola Response in the 26 Jones G, et al. (2003) How many child Forest Region of the Republic of Guinea: An deaths can we prevent this year? Lancet Anthropological Perspective. African Studies 362:65-71. See https://linkinghub.elsevier. Review 59:7-31. doi:10.1017/asr.2016.87. com/retrieve/pii/S0140-6736(03)13811-1. 38 Abramowitz S, et al. (2015) Social 27 Nyhan B, Reifler J. (2010) When science intelligence in the global corrections fail: The persistence of political Ebola response. Lancet 385:330. misperceptions. Political Behavior 32:303-30. 39 Petticrew M, et al. (2003) Evidence, 28 White H. (2009) Theory-based impact hierarchies and typologies: horses evaluation: principles and practice, for courses. Journal of Epidemiology working paper 3. Delhi, India: International and Community Health 57:527-9. Initiative for Impact Evaluation, page 4.
47 | RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR BACK TO ENDNOTES CONTENTS
40 An authoritative and exhaustive list of social 47 Doocy S, Burnham G. (2006). Point- science research frameworks and methods of-use water treatment and diarrhoea is available in Luff R, et al. (2015) Review of reduction in the emergency context: the Typology of Research Methods within an effectiveness trial in Liberia. Tropical the Social Sciences. London, UK: ESRC/ Medicine and International Health 11:1542-52. National Centre for Research Methods. See http://eprints.ncrm.ac.uk/3721. 48 White H, et al. (2014) Randomised Controlled Trials (RCTs). Methodological 41 Department for International Development. Briefs. Impact Evaluation No. 7. Florence, (2014) How to Note: Assessing the Italy: Unicef. See www.unicef-irc.org/ Strength of Evidence. See www.gov.uk/ publications/pdf/brief_7_randomised_ government/uploads/system/uploads/ controlled_trials_eng.pdf. attachment_data/file/291982/HTN- strength-evidence-march2014.pdf. 49 Gertler P. (2000) Final Report: The Impact of PROGRESA on Health. Washington, DC, 42 Adapted from HM Treasury, DECC and DEFRA. USA: International Food Policy Research (2012) Quality in policy impact evaluation: Institute (IFPRI). See www.ifpri.org/ understanding the effects of policy from other publication/impact-progresa-health. influences. London, UK: HM Treasury/DEFRA/ DECC; Frost S, et al. (2006) The Evidence 50 Hainmueller J, et al. (2017) Catalyst or Guide; Using Research and Evaluation in Social crown: does naturalization promote Care and Allied Professions. London, UK: the long-term social integration Barnardo’s; Petticrew M, Roberts H. (2003) of immigrants? American Political Evidence, hierarchies and typologies: horses Science Review 111(2):256-76. for courses. Journal of Epidemiology and doi:10.1017/S0003055416000745. Community Health. 57: 527-9; and Stern E. (2015) Impact Evaluation; A Design Guide for 51 Bozorgmehr K, et al. (2015) Effect of Commissioners and Managers of International restricting access to health care on Development Evaluations in the Voluntary and health expenditures among asylum- Community Sector. London, UK: Big Lottery Fund, Bond, Comic Relief and the Department seekers and refugees: A quasi- for International Development, Table 2, page 18. experimental study in Germany, 1994–2013. PLoS ONE 10: e0131483. 43 Odgaard-Jensen J, et al. (2011) Randomisation 52 to protect against selection bias in Rossi R, et al. (2016) Vaccination health care trials. Cochrane Database of coverage cluster surveys in Middle Systematic Reviews (4):MR000012. Dreib - Akkar, Lebanon: comparison of vaccination coverage in children aged 44 White H. (2013) An introduction to the use 12-59 months pre- and post-vaccination of randomised control trials to evaluate campaign. PLoS ONE 11(12): e0168145. development interventions. Journal of Development Effectiveness 5(1):30-49. 53 Alexander J, Bonino F (2015). A discussion on designs, approaches and examples, 45 See for example: Schulz KF, et al. (2010) ALNAP Discussion Series Improving CONSORT 2010 statement: updated guidelines the quality of EHA evidence, Method for reporting parallel group randomised Note 4, January 2015. See www.alnap. trials. PLoS Medicine 7(3):e1000251; org/system/files/content/resource/ Higgins JP, et al. (2011) The Cochrane files/main/alnap-eha-method-note- Collaboration's tool for assessing risk of addressing-causation-jan2015.pdf. bias in randomised trials. BMJ 343:d5928. 54 Waddington H, et al. (2017) Quasi- 46 Puri J, et al. (2015) What methods may be experimental study designs series paper used in impact evaluations of humanitarian 6: risk of bias assessment. Journal of assistance? working paper 22, Delhi, India: Clinical Epidemiology 89:43-52; and International Initiative for Impact Evaluation. Bärnighausen T, et al. (2017) Quasi- See www.3ieimpact.org/media/filer_ experimental study designs series—paper public/2014/12/08/wp_22_humanitarian_ 7: assessing the assumptions. Journal methods_working_paper-top.pdf. of Clinical Epidemiology 89:53-66.
RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR | 48 BACK TO ENDNOTES CONTENTS
65 55 Nielsen NS, et al. (2013). The Contribution Franco A, et al. (2014) Social science. of Food Assistance to Durable Solutions Publication bias in the social sciences: in Protracted Refugee Situations; its unlocking the file drawer. Science 345:1502-5. impact and role in Bangladesh: A Mixed 66 Sense about Science. (2006) I Method Impact Evaluation, Volume don’t know what to believe: Making I-Evaluation Report, Geneva, Switzerland: sense of science stories. See World Food Program/UNHCR. www. senseaboutscience.org/resources. 56 Carter R. (2012) Helpdesk Research Report: php/16/i-dont-know-what-to-believe. Theory-based evaluation approach. University 67 See for instance, the supplement to the of Birmingham, UK: Birmingham: Governance Magenta Guide: HM Treasury, DECC and and Social Development Resource Centre. DEFRA (2012) Quality in policy impact See www.gsdrc.org/docs/open/hdq872.pdf. evaluation; understanding the effects of 57 Westhorp G. (2014) Realist Evaluation: An policy from other influences. The guidance Introduction. London, UK: Methods Lab, shows how ‘higher quality research designs can help meet the challenge of attributing Overseas Development Institute. See www. measured outcomes to the policy in odi.org/sites/odi.org.uk/files/odi-assets/ question (as opposed to other influences), publications-opinion-files/9138.pdf. whereas lower quality designs reduce 58 Mayne J. (2012) Contribution Analysis: confidence in whether it was the policy Coming of Age? Evaluation 18(3):270-80. that achieved those outcomes’ (page 5).
68 59 For network analysis and process tracing http://gradeworkinggroup.org.
see: Befani B, et al. (2014) Process tracing 69 and contribution analysis: a combined Farrington RP, et al. The Maryland Scientific approach to generative causal inference for Methods Scale, In: Farrington DP, et impact evaluation. IDS Bulletin 45(6):17-36. al. Evidence-based crime prevention. London, UK: Routledge, 2002 chapter 2. 60 Baptist C, et al. (2015) Coffey How To 70 Guide: Qualitative Comparative Analysis For a model of different approaches on – A Rigorous Qualitative Method for quality that includes the four dimensions of (1) methodological quality (2) quality Assessing Impact. See www.coffey. in reporting, (3) appropriateness and com/assets/Ingenuity/Qualitative- (4) relevance to policy and practice; see Comparative-Analysis-June-2015.pdf. Boaz A, Ashby D. (2003) Fit for purpose? 61 Stern E, et al. (2012) Broadening the Assessing research quality for evidence range of designs and methods for impact based policy and practice. Working evaluations, London, UK: Department Paper 11. London, UK: ESRC UK Centre for International Development. for Evidence Based Policy and Practice.
71 62 For example, see Carol Weiss’s guide: Callaway E. (2011) Report finds Weiss C. (1998) Evaluation: methods massive fraud at Dutch universities. for studying programs and policies. Nature 479(15). See www.nature.com/ New York, USA: Prentice Hall. news/2011/111101/full/479015a.html.
72 63 Ioannidis JA. (2005) Contraindications and Boaz A, et al. (2003) Fit for purpose? Initially Stronger Effects in Highly Cited Assessing research quality for evidence Clinical Research. Journal of the American based policy and practice. Working Medical Association. 294(2):218-28. Paper 11. London, UK: ESRC UK Centre for Evidence Based Policy and Practice. 64 For a systematic review of publication bias see 73 Hopewell S, et al. (2009) Publication bias in Qualitative research explores and tries to clinical trials due to statistical significance or understand people’s beliefs, experiences, direction of trial results. Cochrane Database attitudes, behaviour and interactions. It of Systematic Reviews (1):MR000006. doi: generates non-numerical data which might 10.1002/14651858.MR000006.pub3. be gathered through, for example in-depth interviews, focus groups, documentary analysis and participant observation.
49 | RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR BACK TO ENDNOTES CONTENTS
74 Spencer L, et al. (2002) Quality in Qualitative 82 Bangpan M, et al. (2017) The impact Evaluation: A framework for assessing of mental health and psychosocial research evidence. London, UK: Cabinet support programmes for populations Office; CASP. (2018) CASP Checklist for affected by humanitarian Qualitative Research. Oxford, UK: Critical emergencies. Oxford, UK: Oxfam. Appraisal Skills Programme; National Institute for Health and Clinical Excellence. 83 Ganann R, et al. (2010) Expediting (2012) The guidelines manual: Appendix H: systematic reviews: methods and Methodology checklist: qualitative studies. implications of rapid reviews. London, UK: National Institute for Health Implementation science 5:56; and Tricco and Clinical Excellence; and O'Brien BC, et AC, et al. (2015) A scoping review of rapid al. (2014) Standards for reporting qualitative review methods. BMC Medicine 13: 224. research: a synthesis of recommendations. Academic Medicine 89: 1245-51. 84 Begley CG, et al. (2012) Drug development: Raise standards for preclinical cancer 75 Whitty CJM. (2015) What makes an research. Nature 483: 531-3. academic paper useful for health policy? BMC Medicine 13:301. 85 A review of research that had included one or more cumulative meta-analysis 76 See, for example, the discussion of the found many examples showing that stable importance of using existing evidence when results (beneficial, harmful and neutral) designing new studies in Clarke M. (2004) would have been seen had a meta-analysis Doing new research? Don't forget the old: of existing research been done before a nobody should do a trial without reviewing new randomised trial began, which would what is known. PLoS Medicine 1: 100-2; and the have led to earlier uptake of effective history of systematic reviews in Chalmers I, et interventions: Clarke M, et al. (2014) al. (2002) A brief history of research synthesis. Accumulating research: a systematic Evaluation and the Health Professions 25: 12- account of how cumulative meta- 37; and Clarke M. (2016) History of evidence analyses would have provided knowledge, synthesis to assess treatment effects: personal improved health, reduced harm and saved reflections on something that is very much resources. PLoS ONE 9(7): e102670. alive. JLL Bulletin: Commentaries on the history of treatment evaluation. Journal of 86 Chalmers I, et al. (2014) How to increase the Royal Society of Medicine 109: 154-63. value and reduce waste when research priorities are set. Lancet 383: 156-65. 77 Allen C. (2014) A resource for those preparing for and responding to natural disasters, 87 Shea BJ, et al. (2007) Development of humanitarian crises, and major health care AMSTAR: a measurement tool to assess emergencies. Journal of Evidence-Based the methodological quality of systematic Medicine 7: 234-7; and Gurevitch J, et al. reviews. BMC Medical Research Methodology (2018) Meta-analysis and the science of 7:10; and Shea BJ, et al. (2017) AMSTAR research synthesis. Nature 555: 175-82. 2: a critical appraisal tool for systematic reviews that include randomised or 78 Oxman AD. (1994) Checklists for non-randomised studies of health care review articles. BMJ 309: 648-51. interventions, or both. BMJ 358:j4008.
79 For a critique of applying research syntheses 88 Smith V, et al. (2011) Methodology in into policy, see Pawson R. (2001) Evidence conducting a systematic review of systematic Based Policy: In Search of a Method. Working reviews of health care interventions. BMC Paper 3. London, UK: ESRC UK Centre for Medical Research Methodology 11:15. Evidence Based Policy and Practice. 89 See, for example: Brennan RJ, et al. (2005) 80 Donnelly CA, et al. (2018) Four principles for Rapid health assessment in Aceh Jaya synthesizing evidence. Nature 558: 361-4. district, Indonesia, following the December 26 tsunami. Emergency Medicine Australasia 81 Lewis S, Clarke M. (2001) Forest plots: trying to 17: 341e350; and Beebe J. (2014) Rapid see the wood and the trees. BMJ 322:1479-80. Qualitative Inquiry: a Field Guide to Team- based Assessment, second edition. Lanham, Maryland, USA: Rowman & Littlefield.
RESEARCH EVIDENCE FOR THE HUMANITARIAN SECTOR | 50 Karl Blanchet, Director of the Health in Humanitarian Crises Centre, London School of Hygiene & Tropical Medicine [email protected] | crises.lshtm.ac.uk Mike Clarke, Research Director of Evidence Aid, Centre for Public Health, Queen’s University Belfast [email protected] | www.qub.ac.uk
Cover Photo © Patrick Brown/UNICEF/Panos Pictures