Real Time Evaluations: contributing to improve learning and accountability1 Riccardo Polastro Abstract The humanitarian community has recently undertaken measures to foster learning and accountability to enhance the quality of its services. Of growing importance are Real Time Evaluations (RTEs), which provide immediate feedback during fieldwork. When effectively conducted, RTEs improve accountability, bridge the gap between conventional monitoring and evaluation and influence timely decision-making. Furthermore, RTEs identify solutions to problems during humanitarian responses. Keywords Accountability, learning, real time, humanitarian Introduction Every year millions of people, often in the world’s poorest countries, need assistance to survive and recover from conflicts and disasters. According to Principles and Good Practice of Humanitarian Donorship (2003), the objectives of humanitarian assistance “are to save lives, alleviate suffering and maintain human dignity during and in the aftermath of man-made crises and natural disasters, as well as to prevent and strengthen preparedness for the occurrence of such situations2” (Good Humanitarian Donorship, 2003). Following the end of the cold war, the volume of financial resources allocated for humanitarian crises has grown exponentially. The most visible and well funded crises in recent history include the former Yugoslavia following its geo-political disintegration, the Rwandan genocide, the Afghanistan and Iraq wars, the Indian Ocean Tsunami, the Haiti earthquake and the Pakistan 2010 floods. Responding to emergencies in developing countries has become a key development, political and security strategic priority. Nevertheless, historically, humanitarian assistance has been subjected to less rigorous and extensive evaluation procedures than development aid, and it has been plagued by poor monitoring (Hallam, 2000). Today humanitarian assistance evaluation is a growing subset of development aid evaluation. Some factors can help us distinguish humanitarian aid evaluation from development evaluation. In humanitarian settings, objectives are fast changing as the situation on the ground is fluid and rapidly evolves 1 This article draws on a presentation made by the author on ‘Lessons Learned from Recent RTEs, which was given at the 26th ALNAP meeting in Kuala Lumpur, Malaysia in November 2010. Powerpoint slides of the talk are available at http://www.alnap.org/pool/files/ia-rtes-alnap-riccardo.pdf. 2 Principles and Good Practice of Humanitarian Donorship were endorsed by Australia, Austria, Belgium, Canada, Denmark, European Commission, Finland, France, Germany, Greece, Ireland, Italy, Japan, Luxembourg, New Zealand, Norway, Portugal, Spain, Sweden, Switzerland, the Netherlands, the United Kingdom and the United States in Stockholm on June 17, 2003. According to these OECD DAC countries, humanitarian action should be guided by the humanitarian principles of humanity, meaning the centrality of saving human lives and alleviating suffering wherever it is found; impartiality, meaning the implementation of actions solely on the basis of need, without discrimination between or within affected populations; neutrality, meaning that humanitarian action must not favour any side in an armed conflict or other dispute where such action is carried out; and independence, meaning the autonomy of humanitarian objectives from the political, economic, military or other objectives that any actor may hold with regard to areas where humanitarian action is being implemented. Humanitarian action includes the protection of civilians and those no longer taking part in hostilities, and the provision of food, water and sanitation, shelter, health services and other items of assistance, undertaken for the benefit of affected people and to facilitate the return to normal lives and livelihoods. (e.g. continuous civilian population displacements, epidemic outbreaks, etc.); planning tends to be poorly documented and baseline information is often missing; when conflict prevails, actors on the ground quickly adopt polarized positions; media and political profile tends to be higher during emergencies, putting strong pressure on national actors; in contexts such as Afghanistan, Chechnya, Colombia, Pakistan, Somalia, Sudan or Yemen, security and access can be highly deteriorated so humanitarian workers and the civilian population are directly targeted during hostilities. Therefore, poor accountability to national stakeholders and aid recipients generally prevails. Over the last 20 years or so the humanitarian community has introduced a number of initiatives to improve accountability, quality and performance3. Codes of conduct, standards, principles, monitoring frameworks and Real Time Evaluations (RTEs) have all been rolled out, and new humanitarian evaluation architecture has emerged, in which RTEs are becoming a central pillar. Figure 1: Focus group discussion with internally displaced persons during an non food item distribution in Jarma Kohat/Paki; Pakistan. © Riccardo Polastro 2010 What is an RTE? An RTE is a participatory evaluation that is intended to provide immediate feedback during fieldwork. In an RTE, stakeholders execute and manage the response at field, national, regional and headquarters levels. An RTE provides instant input to an ongoing operation and can foster policy, organisational and operational change to increase the effectiveness and efficiency of the overall disaster response (Jamal and Crisp, 2002). 3 E.g. Red Cross code of conduct and with Sphere, HAP and People and Aid setting minimum standards and benchmarks as well as ALNAP RTEs are formative evaluations of intermediary results. They can free up operational bottlenecks and provide real-time learning. An RTE is intended to be a support measure for learning in action. RTEs are also improvement-oriented reviews – dynamic tools used to adjust and improve planning and performance. They can contribute to reinforcing accountability to beneficiaries, implementing partners and donors, and can bridge the gap between monitoring and ex-post evaluation. RTEs are, in principle, carried out in the midst of an emergency operation. They are interactive, involving a wide range of stakeholders and therefore contributing to peer-to- peer learning and accountability. Because the results and recommendations are intended to be applied immediately, RTEs must be rapid, flexible and responsive. In contrast, mid-term evaluations look at the first phase of the response in order to improve the second phase, and ex-post evaluations are essentially retrospective: they examine and learn from the past. Using a medical metaphor, one could say that RTEs are done when the patient is still alive, while ex-post evaluations are done more like an autopsy when the patient is already dead. Their respective prognoses and diagnoses are used very differently. RTEs include an instant therapy, while lessons from ex-post evaluations can be used in future programs when responding to crisis. Monitoring in humanitarian aid is often absent and, when it is in place, is not adapted to the changing realities on the ground. An RTE can help bridge the gap as it provides an immediate snapshot that can help managers identify and address the strengths and weaknesses of the response. At the core of an RTE is the idea that stakeholders - particularly those at country level - don’t have to wait for the evaluation report to come out to start discussing how to implement the recommendations. One way to make this happen – as we did in Pakistan in February 2011 – is to use a series of workshops, both at capital and provincial level, as a main feedback mechanism to allow stakeholders to actively participate in framing of recommendations. In turn, this may contribute to stronger uptake and implementation of recommendations to ultimately improve the overall response. RTEs are one of the most challenging types of evaluations because teams are usually fielded within six weeks to six months following a disaster, when agencies are trying to scale up activities. They have a short timeframe, and findings are made available quickly. The inter-agency RTE carried out in Haiti in 2010 (Grunewald et al., 2010) was deployed just three months after the earthquake struck. In these circumstances, the RTE can become burdensome to the agencies involved, and the exercise can suddenly become a ‘wrong time’ evaluation. RTEs also have to be carried out within relatively short periods of time. In general, teams have only two to three weeks to conduct the analysis and make the evaluation judgment before leaving the field. Findings are then fed back for immediate use. RTEs can potentially identify and suggest solutions to operational problems as they occur and influence decisions when they are being made by feeding back aid recipients’ and providers’ views. RTEs can also reinforce the link between operations and policy formulation. This was the case in Mozambique (Cosgrave et. al, 2007), where the RTE examined how the UN humanitarian reforms were being rolled out in the field. A management matrix was implemented and the recommendations were closely monitored by the UN Emergency Relief Coordinator, looking at how Humanitarian Country Teams were applying lessons on UN humanitarian reform (Beck and Buchanan Smith, 2008). Furthermore, in response to the well-documented failings of the international community’s response to the Pakistan floods and Haiti earthquake, in late 2010 heads of UN agencies formally adopted the ‘transformative