Key Messages from ALNAP' S Review of Humanitarian Ac T I

0387 ALNAP KMS cover x 3 AW 12/12/06 4:30 pm Page 2

Key Messages from ALNAP’ s Review of Humanitarian Ac t i o n

For further information please contact the ALNAP Secretariat:

ALNAP at ODI 111 Westminster Bridge Road London SE1 7JD, UK

Tel: +44 (0)20 7922 0300 The ALNAP Review of Humanitarian Action series aims to Fax: +44 (0)20 7922 0399 Contents advance analysis and understanding of current trends in [email protected] humanitarian action, as a means of supporting improvement Chapter 1 — www.alnap.org in sector-wide performance. On an annual basis, the Review Hugo Slim provides critical reflections on current trends through its Global welfare: A realistic synthesis of evaluation findings; and attempts to assess the ALNAP: Active Learning Network for Accountability and expectation for the Performance in Humanitarian Action international strengths and weaknesses of evaluation practice. humanitarian system? This Review of Humanitarian Action is the sixth in the series. Overview 2 Its first chapter provides an overview of what we can Key Messages 4 realistically expect for the international humanitarian system. The synthesis chapter reviews the performance of the Chapter 2 — humanitarian system in 2005 and 2006, and illustrates and Tony Vaux illuminates questions of distortion and proportion within the Proportion and distortion system. This year, rather than looking at the quality of in humanitarian evaluations, chapter three explores the utilisation of assistance evaluations and suggests answers to the issue of why few Overview 7 evaluations are effective at introducing evident changes or Key Messages 10 improvements in performance. This Key Messages summary aims to amplify the main Chapter 3 — findings of this year’s Review for a wider audience. Copies Peta Sandison are available in English, French and Spanish. For these or for The utilisation of the full version of the Review itself (in English only), contact evaluations the ALNAP secretariat at [email protected] Utilisation in the literature 13 Evidence from case studies, interviews and surveys 13 Limitations of evaluation as an agent of change 15 Concluding remarks 16 Key Messages 17 www.alnap.org 2 Key messages from ALNAP’s Review of Humanitarian Action

1 – Global welfare: a realistic expectation for the international humanitarian system?

dependent, above all, on changes in the outer Overview realm of politics where humanitarian aspirations can be enabled, obstructed or distorted. The inner Expectations of international humanitarian aid realm of humanitarian agency practice in which have changed dramatically in the last twenty humanitarian organisations can professionalise, years. Aid budgets and aid organisations have got innovate, coordinate or fail is the second area that much bigger. Humanitarian action has come to determines improvement. Deep problems of occupy a more central place in world politics, and political distortion and perennial problems of a theory of rights rather than charity is now agency performance and practice continue to driving international assistance and protection in compromise a global, impartial and effective war and disaster. If there has been such a humanitarian system. dramatic development in the humanitarian system ALNAP has now been reviewing humanitarian in the last twenty years, what can we expect of the progress for the last five years. As ALNAP and next twenty years? others begin to assess the performance of the Jan Egeland, the outgoing United Nations system in the next five years, it is sensible to step Emergency Relief Coordinator, is clear about what back and ask whether such high expectations are he expects: ‘We need a humanitarian system that realistic within international society or, at least, is able to respond reliably, effectively and how we should calibrate our progress as we move efficiently across the full range of emergencies. ahead to meet them. Humanitarian goals should Now more than ever, humanitarian aid must be certainly be informed by an ideal of some kind but the responsibility of all nations for the benefit of is it wise ever to expect a functioning global all nations.’ Egeland’s expectation is nothing less system for anything that involves such major and than global welfare, and many other competing elements as war, disasters and the humanitarians share his view. Humanitarian world’s fast-emerging environmental crisis? agencies want to get bigger and better because A discussion of international relations is they want to reach everyone in need, and reach important to any discussion of the performance of them well. Whether people are suffering as a the world’s humanitarian agencies because result of mudslides in Central America or vicious international politics is implicit in almost every displacement in Darfur, the humanitarian system page of this year’s Review of Humanitarian Action wants to help them. Global wealth suggests that it (RHA). How we think the world works, or could can, and global morality says that it should. work, will govern what we can reasonably expect These are ambitious expectations for the of global humanitarian action. What we believe humanitarian system and are not without real about the realistic prospects for global order, political, technical and organisational obstacles. international consensus, equitable resource Meeting them will inevitably require a focus on distribution and recognised legal authority two areas – politics and practice. Any overall determines how we will evaluate the current improvements in the humanitarian system are international system of humanitarian action. Global welfare: a realistic expectation for the international humanitarian system? 3

In short, how much we think the humanitarian Any one of these views of international progress system can be changed and improved depends still needs to start with an analysis of the system on what we believe is possible for international as it actually is. So what is the state of the system society. There are three possible views of today? An objective observation of the progress in international society: optimistic, humanitarian system shows that the ‘formal’ cautiously progressive and pessimistic. If we system is really a well-organised Western have an optimistic political view of international initiative sponsored by around twenty states. This society and think the world is now willing and system has an extraordinarily wide reach and is able to deliver impartial and effective legitimised by the international political authority humanitarian aid, then we will judge current of the United Nations, the Red Cross Movement humanitarian action very severely because, as and the governments that fund and accept its aid. this review and all its predecessors show, it is Despite its impressive global reach, the formal obviously neither fair nor efficient. system is built on a narrow, even parochial, If we have a cautiously progressive view of political base. A range of other, informal international society and think that it is gradually humanitarian systems operates alongside the getting closer to a just global order, then we will formal system. Some of these are politically probably be heartened by the steady but imperfect powerful and politically resistant to the formal progress that humanitarian ideals are making in Western system – like the Muslim system or global consciousness and by the practical effect of China and Russia’s bilateral efforts at humanitarian action in the lives of an increasing reconstruction and recovery. Others are locally number of people. All the ALNAP reviews to date powerful and involve frontline, community-based have identified serious problems in the politics responses to crisis. and practice of humanitarian action but they also Humanitarian action and the idea of global welfare talk of progress in several of these areas. for war and disaster is but one of several If we are pessimistic about the likelihood of a international initiatives working to transform the just global order, we may evaluate current ideals, interests and capacity of contemporary humanitarian action rather favourably. We may international society. Others include world trade, be amazed that some governments are poverty, global warming, disarmament, intellectual prioritising it at all, albeit sometimes distorting property and human rights. The reality of global its ideals as they do so, and that so many competition, the emergence of new powers, the agencies are trying to establish humanitarian volatility of politics and the potential collapse of work as an essential international obligation and the environment surely require moderate and a legitimate profession. We would not expect hedged expectations of progress on all these dramatic results from any project working in issues. As a result, humanitarians would be wise these conditions but might praise the almost to have a long-term ideal goal of global welfare, foolhardy spirit of humanitarians who keep on tempered by very practical mid-term objectives working and trying to improve. for realistic improvement. 4 Key messages from ALNAP’s Review of Humanitarian Action

Key message 3: The formal UN-centric Key messages international humanitarian system is run by a small minority of Western states, but other powerful informal humanitarian systems exist Key message 1: Most humanitarian leaders to challenge or complement this Western argue that the time is right for developing a system. global welfare system for war and disaster, Stepping back to survey the formal humanitarian and that sufficient resources and the pressing system from afar, the first thing we see is the environmental crisis make such global attempt by about twenty morally earnest and welfare both possible and imperative. politically driven European and North American Reading between the lines of ALNAP’s five states to develop a rudimentary global welfare reviews so far, and listening to the statements of system that can provide a high-quality safety net humanitarian leaders like Jan Egeland, the for people suffering from war and disaster optimistic view tends to predominate in the around the world. Already, these Western states humanitarian system’s criticism of itself. have achieved remarkable things. Almost every Optimists clearly believe that conditions are now war or natural disaster now receives the riper than ever before for achieving dramatic attention of this nascent welfare system, usually progress in a global humanitarian system. For mediated in some way through the international humanitarian optimists, the unprecedented authority of the United Nations or the Red Cross sufficiency of resources, communications, know- Movement. Significant and routine transfer of how and moral consensus make this a unique resources, equipment and expertise means that moment for realising the ideal of global welfare in millions of people receive help from this system war and disasters. every day.

But, the formal humanitarian system may well be Key message 2: While agency expectations are seen by states and individuals not party to it – obvious, those of aid recipients are not. It is usually by choice more than discrimination – as a still not clear whether the recipients are relatively closed and particularly Western getting what they need, want and expect. system. Many of these other states and Many people are not getting nearly enough individuals obviously prefer to provide assistance and protection, while others are not humanitarian aid through other highly organised being asked if what they have received was political funding systems or looser social useful. A recurrent theme in this year’s Review is networks. These include: a highly developed and that we really have a view of the expectations of similarly partial Muslim aid system; the only the providers in this system. Still, after many significant practice of remittances in cash and years of emphasising its importance, we have kind that can support people in war and only the slightest inkling of the expectations of the disasters; strong but independent bilateral giving recipients of humanitarian aid and protection – by Russia and China; and the importance of local the people who actually suffer from war and frontline humanitarian systems of local rescue, disaster. These people are the real customers or host families and kinship-based support. end-users of humanitarian action. Any business Many people within Western humanitarian or profession knows that it cannot hope to agencies are mesmerised by ‘our system’ and its succeed without knowing what its clients, patients apparent importance, but other political forces or customers actually need and want. If the last may remain untouched by this system or few years have been critical for revealing what infuriated by its pretension. Some very serious humanitarians expect from the system, their powers have no desire to see the Western system expectations must now be compared rigorously become the global welfare system of tomorrow. and routinely with those on the receiving end. Global welfare: a realistic expectation for the international humanitarian system? 5

This has important implications for those who our jobs. Short-contract recruitments mean that have global expectations of the formal system. individual failures are left to fade away with the end of a contract and are seldom addressed Key message 4: Politics continues to dictate the sharply with disciplinary action or dismissal. practicalities of humanitarian action, and so When operations go badly, NGOs can retreat to global humanitarian resources are applied the warm rhetoric of shared struggle while the unfairly. United Nations remains in a world of its own. Much of the UN still shares the grim culture of Political choices by major humanitarian donors, nepotism where the great bureaucrat lives on. strategic decisions by all permanent members of The humanitarian community needs cultural as the UN Security Council, and local obstruction, well as organisational reform – a nettle that has discrimination and favouritism by governments not been properly grasped in the UN reform and armed groups mean that humanitarian action process. continues to be skewed and blocked.

Key message 7: There is not yet a common Key message 5: The five-year trail of ALNAP vision, coherent thrust or overall report from reviews still consistently suggests that, the many important initiatives to improve regardless of politics, agencies could much quality and practice across the humanitarian improve their day-to-day practice. sector. ALNAP’s findings repeatedly confirm not just the Many people in the humanitarian system care same structural outer-realm political flaws in the deeply about its practice. Several common global fairness of humanitarian action but also problem areas (which have been previously and perennial inner-realm problems in its practice and routinely identified in earlier RHAs) are being efficiency. Evaluations of the formal humanitarian very actively addressed in various inter-agency system continue to show how the combined projects and initiatives. The Humanitarian system struggles in highly strategic areas of Accountability Partnership (HAP) and the Inter- operational response. The formal system Agency Working Group are working hard on continues to embody an unbalanced portfolio of agency accountability, particularly to those who sectoral expertise – a situation confirmed by this need and receive aid. The UN reform process and year’s UN Humanitarian Response Review. It its new cluster approach is a well-supported and remains a mystery why, in an apparently serious effort to address perennial problems of entrepreneurial and still unregulated system, so coordination and the optimal distribution of few innovative and dynamic new agencies have resources and expertise. appeared in response to the many gaps and opportunities in humanitarian response. For their part, donor governments are committing themselves to incremental steps towards their Key message 6: The attitude of humanitarian definition of Good Humanitarian Donorship that, agencies is an important part of the problem among other things, takes important account of of practice. The UN’s humanitarian reforms the principle of needs-based impartiality. SPHERE and the many other improvement initiatives continues to consolidate and develop what need to challenge the culture of the system as constitutes good technical standards of well as its organisation and technique. professional work in key sectors, and People in Aid is building a clear sense of good practice in The principle of being there is more important staff recruitment, retention and development – than being successful to the humanitarian one of the chronic weaknesses of the profession. imagination and its ideal of solidarity. Successful practice is seldom rigorously demanded of us. However, there is no single unit charged with a Nothing usually happens to us if we are bad at responsibility to oversee and report on all these 6 Key messages from ALNAP’s Review of Humanitarian Action

separate initiatives, the progress they may be Key message 9: To monitor its progress making in the system and their impact on the against these ten keynote objectives, the ground. Other than through ALNAP, we still have system needs to set up an independent and little idea of how the system performs as whole. effective oversight mechanism with the power We still have no common and simple measures of to evaluate improvements over the next five global need and success. There is still no years. independent office or organisation that has the This oversight mechanism can be the dashboard oversight and mandate to judge, criticise and and control room for gauging success, which still praise the system from outside. does not exist across the humanitarian system. ALNAP could build on its experience to take up Key message 8: The humanitarian profession this monitoring role and assess how the should take a practical five-year view of its collective efforts of the system are progressing. possibilities and performance, and set 10 In addition to continuing its regular realistic and intermediate objectives for retrospectives, ALNAP could then also be more humanitarian progress. forward-looking and measure performance Five years is a practical and achievable horizon against the system’s keynote objectives. But, to for an interim view. An ideal system of global be taken seriously, ALNAP would need some welfare is not imminent, so whether it remains a high-level power behind it. An independent, high- long-term goal, an inspiring hope or highly level performance panel appointed by the system improbable in the minds of different could actively investigate, consider and report on humanitarians is not critical to where the system individual and collective progress and needs to focus its efforts in the next five years. performance. The call for global welfare is an important motivating vision but the system and international Key message 10: A system-wide culture of society are still some way off from such a goal. improvement must be the driver of progress Small steps towards improvement may be more towards these five-year goals, and must mix useful than grand aspirations for the people consolidation, innovation, positive thinking, agencies can actually reach. Focusing on realistic sanction and praise. expectations in this period means concentrating The approach taken to this five-year plan will be on what is feasible. This could include setting 10 critical and must also generate a system-wide keynote objectives for the system, which are a culture of improvement. This must be motivating, mixture of outer-realm political improvements and widely owned and involve praise as well as inner-realm improvements to agency practice. sanction. In particular, any culture of humanitarian improvement needs to be positive, unbureaucratic, empowering and must value a mix of increasing professionalism with continuing discovery and innovation. To encourage this, at least two of the ten keynote objectives should prioritise innovation and risk-taking in humanitarian politics and practice. If, in the next five years, the humanitarian system has only consolidated, it will have failed to take advantage of new ideas and will not have adapted to new opportunities. Proportion and distortion in humanitarian assistance 7

2 – Proportion and distortion in humanitarian assistance

have set a particularly poor example to local Overview NGOs. By failing to distance themselves from donor interests, they may discredit the notion of Humanitarian need may be the main focus for aid independent NGOs and contribute to failure in workers but it is not the centre of attention for state-building. those who provide the resources. Political interest In situations of high media profile, such as the and public sentiment play a massive role in the Tsunami Disaster, evaluations have shown that allocation of aid, and there is little semblance of agencies become preoccupied with speed, and proportionality in the distribution between focus on token successes. Expatriate staff and countries. Media interest, aid structures and management capacity were drawn from other personal interests add further ‘distortion’ at all areas, undermining other responses. In the rush levels. This chapter explores the consequences of for speed and profile, local capacity was distortion in humanitarian aid using evidence bypassed and there was inadequate consultation from 43 evaluations deposited with ALNAP. It with the affected people. This made it difficult to presents an overview of the workings of the progress from relief to recovery. Relations with humanitarian system, and considers the local civil society deteriorated, especially in Sri underlying problems of defining and addressing Lanka where this may be a contributory factor in humanitarian need. the renewed violence. The dominance of the global security agenda has In the most neglected cases, notably the led to widespread use of humanitarian aid as a Democratic Republic of the Congo (DRC), deaths ‘peace dividend’, not only in countries of the occurred on a large scale from lack of resources. highest political interest, such as Iraq and To a limited extent, agencies compensated for Afghanistan, but also in countries such as Somalia lack of external resources by building closer and Sri Lanka where donors try to ‘manage’ relationships with local communities, leading to conflict using a combination of incentive and what may be more sustainable solutions. But threat. Reports note that there is little evidence to imbalances and distortion in the humanitarian show whether such theories are justified, and system cost lives in DRC. Whereas more than question whether donors have sufficient analysis US$7000 was spent to support each person to pursue them effectively. affected by the Tsunami Disaster, a person in Although donors are trying to make their DRC might die for the lack of a few dollars for approaches more strategic in relation to conflict, health services. The global humanitarian system international NGOs have proved reluctant to is responsive to need to only a limited extent. engage in this and, lacking their own analysis, The impact of the media, in particular, is to focus find themselves caught up in such policies attention on spectacular events. This undermines unawares. Security threats have increased where attempts by the humanitarian system to focus on international NGOs are perceived as tools of preventing disaster and addressing ‘forgotten Western interests. In Iraq, international NGOs emergencies’. 8 Key messages from ALNAP’s Review of Humanitarian Action

For international NGOs (and the Red Cross are powerful enough to wield an influence within movement), the Red Cross Code is the ‘industry the humanitarian system – as in the case of standard’ but is breached on a massive scale. Bangladesh. Principle Four enjoins agencies not to act as At the local level, aid actors tend to see themselves instruments of government foreign policy. But as initiating decisions and actions, when in reality since foreign policy is a fundamental element in they may be tools of local interest. Sometimes they the global allocation of aid, and NGOs depend pursue their own interests and preferences, heavily on institutional funding for their muddling rights with needs and overlooking humanitarian work, humanitarian action follows vulnerable groups in the process. For example, the politics. Principle One of the Red Cross Code system has a general focus on women and this can requires organisations to direct their attention to lead to the relative neglect of boys. ‘need’ above all other considerations, and this is endorsed by the Sphere Charter and Common Needs assessment remains the fundamental flaw Standards. But humanitarian organisations have in the humanitarian system. UN Appeals are often yet to make any serious challenge to the distorted used by donors as the basis for aid allocations but system in which they are embedded. bear little relationship to need. They are sometimes little more than a list of what the UN Over the years the humanitarian system has agencies can most easily provide. Remarkably, developed structures that reflect political interest there is neither any accepted method of needs and sentiment rather than need. It has built an assessment nor any detailed method of architecture of agencies that, especially in the comparing global needs on a global basis. case of the UN, divides problems and people in Distortions become apparent only when they are unsuitable ways. This not only confuses the absolutely glaring, as in the case of the neglect of response but also distorts it, notably towards DRC. The humanitarian system is unable to stand food aid and refugees rather than to cash up for or even define the notion of ‘need’ in the distributions and internally displaced persons. allocation of aid. Existing structures create inertia in the system and this makes change and adaptation difficult. ‘Vulnerability’, particularly in the form of The system is better at addressing the needs that protection against threats to security, has been it can most easily fulfil, rather than at assessing seriously neglected. Vulnerability is a better basis vulnerability. As a result it is poor at prevention for assessment than need because it includes and devotes most of its energy to problems that future threats and therefore has a preventive are already past. It draws personnel from all over quality. But the humanitarian system is only the world to address the aftermath of the beginning to define this term and find ways to Tsunami Disaster, when there are few lives to measure and respond to it. ECHO is now focusing save, but neglects unnecessary deaths by the its attention on ‘forgotten crises’ by using a million in DRC. Vulnerability Index. The UN is also seeking to improve its current methods of assessment. More In disaster response, the humanitarian system comprehensive notions, such as human security, still tends to favour relief and to neglect recovery, have yet to be developed and applied. and is particularly poor at supporting livelihoods. Donors tend to jump from relief to physical Many of these problems are recognised but reconstruction without supporting affected people efforts to address them are inadequate. Donors in in their attempts to earn their own living. This the Good Humanitarian Donorship initiative have reflects a lack of responsiveness in the system to endorsed the principle of impartiality in relation to the demands of the affected people. Only rarely need, but have not developed a common are their voices heard, often through local NGOs, mechanism for measuring need or tackling the but this is translated into action only if the NGOs issue of vulnerability. Meanwhile the ‘Global War Proportion and distortion in humanitarian assistance 9

on Terror’ continues to draw attention towards The UN Humanitarian Response Review focuses global security rather than human security and attention on the relationship between different has trumped all other considerations. There is no elements in the humanitarian system. But the sign that global allocations of aid are less distorted underlying problem is the vigour of the top-down in the three years since the Good Humanitarian dynamic, driven by political and media interests. Donorship initiative was launched. Similarly, the The long-term answer is to strengthen bottom-up UN shows no sign of addressing the structural processes that provide better representation for problems that undermine a comprehensive and countries and people affected by disasters. More objective approach to humanitarian aid. immediately, there is a need for better analysis of International NGOs have noted the distortions in conflict situations, strenuous effort to defend the the humanitarian system, but have done little notion of impartiality and a vigorous attempt by about it. all parties to define what the humanitarian system is about. Only then can a concerted effort be made to address issues of proportion and distortion. 10 Key messages from ALNAP’s Review of Humanitarian Action

Key message 3: The humanitarian system is Key messages not properly directed to saving lives.

The humanitarian system focuses on situations that are spectacular and newsworthy, as in the Key message 1: The humanitarian system is case of the Tsunami Disaster, but these may not not based on a common understanding of present a serious threat to human life. The system ‘need’ and does not reflect proportion in the neglects those situations where there is a serious global allocation of aid. risk to human life but very low profile, as in DRC. There is a serious lack of proportion in the Essentially, the process is resource driven, and distribution of global humanitarian assistance. the allocation of resources is not systematic. Some situations are funded almost to excess while elsewhere lives are lost through lack of the Key message 4: Agencies need better methods most basic resources. The total volume of of supporting human security, especially in humanitarian aid has increased considerably and conflict situations. – if properly distributed – could address a much higher proportion of needs. The main problems In the context of the Global War on Terror, the are the influence of political factors, notably the global security agenda has taken precedence over global security agenda, media pressures, human security. There is a need to reinforce the structural problems in the humanitarian system concept of humanitarian protection. International and various local biases and personal NGOs should note that the model of civil society preferences. In the absence of a common they project to local NGOs may have significant understanding of needs, the influence of these implications for governance. factors is not adequately contested. There is a need for caution in the use of incentives and disincentives in conflict situations. Key message 2: In situations of high profile The West’s increasing willingness to intervene in and funding, local capacity may be neglected; other countries has led to a readiness to ‘manage’ in areas of lower funding, the level of conflict though aid interventions. If such engagement with local capacity may be higher. approaches are to be used they should be The extremes could be eliminated by more grounded in more extensive analysis, made proportionate distribution of aid. explicit and carefully monitored for their effectiveness. Given that such approaches are The contrast between the Tsunami Disaster becoming more common, implementing agencies response and the response in DRC is striking. In such as international NGOs should also engage in the former case heavy funding led to poor conflict analysis in order to protect their relationships with local organisations while in independence and reduce security risks. DRC local organisations had to be involved because international agencies and staff had such a limited presence. The response to the Tsunami Key message 5: Address the deficiencies of the Disaster was relatively wasteful, while in DRC Consolidated Appeals by establishing indices there were unnecessary deaths. of need that take into account vulnerability and human security.

Currently there is no common method for assessing and comparing humanitarian needs on a global basis. This makes it difficult to identify the impact of political, security and media influences. Without such a method the Proportion and distortion in humanitarian assistance 11

humanitarian system is unaccountable to both The ‘Collaborative Approach’ of the UN in relation those who provide the resources and those who to IDPs should not undermine the case for strong receive them. In collaboration with the UN and leadership within the UN in order to overcome its other humanitarian actors, donors should seek inherent distortions. A vigorous style of to establish a common understanding and leadership based on representation of method that reflects notions of vulnerability. A humanitarian interests will be better for the human security approach should be given system as a whole. serious consideration. The Good Humanitarian Donorship initiative may be a useful focus for Key message 8: International NGOs and the such activity, especially because of its Red Cross movement should use the principles commitment to impartiality. of the Red Cross Code and Sphere to uphold humanitarian values in the face of distortion. Key message 6: Focus greater attention on the Codes and principles are an essential defence recovery phase of disaster response. against distortion. The Red Cross Code and In disaster response, donors, the World Bank and Sphere Common Standards are the best available other humanitarian actors should focus more protection against such influences and the nearest attention on the ‘transition’ or recovery phase. there is to a statement of ‘rights’ for beneficiaries. This tends to be neglected. In some cases, this Sphere acknowledges that, although there are may be because of the division between aid many different forms of accountability, ‘our structures focused on humanitarian response and fundamental accountability must be to those we those focused on development. In particular, seek to assist’ (Sphere Project, 2004, p 19). In recovery through livelihood support requires a current times this needs to be actively asserted. higher level of funding and should be recognised Agencies should join together in order to tackle at an early stage. While this activity may be the distortions of the global humanitarian system undertaken by NGOs, adequate funding from in which they are embedded. Otherwise they will donors is still necessary. fail to comply with the first Red Cross Code principle, the ‘humanitarian imperative’, which Key message 7: Minimise the problems arising requires that need comes first. from structural imbalances in the International NGOs and the Red Cross humanitarian system. movement should use advocacy to develop Ideally the UN system of division into separate pressure for a more equitable humanitarian agencies should be re-examined and better system. They should engage with donors in the structures introduced. Failing this, donors should difficult task of defining humanitarian need. They ensure that UN Appeals do not represent vested should also help the media to develop an interests but reflect actual needs. The bias understanding of the issue of global proportion towards food aid should not be allowed to and distortion in humanitarian aid. undermine the case for cash distributions and other non-food approaches where these are more suitable. The separation of populations into groups designated as either refugees or internally displaced persons (IDPs), for example, should not be allowed to add further distortion to the humanitarian system. It is particularly important in such cases that ‘need’ is construed in terms of ‘vulnerability’. 12 Key messages from ALNAP’s Review of Humanitarian Action

Key message 9: Develop mechanisms to ensure Key message 10: Use research to minimise the that ‘accountability to beneficiaries’ takes effects of imbalances in institutional, local precedence over ‘accountability to donors’. and personal interests.

International NGOs and the Red Cross movement The humanitarian system has not been good at have faced strong criticism from evaluations of questioning itself. This allows institutional and the Tsunami Disaster response for failing to personal interests to persist. Research has played establish adequate mechanisms for consulting, an important role in questioning the dominance of informing and involving the people affected by food aid in the humanitarian system. It also helps humanitarian disasters. This repeats a criticism reveal the influence of local political interests on made in many other evaluations over a long humanitarian aid. Agencies should make better period of time. Such agencies should define their use of research, especially in cases where biases role in relation to donors as representing the arise from humanitarian structures or local interests of ‘beneficiaries’. In the current debate interests. They should also recognise that focus about the definition of ‘need’, agencies should on a particular group or issue, such as refugees seek to ensure that beneficiary views are properly or women, may lead to the neglect of others, such represented. This may involve wider use of as internally displaced persons and boys. Peer ‘public opinion’ mechanisms such as beneficiary review may offer an informal mechanism for surveys. They should also ensure that their own questioning local biases. Evaluation also has a activities are more accountable by conducting role to play but must be designed to examine the ‘public audits’ for all humanitarian activity. wider context and ask fundamental questions. International NGOs and the Red Cross movement There is a need to seek out unintended negative should also publish more of their evaluations. consequences systematically, through research. The utilisation of evaluations 13

3 – The utilisation of evaluations

One of the few findings common to all of the Overview studies reviewed is that how, and why, an evaluation is carried out significantly affects the ALNAP commissioned this study because, while likelihood of it being used. Several factors evaluations are generally successful at identifying promote (or inhibit) utilisation. As might be lessons and for institutional memory building, expected, many of the factors relate to the quality only very few evaluations are effective at of the evaluation. But utilisation is not simply introducing evident changes or improvements in related to the credibility and standard of research performance. If this continues, there is a danger and documentation of an evaluation. It is also that continued poor utilisation will undermine the strongly related to the process. The degree of credibility of evaluation as a tool for accountability participation by all the key stakeholders before, and learning in the humanitarian sector. during and after the evaluation, and the extent to which the purpose and approach are designed with the intended use and users in mind, are equally if not more important determinants of Utilisation in the literature utilisation. A serious intention to use an evaluation is indicated by careful planning, The picture of utilisation that emerges from other adequate time and appropriate mechanisms for studies is complex. Utilisation means different follow-up that suit and serve the users. things to different users, and is highly dependent The best evaluation, however, is only as useful as on context. Some studies describe a fairly dismal the organisational and political context allows. Its record of use; others are more upbeat. In all utilisation is greatly affected by the presence of a cases, utilisation is far less predictable than positive culture of learning and accountability, the generally anticipated. absorption capacity of individuals involved and There are also very different kinds of utilisation. the quality of the relationship between the It is relatively rare for users to implement an evaluators and the users. evaluation’s recommendations directly. It is far more likely that an evaluation will be used indirectly – to stimulate debate, help win or lose an argument, add to an organisation’s store of Utilisation in practice: evidence from case knowledge and enhance the understanding of studies, interviews and surveys the participants. One evaluation can lead to a mixture of uses as well as partial or selective The findings from this study largely corroborate use. Evaluations may also be commissioned as a those in the literature. They suggest that when an kind of ritual, ticking an administrative or public- evaluation is planned and conducted in ways that relations box without any real commitment to enhance participation, ownership and high use the results. 14 Key messages from ALNAP’s Review of Humanitarian Action

relevance to the users’ needs, it is far more likely • The users are overwhelmed by an increasing to be useful and therefore used. The ‘utilised’ number of (upwards) accountability demands case-study evaluations tended to exhibit many of or a generally high workload, exceeding their the positive factors described above, patience and absorption capacity. demonstrating that it is feasible and worthwhile to • The evaluation community lacks awareness of invest in a pro-use approach. utilisation-focused practice. These findings also suggest that the adoption of • The formal accountability role of evaluation use-promoting approaches is still in its infancy. demands rigour, common standards and For example, last-minute planning, rushed independence. This may have led to reluctance recruitments and the failure to allocate sufficient to engage in the context-specific, collaborative time for the evaluation appear to be common, and and adaptive approach required by a focus on may be a more significant constraint to utilisation use and users. than is recognised. The sector is still learning how to establish meaningful and dynamic mechanisms for follow-up. There are some good examples of The sheer number and geographic distribution of innovative approaches but generally little stakeholders that characterise humanitarian evidence of adapted and creative dissemination response mean that a user-based approach to methods designed to suit different stakeholders. evaluation demands far more time and The norm continues to be a standard report and engagement than is commonly allocated. The real executive summary. or perceived imposition of an evaluation by one Too many evaluations are implemented without group of stakeholders on another can be the the adequate participation of the primary death-knell of utilisation and, according to the intended users. The purpose of the evaluation is findings here, may be quite common. Serious not always clear, and evaluations may not be participation and a far greater focus on the designed with that purpose clearly in mind. Most intended users would also help to expose the evaluations have more than one stated purpose practice of inappropriate or ritual evaluation and or intended use, such as a mix of field and prevent evaluation further contributing to the organisational learning and accountability to a current mistrust and saturation in the sector. It donor. In some cases, these undermine or implies a shift from quantity to quality – and displace each other. fewer evaluations.

According to many users, evaluation reports are There is an argument for experimenting with often too long and use inaccessible language, approaches that reduce the artificial division particularly evaluation jargon. The quality of the between evaluators and potential users. recommendations is often an issue. Users Evaluators (and evaluation managers) are complained that these are not sufficiently feasible, regarded by some as remote from the operational specific, constructive or relevant. realities of humanitarian work, overly concerned with the independence of the evaluation, prone to Since information on how to optimise utilisation is jargon and protective of their own reputations. For clearly available, why isn’t the humanitarian example, with the intention of preserving the sector consistently adopting a pro-use approach? independence of the report, it is common for There appear to be several common obstacles. evaluators to produce recommendations in relative isolation. Yet the recommendations are not the findings. They are essentially proposals • Too many evaluations are commissioned for made by the evaluators and designed to remedy symbolic or routine reasons, without serious weaknesses or build on strengths identified consideration being given to their utilisation or during the evaluation. One of the case studies to meaningful ownership by key users. The utilisation of evaluations 15

demonstrated that the relevance and feasibility of organisations is not conducive to translating the evaluation recommendations can be greatly learning generated by one part of the organisation enhanced through a collaborative process that into a cross-departmental or sector-wide learning takes account of the organisation’s or process. Or vice versa. programme’s context and constraints without Whether or not policy is changed in response to distorting the findings. evaluations often depends on the convictions of The nature of traditional evaluation itself can also individual stakeholders. The management inadvertently predispose evaluators to be critical, structures and culture of aid agencies are typically rather than constructive. The commercial nature ‘loose’ and decentralised; the translation of new of the relationship with external evaluators may policy into practice is heavily constrained by high also be an issue, particularly in smaller staff turnover and a pragmatic reliance on organisations and with NGO field personnel who relatively autonomous field staff. Translating are more likely to regard consultancy fees as policy into action in the field can be difficult. excessive and the work as mercenary. It is also This study has found little evidence of evaluation relatively rare for evaluators to be involved in leading to penalties or being used as a force for follow-up or to receive any feedback on how their change by external stakeholders such as donors evaluation has been utilised, raising questions or the public. In fact, this is noticeable by its about how evaluators can learn to optimise the absence. Evaluations are regarded as potentially usability and general quality of their work. damaging to an organisation’s reputation and funding and the fear of repercussion from publicised criticism is certainly very real. However, a little chastisement aside, most studies Evaluation in context: limitations of conclude that the repercussions for an evaluation as an agent of change organisation of a critical evaluation are limited or even non-existent.

The effectiveness of evaluation as a means to Nor are humanitarian and development enhance performance is only partly controlled by organisations driven by the power of the its quality and process. This means that, even if customer as a force for change. The the evaluation community gets better at applying questionnaire responses showed that it was very a utilisation-focused approach (as it clearly unusual for the key findings of an evaluation to should), there are real limits to the extent of its be communicated to the programme’s impact. There are many competitors for an beneficiaries in the field. The absence of the organisation’s attention, and evaluation often beneficiaries’ involvement and voice in the loses out to other priorities. Most of those evaluation process means that evaluation findings interviewed said that evaluation was important, cannot be used by them as a means to lobby and but was only one of several influences. It was hold an agency to account. generally given a middle ranking in terms of its value to decision-makers. The utilisation of evaluation, and any other learning tool, is essentially a study of the nature of Organisational imperatives such as the need to change. This is not under the control of any one maintain profile, compete for funds and fulfil person or process. The more complex the use, the various bureaucratic imperatives can and do over- more that this is so. Arguably, the evaluation is ride an organisation’s commitment to learn and but a catalyst, influencing the beginning of a enhance its performance. The humanitarian process and increasingly ceding control to the sector is regarded by some as increasingly risk- impact of other actors and other forces. The averse and its principles progressively buried by evaluation community has the responsibility to protocol. The organisational structure of many aid ensure that it does all it can to facilitate utilisation, 16 Key messages from ALNAP’s Review of Humanitarian Action

but it is not responsible for the actions and It is critical that the job we expect evaluation to commitment of the users. do is realistic, better differentiated from other objectives and designed accordingly. It may not necessarily be the most cost-effective approach for certain uses. It is not, despite its reputation Concluding remarks as a tool for upwards accountability, widely used to hold organisations to account. Learning, This study began with a concern that the another primary objective, is highly dependent credibility of evaluation could be undermined if on genuine participation – which is not its poor record of influencing humanitarian necessarily a feature of many broad-scope and performance continues. In some ways then, the time-poor evaluations. findings indicate a less gloomy outlook; the While the evaluation community can do a lot pessimism partly results from a narrow more to promote the utilisation of evaluation, the perception of utilisation that does not do justice main priority appears to lie in improving our to the rich and often indirect use and influence understanding of how to generate learning and of evaluation. organisational change, adopting a range of The overall power of evaluation as an agent of creative and innovative tools to suit this goal. change is relatively modest. The job we expect Evaluation should be just one tool within a range evaluation to do is often unclear and over- for use in learning and accountability; it will ambitious. Evaluation appears to have become a sometimes be the right tool, sometimes not. catch-all for a range of accountability and learning Either way, it cannot do the job alone. objectives that may or may not be compatible. The utilisation of evaluations 17

would adapt to emerging needs, be less risk- Key messages averse and more creative. The role of the evaluator would also change. If the focus of the evaluation is on ‘softer’ objectives such as Key message 1: The most common forms of learning, then the evaluators and users should be evaluation utilisation are indirect and often in partnership, pooling experience and skills in intangible. the service of improved performance. The uses of evaluation can be rich, diverse and Interpersonal skills for communication, highly specific to organisations and individuals. facilitation and management of organisational The expectation that there should be a neat and change would become more important. linear link between evaluation recommendations External drivers for change could be and subsequent action means that many types strengthened. Audits may better serve the ‘hard’ of utilisation are not recognised. Utilisation objectives of upwards accountability, linked to should be defined in more qualitative terms that regulatory bodies designed for the purpose. reflect the users’ perceptions of use, and the Participatory and impact evaluation designed with, degree to which they have seriously engaged in and for, the affected communities would the process. strengthen downwards accountability.

Key message 2: The learning gained from Key message 4: The evaluation community can evaluation is complex and only partly related significantly increase the utilisation of to the evaluation. evaluations by focusing on use and users.

Utilisation is only partially conditioned by an The general focus of evaluation should now shift evaluation’s quality and process. The process of fundamentally away from methodology and utilisation mirrors the complex nature of change; towards utilisation. There is plenty of pro- it is transformed by the users themselves, is utilisation information and advice available for strongly affected by context and is often evaluators and evaluation managers but its unpredictable. As users appropriate an adoption is inconsistent. Evaluators and users evaluation, their influence on its outcome may not be aware of the examples of good progressively increases, while the evaluation practice, dispersed among numerous studies or itself becomes less significant. The final ‘uses’ described in hefty publications that practitioners may no longer be those predicted or proposed by may not find time to read. There could be value in the recommendations. creating easy-to-use, practical and brief guidelines. Key message 3: The approach to evaluation ALNAP’s Quality Proforma could place far greater should be more creative and adapted to emphasis on assessing the process, rather than diverse needs. content, of an evaluation, monitoring use- Focus on utilisation means that the methodology promoting factors and including participation as a of an evaluation could significantly depart from primary performance criterion. This would mean what is usually viewed as a credible. Evaluation widening the criteria for inclusion in ALNAP’s could become more strategic, and not necessarily database, and continuing from the encouraging tied to its roots in social research. The approach step already taken to interview evaluators and users as part of the assessment. 18 Key messages from ALNAP’s Review of Humanitarian Action

Key message 5: More can be done influence that leads to decision-making. Careful institutionally to close the gap between differentiation between changes that are clearly evaluation and evaluation users. within the control of the sector and those derived from broader political and structural factors Insufficient attention is paid to the impact of would increase the credibility of evaluation and relationships and networks. Evaluators and enhance its use. managers need to generate greater trust and bridge the gap between users and evaluators. More open-minded debate about the real latitude Evaluation practitioners should make concerted of the humanitarian sector to address recurring efforts to be constructive and realistic, to issues would be helpful. This may support the demystify evaluation and resist jargon. Using identification of relevant findings targeted at evaluations is as much a people issue as it is a relevant (senior) decision-makers or politicians technical one, and perhaps more so. and reduce the number of recommendations aimed at an absent audience.

Key message 6: The decision to evaluate This study has not gained much insight into the should be the result of a considered and most influential factors and forces that generate negotiated process involving the key change at either an organisational or individual stakeholders and especially the intended level. This is partly because it does not seem to be users. a topic of enquiry among the strategy-makers in For the evaluation community to enhance the sector. There may be real value in agencies utilisation, a serious intention towards use must conducting what are referred to as tracer studies, be a precondition for the users. If a significant which start with a change and then work number of evaluations are being conducted as a backwards to identify the factors that led to it. type of administrative ritual, then resources and enthusiasm for learning are being diverted and Key message 8: Evaluation is only one of undermined. Decentralising evaluations to the several tools for learning accountability. field, or delegating the management of an The impact of evaluation is enhanced, if not evaluation to the agency in question are enabled, by being an integrated part of a broader promising approaches to increase ownership. menu of approaches to enhancing performance. Stakeholders, including the evaluators, have the Monitoring, for example, remains a poor cousin of right to question intended use, clarifying who the evaluation and has yet to receive equal attention primary users really are and whether the financial from decision-makers. There are a number of and opportunity costs of an evaluation will be existing approaches (such as peer and justified. Quality is more important than quantity. participatory review, community-based monitoring and evaluation, and communities of Key message 7: The ability of evaluation to practice) that are promising candidates for more effect change in the context of powerful established inclusion in a range of methods. obstacles to organisational learning and other drivers is relatively modest.

Great store is placed on evaluation as the main accountability and learning tool of the humanitarian sector. At the same time, the findings suggest that evaluation plays a modest role in the broader stream of information and The utilisation of evaluations 19

Key message 9: Organisational and individual Key message 10: The focus should be on the change requires adequate time to absorb. translation of evaluation-based learning into reliable performance in the field. No learning instrument will generate change unless the sector finds the time and space to Many systemic performance issues have already allow it. There should be far more attention to the been identified by evaluation. It is surely a priority increasing problem of heavy workload and the now to focus energy on field learning. The recent apparent inability of the sector to prioritise. Part of interest in learning innovations drawn from the range of learning tools could simply entail outside the sector (such as mentoring and giving staff the time to absorb information and the coaching) is promising. The participation of field opportunity to consider what is important rather personnel in the development of better learning than urgent, and to generate new ideas outside tools would greatly enhance their impact. the usual parameters of problem-solving. 0387 ALNAP KMS cover x 3 AW 12/12/06 4:30 pm Page 2

Key Messages from ALNAP’ s Review of Humanitarian Ac t i o n

For further information please contact the ALNAP Secretariat:

ALNAP at ODI 111 Westminster Bridge Road London SE1 7JD, UK