External Evaluation ACF International’s Emergency Response to the Pakistan Floods in August 2010 Andy Featherstone, July 2011

!is report was commissioned by Action Against Hunger | ACF International. !e comments contained herein re"ect the opinions of the Evaluator only. Photo Courtesy of Richard Mosse Evaluation of ACF’s Emergency Response to the Pakistan Floods iii

Evaluation of ACF’s Emergency Response to the Pakistan Floods Andy Featherstone, July 2011

EXECUTIVE SUMMARY

1. Introduction

The floods which swept across Pakistan between late-July and September 2010 are considered to be one of the most significant disasters of modern history. Although the death toll was relatively small (estimated to be 1,950 people) the geographic extent of the damage they caused was staggering; affecting over 20 million people, it was the largest disaster ever recorded in terms of affected area, affected people and houses damaged1 and at one stage covered one-fifth of Pakistan’s land mass.

The International humanitarian response was all too often characterised as far too late to be considered life-saving with only a few exceptions and can best be described as providing a second wave of support. While this asks important questions about the capacity of the International humanitarian system, lack of funding in the early stages of the response and the deployment of large numbers of agency staff to Haiti all played an important role in delaying the response.

The initial ACF response to the floods in KPK and Sindh Province took time to scale up and despite a small presence in the country the organisation struggled to recruit staff with the right skills at the right time from across ACF International. Despite this a strengthened team has seen the recovery programme grow to over $9m and deliver important services to those affected by the floods.

2. Scope of the Review & Methodology

Six main themes are outlined in the ToR2 which are consistent with the ACF International Emergency Response Learning Project3 and which provided the basis for the evaluation; (i) needs assessment and strategy development, (ii) communications and coordination, (iii) human resources, (iv) logistics, (v) administration and finance, (vi) monitoring and evaluation. The period under evaluation starts from the onset of the floods in late-July until the end of January 2011.

In addition to an extensive literature review, discussions were held with ACF International members in New York, Paris and London and telecons were held with senior staff from Madrid and Montreal over the phone. Contact was made with former ACF staff deployed to Pakistan and several interviews were conducted. With 11-months having passed since the floods, rather than focus on the operational programme the evaluation focused instead on strategic decision-making and direction, programme support and internal coordination with the aim of contributing to internal organisational learning. While the ToR spoke of an evaluation of the response, in the absence of field work the process was more akin to a management review and this report reflects this.

3. Towards an Assessment of the ACF Response

1 Polastro R et al (2011) Inter-Agency Real-time Evaluation of the Response to Pakistan’s 2010 Flood Crisis, DARA 2 The full ToR is reproduced in annex 3. 3 The project is founded on the belief that the key to more successful emergency responses lies in ACF International’s capacity to critically evaluate its standard operating procedures during emergencies. The aim of the project is to do precisely that; to identify lessons learned from rapid-onset emergencies, and to facilitate their integration into future ACF International responses. In doing so, the project aims to contribute to ACF International’s preparedness efforts, and to provide a framework for futures evaluations that ensures coherent and consistent learning in the organisation.

Evaluation of ACF’s Emergency Response to the Pakistan Floods iv

Needs Assessment & Strategy Development Despite a small presence in the country the process of launching assessments and initiating humanitarian response was laboured. While a modest programme was launched in KPK it took a number of weeks to scale-up in Sindh. The lack of an over-arching, geographic or sectoral strategy to guide programme response made it difficult to mobilise resources. While a proportionate recovery programme was launched in the WaSH and FSL sectors which delivered appropriate programmes through innovative coordination it will be important for ACF to use the opportunity to strengthen preparedness planning.

RECOMMENDATIONS: . A programme strategy document is necessary for guiding programme and mobilising ACF’s resources. It is absolutely essential that the revised EMS framework which stipulates the submission of an International appeal within 24-hours of an emergency, a 1-month strategy within 5-days and a 6-month strategy in the first 2-months is followed; . While an outline of strategic intent is important it cannot provide organisational reassurance that timely and effective programmes are being delivered in the field. To fill this gap it is recommended that an assessment or RTE, preferably independent of the Managing Member is conducted within the first 6- weeks of the response being launched; . One of the key weaknesses for ACF in Pakistan was the lack of preparedness planning. In countries where ACF has a strategic interest in responding to emergencies it is recommended that a preparedness plan which scopes out potential humanitarian needs, defines geographic and sectoral areas of response and which identifies resources for response (both human resource and materiel) would provide the best possible platform to launch a timely response;

Communication & Coordination While the time it took to commence operations made the swift deployment of a communications officer impossible, there was limited communications material available in the initiate weeks after the floods which hampered efforts to engage donors and supporters. The revised EMS document provides much-needed guidance in this respect.

RECOMMENDATIONS: . Communication needs must be viewed through the lens of ACF International as a while and resourced accordingly. The revised EMS stipulates that a decision about the deployment of a Communications person must be made in coordination with other members within 24-hours of an emergency which is strongly endorsed by this evaluation; . Given the frustrations felt by many members at the lack of material made available it would make good sense to prescribe in more detail a set of minimum standards for communications in emergencies which could include a one-page outline of context and key messages within 48-hours, the designation of a spokesperson within 48-hours and a set of expectations around regular updates. A daily/weekly communications conference call between Network members and the programme team in the first month of the response would ensure that expectations are managed and that the combined resources of ACF can be used to support communications work; . The failure of ACF to implement humanitarian response according to the EMT suggests that the document has fallen into disuse. The revised EMS provides a set of minimum requirements for disaster response and a timeline for their delivery. It also strengthens accountability and provides a mechanism for resolving disputes. This tool potentially provides a strong platform for a coordinated response but it is only as strong as ACF’s commitment to abide by it. It is recommended that the document is circulated throughout the members and that key staffs performance objectives are revised in light of it; . Where the managing member lacks capacity in a particular area of a response (comms, operations, HR), this must be communicated to members in a timely manner with a view to delegating responsibility to a supporting member; . To foster greater transparency, to (re)build trust and to guide decision-making about the deployment of emergency pool and roster staff, a system of prioritising emergencies (using criteria such as geographic scale of crisis, number of people affected etc.) is recommended with a commitment for members’ decision-making on deployment of resources to reflect this;

Evaluation of ACF’s Emergency Response to the Pakistan Floods v

. While the coordination structures which have come about as a result of Humanitarian Reform are time- consuming, they are also the means of doing humanitarian business and as such ACF needs to continue to prioritise participation in them. In recognition of this it will be important to ensure that humanitarian programmes are adequately staffed to meet both operational and field coordination needs.

Human Resources That the team was able to scale up from a team of 4 International staff and a handful of national staff to 26-internatioanls and several hundred nationals is an impressive achievement. The responsibility for scaling-up the team in the early days of the response was a shared task and ACF International failed to find sufficient staff with the right skills at the right time. While time has seen the team considerably strengthened it will be necessary to clarity procedures and strengthen mutual accountability if similar problems are to be avoided in the future. Adherence to the revised EMS will be an important part of achieving success in this.

RECOMMENDATIONS: . The decision to make management changes in the Pakistan programme took too long. Where ACF has a long-term programme in a country it is strongly recommended that discussions are held with the Country Director about the size of programme that they have the capacity to manage with a ‘step aside’ policy initiated if a disaster occurs that requires a larger response; . There was a lack of clarity about HR needs. It is essential that an HR plan is devised at the earliest opportunity and communicated across ACF International. The revised EMS stipulates that an initial HR plan is drafted within 7-days. It is recommended that Members start checking their rosters and clarify the status of their emergency pool members within 24-hours of an emergency so that tangible offers of support can be made within 48-hours of a crisis occurring; . There are a number of strategic HR recommendations for ACF-US including the need to (i) strengthen the humanitarian skillset of the ACF-US roster, (ii) increase the number of experienced staff deployed to decrease the proportion of first missions, and (iii) increase the average length of deployments. With the HR team in New York in flux it is recommended that these issues should be high on the agenda of the incoming HR Director; . Once the ACF-US Desk Officer had been deployed to Pakistan there was a bottleneck which constrained the flow of communications and information between ACF International members and the field team. It is recommended that when a managing member assesses staffing capacity after an emergency strikes that it takes into account needs at headquarters in addition to the field.

Logistics ACF delivered its humanitarian and recovery response in Pakistan despite a weak logistics team in the field. While the team has now been strengthened the organisation needs to strengthen its ability to deploy experienced staff at the outset of a crisis.

Recommendations: . The inability of ACF-US and members more broadly to find logistics staff with the right skills at the right time slowed the speed of the response and placed the organisation at considerable risk. ACF needs to promote a zero tolerance approach to under-resourcing such essential parts of the programme and should determine a minimum level of risk beyond which a crisis management approach is adopted to securing the necessary resources.

Administration, Finance & Funding While the recovery response was successful in securing significant donor funds, ACF-US was initially constrained in its ability to secure unrestricted funding to establish a presence and seed programmes. ACF International needs to be far more successful in facilitating access to unrestricted income and in identifying strategic fundraising priorities.

Recommendations: . The approach taken by ACF-US to scaling up the programme was cautious. A culture which supports managed risk-taking is a prerequisite for an organisation that prides itself on being a first-responder to a

Evaluation of ACF’s Emergency Response to the Pakistan Floods vi

crisis. With the ACF-France unrestricted funding facility now part of the EMS and a greater commitment to early communication more likely to yield public donations it will be important for ACF to be more ambitious in their humanitarian response; . While the revised EMS provides an operational coordination forum it misses an opportunity to heighten fundraising and marketing coordination across Network members. By pooling talent and resources ACF International will be far better positioned to focus its support and benefit from cross-Network collaboration. While this currently happens informally it is recommended that a formal process is outlined in the EMS which seeks to determine priorities, fill capacity gaps and provide timely support to fundraising.

Evaluation of ACF’s Emergency Response to the Pakistan Floods vii

Table of Contents

Acknowledgements ii Cover Photograph ii Executive Summary iii Table of Contents vii Acronyms viii Pakistan Flood Extent (date: 07 Sep 2010) ix

1. Introduction 01 1.1 The Impact of the Floods on Life, Livelihoods and Vulnerability 01 1.2 The Humanitarian Response to the Floods 02 1.3 The ACF International Response 03

2. The Scope of the Evaluation & Methodology 04 2.1 The Scope of the Evaluation 04 2.2 Methodology 04 2.3 Limitations 04

3. Needs assessment and strategy development 05 3.1 Phase 1 05 3.2 Phase 2 07 3.3 Recommendations 09

4. Communications and coordination 09 4.1 Phase 1 09 4.2 Phase 2 13 4.3 Recommendations 14

5. Human resources 15 5.1 Phase 1 15 5.2 Phase 2 17 5.3 Recommendations 18

6. Logistics 18 6.1 Phase 1 18 6.2 Phase 2 19 6.3 Recommendations 20

7. Administration, finance and fundraising 20 7.1 Phase 1 20 7.2 Phase 2 21 7.3 Recommendations 22

8. Monitoring and evaluation 22 8.1 Phase 1 & 2 22

Annexes (Separate Document)

Annex 1 Review Participants 24 Annex 2 Terms of Reference for the Review 25 Annex 3 Evaluation Schedule 28

Evaluation of ACF’s Emergency Response to the Pakistan Floods viii

Annex 4 Methodology Matrix 29 Annex 5 Pakistan Floods – Timeline of Key Events 32

Evaluation of ACF’s Emergency Response to the Pakistan Floods ix

Acronyms

ACF Action Contre La Faim ADB Asia Development bank CFW Cash for work CIDA Canadian International Development Agency DEC Disasters Emergency Committee DFID Department for International Development DNA Disaster Needs Assessment DRR Disaster Risk Reduction ECHO European Commission Humanitarian Office EMMA Emergency market Mapping and Analysis EMS Emergency Management System EMT Emergency Management Team FSL Food security & livelihoods GDP Gross Domestic Product IEC International Executive Committee IDP Internally Displaced Person IRC International Rescue Committee KPK Khyber Pukhtunkhwa [Province] McRAM Multi-Cluster Rapid Humanitarian Needs Assessment NDMA National Disaster Management Authority NFI Non-Food items OCHA Office for the Co-ordination of Humanitarian Affairs PEFSA Pakistan Emergency Food Security Alliance PIFERP Pakistan Initial Floods Emergency Response Plan RTE Real Time Evaluation UN United Nations WASH Water, sanitation, hygiene promotion WB World Bank WFP World Food Programme

Evaluation of ACF’s Response to the Pakistan Floods ix

Evaluation of ACF’s Response to the Pakistan Floods 1

1. Introduction

1.1 The Impact of the Floods on Life, Livelihoods and Vulnerability The floods which swept across Pakistan between late-July and September 2010 are considered to be one of the most significant disasters of modern history. Although the death toll was relatively small (estimated by Pakistan’s National Disaster Management Authority, the NDMA to be 1,950 people) the geographic extent of the damage they caused was staggering; affecting over 20 million people, it was the largest disaster ever recorded in terms of affected area, affected people and houses damaged4 and at one stage covered one-fifth of Pakistan’s land mass.

The impact in each of the affected Provinces was different depending on its topography and location. While towns and villages in KPK were devastated by flash floods which lasted days, the cumulative effects of the flood increased as the waters headed south with the floods lasting weeks in Punjab and months in Sindh.

Overall, the economic damage caused by this disaster has been estimated at some 10.1 billion USD, or 5.8% of GDP. Immense infrastructure losses were registered including 2.9 million households of which 1.9 were severely affected or completely destroyed; livelihoods were disrupted with 80% of food reserves estimated as having been lost. As a result of the floods prices were dramatically driven up while the affected population’s purchasing power fell.5

Coming 5-years after the massive earthquake in Kashmir which was estimated to have killed 75,000 people, 4-years after the floods in Balochistan and Sindh Provinces which affected over 2.5 million people and displaced 300,000, and just a year after the army’s offensive against the Pakistan Taliban which displaced over 2-million people there is little doubt that Pakistan is one of the most significant contemporary global disaster hotspots.

Box 1: The Changing Picture of the Needs of Flood-affected People in Pakistan

4 Polastro R et al (2011) Inter-Agency Real-time Evaluation of the Response to Pakistan’s 2010 Flood Crisis, DARA 5 These estimates are based on the findings of the ADB/WB (2010) Preliminary Damage & Needs Assessment, May 2010

Evaluation of ACF’s Response to the Pakistan Floods 2

1.2 The Humanitarian Response to the Floods An analysis of humanitarian appeals for Pakistan flood victims provides some indication of the timeliness of the International humanitarian response. Oxfam launched its Pakistan flood Appeal on 03 August followed a day later by the UK’s Disaster’s Emergency Committee (DEC). The UN appeal for $459m was launched on 11 August 2010 and within a month was 90% funded. However analysis undertaken by Oxfam suggests that funding was slow in comparison to other International disasters, with only $3.2USD for every affected person released within the first 10-days.6

The Inter-agency Evaluation notes the slow pace of joint assessments and the limited utility of single-agency assessments which often elicited incompatible data sets and were often out-of-date before the results were published due to the rapidly changing situation. The humanitarian programme that resulted was considered by many to be patchy with humanitarian assistance concentrated in some areas while other areas went without support. While some consider this a failure to adequately identify needs and coordinate agency activities it likely also reflects the limited capacity that many agencies had, having deployed significant human and financial resources to address the needs of those affected by the Haiti earthquake 6-months earlier.

Box 2: A Comparison of Humanitarian Needs 2004-2010

25,000,000 20,000,000 15,000,000 10,000,000 5,000,000 0

Affected Population

The International humanitarian response was all too often characterised as far too late to be considered life-saving with only a few exceptions and can best be described as a second wave of support which has raised questions about the ability of the sector to mobilise swiftly. Where assistance was provided there were often perceptions that it was supply-driven rather than needs- based with agencies being found to be slow to meet significant (and often foreseeable) requirements such as for winterisation, reconstruction of shelter and livelihoods recovery.

The above said, there were also some successes with agencies cumulative reach increasing with time and the forging of innovative models of coordinated response (such as the Consortium of British Humanitarian Agencies, the CBHA) and innovative practice in cash-based responses in particular. It should be considered a significant achievement that despite the complex aid environment, within a

6 This compares unfavourably with the Haiti Earthquake ($495), the 2005 Pakistan Earthquake ($70) and Cyclone Nargis which hit Myanmar in 2008 ($46). See: Oxfam (2011) Six Months into the Floods – Resetting Pakistan’s Priorities Though Reconstruction.

Evaluation of ACF’s Response to the Pakistan Floods 3 matter of weeks the humanitarian response became ‘the largest emergency operation ever staged by the humanitarian community.’7

1.3 The ACF Response ACF’s first intervention in Pakistan was in 1979 in the context of the Afghan crisis and the arrival of thousands of refugees from across the border. Since then the organisation has returned to the country on numerous occasions with a focus on responding to humanitarian need on an ad-hoc basis rather than establishing a longer-term presence. The most recent emergency response was to the earthquake which hit Pakistan’s Kashmir Province in 2005 for which ACF-US managed the humanitarian response albeit with assistance for the early scale-up from ACF-France. In 2009 a decision was taken not to respond to the IDP crisis on the basis of the insecurity in the affected area, the number of other responders and the complexity of the context.

Preparedness Phase Prior to the floods ACF had a relatively small presence in Thatta in Sindh Province where it was implementing a World Bank (WB) funded project to help coastal farmers develop sustainable solutions for the increasing salinization of soil and water sources. The project only required a staff of 20 nationals and 4 Internationals. In KPK despite the security risks ACF were on the verge of starting an ECHO-funded multi-sector response for conflict-affected populations but had not yet opened an office nor despatched staff to the area. This was the profile of the ACF operation which formed the springboard for the floods response.

Phase 1: July - August Two-weeks after the initial flash floods an ACF WaSH assessment team carried out a rapid assessment in KPK and requested an additional $250,000 for the crisis from ECHO which was added to the proposal which was already being negotiated. This allowed the team to conduct early, rapid assessments, develop emergency interventions, hire personnel, and set up an office in KPK province. The programme focused on providing kits with essential water purification and hygiene items to 52,000 people in addition to the distribution of 32,000 litres of safe drinking water through water trucking, piped water supply and chlorination. Tube wells were rehabilitated in affected villages and emergency latrines were provided along with hygiene promotion sessions. A grant of $228,000 from ACF-France at the beginning of September allowed for the expansion of this work to include food security & livelihoods programmes for 3,500 households (approx. 24,000 people) with a focus on ensuring access to adequate and appropriate food and prevention of asset depletion. ACF also provided human resource support to an Oxfam-led Emergency Market Mapping and Analysis (EMMA) project in the area.

In Sindh, funding from ACF-UK ($15,000k) received on 06 September supplemented by an ECHO grant and ERF funds permitted the team to provide safe water, sanitation and public health promotion to contain the spread of water-borne illness in Thatta Province. While activities took some time to scale-up, an initial 30,000 people were targeted in 10 villages. This programme was further expanded with a donation in mid-September from ACF-France ($225k) which was later followed by funds from CIDA, CBHA and OCHA in October and November.

Phase 2: September - January In the first 6-months of the response, over $9 million dollars of funds were raised with a significant WaSH and FSL programme being delivered by a team of 25 International staff and several hundred national staff in KPK and Sindh Province. During this time in the WASH sector alone project

7 Polastro R et al (2011) Inter-Agency Real-time Evaluation of the Response to Pakistan’s 2010 Flood Crisis, DARA.

Evaluation of ACF’s Response to the Pakistan Floods 4 monitoring estimates suggest that between 28,000 - 35,000 people in KPK and 117 - 147,0008 people in Sindh and KPK benefited from services provided by the programme.

The 2010 ACF International Annual Report estimates that the programme has reached over 300,000 people with access to clean drinking water by trucking in over 50 million litres and rehabilitating 325 water points; over 1,000 emergency latrines and showers have been constructed and more than 27,000 hygiene kits have been distributed to families at risk of contracting illnesses like cholera and dysentery. It reports that ACF has disbursed over 2,500 cash grants to small business owners to re- start their micro-enterprises, distributed food to 5,000 families and provided cash-for-work opportunities to nearly 3,500 people.

2. The Scope of the Evaluation & Methodology

2.1 The Scope of the Evaluation Six main themes are outlined in the ToR9 which are consistent with the ACF International Emergency Response Learning Project10 and which provided the basis for the evaluation;

. Needs assessment and strategy development; . Communications and coordination; . Human resources; . Logistics; . Administration and finance; . Monitoring and evaluation.

These themes were evaluated through the lens of 4 evaluation criteria being timeliness, relevance, effectiveness and coherence. With 11-months having passed since the floods, rather than focus on the operational programme the evaluation focuses instead on strategic decision-making and direction, programme support and internal coordination with the aim of contributing to internal organisational learning. The period under evaluation starts from the onset of the floods in late-July until the end of January 2011.

2.2 Methodology A methodology matrix was prepared prior to the evaluation and circulated to ACF for input. This included a summary of the criteria, key questions to guide the evaluation consultant and a list of information sources.11 Discussions were held with ACF International members in New York, Paris and London and discussions were held with senior staff from Madrid and Montreal over the phone. Contact was made with former ACF staff deployed to Pakistan and several interviews were conducted. Documentation from the programme was shared with the evaluator and a review of this material provided valuable input into the evaluation.

2.3 Limitations

8 These estimates are based on the possibility of a 20% error of inclusion due to double-counting. 9 The full ToR is reproduced in annex 3. 10 See annex 6. The project is founded on the belief that the key to more successful emergency responses lies in ACF International’s capacity to critically evaluate its standard operating procedures during emergencies. The aim of the project is to do precisely that; to identify lessons learned from rapid-onset emergencies, and to facilitate their integration into future ACF International responses. In doing so, the project aims to contribute to ACF International’s preparedness efforts, and to provide a framework for futures evaluations that ensures coherent and consistent learning in the organisation. 11 The matrix is reproduced in annex 4.

Evaluation of ACF’s Response to the Pakistan Floods 5

The evaluation was conducted over a relatively short period of time and the lack of a field component was a significant limitation to field-based programme analysis and skewed the evaluation towards a strategic/headquarters perspective. While efforts were made to contact and arrange telecons with departed field staff this was only partially successful. The implications of this are that some aspects of the ToR were difficult to explore to any detail (particularly those which required field-based evidence). The departure of some members of the New York team (particularly the senior Human Resources staff) made it difficult to piece together some of the history of the intervention. That said the openness of the team and efforts they took to share information helped fill in many of these gaps. While the ToR spoke of an evaluation of the response, in reality, the process was more akin to a management review and this report reflects this.

3. Needs Assessment and Strategy Development

3.1 Phase 1

Preparedness Planning During interviews in New York concerns were raised about the lack of a preparedness plan for Pakistan particularly given the organisation’s history and established presence in the country and the frequency of disasters. While in-country preparedness planning isn’t a requirement for ACF (which for many agencies it is), in a country so vulnerable to natural and man-made disaster it makes good humanitarian sense and a document outlining distinctive competencies, areas of potential response, that identifies suppliers (and standby arrangements) and staff would have gone a long way to easing the initial burden of response and would have helped clarify immediate human resource needs.

Needs Assessment By 29 July the damage from the flash floods in KPK and Northern Districts was considerable and news of the extent of the human cost was becoming widely known both nationally and Internationally. As the flood waters moved south the nature of the damage changed and large areas became completely inundated over a period of several weeks.

With a WaSH Coordinator already in-country it was in this sector that performance was strongest. The first assessment undertaken between 13-16 August in the Thatta district of Sindh where had an established presence was extremely timely and likely pre-dated the actions of many other agencies although follow-up was slow in coming. The assessment in Charsada and Nowshera Districts of KPK Province between 16-20 August followed shortly after the Sindh assessment but was 2-weeks after the flash floods and hence was far less timely. A dedicated FSL assessment was conducted in Sindh between 23-24 August and a follow-up assessment was conducted 6-weeks later between 4-7 October. A multi-sector assessment in Sindh was carried out jointly with Handicap International at the end of October (Oct 28 – Nov 04) by which time communities had started to return home which significantly changed the needs profile. Towards the end of the period under evaluation, an integrated nutritional assessment was undertaken in Thatta in Sindh as a means of verifying the levels of malnutrition to form the basis of a proposed nutritional response.

A review of the assessment reports show that they identified and prioritised interventions. Specifically the first Thatta assessment correctly identifies that ‘the number of camps and camp populations in Thatta will continue to grow as flooding becomes more widespread’. The KPK assessment verified the earlier World Food Programme (WFP) assessment that up to 100,000 households had been affected, detailed a range of needs and made recommendations for a WaSH intervention. That both programmes took some time to commence meant that the opportunity for a timely intervention in Sindh was lost and delays in launching an intervention in KPK meant that ACF’s

Evaluation of ACF’s Response to the Pakistan Floods 6 response was late in coming. Later assessments to update information and analysis are justified given the considerable change in context as people began to return home and re-establish their livelihoods and it is in the this phase of recovery that ACF’s assessments ensured the delivery of a very relevant set of programme responses.

Programme & Strategy While assessments were conducted and priority needs were outlined in the reports that followed a humanitarian strategy wasn’t developed for the initial phase of the response. In the absence of a country, geographic or sector-level strategy, interviews suggest that programme gaols, purposes and outputs were developed on a project basis. While the lack of a field component to the evaluation makes it difficult to evaluate the impact of this it is a reasonable assumption to make that this would have limited the potential for coherence and connectedness of ACF’s work. It would have also complicated the task of planning the response and articulating resource needs.

In discussing programme implementation it is important to highlight the impact of the security situation on activities in KPK where there were considerable restrictions on NGO movements as a result of the recent conflict between the army and Pakistani Taleban. While active conflict in KPK ended in 2009, low-level terrorist activities continue to this day and given that NGO staff have been targeted in the past security remains extremely tight with agencies employing both deterrence and acceptance strategies. The role of the Pakistan military both as a party to the conflict and as the largest (and arguably one of the most effective) providers of humanitarian assistance has further contributed to the blurring of boundaries between the role of the military and that of humanitarian agencies which has been little helped by the insistence of the army on accompanying the provision of humanitarian assistance in some areas. Given the threat to those agencies working in KPK ACF was in good company in seeking to prioritise staff security which at times constrained programming, particularly in terms of the number of staff deployed to the North.

With programmes having got well underway by October and with some success in recruiting in new staff to senior management positions (in particular the Desk Officer and the Country Director) there was a tangible improvement both in terms of addressing some of the areas of weakness and in delivering a much-scaled up recovery programme.

ACF’s initial programme reflected its organisational competencies in WaSH and FSL. The sector which had the greatest capacity was in WaSH and the humanitarian experience of the existing WaSH Coordinator was put to good use in launching assessments and in recruiting national staff into the programme and it was this team that formed the core part of the initial response. That it took time to resource International WaSH posts to supplement the existing capacity meant that the extent of the initial scale-up was geographically limited. Despite the timely assessment of Sindh (see above) it took some time for the team to act on the findings which was hampered by difficulties in identifying human resources and a lack of logistics capacity to procure NFIs and WaSH equipment. Programme work started in mid-September12 and was scaled-up further in October. While important needs were met the trip report of the New York-based Technical Advisor concluded that ACF had missed the initial emergency.

Box 3: Analysing the Timeliness of the Response

In the context of Pakistan the timeliness of the ACF response is an important but complex criterion to assess.

12 The October 01 sitrep reported the following outputs; A total of 1,728,000 liters of safe , chlorinated water were delivered in 3 camps in Thatta District; a total of 84 latrines were constructed in 4 camps covering the needs of 4,200 people; hygiene kits had been provided to 2,200 people

Evaluation of ACF’s Response to the Pakistan Floods 7

While the floods affected KPK Province first (at the end of July), insecurity and modalities of response which sometimes required military escort slowed the overall pace. In Sindh, access and security were far better but the extent of the floods meant that some areas were cut off and as a result the humanitarian community as a whole were quite slow to assist those in need. Perhaps the best comparator available to determine timeliness is the DEC Real Time Evaluation (RTE), undertaken in December 2010 which reviewed the performance of British member agencies13 and concluded the following;

KPK: ‘KPK was the first district to be hit by the flood, and relief started promptly. DEC organizations already had an established presence in the province and had gained considerable expertise in previous disaster responses. Some organizations managed to mobilize and start response within hours of the floods. Availability and expertise of local staff allowed immediate access to areas which would not have been open to International teams, due to security issues and travel restrictions imposed by the government. Some partnerships were already in place, increasing the speed of the response. These partnerships created linkages between organizations beyond coordination in clusters, and allowed the implementation of complementary / integrated approaches.’

Sindh: ‘Most agencies were 2-4 weeks late in responding to the floods in Sindh due to the on-going work in KPK related to the conflict and the subsequent floods. Thus, agencies did not have sufficient capacity in Sindh to immediately launch operations. Those agencies that had partners in Sindh, such as Christian Aid, Concern and Oxfam, were able to start work more quickly. Given the lack of precedence of such a country-wide flood in Pakistan, some delay in starting in Sindh was to be expected and the delay could only have been reduced through proactive disaster preparedness.’

3.2 Phase 2

Preparedness Planning The Disaster Risk Reduction (DRR) assessment which was undertaken in Sindh and KPK Provinces in February 2011 goes some way to meeting the needs to strengthen ACF’s preparedness planning. The report highlights the limited capacity that exists within Pakistan and makes recommendations for integrating DRR measures into existing programmes and in building capacity for response in the future. While measures such as these inevitably have a cost attached to them investment in preparedness has consistently been proved to make good financial and development sense.

Needs Assessment Needs assessments continued throughout the second phase, with an FSL assessment conducted in Sujawal in Thatta District of Sindh which was followed up in the same month with a second joint assessment. These assessments highlighted the imminent return of IDPs to their homes and provided a justification for further scaling up FSL programmes to support communities during and after they had returned. A multi-sector assessment was launched jointly with Handicap International at the end of October in Sindh which by this stage was the area still requiring significant assistance. The report that followed highlighted the shifting needs from emergency shelter to food and livelihoods and access to clean water, all sectors that ACF were ideally placed to support.

Programme & Strategy Once momentum had been built ACF was well-placed to meet recovery needs while continuing to respond to discrete emergencies that came about as a result of the context of protracted displacement in Sindh. The KPK emergency response began to wind up during October which allowed ACF to focus on Sindh where there was still considerable displacement as a result of the floods. ECHO, OCHA ERF and CIDA funds were secured towards the end of the evaluation period to significantly scale-up the response. While the focus on access to potable water, provision of hygiene

13 Murtaza N et al (2010) Pakistan Floods 2010: The DEC Real Time Evaluation Report, December 2010

Evaluation of ACF’s Response to the Pakistan Floods 8 items and construction of sanitation facilities continued, beneficiary numbers were significantly increased and a greater focus was placed on strengthening hygiene promotion.

Work in the FSL sector was slower to start than for WaSH and apart from a modest NFI distribution in KPK in September and despite plans to scale up the response, no significant progress was made until November when cash grants were distributed to communities in Charsada and Nowshera. Project reports show that the long distance to the distribution site and the need to be back at base during daylight hours hampered the process of scale-up but the limited logistics capacity and gaps in staffing were a more significant hindrance to the timeliness of the intervention. The response took even longer to get underway in Sindh with a trip report written in late-October noting considerable concern that 10-weeks after the floods there was still no FSL response in the South.

As the need for humanitarian assistance began to be replaced by longer-term needs the FSL programme began to deliver important support to communities seeking to recover from the impact of the floods on their livelihoods. Rather than seek to deliver standardised responses ACF’s focus on cash-based interventions was highly relevant allowing people to make choices as their situation dictated. With the onset of winter and as people returned to their villages and set about the task of rebuilding their homes, responses which focused on the provision of NFIs struggled to keep pace with the changing context of needs emphasizing the appropriateness of ACF’s approach. Internal monitoring reports document the high degree of satisfaction with cash which was considered to offer flexibility and permit families to establish their own recovery priorities. In addition to this the alliances which ACF entered into as part of the Pakistan Emergency Food Security Alliance (PEFSA) and the Consortium of British Humanitarian Agencies (CBHA) delivered integrated FSL responses which went some way to attaining the sort of reach in terms of beneficiary figures that was required in Pakistan.

The failure to recruit an expatriate nutritionist led to a default position being adopted of not engaging in this sector in either phase 1 or 2. This has raised considerable criticism both in the field team and at headquarters level and is broadly considered to be a gap in the response in a sector that is considered to be a core competence. Interviews highlighted an expectation that ACF would be represented in the nutrition cluster in Pakistan and despite conflicting accounts of post-flood nutritional needs, that the decision was made due to an inability to resource the post rather than on the basis of need is a concern.

The lack of a humanitarian strategy (however brief) is a weakness of the response and it’s a shame that efforts to articulate a formal recovery strategy are yet to come to fruition. The timely dissemination of a response strategy is important for numerous reasons which go far beyond ensuring the delivery of a high quality programme. Senior managers across the ACF International required an indication of action, communications staff needed a clear message for supporters, and fundraisers needed to be able to communicate needs, explain why funds were required and inform potential donors how ACF would spend the funds raised.

The revised Emergency Management System (EMS) requires that an ‘International appeal’ is developed and circulated within 24-hrs to be followed by a 1-month strategy in 5-days and a medium-term (6-month) strategic plan within the first 2-months which is strongly endorsed by this evaluation as it would have clarified strategic intent and allowed a much more coordinated mobilisation of resources from the Network.

Evaluation of ACF’s Response to the Pakistan Floods 9

One of the recommendations of the ACF tsunami evaluation14 was to mandate an independent assessment team early on in the response to act in a real time evaluation (RTE) capacity – to determine whether the initial strategy adopted by the organisation is adequately responding to the crisis, to re-examine needs and to recommend course corrections. Such processes are becoming common amongst larger humanitarian agencies which are keen to be able to prove and improve the effectiveness of their response. In the case of Pakistan a RTE such as this would have provided invaluable feedback and would have potentially allowed issues to be raised and resolved in a timely manner. While an RTE may have limited utility for smaller responses, for the larger ones where there is significant organisational risk it is highly recommended.

3.3 RECOMMENDATIONS: Needs Assessment & Strategy Development . A programme strategy document is necessary for guiding programme and mobilising ACF’s resources. It is absolutely essential that the revised EMS framework which stipulates the submission of an International appeal within 24-hours of an emergency, a 1-month strategy within 5-days and a 6-month strategy in the first 2-months is followed; . While an outline of strategic intent is important it cannot provide organisational reassurance that timely and effective programmes are being delivered in the field. To fill this gap it is recommended that an assessment or RTE, preferably independent of the Managing Member is conducted within the first 6-weeks of the response being launched; . One of the key weaknesses for ACF in Pakistan was the lack of preparedness planning. In countries where ACF has a strategic interest in responding to emergencies it is recommended that a preparedness plan which scopes out potential humanitarian needs, defines geographic and sectoral areas of response and which identifies resources for response (both human resource and materiel) would provide the best possible platform to launch a timely response.

4. Communications & Coordination

4.1 Phase 1

External Communication Clear and regular communications are extremely important for building and maintaining a support base and fundraising and in many of the ACF International members there was an interest in what ACF was doing to help flood survivors. While a team that was delivering a large humanitarian programme would have provided ample opportunity for stories and communications materials, the fact that the start of the ACF programme took time to initiate delayed the deployment of a freelance photo journalist and made it difficult to offer material to ACF International members who were keen to make use of the opportunity. Box 4 below shows some of the early communications material that was released by the New York office and supporting members and also provides an indication of the challenge that the team had in communicating a sense of progress, with the first update on programme delivery provided in early September.

While the deployment of a communications person would have positioned ACF to better able exploit opportunities when they did start delivering programmes it is difficult to get beyond the fact that the slow pace of response made it impossible to communicate a message that ‘ACF were there and were responding’. Despite this shortcoming ACF-UK and ACF-France (where the floods were seen as potentially big news and were of significant interest to constituencies) made efforts to launch communications work based on the knowledge that ACF had a team on the ground; in the UK, a fundraising appeal was launched on 4 August in the UK press but with requests to nominate a

14 Quentrec H and Cunnington P (2006) Indian Ocean Tsunami Evaluation of Action Contre La Faim Response, 26 December 2004 – 30 June 2005, Indonesia and Sri Lanka. The recommendation is made on pp76. No. 460.

Evaluation of ACF’s Response to the Pakistan Floods 10 spokesperson in the Pakistan team turned down and with very little communications material to draw from the advert was stopped after 2-days. This was followed by an email shot on 20 August but it had only limited appeal as a programme was yet to start. ACF-France sought to build profile and momentum in the media but again it was difficult to coordinate and sustain interest. The team in Paris made significant efforts to stimulate interest domestically with a range of press, radio and TV interviews throughout the month of August.

Coming so soon after a successful Haiti mission which was considered by some to be ‘the state of the art’ in terms of communications and fundraising expectations were high which likely made the outcome all the more disappointing. While the complex context of Pakistan made it extremely difficult to deploy a communications person it will be important for ACF to use the revised EMS framework as a forum to discuss how to best meet the communications needs of the network and to manage expectations.

Box 4: External Communications in the aftermath of the floods

03 August, ACF-US PRESS RELEASE: emergency unfolds as floods devastate North Pakistan Situation update from New York-based technical advisor 04 August, ACF-US EMAIL APPEAL: Emergency: Severe flooding hits northern Pakistan 04 August, ACF-UK Appeal: Decision made to launch an appeal based on the success of Haiti, the considerable news interest in the UK and the severity of the disaster 05 August, ACF-UK MEDIA: Advert launched in Daily Telegraph and retained for 2-days in paper and 1 further day online and then removed due to lack of support to sustain the appeal 09 August, ACF-France PRESS RELEASE: Course contre la montre pour venir en aide aux victims des inondations 10 August, ACF-France MEDIA: Pakistan Floods: Aid organisations concerned/donations/ACF Interview with Daniel Holmberg, ACF’s Country Director 11 August, ACF-France WEB UPDATE: 3 questions to Daniel Holmberg, Head of Mission in Pakistan Transcript of interview with ACF’s Country Director 12 August, ACF-US EMAIL APPEAL: Emergency: Crisis in Pakistan Situation update and ACF plans for humanitarian response 17 August, ACF-France PRESS RELEASE: ACF reitere son appel aux dons au Pakistan 17 August: MEDIA: LaCroix.com, Ouest France.fr, France5, La Croix, France 24 18 August, ACF-US WEB UPDATE: Unprecedented Floods Affect 20 million in Pakistan Situation update and ACF plans for humanitarian response 18 August, ACF-France MEDIA: Radio Classique, RCF, RTL 19 August, ACF-France MEDIA: France 24, Mediapart, Liberation 02 September, ACF-US/France PRESS RELEASE: ACF provides clean water & sanitation in flood-affected Pakistan First programme update on the web 09 September, ACF-US WEB UPDATE: Humanitarian emergency in Pakistan Interview with Mohammed Mechmache, New York-based Desk Officer including programme update 09 September: ACF-US EMAIL APPEAL: Support Action Against Hunger’s Emergency Response Programme update

Coordination within ACF International Coordination between ACF International members in the months after the floods can at best be described as flawed. Outside of ACF-US, ACF International members felt that coordination was under-prioritised and given the urgent need for support more frequent telecons could have given a clearer sense of the needs and allowed members to more easily offer and target assistance. It certainly appears that in the early stages there was a dearth of operational information to share with ACF International colleagues which made it difficult for members to understand what progress was

Evaluation of ACF’s Response to the Pakistan Floods 11 being made and where the blockages were. At the International Executive Committee (IEC) level the first detailed formal communication on the Pakistan floods arrived in mid-August which outlined the extent of the needs, ACF assessments and made an urgent request for unrestricted funds. This was followed by similar updates in the weeks that followed.

Coordination in the areas of human resourcing, fundraising and communications are dealt with in the relevant sections and while it would be easy to attribute responsibility for this to a single member the failure to meet humanitarian needs in the weeks immediately after the floods represents a broader failure of governance at an ACF International level.

The ACF Strategic Plan (2010-2015) speaks of ‘a desire to ensure that ACF International is remaining true to its mission of eliminating global hunger and severe malnutrition – and that we are doing so in the most effective and efficient way possible….[by] maximizing the opportunities that flow from being part of the ACF International network’.15 The document provides a challenging mission for ACF which it justifies by the extent of global need, by ACF’s distinctive competencies and the potential that is offered by a unity of purpose between members. Unfortunately the picture painted in the early weeks after the floods in Pakistan was one characterised by a lack of unity rather than the presence of it.

ACF International’s response is guided by an ‘Emergency Management Team (EMT)’ document which was prepared in 2004 and updated in 2008. The first of these documents outlined a broad structure and outlines some basic principles for HR, communications and fundraising. Included in these is notification of HR needs, the deployment of a Press Officer to receive journalists from the Network, the production of regular sitreps (‘daily/weekly EMT Report’), the transfer of emergency funds between Members for humanitarian response and the concept of sub-contracting out elements of the response to other Members. The 2008 document sought to update the EMT document with a focus on how to best support operational capacity building; it explores issue of categorising emergency interventions and Member response capacity. It establishes the concept of an Appeal to be launched within the first 48-hours after a disaster with an ACF International Proposal’ within 96-hours. It speaks of a strategic weekly report being compiled with a daily update for the first 2-weeks and an HR plan being finalised within 48-hours within which time a team should be deployed with a plane of emergency relief items to initiate a response. It also outlines a mechanism for pooled humanitarian financing (‘International Consolidated Proposal’)

It is noteworthy that none of the members used the EMT guidelines as a lens to critique the Pakistan response. Similarly there is no evidence to suggest that the guidelines were used as a means to hold ACF-US to account when, as the managing member it failed to deliver against the expectations outlined in the document. Only in a minority of interviews were the guidelines mentioned at all and very few staff appeared to be cognisant of the requirements that are laid out.

To ensure greater consistency in the first phase of ACF’s humanitarian response will require that the EMT document is strengthened. The minimum standards for response should be reviewed in the context of the lessons learnt in Pakistan and a mechanism needs to be put in place to formally raise concerns when there is a perception that standards are not being met. In addition to outlining responsibilities for the managing member, there also needs to be far greater clarity about the responsibilities of the supporting members. While Pakistan revealed weaknesses in how the response was managed it also highlighted inconsistencies in the way support was provided. Ultimately if ACF is to meet its strategic vision of meeting the needs of greater numbers of people

15 ACF Strategic Plan 2010 – 2015, October 2009, Executive Summary, pp2.

Evaluation of ACF’s Response to the Pakistan Floods 12 efficiently and effectively then the network needs to become more fluent at working collectively towards common goals. One of the lessons of Pakistan is that this doesn’t happen organically.

Beyond this ACF International should also formalise the concept of a provider of last resort whereby the managing member can delegate leadership of specific functions to supporting members. Given the different capacities and competencies of each of the members such an arrangement could see greater efficiencies achieved by allowing members to play to their strengths and would go a long way to realising the vision of unity which the ACF Strategic Plan talks of. This would likely require a greater level of trust and honesty than existed between members during the Pakistan response but it would represent a significant step forwards.

The issue of ‘unity’ came up frequently during the evaluation – as much as a challenge as a strength. Questions were asked about whether there really was unity of purpose across the members and whether a pre-condition for unity was ‘complementarity’ meaning a replication of functions across all members. Questions were asked about whether the Pakistan response would have suffered the same challenges had ASF-US had an emergency pool of its own, or had the response been managed by ACF-France the member with the greatest resources. It is the view of the evaluator that ‘unity’ denotes a common vision rather than a replication of functions across all the members and that while there needs to be sufficient capacity for delivery the diversity of ACF’s members is a strength rather than a weakness. That said success will require that expectations of how members can support each other in humanitarian response need to be much clearer.

Internal (Field – HQ) Coordination Communications and coordination between the field and headquarters was patchy – in some areas there was considerable success while in others it proved problematic with decisions taking too long to be made. Between the Technical Advisors and their field counterparts there was generally very good communication and the New York team worked hard to provide real-time support, recruit much-needed staff, and contribute to the direction of the programme. While field trips from the advisory staff tended to be relatively short (2-3-weeks), they appear to added important momentum and trip reports which were produced were well written and identified action points. While terms of reference and multi-country responsibility were given as reasons why advisory staff didn’t travel out to the field sooner for longer periods of time, this was challenged by some.

At the level of management, communication and decision-making between the field and headquarters appears to have been less effective in the early weeks of the response with several key decisions taking too long to be made. Chief amongst these was the decision to remove the Country Director. Recruited into the Pakistan programme to lead a small team managing a World Bank grant, it is no surprise that his profile was a less than perfect match for a rapid onset emergency response. That it took considerable time to reach a decision that he should be replaced and even longer to find a replacement likely contributed to holding back the programme.

While preparedness is dealt with in greater depth elsewhere in the report a measure which is gaining common currency throughout the sector and which would have facilitated swift decision- making is to have a ‘step-aside’ policy in place which could have been swiftly implemented at the time the disaster hit (see HR recommendations below). By having a management conversation when a Country Director is recruited to determine the capacity of the individual and prescribe the size and scope of programme that s/he is capable and competent to manage, rapid decisions can be made when the programme is required to scale-up. Efforts can also be made outside of emergency response to build capacity of the staff member.

Evaluation of ACF’s Response to the Pakistan Floods 13

There are examples of decisions which were taken which potentially added to the confusion; interviews suggest that the one staff member who was quickly deployed from New York office to the field might have served to add to the confusion rather than reduce it. While the arrival of the Desk Officer in Peshawar could have provided much-needed support the lack of clarity about the role he was playing was divisive within the team. Because the New York role was folded into that of the Director of Operations it also saw a significant reduction in headquarters capacity at the time it was most needed.

4.2 Phase 2

Coordination with ACF International With the emergency phase of the Pakistan programme having come to an end, there was a reduction in the transactions between ACF International network partners. While ACF members continued to liaise on HR issues (see section 5) there were not the same expectations or requirements for communication and coordination and these were effectively serviced by the newly- arrived Desk Officer who was quick to make strategic relations with colleagues throughout the network.

During this time, there was some effort to reflect on the lessons from the Pakistan experience and in June ACF International went a long way to laying the foundations for successful cooperation for humanitarian response. At the Barcelona meeting an ‘Emergency Management System (EMS)’ was agreed which includes a flow chart and road map which identifies critical timeframes for delivery against a core set of expectations for rapid onset emergencies. These include the launch of an International appeal (within the first 24-hours), mobilisation of communications staff and arrangements for their deployment (within the first 24-hours), an initial pledge of funds (within the first 24-hours), a 1-month strategic plan (within 5-days) and a resourcing plan including requirements for emergency pool and roster staff (within 7-days). The EMS defines focal points in both the managing and supporting members and importantly, it outlines a formal mechanism to raise and deal with concerns and to resolve disputes (bilaterally through the Operations Director and/or Executive Director and then through the IEC). An informal mechanism has also been agreed at Executive level as a means to resolve disputes informally without having to escalate them to the IEC. With these mechanisms yet to be trialled it is too early comment on their potential success, but assuming implementation is mandatory, they certainly provide a far stronger platform for timely and coordinated humanitarian response.

While the EMS certainly provides a framework to address many of the problems associated with the Pakistan response the issue of how members prioritise resources is left unclear. To address this it is recommended that a mechanism to prioritise all new emergencies is adopted as part of the EMS framework. This would allow an objective approach to ensuring that resources (particularly human) can be directed towards the largest disasters and would provide a transparent way of allocating roster and emergency pool staff. The process outlined in the document for escalation and resolution of concerns and disputes is particularly critical and it will be important that members prioritise the use of this in order to foster greater trust and accountability across ACF International.

Internal Field – HQ Coordination Phase 2 saw important changes being made in the leadership of the response in Pakistan and with the addition of a Desk Officer to replace the previous incumbent who had been deployed to Pakistan there was considerable improvement in the management capacity of the US and Pakistan team. These changes led to a considerable improvement in communication and decision-making between headquarters and provided a strong foundation for the expansion of ACF’s recovery programme.

Evaluation of ACF’s Response to the Pakistan Floods 14

Participation in Field Coordination Forums ACF staff participated in numerous humanitarian coordination forums including the Pakistan Humanitarian Forum and a number of the clusters (at Capital and Provincial level) and it also contributed to inter-agency appeals and programme delivery mechanisms (PEFSA and CBHA are two examples of this). While there was an eagerness to strengthen coordination there was also recognition of the opportunity costs that this carried.

The WaSH team had a strong commitment to attending the Cluster in Islamabad and where possible at Provincial level also. ACF were also instrumental in lobbying for the creation of a Strategic Advisory Group (SAG) in recognition that the Cluster was over-subscribed and tended towards the lowest-common denominator of information sharing rather than being a strategic forum. The Technical Advisor also participated in the development of the Pakistan Initial Floods Emergency Response Plan (PIFERP). The FSL team were similarly active in coordination, attending relevant clusters when time and resources permitted. ACFs participation in the PEFSA Alliance and CBHA also required significant coordination commitments. A trip report from the New York-based Technical Advisor provides a cogent summary of the trade-offs that had to be made as a result of this commitment;

‘The time and energy resources required to support this coordination is tremendous and regarding the PEFSA, was significantly underestimated in the design of the initial mission organogram. Other members of the Alliance have dedicated staff members to represent their organization in Alliance meetings & trainings and to deliver on various specific responsibilities (M&E, GIS, gender…) while ACF relies solely on the FSL Co for this function. This has undermined her ability to focus on other areas of responsibility including the establishment of monitoring systems; greater contextual analysis; development of technical strategy; greater integration with other departments, etc.’16

The unwieldy and at times ineffectual humanitarian coordination of the Pakistan response was singled out for specific criticism in the Inter-agency Evaluation of the humanitarian response to the floods17 but with the mechanisms that were rolled out as part of the 2005 reform process now an important part of doing humanitarian business it would be short-sighted to opt out and would (at least in theory) place those in need of assistance at ever greater risk of missing out. However that is not to say that the status quo is acceptable and given the limited capacity that the ACF team had the time spent in coordination forums was seen by some as an unnecessary distraction. While an obvious recommendation is to commit dedicated resources to meet coordination needs it is accepted that this has both financial and human resource implications and hence may not always be viable.

The consistent failure of coordination in Pakistan is deeply concerning; successive inter-agency evaluations in Pakistan (for the 2007 flood response in Balochistan, the 2009 IDP crisis in Swat Valley and now the 2010 earthquake response) have been critical of the clusters and appear to be more damning with each subsequent response. It has now become urgent that the humanitarian community find a workable compromise to deliver ‘light-touch’ operational coordination which can successfully balance the need to guide and facilitate humanitarian action with the humanitarian imperative of meeting needs.

4.3 RECOMMENDATIONS: Communication & Coordination

16 Field visit report, Pakistan, FSL Technical Advisor, Muriel Calo, October 2010 17 Polastro R et al (2011) Inter-Agency Real-time Evaluation of the Response to Pakistan’s 2010 Flood Crisis, DARA.

Evaluation of ACF’s Response to the Pakistan Floods 15

. Communication needs must be viewed through the lens of ACF International as a while and resourced accordingly. The revised EMS stipulates that a decision about the deployment of a Communications person must be made in coordination with other members within 24-hours of an emergency which is strongly endorsed by this evaluation; . Given the frustrations felt by many members at the lack of material made available it would make good sense to prescribe in more detail a set of minimum standards for communications in emergencies which could include a one-page outline of context and key messages within 48- hours, the designation of a spokesperson within 48-hours and a set of expectations around regular updates. A daily/weekly communications conference call between Network members and the programme team in the first month of the response would ensure that expectations are managed and that the combined resources of ACF can be used to support communications work; . The failure of ACF to implement humanitarian response according to the EMT suggests that the document has fallen into disuse. The revised EMS provides a set of minimum requirements for disaster response and a timeline for their delivery. It also strengthens accountability and provides a mechanism for resolving disputes. This tool potentially provides a strong platform for a coordinated response but it is only as strong as ACF’s commitment to abide by it. It is recommended that the document is circulated throughout the members and that key staffs performance objectives are revised in light of it; . Where the managing member lacks capacity in a particular area of a response (comms, operations, HR), this must be communicated to members in a timely manner with a view to delegating responsibility to a supporting member; . To foster greater transparency, to (re)build trust and to guide decision-making about the deployment of emergency pool and roster staff, a system of prioritising emergencies (using criteria such as geographic scale of crisis, number of people affected etc.) is recommended with a commitment for members’ decision-making on deployment of resources to reflect this; . While the coordination structures which have come about as a result of Humanitarian Reform are time-consuming, they are also the means of doing humanitarian business and as such ACF needs to continue to prioritise participation in them. In recognition of this it will be important to ensure that humanitarian programmes are adequately staffed to meet both operational and field coordination needs.

5. Human Resources

5.1 Phase 1

Human Resources at Field-level It’s important to prefix this section with a snapshot of what was achieved by ACF International. As box 5 below shows over a period of 4-months the ACF team in Pakistan increased from a size of 4 International staff and a small team of national staff to 25 Internationals and several hundred national staff working across a number of bases across the length of Pakistan. This is no small achievement given the modest size of the team prior to the floods. However the scale-up was far from easy and the HR team in New York struggled with internal problems which exacerbated difficulties it had with successfully recruiting staff with the right skills at the right time in the early stages of the programme. Box 5: Analysis of Phase 1 International Deployments to

Pakistan While on paper the existence of a team in Pakistan should have been a strength, Average # apart from a couple of members who were Pakistan Total # 1st Total # months able to make a contribution to assessment Country missions/ Expats/ on and response, many of the staff were Programme Month month mission July 1 4 6.37 August 1 4 7.3875 September 6 12 3.643333

Evaluation of ACF’s Response to the Pakistan Floods 16 considered ill-suited to the changed context and much time was wasted making up for skills deficits. While ACF-US has experience of successfully scaling-up for humanitarian response in Pakistan, Uganda and Congo, the profile of staff on the roster were found to be ill-suited to a rapid onset crisis in such a complex environment as Pakistan. Efforts were made to explore the possibilities of moving staff from other missions managed out of New York but these were unsuccessful.

Box 5 shows the length of time it took to strengthen the team; throughout August there were only 4 International staff and it was only in September (a month after the floods in KPK) that this number was tripled. While the response highlighted critical gaps in the ability of ACF-US, and by extension ACF International to find staff in a timely manner, interviews with returned staff and ACF member HR staff show a level of concern about the length of time it took to respond to offers of staff and when staff were deployed there was concern raised about changes in roles and responsibilities. While some change is inevitable given the nature of responding to crises a basic level of clarity is important as on at least one occasion this led to the resignation of a loyal and valued roster member.

In terms of identifying the key problem and an important solution, the Evaluation of ACF’s programme in Haiti remains highly relevant;

‘Staff shortages [however] have led to gap in operational performance, and the high turnover of expat staff had led to a lack, at times, of programmatical continuity, a loss of information, and issues for donors and local staff. Recruitment sources need to be augmented in some sectors, and expanded both internally and externally on a global scale.’18

Human Resources at Headquarters-level Throughout the first six-months of the floods response, the only additional post that was created in New York was a 3-month HR post recruited in November to support the significant increases in vetting of CVs and processing of staff to the field. Given the massive increase in ACF-US’ total funding portfolio and given that almost every staff member saw a dramatic increase in their workload, this would appear short-sighted. Rather than strengthen the headquarters team it was further weakened by the decision to deploy the Desk Officer to Peshawar. While this was done for the best of intentions, because the vacant post was not filled for some time it placed an even greater management burden in the hands of the Operations Director who took over the responsibilities.

While it may be argued that staffing up headquarters is counter-intuitive when there is a need to scale-up field teams, with organisational and supporter-expectations around information flow and with the need to maintain high quality management support, it’s essential to ensure that there is a functioning and effective team at headquarters level. Although this necessarily comes at a cost it is common practice to budget for these posts in humanitarian response proposals. A dedicated Desk Officer (with no other geographic responsibilities) or high capacity technical advisory post with a greater expectation of travel between headquarters and programme, who could provide real-time management support and problem-solving capacity could have had a significant impact on the floods response. This builds on one of the recommendations from the ACF East Asian tsunami evaluation which talks of ‘pre-identifying’ potential headquarters staff members.19

The HR Contribution Made by the Network

18 Duncalf J (2010) An evaluation of the Action Against Hunger (ACF) post-earthquake emergency response in Haiti, December 2010. 19 Quentrec H and Cunnington P (2006) Indian Ocean Tsunami Evaluation of Action Contre La Faim Response, 26 December 2004 – 30 June 2005, Indonesia and Sri Lanka. The recommendation is made on pp.77, no. 471 & pp79, no. 501.

Evaluation of ACF’s Response to the Pakistan Floods 17

While ACF-US has experience of resourcing and delivering humanitarian programmes, many staff talked of it having a far greater number of people in its pool with longer-term (recovery or rehabilitation) skill-sets which made it difficult to find recruits for Pakistan. While it would be easy to conclude that this makes it a poor choice for delivering humanitarian programmes, this would be faulty. Given the ambitions of the ACF network to grow its fundraising and to increase its reach in terms of beneficiary numbers and given the existence of emergency rosters managed out Madrid and Paris, in addition to the Emergency pool that both of these members have, an efficient approach would see ACF pooling its resources and establishing clear and accountable mechanisms supported by corporate priorities to guide human resource provision.

Deploying staff from different network members revealed potential ‘grey areas’ in the system. While there were some successes (the ACF -France FSL emergency pool member was deployed for 3-weeks on 18 August), there were also frustrations across all parts of the Network. Interviews suggest that poor communication played a significant part in the failure to adequately resource the Pakistan response in the early stages of the response but the lack of accountability within the prescribed system of managing cross-member deployments also had an important part to play and did little to build trust and confidence. The failure of all members to raise these issues in a timely manner and to proactively, productively and swiftly resolve them suggests a failure of governance at the level of ACF International.

5.2 Phase 2

Human Resources at Field Level Box 6: Analysis of Phase 2 International Deployments to Pakistan As box 6 shows, by October a scaled-up team was in place and it was from this time that programme delivery was expanded. Average # Pakistan Total # 1st Total # months The proportion of staff on first missions Country missions/ Expats/ on continued to be high during this phase (it Programme Month month mission was at 57% at 31st May 2011 the date for October 8 15 3.396667 which figures are available) and although the average length of deployment remains November 11 25 2.5036 relatively low (an average of 4.29 months December 11 20 6.27 at 31st May 2011) a far greater proportion of the team were being put on 12-month contracts which bodes well for stability in the future.

While the heavy reliance on staff on their first mission represents a success in terms of growing a potential pool of candidates for the future it presented a degree of risk to the operation in Pakistan. It also meant that there was a much greater reliance on those with a greater level of experience which tended to mean they got burnt out more quickly as they had to shoulder significant additional responsibility. The heavy reliance on new staff also had repercussions for the team when staff were unable to cope or weren’t competent to deliver against their ToR and at times there was quite high turnover as a result.

The new Country Director worked hard with the team to rationalise lines of communication and to ensure that headquarters support could be directed to where it was most needed. A new organogram was agreed which provided much-needed clarity and which was considered to have made an important contribution to resolving communication gaps.

Human Resources at Headquarters Level With the recruitment of a replacement Desk Officer who arrived in New York 2-months into the response and was operational soon afterwards, coupled with management changes at field level

Evaluation of ACF’s Response to the Pakistan Floods 18 there was a significant increase in staff capacity which led to improvements in both the management and communication of the response. There was also significant improvement in the flow of communications which facilitated the targeting of support from the members.

The HR Contribution Made by the Network During the second phase of the response, the majority of the field staff continued to be deployed from the New York office. Of the 16 field staff who arrived in Pakistan between the beginning of October and the end of January, 10 were deployed from the ACF-US office (3 were from ACF-France, 2 were from ACF-Spain and 1 was from ACF-UK).

5.3 RECOMMENDATIONS: Human Resources . The decision to make management changes in the Pakistan programme took too long. Where ACF has a long-term programme in a country it is strongly recommended that discussions are held with the Country Director about the size of programme that they have the capacity to manage with a ‘step aside’ policy initiated if a disaster occurs that requires a larger response; . There was a lack of clarity about HR needs. It is essential that an HR plan is devised at the earliest opportunity and communicated across ACF International. The revised EMS stipulates that an initial HR plan is drafted within 7-days. It is recommended that Members start checking their rosters and clarify the status of their emergency pool members within 24-hours of an emergency so that tangible offers of support can be made within 48-hours of a crisis occurring; . There are a number of strategic HR recommendations for ACF-US including the need to (i) strengthen the humanitarian skillset of the ACF-US roster, (ii) increase the number of experienced staff deployed to decrease the proportion of first missions, and (iii) increase the average length of deployments. With the HR team in New York in flux it is recommended that these issues should be high on the agenda of the incoming HR Director; . Once the ACF-US Desk Officer had been deployed to Pakistan there was a bottleneck which constrained the flow of communications and information between ACF International members and the field team. It is recommended that when a managing member assesses staffing capacity after an emergency strikes that it takes into account needs at headquarters in addition to the field.

6. Logistics

6.1 Phase 1 The existence of a pre-floods presence and programme in Pakistan, albeit limited in size, provided a platform of sorts from which to launch the humanitarian programme. The small size of the WB- funded programme had led to a decision to recruit a 3-month logistics coordinator to establish systems, train staff and undertake some of the initial procurement. Once preparations had been made it was anticipated that there would be sufficient capacity in the team. There was a significant setback with the resignation of the Thatta logs team shortly before the floods but this was supplemented by the recruitment of a logistics coordinator to start rebuilding the team and to start preparations for the pre-flood ECHO-funded KPK programme.

While it was anticipated that a team of 2-staff would be adequate to launch an initial response this wasn’t the case. With the need to quickly scale-up the programme, skills and capacity gaps became increasingly evident which led to a decision to strengthen the team further. Decision-making and process failures frustrated efforts to do this with the result that it took considerable time to establish a stable International logistics team.

Evaluation of ACF’s Response to the Pakistan Floods 19

There are a number of factors which contributed to the failure to deploy the right staff with the right skills at the right time. Chief amongst these (and documented and analysed in section 3.5 above) was the lack of suitable, available candidates in the ACF-US pool. While there was a successful deployment from the ACF-France pool, given the significant resources of the network this represents a shared failure. Even when staff were deployed compromises had to be made in terms of profile and experience; given the urgent need to deploy staff to scale-up the programme this might have been justified but it did limit the capacity of the team and doubtless impacted on the speed with which logistics tasks were undertaken. This decision had the effect of placing an even greater burden on the few experienced and high-performing staff and is thought to have contributed to early burn- out of some members of the team.

The task of growing ACF’s presence in Pakistan was frustrated in the first phase by the reticence of the country team to request assistance. This was a perception that the team over-estimated their capacity which took too long for Headquarters to challenge. The practical implications of this were that the organisation opened itself up to significant financial risk. Interviews suggest that control frameworks were inadequate in the early days and the impact of this is still being felt by the team.

Box 7: An Analysis of the Logistical Challenges Faced by Humanitarian Agencies in Pakistan

The lack of availability of certain relief items significantly slowed down efforts in a number of sectors, including shelter, water/sanitation and hygiene, and nutrition. A limited production capacity in national and local markets combined with global shortages of certain materials (related in part to the high demand for relief goods in Haiti) led to significant delays in the sourcing of key relief items such as tents, tarpaulins, water bladders, water purification tablets and nutritional supplements.

The key to delivering a timely response is having access to equipment and an offer was made early in the response (on 10 August) from ACF-France of NFIs which were en route to supplement contingency stocks held in Dubai. Interviews in New York showed a reticence to front-end costs for these due to cash-flow constraints; that the field team was also over-confident about their procurement capacity meant that the opportunity was turned down. With the humanitarian community struggling to meet the huge needs of flood victims and with many agencies having greater logistics capacity than ACF local supply was quickly exhausted and as a result it took many weeks to procure the same items that could have been shipped in days.

6.2 Phase 2 By the second phase of the response the team was far from being at full capacity and there continued to be issues with local staff turnover which saw considerable change. In fact wasn’t until December that national staff posts across the programme had been fully recruited. This considerably limited the team’s ability to scale-up programmes in the first months of the response.

One of the consequences of the slow scale-up was that service delivery struggled to keep pace with the needs of the team. Technical staff complained about the slow pace of procurement and an analysis of expenditure against the grants suggests there was a rush to procure project inputs towards the end of the year, shortly before donor contracts ended. While ACF has rigorous systems which have been rolled out across the Network and which have been proven to be robust these took time to be fully embedded in the programme and the lack of capacity in the team was an additional constraint to achieving timely procurement.

With the Logistics Coordination function filled with a long-term staff member in January and the field team capacity strengthened considerably there was a shift in the capacity and competence of

Evaluation of ACF’s Response to the Pakistan Floods 20 the team which has supported the scale-up in programme work although work continues to fill in some of the information gaps from the first days and weeks of the response.

6.3 Recommendations: Logistics . The inability of ACF-US and members more broadly to find logistics staff with the right skills at the right time slowed the speed of the response and placed the organisation at considerable risk. ACF needs to promote a zero tolerance approach to under-resourcing such essential parts of the programme and should determine a minimum level of risk beyond which a crisis management approach is adopted to securing the necessary resources.

7. Administration, Finance and Fundraising

7.1 Phase 1 Interviews suggest that prior to the floods a strong administrative team had been put in place and procedures had been rolled out throughout the programme. Administrative functions had been resourced with national staff that were considered to have the right skills and competencies. Financial reporting was undertaken in a timely way and reports that were submitted to New York were considered complete and to be of good quality. As the programme scaled-up access to additional administrative staff in the local market meant that this aspect of the programme ran relatively smoothly.

While systems and staffing were adequate funding was slow to arrive. The absence of field interviews makes it difficult to determine the reasons for this although the lack of donor funding in the early stages of the response is documented in the inter-agency evaluation.20 That the team struggled to recruit operations staff in adequate numbers early in the response also played a part in this (although the WaSH Coordinator was relatively quick to start assessments and to identify programmes). That the team were in the process of finalising a contract in KPK at the time that the floods happened meant that they were able to re-allocate some of this to the changed context was fortunate. The funding analysis in box 8 shows that apart from the first ECHO contract for KPK (which was dated 01 July although funding was used for floods response work), no further funding was secured until September. Even when lead times for assessments and preparation of proposals are taken into account, this is slow.

Box 8: Donor Funds Raised by Geographic Location and Date in Phase 1

KPK Province . ECHO, Integrated community based response for vulnerable conflict and floods affected populations in Northern West Pakistan, 01 July 2010 . ECHO, Emergency FSL support to flood affected populations in Pakistan, 01 September 2010 . ACF-France, Emergency FSL support to flood affected populations in Nowshera and Charsada Districts, KPK Province, Pakistan, 01 September 2010

Sindh Province . ECHO, Emergency Humanitarian WaSH Response for Vulnerable Flood Affected Populations in Thatta District, Sindh Province, Pakistan, 01 September 2010 . ACF-UK, Emergency Humanitarian WaSH Response for Vulnerable Flood Affected Populations in Sindh Province, Pakistan, 06 September 2010 . OCHA ERF, Emergency Humanitarian WaSH Response for Vulnerable Flood Affected Populations in Sindh

20 Polastro R et al (2011) Inter-Agency Real-time Evaluation of the Response to Pakistan’s 2010 Flood Crisis, DARA.

Evaluation of ACF’s Response to the Pakistan Floods 21

Province, in Pakistan, 15 September 2010 . ACF-France, Emergency Humanitarian WaSH Response for Vulnerable Flood Affected Populations in Thatta District, Sindh Province, Pakistan, 15 September 2010

While an offer of $225,000 unrestricted funds came from Paris on 10 August the funding didn’t arrive until early-September. This was then supplemented by additional funds raised by the members (ACF France and a small contribution from ACF-UK). That coordination between ACF International members wasn’t sufficiently strong to successfully secure such unrestricted funds more swiftly is disappointing. Furthermore, that there isn’t a formal facility for launching a joint appeal between ACF’s members supported by a process to rapidly advance unrestricted funds to the programme is a missed opportunity given the significant marketing and fundraising resources that ACF has and its significant global profile.

While the revised EMS highlights the need to clarify initial funding requirements within the first 24- hours which will provide an opportunity to draw from members’ unrestricted funds, the absence of an explicit coordination mechanism for fundraising at the ACF International level remains an important gap and it’s recommended that at the outset of an emergency a cross-member sub-group is formed to coordinate fundraising initiatives. While this will have time implications the benefits to members in terms of coordinating appeals, identifying donors and securing funds would likely far outweigh the costs and would also help ensure support from across the organisation.

7.2 Phase 2 By the start of the second phase of the programme considerable funding had been raised and the focus changed from active fundraising to programme delivery which was supported by all levels of management. An analysis of the spend rates for the programme confirm the success of the team in expanding programme delivery; in October and November spend rates had increased by 50%, reaching $600,000 with a significant further increase in the month of December to $1.7 million. While this reflected the urgency in delivering against several of the programmes which were due to end, it also shows the considerable determination which the team had to scale-up implementation. While there was a perception in the early parts of the second phase that the programme was administratively light, additional capacity was recruited into the team to support the significant increase in administrative workload.

Box 9: Donor Funds Raised by Geographic Location and Date in Phase 2

Sindh Province . CIDA, Emergency Humanitarian WaSH Response for Vulnerable Flood Affected Populations in Sindh Province, Pakistan, 15 November 2011 . CBHA/DFID, Early recovery (FSL) support to flood affected communities in Pakistan, 15 November 2010 . OCHA ERF, Emergency Humanitarian WaSH services response for the flood affected populations in Thatta district Sindh province, 28 December 2010 . ALAVI, Clean Water and sanitation for at-risk population in Sindh province, Pakistan, 01 January 2011

While the in-country administrative and financial capacity was considered adequate the New York office provided invaluable support to the process of writing and reporting on donor proposals. The relevant experience that the headquarters team had in writing ECHO, CIDA and USAID proposals meant that they were able to help steer the team through the process. To date it is only the ECHO contracts for which reports have been submitted and while a full expenditure analysis has not yet been carried out, that the contracts were spent out and that the reports were written and submitted on time is encouraging.

Evaluation of ACF’s Response to the Pakistan Floods 22

From a fundraising perspective the Pakistan floods response has been something of a success as box 10 indicates with considerable funds raised through a range of donors (both internal & external). ACF-France alone raised Euro 580,000 of which Euro 161,000 was raised online and a further Euro 420,000 from private donors. The fact that in the first 6-months over $9m has been secured with one-tenth of it raised through ACF International shows the strength of the Network and its ability to attract significant investment. It also adds weight to the case for ‘managed’ risk-taking to front-end unfunded programme work and the importance of strengthening cross-member coordination. Had the response been quicker to establish itself there is little doubt that the programme could have been larger still.

Some members spoke of their disappointment that the Box 10: Total Funds Raised by Pakistan response wasn’t larger; given the extent of the need, Donor to January 2011 the established presence ACF had in the country and in the wake of a successful and well-funded Haiti earthquake response programme21 some felt that the organisation should ECHO, $4,977,872 have had greater ambition. Members raised a variety of DFID/CBHA, $2,335,654 UN CERF, $1,000,000 concerns linked to the timeliness and scale of the early ACFIN Grants, $925, 870 programme which included organisational reputation, public Individuals, $96,983 perceptions and donor profile. For members who had global CIDA, $510,500 Cluster commitments, the lack of an ACF programme at the earliest stages of the programme was felt to negatively impact on the reputation of the organisation.

On reflection and in view of the difficulties experienced in scaling up the programme it would be difficult to advocate for a larger programme although with a strengthened EMS now in place there is justification for ACF International to scale-up beyond its current comfort levels in the future – although if this ambition is to be achieved it will be important to rebuild trust and strengthen collaboration between all the members.

7.3 Recommendations: Administration, Finance & Funding . The approach taken by ACF-US to scaling up the programme was cautious. A culture which supports managed risk-taking is a prerequisite for an organisation that prides itself on being a first-responder to a crisis. With the ACF-France unrestricted funding facility now part of the EMS and a greater commitment to early communication more likely to yield public donations it will be important for ACF to be more ambitious in their humanitarian response; . While the revised EMS provides an operational coordination forum it misses an opportunity to heighten fundraising and marketing coordination across Network members. By pooling talent and resources ACF International will be far better positioned to focus its support and benefit from cross-Network collaboration. While this currently happens informally it is recommended that a formal process is outlined in the EMS which seeks to determine priorities, fill capacity gaps and provide timely support to fundraising.

8. Monitoring and Evaluation

8.1 Phase 1 & 2 It is extremely difficult to assess the strength of ACF’s monitoring and evaluation practices in the absence of field work. However a review of the literature suggests that efforts were made to monitor outputs. For WaSH programmes a comprehensive database of all outputs by activity and

21 ACF raised Euro 21 million for the programme in Haiti.

Evaluation of ACF’s Response to the Pakistan Floods 23 geographic location was maintained and regularly updated. The summary document draws on a series of linked worksheets and while closer scrutiny of the document suggests that there might be some errors of inclusion the tool itself is good practice and represents an impressive commitment to documenting the achievements of the programme. There is no evaluative material per se although monthly sitreps and an extensive trip report following a visit by the New York-based WaSH Advisor provide evidence of oversight and analysis of the programme which made course corrections possible. It is noteworthy that the monitoring data is entirely focused on outputs rather than the achievement of minimum standards (be they Sphere or internal standards). This makes it impossible to make a judgement on the extent to which basic needs were met or the level of service provision provided by the field teams.

FSL has a comprehensive set of monitoring data including mid-term reviews, evaluations and learning data. While a significant proportion of this is from the PEFSA programme which was a joint initiative between ACF, ACTED, Care, International Rescue Committee (IRC), Oxfam and Save the Children, it does provide valuable real-time analysis and learning. Given the extent of the needs in Pakistan and the different approaches adopted towards collaborative ways of working, the PEFSA learning documents provide valuable reflections.

The lack of a field component and beneficiary survey work makes it impossible to comment on the extent to which those who received ACF’s assistance valued it or to make a judgment on the quality or quantity of items distributed however, a Post-Distribution Monitoring exercise undertaken for the KPK NFI distribution, cash grants and cash for work (CFW) programme carried out in September provides some feedback. While a 3-month delay in implementing this likely impacted on the quality of the data (an issue raised in the report) the methodology is sound and it goes some way to providing evidence of the effectiveness of the intervention.