INDEPENDENT FINAL EVALUATION JANUARY 2017

Reducing Vulnerability through Disaster Risk Management (DRM) in Khyber Pakhtunkhwa (KPK) Province, Pakistan - Phase 3

Funded by: The Royal Norwegian Embassy By : Zaki Ullah GLOW CONSULTANTS This report was commissioned by Action Against Hunger | ACF International. The comments contained herein reflect the opinions of the Evaluators only.

Action Against Hunger

Author / Contacts This independent final evaluation of “Reducing Vulnerability through Disaster Risk Management (DRM) in Khyber Pakhtunkhwa (KPK) Province, Pakistan - Phase 3” was commissioned by Action Against Hunger and was produced by GLOW Consultants Private Limited. For further information regarding this study please contact:

• Mr. Shahzad Ajmal ([email protected]) at Action Against Hunger Pakistan; or • Mr. Zaki Ullah ([email protected]) at GLOW Consultants.

Cover Page Photo

Evaluation team member meeting a community representative in Upper Dir.

GLOW CONSULTANTS (PRIVATE) LIMITED SECP Registration No: 0088603

Address: GLOW Consultants Private Limited, 4th Floor, Software Technology Park-1, F 5/1, Islamabad, Pakistan / Phone: +92-51-2828 948 / +92 345 85 75 974 Email us: [email protected]

i | P a g e R N E I I I D R R P r o j e c t E v a l u a t i o n R e p o r t

Action Against Hunger

Preface / Acknowledgements

GLOW Consultants is pleased to submit this evaluation report to Action Against Hunger Pakistan. The team would like to say thank you to Action Against Hunger team and other stakeholders whose support made this evaluation possible. The team would also like to acknowledge all those who contributed to our data collection efforts. Without their contribution, this evaluation would not have been possible.

Disclaimer

The views expressed in this evaluation report are those of the GLOW Consultants. They do not necessary represent those of Action Against Hunger or any other institution mentioned in the report.

Evaluation Team

• Mr. Zaki Ullah – Lead Evaluator • Mr. Saeed Ullah Khan - Co Evaluator • Ms. Aisha Batool – Technical Expert • Mr. Zia Ul Islam – Data Analyst

ii | P a g e R N E I I I D R R P r o j e c t E v a l u a t i o n R e p o r t

Action Against Hunger

Project Summary

Reducing Vulnerability through Disaster Risk Project Name Management (DRM) in Khyber Pakhtunkhwa (KPK) Province, Pakistan – Phase 3

Contract Number PAK 3044-14/0009

Partners (if applicable) NA

Districts - Charsadda, Nowshera, Dir Lower and Dir Upper. Location Province - Khyber Pakhtunkhwa Country - Pakistan.

Sector Disaster Risk Management (DRM)

27 Months (but actual implementation period is Duration 13 months)

Starting Date 1st October 2014

Ending Date 31st December 2016

Intervention / Country Office Language English

Donor and Contribution Royal Norwegian Embassy

Country Office administering the Action Against Hunger Pakistan Programme / Project

iii | P a g e R N E I I I D R R P r o j e c t E v a l u a t i o n R e p o r t

Action Against Hunger

Table of Content Author / Contacts...... i Preface / Acknowledgements ...... ii Disclaimer ...... ii Evaluation Team...... ii Project Summary ...... iii List of Acronyms ...... v List of Figures ...... vi List of Tables ...... vi Executive Summary ...... vii Section 1: Background...... - 2 - 1.1 Emergency Context in Pakistan ...... - 2 - 1.2 Overview of the Project...... - 2 - 1.3 Project Details ...... - 3 - Section 2: Evaluation Approach and Case Study Design ...... - 5 - 2.1 Evaluation Questions...... - 5 - 2.2 Evaluation Approach...... - 5 - 2.2.1 Review of Relevant Documents/Secondary Data ...... - 5 - 2.2.2 Primary Data Collection Tools...... - 6 - 2.2.3 Geographical Coverage ...... - 6 - 2.2.4 Group Discussions (GDs)...... - 8 - 2.2.5 Key Informant Interviews ...... - 9 - 2.2.6 Checklists ...... - 9 - 2.2.7 Data Analysis ...... - 9 - 2.3 Limitations of the Study ...... - 10 - Section 3: Key Findings...... - 11 - 3.1 Design...... - 11 - 3.2 Relevance/Appropriateness:...... - 14 - 3.3 Coherence...... - 18 - 3.4 Coverage...... - 19 - 3.5 Efficiency ...... - 20 - 3.6 Effectiveness: ...... - 22 - 3.7 Sustainability and likelihood of impact...... - 25 - Section 4: Significant Change Stories...... - 28 - Section 5: Conclusions...... - 31 - Section 6: Lesson Learnt...... - 35 - Section 7: Recommendations...... - 36 - Appendix...... - 39 - Annexure 1: Good Practice ...... - 39 - Annexure 2: Evaluation Criteria Table ...... - 41 - Annexure 3: List of documents for literature review ...... - 42 - Annexure 4: List of KIIs ...... - 42 - Annexure 5: Data Collection Tools...... - 42 - Annexure 6: Revised Logical Framework...... - 42 - Annexure 7: Evaluation ToRs ...... - 42 - Annexure 8: VDMC and Cluster Disaggregated Data ...... - 42 -

iv | P a g e R N E I I I D R R P r o j e c t E v a l u a t i o n R e p o r t

Action Against Hunger

List of Acronyms

CBDRM Community Based Disaster Risk Management CBOs Community Based Organizations CDPM Centre for Disaster Preparedness and Management DAC Development Assistance Committee DDMU District Disaster Management Unit DRM Disaster Risk Management DRR Disaster Risk Reduction ERT Emergency Response Team EWS Early Warning System GD Group Discussion IDPs Internally Displaced Persons INGO International Non Government Organization KII Key Informant Interview KPK Khyber Pakhtunkhwa NGOs Non-Governmental Organization NoC No Objection Certificate PCVA Participatory Community Vulnerability Analysis PDMA Provisional Disaster Management Authority PLW Pregnant and Lactating Women PWD Person with Disability RNE Royal Norwegian Embassy SBDRM School Based Disaster Risk Management SDMC School Disaster Management Committee ToRs Terms of Reference UCDMC Union Council Disaster Management Committee UC Union Council VDMC Village Disaster Management Committee

v | P a g e DRR Final Evaluation Report

Action Against Hunger

List of Figures

Figure 1: Group Discussions with the communities as part of this evaluation at 4475 ft above sea level in Upper Dir...... - 1 - Figure 2: Discussion with a key informant in Lower Dir ...... - 4 - Figure 3: Geographical Coverage ...... - 6 - Figure 4:Snapshots of CSPro and SPSS ...... - 10 - Figure 5: Evacuation Route Built by Action Against Hunger in Upper Dir...... - 10 - Figure 6: Awareness of Feedback Complaint Mechanism ...... - 12 - Figure 7: Usefulness of search and rescue kits and training in saving lives and property - Community ...... - 13 - Figure 8: Usefulness of search and rescue kits and training in saving lives and property - VDMCs ...... - 14 - Figure 9: Knowledge regarding hazards ...... - 15 - Figure 10: Usefulness of First Aid kit and First Aid training in saving lives - Community ....- 16 - Figure 11: Usefulness of First Aid kit and First Aid training in saving lives - VDMCs...... - 17 - Figure 12: SDMC Interviews and First Aid Kit in Charsadda...... - 19 - Figure 13: Identified safe place/evacuation center have the capacity to cater specific needs of women...... - 20 - Figure 14: Evaluation of DRR Infrastructure in district Charsadda ...... - 20 - Figure 15: Discussion with community in Upper Dir (L) and Nowshera (R) ...... - 21 - Figure 16: Emergency Response Kit and Early Warning System Upper Dir...... - 22 - Figure 17: First Aid Box and stock register...... - 23 - Figure 18: Action Against Hunger Supported Emergency Response Kit in RNE II...... - 27 - Figure 19: Discussion with a key informant in Lower Dir ...... - 30 - Figure 20: Emergency Response Kit Distributed During RNE III ...... - 34 -

List of Tables Table 1: List of Sampled Villages (RNE III)...... - 7 - Table 2: Cluster One and Two ...... - 7 - Table 3: List of Sampled Villages (RNE II) ...... - 8 - Table 4: Details of GDs conducted with General Community ...... - 8 - Table 5: Details of GDs conducted with VDMC...... - 8 - Table 6: Details of Key Informant Interviews ...... - 9 - Table 7: Details of Checklists ...... - 9 -

vi | P a g e R N E I I I D R R P r o j e c t E v a l u a t i o n R e p o r t

Action Against Hunger

Executive Summary

This evaluation report covers the Royal Norwegian Embassy (RNE) Islamabad supported “Reducing Vulnerability through Disaster Risk Management (DRM) in Khyber Pakhtunkhwa (KPK) Province, Pakistan - Phase 3” project implemented by Action Against Hunger. The project was implemented in Charsadda, Nowshera, Lower Dir and Upper Dir districts. The project started on 1st October 2014 and ended on 31st December 2016. Of these 27 months, actual implementation period was only nine months due to access and associated challenges.

The evaluation used the Development Assistance Committee (DAC) criteria for evaluating this project. The evaluation team independently selected 16 villages, four from each project district. Of these sites, three sites were from the current project (RNE III) while one site was from the Phase 2 of the project (RNE II)1, thus enabling the evaluation team to reflect on the success of both interventions and have an informed opinion on this current phase. The evaluation team collected data through a mixed methods approach including the Most Significant Change Technique. Data collection activities included: 32 group discussions with the general community and Village Disaster Management Committees (VDMC); 48 key informant interviews (KII) with government officials, community activists, the Action Against Hunger team, and other relevant stakeholders.

Of the seven DAC criteria, the project scored high on five of them. In terms of relevance, it was clearly evident that all Action Against Hunger activities addressed community needs, especially in the context of project areas given their high disaster vulnerability, particularly in relation to flood (Nowshera and Charsadda) and earthquake (Lower and Upper Dir), and more recently to forest fires, cloud bursts, land sliding amongst others (especially in Lower Dir). In terms of effectiveness, the project reached all of its project targets and the project had impact – on both community and institutional level - despite the challenges associated with access and the fact that the actual implementation period was only 9 months. In terms of sustainability, many of the activities are likely to continue beyond the project duration, although some of the activities may not sustain due to various factors e.g. lack of vertical linkages between VDMCs at Union Council (UC) level. From an efficiency perspective, the project used direct cash transfers to communities instead of other available options such as transfers through mobile phones. It was not possible to make an objective evaluation of whether or not self-transfer was a more efficient option compared to other e.g. use of mobile cash transfers. That said, the evaluators believe that the project was efficient overall as non-project support costs were lower due to no partners being engaged.

Recommendations

Increased Engagement with VDMCs: To ensure sustainability, Action Against Hunger should conduct follow up visits to mobilize VDMCs and Union Council Disaster Management Committees (UCDMCs) to have regular meetings, arrange refresher capacity building trainings.

1 RNE II and RNE III refer to Phase 2 and Phase 3 of the project respectively throughout the report.

vii | P a g e R N E I I I D R R P r o j e c t E v a l u a t i o n R e p o r t

Action Against Hunger

Horizontal and Vertical Linkages between VDMCs: Although the project was successful in establishing VDMCs and to some extent UCDMs, it is necessary to closely link them together i.e. VDMCs with UCDMCs or with District Disaster Management Unit (DDMU).

Cross Learning / Learning Exchange Activities: Currently there is limited cross learning between VDMCs. It is recommended to systematically encourage cross learning, learning exchange activities and information sharing amongst the VDMCs.

Training, Duration, Refreshers: Action Against Hunger should further increase this aspect of the project design as it proves to be an effective ‘gate opener’ to access women.

Including Additional Aspects in the Design: Although DRM plan development at village level was undertaken (however mostly undocumented) Action Against Hunger could consider additional aspects in the design of its future projects, such as hazard and risk maps at village level with proper documentation, and sharing these maps with the communities and government so that they can use them in their planning.

Institutional Strengthening for DRR: Government authorities have come to know Action Against Hunger as a reliable partner with highly skilled and experienced staff, and they would appreciate long-term collaboration with them in DRR, especially from a response perspective. Action Against Hunger could continue to work with them and cover some of their specific needs, such as provision of specialised boats in Lower Dir to enhance their response in case of an occurrence of any disaster.

Plantation Activities: Keeping in mind the relevance of this activity in the project design, this activity should be reintroduced in the future project designs.

Build on DRR Research: The project commissioned five different research studies through the University of Peshawar. Such research can influence policy level changes at district and provincial level and Action Against Hunger should consider further building on the key findings and recommendations from the research.

Support Academia for DRR Research and Creating Government-Academia linkages: Action Against Hunger may like to continue supporting this research and also create a research link between the Provincial Disaster Management Authority (PDMA) and other academic institutes.

Increasing Government Ownership and Profile Building: There is a need to engage other relevant government departments such as civil defence, health, etc. in the project activities.

Increasing Coverage of the Evacuation Routes: In future, the evacuation routes need to be designed and constructed in a manner to cover the needs of the entire village.

Evacuation Centres Better Equipped: The needs of Pregnant and Lactating Women (PLWs) and Persons with Disabilities (PWDs) should also addressed in terms of evacuation centres. In addition, a kit containing some basic items, e.g. water storage vessels with filtration capacity and basic personnel hygiene items could be stored at those centres.

viii | P a g e R N E I I I D R R P r o j e c t E v a l u a t i o n R e p o r t

Action Against Hunger

Figure 1: Group Discussions with the communities as part of this evaluation at 4475 ft above sea level in Upper Dir.

- 1 - | P a g e DRR Final Evaluation Report

Action Against Hunger

Section 1: Background

This section provides an overview of the emergency context in Pakistan, and the specific project implemented by Action Against Hunger with Royal Norwegian Embassy funding. 1.1 Emergency Context in Pakistan

Pakistan lies in a region prone to natural disasters, primarily from earthquake and flood. Over the last decade more than 80% of Pakistan’s 124 districts have been hit by natural disasters, with some 33 million people affected. Since 2005 the country has experienced two major disasters: the massive earthquake in 2005; and devastating floods in 2010, following the annual monsoon rains, which left one-fifth of the country under water. These disasters have compounded the problems of many vulnerable communities already facing hardship across Pakistan. Millions of people face poverty, food insecurity, limited public health provision and education shortfalls. In recent years climate change and environmental degradation, such as deforestation, or settlement on traditional flood plain areas, have resulted in increased exposure to extreme events. The geography of Pakistan makes access to some areas physically challenging, such as the isolated high altitude regions in the north and northwest, embracing Gilgit-Baltistan, Azad, Jammu and Kashmir, and KPK provinces, where infrastructure is poor. The region is burdened with an alarming and diverse spectrum of natural and human-induced disasters including an ongoing complex emergency. KPK also has the shortest warning time to respond to certain types of disasters. Millions of refugees and Internally Displaced Persons (IDPs) fleeing conflict have flooded into the province stretching the resources and capacity of the province beyond its limits. Increased efforts on a joint national and provincial basis are now focusing much more on DRR in recognition that reducing vulnerabilities and limiting the impacts of hazards beforehand can save lives and assets. 1.2 Overview of the Project

Action Against Hunger, being one of the leading International Non-Government Organizations (INGO) in the country, is mainstreaming DRR in their programs as well as managing and implementing standalone Community Based Disaster Risk Management (CBDRM) interventions, particularly in KPK. Building upon the previous work in the region ranging from assistance to Afghan refugees in 1979 to providing humanitarian response in 2010 and subsequent floods has enabled the organization to extend its services from its core areas of food security, livelihood, water, sanitation and hygiene, and link them with disaster preparedness and resilience.

Keeping in view the abovementioned vulnerabilities and hazards in the region, Action Against Hunger International designed and implemented CBDRM projects for the last three years in Nowshera, Charsadda, and Lower Dir and Upper Dir districts in KPK province, which were funded by RNE. Those projects provided services to over 57,000 beneficiaries in 2012 and 60,000 beneficiaries in 2013/2014.

Building upon the lessons learned and best practices of these interventions, Action Against Hunger designed a third phase for the project. A baseline study was carried out in September 2013 to assess the approaches and knowledge of communities towards disaster preparedness and

- 2 - | P a g e R N E I I I D R R P r o j e c t E v a l u a t i o n R e p o r t

Action Against Hunger

resilience. This study also aimed to determine the attitude and practices adopted by communities and their level of information about the disaster management committees and safe places. In addition to this disaster planning, which is crucial in preparedness, was one of the key factors studied under the baseline and it was evaluated whether these communities were covered under the district disaster management plans.

During Phase 3, Action Against Hunger aimed to develop local capacity and introduce risk mitigation in the intervention areas. The project was a multidisciplinary initiative that aimed to involve a variety of stakeholders including ministries, local authorities, disaster management agencies, NGOs, the academic sector and communities. The direct beneficiaries of the project were the vulnerable populations affected by natural and manmade disasters in Nowshera, Charsadda, and Lower Dir and Upper Dir districts which are vulnerable to both serious floods and earthquakes. The basic premise of the project was that communities generally understood local realities and contexts better than outsiders. If ‘at-risk’ groups were involved in all stages of the disaster risk management, a stronger foundation would be created for the development of sustainable programs for risk reduction. Through the use of participatory techniques for data collection and analysis, hazard analysis, and trainings, Action Against Hunger aimed to ensure full participation of the communities. 1.3 Project Details

Project General Objective: To continue reducing the vulnerability and improve the resilience to disaster of KPK province, Pakistan.

Specific Objectives: To enable communities to prepare for and mitigate the impacts of disasters using participatory methods aimed at strengthening the self-reliance capacities of the population and reinforcing or adapting infrastructure.

Results 1. Building Community Disaster Resilience through Community Based Disaster Risk Management plans, preparedness and mitigation measures. 2. Strengthening institutional linkages and capacities for Disaster Resilience.

Activities 1.1 : Baseline and End line survey. 1.2 : Formation and training of VDMC. 1.3: Development of UCDMC Structures. 1.4 : Capacity building of local technicians and artisans. 1.5 : Formation and training of School Disaster Management Committee (SDMC) training. 1.6 : CBDRM sessions at community level. 1.7 : Provision of first aid, search & rescue kits to VDMCs and SDMCs. 1.8 : Facilitate the community to improve the disaster resilience infrastructures. 1.9: Establishment/ improvement of community early warning system.

- 3 - | P a g e R N E I I I D R R P r o j e c t E v a l u a t i o n R e p o r t

Action Against Hunger

Figure 2: Discussion with a key informant in Lower Dir

- 4 - | P a g e R N E I I I D R R P r o j e c t E v a l u a t i o n R e p o r t

Action Against Hunger

Section 2: Evaluation Approach and Case Study Design

At the end of 2016, Action Against Hunger commissioned an evaluation of the RNE funded DRM project activities in KPK. Action Against Hunger subscribes to the DAC criteria for evaluation: Impact, Sustainability, Coherence, Coverage, Relevance / Appropriateness, Effectiveness and Efficiency. Action Against Hunger also promotes systematic analysis of the monitoring system and cross cutting issues such as environment, gender equality, coordination, advocacy and participation. These dimensions served as the main evaluation criteria. The evaluation team used a mixed methods approach and Most Significant Change Technique collecting both qualitative and quantitative information from targeted household and communities. 2.1 Evaluation Questions

The evaluation used the questions provided in the Terms of Reference (ToRs), which listed them under the following evaluation criteria:

• Design • Relevance / Appropriateness • Coherence • Coverage • Efficiency • Effectiveness • Sustainability and Likelihood of Impact

2.2 Evaluation Approach

In order to obtain the information required to evaluate the project progress, different methods were applied to ensure comprehensiveness and quality of data and to cross-check findings from other methods. The final evaluation used qualitative and quantitative tools to collect information from representatives of community, teachers, students and other stakeholders. In addition, the consultant team considered gender, vulnerability, the participation of children and youth, and poor women in the project evaluation process.

Following is a brief description of the methodologies applied to collect data. 2.2.1 Review of Relevant Documents/Secondary Data

Prior to the field work, the evaluation team reviewed the relevant documents, including those related to this project, Action Against Hunger’s general work in Pakistan and DRM globally, and documents related to DRM and emergencies in Pakistan in order to gain a better understanding of the project and its context. This helped in developing the evaluation methodology and instruments. Documentation that was made available included the project proposal, the project logistical framework, activity reports, project reports, research studies, baseline and end line reports, mason training report, and a list of the project targeted villages. The evaluation team carried out a desk study of the available data and reports and developed tools accordingly. The

- 5 - | P a g e R N E I I I D R R P r o j e c t E v a l u a t i o n R e p o r t

Action Against Hunger tools were developed taking into account the mixed method approach (quantitative and qualitative), which coupled with secondary data assisted in the triangulation of data. 2.2.2 Primary Data Collection Tools

Group Discussions (GD) tools were used with general communities and VDMCs. The tools captured both quantitative and qualitative data. This exercise was carried out separately with males and females at each sample village level.

• Key Informant Interviews (KIIs) with stakeholders such as SDMCs, masons / artisans, VDMC and UCDMC members and relevant officials. • Observation Checklists for inspection of structural DRR measures, Early Warning System (EWS), Emergency Response Team (ERT) kit, Search & Rescue kit and First Aid kit. • Most Significant Change Techniques (Stories of Change).

2.2.3 Geographical Coverage

The overall targeted geographical areas were comprised of four districts of KPK including the districts of Nowshera, Charsadda, Upper Dir and Lower Dir (see map below).

Figure 3: Geographical Coverage

- 6 - | P a g e R N E I I I D R R P r o j e c t E v a l u a t i o n R e p o r t

Action Against Hunger

A total of 16 villages were selected for inclusion in the evaluation. To better achieve the intended objectives of the assignment, alongside 12 villages from the RNE III, one village from each of the districts was also selected from RNE II. As RNE III was designed on the basis of lessons learned and were the continuation of RNE II, which was completed in 2014, the impacts and sustainability-related findings can contribute to understand the findings for RNE III.

For selection of the RNE III villages, to achieve a better geographic spread across the project intervention area, a total of 12 villages from eight UCs in the four districts of Nowshera, Charsadda, Upper Dir and Lower Dir were randomly selected. A detailed breakdown of the selected sample areas is provided below:

Table 1: List of Sampled Villages (RNE III) S.No District Tehsil UC Village name 1 Shams abad Lal Qilla Lal Qilla 2 Dir Lower Manyal 3 Timergara Khungi SarPato 4 Palam Palam 5 Dir Upper Dir Upper Batal 6 Tarpatar Galkor 7 Khat Korona Hisar Yasinzai 8 Charsadda Charsadda Gogoshto 9 Agra Veeno ghari 10 Kabul River Ismail Abad 11 Nowshera Nowshera Najeem Abad Pir Payee 12 Laiqat Abad

Based on the similarity in the demographics and terrain, for data analysis purposes the above districts have been merged into two clusters as detailed below:

Table 2: Cluster One and Two Dir Lower Cluster One Dir Upper Charsadda Cluster Two Nowshera

- 7 - | P a g e R N E I I I D R R P r o j e c t E v a l u a t i o n R e p o r t

Action Against Hunger

Similarly for the RNE II villages, a total four villages were selected randomly in each RNE II intervention district. Details are as below:

Table 3: List of Sampled Villages (RNE II) S.No District Tehsil U/C Village name 1 Dir Lower Timergara Rabat Munjai 2 Dir Upper Dir Upper Chaper Cham 3 Charsadda Charsadda Batagram Mased Zai 4 Nowshera Nowshera Chowki Drub Chowki Drub

2.2.4 Group Discussions (GDs)

The evaluation team conducted a total of 32 GDs in selected villages (16 in cluster one and 16 in cluster two) having equal representation from male and females from the general community. The table below provides an overview of the GDs undertaken with gender wise segregation of data.

Table 4: Details of GDs conducted with General Community District # of GDs conducted # of Participants Male Female Total Male Female Total Nowshera 4 4 8 16 19 35 Charsadda 4 4 8 21 17 38 Lower Dir 4 4 8 19 22 41 Upper Dir 4 4 8 22 21 43 Grand Total 16 16 32 78 79 157

Similarly, a total 32 GDs were conducted with members of VDMCs (16 with male and 16 with female VDMC members) in sample villages. The table below provides an overview of group discussions held with VDMC members.

Table 5: Details of GDs conducted with VDMC District # of GDs conducted # of Participants Male Female Total Male Female Total Nowshera 4 4 8 13 23 36 Charsadda 4 4 8 19 23 42 Lower Dir 4 4 8 22 19 41 Upper Dir 4 4 8 16 16 32 Grand Total 16 16 32 70 81 151

- 8 - | P a g e R N E I I I D R R P r o j e c t E v a l u a t i o n R e p o r t

Action Against Hunger

2.2.5 Key Informant Interviews

KIIs were carried out with primary stakeholders including Action Against Hunger staff members, UCDMC members, school teachers (SDMC members), masons / artisans, and government officials. The following table shows the detailed of KIIs with various stakeholders:

Table 6: Details of Key Informant Interviews Action Against Hunger Staff including Technical Advisor (DRR and WASH), Field 8 Co, Logistics Co, Deputy Program Manager etc.(see details in annex) Government Officials including Assistant Commissioner (AC), Education 9 Department, PDMA, Center for Disaster Preparedness and Management (CDPM) SDMCs / teachers 9 Masons / artisans 12 UCDMCs 9 DRR Forum 1 Total 48 2.2.6 Checklists

The field team also designed field observation checklists for DRR structural measures, First Aid kits, ERT Kit, Search & Rescue kits and EWS. The information gathered through the checklists helped in the triangulation of data gathered through other tools. The table below shows the number of each type of checklist used during the evaluation.

Table 7: Details of Checklists Observation Checklist for Community DRR Infrastructure 16 Observation Checklist for Early Warning System 16 Observation Checklist for Emergency Response Kit 16 Total No of Checklists 48 2.2.7 Data Analysis

After collecting data using the abovementioned tools, quantitative data of GDs was entered in CSPro 6.0 and analyzed in SPSS. The data gathered through checklists was entered in excel spreadsheets. Cross tabulation was used to compare clusters as well as for the overall situation. Secondary data helped validate primary data. The information gathered through GDs, in-depth interviews with key informants and physical observation was compared, triangulated and analyzed during the preparation of the report.

- 9 - | P a g e R N E I I I D R R P r o j e c t E v a l u a t i o n R e p o r t

Action Against Hunger

Figure 4:Snapshots of CSPro and SPSS 2.3 Limitations of the Study

Closure of the schools due to winter holidays was one major limitation which hindered the evaluation team's ability to interact with teachers and students. To overcome this issue, the team used project focal points in the communities to identify the students and teachers who were living within the sample villages and met with them.

The project activities were only recently completed, making the evaluation of the project in relation to sustainability and impact challenging. To overcome this constraint a few villages from RNE II interventions were added to the overall RNE III evaluation scope.

Figure 5: Evacuation Route Built by Action Against Hunger in Upper Dir

- 10 - | P a g e R N E I I I D R R P r o j e c t E v a l u a t i o n R e p o r t

Action Against Hunger

Section 3: Key Findings

This section provides the findings according to key evaluation questions and criteria. The findings under each sub-section are based on the triangulation of information from the various sources and tools. 3.1 Design

The project design was in line with the objectives of the PDMA road map, which was to build local capacity and community resilience through community based disaster risk management. The design also took into account the previous RNE project evaluation recommendations to consider local capacities of the communities in the project’s resilience approach. The evaluation found the project to be relevant and adequate and according to the needs of the targeted population. More specifically, the design of the project worked to: enable communities to prepare for and mitigate the impacts of disasters; was based on participatory methods to strengthen the self-reliance capacities of the population and reinforce or adapt infrastructure; worked towards building community disaster resilience through CBDRM; and strengthened institutional linkages and capacities for disaster resilience.

The design of the project envisioned an active role of provincial and district disaster management departments and they were given leading role in the selection of vulnerable districts and villages. Thus, two selected UCs per district and subsequently five villages per UC were identified in consultation with PDMA representatives. Thus, the 40 most vulnerable villages were selected from a list of 75 villages in eight UCs based on set vulnerability criteria.

The starting point of Action Against Hunger’s work for this project was the Participatory Community Vulnerability Analysis (PCVA) exercise. This exercise aimed to assess each village’s vulnerability and possible steps to reduce it through the eyes of the community by using participatory methods to understand the main hazards, community strengths and weaknesses, and possible remedial measures for strengthening the community’s resilience. The use of such an approach helped Action Against Hunger to adopt a highly participatory process and identify the main components of the project in light of community perspectives.

The project had a balanced approach with an appropriate combination of soft and structural activities, which contributed towards the community resilience. Furthermore, provision of search and rescue and first aid kits also enabled community members to prevent and mitigate possible emergencies and disasters.

The establishment and capacity building of VDMCs equipped the target villages with the required skills and knowledge. These VDMCs, once trained on disaster preparedness and risk management, developed their village disaster management plans. These plans were developed in a participatory manner where hazards, risks, vulnerabilities and capacities of communities were assessed. Evacuation routes and safe places were also identified in these plans. EWS, Search and Rescue, and First Aid Kits were also included and provided to communities and schools as prescribed in the project design. Monitoring of irrigation channel construction was also ensured by VDMCs. VDMC and community members helped Action Against Hunger in the

- 11 - | P a g e R N E I I I D R R P r o j e c t E v a l u a t i o n R e p o r t

Action Against Hunger identification of evacuation routes and irrigation channels. Infrastructure was mainly built on identified paths and channels.

The inclusion of locally trained human resources was ensured in the design by undertaking capacity building interventions for local artisans on DRR resilient techniques. Communities could then engage these artisans for the construction of disaster resilient structures in the communities. Other capacity building interventions pertaining to School Based Disaster Risk Management (SBDRM) and CBDRM were included to enhance knowledge and coping skills of communities and schools in the area.

The project was an amalgamation of community based disaster management and promotion of research on DRR related themes. In this context, research studies through CDPM were made part of the project design. Promotion of DRR was included in the project through conduction of exhibitions.

Gender mainstreaming was included as women, being the vulnerable group, need to be equipped with knowledge and skills that can be utilized prior to, during and / or after disasters to mitigate its adverse effects. The representation of women in VDMCs and first aid training was designed to build their coping mechanisms and reduce their vulnerabilities. Linkages of VDMCs and UC DMCs were developed with relevant government line departments including DDMU.

Another important feature in the project design was a feedback complaint mechanism and raising community awareness around it. Accountability of project staff and contractors was ensured by placing complaint boxes in public areas, including hujras and mosques, etc. Alongside complaint boxes, banners were also displayed in various corners of the villages having contact numbers for relevant authorities. In the figure below, level of awareness found in communities and VDMCs regarding the feedback complaint mechanism is shown.

Awareness of Feedback Awareness of Feedback Complaint Mechanism Complaint Mechanism (Community) (VDMC)

18% 29% Yes Yes No No 71% 82%

Figure 6: Awareness of Feedback Complaint Mechanism

As per Group Discussions, 82% VDMC members and 71% general community were aware of the feedback complaint mechanism in their localities. In addition, KIIs with UCDMCs and

- 12 - | P a g e R N E I I I D R R P r o j e c t E v a l u a t i o n R e p o r t

Action Against Hunger

SBDMC also confirmed that a feedback complaint mechanism was present during the project execution.

Another useful aspect of the project design was the search and rescue kits and training. Almost all people found it useful, especially the kits. From a gender perspective, women were less exposed to the kit as compared to the training, and therefore women in both the general community and VDMCs ranked training more highly than the kit, which were mostly accessible to men only. In Figure 5 and 6 below the perceived usefulness of search and rescue kits and training is shown according to the general community and to the VDMCs.

Cluster One (Community): Did search Cluster One (Community): Did and rescue kits help you to save lives search and rescure training help and property you to save lives and property

94% 100% 90% 100% 84% 71% 80% 80% 67% 56% 60% Yes 60% 44% Yes No 40% 29% No 40% 21% Don't Know 20% 12% Don't Know 16% 0 0 20% 6% 0% 0 0 3%7% 0% Male Female Total

Cluster Two (Community): Did Cluster Two (Community): Did search and rescue kits help you to search and rescure training help save lives and property you to save lives and property 100% 100% 100% 79% 79% 80% 80% 69% 53% Yes 60% 47% Yes 60% 49%51% No 40% 31% No 40% Don’t Know Don’t Know 21% 21% 20% 20% 0% 0%0% 0% 0% 0% 0% 0% 0% Male Female Total Male Female Total

Figure 7: Usefulness of search and rescue kits and training in saving lives and property - Community

- 13 - | P a g e R N E I I I D R R P r o j e c t E v a l u a t i o n R e p o r t Action Against Hunger

Cluster One (VDMCs): Did search Cluster One (VDMCs): Did search and and rescue kits help you to save lives rescure training help you to save lives and property and property

100% 90% 92% 89% 100% 89% 91% 90% 80% 80% Yes Yes 60% 60% No 40% No 40% Don’t Don’t 20% 7% 8% 11% Know 3% 20% 11% 9% 0% 0% Know 6%4% 0% 0% 0% Male Female Total 0% Male Female Total

Cluster Two (VDMCs): Did search Cluster Two (VDMCs): Did search and rescue kits help you to save lives and rescure training help you to save and property lives and property 100% 100% 100% 100% 100% 100%

80% 80% Yes Yes 62% 59% 60% 60% No No 36% 40% 29% 40% Don’t Don’t Know 20% 20% 9% 5% Know 0%0% 0%0% 0%0% 0%0% 0% 0% Male Female Total Male Female Total

Figure 8: Usefulness of search and rescue kits and training in saving lives and property - VDMCs 3.2 Relevance/Appropriateness:

According to GDs and KIIs with different stakeholders, beneficiaries were satisfied and found the project both relevant and appropriate for the local social and cultural context. The evaluation team also found that the project was aligned with local needs and priorities, and satisfied the needs of targeted geographic areas. The intervention was in line with government priorities and the PDMA road map, with adherence to Do No Harm, Action Against Hunger’s country strategy, and donor’s agreement. Moreover the intervention was also in line with the National Disaster Risk Reduction policy; Action Against Hunger DRM policy; Action Against Hunger emergency preparedness and response planning; and the Hyogo Framework for Action 2005-2015.

- 14 - | P a g e RNEIII DRR Project Evaluation Report Action Against Hunger

As well, the project was found to be relevant with targeted geographical areas and targeted communities; the relevant sector with which to engage was identified; community driven appropriate approaches were adopted; and relevant soft and hard activities were carried out. The communities and VDMC members were asked, if the project has helped in identification of hazards. The results are provided in the below graph.

Figure 9: Knowledge regarding hazards

As per data analysis of GDs, the VDMCs were found to have greater knowledge about hazards compared to the general community. The VDMCs members also categorized various disasters that happened in their area according to their nature, i.e. natural or man-made. During data analysis, it was revealed that the key hazards were floods, heavy monsoon rains, storms, earthquakes, soil erosion, forest fire, siltation and land sliding. In some cases, respondents also ranked human-induced disaster and hazards as more catastrophic with adverse effects such as militancy.

The relevance of the project was also reflected in the importance given by the community to the first aid kit and first aid training. When communities were asked about the relevance of first aid kits - in cluster one (Lower and Upper Dir), 91% of the male community members found it useful as compared to 80% of females, whereas 63% of men and 80% of women said they believed first aid training helped in saving lives. The percentage of men and women reporting usefulness in general was lower for cluster two (Nowshera and Charsadda districts). This can be primarily due to the fact that communities in cluster two already have access to good health facilities (please refer to graphs above).

- 15 - | P a g e RNEIII DRR Project Evaluation Report Action Against Hunger

Cluster One (Community): Did First Cluster One (Community): Did First Aid Kit help you to save lives Aid Training help you to save lives

91% 100% 86% 100% 80% 80% 80% 80% 70% Yes 63% Yes 60% 60% No No 40% 40% 28% Don't Don't 20% 20% 16% 10% Know 14% Know 20% 6% 20% 9% 3% 0 3% 0 0% 0% Male Female Total Male Female Total

Cluster Two (Community): Did First Cluster Two (Community): Did First Aid Kit help you to save lives Aid Training help you to save lives

100% 86% 100% 85% 82% 80% 72% 78% 80% Yes 59% Yes 60% 60% No No 40% 28% 40% Don’t 21% Don’t 22% 18% Know 14% 14% Know 15% 20% 7% 20% 0 0% 0% 0% 0% 0% Male Female Total Male Female Total

Figure 10: Usefulness of First Aid kit and First Aid training in saving lives - Community

For the VDMCs, the percentage of women who reported both the kits and training useful were significantly higher than women in the general community, especially in cluster one.

- 16 - | P a g e RNEIII DRR Project Evaluation Report Action Against Hunger

Cluster One (VDMCs): Did First Cluster One (VDMCs): Did First Aid Aid Kit help you to save lives Training help you to save lives 100% 94% 100% 92% 100% 90%

80% Yes 80% 64% 60% No 60% Yes Don’t 40% 40% No 29% Know 17% 15% 20% 10% 20% 7% 8% 8% 6% 0% 0% 0%0% 0% 0% 0% Male Female Total Male Female Total

Cluster Two (VDMCs): Did First Cluster Two (VDMCs): Did First Aid Aid Kit help you to save lives Training help you to save lives 100% 100% 93% 93% 100% 88% 100% 88%

80% 80% Yes Yes 60% 60% No No 40% 40% Don’t Don’t Know Know 12% 12% 20% 7% 20% 7% 0%0% 0% 0% 0%0% 0% 0% 0% 0% Male Female Total Male Female Total

Figure 11: Usefulness of First Aid kit and First Aid training in saving lives - VDMCs

The baseline survey of the project indicated a dire need of first aid kits, search and rescue kits, and early warning kits in communities. In the absence of any substantial financial resources, communities were unable to purchase these items. The vulnerability of these communities was compounded with the absence of any reliable source of information which can intimate them beforehand about disasters. With the provision of aforementioned kits, communities stated that they were now capable of saving lives in a timely manner.

In addition to this, the targeted population described the construction of evacuation routes and irrigation channels as appropriate and according to the needs of community. Community members believed that these routes will assist them to reach a safe place in case of a disaster and hence will save the lives of their families. The capacity building of masons on disaster resilient techniques and good construction practices also served as a technical local resource in the communities.

- 17 - | P a g e RNEIII DRR Project Evaluation Report Action Against Hunger

3.3 Coherence

The project was found to be coherent with government interventions and policies and strategies of Action Against Hunger International. The project activities were based upon needs and implemented in a systematic way to achieve the overall project goal. The project maintained a close collaboration and liaison with relevant stakeholders in all areas at the design and implementation stage. The activities designed were in line with the national and international frameworks such as the Hyogo Framework for Action, the national DRR policy, the national disaster management framework, PDMA KPK roadmap, and the DRM policy of Action Against Hunger. Building the capacities of communities and engaging them in disaster planning increased their resilience and contributed to the provincial disaster roadmap developed by PDMA. The project established a viable partnership with government departments, especially with academia, with CDPM being engaged to carry out research on the key themes of DRR.

Both the country strategy and the DRR strategy of Action Against Hunger focus on enhancing the resilience of local communities by engaging them from the onset of the project thus ensuring their participation and ownership. This particular project was developed by building upon previous progress in this area with key components derived from the DRR strategy. These two aforementioned strategies served as guiding documents for designing and implementing the project. Communities, being the first responders during disasters, need to enhance their preparedness to mitigate the adverse effects. Well-equipped communities with trained human resources and stockpiles can utilize their capacities to make their village and area safe. Action Against Hunger, having already implemented two phases of this project in the past, were well aware of the challenges and ground realities and incorporated these into the third phase planning and design.

The coordination with primary and secondary stakeholders was effectively maintained from the onset of the project. Starting from provincial level, the engagement of PDMA included the identification of vulnerable areas and later the facilitation of research by university students which will be published by PDMA. Regular information sharing was maintained to avoid duplication of activities. At the district level, the development of a disaster management plan for Charsadda was carried out in close collaboration with relevant authorities. At village level, communities and village disaster management committee members took a lead in disaster planning after attending relevant training programs and identified safe routes and places for safe havens. However, during KIIs it was revealed that female participation in the identification of routes and safe places was less than men because of local cultural norms and restraints. The first aid training was found to be more effective and appreciated amongst women as they are the primary caregivers at home and in case of minor injuries they are now in a position to provide immediate treatment. Similarly, the training of school teachers and subsequent distribution of first aid kits results in less injuries of pupils.

- 18 - | P a g e RNEIII DRR Project Evaluation Report Action Against Hunger

Figure 12: SDMC Interviews and First Aid Kit in Charsadda 3.4 Coverage

During the interview conducted with PDMA it was revealed that the government priority regarding DRR is shifting from Peshawar city towards northern areas of KPK. Provincial government is requesting organizations to shift their resources towards northern areas of KPK. Therefore, in line with the government priority area, Action Against Hunger selected two districts from northern areas including Lower Dir and Upper Dir where needs were comparatively high compared to other parts. In addition, Action Against Hunger selected villages from district Nowshera and Charsadda that were affected by flash floods. PCVA criteria were followed for village selection in both cluster one and two. The 40 most vulnerable villages (five villages per UC) were selected in eight UCs after the village profiling exercise was completed in more than 75 identified villages based on vulnerability criteria. In order to analyze the potential hazards, vulnerabilities and the existing capacities of the targeted communities, PCVA was conducted in all 40 villages. The findings of the PCVA were considered and incorporated during the implementation of the project activities. Although Action Against Hunger provided kits to communities and schools and imparted knowledge on disaster preparedness, keeping in mind the vast need on the subject more substantial efforts needs to be carried out as this was also reiterated from the evaluation results.

In GDs it was revealed that the provision of ERT, EWS, search and rescue and first aid kits were much needed equipment in the community. However, keeping in view the compounded vulnerability of the areas more efforts are needed. The knowledge retention among communities on the subject was not sufficient. This was observed and verified from the evaluation results given below that in the absence of comprehensive refreshers for communities on disaster management, community members lacked the knowledge on key messages that were delivered to them earlier in the trainings. Furthermore, in line with previous evaluation experiences in other projects, beneficiaries have better retention of key messages when messages are delivered multiple times.

- 19 - | P a g e RNEIII DRR Project Evaluation Report Action Against Hunger

Identified safe place/evacuation Identified safe place/evacuation center has the capacity to cater center has the capacity to cater specific needs of women specific needs of women (Community) (VDMC)

9% 21% Yes Yes 43% 27% No No Don't Know 64% Don't Know 36%

Figure 13: Identified safe place/evacuation center have the capacity to cater specific needs of women

Furthermore, the DRR structural measures such as evacuation routes and irrigation channels were among the most important needs of the communities identified, yet these structures were unable to cater for the collective needs of communities during a disaster. During GDs and KIIs, communities mentioned that evacuation routes were insufficient in times of disasters and evacuation centers were also not serving the needs of the community. As per KIIs with UCDMCs, evacuation centers have no special arrangements for women and people with special needs. This was confirmed by members of both the general community and VDMCs.

Figure 14: Evaluation of DRR Infrastructure in district Charsadda 3.5 Efficiency

It is difficult to measure and compare cost efficiency of the project with other similar programs / projects, as data of other organizations is not publicly available. Similarly, the cost of kits cannot

- 20 - | P a g e RNEIII DRR Project Evaluation Report Action Against Hunger be compared with other organizations implementing DRR projects in different parts of Pakistan. A generic analysis, however, indicates that all targets were achieved on time as per progress reports of RNE III and also validated by beneficiaries during GDs and KIIs. Based on the available information, the project achieved its results efficiently as per the approved budget. Moreover, it is also true for the community physical infrastructure and utilization of local resources. Overall the budget was utilized as per the approved budget, indicating good programme planning and budget.

Figure 15: Discussion with community in Upper Dir (L) and Nowshera (R)

Within a given geographical area and short period of intervention time, Action Against Hunger hired more staff on the ground to deliver services on time and ensure a high quality of work. Action Against Hunger staff carried out a comprehensive PCVA exercise and identified vulnerable villages. On the other hand, formation of VDMCs and UCDMCs added to efficiency whereby response capacity at the grass root levels was created with scarce resources.

Ninety percent of the beneficiaries revealed during GDs and key informants during KIIs that project support was received in an efficient manner across all the targeted sectors. However, there were some delays in payments to the skilled and unskilled labour engaged in the project. This was mainly due to the approval process, which required approval of payments by the signing authorities based in Islamabad; payments were sent to the Islamabad office for approval before issuing cheques or cash payments to the skilled or unskilled labour. As well, the payments were not standard (unlike cash disbursement in food security project per household of PKR 7,500) as the payment depended on the individual engagement in the project. However, the beneficiaries were generally satisfied from with the payment modality and also did not share concerns regarding differences in payments (as they understood that payment was based on the individual's engagement level in the activity). The cellular service provider (MobiCash) refused as there was too much uncertainty regarding number of trips for payment and the security situation in areas of Lower Dir and Upper Dir. Under these circumstances, even if MobiCash agreed to partner with Action Against Hunger the arrangements would have cost around 20% more. This is very high as compared to cash disbursement arrangements with MobiCash or other similar service providers engaged for Cash for Work projects. Later on, Action Against Hunger decided to disburse payment without involving MobiCash in order to clear payments for labour

- 21 - | P a g e RNEIII DRR Project Evaluation Report Action Against Hunger on time. Action Against Hunger obtained donor approval before adopting this payment modality. According to the Action Against Hunger finance section, it was appropriate to adopt the payment modality adopted for this project considering the specific challenges, however, it is not recommended for blanket adaptation across other projects. 3.6 Effectiveness:

Overall there was high level of effectiveness and results were achieved against planned targets proposed in the project proposal. Thus, all Action Against Hunger activities were not only relevant, but also effective. Timeframes and relevant support were extended when it was needed most.

VDMCs (male and female) were formed and trained on first aid, search & rescue and EWS. The trainings imparted to VDMCs and the general community was relevant and much appreciated. Action Against Hunger also equipped them with necessary emergency tools. Training was given mainly on basic search & rescue and emergency response skills like rope management, fire- fighting and stockpile management. During GDs, it was stated that such trainings and tools will be useful and cater to any emergency situation at community and village level. The component of effectiveness was visible when the acquired skills and knowledge was put into practice during an emergency situation. For example, during the evaluation study, the team observed and was also informed by the community about the use of fire extinguishers, during a house fire, so that property and other valuable items could be saved and less damage was done. Hence, coping capacities were effectively enhanced, although there is a strong need of refresher training for the communities.

Figure 16: Emergency Response Kit and Early Warning System Upper Dir

- 22 - | P a g e RNEIII DRR Project Evaluation Report Action Against Hunger

Flood is a major hazard in the area and early warning systems were strengthened by linking up the VDMCs and line departments to attain flood warnings in a timely manner. Communities are well aware of the possible repercussions of the various natural and man-made hazards. These communities also know about the evacuation routes and places of safe havens.

The VDMCs were provided with several types of kits. All of the kits were generally well maintained and occasionally used. Moreover, communities and government line departments also requested from other sources and donors boats and more search and rescue kits due to a perceived needs. Some of items in the kits are perishable in nature and need to be replaced after a certain time, such as chargeable torches in the search and rescue kit. During the evaluation the team observed that some of the items were no longer usable and needed to be replaced.

While a gender perspective remained a key focus of the project, women engagement in selection of evacuation routes was limited due to cultural constraints. Men were more engaged and sensitized in VDMCs. Unexpectedly, gender mainstreaming was visible in more conservative districts of Lower Dir and Upper Dir. A possible explanation of good performance on gender related issues can be attributed to the fact that local staff engaged in implementing project activities worked with the same communities in other non CBDRM related projects, thus they were provided with a solid base to work on. Currently there is a limited role and representation of people with disabilities in the VDMCs. Thus, moving forward, Action Against Hunger should actively consider representation of people with disabilities in the formation of committees and in their program design.

Figure 17: First Aid Box and stock register

Action Against Hunger has prepared the disaster management plans for the all the target villages to identify risk and propose mitigation measures accordingly. Disaster plans were developed in a participatory process, although female participation was comparatively less in comparison with men. These plans are utilized by the communities to identify hazards, vulnerabilities and risk and were used to facilitate comprehensive DRM planning at community and village level. Thus, this activity not only helped in identifying major risks and hazards in these villages, it will also help in better planning for mitigating some of these risks.

Exhibition and research studies were also effective activities in this project as hotspot themes were identified by PDMA. Recommendations in research studies should be incorporated in the

- 23 - | P a g e RNEIII DRR Project Evaluation Report Action Against Hunger guidelines of relevant line departments and in future projects of Action Against Hunger and other organizations.

Action Against Hunger also worked with children and school teachers to reinforce DRR at school level as per the project design. This activity created a multiplier effect as children shared disaster preparedness knowledge with their families. SDMCs were trained in first aid and evacuation procedures during emergencies and disasters. The first aid related learning was put into practice when children were injured while playing games. Even though this activity helped in improving safety for children during a natural or manmade calamity, the transfer of teachers creates a perpetual demand for refresher and/or re-training for the new teachers joining the schools.

Small-scale disaster risk reduction infrastructure was built in the local communities as mitigation measures to save valuable property and lives in the event of disaster. Local communities were involved through the selection of skilled and unskilled labor in the communities for the construction of the infrastructure. This activity can be scaled up to increase effectiveness. The infrastructure projects also helped in generating short-term employment opportunities of the most vulnerable communities in the form of Cash for Work.

During the formation of VDMCs, the existing Community Based Organizations (CBOs), if any, in each of the selected villages were also considered in order to ensure local communities' involvement, ownership and sustainability of the planned interventions. The VDMCs were trained on CBDRM, Search and Rescue and First Aid. Search and Rescue kits and First Aid kits were distributed to each VDMC. In addition to village level disaster management committees, UC level disaster management committees were also established by including members from VDMCs, local councilors, and UC level local government representatives to build the proper linkages between the community and the DDMU. For school based disaster risk management a total of 20 schools were selected (10 boys and 10 girls). Disaster management committees were formed in all the schools and were trained on SBDRM, first aid and search and rescue. Early warning kits were also distributed in each of the selected villages and orientation sessions on its use were conducted for the VDMCs. The communities shared that under the project local masons / artisans were trained on good construction practices/DRR resilience structures. Small scale DRR infrastructure schemes were constructed in targeted villages based on the requirement and selection was jointly done by Action Against Hunger field staff and VDMCs with the aim that these schemes would contribute in the mitigation of risk in the villages. The schemes included evacuation routes, new drainage systems, rehabilitation of existing drainage system and rehabilitation of irrigation channels. DRR exhibitions were also organized in collaboration with CDPM and PDMA. This was much appreciated by PDMA and a continuation of the research component had been requested by them.

During discussions, it was revealed that better coordination should be maintained with concerned government offices at provincial and district level with meetings at the project design, seeking No Objection Certificate (NOC), implementation and exit strategy stages. Project details were not properly shared with DDMU offices and PDMA for record and future use of resources. There is a risk that weak coordination could affect the effectiveness of the project in the long run.

- 24 - | P a g e RNEIII DRR Project Evaluation Report Action Against Hunger

3.7 Sustainability and likelihood of impact

The measure of sustainability and likelihood of impact varied depending on the nature of the activities; some of them were more sustainable then others that which will require continuous follow-up.

During RNE II, VDMCs with male membership were found to be present on the ground and were partially functional – especially in the Lower and Upper Dir. There was no common village across RNE II and III, therefore, for comparison purposes RNE II villages in close proximity to RNE III were selected with similar demographics and terrain for the purposes of this evaluation. Hence, RNE III VDMCs were more recently formed. Members informally met with each other and discussed any issues related to hazards but, there was no formal record of meeting minutes presented to the evaluation team to validate this. On the other hand, VDMCs with female membership were completely diminished on the ground; they may have lost interest in it and/or become redundant. Similarly, there is some likelihood that VDMCs will be only partially functional or completely dysfunctional if no proper follow-up is maintained after the project. The same likelihood applies to SDMCs.

The evaluators came across a particularly successful example of a VDMC functioning well. This VDMC has transformed itself into an NGO (picture of the membership card is given on the cover page). The evaluators believe this was mainly motivated by the youths present in the VDMC, who decided to build upon the structure established by Action Against Hunger and registered the VDMC as an organization. The organization is non-political and focused on common issues e.g. repair and rehabilitation of village access routes. They are meeting on at least a monthly basis. If any of the members become non-active, they are replaced with another more motivated person. They have two major funding streams: a monthly contribution of Rs. 100 by each committee member; and approaching the village diaspora for funding. As they are focusing on physical infrastructure schemes which benefit all the community, the community is taking an interest in the organization and they seem to be performing well.

In RNE II, VDMCs kept kits in safe places and used it for different occasions. The evaluation team observed that once a perishable commodity was used, the community was unable to replace it. For instance, communities are unable to buy new batteries to replace flat ones for megaphones. Keeping in mind the RNE II results, the evaluation team is also getting the same view about kits provided in RNE III project. This will negatively effect the sustainability of the kits.

Training in RNE II for communities was very effective and sustainable. Community members generally recalled key messages and demonstrated how to use different kits on different occasions. They were able to report their knowledge on reducing risks from natural disasters. However, the evaluation team still believes there is a need for refresher training for the communities. In RNE III, evaluation team believes that the trainings will be equally effective and sustainable as the trainings are relevant to the local context and were greatly endorsed by the targeted communities.

The evaluation team found the construction in both RNE II and RNE III project to be of high quality. The infrastructure will likely be sustainable for a long period of time. However, in one of

- 25 - | P a g e RNEIII DRR Project Evaluation Report Action Against Hunger the RNE III villages in Nowshera district, the evacuation route was not built as per community requirement; it was left incomplete and not extended to the evacuation center. Generally, the irrigation channels and protection walls were found to be needed and will be retained in the long run as the communities are taking responsibility of them. Masons training on DRR was also relevant to the local context but less likely to be sustainable in the long run due to increased costs associated with incorporating DRR in house construction such as raised platforms, use of steel bars in construction, etc which the local communities cannot afford. Nevertheless, the trained masons are sensitizing people about disaster resilience houses.

The promotion of DRR through conducting exhibitions and carrying out research studies sensitized the policy makers as well as other actors to help ensure DRR remains a priority theme for saving lives and properties. Information sharing regarding the key project results and lessons learned needs to be disseminated on various forums such as district and provincial DRR forums to be incorporated in project designs, strategic papers or any other relevant documents of civil society organizations.

The evaluation survey results indicated a weak link between government line authorities, communities and Action Against Hunger which can create hurdles in the dissemination of early warning messages in a timely manner.

- 26 - | P a g e RNEIII DRR Project Evaluation Report Action Against Hunger

Figure 18: Action Against Hunger Supported Emergency Response Kit in RNE II.

- 27 - | P a g e RNEIII DRR Project Evaluation Report Action Against Hunger

Section 4: Significant Change Stories

Profile Women as First Responders – Throughout Project Area

Incident Women received training in DRR related subjects. This has enabled women to understand the risks and respond immediately in case of an earthquake, flood and other hazard.

Significant Change Women are better prepared to respond to any unforeseen emergency and will not wait for men to arrive and help them out.

Theme Women Empowerment. Domain Disaster Preparedness

Profile Name: Mr. Farman District: Upper Dir

Incident I am a daily laborer. I am quite poor and live in an area which faces heavy snow fall. During the winter season my house used to get covered in thick snow and I was unable to clean it in the absence of any tools. My nominal income does not allow me to purchase any equipment. As per the practices in the area, we prefer not to borrow anything from others. I was concerned about the safety of my family as a slippery entrance could have caused injuries. However, the situation changed with the provision of tool kits by Action Against Hunger. By utilizing these tools I cleaned my backyard and now the safety of my family has been enhanced. Once there was a fire in my house and I utilized the fire extinguisher given to me in the emergency kit and as a result no severe damage was done. While fixing our house we also used the tarpaulin sheets as temporary shelter until the house was ready to live in.

Significant Change The significant change observed is that the community is taking ownership of the kits provided and are utilizing it during emergency situations.

Theme Saving lives

Domain Disaster preparedness

- 28 - | P a g e RNEIII DRR Project Evaluation Report Action Against Hunger

Profile Name: Ms. Rozina District: Charsadda

Incident I live in a village which had no proper drainage system before. During monsoon season due to the heavy rains, water would get stagnant and convert into dirty ponds with mosquitoes. This situation gave rise to waterborne diseases such as malaria and diarrhea in our area and adversely affected the health of our children. However, with the construction of drainage channels by Action Against Hunger in the area the surplus water flows through these structures. As a result, I have noticed a decrease in disease rates among my children. Because of this I am relieved now and appreciate the work done by Action Against Hunger.

Significant Change The significant change is the reduction in water borne diseases and proper drainage of monsoon water from the village.

Theme Improving health by building structures. Domain Resilient Infrastructure

Profile Name: Ms. Zarmeena District: Nowshera Incident Disaster preparedness and emergency management were not familiar concepts for me. My children used to get injured while playing and I had to spend a substantial amount on their recovery. Later on I got a chance to attend the training conducted on disaster management where I also learned the usage of the first aid kit. I was surprised to know that such basic items if used properly can be of great help. One day my child got minor injuries while playing which I treated by using the first aid kit items. My health related expenditures have been decreased. I am really grateful to Action Against Hunger for including me in the training.

Significant Change The significant change is that women are trained and knowledgeable on the subject and can provide first aid in case of injury.

Theme Empowering women in disaster preparedness. Domain Community Preparedness

- 29 - | P a g e RNEIII DRR Project Evaluation Report Action Against Hunger

Figure 19: Discussion with a key informant in Lower Dir

- 30 - | P a g e RNEIII DRR Project Evaluation Report Action Against Hunger

Section 5: Conclusions

This session summarizes the main findings and conclusions.

From the design perspective, the project was appropriate and coherent with the overall objectives, PDMA road map and relevant to the needs of the targeted population. Action Against Hunger carried out a PCVA exercise for identification of the most vulnerable villages in the four targeted districts of KPK. Based on the PVCA, the 40 most vulnerable villages were selected out of 75 villages in eight UCs based on the set vulnerability criteria. Similarly the project components were well-designed such as DRR infrastructure, early warning system, search & rescue kits, and first aid kits. Village disaster management plans were found to be appropriate and addressed the needs of the communities and selected schools.

There were several other important features of the project design. For example, formation and capacity building of VDMCs and SDMCs and developing Village Disaster Management Plans with the help of the general community and VDMCs. The package of trainings for VDMCs, SDMCs and local masons was very relevant to the needs of the targeted population. Working with academia for conducting research studies, arranging exhibitions on DRR and developing linkages of VDMCs and UCDMCs with DDMU were also part of the design. Gender inclusion and involving women throughout the project life while respecting local culture, and placing a Feedback Complaint Mechanism for ensuring accountability were also included in the design of the project.

From the perspective of relevance and appropriateness, the evaluation team found that the project was relevant and appropriate in the local social and cultural context and aligned with local needs and government priorities. It was aligned with the PDMA road map, Do No Harm principle, country strategy of Action Against Hunger and agreement with donors. Moreover the intervention was also in line with the National Disaster Management Framework; Pakistan Action Against Hunger DRM policy; Action Against Hunger emergency preparedness and response planning; National DRR Policy; and the Hyogo Framework for Action.

During data analysis, it was revealed that the main hazards were floods, heavy monsoon rains, storms, earthquakes, soil erosion, siltation, land sliding, militancy and snowfall and there was dire need of First Aid, Search & Rescue and EWS kits and trainings. Earlier, the communities did not have means to attain reliable information but now with the installation of EWS and linkages with DDMU and capacity building trainings, they are capable to react immediately in effective manner. Local masons training on DRR was also relevant and added value to the objectives of the intervention. The construction of DRR infrastructure enhanced resilience of the communities and decreased their vulnerabilities. Similarly, supporting DRR exhibitions and research studies in different thematic topics in collaboration with PDMA and CDPM-University of Peshawar was also relevant in the context of Pakistan.

From the perspective of coherence, the evaluation team found Action Against Hungerthe project was in line with government policies, detailed above in the design part. Close collaboration with key stakeholders during the design and implementation phases of the project was maintained

- 31 - | P a g e RNEIII DRR Project Evaluation Report Action Against Hunger such as DDMU and PDMA, however linkages with some of the departments likes civil defense, education, and health amongst others could be strengthened. PDMA was involved in the project design and implementation. Beneficiaries and communities ensured that there was no duplication of activities.

From the perspective of coverage, in consultation with PDMA, Action Against Hunger selected four districts of Lower Dir, Upper Dir, Nowshera and Charsadda in KPK for the project. UC and village selection for the intervention was done in consultation with DDMU and district authorities. After conducting PVCA in 75 villages, the 40 most vulnerable villages were identified in eight UCs. The findings of PVCA were considered and incorporated during implementation of the project.

The project mobilized communities, established VDMCs and UCDMCs, built the capacities of the targeted communities through capacity building trainings in CBDRM and SBDRM. Small DRR infrastructure e.g. pathways as safe evacuation routes and drainage were constructed on village level however, it revealed during data analysis that these infrastructures are limited and do not satisfy the needs of the entire communities.

The component of gender inclusion could have been improved with female participation and representation in CBDRM, search and rescue and EWS being nominal (this was not the case for first aid training). Female engagement in comparison to men remained weak, including in VDMCs.

Although the project contributed in the identification of safe routes and evacuation centers, it was found that these evacuation centers can neither serve the needs of the whole population nor cater the specific needs of women. This is mainly due to limited availability of resources and it is unlikely that Action Against Hunger would ever be able to fully meet the requirements of the whole community.

Due to a shortage of time, the formation of UCDMCs was not completed in some areas and linkages were not strengthened between government line departments and VDMC/UCDMCs. In addition to this, capacities of VDMCs were not built as prescribed in project design. Thus, there is a need for continuation of DRR services in the project areas, especially improving the evacuation centres.

From the perspective of efficiency, the evaluation team found during analysis that overall the project achieved the targets within the timeframe and budget. The utilization of local resources was done in an efficient way. In the absence of NOC, there were access issues and regular changes in the planning in remote areas led to some delays in the payment of beneficiaries. Due to overlapping activities, a scarcity of resources was observed. However, despite all the challenges, the project achieved its overall objectives and goal.

From the perspective of effectiveness, the project effectively achieved its targets against the planned activities such as establishment of VDMCs (male and female), SDMCs and UCDMCs, and conducting trainings of First Aid, search & rescue, EWS and CBDRM. In the life of the project there was no large scale disaster or emergency, so the search and rescue and EWS kits were not utilized. Similarly, VDMCs have also not utilized their skills and knowledge after the

- 32 - | P a g e RNEIII DRR Project Evaluation Report Action Against Hunger trainings but they are aware of the existing hazards and in case of a disaster can respond adequately.

Furthermore, the development of village disaster management plans, village evacuation plans and identification of safe routes and safe places in case of hazards are significant contributions of the project. By linking VDMCs and UCDMCs with DDMU and line departments for obtaining information about disasters well in time is another achievement indicating the effectiveness of the project. Similarly, local masons / artisans were trained with mainstreaming DRR in construction and small scale DRR infrastructures e.g. evacuation routes, drains and retaining wall were built, contributing to community resilience. These DRR schemes also helped in generating short-term employment opportunities for the local communities, such as Cash for Work. Arranging exhibition and research studies proved to be effective as findings of these studies will be incorporated into government policies.

Although efforts were carried out for gender mainstreaming throughout the project, women's role in the identification of safe evacuation routes, evacuation centers and their participation in training was limited. In the same manner irregular liaising and coordination was carried out with line departments.

From the perspective of sustainability and likelihood of impact, in different project components the sustainability and likelihood of impact varied depending on the nature of activities. For example, male VDMCs were partially functional in RNE II as they do not formally meet regularly but informally meet and discuss about hazards and disasters. In comparison the VDMCs are more active and sustainable however continuous follow up and capacity building are required to keep them on track and sustainable. On the other hand, female VDMCs in RNE II were completely dysfunctional and members had lost interest. Thus, there is likelihood that VDMCs in RNE III will be only partially functional or completely dysfunctional in case of no follow-up. The same observation can be applied to SDMCs.

The evaluation team found that once perishable commodities in the kits were used; there was no proper mechanism to replace them.

On the positive side, the infrastructure provided by the project will likely be sustainable over a long period of time. Working with research institutions in collaboration with PDMA also contributes to sustainability in terms of influencing government policies regarding DRR.

- 33 - | P a g e RNEIII DRR Project Evaluation Report Action Against Hunger

Figure 20: Emergency Response Kit Distributed During RNE III

- 34 - | P a g e RNEIII DRR Project Evaluation Report Action Against Hunger

Section 6: Lesson Learnt

This section outlines different lessons learned in project design, planning and implementation phases. Action Against Hunger may consider them for future projects.

Mobilization of local resources: Action Against HungerInvolvement of local skilled and unskilled labour in the implementation of construction activities increases the quality assurance and ownership of infrastructure. Moreover, it has financial benefits.

Promotion of Research Studies: Action Against HungerAction Against Hunger support of DRR-oriented research studies via academic institutions can help in identifying gaps and recommendations for making future guidelines or strategic policy making on DRR.

Inclusion of women in emergency response training: As per local context, women usually stay at home. When facing various emergencies and they were first in responding to emergencies that occurred in their house, like fire from short circuiting. This demonstrates the importance of Action Against Hunger including women in emergency response training.

Proper coordination with government departments: It is important to do some background research Action Against Hungerprior to submitting documents to the line departments for the NoC request. As the NoC process is not well defined, the evaluators are of the view that it is useful to share detailed information with the line department about the proposed project and make them realize how useful the project could be for the communities and for the department itself. This approach can be used as a push factor on behalf of Action Against Hunger for granting of the NoCs.

Recruitment of staff: During the program suspension, due to problems in the NoC process, the Human Resources department faced for the first time the challenge of project staff being asked to take forced leave without pay. There were no clear guidelines in Human Resources manual to pass on to the staff e.g. leave accrual, health insurance / benefits etc. Human Resources has now drafted guidelines (still to be approved) to be used in such situations in the future.

Payment mode: The signatory authority being based in Islamabad can cause delays in payment on the field. By shifting this authority to the field office, payment to labor etc. could be made in a timely manner.

- 35 - | P a g e RNEIII DRR Project Evaluation Report Action Against Hunger

Section 7: Recommendations This section provides recommendations for enhancing the quality of the project in the future. The emphasis is on providing recommendations, which would be feasible within the difficult working environment faced by Action Against Hunger in Pakistan.

1st Recommendation: Increased Engagement with VDMCs

(High Priority, Field Team)

VDMCs sustainability should be ensured with proper and long term engagement of Action Against Hunger with them. It was observed in phase 2 and phase 3 of the project, that the longer the interaction of Action Against Hunger with VDMCs, the more likely they were to be functional and performing their activities independently. As well, Action Against Hunger should pay more attention to engagement with female committees.

2nd Recommendation: Horizontal and Vertical Linkages between VDMCs

(High Priority, Senior Management Team)

VDMCs were weakly linked with district departments including DDMU. It was also observed that VDMCs were not closely linked to each other. In future, more efforts should be done to strengthen these linkages, while presently names of focal points at VDMCs can be shared with the DDMUs.

3rd Recommendation: Cross Learning / Learning Exchange Activities

(High Priority, Field Team)

Various significant change examples and stories at community level were observed by the evaluation team. These could provide opportunities to learn and build on each other's experiences, especially for VDMCs. There is currently limited cross learning amongst VDMCs. It is recommended to systematically encourage cross learning, learning exchange activities and information sharing amongst VDMCs. This will not only help in adopting best practices from each other, but also create a positive competition amongst them.

4th Recommendation: Training, Duration, Refreshers

(Medium Priority, Senior Management Team)

Capacity building on First Aid especially from a gender perspective, was one of the most useful components. Furthermore, Action Against Hunger may increase access to women with this aspect in project design as it will be a gate opener. At the same time, in case of a small or major emergency, first aid trained women will be in a position to respond in a timely manner thus saving precious lives.

- 36 - | P a g e RNEIII DRR Project Evaluation Report Action Against Hunger

5th Recommendation: Including Additional Aspects in the Design:

(High Priority, Senior Management Team)

In future project designs, Action Against Hunger should consider additional aspects such as hazard and risk maps at village level. They should also share these maps with the communities and government so that they can be used in their planning activities. Although the activity is time consuming, it is important.

6th Recommendation: Institutional Strengthening for DRR

(High Priority, Senior Management Team)

Action Against Hunger has come to be known by government departments as a reliable partner with highly skilled and experienced staff. Government, especially from a response perspective, will appreciate the continued work of Action Against Hunger in DRR. Action Against Hunger could continue working and cover some specific needs in future such as providing specialised boats in Lower Dir.

7th Recommendation: Plantation Activities

(Medium Priority, Senior Management Team)

In the initial stages of project design, plantation activities were included but later on removed due to a cut in the budget by the donor and other parallel activities like the government billion tree project in the same region. Action Against Hunger should consider reintroducing plantation activities in future project designs, for example, the introduction of plantations such as sea buckthorn plantation for slope stabilisation. This could be helpful in making a DRR project more relevant in the local context.

8th Recommendation: Build on DRR Research

(High Priority, Senior / Field Management Team)

Action Against Hunger commissioned five different research studies through the University of Peshawar. There is a need to build up on this research and link it with policy level changes at district and provincial level. Action Against Hunger should continue promoting DRR related research through academic institutions.

9th Recommendation: Support Academia for DRR Research and Creating Government - Academia Linkages

(High Priority, Senior Management Team)

As mentioned above, Action Against Hunger supported and conducted DRR-oriented research studies through the University of Peshawar. This was an excellent initiative that contributed to a better understanding of DRR-related risks in the region. Action Against Hunger should continue supporting and creating a research link between PDMA and other academic institutes. Moreover,

- 37 - | P a g e RNEIII DRR Project Evaluation Report Action Against Hunger

Action Against Hunger use the research for planning and analysis and may share the data with concerned Government departments like the Geological Survey of Pakistan, Ministry of Environment, Climate Change Division and Disaster Management authorities both at federal and provincial levels.

10th Recommendation: Increasing Government Ownership and Profile Building

(High Priority, Senior Management Team)

The project was mainly focused on DDMU and to some extent, the Education Department, whereas engagement of other government departments was almost negligible. Action Against Hunger should aim to increase government involvement in the project and engage other relevant departments including civil defence, health, irrigation etc. The project should link VDMCs with DDMA and advocate treating VDMCs as relevant structures for any future government grass- roots DRM initiatives. A Memorandum of Understanding should be signed with the government at the time of starting the project, which mentions the roles, responsibilities and inputs that both sides will provide during and after the project. Action Against Hunger should also enhance coordination with different forums such as DRR forums and obtain greater technical inputs from them.

11th Recommendation: Increasing Coverage of the Evacuation Routes

(High Priority, Field Team)

The evacuation routes constructed under this project in many instances were only sufficient to cater for the emergency evacuation needs of a part of the target village. Therefore, in future the evacuation route needs to be designed and constructed in a manner to cover needs of all the village.

12th Recommendation: Evacuation Centers Better Equipped

(Medium Priority, Field Team)

Considering the conservative culture in the project area, the selected evacuation centres should be better equipped to accommodate the needs of women, particularly pregnant and lactating women. Similarly, needs of PWDs should also be addressed. In addition, a kit containing some basic items e.g. water storage vessels with filtration arrangement and basic items of personnel hygiene should be stored at the centres.

- 38 - | P a g e RNEIII DRR Project Evaluation Report Action Against Hunger

Appendix Annexure 1: Good Practice Title of Good Practice

Promoting DRR Related Research through Academic Institutions

Innovative Features and Key Characteristics

To bridge the gap between the practitioners and academia through engagement of students from local educational institutions to obtain field experience and advance their MSc level research studies.

Background of Good Practice

The RNE 3 project had a distinctive characteristic where it has supported 5 research studies at Centre for Disaster Preparedness and Management (CDPM), University of Peshawar. All the DRR related research themes were identified by the Provincial Disaster Management Authority (PDMA). The students of CDPM developed concept notes around these themes. All the concept notes were reviewed by a committee, consisting of representatives from PDMA, CDPM and Action Against Hunger, and 5 studies were approved for conducting the research.

Further explanation of chosen Good Practice

The PDMA road map is an overarching strategy document related to the government priorities in disaster management field. Under the road map, the PDMA also would like to promote research on DRR related topics. Similarly, the CDPM is a DRR specialized department of Peshawar University which is largest public sector university in KPK. Therefore, through Action Against Hunger support CDPM and PDMA came together to support 5 students in their MSc level research. The following are the lists of topics selected:

 Analysis of Sustainable Infrastructure and DRR in KP  Climate Change and its Impacts on KP  Cyclone Disaster Vulnerability and Response Experiences: A Case Study of April 2015 Mini- Cyclone  Disaster Management Trends, its Impacts and Way Forward in KP  Application of GIS in Flood Hazard Mapping: A Case Study of Budni Nullah, District Peshawar, Pakistan.

Action Against Hunger facilitated this whole process of research topic selection. Once the research topics were finalized, Action Against Hunger provided paid internships to the selected students and provided them an opportunity to work in the field. The students not only got rich firsthand experience in the field but also collected primary data besides they had access to data available with Action Against Hunger. The students then completed their draft research studies and submitted to CDPM. After initial internal reviews from their supervisors in the CDPM. The draft research studies were also shared with Action Against

- 39 - | P a g e RNEIII DRR Project Evaluation Report Action Against Hunger

Hunger to provide their inputs before finalization of the researches. These researches are still in the draft form, however, Action Against Hunger intends to incorporate some of the key findings and recommendations from these researches in their future projects. Similarly, the PDMA will publish the researches on their website and are keen to act on any of the recommendations, if required. The CDPM will display the studies in future DRR related exhibition. In addition, they also shared that these researches also help that particular students to get into job market comparatively easily. This case study shows a combined effort on behalf of international community (Action Against Hunger in this case), relevant government department (PDMA) and the academic institutions (CDPM) to contribute towards disaster management.

Practical/Specific Recommendations for Roll Out

This case study could be easily adopted and replicated by other Action Against Hunger programs. The Action Against Hunger office in Pakistan may make it mandatory for all programs to support at least one research study per annum.

How could the Good Practice be developed further?

Long term arrangement through memorandum of understanding or partnership agreements with well reputed university in this regards could be beneficial. In addition, it will facilitate various Action Against Hunger programs to work with their respective subject specialist department within the university without going through the administrative procedures each time.

- 40 - | P a g e RNEIII DRR Project Evaluation Report Action Against Hunger

Annexure 2: Evaluation Criteria Table Rating 1-5 (1 Low, 5 High) Evaluation Criteria Rationale 1 2 3 4 5 Overall, the design of the project was appropriate and coherent with the objectives Design of the project. The project was in line with needs of the targeted population and government priorities. The project was relevant to the needs of the Relevance / targeted communities and was in-line with Appropriateness: government priorities. Overall there was a strong collaboration with stakeholders. The project also fits well Coherence within the overall interventions by government and other organizations.

The project couldn’t cover the needs of the whole targeted union council. Even in some Coverage activities, it won’t cover the needs of the whole village. Besides so many challenges in access due to NoC, constant changes in planning, Action Efficiency Against Hunger’s internal SoPs and working in hard and remote areas the project achieved it targets within time. The project effectively achieved its targets against the planned. However besides the efforts carried out for gender mainstreaming Effectiveness throughout the project women role was limited. In the same manner irregular liaising and coordination was maintained with line departments. In different project components the sustainability and likelihood of impact varied depending on the nature of activities like the provided DRR infrastructures, Sustainability and various tools kits, skills acquired in Likelihood of Impact trainings and linkages of VDMCs with DDMU have more likelihood to sustain. However male VDMCs are more active and sustainable in comparison with women VDMCs.

- 41 - | P a g e RNEIII DRR Project Evaluation Report

Action Against Hunger

Annexure 3: List of Documents

Documents GLOW Response Action Against Hunger Country Strategy Yes National Strategy in Action Against Hunger relevant sectors (DRR, WaSH, etc.) Yes Cluster Strategy in Action Against Hunger relevant sectors (DRR, WaSH, etc.) Yes 02 Days FA and S&R Training report 2016 Yes Research Reports Yes CBDRM training report Yes DME Activity Wise Report –ACF Yes DRM Exhibition Event Report 19 Dec 16 Yes DRM RNEIII-Training Report Aug 2016 Yes DRM training (CBDRM) monthly Report July 2016 Yes Final Directory file Yes Masons-Training Report-Consolidative Yes RNE III-PDM ERT & First Aid Kits Draft report re TR Yes Baseline & Endline Yes Databases RNE III Yes Monthly reports Yes PCVA_KPK _RNE-III 2016- final draft Yes Progress Report RNE III Yes RNE 3 DIP Revised (5) Yes RNE III Docs from Grants Yes Documents required for RNE III Yes Final Report-RNEIII Yes Partnership Agreement-DRM Exhibition Yes RNE III Project Closure meeting Presentations Yes s81FinalReport_110815-Embassy Yes Service Contract Research work Yes

- 42 - | P a g e R N E I I I D R R P r o j e c t E v a l u a t i o n R e p o r t

Action Against Hunger

Annexure 4: List of KIIs

S. No Name Designation ACF Staff 1 Mr. Mushtaq Ex-Field Co/Liaison Co 2 Mr. Rashid Technical Adviser (DRR) 3 Mr. Mazhar Logistic Co. 4 Mr. Arshad Field Co. Sindh 5 Mr. Dildar Deputy Program Manager 6 Mr. Waqas Jadoon HR Head 7 Mr. Raza Haider Finance Co 8 Mr. Abdul Wahab Finance Officer Government 9 Mr. Wajid Deputy Director - PDMA 10 Ms. Ayda Deputy Director - PDMA 11 Ms. Zainab Child Protection & Gender Adviser 12 Mr. Zahir Shah Assistant AC - Charsadda 13 Mr. Abdul Hameed ACO - Nowshera 14 Mr. Fazal Hussain DCO - Nowshera 15 Mr. Janas Khan Director Education - Charsadda 16 Mr. M. Ayaz Mohmand AC - Timergara Academia 17 Mr. Mushtaq Director Clusters 18 Mr. Najeeb Focal Person - DRR Forum SMDCs / Focal Person 19 Mr. Mohabbat Shah Teacher (GHSS Tangi) 20 Mr. Mukamil Shah Teacher (GHS Khungi) 21 Mr. Ijaz Ahmad Teacher (GHSS Samkot Batal) 22 Mr. Rahimullah Teacher (GPS Shekhani No 2 Shahkhani) 23 Mr. Haji Sajjad Ali Teacher (GMS Nawan Kalay) 24 Mr. Marjan Ali Head Teacher (GPS Marozai) 25 Mr. Jalal Shah Teacher (GHS Agra Payan) 26 Ms. Parveen Head Teacher (GGPS Nawan Kalay) 27 Ms. Zakia Begum Teacher (GGHS Palam Samilkot) U/CDMC 28 Mr. Sattan Bacha Upper Dir 29 Mr. Shah Wali Nowshera 30 Mr. Hidayatul Islam Nowshera 31 Mr. Arsala Khan Nowshera 32 Mr. Kamran Khan Charsadda 33 Mr. Masam Jan Charsadda

- 43 - | P a g e R N E I I I D R R P r o j e c t E v a l u a t i o n R e p o r t Action Against Hunger 34 Mr. Bashir Khan Lower Dir 35 Mr. Abdullah Lower Dir 36 Mr. Sardar Hussain Upper Dir Masons 37 Mr. Murad Ali Nowshera 38 Mr. M. Ishaq Nowshera 39 Mr. Anwar Shah Charsadda 40 Mr. Masam Jan Charsadda 41 Mr. Bahre Mand Charsadda 42 Mr. Zubair Khan Charsadda 43 Mr. Rafiullah Khan Lower Dir 44 Mr. Muhammad Ilyas Lower Dir 45 Mr. Noor Zaman Khan Lower Dir 46 Mr. Inamullah Upper Dir 47 Mr. Lal Muhammad Upper Dir 48 Mr. Sohail Upper Dir

- 44 - | P a g e R N E I I I D R R P r o j e c t E v a l u a t i o n R e p o r t Action Against Hunger

Annexure 5: Data Collection Tools

- 45 - | P a g e R N E I I I D R R P r o j e c t E v a l u a t i o n R e p o r t

Annexure 6: Revised Logical Framework

Logical frame work analysis Principal Objective:

To continue reducing the vulnerability and improve the resilience to disasters in targeted community in KPK province, Pakistan. Indicators Risk and Assumption Specific objective: Outcome indicators - Access to beneficiaries possible -Government clearance and support for - 60,000 beneficiaries in 40 communities have improved DRM intervention capacities and reduced vulnerability to disasters - Acceptance of ACF is established at local To enable communities to prepare for and mitigate level and access to beneficiaries is ensured the impacts of disasters using participatory methods -90% of targeted communities have improved knowledge on - Involvement and interest for aimed at strengthening the self-reliance capacities early warning system etc. collaboration of communities and of the population and reinforcing or adapting - 80% of targeted communities have improved knowledge on authorities is adequate. infrastructure. function on DDMU

- 100% of targeted communities identified the evacuation centers and organized themselves to use it during the emergency Result Output indicators

RI: Building Community Disaster Resilience - 80% of targeted VDMC and UC DMC have improved through Community Based Disaster Risk knowledge and are functional Management - 90% of targeted communities covered by disaster management plans. -No of communities are supported with hazard resistant structural mitigation schemes

RII: Strengthening institutional linkages and - 2 of exhibitions supported capacities for Disaster Resilience - 5 research studies conducted and reports developed -No of meetings conducted within DDMU, UC DMC and VDMC Result I: Building Community Disaster Resilience through Disaster Risk Management plans, preparedness and mitigation measures Activities: 1.1: Baseline and End line survey 1.2.: Formation and training of VDMC

1.3: Development of UC level Disaster Management Committee (UCDMC) Structures: 1.4: Capacity building of local technician and artisans. 1.5: Formation and training of SDMC training 1.6: CBDRM sessions at community level 1.7 : Provision of first aid, search & rescue Kits to VDMCs and SDMC 1.8 : Facilitate the community to improve the disaster resilience infrastructures 1.9: Establishments/ Improvement of community early warning system:

Result II: Strengthening institutional linkages and capacities for Disaster Resilience Activities: 2.1 : Support 2 (Two) DRM exhibitions 2.2 : Conducting DRM field research projects by 5 students: 2.3: Linkages of VDMC/UCDMC with DDMUs 2.4 Support DDMU to develop and implement District Charsadda DRM plan 2.5 Support DDMU for Emergency Response (Lower Dir)

TERMS OF REFERENCE

For the Independent Final Evaluation of Action Against Hunger’s

Reducing Vulnerability through Disaster Risk Management (DRM) in Khyber Pakhtunkhwa (KPK) Province, Pakistan - Phase 3,

Summary Table Project Name Reducing Vulnerability through Disaster Risk Management (DRM) in Khyber Pakhtunkhwa (KPK) Province, Pakistan - Phase 3, Contract Number PAK 3044-14/0009 Partners (if applicable) NA. Location (country/ies, Charsadda, Nowshera, Dir Lower, Dir Upper, Khyber region/s) Pukhtoonkwa Pakistan Sector(s) Disaster Risk Management (DRM). Duration 27 Months (But actual implementation period is 13 Months) Starting Date 1st October 2014 Ending Date 31st December 2016 Intervention/Country Office English Language Donor and Contribution/s Royal Norwegian Embassy Country Office administering ACF Pakistan the Programme/Project Responsible Action Against ACF USA Hunger HQ Evaluation Type Independent Final Project Evaluation Evaluation Dates 1st December to 10th Jan 2017

1

1. BACKGROUND of the Project

1.1. Map of Intervention Area Note: Charsadda, Nowshera, Lower Dir and Upper Dir are the Targetted districts.

1.2. Rational for the Project The persistent disasters, combined with structural poverty factors, unequal access to basic services, and a lack of government and community capacity to fully recover from the disasters while absorbing the displaced, has resulted in a systemic cycle of increasing vulnerability amongst communities. While past NGO initiatives in the project area have succeeded in training government staff, creating awareness of DRM, and developing DRM plans in a small percentage of communities, large areas remain uncovered. In addition, those plans that have been created are not well

2

integrated in to district and provincial planning, with district level government actors involved in DRM lacking the capacity to manage and utilize gathered information effectively. The need exists to continue building resilience at the community level, in parallel with government capacity building and strengthening linkages between governments, NGOs, and communities.

Flood is the major disaster affecting the population in the assessed districts in the past years. A base line survey assessing the structural and non-structural knowledge of location community on mitigation measures carried out in September 2013 reveal that people strongly believe that the "Act of God" is the main reason for natural disasters in their area. Moreover, it shows that no proper early warning systems are established at community level and people receive information by observing the rising level of the river. 76.9% of people interviewed said that they built Pakka (strong) infrastructure to mitigate the effect of disaster but 93% did not understand the non-structural measures such as land use policy, reforestation, early warning system. 69 % of respondents said that district disaster management unit (DDMU) was not working properly and the remaining respondents did not know the existence of the DDMU.

Evacuation plans were also assessed through this survey. Only 35.7% of the respondents said they had already identified the evacuation centre to be used during the disaster. ACF in implementing DRR project in the flood effected targeted districts of Khyber Pakhtunkhwa (KPK) Province, this is consecutive third phase of the project funded by Royal Norwegian Ministry.

1.3. Project Objectives

General Objective: To continue reducing the vulnerability and improve the resilience to disaster of KPK province, Pakistan.

Specific Objective: To enable communities to prepare for and mitigate the impacts of disasters using participatory methods aimed at strengthening the self-reliance capacities of the population and reinforcing or adapting infrastructure.

The project log frame is attached in annex I.

1.4. Current Status of the Project The project soft and hard activities are currently ongoing as per initial planning and tentatively completed by mid of December 2016. Initially the project was to be completed by September 2016, but the No Objection Certificate (NOC) seeking from National and Provincial Government took more time than expected and it lead to delays in the implementation, a no cost extension has been approved by RNE and thus the project will be ending in December 2016. Besides that, a few project activities have been redesigned due to parallel government campaigns / interventions across the province like Plantation, and predefined some disaster resilience infrastructure in the proposal has been changed to need based infrastructures, while the rest of the interventions are as proposed initially.

2. PURPOSE AND OBJECTIVES OF THE EVALUATION

2.1. Rational for the Evaluation This evaluation is conducted as an exercise of accountability towards the donor and the beneficiaries at its final implementation stage. The evaluation will be conducted by a third party independent evaluator.

2.2. Objectives of the Evaluation

3

• To determine how effectively and efficiently project outputs/outcomes are achived and has contributed to the overall and specific objectives of the project. • To assess the project interventions are likely to be sustainable beyond the project lifetime. • To examine the preparedness level of communities (in community and schools) to cope with and district authorities to respond to any future emergencies. • To identify the good practices and lesson learnt for future programming • To examine the cross cutting issues like enivornment, Gender equality, coordination, advocacy and participation

2.3. Users of the Evaluation • Direct users: ACF Pakistan Mission, ELA unit - UK, ACF USA (HQ), ACF field teams; • Indirect users: ACF International Network, Royal Norwegian Embassy, Government of KPK, NDMA, PDMA, Provincial government, district and local government, UN agencies and DRR Clusters, INGOs and NGO Consortiums as well as humanitarian learning platforms (such as ALNAP).

2.4. Use of the Evaluation • The learnings from the evaluation will be used to developed new strategies at National, Provincail and District level (if needed). These learning will also be helpful to improve organizational strategy / approach towards DRR and design projects and interventions as per the contextual requirement. • The learning and/or successes on the ground will be dissiminated within ACF networks and learnings will be reaplicated in other similar contaxts. • These learnings will also support to define advocacy and communication contents with different stakeholders. • The evaluation will measure the potential issues faced and recommendations for future, that will guide future programing.

3. EVALUATION SCOPE

3.1. Elements covered by the evaluation • This evaluation will be foucsed on the four intervention districts of Khyber Pakhtoonkhwa (KPK) Province, including Charsadda, Nowshera, Dir upper and Dir lower. The evaluator will consider the goals and objectives of the project in comparison to the project interventions at field level. • The project evaluation will be based on the Development Assistance Committee DAC criteria (Design, Coherence, Coverage, Relevance and Appropriateness, Effectiveness, Efficiency, Sustainability and Likelihood of Impact) that will be followed by the evaluator. • Identify unintended changes, both positive and negative, in addition to the expected results;

3.2. Elements not covered by the evaluation • NA.

3.3. Cross-cutting issues • Throughout the evaluation process, gender concerns should be addressed in line with the ACF Policies. All data should be disaggregated by sex and age. Different needs of women, men and of marginalised groups targeted by the project should be considered throughout the evaluation process. Moreover, the community participation should be emphasised and how ACF ensures that communities were involved throughout the programme cycle.

4. EVALUATION APPROACH AND QUESTIONS

4

As per Action Against Hunger Evaluation Policy and Guidelines1, Action Against Hunger adheres to the Development Assistance Committee (DAC) criteria for evaluating its interventions. Specifically, Action Against Hunger uses the following adapted criteria: Design, Coherence, Coverage, Relevance and Appropriateness, Effectiveness, Efficiency, Sustainability and Likelihood of Impact.The evaluator must complete the DAC criteria rating table (Refer to Annex V) and include it as part of the final evaluation report.

Evaluation questions have been developed to help the evaluator assess the project intervention against these criteria. (Refer to Annex V). The evaluator may adapt the evaluation criteria and questions, but any fundamental changes should be agreed between the evaluation manager and the evaluator and reflected in the inception report.

5. EVALUATION METHODOLOGY

The evaluation methodology will be proposed by the evaluator after thorough study of TOR's requirment. The evaluatior is expected to use mixed methods approach and/or Most Significant Change Technique (MSCT), collecting both qualitative and quantitative information from targeted household and communities. Moreover for triangulation of the information, interviews of other key stakeholders i.e Provincial Disaster Management Athourity, District Disaster Management Athourity, District Disaster managerment officer (DDMO), Deputy Commissioner officer / Assisstant Commissioner officer (ACO/DCO) , Executive District officer (EDO - Education), Targetted Schools Management (Prinicples, Teachers), University of Peshawar (UOP), DRR Clusters, UN Agencies, project key staff etc

5.1. Evaluation Briefing Prior to the evaluation taking place, the evaluator is expected to attend an evaluation technical briefing with the Evaluation and Learning Unit (ELA) ACF-UK focal person and in country with PQA Coordinator.

5.2. Desk review The evaluator will undertake a desk review including the project documents and proposals, progress reports, outputs of the interventions (such as publications, Research studies by students, baseline and endlines, communication materials, etc.), results of any internal planning process and relevant materials from secondary sources.

5.3. Action Against Hunger HQ Interviews As part of the evaluation, the evaluator will interview HQ stakeholders to get preliminary information about the project intervention office being evaluated (DRR Advisor etc.). Briefings by telephone must be agreed in advance.

5.4. Inception Report At the end of the desk review period and before the field work, the evaluator will prepare a brief inception report based on the format provided. The report will be written in English and will include the following sections: • Key elements of the Terms of Reference (TORs) to demonstrate that the evaluator will adhere to the TORs;

1http://www.alnap.org/resource/6199

5

• The methodological approach to the evaluation include an evaluation matrix in annex to specify how the evaluator will collect data to answer the evaluation questions, pointing out the limitations to the methodology if any and the choice of sites per field visit; • The data collection tools; • A detailed evaluation work plan and; • Statement of adherence to Action Against Hunger Evaluation Policy and outline the evaluation report format. The inception report will be discussed and approved by the mission level focal person (PQA Coordinator) while ELA unit (ACF-UK) will support in technical advisory role. The inception report will also be shared with HQ (ACF-USA) Technical staff for validation and approval.

5.5. Field Mission Primary data collection techniques The evaluator will interview key project stakeholders (expatriate/national project staff, local/national representatives, local authorities, humanitarian agencies, or donor representatives, communities) as per the list in Annex X. The evaluator will use the most suitable format for these interviews as detailed in the inception report. The evaluator is also expected to collect information directly from beneficiaries/Communities. For Triangulation of the findings, the evaluator could also conduct Focus Group Discussions (beneficiaries, non-beneficiaries, key informants – teachers and community leaders, government staff etc.) and surveys.

Field visits The evaluator will visit the project sites and the facilities provided to the beneficiaries (if any) according to select methods described in the inception report.

Secondary data collection techniques: Desk review The evaluator will further review complementary documents and collect project monitoring data or of any other relevant statistical data (as and when required).

Debriefing and stakeholders workshop The evaluator shall facilitate a learning workshop in country to present preliminary findings of the evaluation to the project and key stakeholders (ACF management and field staff, Local, Provincial actors and donor representativei); to gather feedback on the findings and build consensus on recommendations; to develop action-oriented workshop statements on lessons learned and proposed improvements for the future.

5.6. Evaluation Report The evaluation report shall follow the following format and be written in English: • Cover Page; • Summary Table to follow template provided • Table of Contents • Executive Summary must be a standalone summary, describing the intervention, main findings of the evaluation, and conclusions and recommendations. This will be no more than 2 pages in length • Background Information • Methodology describe the methodology used, provide evidence of triangulation of data and presents limitations to the methodology • Findings includes overall assessment of the project against the evaluation criteria, responds to the evaluation questions, all findings are backed up by evidence, cross-cutting issues are mainstreamed and; unintended and unexpected outcomes are also discussed

6

• Conclusions Conclusions are formulated by synthesizing the main findings into statements of merit and worth, judgements are fair, impartial, and consistent with the findings • Lessons Learnt and Good Practices presents lessons that can be applied elsewhere to improve intervention or country office performance, outcome, or impact and; identify good practices: successful practices from those lessons which are worthy of replication; further develop on one specific good practice to be showcased in the template provided in Annex VI • Recommendations should be as realistic, operational and pragmatic as possible; that is, they should take careful account of the circumstances currently prevailing in the context of the action, and of the resources available to implement it both locally. They should follow logically from conclusions, lessons learned and good practices. The report must specify who needs to take what action and when. Recommendations need to be presented by order of priority • Annexes These should be listed and numbered and must include the following: logical framework provided in Annex I, Good practice template provided in Annex VI, Evaluation Criteria Rating Table provided in Annex V, list of documents for the desk review, list of persons interviewed, data collection instrument, evaluation TORs

The whole report shall not be longer than 30 pages, 50 pages including annexes. The draft report should be submitted no later than 10 calendar days after departure from the field. The final report will be submitted no later than the end date of the consultancy contract. Annexes to the report will be accepted in the working language (i.e. English) of the country office.

5.7. Debriefing with the ELA Unit – ACF UK The evaluator should provide a debriefing to the ELA unit – ACF UK to discuss any issues related to the quality of the evaluation report.

5.8. Debriefing with Action Against Hunger HQ The evaluator should provide a debriefing with the relevant Action Against Hunger HQ on her/his draft evaluation report, and on the main findings, conclusions and recommendations of the evaluation. Relevant comments should be incorporated in the final report.

6. KEY DELIVERABLES

The following are the evaluation outputs the evaluator will deliver to the evaluation manager:

Outputs Deadlines Inception Report 10th Dec-2016 Stakeholders workshop 26th Dec-2016 Draft Evaluation Report 30th Dec-2016 Final Evaluation Report 10th Jan – 2017

All outputs must be submitted in English and under Word Document format.

The quality of the inception report and the evaluation report will be assessed by the ELA unit (ACF- UK) focal person and mission level PQA coordinator. The evaluator is expected to follow the format, structure and length as defined under section 5.4 and 5.6 above.

7. MANAGEMENT ARRANGEMENTS AND WORKPLAN

These evaluation TORs have been developed in a participatory manner based on inputs from relevant stakeholders and approval from Donor.

7

The evaluator will directly report to the evaluation managerii. The evaluator will submit all the evaluation outputs directly and only to the evaluation manager. The evaluation manager will do a quality check (ensure required elements are there) and decide whether the report is ready for sharing. The evaluation manager will forward a copy to key stakeholders for comments on factual issues and for clarifications. The evaluation manager will consolidate the comments and send these to the evaluator by date agreed between the evaluation manager and the evaluator or as soon as the comments are received from stakeholders. The evaluator will consider all comments to finalize report and will submit it to the evaluation manager who will then officially forward to relevant stakeholders.

7.1. Tentative Work plan NOTE: Consultants are expected to work 6 days a week (either Sundays/Fridays or whatever day the field office has off will not be paid) during their consultancy contract. Travel days are not paid as they are not working days as such.

Activities Evaluator Dates Working Days Evaluation briefing with the evaluation manager 0.5 05th Dec Interviews with HQ 0.5 06th Dec

Desk review, preparation of field work and prepare Inception 3 10th Dec Report Travel to the field (Travel will varies subject to expat and local 1 to 1.5 (not 11th Dec consultant) paid) In country interviews with ACF staff (Key project staff) 1 12th Dec

Field Travel, Field work, collection and analysis of secondary Min. 10 13th – 22nd Dec data and meeting with stakeholders Stakeholders Workshop in country 1 26th Dec

Travel back from the Mission (in case of expatriate consultant) 1 to 1.5 (not TBD paid) Evaluation debriefing with the evaluation manager 0.5 27th Dec

Evaluation debriefing with HQ 0.5 27th Dec

Draft Report Min. 5 30th Dec

Action Against Hunger-UK: Quality check and initial review by Min. 8 07th Jan 2017 the evaluation manager, circulate draft report to key stakeholders, consolidate comments of stakeholders and send to evaluator Final report on the basis of stakeholders, country office, HQ, Min. 3 10th Jan 2017 and Action Against Hunger -UK comments Total: 25 days

7.2. Profile of the evaluator The evaluation will be carried out by an international / Pakistani evaluation consultant with the following profile: • Knowledge about the Community Based Disaster Risk Management (CBDRM)/ Water, Sanitation and Hygiene (WaSH) projects in context of Pakistan and particularly in KPK.

8

• Significant field experience in the evaluation of humanitarian / development projects; • Relevant degree / equivalent experience related to the evaluation to be undertaken; • Significant experience in coordination, design, implementation, monitoring and evaluation of programmes. • Programmes (Well versed with OECD-DAC evaluation Criteria). • Good knowledge and experience of gender equality and non-discrimination; • Good communications skills and experience of workshop facilitation; • Ability to write clear and useful reports (required to produce examples of previous work); • Fluent in English, however for Pakistani national’s fluency in Urdu and Pushto language would be an asset. • Understanding of donor requirements; • Ability to manage the available time and resources and to work to tight deadlines; • Independence from the parties involved. • Applications from Pakistani nationals are strongly encouraged

Others: • Consultant will be responsible for his own health insurance • Consultant will be responsible for its own equioment

8. LEGAL AND ETHICAL MATTERS

The ownership of the draft and final documentation belong to the agency and the funding donor exclusively. The document, or publication related to it, will not be shared with anybody except Action Against Hunger before the delivery by Action Against Hunger of the final document to the donor.

Action Against Hunger is to be the main addressee of the evaluation and its results might impact on both operational and technical strategies. This being said, Action Against Hunger is likely to share the results of the evaluation with the following groups: • Donor(s); • Governmental partners; • Various co-ordination bodies.

For independent evaluations, it is important that the consultant does not have any links to project management, or any other conflict of interest that would interfere with the independence of the evaluation.

8.1. Intellectual Property Rights All documentation related to the Assignment (whether or not in the course of your duties) shall remain the sole and exclusive property of the Charity.

9. Dead line for Proposal: Interested candidates/ institutes should submit their technical & financial proposal along with updated profile on or before December 03, 2016 till 5:00 PM. Organization have right to reject all those proposals which will deliver after deadline. Technical & Financial Proposal need to submit in a sealed envelope with the clearly marked “Propsal for RNE Final Evaluation” at our office address as mentioned below, Logistics Department- Action Against Hunger, 3rd Floor, 65 West Executive Heights, Fazal e Haq Road, Blue Area, Islamabad. Tel: +92-51-2806241-2

9

10. ANNEXES TO THE TORs

I. Log frame Project Log frame II. Evaluation Criteria and Detailed Evaluation Questions III. List of Project documents for the desk review IV. List of people to be interviewed V. Evaluation Criteria Table VI. Good Practice Template

10

Annex I: Project Log frame

Annex D_Revised Annex E_Revised Logical Framework An Performance Measur

11

Annex II: Evaluation Criteria and Detailed Questions

To assess the project intervention against each evaluation criteria, the evaluator will respond to the following evaluation questions:

Design: • Have local capacities been considered and integrated into the project’s resilience approach?  Were project objectives are clear and realistic  Did a regular collection of monitoring data apart from the initial concept and design? Have systems been put in place by ACF to improve the monitoring and the delivery of support services?  To what extent have previous applicable project evaluation recommendations been considered in the design of this project? • To what extent did the project design take into account gender issues as per the ACF Gender Policy? To what extent the project mainstream gender equity in the design and delivery of activities (e.g. integrating women’s specific needs, Needs of persons with disabilities & oldage and capacities in the resilience approach)?  What measures have been taken to ensure transparency and accountability? How could ACF improve its communication with communities in order to improve transparency and accountability?

Relevance/Appropriateness:  Are the interventions aligned with local needs and priorities as well as government priorities.  Was the intervention in-line with donor policies and priorities?  What were the challenges faced (operational and programmatic) and how could they mitigated? Coherence:  Is the project implementation strategy in-line with the national strategies and policies to ensure consistency?  Does the project design fit within and complement existing initiatives by other organizations?  Are the project interventions being align with ACF country strategy and DRR strategy?  To what extent has ACF managed to coordinate effectively among different stakeholders?

Coverage:  To what extent were the most vulnerable members of the target population effectively covered by the project?  Were the criteria and indicators defined in the project suitable to identify the vulnerable population?  Is the geographic coverage being relevant and as per government priorities?  What are the remaining gaps or continuation of intervention that need to be covered within the target area?

Efficiency:  How economically were the resources/inputs converted into results?  Is the current operational model cost effective and efficient to achieve the results?

Effectiveness:  To which extent does the project reach the planned targets against the proposal indicators in a timely manner? What were the main challenges?

12

 What was the level of participation of the communities during the targeting process and the implementation of the program?  How has the project responded to positive and negative factors (both foreseen and unforeseen) that arose throughout the implementation process? Has the project team been able to adapt the implementation process in order to overcome these obstacles without hindering the effectiveness of the project?  How effectively were the project stakeholders (especially PDMA, DDMA, NDMA and academia) engaged in the project interventions?

Sustainability and likelihood of impact:  Does the project design include a sustainability strategy involving local communities and beneficiaries as well as other relevant stakeholders?  What measures, and with what success, did ACF ensure that all interventions were sustainable (including training, quality hardware, integration with government departments and resilience)? • Assess and evaluate ACF’s project exit strategy.  To what extent is there ownership and buy-in of the approach? Is it likely to be continued once the program ends?

13

Annex III: List of project documents for the desk review

The following documents will be reviewed by the evaluator during the desk review phase:

Documents Action Against Hunger Country Strategy National Strategy in Action Against Hunger relevant sectors (DRR, WaSH, etc.) Cluster Strategy in Action Against Hunger relevant sectors (DRR, WaSH, etc.) Any prior assessments for Design Intervention proposal and budget (and amendments if applicable) Partnership Agreement/Memorandum of understanding with University of Peshawar Logical framework Monitoring plan Project/Activity Scope Document (ASD/PSD) Activity Progress Reports Field visits reports Project implementation plan Training and supervision reports Beneficiaries monitoring reports Baseline and endline reports Surveys and research studies Intervention Villages and Beneficiries List Any other relevant document

Annex IV: List of people to be interviewed

The evaluator will interview the following stakeholders2:

Internal

Position and Organisation Country Director Deputy Country Director-Prog Deputy Country Director-Support WASH Co PQA Co Field Co - KPK Grants co Finance Co Logistic Co HR Co Senior Project Manager PQA Supervisor KPK Support units Program Team

2 This list is non-exhaustive.

14

External

Name Position and Organisation To be confirmed Clusters (Nut / FSL / WaSH) To be confirmed DOH To be confirmed PPHI / To be confirmed PDMA To be confirmed UN agencies To be confirmed Military Intelligence To be confirmed Agriculture department Target Communities and Based on sampling Beneficiaries

Annex V: Evaluation Criteria Table

The evaluator will be expected to use the following table to rank the performance of the overall intervention using the DAC criteria. The table should be included in annex of the evaluation report.

Criteria Rating Rationale (1 low, 5 high) 1 2 3 4 5 Design Relevance/Appropriateness Coherence Coverage Efficiency Effectiveness Sustainability Likelihood of Impact

Guidance for rating the evaluation criteria:

Rating Definition

1. Unsatisfactory Performance was consistently below expectations in most areas of enquiry related to the evaluation criteria. Overall performance in relation to the evaluation criteria is not satisfactory due to serious gaps in some of the areas. Significant improvement is needed. Recommendations to improve performance are outlined in the evaluation report and Action Against Hunger will monitor progress in these areas.

2. Improvement Performance did not consistently meet expectations in some areas of needed enquiry– performance failed to meet expectations in one or more essential areas of enquiry. Some improvements are needed in one or more of these. Recommendations to improve performance are outlined in the evaluation

15

report and Action Against Hunger will monitor progress in these key areas.

3. On average On average, performance met expectations in all essential areas of enquiry meets and the overall quality of work was acceptable. Eventual recommendations expectations over potential areas for improvement are outlined in the evaluation report.

4. Meets Performance consistently met expectations in all essential areas of enquiry, expectations and the overall quality of work was fairly good. The most critical expectations were met.

5. Exceptional Performance consistently met expectations due to high quality of work performed in all essential areas of enquiry, resulting in an overall quality of work that was remarkable.

Annex VI: Good Practice Template

The evaluation is expected to provide one (1) key example of Good Practice from the intervention/country office. This example should relate to the technical area of intervention, either in terms of processes or systems, and should be potentially applicable to other contexts whereAction Against Hunger operates. This example of Good Practice should be presented in the Executive Summary and the Main Body of the report.

Title of Good Practice (Max. 30 words)

Innovative Features and Key Characteristics (What makes the selected practice different?) Background of Good Practice (What was the rationale behind the good practice? What factors/ideas/developments/events lead to this particular practice being adopted? Why and how was it preferable to other alternatives?) Further explanation of chosen Good Practice (Elaborate on the features of the good practice chosen. How did the practice work in reality? What did it entail? How was it received by the local communities? What were some of its more important/relevant features? What made it unique?) Practical/Specific Recommendations for Roll Out (How can the selected practice be replicated more widely? Can this practice be replicated (in part or in full) by other Action Against Hungerprogrammes? What would it take at practical level? What would it take at policy level?) How could the Good Practice be developed further? (Outline what steps should be taken for the practice to be improved and for the country office to further capitalise on this good practice)

i Subject to their availability ii ELA Unit Focal Person – ACF UK while in field it will be PQA Coordinator.

16

Annexure 8: VDMC and Cluster Disaggregated Data

Cluster 1- Lower Dir and Upper Dir Districts Cluster 2 – Nowshera and Charsadda Districts

Cluster 1: Is First aid kit helped Cluster 1: Is First aid training you to save lives helped you to save lives 80% 91% 100% 86% 80% 80% 70% 70% 63% 80% 60% 60% Yes 50% Yes 40% 40% No 28% No 20% 30% 20% Don't Know 10% Don't Know 20% 6% 161%4% 3% 0 3% 20% 9% 0% 10% 0 0%

Male Female Total

Cluster 2: Is First Aid kit helped Cluster 2: Is First aid training you to save lives helped you to save lives

86% 90% 85% 82% 90% 78% 80% 72% 80% 70% 59% 70% 60% Yes 60% 50% 50% Yes No 40% 40% 28% Don’t Know 30% 21% Don’t Know 30% 22% 18% 20% 14% 14% 15% 7% 20% 10% 0 10% 0% 0% Male Female Total Male Female Total

Figure: Usefulness of First Aid kit and training in saving lives

Is First Aid kit helped you to save lives Is First Aid training helped you to save lives Male Female Total Male Female Total Cluster 1 Yes 64% 92% 77% 90% 100% 94% No 29% 0% 15% 10% 0% 6% Don't know 7% 8% 8% 0% 0% 0% Cluster 2 Yes 100% 88% 93% 100% 88% 93% No 0% 0% 0% 0% 0% 0% Don't know 0% 12% 7% 0% 12% 7%

Cluster 1: Is search and rescue kits Cluster 1: Is search and rescure helped you to save lives and training helped you to save lives property and property 94% 100% 90% 100% 84% 80% 80% 71% 67% 56% 60% 60% Yes 44% Yes No 40% 29% No 40% 21% 20% 12% Don't Know Don't Know 16% 0 0 20% 6% 7% 0% 0 0 3% 0%

Male Female Total

Cluster 2: Is search and rescue kits Cluster 2: Is search and rescure helped you to save lives and training helped you to save lives property and property 79% 79% 100% 80% 100% 70% 80% 69% 60% 51% 49% 60% 53% 50% 47% Yes Yes 40% 31% 40% Don't Know 30% 21% 21% Don't Know 20% 20% 0 0% 10% 0%

Male Female Total

Figure 2: Usefulness of search and rescue kits and training in saving lives and property Is search and Rescue kits helped you to Is search and Rescue training helped save lives and property you to save lives and property Male Female Total Male Female Total Cluster 1 Yes 90% 92% 91% 89% 91% 90% No 7% 0% 4% 11% 0% 6% Don't Know 3% 8% 6% 0% 9% 4% Cluster 2 Yes 100% 29% 59% 100% 100% 100% No 0% 9% 5% 0% 0% 0% Don't Know 0% 62% 36% 0% 0% 0%

Cluster 1: Is early warning system Cluster 1: Is early warning system kit helped you to save lives and training helped you to save lives property and property 100% 97% 96% 98% 92% 100% 100% 86%

80% 80%

60% Yes 60% Yes 40% No 40% Don't Know 14% 20% Don't Know 6% 3%0 0 2% 20% 4% 0% 0 2% 0%

Male Female Total

Cluster 2: Is early warning system Cluster 2: Is early warning kit helped you to save lives and system training helped you to property save lives and property 100% 92% 100% 100% 74% 80% 71% 80% 59% 52% 60% Yes 60% 48% Yes 36% 40% 26% No 40% 26% Don't Know 20% 8% 5% 3% Don't Know 20% 0 0% 0 0%

Male Female Total

Figure 3: Usefulness of EWS kit and training in saving lives and property Is early warning system kit helped you Is early warning system training helped save lives and property you to save lives and property Male Female Total Male Female Total Cluster 1 Yes 86% 100% 93% 86% 91% 89% No 7% 0% 4% 9% 0% 5% Don't 7% 0% 4% 5% 9% 7% Know Cluster 2 Yes 100% 82% 90% 100% 100% 100% No 0% 0% 0% 0% 0% 0% Don't 0% 18% 10% 0% 0% 0% Know

Cluster 1: Contribution of Project Cluster 2: Contribution of Project in the development of DRM plan in in the development of DRM plan the village in the village

100% 100% 85% 85% 69% 80% 69% 80% 52% 52% 60% 48% 60% 48% Yes Yes 40% 31% 40% 31% Don't Know 15% Don't Know 20% 15% 20% 0%

0%

Male Female Total

Figure: Contribution of project in the development of DRM plan in the village

Contribution of project in the Cluster 1 Cluster 2 development of DRM plan in Male Female Total Male Female Total the village Yes 90% 82% 87% 100% 91% 95% No 7% 0% 4% 0% 0% 0% Don't know 3% 18% 9% 0% 9% 5%

Cluster 1: Contribution of project Cluster 2: Contribution of project in helping identify of various kind in helping identify of various kind of hazards of hazards 120% 120%

100% 100%

29% 29% 80% 47% Don't Know 80% 47% Don't Know 60% 15% 72% 60% 15% 72% No No 8% 8% 40% Yes 40% Yes 56% 0 56% 0 20% 44% 20% 44% 28% 28% 0% 0% Male Female Total Male Female Total

Contribution of project in Cluster 1 Cluster 2 helping identify of various kind Male Female Total Male Female Total of hazards Yes 86% 95% 90% 100% 100% 100% No 7% 0% 4% 0% 0% 0% Don't know 7% 5% 6% 0% 0% 0%

Cluster 1: Is the project provided Cluster 2: Is the project provided support in identifying safe areas support in identifying safe areas in the village in the village

20% 20% 10% 10% Total 69% Total 69% 24% 24% 4% 4% Female 72% Female 72% 18% 18% 15% 15% Male 68% Male 68% 0% 20% 40% 60% 80% 0% 20% 40% 60% 80%

Don't Know No Yes Don't Know No Yes

Is the project contribution in helping Cluster 1 Cluster 2 identify safe areas in the village Male Female Total Male Female Total Yes 72% 95% 82% 100% 100% 100% No 21% 5% 14% 0% 0% 0% Don't know 7% 0% 4% 0% 0% 0% Total 100% 100% 100% 100% 100% 100%

Cluster 1: Received Community Cluster 2: Received Community based DRM session / based DRM session / training by the project training by the project

100% 100% 21% 20% 20% 21% 20% 20% 80% 80% 26% 36% 31% 26% 36% 31% 60% Don't Know 60% Don't Know 40% No 40% No 53% 49% 53% 49% 20% 44% Yes 20% 44% Yes 0% 0%

Received any community Cluster 1 Cluster 2 based disaster risk Male Female Total Male Female Total management session/training by the project Yes 86% 100% 92% 100% 100% 100% No 7% 0% 4% 0% 0% 0% Don't know 7% 0% 4% 0% 0% 0%

Cluster 1: After receiving training, Cluster 2: After receiving training, have you taken any action have you taken any action 60% 50% 48% 49% 60% 55% 50% 45% 50 % 4 8% 40% 36% 35% 50% 42% 40% 40% Yes Yes 30% 30% No No

20% 14% 16% 20% Don't Kno w Don't Know

10% 7% 10% 3% 0 2% 0% 0% Male Female Total Male Female Total

After receiving training Cluster 1 Cluster 2 have you taken any action Male Female Total Male Female Total Yes 86% 59% 76% 53% 61% 58% No 7% 41% 20% 47% 36% 40% Don't know 7% 0% 4% 0% 4% 2%

Cluster 1: Are the sessions/training Cluster 2 - Are the received will decrease the sessions/training received will vulnerability of community decrease the vulnerability of community 100% 88% 68% 70% 80% 68% 58% 60% 50%50% 60% 49% Yes 50% 47% 40% Yes No 40% 40% 26% 26% No Don't Know 30% 20% Don't know 20% 12% 6% 5% 0 4% 10% 0 2% 0% 0% Male Female Total Male Female Total

Are the sessions training Cluster 1 Cluster 2 received will decrease the Male Female Total Male Female Total vulnerability of community Yes 92% 88% 90% 100% 56% 75% No 4% 6% 5% 0% 18% 10% Don't know 4% 6% 5% 0% 26% 15%

Cluster 1 - Can you Cluster 2 - Can you perceive/think perceive/think that old people, that old people, PWDs, Women PWDs, Women and Children feel and Children feel empowered for empowered for future crisis future crisis

59% 56% 100% 81% 60% 54% 80% 62% 46% 50% 38% 60% 48% 36% Yes 40% Yes 25% 40% 24% No 30% 15% 19% 16% 19% No 20% 20% 0 Don't Know Don't know 0% 10% 3% 0 0%

Male Female Total

Can you perceive/think Cluster 1 Cluster 2 that older people, PWDs, Women and children feel empowered for futre Male Female Total Male Female Total crisis Yes 79% 44% 67% 76% 24% 46% No 14% 38% 22% 0% 47% 27% Don't know 7% 19% 11% 24% 29% 27% Total 100% 100% 100% 100% 100% 100%

Cluster 1: Do you have evacuation Cluster 2: Do you have plan for your village evacuation plan for your village 80% 76% 71% 90% 82% 70% 68% 80% 60% 70% 61% 50% 60% Yes Yes 40% 50% 39%43% No 40% No 30% 23% 24% Don't Know 30% 21% Don’t Know 18% 18% 18% 20% 131%5% 10% 20% 10% 10% 0 0 0% 0% Male Female Total Male Female Total

Do you have an evacuation plan for your Cluster 1 Cluster 2 village Male Female Total Male Female Total Yes 66% 100% 78% 88% 91% 90% No 28% 0% 18% 12% 0% 5% Don't know 7% 0% 4% 0% 9% 5%

Cluster 1: Is there any safe Cluster 2: Is there any safe place/evacuation center for the place/evacuation center for the community members in case of community members in case of hazards hazards

17% 2% 15% 11% Total 67% Total 88% 33% 0 Female 0 21% 67% Female 79% 10% 4% Male 23% 0 68% Male 96%

0% 20% 40% 60% 80% 0% 20% 40% 60% 80% 100%

Don't Know No Yes Don’t Know No Yes

Is there any safe place evacuation Cluster 1 Cluster 2 center for the community members Male Female Total Male Female Total in case of hazards Yes 76% 100% 84% 100% 79% 88% No 17% 0% 11% 0% 18% 10% Don't know 7% 0% 4% 0% 3% 2%

Cluster 1: Does the identified safe Cluster 2: Does the identified safe place/evacuation center has the place/evacuation center has the capcity to cater specific needs of capcity to cater specific needs of women women

80% 71% 61% 65% 100% 82% 60% 80% 68% Yes 54% 60% Yes 40% 29% No 40% 29% 23% 26% 18% 18% No 19% 9% 15% 20% 0 20% 13% Don't Know Don’t Know 0% 0 0%

Male Female Total

Does the identified safe Cluster 1 Cluster 2 place/evacuation center has the capacity to cater specific needs of women Male Female Total Male Female Total Yes 52% 0% 36% 100% 65% 80% No 41% 100% 60% 0% 18% 10% Don't know 7% 0% 5% 0% 18% 10%

Cluster 1: Will community use the Cluster 2: Will community use the evacuation center evacuation center

96% 76% 80% 68% 71% 100% 88% 70% 79% 60% 80%

50% Yes 60% Yes 40% No 30% 23% 24% Don't Know 40% Don't know 21% 20% 131%5% 20% 12% 10% 4% 10% 0 0% 0% Male Female Total Male Female Total

Will community use the evacuation center Cluster 1 Cluster 2 Male Female Total Male Female Total Yes 83% 81% 82% 100% 65% 80% No 7% 19% 11% 0% 0% 0% Don't know 10% 0% 7% 0% 35% 20%

Cluster 1: Do you think project Cluster 2: Do you think project was successful in creating safe was successful in creating safe evacuation routes evacuation routes

76% 80% 60% 55%

70% 58% 50% 43% 60% 56% 39% 36% 35% 50% 42% Yes 40% Yes 28% 26% 35% 40% No 30% 21% No 30% 24% 17% Don't Know 20% Don't know 20% 10% 10% 10% 0 0 0% 0% Male Female Total Male Female Total

Do you think project Cluster 1 Cluster 2 was successful in creating safe Male Female Total Male Female Total evacuation routes Yes 76% 100% 84% 64% 41% 51% No 17% 0% 11% 36% 44% 41% Don't know 7% 0% 4% 0% 15% 8%

Cluster 1: Was any disaster Cluster 2: Was any disaster resilience infrastructures resilience infrastructures constructed in your community by constructed in your community project by project

92% 100% 85% 100% 76% 80% 80% 60% 46% 48% 424%4% Yes 60% Yes 39% 38% No 40% 14% 14% 14% No 40% 24% Don't Know 20% Don't know 20% 8% 11% 0% 0 0 4% 0%

Male Female Total

Was any disaster Cluster 1 Cluster 2 resilience infrastructures Male Female Total Male Female Total constructed in your community Yes 86% 100% 91% 64% 41% 51% No 7% 0% 4% 36% 44% 41% Don't know 7% 0% 4% 0% 15% 8%

Cluster 1: Are you aware of any Cluster 2: Are you aware of any complaint mechanism complaint mechanism installed during the project 100% 94% 87% 75% 76% 80% 80% 70% 56% 60% Yes 60% 50% 38% Yes No 34% 40% 40% No 28% 26% Don't know 30% 25% 18% 19% Don't know 20% 10% 20% 3%3% 5% 4% 10% 0 0% 0% Male Female Total Male Female Total

Are you aware of any Cluster 1 Cluster 2 complaint mechanism Male Female Total Male Female Total

Yes 72% 77% 74% 100% 80% 89% No 21% 23% 21% 0% 20% 11% Don't know 7% 0% 5% 0% 0% 0%

Cluster 1: Have you received Cluster 2: Have you received timely feedback on it timely feedback on it 80% 100% 80% 100% 70% 75% 80% 60% 55% 474%7% 50% Yes 60% 505%0% No 36% 40% 30% No 40% Don't know 20% Don't know 25% 20% 9% 20% 6% 0 10% 0 0% 0% Male Female Total Male Female Total

Cluster 1 Cluster 2

Have you received Male Female Total Male Female Total timely feedback on it Yes 26% 19% 23% 20% 0% 8% No 22% 81% 46% 0% 0% 0% Don't know 52% 0% 31% 80% 100% 92%

Action Against Hunger

Annexure 3: List of documents for literature review

List of Documents for Literature Review.do

Annexure 4: List of KIIs

List of Key Informants Interview

Annexure 5: Data Collection Tools

Data Collection Tools.docx

Annexure 6: Revised Logical Framework

Annex D_Revised Logical Framework An

Annexure 7: Evaluation ToRs

RNE End of project Eval ToRs Draft V2.p

Annexure 8: VDMC and Cluster Disaggregated Data

VDMC and Cluster Disaggregated Data.

- 46 - | P a g e R N E I I I D R R P r o j e c t E v a l u a t i o n R e p o r t