Terms of Reference for the Evaluation of the Tropical Cyclone Pam Emergency Response Program VUT151 – March 2016
1.0 Background The Cyclone Pam Emergency Response Program VUT151 - was developed following the destructive impact of Tropical Cyclone Pam which struck the central and southern islands of Vanuatu in March 2015 causing widespread destruction across the country and affecting up to 70% of the archipelago’s population. International assistance was sought by the Vanuatu government given the scale of the disaster.
The humanitarian response was led by the Government’s National Disaster Management Office (NDMO) and supported though UN and non-government agencies and the management of an activated cluster system.
Early recovery assessments in the following months shaped a National Humanitarian Action Plan, which shifted the focus from life-saving support to strengthening basic services and supporting the restoration of livelihoods. This work has included the rebuilding of key infrastructure but also a particular emphasis on strengthening livelihood resilience; build back better principles and disaster risk reduction awareness and preparedness. Risk resilience work will remain a key platform in the government disaster management strategy given Vanuatu’s status as the most disaster prone country in the world.
2.0 ACT emergency response The Cyclone Pam response program (VUT151) included both immediate emergency response activities (3 months) and rebuilding and recovery initiatives (9 months) and engaged three ACT Alliance members and local implementing partners in Vanuatu. Act for Peace as the lead agency worked through its local partner Vanuatu Christian Council (VCC), Uniting World (UW) worked through Presbyterian Church of Vanuatu (PCV) and Anglican Overseas Aid (AoA) with support from Anglican Board of Mission (ABM) jointly worked with Anglican Church of Melanesia-Vanuatu (ACOMV). Response and recovery activities included food and non-food item and shelter distribution, provision of seeds and seedlings and food security training, fish farming pilot trials, water and sanitation services (drinking water, water purification and water tank and pump repair), small grants for evacuation centre repairs, repair of key infrastructure, psychosocial support training and church coordination initiatives. Activities were carried out across a number of Islands primarily in the central and southeast areas of Vanuatu including Tongoa, Epi, Efate, Erromango, Ambrym and in Torba and Penama provinces.
The principle goal of the VUT151 appeal was to ensure the safety, well-being and resilience of disaster affected populations in Vanuatu. Specific objectives determined by implementing partners were –
Act for Peace-Vanuatu Christian Council (AfP-VCC)
§ Improved health of cyclone affected populations on Tongoa, Epi, Tanna, Emao, Ambae, Efate and Ambrym Islands § Increased access to immediate temporary shelter on Efate and Tongoa Islands § Increased levels of food security for families through household and community gardens on Erromango, Ambrym, Tongoa and Efate Islands § Greater community access to safe disaster evacuation facilities on Efate Island § Improved capacity of church Communities to coordinate disaster risk reduction and response initiatives on Efate Island § Increased access to safe water supplies on Tongoa, Ambrym, Tanna, Emao, Ambae and Efate Islands
Tropical Cyclone Pam Emergency Response Program VUT151 Evaluation TOR
Uniting World- Presbyterian Church of Vanuatu (UW-PCV)
§ To increase capacity of church leaders to provide psychosocial support to crisis affected populations supporting improved mental health and well-being. § Increased ability for the Women’s Fellowship in Efate to support training for their women’s fellowship members as well as for the broader community across Vanuatu
Anglican Overseas Aid- Anglican Church of Melanesia-Vanuatu (AOA-ACOMV)
§ Increased levels of food security for communities affected by Cyclone Pam in PENEMA and TORBA Province
The ACT response program commenced on April 1st 2015 and will conclude on the 31st of March 2016. This is the first time the PANZ forum had directly responded to a disaster through an ACT Alliance appeal.
3.0 Scope of the evaluation Act for Peace seeks an experienced consultant to undertake an end of program evaluation of Cyclone Pam Emergency Response VUT151 program in Vanuatu. This will be carried out during April 2016 over a period of 15- 17 days. The evaluation will focus on the performance and deliverables across all three ACT members and their implementing partners engaged in the response from March 2015 – April 2016. This includes reflecting on key performance against project objectives as well as key learning’s specifically in relation to the response of church agency networks engaged in the Cyclone Pam response. It is expected the consultant will undertake a review of documents and project materials; will also solicit input via key informant interviews and will include field visits to some community project locations. The choice of locations for sites visits will be decided in consultation with the consultant.
The proposed evaluation will build on several other key studies already undertaken in the aftermath of Cyclone Pam and constitute part of the desk review. These include: • Act Alliance Situation Reports (SitRep) • Act for Peace Reflection Report • VCC Lessons Leaned Report • VCC mid-term report, • TC Pam Donor reports related case studies § Church working group lessons learned report § Field reports
4.0 Evaluation objectives The objectives of the evaluation should focus on the following points:
§ To the fullest extent possible, assess the relevance, effectiveness and efficiency of ACT partner and local implementing partner responses to Cyclone Pam under VUT151 § To assess actual partner performance and deliverables against proposed activities under VUT151 § To assess how responding agencies worked in coordination with humanitarian coordination mechanisms and the Vanuatu National Disaster Management Office during the response § To assess the effectiveness of the PANZ forum mechanism during the response and its engagement with ACT and internally with ACT partners § To assess how well the response identified and addressed key cross cutting issues including gender disability and protection issues1
1 See http://actalliance.org/documents/act-gender-policy/ and http://actalliance.org/documents/1134/ Tropical Cyclone Pam Emergency Response Program VUT151 Evaluation TOR
§ To identify both key challenges to the response and examples of ‘good practice’2 by ACT responding agencies and implementing partners § To provide clear, concrete recommendations on how responding members (international and local) can better prepare and respond to disasters through the ACT forum mechanism specifically in Vanuatu
5.0 Target groups: The evaluation will focus on a range of key stakeholder groups to ensure that a diverse range of opinions and experiences are represented. Key stakeholder groups will likely include: § Key community representatives from Cyclone affected communities § Implementing partner program staff § ACT partner program staff § Government officials i.e. from the National Disaster Management Office (NDMO) § Church leaders engaged in the response § NGO representatives engaged in the response
The evaluation’s primary audiences for the evaluation report will include - § ACT responding members § ACT partner organizations § ACT Forums § ACT funding members § Implementing partners § Other responding NGOs § ACT Secretariat § Back-donors § Wider constituency of ACT members
6.0 Key Evaluation Questions Key evaluation questions (KEQs) to guide the monitoring and evaluation for the Cyclone Pam Emergency Response VUT151 program have been identified. It is expected the consultant will review and confirm the KEQs and sub- questions with the Act for Peace Disaster Response Team who will consult with the other implementing partners and also in finalising the evaluation design. Draft Key Evaluation Questions developed for the Cyclone Pam Emergency Response VUT151 program are:
Achievements against proposed objectives and outcomes a. Considering the need to meet both the immediate and the early recovery needs of disaster affected populations, how relevant, were the ACT members’ responses? b. Were partner activities delivered efficiently, within budget and within the proposed timeframe? c. Were the interventions and outputs of the program consistent with the overall goal and the attainment of its objectives? d. What were the major factors influencing the achievement or non-achievement of the objectives?
Quality & Accountability a. To what extent were beneficiaries engaged at each stage of the response in terms of input and feedback? Did responding organisations develop appropriate accountability practices and feedback/complaints mechanisms for the response?
Gender, protection and cross-cutting issues a. How effective were ACT members in identifying the most vulnerable (including the elderly, persons living with disabilities, female headed households and children) through beneficiary selection processes?
2 See ACT Evaluation Policy and Guidelines - ACT humanitarian response May 2012 pg. 18 Tropical Cyclone Pam Emergency Response Program VUT151 Evaluation TOR
a. To what extent did the assistance address different protection needs and priorities of women, men, girls and boys through gender sensitive programming?
ACT policies, coordination and capacity (For consultation through ACT Australia based members) a. How effectively did PANZ members utilise ACT emergency response protocols during the response period? b. What were some of the key challenges and successes of working together under the PANZ forum? How might weaknesses be redressed/managed in future events?
Effectiveness of Faith Based Organisation (FBO) coordination a. What contribution did the program make in preparing communities for future disasters and to what extent are FBO institutions now better equipped to plan for and respond to disasters? b. How effective was coordination within the FBO network as well as with the National Disaster Management Office, and with other humanitarian actors during the response? Specifically, how effective was the Church Working Group in improving FBO coordination in the aftermath of the response? c. How might shortcomings in the effectiveness of FBO coordination be redressed for future events?
7.0 Evaluation methodology ü Need for mixed qualitative and quantitative methods ü Use of existing monitoring and evaluation data ü Must be participatory and may include but not limited to: selected interviews, focus group discussions, case study, survey, desk review ü Site visits to be decided but will include Efate and possibly one site visit to Tongoa or Ambrym
8.0 Expected outputs ü Methodology being used for the evaluation including expected data collection and analysis methods, with descriptions of any instruments used to collect needed information and any outcome and output indicators that are being proposed to measure performance, along with any associated baseline and target data ü Agreed Key Evaluation Questions ü Draft evaluation report ü Availability of any other relevant data, such as existing local, regional, or national data, or data from similar programs collected as part of the evaluation ü A process for verifying findings with key stakeholders ü Final report
9.0 Evaluation Team The consultant evaluator(s) should have excellent knowledge of monitoring and evaluation in theory and practice, with strong experience in an evaluation lead or evaluation facilitation role.
• Experience using mixed-methods (qualitative and quantitative) approaches to evaluation • Strong commitment to participatory and learning focused evaluation and knowledge/use of participatory tools • Knowledge of the Pacific context and the unique challenges of small island developing states • High level analytical skills, including experience in applying conceptual frameworks of analysis • Awareness and familiarity with Australian Government Minimum Standards for Monitoring and Evaluation • Fluent written and spoken English. Ability to speak Bislama an advantage.
Tropical Cyclone Pam Emergency Response Program VUT151 Evaluation TOR
10 Schedule and deliverables The consultancy is expected to commence the first week of April and be completed by 9th May2016. An indicative timeframe for key activities for the evaluation is below:
Deliverable Date for completion Responsibility Total Days Development of Terms of Reverence 1st -8th March Act for Peace (AfP) (ToR) for Evaluation Review and approval of ToR by PANZ 11th 14th March PANZ and ACT members members and ACT office Bangkok Identify and appoint 29 March AfP/ACT Evaluator/Evaluation team Jointly develop and approve 4th -6th April AfP/Consultant/ACT members 2 methodology and Key Evaluation questions Evaluation team undertakes desk 6th-9th April Consultant 3 review Visit schedule finalised 7th April AfP/ACT/implementing partners Evaluation commences 11th April Consultant/AfP/Implementing 5 partners Field evaluation completed 15th April Consultant/AfP/implementing partners Writing up of findings 18th-22 April 5 Submission of draft evaluation report 29th April Consultant Finalisation of evaluation report 6th May Consultant 1 Delivery presentation of report to key 9th May Consultant 1 stakeholders Total days for consultant 17 days
11. Final Report Production of final report should be t o a standard suitable for publishing (indicative outline below)
1. Cover page 2. Table of contents 3. List of abbreviations. 4. Executive summary that can be used as a stand-alone document 5. Introduction, stating objectives of the evaluation and evaluation questions 6. Program summary (objectives and key activities) 7. Methodology, including an indication of any perceived limitations of the evaluation 8. Presentation of the findings and their analysis 9. Conclusions 10. Learning and Recommendations 11. Appendices: • Terms of reference • Evaluation program (main features of data and activities carried out). • A list of interviewees (name, function and working environment) and places visited. • List of documents and bibliography used. • Details on composition of evaluation team • Link to methodological appendices:
Tropical Cyclone Pam Emergency Response Program VUT151 Evaluation TOR
• The evaluation proposal • Evaluation instruments
12. Roles & responsibilities
Consultant Reporting to Act for Peace Disaster Risk Response Coordinator, the consultant will: • Undertake consultancy within the timeframe agreed; • Prepare and submit a plan for the consultancy and amend as required; • Undertake desk based review of program documentation; • Develop the methodology for undertaking the evaluation with consortium staff including data collection and analysis; • Manage the evaluation; • Lead data collection and analysis processes; • Ensure information collected is jointly analysed and recommendations developed with team members; • Produce and submit an evaluation report including recommendations; and • Amend report in light of feedback from the evaluation steering group
Act for Peace; Uniting World (UW); Anglican Overseas Aid (AoA) Act for Peace will: • Develop, review and sign off on the Terms of Reference; • Participate in Consultant selection; • Provide all documents, information and materials relevant to project to enable Contractor/Consultant to understand the context of the Project for this assignment. • Coordinate the Country Program Partners involvement in the evaluation, including liaising with program teams to facilitate their participation in the evaluation and inputs into draft documents; • Coordinate the development of the terms of reference for the evaluation; and • Commission and manage the consultant • Provide one local organization contact person who is knowledgeable about the partner projects and who understands the goals of the evaluation • To respond in writing to Consultant written outputs/reports, within 5 days of receipt from the consultant.
Country Program implementing Partners (Vanuatu Christian Council; Presbyterian Church of Vanuatu (PCV) Anglican Church of Melanesia-Vanuatu (ACOMV).) • Coordinate field visits • Provide information as required to the evaluation team including ensuring other staff are available for meeting with evaluation team; • Provide/select the project staff participating on the evaluation team; • Ensure all implementation staff are available to participate in the reflection and analysis workshop; • Review and provide feedback on draft evaluation report; • Disseminate the evaluation report and lessons for accountability and learning purposes
13. Selection Process Expressions of Interest are invited from suitably qualified consultants (individuals or teams) with the skills and experience to undertake the evaluation. Proposals should include:
1. A cover letter introducing the evaluator/s and how the skills and competencies described above are met, with concrete examples. 2. A 2-page outline of the proposed evaluation process including: a. Proposed outline methodology. b. Management arrangements.
Tropical Cyclone Pam Emergency Response Program VUT151 Evaluation TOR
3. A budget covering consultancy fees, and expenses (including transport). An indicative budget from the consultant should is expected to be in the range AUD$9,000-12,000. Act for Peace will pay for all expenses associated with travel outside of Port Vila for the evaluation team as well as any workshop costs. 4. A CV for the evaluator(s). 5. One example of a previous evaluation, preferably with a participatory focus.
The consultant will be covered under Act for Peace general insurance whilst undertaking field activities but will need to provide evidence of professional indemnity cover.
The consultant will be required to sign contractor compliance with Act for Peace Code of Conduct, Child Protection Policy, Privacy Policy and Confidentiality Agreement.
Expressions of Interest should be submitted electronically to Tracey Robinson, Disaster Response Coordinator with “The Cyclone Pam Emergency Response Program Evaluation” in the Subject heading to [email protected] by 5pm Wednesday 23rd March 2016.
Proposals will be assessed according to the flowing criteria:
• Clear, credible, structured proposed methodology. • Profile, competencies and availability of the evaluator • Suitability of the financial proposal as per the methodology and program budget • Ability to complete the evaluation within the required timeframe
It is expected shortlisted candidates will be contacted by the Tuesday 29th March. Act for Peace is not bound to accept the lowest or any tender.
For further information please contact Tracey Robinson on +678 555 3517 or by email [email protected]
Tropical Cyclone Pam Emergency Response Program VUT151 Evaluation TOR