Australian Humanitarian Partnership
Response to the 2018 Papua New Guinea Highlands Earthquake
Evaluation August 2019
This humanitarian response to the 2018 Highlands earthquake in Papua New Guinea was funded by the Australian Department of Foreign Affairs and Trade (DFAT) and the New Zealand Ministry of Foreign Affairs and Trade and implemented by two NGOs - CARE and CAN DO.
This evaluation of the Papua New Guinea 2018 Highlands earthquake response was led by independent consultant Peter Chamberlain and was funded by DFAT and MFAT and implemented as a joint review by the donors and the implementing NGOs.
The evaluation was supported by the Australian Humanitarian Partnership Support Unit and Geosciences Australia.
Cover photo: An enumerator undertakes an interview in, Kundaka, Southern Highlands, using the SenseMaker tool.
Contents Acronyms ...... ii Executive Summary ...... iv 1. Introduction ...... 1 1.1 CARE proposals - summary ...... 1 1.2 CAN DO proposals - summary ...... 2 2. Methodology ...... 4 2.1 Assessment Rubric ...... 4 2.2 SenseMaker ...... 6 2.3 Evaluation Team ...... 7 2.4 Field Work ...... 7 2.5 Constraints and Limitations ...... 7 3. Findings – CARE ...... 9 3.1 Context ...... 9 3.2 Was the CARE earthquake response appropriate and relevant? ...... 11 3.3 Was the CARE response effective? ...... 13 3.4 How inclusive was the CARE earthquake response? ...... 17 3.5 How efficient was the CARE earthquake response? ...... 19 3.6 Did the CARE earthquake response reinforce local capacity/leadership? ...... 20 3.7 How transparent and accountable was the CARE response? ...... 21 3.8 How adequate were CARE’s M&E practices? ...... 22 4. Findings – CAN DO ...... 23 4.1 Challenges in Assessing the CAN DO Project ...... 23 4.2 Context ...... 24 4.3 Was the CAN DO PNG earthquake response appropriate and relevant? ...... 25 4.4 Was the CAN DO earthquake response effective? ...... 28 4.5 How inclusive was the CAN DO PNG earthquake response? ...... 30 4.6 How efficient was the CAN DO PNG earthquake response? ...... 31 4.7 Did the CAN DO PNG earthquake response reinforce local capacity/leadership? ...... 33 4.8 How transparent and accountable was the CAN DO earthquake response? ...... 34 4.9 How adequate were CAN DO’s M&E practices? ...... 34 5. Findings – AHP and Donors ...... 35 6. Conclusion ...... 36 Annex 1: Evaluation Team and Schedule ...... 38 Annex 2: SenseMaker Questions ...... 41 Annex 3: CARE Performance Against Targets ...... 50 Annex 4: Bibliography ...... 52 Annex 5: List of Key Informants ...... 53 Annex 6: Map of Affected Areas ...... 55
Acknowledgements
This evaluation could not have been completed without the cooperation and support of many staff representing the implementing partners, donors and the Australian Humanitarian Partnership. Particular thanks go to Jordan Hoffman, (Senior Program Officer, CARE), André Breitenstein, (Program Coordinator, CAN DO Coordination Unit), Julius Nohu (Disaster Hub Manager, Caritas), Jessica Kenway, AHP Support Unit Monitoring, Evaluation and Learning Manager and Martyn Hazelwood, Geoscience Australia, Director, Regional Development Section, Community Safety Branch.
Page | i
Acronyms
ADAPT Agriculture Drought Recovery and Adaption Training
ADF Australian Defence Force
ADRA Adventist Development and Relief Agency
AHC Australian High Commission
AHP Australian Humanitarian Partnership
AUD Australian dollars
CAN DO Church Agencies Network – Disaster Operations
CHS Core Humanitarian Standard on Quality and Accountability
CLTS Community Led Total Sanitation
DFAT Department of Foreign Affairs and Trade (Australia)
DMT Disaster Management Team
ECHO Directorate-General for European Civil Protection and Humanitarian Aid Operations
ECPNG Evangelical Church of Papua New Guinea
ELCPNG Evangelical Lutheran Church of Papua New Guinea
GBV Gender-Based Violence
HHH Highlands Humanitarian Hub
IDP Internally Displaced Person
IOM International Organisation of Migration
M&E Monitoring and Evaluation
MFAT Ministry of Foreign Affairs and Trade (New Zealand)
NARI National Agricultural Research Institute
NDC National Disaster Centre
NGO Non-governmental Agency
NZD New Zealand dollars
NZDF New Zealand Defence Force
NZDRP New Zealand Disaster Response Partnership
PDC Provincial Disaster Coordinator
PDM Post-distribution Monitoring
PHAST Participatory Hygiene and Sanitation Transformation
Page | ii
PIP Project Implementation Plan
PNG Papua New Guinea
PNGADP Papua New Guinea Assembly of Disabled Persons
RUTF Ready to Use Therapeutic Food
SBF Strickland-Bosavi Foundation
TORs Terms of Reference
UNFPA United Nations Population Fund
UNICEF United Nations Children’s Fund
WASH Water, Sanitation and Hygiene
Page | iii
Executive Summary
Overview
On 26 February 2018 the Highlands of Papua New Guinea (PNG) were struck by an earthquake with a magnitude of 7.5 followed by a powerful series of aftershocks. It was the largest earthquake to strike the region since 1922. An estimated 544,368 people were affected and 270,442 were in need of assistance.
This evaluation examines two one-year projects, implemented by CARE and the Church Agencies Network – Disaster Operations (CAN DO) using funds from the Australian Humanitarian Partnership (AHP) and the New Zealand Ministry of Foreign Affairs and Trade (MFAT). CARE received a combined total of just under AUD$2m from AHP and MFAT and CAN DO received AUD$2.36m.
These projects have operated in very difficult circumstances, in remote and challenging environments with little infrastructure and low levels of civil society and local government capacity. Logistics in these Highland Provinces is immensely challenging and insecurity and violence have been a constant concern particularly in CAN DO’s areas of operation.
CARE worked across 44 villages in four remote areas; Dodomona, Mougulu (Western Province) Huiya and Walagu (Hela). All are close to the epicentre of the earthquake and include host populations and Internally Displaced People’s (IDPs) whose mountain villages were devastated by the disaster and moved to safer locations nearby. CARE’s work focused on the early recovery phase as part of a holistic response including WASH, shelter, food security and livelihoods, nutrition, gender and protection.
CAN DO implemented its response through four church organisations: Caritas, the United Church, the Adventist Disaster and Relief Agency (ADRA) and the Evangelical Lutheran Church of PNG (ELCPNG). The project worked across 40 separate villages and satellite settlements located in Hela and Southern Highlands Provinces. Its main focus was on WASH based on the provision of water systems to public buildings. Security conditions affected implementation in both provinces. Logistics problems led to considerable delays in implementation for CAN DO and the project was granted a no-cost extension until the end of June 2019, which, as a result of initial findings of the evaluation has been extended until the end of August.
This evaluation used a standard mixed methods approach including a review of relevant documents, key informant interviews and focus groups. Of note is that the evaluation also trialled the SenseMaker monitoring and decision-support tool in collaboration with Geoscience Australia and Cognitive Edge.
CAN DO projects are spread across two large, remote provinces and a combination of logistics and security issues meant that the evaluation team was unable to spend sufficient time with what appear to be its stronger implementing partners which must be considered when reading this report.
An absence of reliable data about the target areas – for example the lack of reliable population figures – was an additional challenge for both agencies.
This evaluation is one of four pilot evaluations designed to contribute to learning about AHP activations and as such it also includes a focus on four cross-cutting issues; inclusion, transparency, localisation and cost effectiveness.
Page | iv
Main Findings The AHP response implemented by Care and CAN DO was appropriate and relevant, and both responses reinforced local capacity and leadership. The CARE response achieved good standards of inclusivity, efficiency and effectiveness and transparency and accountability to local populations. The CAN DO response did not reach acceptable standards in these areas, but it is understood that the agency is addressing these issues during an extension granted to complete the project. The evaluation identified significant lessons both of good practices, and areas to improve, that if heeded will significantly strengthen future NGO responses in PNG.
Was the AHP/NZDRP PNG earthquake response appropriate and relevant? CARE: Highly appropriate and relevant. CARE focused on some of the worst affected communities in remote areas which have very little infrastructure or government services. CARE was the only agency to continue to provide support through the recovery phase in these areas. CARE documented a needs assessment and local people confirmed that the project interventions – WASH, shelter and livelihoods - reflected their needs. The project is consistent with the terms of the AHP activation, Australia’s Humanitarian Strategy and New Zealand Aid Programme Strategic Plan 2015-19. CARE’s response was strongly engaged with coordination mechanisms at the national level and took the lead in establishing the Highlands Humanitarian Hub.
CAN DO: Appropriate and relevant. CAN DO worked in a range of communities affected to differing degrees by the earthquake which received little outside assistance. The focus on WASH, using a simple, standardised water supply approach was appropriate and reflected the needs of the affected population. The four CAN DO partners largely used locally generated needs assessments done at different times which were hard to consolidate.
CAN DO interventions are consistent with Australian and New Zealand strategies and policies and with the terms of the AHP activation. CAN DO partners were active in national level coordination – but like CARE found less value in provincial coordination bodies and focused instead on the Highlands Humanitarian Hub. CAN DO’s response is also considered appropriate and relevant.
Was the AHP/ NZDRP PNG earthquake response effective? CARE: Effective. Outputs and outcomes were clear and – in most cases met or exceeded. Livelihoods training, seeds and tools reached almost double the intended targets. Training targets including nutrition training for health workers, and effective family partnerships were met as were WASH and shelter targets. Gender based violence and sexual exploitation and abuse training had creditable results, but did not meet their targets due to reliance in the proposal on Government figures, which significantly overestimated the population in affected villages. Social infrastructure (such as WASH committees) has not uniformly survived the project and external factors mean that food security remains problematic despite the best efforts of the project. Innovative approaches – such as the use of a mobile cinema to spread hygiene education and other messages were noted by the evaluation.
CAN DO: Not effective to date. The extremely challenging operating conditions have contributed to a failure to complete the proposed activities at the time of the evaluation. Projects should be reassessed at the end of the no-cost extension. At the time of the evaluation a significant proportion of the WASH infrastructure was incomplete and the related activities (hygiene promotion, WASH groups) had consequently been delayed. A trauma counselling and peacebuilding component had been successfully completed and was well received by participants.
Page | v
How inclusive was the AHP/ NZDRP PNG earthquake response? CARE: Highly inclusive. CARE has made remarkable efforts to ensure inclusion throughout the response, conducting a rapid gender analysis of the disaster context and consulting women as part of the needs assessment. It provided gender training for its staff and provided gender-based violence, child protection and family partnership training as part of the project. From the first week of the response, CARE worked with PNG disability organisations to conduct a disability needs assessment and involved the same agencies in responding to these needs. Working closely with the Papua New Guinea Assembly of Disabled Persons (PNGADP), which is also a partner of the AHP Disaster Ready program has also supported sustainability.
CAN DO: Not inclusive to date. The CAN DO response was not systematically inclusive, although some examples of good practice were found – such as prioritising access to toilets for women (ELCPNG) and people with a disability (ADRA). Completion of remaining project activities may afford opportunities to build on these approaches. A definitive assessment of inclusion should wait until the conclusion of the project.
How efficient was the AHP/ NZDRP PNG earthquake response? CARE: Efficient. CARE completed its project on time and on budget and is assessed as efficient. Some activities occurred later than planned, mainly due to the challenges of working with local government bodies which had different processes and priorities. Staff turnover and sometimes inconsistent contextual knowledge of some local and international staff contributed to confusion at times with the local partner and beneficiaries which could have been mitigated by drawing more systematically on the local knowledge of the key community based partner.
CAN DO: Not efficient. The project has struggled with a difficult security and logistical context. This has resulted in a failure to deliver project supplies in a timely manner – and logistical systems require urgent attention before further disaster responses are undertaken. CAN DO worked with local churches which provided some cost savings. For Caritas and the United Church these were outweighed by communication problems with their local church representatives who were poorly informed about the project.
Did the AHP/ NZDRP PNG earthquake response reinforce local capacity/leadership? CARE: Successful. CARE worked extensively with PNG partners including government agencies, local NGOs, disability organisations and the main local church and has generally reinforced their capacity. Some challenges occurred in the relationship with its key local partner based around the informal, community based nature of the organisation, an unregistered PNG local NGO, with no formal management or governance structure and no reliable funding source beyond its own members. CARE could have worked more closely with the partner at the beginning of the response to ensure more clarity on the terms of the partnership. This might have avoided some of the misunderstandings which occurred.
CAN DO: Successful. Without the involvement of the churches and the respect which communities accord them, secure access to many of the project areas would be severely constrained. The four church partners are critical elements in civil society in the Highlands and their involvement in the response has strengthened their capacity to varying degrees. Developing their humanitarian capacity will be a long term process and it is important that project activities are completed successfully to strengthen their competence, confidence and credibility.
Page | vi
How transparent and accountable was the AHP/ NZDRP PNG earthquake response? CARE: Highly transparent and accountable. CARE has taken significant steps to ensure accountability and transparency. It established disaster committees in each community and the members of these groups provided the main channel of communication with the project. CARE conducted extensive post distribution monitoring. It kept a register of community feedback and the agency acted on the feedback it received. Management of community expectations by the provision of project plans is one suggested improvement: some community members had incorrect and unrealistic impressions of what the project would do.
CAN DO: Not transparent and accountable. Local church officials are regarded as key intermediaries by communities and the project, rather than formal mechanisms. Accountability and transparency varied between the partners. ELCPNG and ADRA had some mechanisms in place but this was less evident for Caritas and the United Church.
How adequate were the NGOs’ M&E practices during the response? CARE: Highly successful. Project documents provide a sound basis for monitoring and evaluating the response. Changes to the original design have been documented. Extensive post distribution monitoring and an After Action Review were conducted and CARE’s response to these exercises is documented.
CAN DO: Not successful. Project documents provide inconsistent and less tangible targets against which to monitor progress. Some M&E activities have occurred but on the evidence provided, it is not clear that they informed or influenced management of the project.
The evaluation also noted generally strong relations between donors and implementing partners, although formal reporting did not sufficiently update stakeholders on developments and problems with implementation, particularly for the DFAT Post – which is required to report to the PNG Government on the progress of activities it supports.
Key issues for the sector in PNG First, the conditions in PNG are very challenging - particularly in the Highlands which are faced with chronic underdevelopment, frequent insecurity and very weak infrastructure and government services. It is important to set realistic expectations for what can be achieved in relatively short emergency response and early recovery interventions. For example there are likely to be trade-offs between timeliness and developing the skills of local partners.
CARE has made impressive efforts to address gender and disability inclusion which deserve to be documented and shared as an example of good practice – but again there need to be realistic expectations of what can be achieved in an isolated project.
Localisation must be seen as a long term process; both government and civil society are relatively weak, particularly at provincial and sub-provincial levels, which is important in a country as ethnically diverse as PNG. Churches provide critical leadership and are major providers of health and education services, but they will require prolonged engagement to become consistently effective humanitarian actors which embrace the standards and principles of the sector.
The role of the private sector in this response was remarkable and requires deeper consideration than is possible in this evaluation. It is important that dialogue between humanitarian and private sector actors continues between disasters. Whilst there was an inevitable clash of organisational cultures
Page | vii
during the earthquake response, they have skills and capacities which should be complimentary. The private sector has resources and logistical capacity which are critical in the first phase of a response, whilst the NGOs can offer humanitarian expertise and the capacity to support both the initial response and the longer term recovery process.
The evaluation used SenseMaker tools and methodology for the first time in PNG, which proved to be a potentially important innovation. Whilst there are lessons to be learned about its use, it undoubtedly provided important evidence based on the perspectives and experience of those affected by the disaster. It allowed evaluators to disaggregate data according to gender, age, location and between IDPs and host populations.
The recommendations of the evaluation are as follows:
CARE: Recommendation 1: Depending on available funding, CARE should consider whether it can facilitate follow-up nutritional training by the Ministry of Health either directly or through other stakeholders. This would reinforce the gains made by previous nutrition training and allow monitoring of nutrition and food security in the project areas.
Recommendation 2: CARE should document its work on disability in the earthquake response as an example of best practice and as something that can be built upon in future humanitarian and development work in the Pacific.
Recommendation 3: CARE should review its approach when partnering with informal, community based groups and attempt to formalise its relationships as it would do with a local NGO so that mutual expectations are clearer. This should be based on an assessment of the community groups’ capacities and discussion of the aims and objectives of the response.
Recommendation 4: CARE should review its recruitment deployment and induction for future PNG disasters. If recruits with PNG experience, or with experience of the project area for PNG nationals, are not available they must be thoroughly briefed on their roles and the operating context.
Recommendation 5: For future responses, CARE should put up in addition detailed written summaries of its planned project activities noting the role expected of beneficiary communities to further enhance transparency and to minimise potential misunderstandings.
CAN DO: Recommendation 6: The following actions are required for the successful completion and documentation of the CAN DO Project:
6a CAN DO’s final project report should be informed by verification of the status of project activities in the forty villages included in the project and of overall progress against the targets, outputs and outcomes in the donor proposals.
6b CAN DO and its implementing partners must urgently provide the management logistical and technical resources in the field to ensure that the project is completed successfully by the end of the no-cost extension.
6c CAN DO and its implementing partners should develop a follow up plan for its hygiene promotion activities in accordance with the recommendations of the training provider.
Page | viii
Recommendation 7: In order to strengthen its operational capacity for future disaster responses, CAN DO should:
7a Following the completion of the project CAN DO partners should conduct and document a workshop to learn from their different approaches to WASH implementation.
7b CAN DO should conduct and document a review of its procurement and logistics systems. It should commit to a clear timeline for implementing the findings of this review.
Recommendation 8: Future CAN DO projects require better documentation. Targets and revisions to plans should be transparent and ideally broken down by implementing partner. There should be greater clarity on how donor funds are combined.
Recommendation 9: CAN DO should consider options for supporting further trauma counselling and peacebuilding training in the Highlands Region.
Recommendation 10: CAN DO and its partners need to reflect on their approach to gender and inclusion and strengthen it. This should involve the identification of good practice within their network – but also engagement with agencies with specialist skills and experience.
Recommendation 11: Caritas and United Church must develop more effective ways of working with their local branches and building their capacity. These should include involvement in needs assessment, clarity on project plans and timely provision of funds as required. A dialogue is necessary between the different levels of church partner organisations at all stages of the project.
AHP and Donors: Recommendation 12: An in-country workshop involving the AHC and implementing partners to discuss project developments should be considered in large-scale disaster responses. It should be timed to coincide with existing AHP activities and reporting requirements to reduce the additional workload.
Recommendation 13: The Australian Humanitarian Partnership should continue to use and develop SenseMaker as a tool for humanitarian monitoring and evaluation.
Page | ix
1. Introduction
On 26 February 2018 Papua New Guinea (PNG) was struck by an earthquake with a magnitude of 7.5, with its epicentre in a remote area of the Highlands Region. It was the largest earthquake to strike the region since 1922 and was followed by aftershocks, including one measuring 6.7M in the same area on 8 March. On 1 March 2018, the Government of PNG declared an emergency in Hela, Southern Highlands, Western and Enga provinces and estimated that 544,368 people were affected and that 270,442 were in need of assistance.1
The affected areas are remote and difficult to access, and experience high levels of insecurity. These factors led to an initial response dominated by major private sector actors, particularly Oil Search and Exxon Mobil who have a long term presence in the region.2 The initial delivery of relief supplies was also supported by military resources particularly the Australian and New Zealand Defence Forces (ADF and NZDF).
PNG’s management of the response evolved in the early days of the response. Oversight is usually through the National Disaster Centre (NDC) and a Disaster Management Team (DMT) is set up to coordinate the response. However, on 1 March, the Government appointed an Emergency Controller to lead the National Emergency Disaster Restoration Team, overseeing relief and recovery efforts.3
In addition to the provision of relief supplies through the ADF and private sector responders, the Australian Department of Foreign Affairs and Trade (DFAT) funded UN organisations including UN Women, United Nations Children’s Fund (UNICEF) and United Nations Population Fund (UNFPA). It also responded through its NGO network, the Australian Humanitarian Partnership (AHP). On 20 March it activated the AHP announcing the availability of up to AUD$3m for projects of up to 12 months’ duration. It stipulated that proposals should have: “…a focus on early recovery, provide support to re-establish livelihoods, and improve shelter, WASH with a particular focus on the needs of women, girls, and people with a disability. In doing so, partners must work in coordination with other partners Australia is funding including UN agencies, particularly on protection issues.”
It also noted that: “Only partners who are currently responding to the crisis, have well established distribution networks and proven access to affected areas will be considered.4”
CARE and CAN DO submitted successful proposals on 23 March and grant orders were signed to allow implementation to begin by the end of the month. Extensions were subsequently granted. In CARE's case, activities were completed as scheduled by the end of March 2019 – and the extension simply allowed for the completion of this final evaluation. In CAN DO’s project, logistical delays meant that extra time was needed for the completion of the project. Final reporting was moved to 31 August 2019.
1.1 CARE proposals - summary CARE’s project focused on the needs of 44 remote villages, 32 of which are located around 4 central villages, close to the epicentre of the earthquake: Huiya and Walagu in Komo-Magarima District, Hela Province and Dodomona and Mougulu in North Fly District, Western Province.5 The AHP project
1 Papua New Guinea: Highlands Earthquake Situation Report No. 1, 10 March 2018 2 It is estimated that the private sector delivered 64% of the assistance by value in the response – see Extractives and Emergencies the Papua New Guinea Earthquake Response, the Humanitarian Advisory Group, December 2018 3 Papua New Guinea: Highlands Earthquake Disaster Management Team Response Plan, 28 March 2018 4 E-mail from Mark Tattersall, 20 March 2018 5 A map showing earthquake affected areas is provided in Annex 6
Page | 1
provided AUD$1,500,000 and focused on early recovery needs in a range of sectors: WASH, food security / livelihoods, and gender and protection. CARE aimed to work with local partners - the Strickland-Bosavi Foundation (SBF) and the Evangelical Church of Papua New Guinea (ECPNG). The objectives and outcomes proposed in the Project Implementation Plan (PIP) are as follows:
Overall Support the immediate relief and early recovery needs of 10,301 (48% women) people directly objective and including indirectly 33,000 severely affected by the Highlands Earthquake across Western Highlands and Hela provinces by providing WASH, Livelihoods, and Gender and Protection support.
Outcome 1 An estimated 30,311 of the most vulnerable women, men, girls and boys including those with disabilities, have their basic water, sanitation and hygiene needs met
Outcome 2 Support 500 Households or 2,500 earthquake-affected people through livelihoods rehabilitation to restore food security, dignity and quality of life
Outcome 3 6000 girls, boys, men and women, including those with disabilities have access to information, training and support on gender, protection and inclusion. MFAT provided funding that was designed to complement the AHP project. CARE was assessed and funded by the New Zealand High Commission as a new partner. A proposal for NZD 500,000 was submitted in April 2018 and reports give the project period as 26 June 2018 - 30 June 2019. Key outcomes were:
Number Outcome
Outcome 1 Decreased human suffering associated with natural disasters and armed conflict.
Outcome 2 Improved food security for earthquake-affected households
Outcome 3 Improved resilience, dignity, and quality of life through restored shelter for earthquake- affected households
Outcome 4 Improved nutritional status among earthquake-affected communities
It should be noted that the largest component of the MFAT project was nutrition. The shelter component was relatively minor, designed to complement a larger Canadian funded initiative.
1.2 CAN DO proposals - summary The CAN DO AHP project focused on WASH and - to a limited extent shelter.6 Activities were to be implemented in 40 villages in Southern Highlands and Hela Provinces,7 using the capacity of four member churches present in the disaster affected areas: the Catholic Church, the United Church of PNG (UCPNG), The Evangelical Lutheran Church of PNG (ELCPNG) and Adventist Development and Relief Agency (ADRA). It should be noted that the CAN DO project was significantly revised after its initial submission and shifted its focus from disaster response to early recovery in recognition of the changes which took place as needs assessments were being conducted. Following discussions with the DFAT Post in April 2018, this change was agreed and the project expanded its reach from 12
6 The shelter component refers to the repair of buildings to be used as rain water catchments 7 The initial proposal and the evaluation TORs also mention Enga and Western Province – but these are not included in the Project Implementation Plan.
Page | 2
villages to 40 and CAN DO added a fourth Church organisation – the Evangelical Lutheran Church of Papua New Guinea (ELCPNG) to the initiative. The overall objective as defined in the PIP is:
58, 972 individuals (59% female, 41% male, 15% people with disabilities) are reached with life-saving Water, Sanitation, Hygiene (WASH) and shelter solutions Project outcomes are8:
Outcome 1 Affected communities have access to adequate Water, Sanitation and Hygiene
Outcome 2 Community buildings that have been damaged or destroyed attain a basic shelter and water catchment solution
CAN DO accessed MFAT funding through Caritas and ADRA in New Zealand and is entitled: Caritas & ADRA Supporting Early Recovery - A Joint Approach in PNG. The proposal is for NZD$1,005,115 and articulates the following outcome and outputs:
Outcome Improved physical and mental health of affected communities
Output 1 To provide access to safe water to targeted affected communities and health care centres
Output 2 To provide adequate sanitation facilities in targeted affected communities
Output 3 To provide medical supplies to clinics in targeted affected areas
Output 4 To build local mental resilience of targeted affected communities
Funds for the WASH activities appear to have been pooled. A joint progress report indicates that AHP funds supported 31 villages rather than the 40 indicated in the PIP, with the remaining nine villages funded by MFAT. It does not state which villages are supported by which donor. The report does not mention the provision of medical supplies proposed in the MFAT proposal.9 This evaluation is one of four pilot evaluations designed to contribute to learning about AHP activations and as such it also includes a focus on four cross-cutting issues, inclusion, transparency, localisation and cost effectiveness.10
8 It should be noted that the document is not consistent. Annex 3 – the Project Implementation planning matrix – includes a third outcome relating to Monitoring and evaluation. 9 AHP/NDRF Activation - Progress Report 31 March – 30 October 2018 10 See: “ Pilot Evaluations of four humanitarian responses through the Australian Humanitarian Partnership”
Page | 3
2. Methodology
This evaluation was led by an independent team leader, supported by representatives of both NGOs which ensured proper accountability, but also learning. The methodology for the evaluation is defined in the TORs as a participatory mixed methods approach. It should be noted that the key questions included in the TORs are consistent with the Strategic Result Statements provided by the MFAT DFAT Humanitarian Monitoring and Evaluation Framework for the Pacific (2018) and that this in turn reflects commitments to The Grand Bargain, the Sphere Humanitarian Charter and the Core Humanitarian Standard on Quality and Accountability (CHS).
A document review was conducted of material provided by implementing partners and donors, including key project documents, monitoring data and relevant policies and strategies. This was supplemented by publicly available documents on the overall humanitarian response to the Highlands Earthquake. Fifty-two key informant interviews (41 male, 12 female) were conducted by the team leader at a range of levels involving implementing agency staff, donors, other humanitarian actors, local partner organisations, government, community and religious leaders. A list of key informants is provided in AAnnex 5. Nine focus groups including around 135 people (74 male, 61 female) were conducted in affected communities and specific interviews were held with vulnerable groups, such as people living with a disability (six interviews).
2.1 Assessment Rubric The evaluation plan also includes a rubric (see below) designed to assess the relative success of each project against the evaluation questions and these ratings are included in the text. Each evaluation question references relevant Core Humanitarian Standards (CHS). It should be noted that for some questions the ratings for CAN DO projects cannot be definitive because the evaluation was conducted during the no cost extension before completion of most activities.
Rating Criteria
Highly Appropriate, The response will be considered highly appropriate, effective and inclusive etc. if Effective, Inclusive etc. there is significant evidence for all of the criteria for each evaluation question.
Appropriate, Effective, The response will be considered appropriate, effective and inclusive etc. if Inclusive etc. evidence is available to meet all of the assessment criteria for each of the evaluation questions.
Partially appropriate, The response will be considered partially appropriate, effective and inclusive etc. effective, inclusive etc. if half or more of the criteria are met for each of the evaluation questions.
Not appropriate, effective, The response will be considered to not have been appropriate, effective, or inclusive etc. inclusive etc. if less than half of the criteria are met for each of the evaluation questions.
Evaluation Question Assessment Criteria
1. Was the AHP/NZDRP 1. Project design and implementation is consistent with DFAT/MFAT policies PNG earthquake response and priorities
Page | 4
appropriate and relevant? 2. Activities are consistent with agency and overall response needs (CHS 1, CHS 6)? assessments 3. Activities/ outputs are deemed appropriate by the majority of target community members interviewed 4. Technical solutions (e.g. WASH or shelter infrastructure, Food Security/Livelihoods approach) are optimal according to the context 5. Project activities are coordinated and complimentary to those of other humanitarian actors at local, provincial and national levels 6. Projects and strategies adapt to changing needs over time (initial relief phase to early recovery) 7. Considerable efforts have been made to coordinate with private sector actors, where appropriate
2. Was the AHP/ NZDRP 8. Outputs defined in project documents delivered to an acceptable quality, or PNG earthquake response adequate explanation provided as to why changes were made effective? (CHS 2)? 9. Outputs contributed to achievement of intended outcomes 10. Community/family level resilience for future disasters improved as a result of the project 11. Projects meet accepted principles and standards of humanitarian assistance (CHS, SPHERE etc)
3. How inclusive was the 12. Data disaggregated by gender, age and disability maintained and used by the AHP/ NZDRP PNG project earthquake response? 13. Agency and donor policies on gender, inclusive development and child protection are followed by the project 14. Active strategies and partnerships developed for inclusive programming 15. Input from disadvantaged groups, particularly women, in project decision making and leadership 16. Women and men benefited equally 17. People living with a disability also benefited
4. How efficient was the 18. Project implemented according to budget and timeline as far as possible AHP/ NZDRP PNG 19. Where unforeseen circumstances have arisen, revisions to plans have been earthquake response (CHS managed and communicated to key stakeholders 2, CHS 9)? 20. Budgeted and actual costs are commensurate with outputs and outcomes of project
5. Did the AHP/ NZDRP 21. Humanitarian assistance delivered with and through accountable local PNG earthquake response organisations reinforce local 22. Local organisations are strengthened as a result of the partnership capacity/leadership (CHS3, 23. Activities coordinated with other actors through engagement with PNG CHS 4, CHS 6)? authorities, other agencies and coordinating bodies (e.g. Highlands Humanitarian Hub, Clusters, DMT) 24. Local input into project design planning and implementation
6. How transparent and 25. Implementing agency and partners have strategies to ensure content and accountable was the AHP/ strategy of activities is clear to affected populations NZDRP PNG earthquake 26. Mechanisms for questions and complaints relating to the project are in place response (CHS 4, CHS 5)? and familiar to affected population
Page | 5
27. Changes were made to the project as a result of feedback from the communities
7. How adequate were the 28. Comprehensive M&E data enables judgements in relation to the project NGOs’ M&E practices outputs and outcomes during the response to 29. M&E processes and data support learning processes within agency and AHP inform their management, and to enable them to assess the effectiveness and inclusion of their response?
2.2 SenseMaker This approach was supplemented by the use of the SenseMaker methodology, which has been developed by Cognitive Edge. This is an innovative approach to collecting and analysing data. It combines storytelling with the ability to provide statistical data which is generated by large numbers of interviews conducted in the field.11 The Australian Humanitarian Partnership partnered with Geoscience Australia to trial SenseMaker with additional funding being provided for the trial by DFAT Post in PNG. Geoscience Australia’s interest in the evaluation was to identify the information needs of disaster affected populations in PNG in order to support better communication and early warning with these communities.
The SenseMaker questions were developed with input from the evaluation team, the selected Papua New Guinean enumerators, and with support from representatives from SenseMaker.
It is important to note that SenseMaker was used to enhance the insights provided by other evaluation methods, not to replace them.
CARE and CAN DO each provided two local enumerators familiar with the project context. In PNG representatives of Cognitive Edge and Geoscience Australia provided three days of training in the approach. SenseMaker questions were refined and enumerators were issued with iPad minis for capturing stories and data. A half-day supervised field trial was conducted in an affected community12 to ensure that the team were ready to use SenseMaker as part of the evaluation.
In order to generate sufficient data, SenseMaker interviews were conducted before, during and after the other field work conducted for the evaluation. For CARE sites, enumerators operated largely independently, but for CAN DO sites, which are much more dispersed and where a security escort was deemed necessary, interviews were largely conducted alongside other evaluation fieldwork.
Around 130 SenseMaker interviews were completed (57 female 73 male) of which 115 relate to the CARE project and only 15 to the CAN DO project, although it is possible that more will be provided13. 67 interviewees were from host communities and 63 were displaced persons. All age ranges from 16 to over 65 were represented and the interviewees can be further disaggregated according to marital status and main occupation. The SenseMaker questionnaire is provided in AAnnex 2.
11 A brief explanation of the SenseMaker approach can be found at: http://cognitive-edge.com/sensemaker/ 12 Kundaka village, LLG Upper Mendi, Southern Highlands 13 For this reason SenseMaker is only referred to in the CARE findings, as the number of completed interviews for CAN DO is not sufficient to draw reliable conclusions.
Page | 6
2.3 Evaluation Team The Evaluation Team included an independent Team Leader and representatives from CARE and CAN DO who participated in their own field work.14 The team conducted daily debriefs together with SenseMaker enumerators to share learning and identify problems or new lines of enquiry. In Australia the AHP Support Unit M&E Manager was a key member of the team, liaising with the AHP Evaluation Steering Committee, and contributing to the development of the evaluation plan and the integration of the SenseMaker methodology. Full details of the evaluation team are provided in Annex 1.
2.4 Field Work The Team Leader requested visits to two of the four CARE sites, which are remote and only accessible by chartered flight. The choice of sites was largely determined by logistical factors, but there is no reason to believe that they were atypical. CAN DO were asked for a representative sample of each of the four implementing partners. This constituted a strong focus on field work with eleven of sixteen days in country spent visiting or travelling in affected communities. Unfortunately security issues prevented the team from visiting ADRA project sites in Hela, so additional time was spent on Caritas and United Church sites in Southern Highlands Province. Security conditions meant that the team was accompanied by guards throughout the CAN DO field work which may have affected the dynamic of field work.
At the end of field work the Team Leader conducted a debrief in Port Moresby including donor representatives presenting initial conclusions based on the field work, based on a written aide memoire.
2.5 Constraints and Limitations The analysis of CAN DO’s progress was influenced by some important limitations. As a joint evaluation roughly half of the time was spent with CARE and half with CAN DO. However CAN DO is a complex network and this project involved stakeholders across four separate churches operating relatively independently from each other and with decision makers at the local, diocesan, and national level as well as their counterparts in Australia and New Zealand. There was not sufficient time to consult all of these stakeholders or to search for all of the key documents related to each church response. The findings of the evaluation reflect the field work and relatively small selection of documents relating to the response.
The nature of CAN DO’s project, including forty villages across two large, remote and insecure provinces also presented challenges for the team in visiting a representative sample of project activities.
As a result the field work did not focus what were perceived to be CAN DO’s more successful partners. ADRA has considerable humanitarian experience: reports and interviews suggest that it is on target to complete its activities and that its program contains significant examples of good practice – for example toilets designed for people living with a disability and prioritising access for women and children – but the evaluation team was unable to visit its project areas due to security concerns. Only one day was allocated to ELCPNG projects which are also on track for completion. Five days were spent
14 CARE and CAN DO staff did not participate in each other’s field work to avoid the impact of a large evaluation team on small communities – and due to other operational commitments.
Page | 7
with Caritas and United Church projects – the two partners who appear to have struggled most with implementation.
Analysis of documents was also challenging: each church conducted separate needs assessments and developed its own implementation plans and relatively limited written data was available prior to the commencement of fieldwork. Even if all data had been available, analysing four separate sub-projects would have been challenging in the time allocated.
Due to security and logistical considerations, conventional evaluation fieldwork was done simultaneously with SenseMaker interviews in CAN DO sites, which enumerators found challenging. Conditions in CARE’s operational areas allowed the teams to operate more independently, allowing more time and space for interviews to be conducted. This may, in part account for the greater quantity and quality of data from CARE sites.
People with a disability were not included in the Sense Maker interviews: this may reflect a lack of briefing for enumerators and protective attitudes by families towards family members with a disability.
It should be noted that the timing of the evaluation was a little too late for CARE, occurring 2-3 months after the completion of activities and 15 months after the disaster when memories of the early stages of the program were fading – but too early for CAN DO, which had not completed many of its activities making comparisons very difficult.
Page | 8
3. Findings – CARE
This report assesses the performance of CARE and CAN DO separately. This is because the circumstances and issues faced by the two implementing organisations are quite different. It also allows each agency to understand and respond to specific findings more easily. As part of the methodology, a rubric was developed to assess performance against each evaluation question and this is provided as part of the analysis. The rubric assessment criteria can be found in Annex 1. Some overall observations relevant to the AHP and donors are provided after this and in the conclusion.
3.1 Context It is important to understand how and why CARE chose its project areas – and some of their key features of these areas. CARE had significant experience of humanitarian response in the Highlands particularly during the El Nino drought in 2015-16. In addition to AHP and The Directorate-General for European Civil Protection and Humanitarian Aid Operations (ECHO) funded projects in Eastern Highlands, Chimbu and Western Highlands provinces, it also cooperated with the World Food Programme in provinces including Enga and Hela. This 2015-16 El Nino response across the Highlands region highlighted for CARE the considerable challenges of operating in insecure parts, which had included at times unacceptable risks to staff, including death threats. CARE was thus well prepared for the earthquake response.
The Choice of Target Areas Following the earthquake, CARE sent a team to Hela Province and supported a disability assessment in Southern Highlands. However, it quickly identified its four target areas with help from its partner, the Strickland Bosavi Foundation, which has excellent relations with these communities. 32 of the 44 villages are situated close to the epicentre of the earthquake and are amongst the worst affected for this reason. Their remoteness meant that other agencies were unlikely to provide support after the initial relief phase. Although PNG’s administrative divisions put part of the target area within the Highlands region, ethnically and geographically they are inhabitants of the Papuan Plateau and have different cultures and livelihoods15. Another key aspect of this difference is that there are limited security issues in these areas – another significant consideration for CARE.
These locations also brought challenges for CARE. These areas can only be accessed by specially chartered small planes and schedules are liable to frequent disruption due to weather conditions. This makes the provision of inputs very difficult and rules out some interventions. For example some community members expressed a desire for corrugated iron roofing materials – but this would have been prohibitively expensive to transport. Remoteness and lack of facilities meant that CARE teams had to use a “fly in, fly out” approach with teams going to target areas on two to three week rotations.
Conditions in the Target Areas These areas also suffered from chronic under-development even before the earthquake: Strickland- Bosavi Foundation describe their inhabitants as “the forgotten people” and one informant told the evaluation: “what you see in the houses – it all came from CARE.” Huiya in particular has very low level of development with no school or clinic.16
15 See Strickland Bosavi Community Development Plan 2017-25 16 The evaluation team were told of plans for both but they did not exist in June 2019
Page | 9
These challenges were exacerbated by high numbers of earthquake Internally Displaced People (or IDPs). The pre-earthquake population of Huiya was estimated by local leaders to be between 120-150 people; it is currently more than 1,500 – mainly IDPs. Figures for Walagu also suggest that IDPs outnumber the original villagers. The IDPs come from surrounding villages in steep mountainous terrain. These villages were the worst affected by the earthquake: most houses collapsed or were severely damaged and a number of people were buried alive by landslides. These landslides also destroyed much of the best agricultural land – and by blocking rivers and streams polluted the water supply and caused flooding which destroyed most of their sago plantations which are their key crop, providing food, roofing material and reportedly a cash crop which was used to exchange for commodities such as soap and salt.
Interviews with these IDPs suggest that they were badly traumatised by the earthquakes which were the first of this severity since 1922 and they are still too scared to return to their original villages. Some go back for short periods to re-establish gardens, but most currently have no intention of returning permanently.
SenseMaker interviews asked people how severely they were impacted by the disaster and their responses underlined that the level of impact was high. Women in particular said that they were strongly impacted. (See Figure 1 below)
Figure 1: How seriously were you affected by the earthquake?
- - - mean rating
X axis = severity of impact; y axis = number of respondents
Page | 10
3.2 Was the CARE earthquake response appropriate and relevant?
Assessment: Highly appropriate The selection of project target areas is certainly appropriate. One informant noted: “CARE was in the epicentre…they took the most remote areas…most agencies took on areas near the big towns.” These communities sustained significant casualties and their shelter and livelihoods were devastated. High IDP numbers and chronic poverty also make them priority areas for the response.
Needs Assessment and Community Perspectives CARE’s activities were based on a thorough and well-documented needs assessment.17 This was not completed until late May 2018, although initial assessments, the Rapid Gender Analysis and Disability analyses were conducted within the first month. Community members distinctly remembered being consulted by this mission and considered that the project had responded to the needs they had expressed. For example, the contents of hygiene kits reflected feedback from the community.
In consultations community members consistently nominated shelter, WASH and food security as their key needs after the earthquake and these were prioritised in CARE’s response.18
CARE also made sure that it understood the specific needs of women and people living with a disability. Within the first month of the response it conducted a Rapid Gender Analysis of earthquake affected areas which was an important tool for understanding the differential impacts of the disaster on men and women.19 It also undertook two needs assessment processes aimed at understanding and responding to the needs of people living with a disability, first in Mendi, Southern Highlands and later in its project areas.
Adapting to Changing Needs CARE adjusted its activities as its understanding of the project evolved. For example, when CARE realised that there were continued food security problems in target areas it asked for a project revision to allow a second food distribution to both IDP and host populations.
Consistency with DFAT/MFAT Policies CARE’s response has been consistent with Australia’s Humanitarian Strategy20 and addresses its thematic priorities, such as: gender equality, women’s empowerment, disability inclusion and protection. For example, CARE provided training to community members on child protection, gender based violence and community leadership for men and women. It is also consistent with the guidelines stipulated by DFAT in its AHP activation and with New Zealand’s Aid Programme strategic plan; it has a clear Pacific focus and can be categorised as “targeted practical relief and recovery assistance.”21
17 Post Disaster Needs Assessment Report; Huiya and Walagu, Hela Province, Dodomona and Mougulu, Western Province 7-17 May 2018 18 Shelter kits and “build back better” training were mainly covered by a Canadian Humanitarian Assistance Fund (CHAF) contribution and so are not covered in depth in this report. 19 CARE Rapid Gender Analysis – Papua New Guinea Highlands Earthquake 22 March 2018 20 DFAT Humanitarian Strategy, May 2016 21 New Zealand Aid Programme Strategic Plan 2015-19, p18
Page | 11
Coordination At a national level, co-ordination was through the National Disaster Centre (NDC) and the Disaster Management Team (DMT) convened for the response. However, NDC and DMT were less involved once the Emergency Controller was appointed by the Prime Minister. CARE was noted as a strong contributor to the clusters – but many informants questioned the value of coordination efforts in Port Moresby, far from the affected areas.
Provincial Disaster Coordinators were under-resourced and undermined by conflict and insecurity and therefore ineffective. The absence of meaningful support from national and provincial governments is also confirmed by the SenseMaker data. The following triad plots the responses of interviewees when asked about their sources of information in relation to the earthquake response and shows that very few relied upon government sources:
Figure 2: Who are the trusted sources of information?
CARE therefore focused on the establishment of a regional coordination body, which it had found to be effective during the 2015-16 El Nino Drought and led the establishment of the Highlands Humanitarian Hub (HHH) in Mount Hagen and provided secretariat support. Mount Hagen is larger than Mendi and Tari and is the main entry point for those travelling to the Highlands. It is also the best logistical base for operations and also has an enthusiastic and competent Provincial Disaster Coordinator (PDC). This made it a logical locus of coordination even though the Western Highlands Province was not much affected by the earthquake.
Page | 12
The Hub, chaired by the PDC and actively supported by key NGO actors including CAN DO, became the main coordination body for civil society groups in the early months of the response. One of those involved in national level coordination noted that it cooperated well with national level structures, providing “four Ws” information on the activities of its members and liaising regularly.
It should be noted that at field level there were no other agencies to coordinate with after the first months of activities. After an initial burst of distributions supported by military and private sector flights there were no other humanitarian actors in the CARE areas visited by the evaluation. It therefore made sense for CARE to focus on early recovery and to provide a holistic project.
Private Sector Engagement CARE did make a concerted effort to engage with the private sector, particularly Oil Search, one of the major companies in the Highlands Region. Coordination proved problematic on both sides. The Oil Search Foundation representative noted that it found cluster meetings unhelpful and agencies slow to respond. It was hard to distinguish between significant responders and “organisations that just wanted us to take three boxes of supplies and a cameraman” to the affected areas. They also found first responders “weren’t there to listen; they were there to tell,” despite many of them having no PNG experience. CARE for its part found it difficult to engage with Oil Search – sometimes getting mixed messages from Port Moresby and the main logistics base in Moro. An Oil Search representative was positive about CARE’s role and said “If it happened again I would take the NGOs with me from the start so that we are on the same page from day one. They could have helped us avoid mistakes.”
Despite these problems the sequencing of the response worked well: the private sector used its resources and logistics to start the humanitarian response which allowed NGOs to prepare and mobilise for early recovery in project areas. The emphasis on early recovery in the AHP activation and the timing of donor funding suggests that donors understood this.
3.3 Was the CARE response effective?
Assessment: Effective CARE has generally delivered most of the outcomes and outputs presented in the AHP PIP and in its New Zealand Proposal. A table listing performance against each output target can be found in Annex 3 which includes gender and child disaggregated data for the totals below. It should be noted that the project focused on food security and nutrition to a greater degree than planned because of the problems it encountered in these areas during the course of implementation. Key points to note are:
• WASH targets including provision of water supply, sanitation and hygiene promotion were met. 11,128 people gained access to water supplies and 31,223 direct and indirect beneficiaries benefited from health and hygiene promotion. This figure includes populations outside the four main target areas where health workers from adjacent areas received training. Huiya stakeholders were very positive about the water system which uses a spring source and distributes water to six water points around the community. Demonstration latrines are of good quality and as a result 78% of people reported that they have access to a toilet in the course of post-distribution monitoring. Toilets adapted for disabled people were also in evidence. People were generally aware of health messages – such as hand washing and the avoidance of open defecation. • Provision of seeds, tools and related training were considerably exceeded. 1,052 Households received seeds and cuttings – double the initial target and 808 people received livelihoods
Page | 13
training. Community gardens have been established and help to multiply improved crop varieties provided by the National Agricultural Research Institute (NARI) and demonstrate improved agricultural practices. CARE modified the Agriculture Drought Recovery and Adaption Training (ADAPT) used successfully in its 2015-16 El Nino drought response and focus groups and interviews showed a good awareness of practices encouraged by the training – such as composting, mulching and mixing wood ash to improve soils. Huiya IDPs have been given access to land on which to cultivate crops: those in Walagu have not - but use these practises on a small scale adjacent to their homes and when they cultivate land in their original villages. • Distribution of dignity kits and family partnership training targets met; Gender based violence training did not attain an ambitious target. Initially the project planned to reach 6,000 people – but this figure was based on a much higher estimation of the population of the four target areas which proved to be unreliable. Ultimately 1,706 people were trained out of a total population of 5,809. Interviewees remembered and valued the contents of the dignity kits. Solar lamps – which allow safe movement at night, were hugely popular. Female and male participants spoke positively of the training they had received. Family Partnership training for example, taught people the advantages of child spacing, sharing workloads, mutual trust and how to manage budgets. • Shelter materials distribution exceeded. Most shelter support came from another donor but hinges and nails were provided from the MFAT budget as a way of adapting and improving the initial intervention. • Nutrition training target exceeded and extra food distribution conducted. Nutrition activities were expanded in the course of the project. Initially 20 health workers were to benefit from nutrition training but this expanded to 106. 10,665 benefitted from food distributions which were unforeseen in the initial proposal. Nutrition training was delivered very late in the program in February 2019. This was as a result of the Ministry of Health and UNICEF – who provided the Ready to Use Therapeutic Food (RUTF) insisting that they provide training if RUTF was to be supplied. Nevertheless the training was regarded as a major success by the Ministry of Health, community health workers and local partners. Treatment of malnourished children is ongoing by the community health workers trained and supplied by the project.
For a more detailed summary of CARE’s performance against targets, see Annex 3.
As a general point the impact of training seemed to be stronger in Walagu. Key informants and focus group participants were able to describe the training they had received and seemed more enthusiastic about using it. The community garden inspected in Walagu was much better maintained than that in Huiya. This may reflect better education levels and socio-economic status than in Huiya where responses were less detailed and engaged.
Challenges in Sustaining the Project’s Outcomes While the outputs and outcomes described in the two proposals have been attained, there is an important limitation: some outcomes that have been attained are probably not sustainable due to factors outside of the project’s control. First, it must be asked if it is reasonable to achieve sustainable behavioural change as the result of a twelve month project in a context which is extremely challenging. In the project areas, government services could be described as minimal: Huiya for example has no school, clinic or transport infrastructure. Most of the affected populations are IDPs and likely to remain so. Their isolation means that there are few economic opportunities to support self-recovery.
Page | 14
This threatens some of the gains made by the project. For example, many focus group members could explain when and how hand washing should be done – but they no longer have access to soap and have no income to buy any.
Some of the fundamental social changes envisaged in the training provided – for example re-casting attitudes on gender based violence or child protection - will require a longer and more extensive engagement. The aspiration that “CARE’s gender and protection work…aims to transform gender relations and norms…22” was probably too ambitious. It was also noticeable that communities remain deeply traumatised by the experience of the earthquake which probably affects their ability to learn and promote change.
The project was assiduous in setting up groups to continue the momentum of the project – WASH Committees played an active role in setting up water systems and promoting sanitation and hygiene education, and groups based on GBV and family partnerships. These have had a positive impact during the course of the project, but they have generally not survived the project.
Livelihoods recovery has been slow, again due to factors outside the control of the project. The staple crop grown in these areas is sago which was devastated by the earthquake and accompanying landslides. Sago palms take 10-15 years to mature. CARE sensibly focused on fast growing crops to address immediate food security needs – but the absence of sago is a significant problem. Additionally farmers in both project areas were unanimous in saying that yields of other crops such as sweet potato and taro were significantly lower with many blaming this on the earthquake. Other practical factors are likely to contribute to the problem. Much of the best land in the IDP’s home villages has been lost and farmers now have to adapt to new areas and conditions. Newly cleared bush reportedly takes a couple of years to produce good yields – particularly of root crops – as the root systems of the old vegetation gradually break down. The best land available in host villages is already occupied by the host population and not available for IDPs. The Huiya host population has been remarkably generous in providing agricultural land to IDPs but those in Walagu have no such access.
This situation is concerning: low yields may undermine confidence in the techniques promoted by CARE and the risk of other shocks – such as drought, means that these populations remain highly vulnerable.
Although the focus of AHP and MFAT activities was not on shelter, it is worth mentioning that tarpaulins provided by CARE and other agencies early in the response are now deteriorating – but with the loss of sago plantations, the traditional roofing material is not available in sufficient quantities to replace them.
22 CARE MFAT Proposal p15
Page | 15
These concerns around recovery should be tempered by the findings of SenseMaker interviews. Although responses varied considerably, a significant number of IDPs and host populations considered that they had substantially recovered from the effects of the earthquake. Figure 3 shows that a considerable number of women felt that they had recovered less fully than men. Figure 3: Earthquake impact – and extent of recovery
Interestingly further analysis shows that IDP women felt that their recovery needs had been more successfully met than women from host communities even though they were still displaced. It is unclear why this should be the case: perhaps women in host communities feel the burden of the IDP presence more acutely.
The comparatively late completion of the nutrition program may offer an opportunity for future engagement. One round of nutrition training was provided and the initial use of RUTF has been popular and successful. However, it is too soon to say what the impact has been and Ministry of Health Officials involved in the training felt that follow up training would be beneficial. The educational level of trainees was sometimes poor and there are question marks over how well the documentation of the use of RUTF is understood. If the documentation is good, further supplies of RUTF can be supplied by the provincial authorities. If it were possible either directly – or through a partner23 to conduct follow up training involving the Ministry of Health it might contribute to longer term nutritional improvements – and allow monitoring of the uncertain food security situation on which it depends.
Recommendation 1: Depending on available funding, CARE should consider whether it can facilitate follow-up nutritional training by the Ministry of Health either directly or through other stakeholders. This would reinforce the gains made by previous nutrition training and allow monitoring of nutrition and food security in the project areas.
Innovation CARE deserves credit for using an innovative and flexible approach. In order to pass on key training messages it used a mobile cinema, something completely unknown in the target area. This proved very popular, engaging many people who might not have been able to undertake more formal training. More work is needed to understand the effectiveness of this cinema approach as a training tool for
23 CARE’s MFAT proposal seems to contemplate this: “…CARE will coordinate with longer term nutrition initiatives and research (e.g. multi -year planned research that is being started through the Strickland-Bosavi Foundation)”. P8
Page | 16
communities24 but it is a promising initiative. When CARE learned that women were not allowed to give birth in their own houses and frequently had to do so in the bush, they provided a birthing bed to the health post in Walagu and helped to build a birthing hut in Huiya.
CARE also showed adaptability in its provision of water supply. In the early stages of the response the government airlifted twelve water tanks and dropped them in the project area – but with no means of connecting or utilising them. CARE assessed the situation and established the tanks as functioning water systems, providing pipes guttering, taps and other fittings.
One major barrier to the design and delivery of the project was the lack of reliable data. Beneficiary figures had to be revised when it became clear that 2011 census data was unreliable – and early in the project there were even disputes about which province specific villages were in and who was responsible for their services.25
3.4 How inclusive was the CARE earthquake response?
Assessment: Highly inclusive CARE deserves considerable credit for its efforts to make its response as inclusive as possible. From the early days of the response it tried to understand the needs of different groups in the community and shaped its response accordingly. Data disaggregated to identify men, women, boys, girls and people with a disability can be found in project reports.
Gender CARE conducted a Rapid Gender Analysis of the impact of the earthquake within a month. During the evaluation focus group participants clearly recalled that CARE’s needs assessment team had conducted separate meetings with women to ascertain their needs and that they were challenged by CARE as to how they would support marginalised people such as widows and the elderly in the course of the project.
CARE provided training in Gender Based Violence, Child Protection and Family Partnership. One observer noted with approval that “…they emphasised strengthening the family unit and the household rather than just talking about women.”
Women were consulted on the contents of dignity kits and through WASH committees on the location of water points. Gender disaggregated data was kept for project outputs (see Annex 3 for details) and in most distributions and training it shows gender balance in those receiving support for the project.
CARE conducted two Gender in Emergencies training workshops for its staff to improve their awareness and skills in this respect. Post-distribution monitoring was conducted on several occasions and included 48% women as well as a representative sample of female headed households, young people and the elderly.26
24 For example women seemed to have been more engaged than men and were better at describing what they had learned. 25 The initial CARE proposal puts Walagu in Southern Highlands Province but it was subsequently found to be in Hela Province, created in 2012. 26 Monitoring and Evaluation Report, Papua New Guinea Earthquake and Landslides Response, March 2019
Page | 17
Disability CARE made a particular effort to identify and support people with a disability. For example 402 people in the target areas underwent disability inclusive training and 174 people with a disability received livelihoods kits.
CARE was quick to engage with groups working on behalf of people living with a disability, particularly the PNG Assembly of Disabled Persons (PNGADP). This organisation was engaged to conduct a disability assessment in Southern Highlands and then in CARE’s target areas along with Callan Services for Disabled Persons.27 The Prosthetics Division of the Ministry of Health was also involved in the assessment of the target areas.
CARE convened workshops with disabled people’s organisations on a national and provincial basis resulting in the establishment of small provincial disabled people’s organisations in Southern Highlands and Western Highlands Provinces.
As a result a range of support was provided to people in the target areas including disabled toilets, raised garden beds, prosthetic limbs, wheelchairs, glasses and hearing aids. Mendi Callan has registered recipients and should provide follow up support.
The disabled organisations involved in these initiatives were uniformly positive about CARE’s collaboration. One noted that CARE approached him within a week of the earthquake asking for proposals on how they could work together. He regarded their collaboration as ground-breaking: “I must say we have never done this before in the Pacific; it is a success story…I am currently at an international conference with people from Africa and Asia…people haven’t heard of this kind of intervention.”
Another remarked: CARE is so helpful and respectful to me…We can see that CARE is committed to helping people with special needs.”
Partly as a result of this successful collaboration PNGADP’s existing work with AHP’s Disaster READY project has been strengthened, which offers the prospect of improvements in disability inclusion for other key NGOs in the country.
The issue of sustainability is again relevant for small provincial organisations. Whilst enthusiastic the Chairman of the Western Highlands Disabled Persons Organisation has no resources and is struggling to get any kind of engagement with the provincial government.
Recommendation 2: CARE should document its work on disability in the earthquake response as an example of best practice and as something that can be built upon in future humanitarian and development work in the Pacific.
One further marginalised group should be briefly mentioned. Whilst the villagers of Huiya were welcoming to most IDPs, a long running clan or family dispute meant that those from Damalia were not welcome. These villagers have chosen to resettle outside Huiya’s traditional lands for this reason and they claim that they have been excluded from some benefits of the project and were not told of one distribution. Although a sense of grievance remains, it is to CARE’s credit that they identified the problem and ensured that these IDPs were not excluded from any of the subsequent distributions.
27 A disability service established by the Christian Brothers, generally referred to as “Mendi Callan” in the project area
Page | 18
3.5 How efficient was the CARE earthquake response?
Assessment: Efficient CARE has successfully implemented the planned response in the time allocated. There were, of course challenges in staffing and logistics, but these were overcome. Where delays occurred these were largely as a result of trying to work with government institutions in a responsible manner. Several examples should be mentioned.
CARE’s nutritional support work was included in the MFAT proposal in May 2018 and the agency was ready to commence this work from around September. However UNICEF and the Ministry of Health insisted that RUTF would only be provided if the Ministry was involved in the provision of training for Community Health Workers. The unavailability of staff to implement the training meant that it was delayed until February 2019. The training was regarded as highly successful but its lateness made it impossible to measure the impact or conduct follow up activities. Health Ministry staff interviewed said it was their first exposure to remote rural locations and they were keen to provide ongoing support – but working in this way took longer.
Similar delays were experienced with the inclusion of the Prosthetics Division of the Ministry of Health and the provision of seeds and cuttings through NARI, although in the latter case recipients were happy that these had been provided in time for planting.
It has already been noted that it took nearly three months to complete the comprehensive needs assessment (on the back of initial assessments, the Rapid Gender Analysis and Disability analyses within the first month) and although timeliness was not a significant issue in interviews and focus groups it did appear as a significant issue in SenseMaker interviews. When asked if future relief projects should focus on meeting needs or providing timely assistance there was a strong preference for rapid relief (see Figure 4 below):
Figure 4: Is it more important that support is timely, or that it meets your needs?
Page | 19
If simple cost per beneficiary calculations are done it may appear that the CARE intervention was expensive: AHP and MFAT combined provided nearly AUD$2m for just over 10,000 direct and 33,000 indirect beneficiaries. However CARE provided a comprehensive response including WASH, food security/livelihoods, nutrition and gender and protection initiatives. It was also clear to CARE and the donors that choosing to work in inaccessible areas close to the epicentre of the earthquake, all inputs and staff would need to be flown in using small charter planes and the cost would therefore be considerable. For example CARE estimates that the small shelter component of the MFAT project required the shipment of around 3 tons of nails and hinges. The light planes most commonly used for the flight have a maximum payload of around 950kg and cost around K9000 (nearly AUD$4,000). In this context the cost per beneficiary is reasonable.
3.6 Did the CARE earthquake response reinforce local capacity/leadership?
Assessment: Successful CARE worked extensively with local partners. As has already been discussed CARE worked with a number of partners in disability services and worked extensively with the Ministry of Health and NARI.
The two key partners identified in its target areas were the Strickland-Bosavi Foundation (SBF) and the Evangelical Church of PNG (ECPNG). The two are quite closely related as SBF was founded by a family member of the original ECPNG missionaries. Both organisations are very well-known and respected in CARE’s target areas.
CARE’s relationship with SBF was essential to its acceptance in the communities and to its understanding of the culture in which it worked, and to this extent it was successful. The leader of SBF was employed as a part time consultant by CARE and some of its volunteers made important contributions to the project. The limitations of SBF should also be remembered. It is only four years
Page | 20
old and is as yet unregistered in PNG. As yet it has no real management or governance structure and no reliable funding source beyond its own members. Their small size, and limited capacity meant they were not really a full National NGO implementing partner, but more an informal network or Community Based Organisation.
As a result of the informal nature of SBF’s structure, the relationship was sometimes strained and frustrating to both sides. “I don’t think we thought it through” said one CARE manager: “we need to be much clearer about what we can offer (to such informal community partners) and what our expectations are…it was a bit frustrating for staff and confusing to communities.” For its part SBF didn’t feel involved in the planning or design of the project “We were involved, but it was after the fact – picking up the pieces.” Given the fact that secular civil society, and local secular NGOs are nascent across PNG, many community groups and informal networks that are not formally registered can be key partners to larger NGOs. Reflecting on how best to approach these informal networks will be important for future responses and ongoing work.
Recommendation 3: CARE should review its approach when partnering with informal, community based groups and attempt to formalise its relationships as it would do with a local NGO so that mutual expectations are clearer. This should be based on an assessment of the community groups’ capacities and discussion of the aims and objectives of the response.
The relationship with ECPNG did not develop to its potential. CARE did try to formalise the relationship, eventually signing an MoU at the national level but this did not filter down to the field. Pastors interviewed during the evaluation knew the project and its staff well but were unaware of any formal partnership.
Whilst most partners consulted were positive about the CARE program a number spoke of confusion because of staffing changes. These changes were seen to result in changes in priorities and in some instances, cultural insensitivity was reported. This applied to international staff – but also to PNG nationals from other parts of the country. Better cultural briefings might have helped and SBF could have been a resource in addressing this.
Recruiting staff for such disasters is always challenging. One informant mentioned that the earthquake occurred a week after Cyclone Gita had devastated Tonga which had absorbed many of the cadre of suitable internal CARE Pacific-experienced humanitarian professionals. However a number of informants felt that the recruitment and orientation of staff for future humanitarian projects in PNG could be improved.
Recommendation 4: CARE should review its recruitment deployment and induction for future PNG disasters. If recruits with PNG experience, or with experience of the project area for PNG nationals, are not available they must be thoroughly briefed on their roles and the operating context.
3.7 How transparent and accountable was the CARE response?
Assessment: Highly transparent/accountable Focus groups showed that affected communities had a good general knowledge of the project and what it was trying to achieve. CARE set up disaster committees in each of the host and IDP communities in Huiya and Walagu and these have been active in the work of the project. Most informants said that in case of a problem with the project they would go to the disaster committee – or one of its leaders to communicate with CARE.
Page | 21
CARE kept a detailed complaints and feedback register which included information from multiple sources including post distribution monitoring, information provided to field staff and beneficiary feedback forms. Details of each complaint and what was done about it were maintained throughout the response.
When problems have arisen, CARE has generally responded. For example post-distribution monitoring and other sources indicated that there was a problem with the quality of axes distributed as part of the shelter kits.28 CARE used its own funds to buy replacements.
CARE also maintains notice boards which spread health, nutrition and other messages and identified the project’s donors.
However, one issue which could be addressed in future interventions is to provide clear written information about project activities. While verbal briefings were provided, the nature of rumour in PNG can distort messaging. Even though areas are highly illiterate, written overviews on project activities posted on the existing noticeboards with copies provided to the response committees could potentially mitigate these rumours. For example, the evaluation heard several – probably spurious -- claims that CARE or its staff had made unfulfilled ‘promises’ that were outside the project scope. In Huiya they had “promised” to build more water points and to build the birthing house without mention of a community contribution. In Walagu they had “promised” to build a new water system (an extensive system of tanks fed by rain water harvesting already exists) – or latrines for everyone. While it is unlikely to avoid completely these misunderstandings, a clear written statement of project activities might minimise this, or at least provide another reference point for CARE staff and communities should misperceptions arise.
Recommendation 5: For future responses, CARE should put up in addition detailed written summaries of its planned project activities noting the role expected of beneficiary communities to further enhance transparency and to minimise potential misunderstandings.
3.8 How adequate were CARE’s M&E practices?
Assessment: Highly successful CARE’s M&E practices have been strong. The implementation plans for both AHP and MFAT provide measurable targets and indicators and these have been used in reporting. It is easy to distinguish which components are covered by which donor. Separate but consistent reporting has been provided to each donor. Where changes have been made to project activities – for example the decision to adjust the use of MFAT funding to allow a second food distribution – such changes are clearly documented. Reports contain gender disaggregated data and figure for disadvantaged groups such as people with a disability or female headed households.
CARE conducted at least three rounds of post distribution monitoring and also conducted an After Action Review. Although final documentation of this exercise was not available, CARE prepared a report on the findings of these exercises and how it had responded to them.29
28 Not funded by AHP or MFAT 29 Monitoring and Evaluation Report: Papua New Guinea Earthquake and Landslides Response, March 2019
Page | 22
SenseMaker interview, Kundaka, Southern Highlands
4. Findings – CAN DO
4.1 Challenges in Assessing the CAN DO Project The following section on CAN DO’s activities is necessarily less detailed than that dealing with the CARE project. At the time of the evaluation many CAN DO activities were incomplete. Whilst water tanks were delivered to partners and to sites in late 2018 and early 2019, at the time of the evaluation most of these had not been installed due to the lack of pipes, fittings and other materials which left communities reliant on pre-existing water sources. CAN DO was granted a no cost extension which shifted the deadline for implementation to 30 June,30 the final reporting date to 31 August 2019 and the acquittal to 31 October. The reasons for the delays experienced by the project are discussed in detail below. The challenge for the evaluation is that the timing of most activities is contingent on the installation of WASH hardware: water user groups and health promotion cannot be done in isolation and should accompany the installation of water systems and toilets. As one informant put it “there’s no use in training people in handwashing when they don’t have water.”
30 Subsequently extended until 31 August as a result of the initial findings of this evaluation.
Page | 23
It is similarly challenging to make definitive judgements on aspects such as inclusive development and gender, when the activities into which they were to be mainstreamed are not well advanced.
It was unfortunate that security considerations and timetabling meant that it was not possible to visit ADRA and only one day with the ELCPNG the two agencies which have made the most progress with implementation.
Since initial findings of the evaluation were reported to CAN DO, there has been a concerted drive to accelerate activities. The main shipment of pipes and fittings arrived in mid-June and a senior manager was dispatched from Sydney to focus on the completion of project activities.
Nevertheless it is important that CAN DO conducts a rigorous follow up to ensure that work has been completed to an acceptable standard in the forty villages supported by the project at the end of the no-cost extension. This should provide CAN DO and its donors with assurance that the proposed targets, outputs and outcomes have been delivered. This can be incorporated with the final reporting process – but it needs to provide greater detail than the progress report (October 2018).
Recommendation 6: The following actions are required for the successful completion and documentation of the CAN DO Project:
6a CAN DO’s final project report should be informed by verification of the status of project activities in the forty villages included in the project and of overall progress against the targets, outputs and outcomes in the donor proposals.
6b CAN DO and its implementing partners must urgently provide the management logistical and technical resources in the field to ensure that the project is completed successfully by the end of the no-cost extension.
6c CAN DO and its implementing partners should develop a follow up plan for its hygiene promotion activities in accordance with the recommendations of the training provider.
4.2 Context The CAN DO alliance is a group of Australian church based aid organisations31 which works through its national counterparts. In PNG CAN DO works with members of the Church Partnership Program, set up in 2004 with assistance from DFAT.
According to the 2011 census, 96% of PNG’s population identify as Christian and the seven churches which make up the Church Partnership Program32 claim 78% of the population as adherents. In addition the churches provide around 50% of PNG’s health service and 40% of its primary and secondary education.33 Churches are clearly a critical part of civil society in the Highlands Region.
CAN DO is working in forty villages across Southern Highlands and Hela Provinces. These include a variety of sites, including both remote rural communities and urban settings. The logistical challenges of working in rural communities in the Highlands should not be underestimated. The evaluation team’s initial schedule was revised when it was realised that one site selected would require a day’s
31 Caritas Australia, Australian Lutheran World Service, Act for Peace, Adventist Relief and Development Agency Australia, Anglican Board of Mission, Anglican Overseas Aid, Baptist World Aid Australia / Transform Aid International and Uniting World. 32 United Church, Seventh Day Adventist, Evangelical Lutheran Church, Catholic Church, Anglican Church, Baptist Union and the Salvation Army 33 Highlands Earthquake Joint Response Plan The collaborative efforts of the Australian PNG Church Partnership 17th April 2018, p2
Page | 24
travel just to get there. One United Church site will require helicopter transport to deliver the water tanks because usable roads do not exist.
It is also important to stress that insecurity is a serious problem in these two provinces and has disrupted the activities of most humanitarian actors. The situation in Hela after the earthquake was described by one informant as “all-out war” and the UN suspended operations there from April-July 2018. The Southern Highlands was beset by rioting and the burning of an aircraft at Mendi airport in June 2018 following a dispute over the legitimacy of the provincial authorities. Tribal and clan disputes are common in both provinces and can result in multiple deaths, the destruction of infrastructure and roadblocks.
The respect which people have for the churches is critical to working in these areas and it would be difficult to gain secure access without them.34 The courage and enthusiasm of local church pastors, staff and volunteers, who continue to work in these conditions merits great respect.
4.3 Was the CAN DO PNG earthquake response appropriate and relevant?
Assessment: Appropriate and Relevant
Choice of Sites and Sectors CAN DO’s choice of sites and activities can be affirmed as appropriate on the basis of the field work conducted by the evaluation. Focus groups and interviews consistently nominated water supply and shelter as the key needs after the earthquake with disruption to livelihoods in third place. Informants also confirmed that most people had been able to repair their houses using locally available materials although this had not yet been done to pre-earthquake standards. The focus on an integrated WASH program is therefore justified as a way of maximising the benefits of providing water supply.
One question which must be considered is whether the project was too ambitious. In its initial AHP proposal, CAN DO planned with three Church partners to work in 12 villages with a total population of 37,500. Activities were to be relatively straightforward distributions of items such as water containers, hygiene kits and shelter inputs. When the focus moved from immediate relief to early recovery, the Project Implementation Plan included four partners, 40 villages and a population of 58,000 – almost twice the number targeted by CARE. Activities too became more complex with the implementation of community water systems, and a sanitation component. The challenges of working through relatively inexperienced local churches in so many villages spread across two Highland provinces were too great.
Another informant questioned whether the project necessarily selected the areas of greatest need. The reports of damage certainly varied and some selected communities were further from the epicentre than others35, but all sites visited reported significant earthquake damage and are legitimate areas for intervention.
Technical Approaches to Implementation One interesting distinction is that each of the four Church partners had a different approach to selecting sites within communities. The United Church tended to concentrate resources in key
34 In one village the community visited by the evaluation, it was made clear that we would not have been welcome – and might have been subject to violence had we not come with the authority of the church. 35For example the legitimacy of working in Ialibu district, Southern Highlands was questioned by the chair of the Highlands Humanitarian Hub see HHH minutes, 18 March 2018
Page | 25
locations with as many as eight tanks in a single location being used for United Church centres, clinics and schools. This has clear logistical advantages but potentially runs the risk of being perceived as biased by communities which missed out.
Caritas has spread resources more widely with concentrations of 2-8 tanks per site in the locations visited. ELCPNG went to considerable lengths to avoid any appearance of self-interest by allocating only one tank per site and focusing on schools and clinics rather than church buildings. Spreading resources widely is perhaps the fairest approach – but has the drawback of not necessarily solving WASH problems. For example in one school visited only a single latrine could be provided which has been allocated to teachers, leaving children with the existing open pits for defecation36.
ADRA worked in communities which experience high levels of conflict and chose to provide water systems only to Churches, as these are less likely to be damaged in inter-communal violence. However they chose to provide systems not only for Adventist churches but for a number of denominations presumably to avoid any perception of bias.37
There is no right or wrong approach but CAN DO should certainly learn from these approaches when implementing future WASH projects. The projects vary in other ways: for example partners appear to be using different latrine designs and it will be important to learn the advantages and disadvantages of each. ADRA has done its own health promotion whereas the other partners have contracted a specialist WASH NGO to train the trainer for their communities. Whilst learning opportunities exist, it must be acknowledged that multiple approaches have made the project more difficult to implement and to monitor.
Recommendation 7: In order to strengthen its operational capacity for future disaster responses, CAN DO should:
7a Following the completion of the project CAN DO partners should conduct and document a workshop to learn from their different approaches to WASH implementation.
7b CAN DO should conduct and document a review of its procurement and logistics systems. It should commit to a clear timeline for implementing the findings of this review.
One informant questioned whether the exclusive focus on rainwater harvesting was appropriate as such systems can run dry in periods of drought – and in some areas more reliable spring sources and shallow bores might be feasible. This point is valid – but opting for one simple solution is justified in the view of the evaluation. The additional logistical demands and the need for greater technical skills for such an approach would have been challenging for the project.
Trauma and Peacebuilding Training Another component of the MFAT project was a combined trauma counselling and peacebuilding training of trainers. Focus groups conducted during the evaluation confirmed that the trauma caused to victims of the earthquake has been a serious problem. This training was therefore highly relevant. Peacebuilding was a necessary auxiliary activity in order to access earthquake affected areas.
Needs Assessment The CAN DO projects appear to have done their own needs assessments for this project which presents challenges from the point of view of accountability. A needs assessment conducted by the United
36 Pomowi school, Kewabi LLG, Yalibu-Pangia District 37 ADRA: CASER CAN DO Progressive Report November 2018 (?) p3
Page | 26
Church, PNG Council of Churches and International Organisation for Migration (IOM) in three communities in Hela is cited in the AHP and MFAT project documents38. This certainly informed the project – but it relates only to a few communities. The AHP Project Implementation Plan mentions a validation workshop conducted in Port Moresby on 12 May 2018 to reflect on field assessments and this resulted in a change of focus from the initial proposal which focused on first phase response to the PIP which concentrates on early recovery.
The individual partner assessments and how these translate into plans do not seem to be readily available to the local churches for Caritas and United Church. A more consolidated approach to needs assessment and planning would be clearer and more consistent.
Consistency with DFAT/MFAT Strategies The CAN DO project design is consistent with DFAT’s Humanitarian Strategy (2016) although it will be important to confirm that key thematic priorities such as women’s empowerment and disability inclusiveness are thoroughly addressed as project implementation proceeds. The project is also consistent with the stipulations of this particular AHP activation and meets the requirements of New Zealand’s Strategic Plan for the Aid Programme, 2015-19 in relation to humanitarian programming.
Coordination CAN DO partners participated constructively in co-ordination at a national level participating in the Disaster Management Team and in clusters – particularly the WASH cluster. Caritas and United Church were particularly active. ADRA was initially less good at reporting its activities but this improved in the course of the response. As with CARE, CAN DO partners found that there was little value in provincial level coordination, but were active participants in the Highlands Humanitarian Hub.
The fieldwork conducted by the evaluation suggests that there were few other humanitarian actors to work with; in some cases there were none. UNICEF also provided WASH for schools in some villages visited but this seems to have been complimentary to the activities of the project.
Private Sector Engagement Private sector engagement was integrated into initial proposals. The PIP mentions that: “CAN DO will be working with private organisations such as Exxon Mobil.” The MFAT proposal envisages a three phase project with the initial response funded by AHP, early recovery by MFAT and longer term recovery which “will be supported by Salvation Army using funding provided by Exxon Mobil.” A Salvation Army website suggests that the partnership did provide assistance – but it was in the vicinity of Exxon Mobil’s Hides gas plant in Hela and not necessarily connected to any of the CAN DO partners or locations.39
Exxon Mobil has also reportedly supported ELCPNG to support food security as part of the response. As has been observed in relation to the CARE project, the main focus of private sector engagement was the first weeks of the response, so it is possible that it proved difficult to forge a partnership for longer term activities.
38 Assessment Report for Pureni, Levani and Fugwa, Hela Province, 16 March 2018 39 https://others.org.au/news/2018/03/02/salvation-army-responding-to-papua-new-guinea-earthquake/
Page | 27
4.4 Was the CAN DO earthquake response effective?
Assessment: Not effective to date40 It is difficult to assess the effectiveness of the CAN DO project at this point for two reasons. First, as has already been noted, most activities were incomplete at the time of the evaluation and can only be definitively assessed at the end of the project. The second reason is that the proposals which establish the targets, outputs, and outcomes for the project are inconsistent. Activities, outputs and even outcomes are not consistently stated between and within project documents making progress difficult to monitor and report against.
It is also unclear how the AHP and MFAT proposals fit together. The AHP/NDRF Activation Progress Report for the period March – October 2018 is confusing on a number of issues:
• The AHP proposal is for 40 communities. The MFAT proposal is for nine communities. The combined report is for a combined total of 40 communities with AHP communities reduced to 31. Does this represent an increase in costs for AHP communities? Which communities relate to which donor? This is not explained. • The provision of six emergency health kits (output 3, MFAT) is not mentioned in the report. • The report indicates that around a third of the beneficiaries are already receiving water by the end of October 2018, which seems surprising given the logistical problems experienced by the project.41 It is unclear which implementing partners this relates to and overall outputs and targets are not disaggregated to allow this.
The AHP and MFAT projects have necessarily and legitimately evolved over time and it makes perfect sense to combine reporting on them. However there is no clear documentation which sets out definitive outputs and targets and how and why they have evolved. This should be a requirement for future multi donor projects and amendments should be clearly indicated.
Recommendation 8: Future CAN DO projects require better documentation. Targets and revisions to plans should be transparent and ideally broken down by implementing partner. There should be greater clarity on how donor funds are combined.
Based on the available evidence, the following observations can be made about the progress to date of the four implementing partners with respect to WASH infrastructure:
ADRA: ADRA’s progress report suggests that water systems will be installed at 73 sites. It was not possible to visit ADRA project sites but the most recent report available (November 2018) and discussions with the Project Coordinator indicate that the project is on track to achieve its WASH targets within the time allocated by the no-cost extension – assuming that logistical problems with pipes, fittings and other materials can be resolved. Community contributions in the form of volunteers and funds have helped with progress. WASH Committees have been set up.
ELCPNG: ELCPNG is on track to complete activities if the necessary pipes fittings and supplies are provided. Visits to two school sites found that water systems and latrines had been completed to a good standard. The project had resolved the problem of missing pipes and fittings by persuading the communities and schools to make a contribution to the completion of the project. WASH Committees
40 The evaluation was conducted before completion of most project activities: It is probable that a higher rating would be appropriate when activities are completed. 41 “Water management committees estimate that 10,100 males and 8,300 females, 3,300 boys and 2,700 girls and a total of 4,200 people with disabilities so far have access to water.” AHP/NDRF Activation Progress Report 31 March – 30 October 2018, p6
Page | 28
have been established and their future role in the maintenance of the facilities is contained in an MoU which is signed as part of a formal handover process.
United Church: United Church has been allocated 70 water tanks. It has completed the installation of eight tanks and distribution systems at its own headquarters, the Bernie Collins Bible College in Mendi, by finding its own funds to pay for pipes and fittings. Other systems are incomplete according to United Church officials. Tanks have been distributed but not installed because of a lack of pipes, guttering etc. No latrines have been completed so far.
Caritas: Tanks have been distributed but are generally not connected. At one site visited by the evaluation a water supply and two latrines had been installed but they were incomplete and quality was compromised by a lack of materials.
(See recommendation 6b above in relation to this issue)
The evaluation team were able to meet the hygiene promotion training team and briefly observe their activities. The project is using an experienced PNG NGO, Village Kit based in the Eastern Highlands42. They have designed a curriculum to train master trainers. It is based on Community Led Total Sanitation (CLTS) and Participatory Hygiene and Sanitation Transformation (PHAST) approaches. Hygiene Promoters were given practical training over three days and then accompanied to villages to practice their training skills with assistance from Village Kit staff over a further three days.
The training is appropriate to needs and is being carried out professionally – but it has occurred late in the project and it will be important to provide follow up as recommended by the trainers to ensure that WASH committees are set up and maintenance of WASH infrastructure is continued43. This will be an important factor in ensuring the sustainability of the WASH activities of the project.
(See recommendation 6c above in relation to this issue).
Peace Building and Trauma Counselling Training of Trainers was included as part of the MFAT project. Trauma – both as a result of intercommunal violence and the earthquake is at the heart of the cycle of conflict experienced in the Highlands. The training was organised and conducted by the United Church in Southern Highlands and Hela, although it also involved outside trainers – for example Child Fund provided gender based violence training. Refresher training was also provided to participants in March 2019, and this workshop also helped to organise outreach activities. A total of 97 people (75 men, 22 women) participated.
Feedback from participants was very positive and reporting provides positive examples of how the training has led people to renounce violence.44 The training seems to have had a positive impact – but there is a sense that more support is needed: in the words of the training report: “we were only given a boat to sail across the ocean without fuel.”45
Recommendation 9: CAN DO should consider options for supporting further trauma counselling and peacebuilding training in the Highlands Region.
42 Note that ADRA have conducted their own health promotion training 43 Hygiene Promotion Training of Trainers Report, Village Kit, Mendi, June 2019 44 See for example ADRA CASER CAN DO Progressive Report, November 2018 p18 45 Peace Building and Trauma Counselling Report, Weston Kuling, United Church, March 2019 p9
Page | 29
4.5 How inclusive was the CAN DO PNG earthquake response?
Assessment: Not inclusive to date The evaluation found some examples of good practice in relation to inclusion but the project has not articulated a systematic approach in relation to gender or disability inclusion.
The AHP proposal contains a Gender Disability and Social Inclusion Plan but commitments tend to be vague or difficult to assess. For example it notes that the project “will build on the learning from the El Nino response” and gives as an example the need to provide distributions of relief items to women rather than male heads of household – but there are no such household distributions in the AHP earthquake project. It seems that some of the partner organisations for disability inclusion have not yet been involved: PNG ADP said that it was not involved with CAN DO in the disaster response – although they are now collaborating through the Disaster Ready project. Despite the proximity of Mendi Callan the evaluation was told that they had not yet been involved in the response.
The project has maintained data which disaggregates women, children and people with a disability and there is evidence of some positive initiatives. For example, when ADRA realised that it would not have the resources to provide latrines for men and women, it decided to prioritise women as well as people with a disability. Part of ELCPNG’s MoU and written handover agreement for its WASH facilities is agreement that women will have equal access to facilities.
The trauma and peacebuilding training contains a section of Gender Based Violence which was run by a trainer from ChildFund.
Despite these positive signs change within the churches is likely to be a slow process. Only 22 of the 97 trauma/peacebuilding trainees were women: only 2 of 10 hygiene promotion trainees were women. This probably reflects the current staffing of the churches rather than a bias on behalf of the project, but more might have been done to locate appropriate women trainees from elsewhere.
Where the churches themselves are making progress with women’s advancement, the project is likely reflect this. For example the Catholic parish of Pombrel46 has an impressive Women’s Leader, working closely with the priest and catechist. Church members had received gender training from UN Women and the parish was part of a diocesan program designed to protect accused sorceresses. One could be optimistic that women would be consulted in the siting and use of water facilities. The parish had a clear idea of the number of people with disabilities and where they lived. In other sites visited women were less obviously included in local leadership and information about vulnerable groups was less detailed.
The October 2018 progress report noted that “a participatory reflection revealed the partners could do better in terms of gender and disability inclusion.” One national level stakeholder suggested that the church partners were “a little bit behind” on protection and gender. He suggested that they had not been engaged in the code of conduct for the response developed by the protection cluster or the ‘gender marker’ – a self-assessment checklist for gender inclusion. He concluded that “It is important to support them on this journey.”
There are still opportunities to improve as the project is completed and potential follow up activities – for example in the involvement of women in hygiene promotion and in water user groups and it may be possible to seek support from local disability organisations such as Mendi Callan.
46 Nembi Plateau Rural LLG, Nipa Kutubu District, Southern Highlands
Page | 30
Recommendation 10: CAN DO and its partners need to reflect on their approach to gender and inclusion and strengthen it. This should involve the identification of good practice within their network – but also engagement with agencies with specialist skills and experience.
4.6 How efficient was the CAN DO PNG earthquake response?
Assessment: Not efficient At the time of the evaluation the project was still beset by significant delays and key activities were incomplete. CAN DO and the Churches in PNG face considerable challenges in working with their local counterparts and this situation, combined with major delays in the flow of equipment and funds have caused major delays.
First, the vertical communication between the churches at a diocesan level and their national and international counterparts has been poor in the cases of Caritas and the United Church.47 In both cases local staff members were frustrated about the lack of information provided to them and the lack of funds to complete the project. A key Caritas official at the Mendi diocese said: “When I took over there were no designs, no nothing.” He felt that the project had stalled for the lack of a small amount of funding. “We could get these things done in a month if we had the money – but how can I build sanitation with nothing?” He explained his absence from the evaluation’s field activities because of the need to plead with the diocese to borrow money to run the hygiene promotion training as funds for this had not come through from headquarters.
At the parish level there is lack of clarity on the project plan; in one instance the parish priest had received tanks and was told that the community could decide how to use them. He did not know of any plans for sanitation or health promotion. There is a danger that the lack of knowledge of the plans at field level will exacerbate logistical problems: how can the project procure the right materials if the plan for their use is uncertain?
United Church officials in Mendi claimed not to have been involved in the needs assessment for the project and admitted to a poor understanding of what was planned. This means that local church officials at the field level are also in the dark. A Superintendent Minister said: Even our office in Mendi doesn’t know what the project is… the Mendi office is sidelined…I do not know anything about it…I had no idea how many tanks we would have; I only knew when they arrived.”
Discussions with CAN DO underline the political complexity of project implementation in the Highlands which affects the other churches - not just United Church. Church leaders are often posted away from their tribal area or are recruited from less powerful clans. In these cases it is possible that they will not be informed of developments affecting the project, or have very limited ability to affect decision making. Managing these cultural dynamics is a significant challenge.
Recommendation 11: Caritas and United Church must develop more effective ways of working with their local branches and building their capacity. These should include involvement in needs assessment, clarity on project plans and timely provision of funds as required. A dialogue is necessary between the different levels of church partner organisations at all stages of the project.
47 It should be emphasised that the evaluation was not able to meet the relevant staff in relation to ELCPNG and ADRA.
Page | 31
Logistics Management of logistics has been the biggest barrier for the efficient delivery of the CAN DO project and the main cause of delays. An operations base was established by CAN DO in Mount Hagen but it did not manage the main procurement processes and both of the key logistics related staff had left before the evaluation occurred. The project centralised the purchase of the key WASH equipment necessary for the completion of the project. Consolidated tenders were developed to meet the needs of all partners. These used PNG suppliers but were managed and signed off at a senior level in the Caritas office in Sydney.
The rationale for this approach was to obtain the best possible prices by making bulk purchases and also manage the risk of errors or fraud. The partner churches would have had very limited experience of procurement on this scale. However it has failed to deliver the necessary equipment and fittings on time.
The first major purchase appears to have gone well – and is cited as one of the key achievements in the October 2018 Progress Report: 329 water tanks were purchased and by early 2019 had largely been delivered to sites for installation. However the purchase and delivery of other key components such as pipes, guttering and fittings for the rainwater harvesting systems and materials for latrine construction did not arrive and had still not been delivered at the time of the evaluation field work in early June 201948.
This was enormously problematic. Tanks were left on sites open to potential loss or damage. Local people became disengaged from the project and local church representatives have received unjust criticism. One example is Keina49, a Catholic run health post in Mendi Munihu District. A Community Health Worker said: “When people heard that there would be a water tank they were excited. People cleared the ground (for the platform) with digging sticks. It was very hard work. When nothing happened, people got disappointed. They say “these people are telling lies.” We desperately need water.” A Caritas representative in Mendi added: “The people from Keina built a platform four months ago. They come to my door to ask what is going on. I take the blame for what you failed to deliver.”
Key informants gave different explanations as to what had gone wrong and each perspective should be considered. One manager noted that the project wasn’t well prepared for the procurement: “we were describing parts to suppliers rather than giving specifications.” This suggests that supply chain work for key relief items needs to be done to prepare for future disasters. Key relief items do not vary very much and prior discussions about technical and quality specifications with suppliers would be useful.
Other informants criticised the tender process. Since this had been conducted in Sydney it was hard for those in PNG to influence it or put pressure on the suppliers directly. It was also suggested that the tender documents did not include penalty clauses so that when delays occurred there was little that could be done. Was it the best approach to use one supplier? Some suggested that using several suppliers in competition would have given the project more leverage when things went wrong. One informant questioned what technical input was provided to the tender process.
It was also suggested that the supplier was negligent in providing the required materials to the required specifications. Two Church partners mentioned receiving the wrong quantities of materials and complaints about quality have also been heard. One estimated that there was a 8-10% variation
48 They reportedly arrived shortly afterwards in mid-June 49 Also known as Kema.
Page | 32
between what was ordered and what was delivered including mistakes in the quantity of materials and in their specifications.
Finally, the failure to delegate funds to the field was criticised. Several stakeholders believed that with relatively little funding they could have purchased some materials locally and kept up the momentum of the project.
It is beyond the scope and resources of this evaluation to produce definitive answers to these issues but they must be addressed so that lessons can be learned for the next humanitarian response. This is included in recommendation 7b above.
It should however be acknowledged that working through local church organisations has undoubtedly had some real advantages in terms of efficiency. Their knowledge of the local context, including local markets allows them to identify practical low cost solutions to problems. For example, when faced with the problem of how to connect eight water tanks with no pipes or fittings at the Bernie Collins Bible Centre the United Church estimated that they were able to do this at a cost of only AUD$250- 300. The churches can also call upon their unpaid networks of pastors, catechists, volunteers and parishioners who have made an immense contribution to the project.
4.7 Did the CAN DO PNG earthquake response reinforce local capacity/ leadership?
Assessment: Successful The key area where the project had the potential to develop local capacity is in strengthening the local church organisations themselves. It is important to have realistic expectations of what can be achieved in the space of a year. Local churches are not “implementing partners” in the conventional sense. They are faith-based organisations primarily focused on the spiritual needs of their parishioners. Their faith demands that they help alleviate suffering, but they are not professional cadres trained for such a role. They are made up for the most part of committed volunteers who must deal with the daily struggles of living in poor and often conflict ridden rural communities; they themselves are victims of the earthquake, struggling to recover. With some exceptions they have had little experience of responding to a large scale disaster.
The task of developing their skills as disaster responders will take years and will be based on learning from their own experience and that of other church partners. The current response has undoubtedly produced some gains in the capacity of the Churches, particularly within ADRA and ELCPNG. For Caritas and the United Church delays and miscommunication have caused some frustration and disillusionment amongst their officials at local and provincial levels: they do not feel that they “own” the project plans. For this reason it is particularly important that the project is completed as planned in a way that will strengthen the reputation and confidence of local stakeholders.
The project has made efforts to work with other stakeholders, but this has been challenging. Focus groups and key informants have been almost unanimous in criticising the national and provincial governments which have been completely ineffective in leading or coordinating the response at ground level, so collaboration has necessarily been limited.
The use of local NGO, Village Kit to provide hygiene promotion training is a good example of collaboration with a local Highlands based organisation. However delays and blockages in implementation have limited such collaborations so far, and with limited time left to complete the project this is unlikely to change significantly.
Page | 33
4.8 How transparent and accountable was the CAN DO earthquake response?
Assessment: Not transparent/accountable The degree of transparency and accountability varies between the churches. ELCPNG has a strong system for engagement of local stakeholders. Sites for WASH facilities were chosen by district committees involving local officials and representatives of the schools and clinics involved. In each case a project focal point was appointed – usually a pastor. These focal points are the first point of contact for local stakeholders but they are aware that if they cannot solve a problem or provide adequate information they can contact the project co-ordinator or the church headquarters in Lae.
ELCPNG signs an MoU with each school or clinic which outlines the responsibilities of each party – particularly the ongoing responsibility of the recipient to manage and maintain the facilities appropriately. When work is completed there is a formal handover process which includes inspection of the facilities and the signing of a certificate of completion.
For Caritas and United Church there is accountability through the church hierarchy but it appears to have broken down at times during the response. Both churches have a designated official in the diocese responsible for the project and in each case they are well known to pastors, and clinic and school staff in the field. They clearly have spent time engaging in target villages – but their role has been hampered by their inability to explain what is happening, particularly with the delivery of materials. Dissemination of information is usually through the local parish; there are no phone lines or notice boards with which to address problems. As with the CARE project, beneficiaries seem to prefer to raise questions or problems through respected local leaders or project stakeholders.
4.9 How adequate were CAN DO’s M&E practices?
Assessment: Not successful The CAN DO Project Implementation Plan (PIP) makes a clear statement that Monitoring and Evaluation (M&E) is an important aspect of the project. It suggests that 5% of the budget will go to M&E and notes that an M&E Coordinator will be appointed in PNG50. The Project Implementation Planning Matrix even suggests that Monitoring and Evaluation is the third outcome of the project although this is not included in the logistical framework. The PIP is however quite vague about what M&E initiatives will occur: these include “monitoring visits” “mentoring support to direct project participants” and “focus group discussions and key informant interviews to evaluate and assess the application of trainings and strategies.”
The MFAT proposal includes a section on proposed M&E but it is also not very helpful. The main monitoring tasks are “distribution monitoring” (the project doesn’t involve distribution at a household level) and “Assess adequacy of provisions” which is also hard to interpret in relation to the proposed activities.
The AHP/NDRF progress report mentions a “real time evaluation” of the trauma counselling but this appears to have been a relatively informal consultation with participants.
Given the lack of clarity and consistency in the PIP and the MFAT proposal the construction of an M&E framework was always likely to be challenging. Since only one combined interim report was required for both donors it is hard to see how M&E informed reporting or the management of the project. The
50 The M&E Coordinator was on maternity leave at the time of the evaluation, so it was not possible to consult her.
Page | 34
relatively late realisation that much of the projects key outputs had not yet been delivered suggests that they did not.
The most significant step to addressing this problem will be involving M&E personnel in the design of future projects to ensure that outcomes and outputs are specific, measurable and consistent as
suggested in recommendation 8.
Focus group, Keina, Southern Highlands
5. Findings – AHP and Donors
The focus of the evaluation has been on the progress made in the delivery of the project by the implementing partners. Nevertheless there has also been some interest in what the response tells us about donor collaboration,51 and about the performance of the relatively new AHP processes.
The collaboration between the donors and between the donors and the implementing partners has generally been positive, supported by good relationships and informal as well as formal communication by the in-country stakeholders. One Australian High Commission (AHC) participant said “we blended our support (with MFAT) well…it’s a good news story.” It should be noted that the combination of the funds of the two donors is clearer with the CARE project than the CAN DO project – but this is an issue for implementing partners to address.
Combining the funds from the two donors provided some advantages; clearly it allowed an increased range of complimentary and necessary activities within the same target areas – such as the addition of nutrition in CARE areas and peacebuilding and trauma counselling training in CAN DO areas – and a joint evaluation has saved resources and increased the potential for learning. However, careful coordination of proposals is necessary particularly with multi agency consortia: CAN DO’s MFAT proposal submitted states that DFAT funds will be used for response activities and MFAT’s for early recovery,52 whilst the AHP PIP makes it clear that AHP funds are also to be used for early recovery.
51 One suggestion from the AHP M&E Steering Committee in approving the TORs for the evaluation was that it should include “a process element… that focuses on the multi-donor aspect of the grants. There is a big opportunity here to judge the efficacy of multi-donor, coordinated responses in the Pacific that we’d be remiss to overlook.” 52 MFAT proposal, p7
Page | 35
However formal communications concerning the project were felt to be slow by the AHC. With only one interim report – which was submitted in the first instance to Canberra and then passed on to the AHP Support Unit for scrutiny, AHC staff felt that they were “out of the loop” on important information which they needed to engage with and report to the PNG Government on a regular basis.
One suggestion was that the AHC and implementing partners could conduct a workshop to discuss progress ahead of the preparation of the six month report, to provide an in depth update on the project and the challenges faced in its implementation. This would also encourage information sharing between AHP implementing partners. In multi-donor projects such as this the NZHC could also be invited and AHP could participate by video link to ensure a common understanding of the project and its progress. It was recognised that not every High Commission would have the capacity to engage in an exercise of this type, but it should be considered as an option at the time of AHP activations.
Recommendation 12: An in-country workshop involving the AHC and implementing partners to discuss project developments should be considered in large-scale disaster responses. It should be timed to coincide with existing AHP activities and reporting requirements to reduce the additional workload.
6. Conclusion
The implementation of these earthquake responses has been accomplished under extremely difficult conditions. The affected areas are remote and difficult to access. Outside of provincial capitals there is little evidence of government investment or development and the response had to deal not only with the effects of the earthquake but with the effects of chronic poverty and disadvantage. During the evaluation, more than a year after the disaster the high levels of trauma experienced by a population that had never experienced an earthquake on this scale were very much in evidence. Mandated coordination mechanisms did not work well at the provincial level and those in Port Moresby were too far from the realities of the disaster. Conflict has been a constant threat throughout the response.
“Everyone comprehends how difficult this operating environment is,” commented one donor representative. It is greatly to the credit of CARE and CAN DO that they were prepared to confront these challenges in fulfilment of the humanitarian imperative when others were not.
However under these circumstances it is important that implementing partners – and donors are realistic when agreeing project plans. CARE perhaps overestimated what change could be achieved in its eight month New Zealand proposal in arguing that “…community participation reflects the much longer term value for money that the interventions create, in the form of the sustainability.”53 CAN DO was too ambitious in attempting to provide assistance to 40 communities spread across two remote provinces working with local churches, some with very limited humanitarian experience.
53 CARE in Papua New Guinea activity proposal for Papua New Guinea (NZ) p10
Page | 36
These projects provide useful insights into the challenges of localisation in PNG. Some functions of the PNG Government lack funds and direction and civil society is weak. Church or secular partners will require sustained engagement if they are to provide the basis of a humanitarian response. Bringing them together to collaborate and to learn from one another will be an important part of the process. One donor representative commented: “We supported the Highland Humanitarian Hub: it was the closest thing to a localisation agenda that we saw.” The Hub involved church organisations, a range of NGOs, the local Red Cross and local government (it was chaired by the Western Highlands PDC) but also supported by international organisations including CAN DO members and resourced in part by CARE.
Opportunities for development of promising PNG NGOs should also be identified and pursued. The PNG ADP has been energised by its work with CARE in the earthquake response and is now keen to work with the AHP’s Disaster READY project on a longer term basis. Village Kit which has provided hygiene promotion training for CAN DO has an impressive cadre of staff who have learnt their skills while working with international NGOs over a long period.
One other aspect of the response which deserves further reflection is the engagement with the private sector which, as we have seen dominated the early stages of the response providing 64% of the total input and that: “Extractives contributed significantly to the earthquake response with greater speed and reach than traditional humanitarian actors could achieve.”54 Since humanitarian agencies and the extractives have not worked together it was unsurprising that there were misunderstandings on both sides and that collaboration was not optimal. However both CARE and CAN DO have demonstrated that they are willing to work with the private sector to meet humanitarian needs. It makes sense to maintain the contacts that have been established to develop relationships and an understanding of their respective capacities.
It is also useful to reflect on the trial use of SenseMaker as part of this evaluation. It has provided important insights into the perspectives of the disaster affected population, sometimes confirming and sometimes challenging the conclusions drawn from other sources of evidence. Its capacity to drill down into interview data and to rapidly analyse it according to age, gender, location and other variables – and to supplement this with individual stories has great potential for humanitarian monitoring and evaluation.
One lesson that has been learned is that it needs careful preparation and is best used in an iterative way. Questions need to be tested and refined and enumerators need to become familiar with a completely new technology and approach in order to provide the best insights which SenseMaker can provide. It would work best if introduced as a monitoring tool earlier in the project cycle and not as a one off tool for an ex-post evaluation.
Recommendation 13: The Australian Humanitarian Partnership should continue to use and develop SenseMaker as a tool for humanitarian monitoring and evaluation.
54 Extractives and Emergencies the Papua New Guinea Earthquake Response, the Humanitarian Advisory Group, December 2018,p2
Page | 37
Staff and students at Karanas Community School, Yalibu District, Southern Highlands
Annex 1: Evaluation Team and Schedule
Team Members
Position Name
Independent Team Leader Peter Chamberlain
CARE Representative Jordan Hoffman, Senior Program Officer
CAN DO Representative André Breitenstein, Program Coordinator, CAN DO Coordination Unit
AHP Support Unit Monitoring, Evaluation and Jessica Kenway Learning Manager
Geoscience Australia, Director, Martyn Hazelwood
Page | 38
Regional Development Section, Community Safety Branch
CARE SenseMaker enumerator Sampson Dialobe
CARE SenseMaker enumerator Helen Dialobe
CAN DO SenseMaker enumerator Albertine Kabaru (United Church)
CAN DO SenseMaker enumerator Julius Nohu (Caritas)
Timetable
Date Item
6-10 May: Evaluation commences – briefing, document analysis, preparation of Evaluation Plan
13-17 May Review and finalise Evaluation Plan
20-24 May - Data collection (Aus &NZ) by phone/skype - Logistics details finalised (transport meeting schedules etc.)
Monday, 27 May Team Leader and Cognitive Edge consultant arrive in Port Moresby: Briefings with DFAT and MFAT;
Tuesday 28 May Cognitive Edge Consultant: travel to Mount Hagen and prepare for workshop Team Leader: Port Moresby Interviews
Wed-Thur 29- 30 May SenseMaker workshop Mount Hagen
Friday 31 May Field testing of SenseMaker: Kundaka,(Caritas) Upper Mendi LLG, Mendi Munihu District, Southern Highlands
Saturday 1 June Field Kundaka, (Caritas)Upper Mendi LLG; Bernie Collins Bible College (United Church) Mendi Urban District
Sunday 2 June Mendi: interviews
Monday 3 June Kema (Caritas) Muniho Health centre (United Church), Lai Valley LLG Mendi Muniho District
Tuesday 4 June Enjua Primary School,(United Church) Pombrel parish (Caritas), Nembi Plateau LLG, Nipa Kutubu District CARE SenseMaker interviewers go to Huiya/Walagu
Wednesday 5 June Pomowi School (ULCPNG), Karanas School,(ULCPNG) Kewabi Rural LLG, Yalibu-Pangia District
Thur-Fri 6-7 June Huiya (CARE) Mount Sisa LLG, Komo Magarima District, Hela
Sat 8-Mon10 June Walagu (CARE) Mount Sisa LLG
Tuesday 11 June Travel to Port Moresby: validation/debrief workshop
Page | 39
Wednesday 12 June Team Leader returns to Australia
Page | 40
Annex 2: SenseMaker Questions
Page | 41
Page | 42
Page | 43
Page | 44
Page | 45
Page | 46
Page | 47
Page | 48
Page | 49
Annex 3: CARE Performance Against Targets
AHP Output target Actual
Output 1.1 10,301 people (4,945 women, 5,356 men) will 11,128 (5,341 women, 5,787 men, of which be provided with access to clean water. 7,122 are children)
Output 1.2 5,000 people (2,400 women, 2,600 men) will be 5,750 people (1,376 women, 1,416 men, 1,572 provided with access to sanitation facilities. boys and 1,386 girls)
Output 1.3 30,311 people will be reached through Total Beneficiaries: 31,233 Hygiene/Health awareness campaign, knowledge products Direct: 27 Community Health Workers (24 men, and hygiene materials (direct and indirect) 3 women) Indirect: 31,233 (14,992 females, 16,241 males)
Output 2.1 Seed/cutting distributions to support 500 HH or 1,052 HH (5,809 people: 1,390 women, 1,431 2,500 beneficiaries (approximately 1,300 male & 1,200 men, 1,588 boys and 1,400 girls) female) in Huiya, Walagau, and Dodomona to re-establish sago and food gardens.
Output 2.2 Livelihood Kit (Tools) distribution to support 500 773 HH (5,025 people, including 174 PWD and HH or 2,500 beneficiaries (approximately 1,300 male & 37 women-headed households) 1,200 female) in Huiya, Walagau, Mougulu and Dodomona to re-establish sago and food gardens.
Output 2.3 Livelihoods training to 500 beneficiaries to 808 people (486 males, 322 females) support adaptive agricultural practices to rehabilitate or strengthen food gardens and other livelihood activities.
Output 3.1: 6000 girls, boys, men and women (including 1,716 (987 women, 729 men) including those with disabilities) receive information and education 402 Individuals provided with Disability Inclusive materials on gender based violence (GBV), sexual Training, and 677 on GBV and CP. exploitation and abuse (SEA), protection and reproductive 66 Individuals (29 women, 37 males) trained on and sexual health rights Gender in Emergency (GiE) through 2 Workshops realised in POM and Mt Hagen. 1,550 girls, boys, men and women reached through videos via roving cinema.
Output 3.2: 500 Households within the target communities 587 HHs or 3,816 beneficiaries have received training on effective family partnerships
Output 3.3: 600 women and girls, including those with 868 women and girls disabilities, of reproductive age receive dignity kits
MFAT Output targets Actual
Output 1 (see AHP output 2.1 above)
Output 2: Shelter materials distributed and building best 5,461 people: 1,306 women, 1,346 men, 1,493 practices promoted amongst earthquake-affected families boys and 1,316 girls whose homes have been destroyed. 3,165 people (1,646 males, 1,519 females)
Output 3: Training provided to health workers to screen 106 people; (52 males, 64 females) and manage Severe Acute Malnutrition (SAM) and Moderate Acute Malnutrition (MAM) cases. 20 people (10 males, 10 females)
Page | 50
Output 4: Nutrition screening, treatment/prevention 211 SAM and MAM cases being treated assistance, and referrals for SAM cases with complications Additionally 10,665 people (2,354 men, 2,216 provided at health centers and aid posts. 500 people women, 3,190 boys and 2,905 girls) Benefit from supplementary food baskets following findings of nutrition survey NB: Insufficient data is currently available to provide this information for the CAN DO project.
Page | 51
Annex 4: Bibliography
ADRA: CASER CAN DO Progressive Report, November 2018
AHP Disaster Ready PNG Country Plan, December 2017
CARE Monitoring and Evaluation Report, Papua New Guinea Earthquake and Landslides Response, J Hoffmann, March 2019
CARE Post Disaster Needs Assessment Report; Huiya and Walagu, Hela Province, Dodomona and Mougulu, Western Province, Papua New Guinea,May 7th – May 17th 2018
CARE Rapid Gender Analysis, Papua New Guinea -Highlands Earthquake, D. Brun March 2018
Church Partnership Program, Highlands Earthquake Joint Response Plan: the collaborative efforts of the Australian PNG Church Partnership, 17th April 2018,
DFAT Development for All 2015–2020; Strategy for Strengthening Disability-inclusive Development in Australia’s aid program, May 2015
DFAT Gender Equality and Women’s Empowerment Strategy, February 2016
DFAT Humanitarian Strategy, 2016
DMT Highlands Earthquake Response Lessons Learnt Workshop, 22-23 November 2018,
Workshop Report
Highlands Humanitarian Hub, Terms of Reference, March 2018
Humanitarian Advisory Group, Extractives and Emergencies the Papua New Guinea Earthquake Response, December 2018
MFAT/ DFAT Humanitarian Monitoring and Evaluation Framework for thePacific, December 2018
NDC Papua New Guinea Highlands Earthquake, Situations Reports 1-5, March 2018
New Zealand Disaster Response Partnership Activity Decision Making Process Guideline, April 2013
New Zealand Aid Programme Strategic Plan 2015-19
Papua New Guinea, Disaster Management Act ,1987
Papua New Guinea: Highlands Earthquake Disaster Management Team Response Plan, 28 March 2018
Strickland Bosavi Community Development Plan 2017-25
United Church, Hela Council of Churches, IOM Assessment Report for Pureni, Levani and Fugwa, Hela Province, 16 March 2018
United Church, Peace Building and Trauma Counselling Report, Weston Kuling, March 2019
Page | 52
Annex 5: List of Key Informants
Handale Alfa, Huiya Agnes Ake, Women’s Leader Pombrel Parish, Nembi Plateau Kevin Akiko, Chairman, PNG Association of Disabled Persons. Mark Alu, Catechist Kundaka Catholic Parish, Upper Mendi Jeff Barik, Hygiene Promotion Trainer, Village Kit, Mendi Efara Babe, LLG Secretary, Mount Sisa Jason Brown, Director Australian Humanitarian Partnership Anna Bryan, Program Director, CARE PNG Bishop Buka, United Church, Highlands Region Father Mark Buka, Disaster Relief, Caritas Mendi Diocese Darian Clark, Former First Secretary, Australian High Commission Pastor Peterson Fuga, ECPNG District Chairman Onobasulu, Walagu Ingrid Glastonbury, Executive Advisor to the Board, Oil Search Foundation Mike Haewi, CHW, Officer in Charge Walagu Health Post Mark Haleba, Magistrate, Peace Officer, Huiya Pastor Jonathan Hauwo, Area Coordinator, ECPNG, Walagu Reverend Hingopol Hindit, Superintendent Minister, Nipa South Circuit, United Church John Hosea, ELCPNG Coordinator, Southern Highlands, Epa Mark Ihania, Director, Village Kit, Mendi Naomi Ihania, Assistant Director Village Kit, Mendi Reverend Vala Illa, Chaplain, United Church, Highlands Region Wilson Karoke, Nutrition Officer National Health Department Jack Kep, Headmaster, Injua Primary School, Nembi Plateau Pasipe Kipa, Village Leader Futulaia, Huiya Stefan Knollmayer, Manager, Humanitarian Response Unit, CARE Australia Michelle Kopi, Development Program Coordinator (Focal point for gender and for disasters) Willy Kunei, Project Coordinator, ADRA Joseph Lakai, General Secretary, Baptist Union, Mount Hagen Sally Lloyd, Strickland – Bosavi Foundation Justine McMahon, Country Representative, CARE International in PNG Jennifer Mati Luvahike, Nutrition Trainer Ministry of Health Amos Mai, Disaster Committee Member, Damalia, Huiya Mark Minape, Village Leader Bobole, Huiya Charles Mogo, Village Recorder, Aiya Village, Huiya Yasulume Muga, Bobole, Huiya Gerard Ng, Humanitarian Coordination Specialist, UNDP Julius Nohu, Disaster Hub Manager, CARITAS Kawe Oleiba, Village Leader, Fau, Huiya James Pesop, Community Health Worker, Kema Health Post, Mendi Munihu Johnson Pisa, Education Secretary/ Team Leader WASH, United Church Highlands Region Nicola Ross, Counsellor, Program Strategy and Gender, Australian High Commission Beni Rowa, Officer in Charge Munihu Health Centre, Mendi Munihu Peter Rumints, Chairman, Western Highlands Disabled Persons Organisation Father Peter Seo, Pombrel Parish, Nembi Plateau Mr Siroyoko, Headmaster, Pomowi primary School, Yalibu-Pangia
Page | 53
Pastor Turlubu, (ECPNG) Chairman, Church District Edoro, Huiya Dianne Unagi, Country Representative, CARITAS Paolus Waik, Catechist Pombrel Parish, Nembi Plateau Reverend Justin Wapu, Regional Secretary - Highlands, United Church Matthew Wasa, Landowner, Huiya Koni Wembu, Headmaster, Karanas Primary School, Yalibu-Pangia Agime Yaluma, Futulaia, Huiya Robin Yakumb, Provincial Disaster Coordinator, Western Highlands Province, Chairman, Highland Humanitarian Hub
Page | 54
Annex 6: Map of Affected Areas
Page | 55