Care International in Png

CARE INTERNATIONAL IN PNG

CARE PNG AHP DISASTER READY PROJECT

Monitoring, Evaluation and Learning Plan

MEL Plan: Care International in Papua New Guinea

OVERVIEW

The Monitoring & Evaluation (M&E) plan is oriented to support reflection-based learning by the project team and other stakeholders, as well as collecting robust data to monitor implementation and outcomes and meet accountability and reporting requirements. A full set of M&E tools is outlined later in this plan.

KEY EVALUATION QUESTIONS (KEQ) 1. To what extent and how has the project contributed to achieving the Disaster READY objectives? Specifically, detail the extent to which:  communities are better prepared for rapid and slow onset disasters  women, people with disability, youth and children have greater leadership in and influence on systems for disaster preparedness and response  Government, NGOs and communities are co-ordinating more effectively for disaster preparedness and response  National NGOs and churches have more influence in the country humanitarian system  AHP partners have worked together to achieve common aims 2. What unexpected changes have occurred as a result of the program? How has the operating context (including between EHP and ARoB) affected program achievements? 3. To what extent can or will the changes be sustained? 4. To what extent has there been a change in community and humanitarian actors’ attitudes towards the role of marginalised people (women, people with disabilities, youth and other marginalised people) in preparing for and responding to disasters as a result of this project? Who in the community/areas of government and humanitarian system is changing their attitudes and why? 5. Has the project influenced any other changes in the lives of women, people with disabilities and marginalised people? 6. Which aspects of the project generated the most (or least) value, given the time, money and effort required? 7. What aspects of the program need to be improved? What are we learning about building preparedness and response capability at different levels in PNG?

DEFINITION OF KEY VARIABLES

The table below provides definitions for terminologies and variables used in this M&E Plan that best suits the PNG Disaster READY context.

TABLE 1. DEFINITIONS OF TERMINOLOGIES AND VARIABLES USED IN THE M&E PLAN Terminology/Variable Definition DIRECT beneficiary The definition of ‘who directly benefits’ is going to vary per indicator and per activity. Possible definitions include:  Only those people participating in the activity; or  Those people plus their households (utilising a standard multiplier to estimate this based on average household size in that community or country); or  The total number of people in a community (e.g. in a village which is preparing a disaster plan).

Gender Gender in the PNG context, and for the purpose of the disaster ready project refers to Male or Female. Age (adult/child + Age in the PNG Disaster Ready context refers to: youth?)  Adult 18+  Child under 18  Youth is defined according to National Youth Policy as: “any person between the ages of 12 to 29, but any person can participate in youth activities because culturally age is defined by traditional roles and responsibilities”.

Urban /Rural The terms Rural/Urban are defined at the Ward Level as defined nationally in accordance with the Local Level Government Act.

Ability The term ‘Ability’ in the context of disaster ready in PNG is often categorised as:  People with disabilities  people without disabilities

People with disabilities is used with recognition that this is not a homogenous group of people, and that gender, age, type of disability and other socioeconomic factors shape individual experience and capacities.

Community The word community is defined in the disaster ready project context as: ‘village’ or ‘settlement’.

Sub-national The term sub-national government in this context refers to: Government  Province  District  LLG  Ward  Autonomous Bougainville Government  Regions (Bougainville)  Community Government

National NGOs National NGOs include local NGOs (national, regional and community-based), and national offices of international NGOs (including national offices of AHP NGOs). Partners Partners refers to whichever organisations are involved in implementing the M&E in country. This might include the lead NGO (i.e. the country office of the INGO, or a national NGO where relevant). The project aims to strengthen, not overwhelm the M&E system of the local NGOs involved Kobo Toolbox The Kobo Toolbox is a tool based on Open Data Kit used for collecting quantitative survey data. Refer to http://www.kobotoolbox.org/

M&E MATRIX OF INDICATORS

TABLE 2. MONITORING & EVALUATION MATRIX OF INDICATORS Participant Tracking & Direct Beneficiary Monitoring (all outcomes) Relevant outcome Project specific Indicators (based on Open Data Kit) How will you assess How and when Who will lead areas from country outcomes this? (e.g. method) will a baseline be this work? plan collected? ALL N/A DR-A: Number of individuals who  Signed participant Baseline is 0 Disaster directly benefit from AHP activities (by sheets for all activities READY gender, ability, age, urban/rural).  Participant tracking Project database manager & (disaggregated) staff  Community profiles (based on census data and updated based on community consultation)

Disaster READY Objective 1: Communities are better prepared for rapid and slow onset disasters CARE PNG Long-Term Outcome 1: Women and men in communities in 2 districts are better prepared for rapid and slow onset disasters at household and community level. Relevant outcome Project specific I: Indicators/Q: Evaluation Questions How will you assess this How and when Who will lead areas from country outcomes (e.g. method) will a baseline be this work? plan collected? 1.1 Communities Women and men in Q 1: How has people’s understanding of Rolling baseline Disaster  Participant and output understand likely communities understand disaster risk, and steps they can take to over the first 4 READY tracking hazards and risks and disaster risk, and steps minimize it, changed over the project? months of the Project have knowledge, skills that they can take to plan  Pre and post-training project which may Manager & I-i: Number of people who have completed tests and resources to for disasters and save training repeated as the staff manage these lives and property.

I-ii: % people who can name key hazards  Baseline and project expands and steps they can take to managing monitoring data of into new areas hazards and risk knowledge and I-iii: # and type of IEC materials developed practices/Activity or updated tracking

 Qualitative indicators of change (survey) Women and men in Q 2: What steps are people taking to As above Disaster  Quantitative Survey target districts are reduce risk at household level? Why? READY planning for disasters as  KIIs and FGDs Project DR-B: Number and percentage of Community profiles families and  Post training tests and Manager & communities which have disaster plans collected during communities, and taking discussions (follow up staff to reduce risks and respond to disaster visits) initial visit to action to reduce risk. (by new or updated plan)  KIIs communities and I-iv: # and % HH in target communities  Debriefs updated over time. with a plan for preparing for and  Community profiles and responding to disasters ongoing monitoring Community groups Q 3: Are there groups in each community As above Disaster 1.2 Community disaster (including farmer groups) who take responsibility for preparing for READY mechanisms are and local level and responding to disasters? Project prepared for and  Quantitative Survey government Manager & responding to rapid DR-C: Number and percentage of representatives better  KIIs and FGDs staff and slow onset communities which have tested their understand roles and  Community profiles and disasters response plan in the last 12 months responsibilities in monitoring DR-D: Number and percentage of disasters.  Quantitative Survey Disaster communities that have implemented READY action plans to reduce risks  KIIs and FGDs Project I-v: % of communities that have  Community Monitoring Manager & implemented action plans to reduce risks & Activity Tracking staff 1.3 Communities Strengthened Q 4: Did government representatives As above Disaster  Quantitative Survey understand and seek communication and participate in community level activities? READY support from sub- support between Who? Where?  KIIs and FGDs Project national government provincial, district and  Community Monitoring Community Government & Activity Tracking

before, during and after DR-E. Number of communities seeking Manager and disasters. financial support for disaster plans from staff government 1.4 Women, men, people Women and men, people Q 5: Have you received early warning Baseline and end Disaster  Quantitative Survey with disabilities and with disabilities and information in the last 6 months? Did you line survey READY  FGD children demand, children receive some understand what it meant? Project access, understand and form of early warning  Annual monitoring and Manager and Annual act on early warning information for slow onset reflection visits to staff DR-F: Number and % of communities information for rapid and hazards, with relevant communities who are receiving and acting on early slow onset disasters advisories on how to act warning information for rapid and slow (TBC). (TBC). onset disasters Objective 2: The rights and needs of women, people with disabilities, youth and children are being met in disaster preparedness and response at all levels CARE PNG long-term outcome 2: The rights and needs of women, people with disabilities, youth and children’s rights and needs are more fully met in disaster preparedness and response at all levels. 2.1 Increased Women and people with Q 6: What are the ways that women, Baseline and end Disaster representation and disabilities have greater people with disability, youth and children line survey READY capacity of women, voice in humanitarian are now included in disaster preparedness Project people with disabilities preparedness and and response? Manager and Mid-term review and youth and children in response at all levels. staff Q 7: How has the capacity of women, disaster committees and people with disabilities on disaster planning processes, committees increased? particularly at sub- DR-H: Number and percentage of  Plans will be assessed national levels. community disaster plans that are (using a participatory inclusive of women, youth, children and process with people with disabilities communities) using a criteria that has been DR-J: Number (and %) of disaster jointly developed by committees that have women AHP Partners represented, and the percentage of members that are women (by national or  KIIs sub-national level)  Monitoring of committee membership

I-vi: Number of community disaster in ARoB, EHP and committees that have women represented Selected Districts  Participant monitoring DR-K: Number of people with disabilities on national and sub national disaster committees.

I-vi: % of Female Participants in CARE-led  Participant List activities  Meeting Minutes I-vii: Number of disaster committees at  De-briefs national and sub-national level that have  KIIs people with disabilities represented I-viii: Number of community disaster committees that have people with

disabilities represented

 Participant tracking  Meeting Minutes  De-briefs 2.2 Humanitarian Strengthened policies Q 8: How has the project contributed to the End of October Disaster  KIIs operating practices, and procedures on integration of gender, protection and 2018 READY procedures, policies, gender, protection and inclusion in the PNG Humanitarian  Document Review Project laws and tools from inclusion in the national System?  Document Reviews Manager community to national humanitarian system. Q 9: How has CARE strengthened the level incorporate and are Protection Cluster’s focus on gender and responsive to rights and The PNG Protection inclusion? needs of women, people Cluster has a central DR-I: # disaster assessment tools with disabilities, youth focus on Gender and mandated by national committees that and children. Inclusion include questions on gender, disability and children I-ix: # humanitarian operating procedures, policies, laws and tools from community to national level that have been

created/revised to be responsive to the rights and needs of women, people with disability, youth and children.

Officials and Q 10: How are men, boys, church leaders, Baseline and end Disaster 2.3 All community  Baseline and Endline humanitarian actors at all chiefs and government staff working with line survey READY members, including men key informant levels have improved women, people with disability, youth and Project and boys, church and interviews understanding of the children to address negative driver of Manager and other community leaders Mid-term Review needs and rights of exclusion?  Quantitative survey staff and government staff  Case study evidence women and people with address the barriers that  KIIs disabilities in disasters DR- M: Examples of men and boys, prevent women, people  FGDs with disabilities, youth church and other community leaders and government staff addressing  HH Surveys and children from having  De-briefs their rights and needs barriers to inclusion on disaster met in disaster preparedness and response preparedness and response 2.4 AHP NGOs apply DR-N: Number and percentage of AHP At Baseline & Mid- Disaster more inclusive NGOs who have preparedness and term review READY approaches in their own response plans that explicitly prioritise Project preparedness and gender. Manager response planning I-x: % of CARE preparedness and  Document Reviews response projects rated “Gender (using criteria jointly developed by AHP partners)

Responsive” or above using the CARE Gender Marker I-xi: # AHP staff and partners who have received basic Gender in Emergencies training  Participant List % of AHPCWG meetings attended by DPO representative  Meeting Minutes Objective 3: Government, NGOs, the private sector and communities co-ordinate more effectively for inclusive disaster preparedness and response CARE PNG long-term Outcome 3: The EHP Government and ABG, and position holders in selected Districts, co-ordinate more effectively with NGOs and communities for inclusive disaster preparedness and response 3.1 National and sub EHP, Bougainville and Q 11: How did CARE support more Baseline/Mid-term Disaster national committees are district disaster effective and inclusive co-ordination for Review/End-line READY  KIIs functioning committees are meeting disaster preparedness and response? Survey Project at least twice a year  Meeting minutes Manager & DR-O: # & % of national and sub-  Participant Lists national disaster committees that meet staff regularly and engage actively with the community. I- xii: % Women and people with disability in committee meetings Inclusive Disaster Q 12: How do Plans and SOPs address the At Baseline Disaster  Document Reviews Management Plans and needs of women and people with READY Standard Operating disabilities?  KIIs Project Procedures in place in I-xiii: # Capacity development plans and Manager EHP and ARoB activities completed 3.2 Sub National Increased skills and DR-P: Examples of sub-national disaster At Baseline & mid- Disaster  Case Studies-Stories of Governments are better resources disaster committees which have improved term review READY Change able to respond to management in ARoB disaster preparedness practices (e.g. Project community needs during and ABG they have mapped evacuation assets or  De-briefs Manager & rapid and slow onset risk profiles for communities)  Output tracking staff disasters I-xiv: # Capacity development plans and activities completed

3.3 Improved two-way Provincial and district DR-R: Examples of improved At Baseline Disaster  Participant List & communications staff are able to co- communication between communities READY Output tracking between communities facilitate community level and government (e.g. community Project and governments for training assessments have informed  Meeting Minutes Manager & disaster preparedness government-led responses)  Case Studies-Stories of staff Change and response I-xv: # Trainings co-facilitated with  KIIs Procedures and Government staff

communication channels I-xvi: # Trainings led by government staff support effective, co- independently of CARE ordinate response

AHP Objective 4: National NGOs and faith-based organisations have more influence and capacity in the country humanitarian system AHP Objective 5: AHP NGOs work well together and with other relevant stakeholders CARE long-term outcome 4: CIPNG is an effective, inclusive and responsive humanitarian actor in PNG, working to increase our impact by building resilience to disasters 4.1 Local NGOs and DR-T: Number of CSOs and churches Baseline audit Disaster  Meeting Minutes faith-based organisations represented on national disaster READY are better represented in clusters or coordination committees  Participant List Project  KIIs with local CSOs national and sub-national Q 13: How has CARE supported with local Manager/AHP disaster coordination partners. CSOs to engage with the national or mechanisms subnational humanitarian system 4.2 National NGOs have CARE’s staff, offices and DR-V: Examples of increased influence Baseline Analysis Disaster  Document Reviews improved organizational communication systems by national INGO offices relative to their within first 6 READY capacity for disaster are disaster ready headquarters during a disaster  Case Studies-stories of months of the Project preparedness and response. change within CARE project Manager response including Q 14: Are CARE’s own offices and systems, policies, communication systems disaster ready? processes, equipment Have they been tested? and distribution Q 15: What have we learnt from operating in EHP and ARoB? I-xvii: # Emergency preparedness plan updates

I-xviii: # Simulations completed

5.3 AHP NGOs are using CARE’s long term DR-X: Examples of where AHP NGOs Baseline gender CI PNG good practices from programs and have integrated DRR into their other marker analysis will Program humanitarian programs partnerships are building work be conducted Managers  Participant List to mainstream disaster community resilience to I-xix: # of CARE staff from other projects within the first 6 preparedness and risk disaster and climate trained in Disaster Management tools  Meeting Minutes months of the reduction into other work. change using inclusive  KIIs/FGDs with CARE project. I-xx: % of CARE projects and programs approaches staff achieving a “sensitive” score or higher  Document Reviews using the resilience gender marker.

TOOLS & METHODS

TABLE 3. TOOLS & METHODS Method Purpose / description Frequency Quantitative Quantitative survey information (disaggregated by sex and Baseline and endline. Survey disability) will be collected using KOBO toolkit from ODK, and Baseline data may designed and analysed with remote support. An initial baseline, need to be updated as data management and analysis system will be set up, but the project expands additional baseline data may be collected as the project moves into new areas. into new areas. Qualitative Key Key informant interviews (KIIs) and focus group discussions Baseline and endline. Informant (FGDs) with community, government and other stakeholders. Interviews and Attention will be paid to seeking out and highlighting the voice of Focus Group women, people of disabilities and other stakeholders. Discussions Document Review of key policies and plans and operating procedures for Baseline and endline review gender and disability inclusion. Will also be used to review CARE documents using the Gender Marker and Resilience Marker Participant & A database of project participants will be maintained to monitor Updated following each output tracking the participants in the Disaster READY Program over time, field visit & output calculate the number of direct beneficiaries and track key produced outputs. Verified by signatures of participants. Post training tests and discussions (follow up visits) will be conducted following training. Sub-national Community profiles will be developed collaboratively by CARE, Updated annually and Community community members and local government officials. They will Profiles (See include basic information about target districts and table 1 of key communities, including population, # HH, disaster/hazard definitions) profile, vulnerable households and groups, key people/groups responsible in disasters (and contact details), status of community plans. Copies of these profiles will be stored by CARE and shared with Government Partners as a key information sources for disaster response. Annual Annual monitoring visits to project selected locations will be Annually monitoring & used to update project monitoring (indicators), and bring reflection together local stakeholders to reflect on progress and inform CARE’s direction for the following year. Annual reflection Interpret output and outcome data and reflect on evaluation Annually workshop & questions (“sense making” of data); Identify the key contextual quarterly update factors which are enabling or constraining the effectiveness of calls the project; Develop annual reports; Reflect on and update the theory of change and develop annual plans.

EXISTING STRENGTHS AND WEAKNESSES OF PARTNERS’ M&E SYSTEMS

CARE International in PNG has experience reporting on activities and outcomes to multiple donors, including DFAT and USAID, on long-term development and humanitarian response. CARE PNG is making continued efforts to strengthen project M&E systems, national staff and partner capacity as part of our commitment to localization, but recognizes that this takes time and continued improvement. Key initiatives will include:

 Using the KoBo Toolbox (based on Open Data Kit) for quantitative survey data collection. This will build on CARE’s experience using KoBo for the El Niño response, and minimise scope for data entry error. Using KoBo for Disaster READY supports CARE’s own preparedness for disaster response, contributing to Outcome 4.  Developing staff skills and confidence in collecting and analysing qualitative interview and focus group discussion data.

EVALUATION AUDIENCES

TABLE 4. EVALUATION AUDIENCES Stakeholder Information to be provided How will they use the information?

Local Community profiles will be developed Community profiles maintained and updated to implementing collaboratively by CARE, community support emergency preparedness and partners & members and local government response. Government officials. They will include basic Stakeholders information about target districts and communities, including population, # HH, disaster/hazard profile, vulnerable households and groups, key people/groups responsible in disasters (and contact details), status of community plans Community Community profiles will be developed Community profiles maintained and updated to members collaboratively by CARE, community support emergency preparedness and members and local government response. officials. They will include basic information about target districts and communities, including population, # HH, disaster/hazard profile, vulnerable households and groups, key people/groups responsible in disasters (and contact details), status of community plans National Any reports and evaluations produced, Supporting “vertical” communication and co- Disaster Co- for example the end of year report ordination and promoting community-based ordination (through AHPSU) approaches Mechanism Tracking the state of gender and disability within the country humanitarian system.

Country working Any reports and evaluations produced, Tracking the state of gender and disability group for example the end of year report within the country humanitarian system (through AHPSU) Using evidence of success and lessons to develop targeted knowledge and advocacy products through Shared Services.

DFAT Post and Any reports and evaluations produced, Aggregation of indicators, results and lessons Canberra for example the end of year report from end of year report (via AHPSU) (through AHPSU) Source of evidence for annual/periodic aid effectiveness reviews and quality checks.

Case studies, human interest stories Content for public diplomacy, online case studies and social media (see communication section)

ANGO Head Baseline and endline results, regular Input to country portfolio discussion (Annual – quarters verbal communications, annual CARE Aus) reflection outcomes and annual report. Comparison and aggregation of outcomes across the 5 AHP countries, taking account the differing operating contexts output – learning briefs.

Informing understanding of CO capacity to inform future humanitarian response.

Feeding into CI Global Reporting on Resilience and Emergency Preparedness.

REVIEW AND REPORTING REQUIREMENTS

Central to the MEL approach will be face-to-face Annual Reflection Workshops that bring together Buka and Goroka-based staff and key stakeholders to:  Interpret output and outcome data and reflect on evaluation questions (“sense making” of data)  Identify the key contextual factors which are enabling or constraining the effectiveness of the project  Develop annual reports  Reflect on and update the theory of change and develop annual plans.

All quantitative evaluation data will be disaggregated by sex, disability and age. Qualitative data collection will seek to include and promote the voice, stories and experience of women, people with disabilities through direct targeting and collection of most significant change stories. Male power holders will be interviewed to understand and assess change in attitudes toward women and people with disabilities as a result of project activities.

TABLE 5. KEY DATES AND REPORTING DEADLINES 29 June 2018 Detailed M&E Plan submitted to the AHP Support Unit

1-2 October Review, Learning & Planning Forum in Port Moresby 2018 (Regional learning event also held in 2019 and 2021 – potentially at a different time of year)

31 October Annual report for Year 1 Disaster READY funding submitted1 2018 Annual plan proposal submitted January 2019 DFAT annual funding released Mid Feb 2019 NGOs provide brief update on highlight results to inform DFAT’s Annual Quality Check (e.g. onwards 2 pages max) March 2019 Support Unit compiles collated country annual report

Within the first 6-12 months of the program, the SU will facilitate a small technical and possibly peer review of the NGOs M&E systems at the country-level. This might be held concurrently with the learning and planning forum scheduled to take place from 1st -2nd October 2018.

COMMUNICATIONS

CARE is committed to demonstrating and promoting the project approach and sharing success and lessons. CARE sees effective communications as a shared responsibility: seeking collaboration with CARE Australia, the AHPSU communications focal point and DFAT post. Effective communication will be accomplished in the following ways:  CARE has years of experience in identifying, developing and compiling content that promotes its programs. For this Project, field staff will be trained in gathering stories (with photographs) of most significant change. Particular attention will be paid to content that captures positive shifts in behavior and attitude related to gender, age and disability.  Senior staff will collate case studies to be passed on to management, whose responsibility will be assessing and filtering appropriate content before public exposure.  CARE PNG will communicate with CARE Australia, the AHPSU and DFAT post to generate content that can be used for communications and social media and soft diplomacy by DFAT Post and Canberra.  Any publications, media releases, signage etc. will be branded by both CARE International and, when appropriate, Australian Aid (DFAT). CARE knows and follows the applicable branding guidelines.

RESOURCES (FINANCIAL AND STAFFING ALLOCATED TO M&E)

It is important to note that the M&E arrangements for this project reflect the constraints of the small grant size to CARE PNG. M&E will be led by the AHP Program Manager and Disaster READY Project team, with support from CARE PNG senior management and limited remote support from CARE Australia. The project baseline will be carried out by the country team with remote support from an external consultant to design data collection instruments, support the analysis and reporting of data, as well as supporting the

1 NB in year one this will also include the bridging funds.

14 establishment of fit for purpose monitoring systems. The baseline approach may need to be replicated on a rolling basis as the project progresses. Internal/external evaluation will be carried out at key strategic points.

Flexibility will be needed in the event of a humanitarian response, and CARE would seek to collaborate with other AHP WG members and humanitarian stakeholders to carry out an after-action review and lessons learned process should the disaster occur within our target area.