Evaluation

Summary International Labour Office

Evaluation Office Shan State: Peace, Reconciliation, and Development through Community Empowerment – Independent Midterm evaluation

Quick Facts 2019), €7 million European Union (EU) effort to promote the inclusion of community voices in Countries: Myanmar Myanmar’s national peace process. The programme is Mid-Term Evaluation: June 2017 based on an overarching theory of change (ToC) that “ceasefires have made possible efforts in the Evaluation Mode: Independent empowerment of conflict-affected communities and Administrative Office: ROAP such empowerment can make a measurable contribution to peace, reconciliation and Technical Office: ILO Myanmar development at the local level.” Evaluation Manager: Pamornrat Pringsulaka PRD expects to reach over 75,000 beneficiaries across Evaluation Consultant(s): Mathias Kjaer, Angela 80 villages in Shan state and achieve three overarching Thaung Strategic Objectives (SOs):

Project Code: MMR/14/01/EEC 1. Provide opportunities for communities Donor(s) & Budget: EU, € 24,584 and local actors, including women and children, to be engaged inclusively in the Keywords: Midterm Evaluation, peace and reconciliation process. Myanmar, Peacebuilding, Reconciliation, Community Empowerment, Shan State, Ethnic Armed 2. Support all stakeholders to create a safe and protective environment that supports Organization, Community Protection, Community effective and sustainable reintegration of Infrastructure, Conflict-Affect Areas, EU. children affected by conflict. Background & Context 3. Facilitate participatory development in conflict-affected communities based on Summary of the project purpose, logic and community empowerment. structure PRD is implemented by a consortium of five The “Shan State: Peace, Reconciliation, and implementing partners—AIDS Support Group (ASG), Development through Community-Empowerment” Ethnic Peace Resources Project (EPRP), Maggin (PRD) programme is a four-year (March 2015-March Development Consultancy Group (MDCG), Save the

ILO Evaluation Summaries - Page 1

Children International (SCI), and the International (OECD/DAC) evaluation criteria: relevance, efficiency, Labour Organisation (ILO). Each partner is responsible effectiveness, and sustainability. Given the relatively for implementing activities in their own locations, recent start of programme activities and aligned with while ILO, as the Consortium Lead, has the additional the formative nature of the evaluation, the evaluation responsibility of ensuring the quality of reporting to only partially analyses impact, the fifth OECD/DAC the EU. evaluation criterion. Specifically, this is limited to unintended/unanticipated results (positive/negative Present Situation of the Project and primary/secondary). The evaluation has added an Midway through its implementation, that programme additional “coherence” criteria to more thoroughly has already achieved some notable results: investigate the continued validity of the programme’s ToC, as well as the complementarity of Consortium • Holding dialogue sessions benefitting over partner inputs. 4,000 community members; The evaluation is also particularly focused on • Strengthening the capacity of over 20 analysing issues specific to gender and age. This CBOs in advocacy, community includes not just collecting sex and age disaggregated engagement, and organizational data but also analysing the relevance and management; effectiveness of the programme’s gender strategy and • Sensitizing nearly 20,000 community activities involving youth. members on forced labour, child Finally, the evaluation analyses the conflict sensitivity protection and child rights issues; of both the programme as well as the evaluation itself. • Starting or completing work on seven Instruments, site and interviewee sampling, and data community infrastructure programs analysis and presentation were reviewed to generating more than 4,500 workdays. understand how they might be influenced by the Purpose, scope and clients of the evaluation conflict, and in turn possibly influence the conflict and programme stakeholders interviewed. This midterm evaluation was commissioned for several reasons. Firstly, the midterm evaluation is part Purpose and Objectives of the programme’s MEAL plan and follows a As expressed in the Terms of Reference (ToR), the commitment listed in the programme’s original “main purposes of the midterm independent proposal to the EU.1 Secondly, it follows general ILO evaluation are for programme improvements and to evaluation guidance requiring all projects over US$5 promote accountability to ILO key stakeholders and million and/or lasting more than 30 months to donor, and to enhance learning within the ILO and key undertake a midterm evaluation.2 Finally, less on the stakeholders.”3 accountability side and more on the learning side, the programme is interested in a candid, external In line with the OECD/DAC criteria described above, assessment on its current coherence and relevance the objectives of the evaluation are to: given a substantial deterioration of the peace and • reconciliation context of Shan. Assess the programme design and its theory of change, specifically whether it is The evaluation utilises four of the five standard still valid; Organisation for Economic Co-operation and Development/Development Assistance Committee

1 Organisation. Pg. 23. 3 International Labour Orgnanisation. Pg. 3. 2 Orgnanisation.

ILO Evaluation Summaries - Page 2

• Assess the relevance of the programme in evaluation based on a review of project documents responding to peace and reconciliation in and early-stage interviews; (2) a fieldwork phase to Myanmar; collect and analyse primary data; and (3) a data analysis and reporting phase to consolidate and report • Assess the effectiveness of the the findings, conclusions, and recommendations of programme and the effectiveness of its the evaluation. management arrangement; • Assess the programme implementation efficiency; Main Findings & Conclusions • Provide recommendations; and • Identify emerging potential good practices 1. Programming Assumptions. While some and lessons learnt of the program’s underlying assumptions have materialized, particularly those Scope around the program’s community- The evaluation covers programme activities beginning empowerment and armed actor from the commencement of the programme (March sensitization work, other important 2015) until its midway point (March 2017). Fieldwork assumptions around duty-bearer took place between March 8-24, 2017. It included 48 participation and response have not. This interviews (key informant, small groups, and focus has constrained the program’s ability to groups) with 156 stakeholders (71 or 46% female) deliver on outcome-level results. across seven locations (Taunggyi, Tachileik, Laikha, 2. Coherence. The programme has struggled Lashio, Kutkai, and Muse in Shan state and Yangon). to develop and follow a coherent The team attempted to interview a diversity of approach to implementation across its stakeholders to help triangulate intentions, consortium. This is partly caused by perceptions, and on-the-ground realities of the institutional agreements between activities reviewed. Consortium partners; partly by the Audience different thematic and geographical focus areas of partners; and partly by The primary intended audience for the evaluation is incomplete use of central programme the ILO Consortium members and the EU as the management tools. funding agency. Secondary audiences including township, state, and Union level officials and EAO 3. Relevance. PRD is strongly aligned to the representatives, project beneficiaries, ILO Regional empowerment needs expressed by and Headquarters staff, and other development communities. It is also aligned to the partners (current and future) interested in supporting priorities of the EU. However, the lack of peace, reconciliation, and development through government participation indicates that community empowerment in Shan state. PRD may not be aligned to the peacebuilding priorities of the Shan state Methodology of evaluation government, which admittedly have been The evaluation followed a non-experimental, vague and poorly defined. qualitatively-dominant design. It utilised a theory- 4. Effectiveness The lack of updated M&E based approach to collect and analyse information data at the time of the evaluation, which guided by an overarching framework. The evaluation occurred prior to end-of-year reporting, was designed to be implemented in three phases: (1) limits its ability to make a definitive an inception phase to revise the design of the

ILO Evaluation Summaries - Page 3

assessment on the program’s Recommendations effectiveness. However, based on

information available and collected during 1. PRD should scale back its ambitions on the evaluation, PRD appears unlikely to the targets and results feasible in reach all its current targets and intended Myanmar’s current peacebuilding results. context. 5. Efficiency. Issues related to reporting and PRD should update its programming assumption and subsequent release of funds have caused revise its targets and intended results to better reflect disruptions to activities and even its current operating environment. The programme threatened the continued participation of should also adopt a more unified and coherent some partners. Additionally, while the approach to implementation. This should include PAC has met on a regular basis, it has had increased joint planning, monitoring, and limited effect on improving joint planning implementation. and efficiencies in implementation across the Consortium. Responsible Priority: Timeframe: 0- entity:PRD High 6 months. 6. Sustainability. The sustainability of PRD’s Partners and EU. activities and results appears mixed.

While staff and beneficiaries hold positive 2. PRD should consider focusing more effort expectations that activities and results will on promoting the participation of MPs as continue, the evaluation found few part of the program’s effort to encourage examples of tangible steps taken to “duty-bearer” response. ensure sustainability. The exception is activities under SO3 which have the The programme should focus on encouraging and highest chance for sustainability, in part supporting MPs to meet regularly with their due to the strong design addressing constituents in project areas and provide workshops sustainability from the outset. to community members about how they can get in touch with their local MPs to express concerns. This 7. Special Aspects. PRD has taken notable will provide a new avenue to encourage government steps to capture and report sex- participation and help improve the prospects for disaggregated data, however additional government buy-in and strengthen the sustainability effort is needed to understand differences of results. in how activities are internalized by women, men, girls, and boys. PRD has had Responsible Priority: Timeframe: 0- a strong focus on ensuring a conflict entity:PRD High 6 months. sensitive approach and has succeeded in Partners. establishing mechanisms to investigate, understand, and monitor sensitivities in 3. PRD should consider working with fewer areas where it works. The programme has local partners and instead provide more already experienced several in-depth and sustained engagement, unanticipated results, both positive and including conducting fewer but more negative, the most concerning of which joint and high visibility events and relates to protection issues for staff and awareness-raising platforms. beneficiaries. PRD should consider adopting the approach implemented by SCI where it has identified five main

ILO Evaluation Summaries - Page 4

CBO partners that it then engages, supports, and capacitates over the life of the programme. Additionally, PRD should look to support larger joint events, seminars, and platforms to bring more visibility. ILO and SCI should look to utilise their “convening authority” to encourage the participation of government and MPs, as well as EAGs members to the extent that laws allow. Responsible Priority: Timeframe: 0- entity:PRD High 6 months. Partners.

ILO Evaluation Summaries - Page 5