Mercy Corps: Baseline Evaluation - PROSPER
Total Page：16
File Type：pdf, Size：1020Kb
Photo: Melyn McKay Mercy Corps: Baseline Evaluation - PROSPER Submitted to: Programme Director Ms. Jenny Vaughn 1 March 2016 Executive Overview Introduction This report communicates findings and recommendations derived from the baseline data collection and analysis conducted by Europe Conflict and Security Consulting (ECAS) on behalf of the European Union (EU) funded programme ‘Promoting Sustainable Peace & Resiliency’ in Kayah State (PROSPER, or Paung Si Lett in Myanmar language). It briefly discusses the project work completed in advance of, and in preparation for, the Baseline, including the findings of an Evaluability Assessment (EA) and Monitoring, Evaluation, and Learning (ME&L) plan redesign, as well as the baseline methodologies. It outlines the most salient findings, particularly those that speak to the overall context, stakeholder relations and perceptions, as well as external factors that are likely to affect the implementation of PROSPER and/or confound or complicate the measurement of desired or expected results. Finally, the report analyses this data against the ME&L framework in order to produce both: 1) recommendations for programmatic changes that are likely to produce greater, more positive impacts and, 2) recommendations for slight changes to both the ME&L framework and methods that are expected to improve the measurement of these impacts, if and when they are achieved. Programme Description Launched in March 2015, PROSPER is a three-year intervention implemented by a consortium comprised of International Rescue Committee (IRC), The Border Consortium (TBC), the Association of Volunteers in International Service (AVSI), Nyein Foundation, and Karuna Mission Social Solidarity (KMSS), and led by Mercy Corps. The programme will be implemented in all seven townships of Kayah State with the aim to deliver results across three interrelated areas known as ‘Expected Results’ (ERs): 1. Governance; 2. Community Driven Development and CSO Capacity Development; 3. Peace building. PROSPER targets four primary stakeholders: government, ethnic armed organisations (EAOs), Civil Society Organisations (CSOs), and ‘communities’, with the latter being broadly defined by the programme but largely conceptualised by consortium partners in terms of villages and village tracts. PROSPER is further divided into ten ‘Activity Areas’ (AAs), delivered by an individual consortium member or, in one instance, by two members in tandem. Otherwise, the programme elements are complementary but not necessarily overlapping, giving the intervention a broad, if not at times diffuse, remit. The Baseline Assessment ECAS believes that external assessment and evaluation should be a consultative, collaborative, and iterative process. As such, ECAS first undertook exploratory research (EA) in order to establish shared understandings of programme and learning objectives, as both implementing 2 partners and beneficiaries articulate them. The EA sought to assess the evaluability of the programme in order to understand the extent to which the programme activities and objectives, as they were initially stated, could be evaluated in a reliable and credible manner. Furthermore the EA looked at the “in principle and in practice” aspects of the programme and sought to present an overview of applicable design approaches and their relative advantages and disadvantages given the programming environment, operational requirements, and available resources. Following on from the EA findings, ECAS worked with PROSPER’s M&E and programme teams, as well as with consortium partners, to redesign the Programme Monitoring Plan (PMP) in order to improve evaluability. This helped to ensure that the ME&L approach would be sufficiently rigorous to monitor and evaluate impact, as well as appropriately aspirational and adaptive, so as to encourage continuous learning and testing of accepted Theories of Change (ToCs). Importantly, the ME&L redesign made the key differences clearer between what Mercy Corps and PROSPER are accountable for, and what they want to learn. Following on from the EA and ME&L redesign, primary data collection for the baseline assessment took place in January 2016 in three townships: Demoso, Hpruso, and Loikaw. An additional wave of data collection occurred in February 2016 in Hpawasng Township. The baseline is qualitative in nature; focus group discussions (FGDs) and Key Informant Interviews (KIIs) were held with numerous stakeholders, existing and target beneficiaries including: • Government representatives (ER1); • EAO representatives (ER1); • CSO members (ER2.1); • Communities benefiting from land rights and sustainable agricultural training (2.2); • Youth enrolled in vocational trainings (ER2.3); • Ceasefire Monitors (ER3.2 and 3.3); • Community Leaders (ER3.4); • Youth engaged as Peace Fellows (ER3.5); • Community members from 16 Village Tracts. Given the scope and scale of the programme activities, the evaluation is rooted in a theory- based contribution analysis. Theory-based approaches attempt to understand an intervention’s contribution to observed results through a mechanistic or process interpretation of causation, rather than determining causation through comparison to a counterfactual. Simultaneously, theory-based approaches seek to identify and assess any significant influencing factors (i.e. contextual factors) that may also play a role in the causal chain and thus affect the contribution claim. ECAS consultants Melyn McKay, Sharon Low, and Khin Chit Win, carried out the baseline research with the support of Karen Sherlock and Simon Richards. ECAS would also like to thank the PROSPER M&E and programme staff, in particular, Yamin Kyaw, Thuta Maung Maung, and Julian Pinzón Godoy, who were highly involved in the evaluation design and data collection. Further, the baseline would not have been possible without the dedicated work of the entire Mercy Corps team in Loikaw, committed consortium partners, and the incredible support of Mercy Corps leadership in both Yangon and Loikaw, notably, U Myat Thu and Ms. Jenny Vaughan. 3 Key Recommendations The following Programmatic Recommendations highlight minor shifts or reconfigurations of the programme approach to improve or maximize impacts. ER1 1) PROSPER is likely to see the greatest impacts under ER1 if the programme focuses on working with government representatives in positions at, and above, the township level. 2) Similarly, under ER1, ECAS recommends that PROSPER identify ways to engage with government representatives who are not working directly in development, specifically those concerned with security, governance, and politics, as these are the representatives least familiar with community-led processes. This may not align with the results of the governance mapping, but reflects the representatives with whom Mercy Corps are likely to achieve the most significant impact in improving knowledge and shifting attitudes. Practice will continue to be constrained by funding and access. 3) PROSPER should aim to encourage government representatives to communicate in more detail about development initiatives. Specifically: A) development initiatives should be communicated both to planned intervention villages and village tracts, but also to all of the villages and village tracts in the immediate surrounding areas; B) community consultations should include a clear and consistent presentation of the overall, long term strategic development vision for the entirety of Kayah state, as well as a discussion of decision making and prioritization criteria, so that communities can situate their own experience of development within the larger framework of change and progress. 4) ECAS recommends that PROSPER focus on creating and supporting opportunities for government representatives to engage with communities around issues of peace and security, specifically targeting rural areas as government representatives at or below the township level already interact with communities regularly. 5) PROSPER should consider combining IBN training elements into EAO representative training as a means of helping EAOs to better support community conversations around immediate development concerns (little d) and larger development goals (big D) as a means of more effectively representing community needs in negotiation with government. IBN training may be similarly beneficial for CSOs implementing under ER2.1, and for local implementing partners, particularly KMSS, if they have not already received commensurate training. ER2 6) CSOs are already engaging in community-led development. However, the evidence to support community-engagement and two-way dialogue around peace and security is less robust, suggesting that CSOs may benefit from support in this area specifically. 4 7) CSOs are at times perceived as offering short term, material solutions. However, there is an issue of scale that likely cannot be addressed by PROSPER given the size of the grant making available in ER2.1. As such, PROSPER should instead focus on addressing sustainability of these small projects, namely, the issue of continued technical assistance and knowledge- transfer related to maintenance of things such as water pipes and rainwater collection systems. 8) There may be an opportunity to engage EAO company members, as they are indirect EAO representatives, and they are likely to overlap with both political structures, and appear somewhat more politically neutral. 9) The ability to articulate long term vision and understanding of community impacts (including plans