Impact Evaluation of the Municipal Climate Change Strategies Pilot in Macedonia
Total Page:16
File Type:pdf, Size:1020Kb
Impact Evaluation of the Municipal Climate Change Strategies Pilot in Macedonia April 2017 (Revised July 2018) This publication was produced at the request of the United States Agency for International Development for the E3 Analytics and Evaluation Project. It was prepared independently by Management Systems International, A Tetra Tech Company; and Development and Training Services, a Palladium company. ABSTRACT The United States Agency for International Development (USAID) commissioned an impact evaluation (IE) of its Municipal Climate Change Strategies (MCCS) pilot in Macedonia to inform the Agency’s understanding of effective investments to improve climate change awareness, adaptation, and mitigation. The IE also tested whether and how participatory processes between local government, civil society, and citizens that focus on strengthening local resiliency to climate change can strengthen civic activism, intra-community engagement, and social cohesion. The IE used a mixed-methods approach with a quasi-experimental design and case studies. The quasi- experimental design matched comparable non-MCCS municipalities to each of eight MCCS treatment municipalities, and compared changes in outcomes of interest among households within these municipalities using propensity score matching with a difference-in-differences estimation model. To provide greater context and depth to the quantitative findings, particularly regarding intra-community engagement and social cohesion, the evaluation included in-depth case studies of four treatment municipalities. Overall, the evaluation concluded that MCCS had a significant and positive impact on treatment municipality residents vis-à-vis their (1) general awareness of climate change, (2) overall awareness of the local impacts of climate change, and (3) overall awareness of the global impacts of climate change. MCCS did not, with a few exceptions, have a significant and positive impact on (1) awareness of the specific impacts of climate change, (2) attitudes toward climate change, (3) actions to improve adaptation to climate change, (4) actions to mitigate climate change, (5) attitudes toward civic activism, (6) levels of civic activism, (7) attitudes toward intra-community engagement, (8) levels of intra-community engagement, (9) attitudes toward social cohesion, or (10) levels of social cohesion. The lack of positive impact may be due to factors including (1) weaknesses in the activity design, particularly its theory of change, (2) political polarization in pilot municipalities, which MCCS could not overcome (in part due to weaknesses in its theory of change), (3) institutional weaknesses inherent in Macedonia’s system of municipal government, and (4) issues related to activity implementation. Impact Evaluation of the Municipal Climate Change Strategies Pilot in Macedonia April 14, 2017 (Revised July 18, 2018) Contracted under AID-OAA-M-13-00017 E3 Analytics and Evaluation Project Prepared By: Dr. Gary Woller, Principal Investigator (Development and Training Services) Dr. Gwendolyn Bevis, Senior Qualitative Researcher (Management Systems International) Photo Caption: The Municipal Climate Change Strategies pilot holds the “It’s Up to Us” Climate Change Awareness Campaign in Delcevo, Macedonia on October 24, 2015. Credit: Milieukontakt Macedonia DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. CONTENTS Abstract ............................................................................................................................................i Contents ......................................................................................................................................... iii Acknowledgements ..................................................................................................................... viii Executive Summary ..................................................................................................................... xi Introduction ........................................................................................................................................................ xi MCCS Pilot Overview ...................................................................................................................................... xi Activity Context ................................................................................................................................................ xi Evaluation Purpose and Questions ................................................................................................................ xi Evaluation Design .............................................................................................................................................. xii Key Findings and Conclusions by Evaluation Question ........................................................................... xii Overall Conclusions and Recommendations ............................................................................................. xv Introduction ................................................................................................................................... 1 Activity Background...................................................................................................................... 1 Pilot Overview ..................................................................................................................................................... 1 Development Hypothesis ................................................................................................................................. 3 USAID Policy Context ...................................................................................................................................... 4 Country Political Context ................................................................................................................................ 4 Evaluation Purpose and Questions .............................................................................................. 5 Evaluation Purpose ............................................................................................................................................. 5 Evaluation Questions ......................................................................................................................................... 5 Evaluation Design .......................................................................................................................... 6 Mixed-Methods Design ..................................................................................................................................... 6 Data Collection and Analysis Methods .......................................................................................................... 9 Gender Considerations ................................................................................................................................... 16 Limitations of the Evaluation Design ............................................................................................................ 17 Evaluation Team ................................................................................................................................................ 19 Findings and Conclusions ............................................................................................................ 20 Awareness of Climate Change and Climate Change Impacts ................................................................ 20 Attitudes toward Climate Change ................................................................................................................ 27 Actions to Improve Climate Change Adaptation and Mitigation .......................................................... 29 Civic Activism .................................................................................................................................................... 34 Intra-Community Engagement ....................................................................................................................... 38 Social Cohesion ................................................................................................................................................. 43 Comparison of MCCS Endline Survey Findings to the Macedonian Democracy and Governance Survey Findings .......................................................................................................................... 45 MCCS Climate Change Informational Events ............................................................................................ 49 MCCS Climate Change Projects ................................................................................................................... 56 Gender ................................................................................................................................................................ 62 Overall Conclusions .................................................................................................................... 65 Impact Evaluation: Municipal Climate Change Strategies Pilot in Macedonia iii Weaknesses in Activity Design...................................................................................................................... 65 Political Polarization ........................................................................................................................................