Office IX DGCS Evaluation Unit Macedonia, Cities of Stip and Prilep: Improving the quality of life of Roma and promoting integration INSERIRE UNA FOTOGRAFIA RAPPRESENTATIVA DEL PROGETTO 2013 EVALUATION Evaluation-Macedonia, Cities of Stip and Prilep Document prepared by: Serena Rossignoli and Annarosa Mezzasalma, Scuola Superiore Sant'Anna 1 Evaluation-Macedonia, Cities of Stip and Prilep Acknowledgments The contents of this report are the product of the work of the team selected by the Scuola Superiore Sant'Anna. Overall scientific supervision and internal quality control were insured by Professor Andrea de Guttry of the Scuola Superiore Sant'Anna. Although not formally part of the evaluation team, two additional persons contributed to the work and deserve to be thanked herein: Dr. Daniel Carpita, who contributed substantially to the analysis of the data and Dr. Ilija Talev, who, with local expertise, supported the team during the data collection in Macedonia. Special thanks must also be extended to the staff of COSV, and in particular to Goce Bogoevski and Alessandro Botta, who, although not directly involved in the implementation of the evaluated project, supported the organization and implementation of the evaluation field mission with professionalism and sincerity. Finally, we would like to thank all those who kindly agreed to be interviewed during the mission, and who provided important information and key details on the project intervention which were essential in the formulation of this report. We would also like to extend our sincere thanks to the Italian Ambassador in Macedonia, for his kind availability and willingness to receive and support the evaluation team during the mission in the field. 2 Evaluation-Macedonia, Cities of Stip and Prilep Table of Contents Acknowledgments ............................................................................................................................................. 2 Abbreviations and Acronyms ............................................................................................................................ 5 List of Tables and Figures .................................................................................................................................. 6 Executive Summary ........................................................................................................................................... 7 Introduction ..................................................................................................................................................... 10 Objectives and Scope of the Evaluation ...................................................................................................... 10 Project Context ............................................................................................................................................ 10 Project Summary ......................................................................................................................................... 15 Evaluation ........................................................................................................................................................ 17 Methodology ............................................................................................................................................... 17 Evaluation Questions ................................................................................................................................... 18 Sources of Information ................................................................................................................................ 18 Limits of the Evaluation ............................................................................................................................... 19 Evaluation Team .......................................................................................................................................... 19 Analysis of Evaluation Questions ..................................................................................................................... 21 Relevance .................................................................................................................................................... 21 1. Is the project strategy relevant?.............................................................................................................. 21 1A Do the beneficiaries feel the project is useful? ................................................................................... 21 1B Is the project strategy consistent with the needs of the beneficiaries? ............................................. 21 Efficiency.......................................................................................................................................................... 30 2. Was the project efficient? ....................................................................................................................... 30 2A Were the results achieved with the expected costs? .......................................................................... 30 2B To what extent were the inputs converted into results? .................................................................... 37 2C Was the alternative utilized the most efficient? ................................................................................. 37 2D Were the results achieved within the anticipated timeframe? .......................................................... 42 Effectiveness .................................................................................................................................................... 45 3. Is the project design valid? ...................................................................................................................... 45 3A Is there a logical and consistent connection between inputs, activities, outputs, outcomes and impact? .................................................................................................................................................... 45 4. To what extent were the project objectives met? .................................................................................. 47 4A What percentage of the population has access to and uses the hygiene/sanitation facilities? ........ 47 4B Is the drainage canal consistently clean? ........................................................................................... 48 4C Did the population participate in the cleaning of the canal? ............................................................. 49 3 Evaluation-Macedonia, Cities of Stip and Prilep 4D Has the presence of the paved road improved the image of the neighbourhood in terms of health and cleanliness? ....................................................................................................................................... 50 4E Has paving the road facilitated movements between Trizla 1 and Trizla 2? ...................................... 50 4F To what extent there has been an increase in school attendance among children targeted by the project? .................................................................................................................................................... 50 4G To what extent have primary school dropout rates decreased for targeted children within the entire schooling path? ........................................................................................................................................ 53 4H Are Roma and non-Roma organizing communal cultural activities? ................................................. 55 4I Are Roma and non-Roma communities participating in the activities of the Multicultural Centres? . 55 Impact .............................................................................................................................................................. 59 5. What was the impact of the project intervention? ................................................................................. 59 5A Has the project contributed to the reduction of diseases linked to poor personal hygiene in Stip? .. 59 5B Has the project contributed to the reduction of diseases linked to poor personal hygiene in Prilep? 59 5C Has the population of Prilep contributed to environmental care in the neighbourhood, in addition to the other project activities? ..................................................................................................................... 59 5D Has there been a % increase in the education of Roma in Prilep and Stip as a direct consequence of the project? .............................................................................................................................................. 60 5E Has the project contributed to greater integration between Roma and non-Roma in the target areas? ...................................................................................................................................................... 63 Sustainability ................................................................................................................................................... 65 6. To what extent do the benefits of the project continue beyond the project’s life? ................................. 65 6A Has the project adopted a management system for the maintenance of the services established (hygiene and sanitation, cleaning the canal, pavement of the road, MCCs, parallel classes,