Office IX DGCS Evaluation Unit

Macedonia, Cities of Stip and Prilep: Improving the quality of life of Roma and promoting integration

INSERIRE UNA FOTOGRAFIA RAPPRESENTATIVA DEL PROGETTO

2013 EVALUATION Evaluation-Macedonia, Cities of Stip and Prilep

Document prepared by: Serena Rossignoli and Annarosa Mezzasalma, Scuola Superiore Sant'Anna

1

Evaluation-Macedonia, Cities of Stip and Prilep

Acknowledgments

The contents of this report are the product of the work of the team selected by the Scuola Superiore Sant'Anna. Overall scientific supervision and internal quality control were insured by Professor Andrea de Guttry of the Scuola Superiore Sant'Anna.

Although not formally part of the evaluation team, two additional persons contributed to the work and deserve to be thanked herein: Dr. Daniel Carpita, who contributed substantially to the analysis of the data and Dr. Ilija Talev, who, with local expertise, supported the team during the data collection in Macedonia.

Special thanks must also be extended to the staff of COSV, and in particular to Goce Bogoevski and Alessandro Botta, who, although not directly involved in the implementation of the evaluated project, supported the organization and implementation of the evaluation field mission with professionalism and sincerity. Finally, we would like to thank all those who kindly agreed to be interviewed during the mission, and who provided important information and key details on the project intervention which were essential in the formulation of this report.

We would also like to extend our sincere thanks to the Italian Ambassador in Macedonia, for his kind availability and willingness to receive and support the evaluation team during the mission in the field.

2

Evaluation-Macedonia, Cities of Stip and Prilep

Table of Contents

Acknowledgments ...... 2 Abbreviations and Acronyms ...... 5 List of Tables and Figures ...... 6 Executive Summary ...... 7 Introduction ...... 10 Objectives and Scope of the Evaluation ...... 10 Project Context ...... 10 Project Summary ...... 15 Evaluation ...... 17 Methodology ...... 17 Evaluation Questions ...... 18 Sources of Information ...... 18 Limits of the Evaluation ...... 19 Evaluation Team ...... 19 Analysis of Evaluation Questions ...... 21 Relevance ...... 21 1. Is the project strategy relevant?...... 21 1A Do the beneficiaries feel the project is useful? ...... 21 1B Is the project strategy consistent with the needs of the beneficiaries? ...... 21 Efficiency...... 30 2. Was the project efficient? ...... 30 2A Were the results achieved with the expected costs? ...... 30 2B To what extent were the inputs converted into results? ...... 37 2C Was the alternative utilized the most efficient? ...... 37 2D Were the results achieved within the anticipated timeframe? ...... 42 Effectiveness ...... 45 3. Is the project design valid? ...... 45 3A Is there a logical and consistent connection between inputs, activities, outputs, outcomes and impact? ...... 45 4. To what extent were the project objectives met? ...... 47 4A What percentage of the population has access to and uses the hygiene/sanitation facilities? ...... 47 4B Is the drainage canal consistently clean? ...... 48 4C Did the population participate in the cleaning of the canal? ...... 49

3

Evaluation-Macedonia, Cities of Stip and Prilep

4D Has the presence of the paved road improved the image of the neighbourhood in terms of health and cleanliness? ...... 50 4E Has paving the road facilitated movements between Trizla 1 and Trizla 2? ...... 50 4F To what extent there has been an increase in school attendance among children targeted by the project? ...... 50 4G To what extent have primary school dropout rates decreased for targeted children within the entire schooling path? ...... 53 4H Are Roma and non-Roma organizing communal cultural activities? ...... 55 4I Are Roma and non-Roma communities participating in the activities of the Multicultural Centres? . 55 Impact ...... 59 5. What was the impact of the project intervention? ...... 59 5A Has the project contributed to the reduction of diseases linked to poor personal hygiene in Stip? .. 59 5B Has the project contributed to the reduction of diseases linked to poor personal hygiene in Prilep? 59 5C Has the population of Prilep contributed to environmental care in the neighbourhood, in addition to the other project activities? ...... 59 5D Has there been a % increase in the education of Roma in Prilep and Stip as a direct consequence of the project? ...... 60 5E Has the project contributed to greater integration between Roma and non-Roma in the target areas? ...... 63 Sustainability ...... 65 6. To what extent do the benefits of the project continue beyond the project’s life? ...... 65 6A Has the project adopted a management system for the maintenance of the services established (hygiene and sanitation, cleaning the canal, pavement of the road, MCCs, parallel classes, extra- curricular activities)? ...... 65 6B Are appropriate financial and human resources available for the established services? ...... 66 Conclusions ...... 70 Recommendations ...... 74 Lessons Learned ...... 76 Annex 1 - Design Matrix...... 78 Annex 2 - Theory of Change ...... 82 Annex 3 - Country Context 1991 - 2013 ...... 85 Annex 4 - Agenda of the Evaluation Field Mission ……………………………………………………………………………………..88 Annex 5 - Photography Report……………………………………………………………………………………………………………………90 Annex 6 - Bibliography………………………………………………………………………………………………………………………………100 Annex 7 - List of Persons Interviewed ………………………………………………………………………………………………………104 Annex 8 - Terms of Reference……………….………………………………………………………………………………………………….106

4

Evaluation-Macedonia, Cities of Stip and Prilep

Abbreviations and Acronyms

AHP Aid for Handicapped and P oor MC C Multicultural Centre COSV Coordination of Orga nizations for Voluntary Service DGC D Director General for Cooperation and Development ECHO European Community Humanitarian Office MFA Minis try of Foreign Affairs of Italy OECD -DAC Organisation for Economic Co -operation and Development -Development Assistance Committee NGO Non -Governmental Organization ET Evaluation Team EU European Union UNDP United Nations Development Programme UNICEF United Nations Children’s Emergency Fund CTU Central Technical Unit WB World Bank WHO World Health Organisation

5

Evaluation-Macedonia, Cities of Stip and Prilep

List of Tables and Figures

Tab. 1 Financial Plan - Budget and Final Expenditure p. 30 Tab. 2 Total Project Costs by Co -financing Shares p. 3 1 Tab. 3 Detailed Summary of Project Funding Disbursements p. 3 1 Tab. 4 Summary of Project Funding Disbursements p. 3 1 Tab. 5 Breakdown and Analysis of the Submitted Financial Plan, Modifications and Final p. 3 3 Expenditure Tab. 6 Cost Analysis “Sanitation and Hygiene Facilities Stip” p. 3 7 Tab. 7 Average Cost per Beneficiary “Sanitation and Hygiene Facilities Stip” p. 38 Tab. 8 Cost Analysis “Water Adduction Network Stip – Canal Cleaning Prilep – Pavement p. 38 Prilep” Tab. 9 Cost Analysis “Increase in Schooling” p. 39 Tab. 10 Average Cost per Beneficiary “Parallel Classes and Insertion in Regular Classes” p. 39 Tab. 11 Cost per Child in Primary and Secondary School (1998 - 2003) p. 40 Tab. 12 Cost per Child in Primary School Dimitar Vlahov - 2013 p. 40 Tab. 13 Cost Analysis “Multicultural Centres ” p. 41 Tab. 14 Activity Workplan p. 42 Tab. 15 Use of “Sanitation and Hygien e Facilities ” and Revenues Recorded p. 48 Tab. 16 Educational Activities Data 2004 – 2007 p. 52 Tab. 17 Distribution of Clothing and School Supplies 2004 – 2007 p. 5 3 Tab. 18 Extra -curricular Activities 2004 – 2007 p. 5 3 Tab. 19 Multicultural Centre s Initiatives p. 58 Tab. 20 Deaths per 100, 000 p. 60 Tab. 21 Number of Roma Children Enrolled in Primary School in Stip and Prilep p. 6 1

Figure 1 School Attendance in Parallel Classes p. 54 Figure 2 School Attendance in Stip and Prilep p. 54 Figure 3 Enrolment of Roma Children – Stip, Prilep, Na tional p. 62 Figure 4 Increase in Roma Children in Primary School 2005 -2012 p. 6 2

6

Evaluation-Macedonia, Cities of Stip and Prilep

Executive Summary

The Ministry of Foreign Affairs of Italy commissioned the Scuola Superiore Sant’Anna to conduct the ex- post evaluation of the project “Macedonia, Cities of Stip and Prilep: Improving the quality of life of Roma and promoting integration” (Aid7478), implemented by the non-governmental organization (NGO), InterSOS, and its local counterparts, the associations of Cerenja and Aid for Handicapped and Poor. The project was carried out from 2004 to 2007.

The purpose of the evaluation is: i) to be accountable to various external stakeholders (including Parliament and the general public) on the activities conducted based on a structured analysis of the results achieved and; ii) to share experiences and lessons learned in order to inform any future funding initiatives in this sector in Macedonia. Therefore, the main objectives of the evaluation include: a. to determine the relevance of the project objectives and the level of achievement; b. to determine the level of efficiency, effectiveness, impact and sustainability of the project; c. to analyse the strategies and methods of implementation and how to provide recommendations for integration into the program of decentralization of culture, education and training in the Republic of Macedonia; d. to consider the sustainability factors and impact of the program on educational, hygiene, health and cultural conditions in the country; e. to estimate the existing management capacity of local partners, including local authorities.

The evaluation commenced with an examination of the terms of reference and was subsequently divided into three main phases: i) desk review, ii) field mission and data collection, iii) synthesis. During the desk review phase , the Evaluation Team (ET) reviewed evaluation questions proposed by the Ministry of Foreign Affairs of Italy (MFA) and examined existing project documentation. The team then proceeded to reconstruct the project’s intervention logic utilizing the Theory of Change methodology (Annex 2). Based on this initial analysis, the ET developed an additional set of questions and sub-questions to be answered during the course of the evaluation, as well as a list of appropriate indicators and means of verification. The ET also determined the methodologies best-suited to conduct the data collection in the field and subsequent analysis. The field mission and data collection phase included the involvement of two evaluation experts and lasted for a total of seven (7) days. During the mission, a total of 23 interviews were conducted and additional meetings were held with approximately 32 persons. The objective of the field mission was to gather further information to supplement that of the project’s logical framework and to fill existing information gaps. Following the field mission, the synthesis phase took place, during which a critical analysis of the collected data was conducted in order to respond to the evaluation questions, produce recommendations, formulate conclusions and identify lessons learned.

The project implemented activities concerning: i) community and environmental health and sanitation; ii) primary education; iii) cultural integration to benefit the Roma community in the cities of Stip and Prilep (in particular, within the Radanski Pat district of Stip and the neighbourhoods of Trizla 1 and Trizla 2 in Prilep – where the majority of Roma live. Prilep and Stip host the second and seventh largest Roma communities in Macedonia, respectively). The project identified three principle challenges . The first concerns the precarious environmental-hygiene conditions of the neighbourhoods where Roma communities primarily live in Stip and Prilep. In general, the

7

Evaluation-Macedonia, Cities of Stip and Prilep relevance of the project activities - the enhancement of access to water services and the construction of sanitation and hygiene facilities in Stip and the paving of road and cleaning of the drainage canal in Prilep - with respect to this problem is believed to be positive; however, the main weakness of the project concerns the fact that a detailed analysis of the incidence of diseases related to personal hygiene or environmental pollution was not conducted at any stage of the project, despite the fact that a reduction in the incidence of diseases was a specific indicator for the project objective. This was a critical factor, negatively affecting the effectiveness, impact and sustainability of the intervention. The second issue concerns the low level of primary education among Roma children. The project aimed to achieve at least primary education for all Roma children in the target areas who were not already enrolled in school. The establishment of parallel education classes (an experimental methodology), distribution of school supplies and necessary clothing and material support to cover schooling costs are among the priority actions identified in the National Strategy for Roma Inclusion , launched in Macedonia in 2004, as a strategy aligned with the Decade of Roma Inclusion Initiative , 2005-2015. The third issue addressed by the project was the significant fragmentation among Roma groups in target neighbourhoods and the lack of a common organization, as well as the poor level of integration between Roma and non-Roma groups in these areas. Project initiatives supporting the creation of Roma organizations and joint Roma-Macedonian groups proved not to be significant, as these types of organizations already existed prior to project implementation. However, activities to promote the integration of Roma and non-Roma are of particular importance, given that the Roma population faces extensive prejudice in the country. Thus, the various initiatives promoted by the project, including the construction of multicultural centres (MCCs) as community gathering places to increase inclusion and integration are considered to be relevant. The reconstruction of the project intervention logic through the use of the Theory of Change tool has shown the project to be, as was believed in the initial design phase, coherent and that it benefited from the presence of relevant linkages between the three thematic pillars (Pillar A: Improvement of community health and hygiene; Pillar B: Increase in schooling; and Pillar C: Organized management of activities and cultural exchanges). For each pillar, the ET analysed the logical steps, starting with the project inputs up until the intended impacts. In reconstructing the logic of the intervention, the ET detected the absence of appropriate targets necessary to quantify the degree to which the project achieved its intended objectives ; the project did not include specific indicators for the level success it aimed to achieve in terms of quantifiable results in the medium and long term. All project activities were carried out regularly and with positive results in terms of the beneficiaries reached . However, with respect to Pillar A: Improvement in community hygiene, the analysis on the level of effectiveness achieved by the construction of sanitation and hygiene facilities in Stip revealed that the actual percentage of the population with daily access to such services (0.2%), was much lower than initially estimated (13%). In terms of the schooling of Roma children (Pillar B), the results achieved were rather satisfactory, although the target set by the National Strategy was much more ambitious. The success rate of the project in terms of the schooling of Roma children from 2004 to 2007 was 61.25%. Less satisfactory results were believed to be recorded in Stip in terms of reducing the dropout rate. Between the first and the third grades of parallel classes in Stip, there was a dropout rate of 41%, in Prilep, the rate was 6%, compared to an overall average dropout rate of 10% in Stip and 16% in Prilep. Pillar C encountered difficulties with respect to the creation of both Roma and non-Roma associations which were supposed to manage multicultural centres (MCCs) established (from scratch) by the project. However, the high number of both Roma and non-Roma beneficiaries reached through the MCCs activities in Prilep and Stip suggests the initiatives promoting integration and community involvement of Roma and non-Roma were effective. It should be noted that the community involvement of non-Roma in Stip was achieved only through high

8

Evaluation-Macedonia, Cities of Stip and Prilep visibility events organized outside the MCC, which during the years the project was, in fact, a gathering place for the Roma community alone. In terms of the impact, the ET has recorded negative results with regard to the Pillars A and C . The team was unable to determine whether the project contributed to the reduction of diseases associated with poor personal hygiene in Stip or to the reduction those linked to environmental pollution in Prilep, given that data on these topics were not collected at any phase of the project. A survey on current diseases, would not be relevant or significant in the absence of baseline information. In particular, in Stip, it should be noted that the closing of the sanitation and hygiene facilities at end of the project resulted in the action having no impact with respect to the creation of community sanitation services. The ET observed a positive impact in Stip with relation to the construction of the pumping station and the water adduction/distribution network, given the large number of people (500) connected to the municipal water system. In Stip, project activities did not lead to an impact with respect to greater interaction between Roma and non-Roma communities. In Prilep, on the basis of information obtained during the field mission, the ET believes there was a positive impact on the integration of Roma and non-Roma groups thanks to the continuation of the project’s integration activities by the COSV project implemented from 2007 to 2013. There is no doubt with respect to the value produced by the project through parallel classes, which provided children who would otherwise be excluded from schooling with an opportunity for an education. To this end, the ET notes that the parallel education classes have continued to be implemented today, since their initial years of introduction - 2004 in Stip and 2005 in Prilep. Regardless of whether they completed primary school, children with access to the parallel classes have learned to read and write and have acquired basic mathematic/calculation skills. The increase in Roma children enrolled in primary school in Prilep followed the national trend, which suggests this particular aspect of the project did not have a significant impact in Prilep. However, in Stip, the increase in primary school enrolment followed the national trend until 2008/2009, while a major increase was noted in 2009/2010 and 2011/2012, with a slight decline in 2010/2011. While the precise extent to which the project directly contributed to this impact cannot be determined in light of multiple variables contributing to this trend , including assistance from other international donors, it is nevertheless possible to deduce that the project has led to a positive result. A critical factor affecting the sustainability of the project is linked to the absence of an ad hoc strategy , which was not accurately defined in either the planning or in the implementation phase. This absence was even more significant due to the fact that, at the end of the project, InterSOS closed its mission in Macedonia . The lack of a clear business plan for the sustainability of sanitation facilities exemplifies this statement. The sanitation and hygiene facilities in Stip, in fact, ceased to exist in their functional capacity as of 2007. In general, the sustainability of select services enabled by the project – i.e. the cleaning of the drainage canal, parallel education classes and the operations of the MCC in Prilep, as well as the parallel education classes in Stip - is attributable to the goodwill of local partners (municipalities, schools) and independent interventions implemented following the end of the evaluated project. In terms of efficiency, the project did not prove to have any critical issues, neither relative to the budget nor to the timing of the implementation. It should be noted, however, that the budget required a series of adjustments relating to current costs of items of expenditure as a result of the time which elapsed between the design/submission of the project (2001) and the actual start of implementation (2004).

9

Evaluation-Macedonia, Cities of Stip and Prilep

Introduction

Objectives and Scope of the Evaluation

The Ministry of Foreign Affairs of Italy - Directorate General for Development Cooperation (MFA-DGDC) requested the Scuola Superiore Sant'Anna to conduct the ex-post evaluation of the project “Macedonia, cities of Stip and Prilep: Improving the quality of life of Roma and promoting integration” (AID 7478), implemented by the NGO InterSOS and its local counterparts, the associations of Cerenja and Aid for Handicapped and Poor (AHP). The project was carried out from 2004 to 2007.

The purpose of the evaluation is: i) to be accountable to various external stakeholders (including Parliament and the general public) on the activities conducted based on a structured analysis of the results achieved and; ii) to share the experiences and lessons learned in order to inform any future funding initiatives in this sector in Macedonia. Therefore, the main objectives of the evaluation include: a. to determine the relevance of the project objectives and the level of achievement; b. to determine the level of efficiency, effectiveness, impact and sustainability of the project; c. to analyse the strategies and methods of implementation and how to provide recommendations for integration into the program of decentralization of culture, education and training in the Republic of Macedonia; d. to consider the sustainability factors and impact of the program on educational, hygiene, health and cultural conditions in the country; e. to estimate the existing management capacity of local partners, including local authorities.

The results of the analysis of the elements mentioned above constitute a significant source of information to support the formulation of possible strategies and actions in the future. The evaluation provides detailed and specific recommendations with the aim of fostering a culture of “lesson learning”.

Project Context

The Council of Europe estimates that between 10 and 12 million Roma and Travellers live in Europe (2010 data). Macedonia is home to 9.59% of the total Roma and Gypsy population, which is the second largest country presence after Bulgaria (10.33%), followed by Slovakia (9.17%), Romania (8.32%), Serbia (8.18%), Hungary (7.05%), Turkey (3.83%), Albania (3.18%), Greece (2.47%) and Spain (1.52%). 1

According to official figures from the 2002 Census, 53,879 people living in Macedonia were ethnic Roma. However, the figure is likely to be an underestimate given the following factors: i) the mobility of the Roma population; ii) many Roma are not officially registered; and iii) many Roma often claim to belong to another ethnic group or mother-tongue to avoid discrimination.23 Based on official data, the Roma constitute 2.66%

1 Council of Europe, Protect the Rights of the Roma , 2010. 2 In addition, in some cities, the most important ethnic communities urge the Roma to register and declare membership to other ethnic groups. Finally in 1991, after independence, many Roma did not have the necessary requirements to become Macedonian citizens and were not be counted in the census carried out in 1994 and 2002. 3It should be noted that there are no updated data after 2002 in relation to the ethnic composition of the population of Macedonia. In fact, the census carried out in 2011 due to some irregularities was not considered valid by the relevant governmental authorities themselves. 10

Evaluation-Macedonia, Cities of Stip and Prilep of the Macedonian population and represent the third largest minority after ethnic Albanians and Turks. 4 However, in 2004, the Roma Education Fund estimated the Roma population in Macedonia to include 113,123 persons in 2002, and in 2010, the Open Society Institute estimated the Roma population to be 135,490 (approximately 6.77% of the population of the country).5 It must also be considered that Macedonia is home to around 2,240 refugees, mostly Roma (about 1,000) from Kosovo. 6

Roma communities usually live on the outskirts of the city in separate quarters or neighbourhoods; only in cities and rural villages where there are a limited number do Roma live integrated with the Macedonian population. The Roma are concentrated in 64 of the 85 municipalities of the country. 7 The largest Roma community in the country is located in the capital, Skopje. The cities of Prilep and Stip, the target areas of the evaluated project, host the second and the seventh largest Roma communities in Macedonia, respectively. In 2002, 4,433 Roma were registered in Prilep out of a total of 76,768 inhabitants (5.77% of the population), while in Stip, 2,195 Roma were registered out of 47,796 inhabitants (4.59% of the population). Unofficial data recorded in 2003 estimated a total of 12,000 Roma in Prilep and 5,000 in Stip. 8

According to estimates by UNICEF (2007), in Macedonia, the percentage of the population under the age of 19 years is between 22% and 29%; among the Roma minority, the average rises to between 41 and 47%.9

About 90% of Roma in Macedonia are Muslim; religion does not represent a factor for discrimination. Approximately 80% of Roma speak Romani as a first language and this constitutes an obstacle for learning, especially for children who need to access primary education. In the city of Prilep, in early 2000, only 15% of the total population spoke Romani. There are no precise data on Stip.10

The economic and social conditions of the Roma have been have remained, since early 2000 until today, very critical. Recent surveys show that 88% of Roma live below the poverty line, compared to a national average of 27.5%.11

Currently, the majority of Roma live in irregular dwellings, in separate quarters/neighbourhoods (ghettos), with an average area of 11 square metres per person; this corresponds to half the national average. Twenty-five percent (25%) of Roma live in unsafe homes, and 10% do not have toilets at home, as compared with only 5% and 2% of the total Macedonian population, respectively.12 Data shows the housing situation of Roma was significantly worse in the early 2000s, when the area available to each member of a family ranged from three to five square metres, with roughly 60% of the Roma population reporting not to have the necessary conditions (toilets and access to water) to ensure good personal hygiene.13

4 The 2002 Census recorded a population of 2,022,547 inhabitants in Macedonia, made up for 62% Macedonians, 25.2 % Albanians, 3.9 % Turks, 2.66% Roma, 0.2% from Serbs and 3.9% from other ethnic minorities. State Statistical Office - Republic of Macedonia, Census of Population, Households and Dwellings in the Republic of Macedonia , 2002. 5 Roma Education Fund, Needs Assessment Study for the Roma Education Fund. Background paper. Republic of Macedonia , 2004, p. 5. Open Society Institute Report, No Data - No Progress, Data Collection in Countries Participating to the Decade of Roma Inclusion 2005-2015 , 2010, p.53. 6 European Commission, The Former Yugoslav Republic of Macedonia 2013 Progress Report , SWD (2013) 413 final, p.13. 7 European Commission, The Former Yugoslav Republic of Macedonia 2013 Progress Report , SWD (2013) 413 final, p.13. 8 State Statistical Office - Republic of Macedonia, Census of Population, Households and Dwellings in the Republic of Macedonia , 2002. 9 UNICEF, Breaking the cycle of exclusion. Roma children in South East Europe , 2007, p. 6. 10 European Roma Rights Centre, Macedonia Country Profile 2011-2012 , 2013, p.7. Roma Education Fund, Needs Assessment Study for the Roma Education Fund. Background paper. Republic of Macedonia , 2004, pp. 10-11. 11 UNDP-WB-European Commission, Regional Roma Survey 2011 . UNDP website on the Former Yugoslav Republic of Macedonia (http://www.mk.undp.org/content/the_former_yugoslav_republic_of_macedonia/en/home/countryinfo/). 12 UNDP-WB-European Commission, Regional Roma Survey 2011 . 13 Ministry of Labour and Social Policy - Republic of Macedonia, Strategy for Roma in the Republic of Macedonia , 2004, p. 24. 11

Evaluation-Macedonia, Cities of Stip and Prilep

The data on the health of Roma in Macedonia have been and continue to be of concern, given that the life expectancy for Roma is now 10 years lower than that of the national average and the infant mortality rate is twice the national average. Poverty-related diseases are widespread (e.g. child malnutrition, tuberculosis) as are those related to poor hygiene (e.g. hepatitis, diarrhoea, respiratory problems). There are also a number of social pathologies related to poor levels of education and poverty status (alcoholism, tobacco addiction, cardiovascular disease). Although significant discrepancies are not recorded between Roma and non-Roma with respect to accessing the National Health System (92% vs. 98%), serious challenges are apparent with regard to the purchase of medicines and vaccinations. In 2011, 68% of Roma reported not to have enough money to purchase required medicines, compared to 29 % in 2004. 14 In 2011, 15% of Roma children under the age of 14 were not vaccinated; compared with 4% in 2004. The system of care for pregnant Roma women is also highly problematic, alongside the issue of the prevention of teenage pregnancy, linked, moreover, to early marriage. 15

The unemployment rate of the Roma was 53%, compared with 27% of the non-Roma population (2011); Roma women are particularly vulnerable; the unemployment rate among Roma women is 70%, which is twice the rate of non-Roma women. Also of concern is the unemployment rate among youth (15-24 years), which is 71%. In 1999, UNDP recorded that unemployment rates were not proportional to the size of the Roma community. In fact, although the Roma accounted only for 2.2% of the total population in the country, the unemployment rate was 4.3%; on the contrary, the unemployment rate proved to be more equal for Macedonians (66.6% of the population) at 67.4%, for Albanians at 20.2% (22.7% of the population), for Turks at 3.6% (4% of the population), for Serbs at 1.6% (2.1% of the population) and for Vlachs at 0.2% (1.9 % of the population).16 The majority of Roma workers, men and women, undertake informal activities, mainly seasonal jobs and/or casual work which requires a low degree of professional qualification. The phenomenon of child labour is widespread. The low level of education has always been the main obstacle to access to better-paid and more skilled jobs; in 2003, one-third of the Roma population who was unemployed had not completed primary school.17

With regard to literacy levels (2011 data), 17% of Roma above the age of 16 years are illiterate. This represents a 13% difference in comparison with the non-Roma population. The difference in percentage is exacerbated when considering the illiteracy rate among women (20%). The rate of primary school enrolment is significantly lower among Roma children (74%) when compared to that of non-Roma (90%). The enrolment rate is even lower with reference to secondary school (27% among Roma and 65% among non-Roma), this is considering the fact that since the 2009/2010 school year secondary school is mandatory. In addition, enrolment in kindergarten is less common among Roma (16%) than among non- Roma (25%). 18 In 2004, 24% of Roma above 24 years of age were illiterate and 35% had not completed primary school. During the 2001/2002 school year, the percent of children who dropped out during primary school among Roma children was 48.63%. 19 Only 20% of Roma children who completed primary school were enrolled in secondary school. 20 One issue of particular importance has been and continues to be assistance to Roma children with disabilities or learning difficulties.

14 32% of Macedonians were unable able to purchase medicines in 2011. 15 UNDP-WB-European Commission, Regional Roma Survey 2011 . Ministry of Labour and Social Policy - Republic of Macedonia, Strategy for Roma in the Republic of Macedonia , 2004, pp.53-60. Roma Education Fund, Needs Assessment Study for the Roma Education Fund. Background paper. Republic of Macedonia , 2004, pp. 25-27. 16 UNDP, National Human Development Report 2001. Social Exclusion and Human Insecurity in FYR Macedonia , 2001, p. 25. 17 Ministry of Labour and Social Policy - Republic of Macedonia, Strategy for Roma in the Republic of Macedonia , 2004, p. 31. 18 UNDP-WB-European Commission, Regional Roma Survey 2011 . 19 Roma Education Fund, Needs Assessment Study for the Roma Education Fund. Background paper. Republic of Macedonia , 2004, pp. 30-33. 20 Ministry of Labour and Social Policy - Republic of Macedonia, Strategy for Roma in the Republic of Macedonia , 2004, p. 41. 12

Evaluation-Macedonia, Cities of Stip and Prilep

Illiteracy and low levels of education clearly constitute an obstacle to the improvement of the socio- economic conditions of the Roma population, in addition to representing an objective difficulty for the Roma in having equal access to the Macedonian social assistance system.21

In 2005, nine countries of Eastern Europe, including Macedonia, joined the “Decade of Roma”, an initiative aimed at eliminating the racial discrimination suffered by Roma throughout Europe. The objective of the initiative is to contribute to the improvement of the economic and social conditions of the Roma population. The Decade brings together governments and intergovernmental organizations, including the Council of Europe and NGOs representing Roma communities. The emphasis is placed on education, employment, health, housing conditions and the elimination of poverty, as well as racial discrimination and gender. Twelve states currently participate in the initiative. 22

In conjunction with the launch of the “Decade of Roma”, the “National Strategy for Roma Inclusion 2005- 2015” was launched in Macedonia in 2004. The Strategy, which is considered to be a national effort to reduce poverty and address the fact that the Roma constitute the most vulnerable minority in the country, aims to promote the improvement of living conditions among Roma and their integration into the economic and social fabric of Macedonia. Through the Strategy’s adoption of a multidisciplinary approach, the Macedonian Government intends to achieve the following objectives: i) to achieve the full integration and participation of Roma in Macedonian society; ii) to provide conditions for equal access to existing social opportunities; iii) to reduce the poverty and social exclusion of Roma; iv) to reduce the differences in education levels between Roma and non-Roma; and iv) to reduce stigma and discrimination against the Roma. The Strategy adopts the key priorities of the Decade of Roma programme – i.e. housing, education, health and employment - as the main areas for action at national level. 23

The adoption of this Strategy by the Macedonian Government also responds, in particular, to the need to align with the acquis communautaire in view of EU accession 24 and to implement the numerous international agreements to which the Republic of Macedonia is a party.

As noted by the European Commission, after almost ten years of policies focused on Roma and following the revival of an ad hoc strategy for Roma inclusion from 2012-2014, 25 in Macedonia:

“(…) the overall implementation of the Roma strategy has slowed. The sustainability of ongoing projects in the area of health, education, employment, housing and social welfare is at risk, as they are mainly donor funded. Evaluation and monitoring mechanisms are weak and ineffective. The unit for implementation of the Roma strategy in the Ministry of Labour and Social Policy remains understaffed. Segregation of Roma in schools still persists. No comprehensive measures have been taken to address the overrepresentation of Roma children in special schools, nor to mainstream street children into education. Open discrimination against the Roma continues, particularly in employment. Legislative measures still need to be taken to address civil registration and personal

21 Roma Education Fund, Needs Assessment Study for the Roma Education Fund. Background paper. Republic of Macedonia , 2004, pp. 15-18. 22 The countries participating in the "Decade of Roma" initiative are: Albania, Bosnia and Herzegovina, Bulgaria, Croatia, Macedonia, Montenegro, Czech Republic, Romania, Serbia, Slovakia, Spain and Hungary. Slovenia has observer status. The initiative is funded by governments, multilateral organizations, and private sources. For details, see the website for the Decade of Roma Inclusion 2005-2015 (http://www.romadecade.org). 23 Ministry of Labour and Social Policy - Republic of Macedonia, Strategy for Roma in the Republic of Macedonia , 2004, pp. 4-8. 24 Macedonia has been one of the candidate countries to join the EU since 2005. 25 Ministry of Labour and Social Policy - Republic of Macedonia, Strategy for intensifying of the social inclusion of Roma in the social protection system in the Republic of Macedonia for the period 2012-2014 , 2011. 13

Evaluation-Macedonia, Cities of Stip and Prilep

identity documentation. The Roma continue to live in a cycle of poverty and unemployment, and in substandard living conditions” .26

Therefore, despite some progress with respect to the previous years, and as evident in reports on the progress of the “Decade of Roma”27 initiative and described briefly herein, further work is required to promote the integration and socio-economic inclusion of Roma in Macedonia. In concluding this section, it should be noted that the Municipalities of Stip and Prilep, the project’s target cities, are among eight Macedonian cities that have developed local plans to improve the status and conditions of Roma communities living within their jurisdictions and are among the 19 Macedonian cities that have signed a cooperation agreement with the Government for the joint Implementation of the National Strategy for the Roma. 28 29

26 European Commission, The Former Yugoslav Republic of Macedonia 2013 Progress Report , SWD (2013) 413 final, p. 46. 27 Progress Reports 2010, 2011, 2012 and the Civil Society Monitoring Report 2012 available at the following link: http://romadecade.org/article/macedonia/9309. 28 Ministry of Labour and Social Policy - Republic of Macedonia, Strategy for intensifying of the social inclusion of Roma in the social protection system in the Republic of Macedonia for the period 2012-2014 , 2011. 29 A more in-depth description of the country context in Macedonia in from 1991-2013 is provided in Annex 3. 14

Evaluation-Macedonia, Cities of Stip and Prilep

Project Summary

MACEDONIA, CITIES OF STIP AND PRILEP: IMPROVING THE QUALITY OF LIFE OF ROMA AND TITLE PROMOTING INTEGRATION

CODE 7478/INTERSOS/MCD

COUNTRY OF MACEDONIA , suburban areas with a high prevalence of Roma in the CITIES OF STIP (Radanski INTERVENTION Pat) AND PRILEP (Trizla1 e Trizla 2)

EXECUTING INTERSOS NGO STIP CERENJA ASSOCIATION LOCAL

PARTNERS PRILEP AID FOR HANDICAPPED AND POOR (AHP) ASSOCIATION

STIP Municipality of Stip 4 primary schools within Radanski Pat District: Dimitar Vlahov, Goce Delcev, Tosho Arsov, Vancho Prke OTHER AGENCIES PRILEP INVOLVED Municipality of Prilep Primary school Dobre Javanoski

Ministry of Education – Republic of Macedonia

START 15/06/2004 END 14/06/2007

DURATION 36 months (no extension requested)

EXPECTED COSTS ACTUAL COSTS APPROVED EXPENDITURE

TOTAL COST TOTAL COST TOTAL COST € 1,560,195.00 € 1,550,265.00 € 1,523,365.57

COSTS CHARGED COSTS CHARGED COSTS CHARGED COSTS MFA-DGCD € 774,009.00 MFA-DGCD € 764,001.00 MFA-DGCD € 737,101.00 INTERSOS € 237,831.00 INTERSOS € 238,039.00 INTERSOS € 238,039.00 PARTNERS € 443,570.00 PARTNERS € 412,485.00 PARTNERS € 412,485.00 AHP-CERENJA AHP-CERENJA AHP-CERENJA

CO-FINANCING € 114,786.00 CO-FINANCING € 135,740.00 CO-FINANCING € 135,740.00 Municipalities of Stip and Prilep Municipalities of Stip and Prilep Municipalities of Stip and Prilep

15

Evaluation-Macedonia, Cities of Stip and Prilep

OVERALL Improvement in living conditions and integration of Roma communities in Stip and Prilep OBJECTIVE

A. Improvement in community hygiene SPECIFIC B. Increase in schooling OBJECTIVES C. Organized management of activities and cultural exchanges

OBJECTIVE A Stip - Radanski Pat District: water adduction/distribution network constructed; community sanitation and hygiene facilities constructed, functioning and properly used.

Prilep – District of Trizla 1 and Trizla 2: drainage canal is cleaned regularly and running properly; the main road dividing the two districts is paved.

OBJECTIVE B In Stip and Prilep, a total of 800 school-aged children re-enter primary school and regularly attend EXPECTED classes. RESULTS

Approximately 100 children with substantial difficulties supported through after-school activities.

OBJECTIVE C In each district (Radanski Pat, Trizla 1 and Trizla 2) a neighbourhood association exists.

In each of the two cities of Stip and Prilep, a multicultural centre (MCC) is established, managed by a district-appointed committee and operates regularly.

OBJECTIVE A A1 Construction of the water adduction/distribution network in Stip A2 Community hygiene and sanitation facilities in Stip A3 Cleaning of the drainage canal in Prilep A4 Pavement of the road in Prilep

OBJECTIVE B B1 Distribution of clothing and scholastic materials/school supplies B2 Contribute to school running costs B3 Establish parallel education classes ACTIVITIES OBJECTIVE C C1 Meeting with the Roma community and their leaders C2 Creation of neighbourhood associations and management committees C3 Construction and launch of the Multicultural Centres (MCC) C4 Cultural initiatives and activities C5 Afterschool programs C6 Educational course in hygiene and sanitation for women C7 Adult literacy course C8 Course in conflict dynamics and management C9 Adm. and management course for members of MCC management committees

The report will identify the specific objectives according to each of the principal pillars of the project

16

Evaluation

Methodology

The evaluation was initiated by examining the terms of reference provided and was divided into three main phases: i) desk review phase; ii) field mission and data collection phase; iii) synthesis phase. During the desk review phase , the ET initially focused on analysing the evaluation questions proposed by the MFA and examined all supporting project documentation, including the original proposal developed in 2000, the technical feedback of the Central Technical Unit (CTU), three technical and financial reports produced by InterSOS during project implementation, documentation required analyse the context of the intervention from an economic and socio-cultural point of view and other additional documents provided by the MFA at a meeting held on 15 February 2013. The ET then proceeded to reconstruct the intervention logic utilizing the Theory of Change tool (Annex 2), a method through which is possible to highlight logical connections linking inputs, activities, outputs, outcomes and impacts. In this way, the ET was able to both test the consistency of the intervention logic, identifying possible problems, and to identify the main sectors of intervention and the overall project strategy.

Based on this initial analysis, the ET developed a set of evaluation questions and sub-questions (see the section on “Evaluation Questions”), appropriate indicators, means of verification and methodologies best suited to conduct the data analysis during the synthesis phase. During the desk review, it was also decided to plan and organize the mission field.

The field mission phase included the involvement of two evaluation experts, Dr. Serena Rossignoli and Dr. Annarosa Mezzasalma. The field mission lasted for seven days, 20-26 October 2013. During the mission, around 23 interviews were conducted and meetings were held with approximately 32 persons. The objective of the mission was to gather more information to complete the logical framework and fill in information gaps. To strengthen the scope and quality of the data collection, the ET relied on the local expertise of Dr. Iljia Talev, who supported further analysis and research, enabling access to information that would have otherwise been inaccessible to the ET given existing linguistic and cultural barriers. Given the nature of the ex-post evaluation, the field mission was particularly useful to verify certain aspects of the project, including the impact and sustainability of the intervention. In addition, interviews with four InterSOS staff and the MFA project focal point were conducted in Italy.

Finally, during the synthesis phase , substantial data analysis was conducted, allowing for the formulation of responses to the evaluation questions and the development of recommendations, conclusions and lessons learned. The final version of the evaluation report has been translated into two languages, Italian and English.

17

Evaluation-Macedonia, Cities of Stip and Prilep

Evaluation Questions

The evaluation questions developed by the ET are consistent with the Principles for the Evaluation of Development Assistance OECD-DAC, as suggested in the terms of reference, namely: - Relevance: the extent to which the project has taken into account the context and the problems of the country and the extent to which the project objectives were consistent with the requirements and needs of the recipient. - Validity of the design : the extent to which the design of the program was logical and consistent. - Efficiency: a) if the results have been achieved within the anticipated budget; b) if the results have been achieved in the allotted timeframe; c) if any alternative used was more efficient (lower costs and shorter time ) with respect to others. - Effectiveness : 1) whether the objectives, general and project-specific, have been clearly identified and quantified; 2) whether the characteristics of the project are consistent with its overall objective and specific objectives; 3) the extent to which the overall and specific objectives were achieved; 4) the main factors that influenced the achievement of the objectives. - Impact: The extent of the direct and indirect effects as well as real changes caused by the project within the context and to what extent this could be attributed to the project. - Sustainability : the extent to which the project's benefits will continue even after the assistance from the DGDC has ceased and the main factors influencing the achievement or non-achievement of sustainability of the project.

On the basis of the criteria and guidelines described above, the ET developed an evaluation framework (Annex 1), comprising six evaluation questions, each divided into 23 sub-evaluation questions. The methods used by the ET to process the answers to each question included qualitative and quantitative analyses. The questions are elaborated as follows: • Is the strategy of the project relevant? • Has the project been efficient? • Is the design of the project valid? • To what extent have the project objectives been achieved? • What has been the impact of the initiative? • To what extent have the project's benefits continued even after its conclusion?

Sources of Information

The evaluation utilized a variety of sources and means of verification to ensure the cross-referencing of information, data reliability, and credibility of results, including: • Analysis of relevant documents on topics such as Roma integration, the education sector in the Republic of Macedonia, the economic, institutional and socio-cultural relations of international agencies relating to the areas of intervention. • Documents concerning the project and provided by the DGDC. • Semi-structured interviews and occasional focus groups, with a wide variety of stakeholders including officials of the MFA, InterSOS, Cerenja, AHP, officials and mayors of the two municipalities involved - Stip and Prilep - Ministry of Education of the Republic of Macedonia, project beneficiaries, principals of the schools targeted by the initiative.

18

Evaluation-Macedonia, Cities of Stip and Prilep

• Direct observation through visits conducted to the main sites affected by the project, such as the Prilep canal, the road separating the districts of Trizla 1 and 2 in Prilep, the multi-cultural centres in Stip and Prilep, hygiene and sanitation facilities in Stip and the water adduction/distribution network in Stip.

Limits of the Evaluation

The principle limitation of the evaluation is associated with the time interval between the end of the project and the implementation of the evaluation . The evaluation was conducted seven years after the end of the project. While this made it possible to analyse certain aspects such as the sustainability and the impact of the intervention, it was more difficult with respect to the collection of data related to the project’s effectiveness and efficiency. In many cases, respondents could not remember details and accurate information about events which occurred seven years earlier. In addition, it was not possible to reach many staff of InterSOS, partner organizations, beneficiaries and local stakeholders who had participated directly in the project. Given the mobility that characterizes the Roma, it was extremely difficult to contact beneficiaries. In fact, in Stip, it was not possible to interview a single project beneficiary. In Prilep, the ET was able to organize a focus group discussion with youth who participated in Pillar B activities, with support from the local partner, AHP and was able to meet some current inhabitants of Trizla 1 and to discuss the community sanitation and hygiene conditions in an attempt to understand information from the past.

Another limitation of the analysis was the availability of data , especially related to health and education aspects of the project. In the Macedonia, much information which is recorded and officially registered is kept only for a few years. This complicated the ability to assess the impact of the project and required the ET to make changes to initial evaluation framework design. Nevertheless, the ET proposed alternative methods of analysis in order to formulate conclusions regarding the impact of the project.

In addition, in the reconstruction of the intervention logic, the ET identified the absence of appropriate targets to quantify the extent to which the project achieved its intended objectives. This absence could have been overcome had it been possible to consult with those who designed the project and who contributed to its realization. However, due to the circumstances outlined above, this was not possible. In order to overcome this difficulty, the ET referred, where possible, to national targets provided by the competent Macedonian ministries and/or by relevant international organizations.

Finally, another difficulty was caused by the amount of time that elapsed between the submission of the initial project document to its subsequent approval and activation. This aspect resulted in the need to reconstruct the intervention logic and the subsequent modification of project steps and justifications were particularly complex and costly in terms of time.

Evaluation Team

A team of three persons from the Scuola Superiore Sant'Anna contributed to the evaluation: Professor Andrea de Guttry coordinated and supervised all phases of the evaluation and Dr. Serena Rossignoli and Dr. Annarosa Mezzasalma conducted all phases of the evaluation (desk review, field mission and synthesis).

19

Evaluation-Macedonia, Cities of Stip and Prilep

The team relied on the local expertise of Dr. Iljia Talev , who made a valuable contribution to the contextual analysis and supported the retrieval and refinement of key data. In addition, Dr. Daniel Carpita investigated all aspects of the development of the design matrix , including the reconstruction of the intervention logic (logical framework) and assessment of the efficiency of the project, reconstructing the single elements of the project from both a financial and timing point of view.

Overall, the team utilized a multidisciplinary approach which benefited from the presence of qualified professionals with a background in evaluation and experience in the region.

20

Evaluation-Macedonia, Cities of Stip and Prilep

Analysis of Evaluation Questions

Relevance

1. Is the project strategy relevant?

1A Do the beneficiaries feel the project is useful?

1B Is the project strategy consistent with the needs of the beneficiaries?

The project implemented activities regarding i) community hygiene and environmental health; ii) primary education; and iii) cultural integration to benefit Roma communities in Stip and Prilep, notably in neighbourhoods with a high prevalence of Roma: Radanski Pat District in Stip and Trizla 1 and Trizla 2 in Prilep.

As already mentioned, in 2002, 4,433 Roma were registered in Prilep, out of a total of 76,768 inhabitants, or 5.77% of the population; in Stip, 2,195 Roma were registered out of a total of 47,796 inhabitants, or 4.59% of the population. Unofficial data recorded in 2003 in Stip and Prilep, suggests a population of 5,000 and 12,000 Roma, respectively. 30 The cities of Prilep and Stip host the second and seventh largest Roma communities in Macedonia, respectively.

In the absence of specific data on the project’s target cities, information already provided in the section on the “Project Context" also highlights the condition of Roma communities in Stip and Prilep. It is important to note that during the years when the project implementation was planned, the country context was characterized by an emergency situation, following the dissolution of the Federal Republic of Yugoslavia, the Balkan wars and the conflict in Kosovo (see Annex 3). In light of this situation and considering previous interventions implemented in Macedonia by Italian humanitarian NGOs with ECHO in the late 1990s and early 2000s, the idea developed for a more comprehensive intervention to assist Roma communities. The contextual analysis by InterSOS was based on requests from local partners; considering the emergency situation in the country at the time, this analysis was unable to benefit from detailed research or studies.

The project identified three main challenges it aimed to address during the three year implementation period.

PILLAR A. IMPROVEMENT IN COMMUNITY HYGIENE

The first problem identified concerns the precarious community hygiene and environmental conditions in neighbourhoods where Roma communities predominantly live in Stip and Prilep. In the development phase of the project, meetings with the Roma community and the local partner organization, Cerenja, highlighted the lack of access to water and lack of adequate sanitation facilities in the Radanski Pat District of Stip. These elements contributed to poor levels of community hygiene and consequently impacted the health of the inhabitants of the district. Therefore, the project intervention included the construction of a water adduction/distribution network, linked with neighbouring supply tanks in the city and the construction of community sanitation and hygiene facilities connected to the new community water adduction/distribution network ( Activity A1 and A2 ).

30 State Statistical Office - Republic of Macedonia, Census of Population, Households and Dwellings in the Republic of Macedonia , 2002. 21

Evaluation-Macedonia, Cities of Stip and Prilep

In Prilep, at the start of project implementation, the central access road to the districts of Trizla 1 and Trizla 2 was in extremely bad condition, made of dirt, poorly maintained and subject to constant flooding. During the first year of implementation, it became apparent that the road was also subject to a continuous accumulation of litter and organic matter, given the absence of a sewage/drainage system and the malfunctioning of the collection tank at the front of the street. In addition, the canal receiving waste water from Trizla 1 was blocked with rubbish and organic matter which prevented the normal flow of water. The conditions of the road and canal presented serious challenges to the health of the inhabitants of the two neighbourhoods and thus project activities included the annual cleaning of the canal and pavement of the access road ( Activity A3 and A4 ).

No studies or statistics are available on diseases associated with low levels of personal hygiene or environmental pollution among the Roma community in Macedonia from 2000-2013. In the case of Stip and Prilep, no detailed analysis on the incidence of such diseases in the target neighbourhoods was carried out either in the design phase or in the later stages of the project. Although the first progress report by InterSOS states a study was conducted during the first year of the project through individual interviews on the health and sanitary conditions in Prilep, the results of such research are not indicated in the progress report. In addition, in Prilep, a wealth of information on the actual condition of the access road to Trizla 1 and Trizla 2 was only recorded during the first year of the project, ensuring the Municipality undertook necessary preliminary work to pave the access road and construct the sewer system in Trizla 1, in accordance with the master plan of the city. In this regard, it must be noted that a reduction in the percentage of diseases, although not clearly identified, was an indicator in the project’s logical framework to measure the achievement of specific objective A: “Improvement in community hygiene”.

Data was found by the ET on the principle effects related to children's health and access to the public healthcare system by the Roma population. 31 The data, however, are not relevant to the evaluation of specific project activities. The only data on community hygiene conditions and on environmental diseases - provided by the WHO - indicate there were no particular issues related to poor hygiene or environmental pollution in Macedonia ; from 2000-2010, the data show that 100% of the population had access to drinking water and that at least 20% of the population did not have access to adequate toilets connected to the sewage system (2005-2006).32 The most worrying trend detected at the national level in 2005-2006 relates to the percentage of children exposed to contact with the faeces (50%); however, this figure decreased significantly in 2011 (17.3% nationally, 25% within Roma settlements). The National Bureau of Statistics recorded that 10% of Roma children did not use a toilet in 2005-2006. 33

During the on-site field mission, the ET was able to observe the living conditions of Roma communities in Stip and Prilep. Although the situation has definitely improved since 2001 (design of the project) to 2004 (commencement of operations), it was evident that the districts where Roma communities are concentrated continue to face problems related to the sewage system and pavement of roads . In fact, the

31 European Roma Rights Centre, Macedonia Country Profile 2011-2012 , 2013. UNDP-WB-European Commission, Regional Roma Survey 2011 . Website UNDP in the former Yugoslav Republic of Macedonia (http://www.mk.undp.org/content/the_former_yugoslav_republic_of_macedonia/en/home/countryinfo/). Ministry of Labour and Social Policy - Republic of Macedonia, Strategy for intensifying of the social inclusion of Roma in the social protection system in the Republic of Macedonia for the period 2012-2014 , 2011. UNICEF, Breaking the cycle of exclusion. Roma children in South East Europe , 2007. Roma Education Fund, Need s Assessment Study for the Roma Education Fund. Background paper. Republic of Macedonia , 2004. Ministry of Labour and Social Policy - Republic of Macedonia, Strategy for Roma in the Republic of Macedonia , 2004. 32 WHO, The Former Yugoslav Republic of Macedonia: Health profile , 2013. State Statistical Office, Republic of Macedonia-Multiple Indicator Cluster Survey 2005-2006 , p. 38. 33 State Statistical Office, Republic of Macedonia-Multiple Indicator Cluster Survey 2005-2006 , p. 38. State Statistical Office, Republic of Macedonia- Multiple Indicator Cluster Survey 2011 , pp. IV, 61. 22

Evaluation-Macedonia, Cities of Stip and Prilep majority of homes were constructed in an irregular manner: the District of Radanski Pat in Stip is not included in the Municipality’s master plan and the District of Trizla 1 in Prilep was not included in the development plan of the Municipality until the intervention of InterSOS; and in fact, it was only included partially therein.

In addition, in Prilep, the district of Trizla 1 lacks a system of rubbish collection, which is the main reason the residents dispose of waste in the canal. The project aimed to progressively educate the public on environmental responsibility, by actively involving them in the cleaning of the canal in collaboration with city workers and through an awareness campaign on environmental damage and health problems caused by the stagnation of wastewater, implemented in the first year of the project. However, as the intervention did not put in place specific actions to establish and maintain the systematic collection of rubbish - either self-managed by the community or promoted by the Municipality – this was not been achieved in Prilep and one of the main problems continues to be linked to the presence of rubbish in the canal.

The interviews conducted in Stip and Prilep with project partners and with representatives from the two Municipalities highlighted the usefulness of the intervention and its achievements in light of the needs of the beneficiaries. In Stip, the ET was unable to meet with any direct project beneficiary from the years when the community sanitation and hygiene facilities established by the project were operating; however, information from the desk review and on-site observations as well as project progress reports demonstrate the strategy put in place was consistent with the needs of beneficiaries. In Prilep, the ET was able to meet with members of the Roma community living in Trizla 1 and in the vicinity of the drainage canal, who confirmed the relevance of the actions implemented by the InterSOS project to their needs.

PILLAR B. INCREASE IN SCHOOLING

The second issue addressed by the project concerns the low level of primary education and schooling among Roma children . The project aimed to achieve at least primary education for all children who had never attended school in the neighbourhood of Trizla 1(Prilep) and Radanski (Stip), either because of family poverty or the inability to cover the cost of school supplies and clothing. The project operated on the assumption that Roma families recognized that a positive future for their children was linked to the precondition of receiving a basic education.

Article 44 of the Constitution of the Republic of Macedonia states that all persons have an equal right to education and that primary education is compulsory and free. In line with Article 48 of the Constitution, the Ohrid Accords of 2001 and the 2008 Law on the Promotion and Protection of Minority Rights, when a community constitutes at least 20% of the population of the municipality of residence, the community has the right to receive primary and secondary education in their mother tongue, with Macedonian being taught as a second language.

The public education system in Macedonia, with the exception of university education, is organized as described below: • Kindergarten: optional for children from 1 to 5 years of age. • Primary School: Until 2004, primary school consisted of eight years of study for children aged 7 to 15 years. In 2007, the state introduced the program in which nine years of school are compulsory for children aged 6 to 15 years. Thus, beginning with the 2005/2006 school year, it was mandatory for children aged 6 it to attend “year zero” in preparation for primary school, followed by two, four- 23

Evaluation-Macedonia, Cities of Stip and Prilep

year cycles of class (as in the previous system). The first cycle, from 1st to 4th grade, includes one teacher per class (classroom teaching) and no examination required to enter the next grade. In the second cycle, from 5th to 8th grade, classes are based on specific subjects (subject teaching) and students must pass an examination each year to proceed to the next grade. In addition, a final examination is required to assess the level of achievement at the end of primary school. • Secondary School: general or specialized secondary schools (languages or science). Schools offer technical, artistic or professional programs. Secondary school lasts for four years and includes examinations. Since 2009, following legislative changes in 2007, secondary school is now compulsory and free. 34

Based on 2002 Census data, the Roma community represents the youngest segment of the Macedonian population, meaning that the percentage of school-aged children is higher than that of other ethnic groups. The “National Strategy for Roma Inclusion 2005-2015” recognizes the absence of accurate statistics in the education sector and the availability of data disaggregated by ethnicity . The data demonstrate, however, the low level of education among the Roma minority in comparison with the national average. In 2002, more than half (51.8%) of the adult Roma population had no formal education or had not completed primary school, compared to a national average of 18%. There are no precise statistics on the participation and completion of school cycles for Roma in Macedonia and no figures are available on the number of children who have never been enrolled in school. The absence of this information is associated with the lack of birth registration among children, the mobility of families, and the numerical discrepancies between the official and unofficial Roma population, as already highlighted in this report in the description of context of the project.

Limited data is available on the presence of Roma children in kindergarten; it is estimated to be quite low (4%, as compared to 17% of the Macedonian population).35 This lack of pre-school education means that, given the widespread prevalence of the Romani language (80% of the Roma population speaks Romani), many children begin formal school without understanding the Macedonian language in which they are being instructed. 36

The rates of enrolment of children in primary school differ between the national average and those of the Roma ethnic group. It is important to remember that the percentage of school-aged children among the Roma population is higher than the rest of the national population. The 2002 Census data show that 11% of the national population is aged 7-14 (221,961 children), while the corresponding percentage among Roma is 16% (8,621 children). The total number of Roma children enrolled in primary school turns out to be much lower than the national average. In 2005, UNDP estimated that the rate of primary school enrolment of Roma children was 76% (81% males and 71% females), compared with a rate of 98% for children living in the vicinity of the Roma communities and a national average of 82%. 37 The National Bureau of Statistics reports a primary school attendance rate of 61% among Roma children compared to a national average of 90% from 200-2006. 38 In 2011, the level of schooling among Roma children improved, with the rate of enrolment in 1st grade reaching 84%, and the attendance rate being 86%. About 14% of Roma children did

34 Articles 44 and 48 of the Constitution of the Republic of Macedonia (http://www.sobranie.mk/en/default.asp?ItemID=9F7452BF44EE814B8DB897C1858B71FF). Roma Education Fund, Country Assessment Macedonia, 2011 , pp.18-28 . UNESCO, World data 7th Edition 2010/2011- The Former Yugoslav Republic of Macedonia , December 2011. 35 State Statistical Office, Republic of Macedonia-Multiple Indicator Cluster Survey 2005-2006 , p. 47. 36 Only in the schools of Shuto Orizari (Roma Municipality of Skopje, the largest Roma settlement in Macedonia) is Romani used as a first language in accordance with the legislation, and only after 5th grade can Romani be an optional subject chosen by the student. 37 UNDP, National Vulnerability Report for Macedonia - Focus on Roma , 2006, p. 20. 38 State Statistical Office, Republic of Macedonia-Multiple Indicator Cluster Survey 2005-2006 , p. 49. 24

Evaluation-Macedonia, Cities of Stip and Prilep not attend school in 2011, compared to 20% recorded at the start of 2000 (UNICEF research data from the settlement of Roma Shuto Orizari - Skopje, the largest in Macedonia).39

In 2004, the Roma Education Fund indicated the dropout rate of Roma children in primary school had declined to 48.63%, compared to the low rate of 0.9% for the rest of the school-aged population in Macedonia. 40 The abandoning school studies occurred particularly during the transition from the classroom teaching cycle (from 1st to 4th grade) to the cycle organized based on subject teaching (from the 5th to 8th grade), when the commitment to school becomes more burdensome in terms of the subjects studied, requirement to pass exams to advance to subsequent grades, as well as increased costs for families due to the greater number of subjects. Moreover, considering that there are no exams required during the first cycle, children often reach the 5th grade without proper preparation, an element which penalizes them further.

The main barriers to enrolment in and completion of primary education for Roma children are : i) failure to register the births of many children; ii) distrust of parents towards the formal education system, given that Roma culture is largely oral and based on the concept of learning within the family and community, and the inability of parents to follow children throughout the school curriculum considering their economic and social conditions; iii ) the fact that many Roma children do not speak Macedonian, the official language of instruction; iv ) low levels of registration in “year zero” (nursery school or kindergarten) that would assist with breaking down the language barrier in preparation for primary school; v) high costs to attend school that increase substantially in the transition from the 4th to 5th grade (from € 150-200 in the first year to € 300-400 in higher grades 41 ); vi) dropping out of school to go to work and help support the family (in 2011 the rate of child labour among Roma was 10.3%, compared to a national average of 16.6%42 ); vii) the mobility of families; and viii) early marriage for girls (12% of Roma girls are married before the age of 15 and 47% before age 18, one out of five Roma girls was married between 15 and 18 years 43 ).

It should be noted that in 2008, the Ministry of Education and Science launched a reform process in which it addressed issues referred to in points i), ii) and v) above. Firstly, the Ministry’s own inspectors conducted stricter oversight and control on parents failing to send their children to school. This was done though the activation of a signalling, warning and follow-up procedure with parents which may lead to sanctions for those who continue to not send their children to school. Secondly, the Ministry launched a programme providing free books for children in primary schools, thereby reducing the cost for parents.

In 2011, the Roma Education Fund estimated that about one quarter of Roma children did not continue their education beyond the 4th grade and about 45% of Roma children did not complete their primary education; In 2011, this rate decreased by about 10 percentage points. 44 Only 10-12% of Roma students continue to secondary school. Among boys between the ages of 15 and 19 years, only 4% attended primary school and 58% did not attend school at all. 45 However, in 2007, the Macedonian government eliminated

39 State Statistical Office, Republic of Macedonia-Multiple Indicator Cluster Survey 2011 , p. 111. UNICEF-WB, Study project - Vulnerability of Roma children in the dispersed Roma communities in Skopje , 2000. 40 Roma Education Fund, Needs Assessment Study for the Roma Education Fund. Background paper. Republic of Macedonia , 2004, p. 28-36. 41 During the mission, the ET found that a Macedonian worker can earn from €150 to € 200 per month, while a Roma worker who works occasionally can earn up to € 90/100 per month. 42 State Statistical Office, Republic of Macedonia-Multiple Indicator Cluster Survey 2011 , p. VII. 43 State Statistical Office, Republic of Macedonia-Multiple Indicator Cluster Survey 2011 , p. 127. 44 Roma Education Fund, Needs Assessment Study for the Roma Education Fund. Background paper. Republic of Macedonia , 2004, p. 28-36. State Statistical Office, Republic of Macedonia-Multiple Indicator Cluster Survey 2011 , p. 112. 45 Roma Education Fund, Needs Assessment Study for the Roma Education Fund. Background paper. Republic of Macedonia , 2004, p. 28-36. 25

Evaluation-Macedonia, Cities of Stip and Prilep the minimum grade required to enter secondary school for Roma children and has promoted an ad hoc scholarship programme to increase the frequency of compulsory secondary school since 2009.

In order to increase the participation of Roma children in the school system (target 90%) and the number of children who complete primary school (60-70%), in 2004 the National Strategy for Roma included, among others, the following measures:
increase in the complete registration of children who do not have documents to enrol in the 1st grade;
identify Roma children up to eight years of age and promote their inclusion in education following the regular learning programme established for that purpose;
allow children who have dropped out of school or have never been included in the educational process to complete primary education through specialized programs;
sensitize parents on the importance of school, involving social workers and psychologists in school activities of their children;
promote pre-school education;
organize after-school programmes to teach the Macedonian language;
include Roma language assistants in classes from 1st to 5th grade;
promote ethnic diversity and integration in schools and primary classes, and in extra-curricular activities.
provide scholarships, textbooks, school supplies and clothing. 46

The Macedonian Government has not instituted specific programmes of inclusion for Roma children, as evident in the National Programme for the Development of Education in the Republic of Macedonia 2005- 2015 and the promulgation of the 2008 law concerning adult education from 15 years of age. 47 After the launch of the Strategy in 2004, programmes have focused primarily on raising awareness among parents and the inclusion of Roma children in pre-school and kindergarten, an approach UNICEF has also confirmed to be successful in improving levels of education among Roma children in Macedonia, allowing them to gain a better command of the Macedonian language before starting their studies. 48 The National Strategy for Roma Inclusion 2012-2014 has since concentrated more on secondary education.49 50 In the complex environment concerning the education of Roma children in Macedonia, the InterSOS project had an ambitious goal of increasing the schooling of Roma children in Stip and Prilep, through activities in line with the National Strategy, including:
the distribution of clothing and school supplies – during the implementation phase, additional activities were added to this component, including the enrolment of children in the 1st grade, enrolment of children in “year zero” (beginning in the second year of the project, when pre-school became compulsory under Macedonian law), the creation of parallel classes from 1st to 3rd grade for

46 Ministry of Labour and Social Policy - Republic of Macedonia, Strategy for Roma in the Republic of Macedonia , 2004, pp. 43-46. 47 Ministry of Education and Science - Republic of Macedonia, National Programme for the Development of Education in the Republic of Macedonia 2005-2015. Ministry of Education and Science - Republic of Macedonia, Development and State of the adult learning and education - National Report of the Republic of Macedonia , 2008. 48 State Statistical Office, Republic of Macedonia-Multiple Indicator Cluster Survey 2005-2006 , p. 47. State Statistical Office, Republic of Macedonia- Multiple Indicator Cluster Survey 2011 , p. 104. UNICEF, Breaking the cycle of exclusion. Roma children in South East Europe, 2007 , p. 46. 49 Ministry of Labour and Social Policy - Republic of Macedonia, Strategy for intensifying of the social inclusion of Roma in the social protection system in the Republic of Macedonia for the period 2012-2014 , 2011. 50 To reconstruct the situation regarding the lack of primary education among Roma children in Macedonia, the ET referenced, in addition to the sources already mentioned, the publication: Open Society Institute, Equal Access To Quality Education For Roma. Volume 2 - Croatia Macedonia Montenegro Slovakia , 2007, pp. 167-275.

26

Evaluation-Macedonia, Cities of Stip and Prilep

the subsequent reintegration of children who had never attended primary school and who were no longer of a suitable age to enter 1st grade (8-14 years);
provision of grants to support the costs of the partner schools ;
extra-curricular activities, including after-school courses and recreational-leisure courses for cultural integration between Roma and Macedonia (Activity B1, B2 and B3 ).

InterSOS worked with four primary schools in Stip, two of which (Dimitar Vlahov and Goce Delcev) were located in the vicinity of the Radanski Pat District, which consists of the largest number of Roma students in the city. In Prilep, InterSOS worked with only one primary school, located in the district of Trizla 2. The school (Dobre Jovanoski) is the school in Macedonia with the second largest number of Roma children.

The desk analysis and interviews conducted during the field mission (with directors of schools, teachers, beneficiaries, partners and municipalities) revealed that the project activities responded to the needs of the beneficiaries were perceived to have been useful.

In particular, a focus group with five girls between 14 and 17 years of age who benefited from the distribution of clothing and after-school activities highlighted the relevance and usefulness of the project in supporting school attendance. The girls interviewed are still in the school system, two are finishing primary school and the other three are attending in high school. These girls have internalized the relative importance of education for personal growth and have confirmed that many Roma parents do not fully appreciate the utility of an education in contributing to the positive development of children.

In terms of the beneficiaries reached, it is predicted that the project did not reach the ambitious goal of reintegrating 800 children into the school curricula, but that the intervention did, as discussed in the section on effectiveness, reinsert a large number of children in primary school and assisted a considerable number of children through the distribution of school supplies and clothing.

With regard to the inclusion of children in “year zero”, the project demonstrated particular coherence in following the changes in national legislation taking place during the implementation of the project. With regard to the initiation of parallel education classes, the project, subject to the approval of the Ministry of Education and Science, and in consultation with target schools in Stip and Prilep, established a specific programme for the reintegration of children into primary school with an age greater than that permitted by law for placement in 1st grade; this is the only programme of its kind in Macedonia. The classes, composed entirely of Roma children aged 8 to 14 years, aimed to address the existing educational gap with their peers attending mainstream classes. To avoid problems associated with discrimination, these children were constantly involved in extracurricular integration programmes with other children attending the target schools (Goce Delcev in Stip and Dobre Jovanoski in Prilep) through the multicultural centres supported by the project. However, it is important to note that the parallel classes covered a period of three years, yet this specific aspect of the project did not include measures to support those participating in the parallel classes to join others for the completion of primary school. Some children have continued their schooling, enrolling in regular classes while others have abandoned their education.

It should be noted that the project intervention logic was defined in 2001, in the absence of legislative references and strategic government policies which developed since 2004. The project was therefore able to immediately identify a strategy relevant to the poor education and schooling of Roma children, based

27

Evaluation-Macedonia, Cities of Stip and Prilep on the work carried out by international organizations present in Macedonia, and was able to contribute to the realization of national directives issued during the project implementation period.

PILLAR C. ORGANIZED MANAGEMENT OF ACTIVITIES AND CULTURAL EXCHANGES

The third issue addressed by the project is twofold and involves: i) the significant fragmentation of groups of Roma existing in the target districts - at least 10 were documented in the initiative - and the lack of a common organization to enable Roma communities to better address and collectively confront issues of concern; and ii) the lack of integration between the Roma and non-Roma populations .

In order to promote solidarity among the Roma communities and the integration of the Roma and Macedonian populations, project activities included: the creation of neighbourhood associations including both Roma and non-Roma ( Activity C1 and C2 ); the construction of multicultural centres (MCCs) in the Radanski Pat District in Stip and Trizla 2 in Prilep (Activity C3 ); the realization of initiatives for the Roma community and joint initiatives for Roma and non-Roma through cultural, recreational, educational and social activities organized by the MCCs and in the target areas ( Activity C4, C5 , C6 , C7 , C8 and C9 )

In the documentation analysed by the ET to understand the context of the project, problems concerning the lack of solidarity and organization among Roma communities is not to be addressed, but is hinted at given the vast number of Roma organizations in the country. The Roma Education Fund reported that there were approximately 120 officially registered Roma NGOs in 2004, of which half were inactive. On the basis of their activities NGOs were categorized as follows: 48% as humanitarian organizations; 6.67% as organizations for women, children and young people; 2.67% as organizations operating in the cultural sphere; 34.67% as those promoting the protection of human rights; 2.67% as organizations of general character; and 5.33% as various other organizations. These organizations appeared to be relatively weak in terms of capacity to work independently with respect to donors and international NGOs as well as the ability to work through networks and with institutions. 51 In 2011, the Ministry of Labour and Social Policy reported that the country had 120 active Roma organizations.52

The reports analysed also did not deal with specific instances of hostility between Roma and non-Roma communities, except in relation to the dynamics of the socio-economic exclusion of the Roma. Clear problems which emerged included: the segregation of Roma communities to the peripheral areas of the cities; and the discrimination of Roma based on prejudices of the host population and among other ethnic minorities, resulting in the exclusion of Roma from the labour market. Exclusion also stems from the low level of education and professional qualifications of the Roma. In the country reports from the EU, UNDP, UNICEF, and WB it is evident that the Roma constitute the poorest and most socially marginalized in ethnic group in Macedonia. This assessment is constant for the 2000-2013 period.

With regard to the creation of Roma neighbourhood associations, the ET learned during the mission that these types of organizations already existed prior to the project, including those with Roma and with mixed Roma and Macedonian populations. There were at least three in Stip and about thirty in Prilep. In particular, the association formed in Stip, CerenjaSOS, was a duplicate of the local partner Cerenja in terms of members and activities conducted. That said, and given that the organizations set up by the project were

51 Roma Education Fund, Needs Assessment Study for the Roma Education Fund. Background paper. Republic of Macedonia , 2004, p. 43. 52 Ministry of Labour and Social Policy - Republic of Macedonia, Strategy for intensifying of the social inclusion of Roma in the social protection system in the Republic of Macedonia for the period 2012-2014 , 2011. 28

Evaluation-Macedonia, Cities of Stip and Prilep dissolved before (in Prilep) or upon its completion (in Stip) and although the situational analysis revealed a need to strengthen existing Roma associations, the action carried out by the project is not considered to be particularly relevant or useful as assessed by the beneficiaries, or by those who were members of the associations formed.

On the issue of Roma inclusion and integration between Roma and non-Roma, the ET understands there exists a significant problem in Macedonia in this regard, based mainly on the lack of mutual understanding between these groups and the persistence of prejudice. During the field mission, however, it was apparent, in particular through the interviews conducted with local partners and various organizations currently collaborating with MCCs, that this lack of integration does not correlate with an inability to work together or with a high level of conflict between Roma and non-Roma; instead, it translates into mutual distrust and a lack of cooperation.

Given the numerous campaigns promoted by the EU 53 , the Council of Europe 54 and other international organizations in favour of socio-economic inclusion and Roma integration, the ET considers the various initiatives promoted by the project to increase inclusion to be relevant. Partners, trainers, teachers, managers of Roma and non-Roma organizations involved in the project, with whom the ET was able to consult during the course of the mission, confirmed the usefulness and consistency of project actions to the needs of the beneficiaries.

In addition, initiatives under Pillar C of the project are integrated with activities under Pillar A. Improving community hygiene, refers to Activity C6 - courses in health and hygiene education for women, and Pillar B. Increasing education is particularly linked with Activities C4 Cultural initiatives, C5 Afterschool activities , C7 Adult literacy course and C8 Course in managing the dynamics of conflict . This enabled a high degree of interaction between the various Pillars of the project.

In concluding the analysis of the relevance of the project, the ET notes that, in the first progress report InterSOS claimed to have carried out two studies - one on the sanitary conditions of the population in Prilep and one aimed at identifying the socio-economic structure and the level of education of Roma families residing in project target districts (Radanski Pat in Stip and Trizla 2 in Prilep). In the documentation received by the ET, there is no analysis of the data collected, except that which is contained in the narrative summary of the progress report from the first year of implementation. Both studies should have been conducted; however, in the development phase of the project rather than in the course of the first year in order to better understand the local context and situation in more detail (the initial project description in the documentation provided is quite vague). The evaluation above is confirmed by changes which occurred during the first year of the project regarding the issue of paving of the road in Prilep, as already pointed out with respect to the structural work required by the Municipality on the sewer system, and the change in the project strategy regarding the objective linked to the inclusion of children in the school system.

53 For more in-depth information see the website on EU and Roma (http://ec.europa.eu/justice/discrimination/roma/index_en.htm). 54 For more in-depth information see the website of the Council of Europe dedicated to the promotion of the human rights of Roma (http://hub.coe.int/web/coe-portal/roma) and the website for the Decade of Roma Inclusion 2005-2015 (http://www.romadecade.org/). 29

Evaluation-Macedonia, Cities of Stip and Prilep

Efficiency

2. Was the project efficient?

2A Were the results achieved with the expected costs?

In order to respond to the question above, the ET conducted an analysis of the project financial records and its financial plan, budget, variations in the project pipeline, final financial report and approved expenditure. In addition, data and information collected during the on-site field mission was utilized.

The ET followed a series of predisposed calculations to understand the overall financial framework of the project based on details from the estimated budget, final statement of expenditure and approved expenses. The information provided below is detailed according to each year of implementation and based on the financial costs and expenditure for each actor involved.

Tab. 1 FINANCIAL PLAN – BUDGET AND FINAL EXPENDITURE in euro TOTAL PROJECT COSTS DIVIDED BY CO-FINANCING SHARES TOTAL PROJECT COSTS Total Project Share Share 1st 2nd 3rd Share InterSOS InterSOS Share Others Total Costs year year year DGCD CASH Val. counterparts (municipality) TOTAL BUDGET* 1,560,196.00 600,450.00 502,158.00 457,588.00 774,009.00 78,111.00 159,720.00 433,570.00 114,786.00 TOTALE EXPENDITURE 1,550,265.00 600,450.00 498,912.00 450,903.00 764,001.00 78,548.00 159,491.00 412,485.00 135,740.00 RESIDUAL BALANCE 9,931.00 0.00 3,246.00 6,685.00 10,008.00 -437.00 229.00 21,085.00 -20,954.00 TOTAL RESIDUAL BALANCE 9,931 .00

From the analysis of Table 2, it is evident that value of macro-items included in the estimated budget and in the final financial plan were realized for nearly all expected costs (99.36%), compared to a residual balance in expenditure of only 0.64%. It must be emphasized that in the impact of the variations in expenditure between the co-financing “counterpart" share and the co-financing “others (municipality)" share were not significant and approval of the request for changes was obtained by InterSOS from the MFA.55 These budget modifications, although within the parameters of those permitted by the MFA-DGDC 56 and not having significant impact on project expenditure, do highlight that the initial project budget required a review and adaptation on the basis of changes and new needs among beneficiaries and partners. It should be noted that the project and its costs were initially submitted by InterSOS in 2001; the project was then approved in 2003 by the MFA-DGCD and commenced operations in 2004. The period of time which elapsed between the initial submission of the project and the commencement of operations justifies the need to review and recalculate certain costs. To provide a complete picture of the overall project financial framework, Table 3 outlines the full succession of financing from the MFA-DGCD from the start of operations until the close of the financial statements.

55 2 modifications in 1st year (Prot. no. 337P/VII/0387001 of 02/09/04 and Prot. no. 337P/VII/0362116 of 14/09/05), 1 modification in 2nd year (Prot. no 337P/27/12/0509180 of 27/12/09) and 1 modification in 3rd year (Prot. no. 337P/VII/0655820 of 16/12/07) (Ref. Document “Reporting of Account Statements n.1 (III Final). 56 Ref. paragraph IV “Project management and reporting” sub paragraph 5 “Changes in the financial plan “document “Projects promoted by NGOs reporting format and procedures” – MFA-DGDC Office VII (Approved by the executive committee by resolution no. 36 of 13.05.97). 30

Evaluation-Macedonia, Cities of Stip and Prilep

Tab. 2 TOTAL PROJECT COSTS SPECIFIED BY CO-FINANCING SHARES in euro 1st year 2nd year 3rd year Share Share Share Share Share Share Share InterSOS InterSOS Share Others Share InterSOS InterSOS Share Others Share InterSOS InterSOS Share Others DGCD CASH Val. counterpart (municipality) DGCD CASH Val. counterpart (municipality) DGCD CASH Val. counterpart (municipality) TOTAL EXPECTED COSTS* 258,069.00 68,860.00 93,482.00 127,154.00 52,885.00 257,966.00 6,705.00 47,400.00 159,120.00 30,967.00 257,974.00 2,546.00 18,839.00 147,296.00 30,934.00 TOTAL EXPENDITURE 258,069.00 68,860.00 93,482.00 119,864.00 60,175.00 254,645.00 6,807.00 47,504.00 165,189.00 24,767.00 251,287.00 2,881.00 18,505.00 127,432.00 50,798.00

REMAINING BALANCE 0.00 0.00 0.00 7,290.00 -7,290.00 3,321.00 -102.00 -104.00 -6,069.00 6,200.00 6,687.00 -335.00 334.00 19,864.00 -19,864.00 TOTAL RESIDUAL BALANCE 0. 00 TOTAL RESIDUAL BALANCE 3,246. 00 TOTAL RESIDU AL BALANCE 6,686. 00

Tab. 3 DETAILED SUMMARY OF ANNUAL PROJECT FUNDING in euro DGCD Actual Residual Difference contribution Project accounted Funding Actual annual Non- Funding in project net Contribution Total DGCD budget gross Approved Funding expenditure Surplus deductible Closing provided credit or expenditure to DGCD funding Balance to expenditure expenses DGCD DGCD approved DGCD expenses balance DGCD debit NGO be excluded % recognized disbursed be paid* 1st year 600,449.00 600,449.00 258,069.00 258,069.00 0.00 0.00 1.00 258,068.00 258,069.00 -1.00 - - - -1.00 2nd - year 502,158.00 498,913.00 257,966.0 0 254,645.00 3,245.00 3,321.00 22,398.00 232,247.00 257,966.00 -25,719.00 - - - 25,719.00 3rd year 457,588.00 450,903.00 257,974.00 251,287.00 6,685.00 6,687.00 4,500.43 246,786.57 178,310.30 68,476.27 - - - 87,112.71 Total 1,560,195.00 1,550,265.00 774,009.00 764,001.00 - 10.008.00 26,899.43 737,101.57 694,345.30 42,756.27 755,738.01 755,738.01 694,345.30 61,392.71

Tab. 4 SUMMARY PROJECT FUNDING in euro Actual net of non-deductible Actual expenditure Recognized contribution (% of initial DGCD funding Final balance to be expenses DGCD funding, 49.61) provided paid*

1,523,365.57 764,001.00 755,738.01 694,345.30 61,392.71 57

57 Figure calculated as the difference between € 1.00 for the negative balance on the 1st year of € 25,179.00 by way of a negative balance on the 2nd year of € 87,112.71, and as a positive balance in the 3rd year, including the adjustment as a result of the calculation provided for in the opinion of the Council of State no. 1183/2009 of 22/10/2009 (recalculation proportional share of initial DGCD, or 49.61% of total eligible costs). Moreover, this figure has been subjected to in Article 10 of Law no. 426/96 on the application of the “principle of compensation” (Ref. “Decree approving statements of expenditure” MFA-DGCD Office VII - 15/02/2011). 31

Evaluation-Macedonia, Cities of Stip and Prilep

Below is a detailed analysis of the project financial plan conducted by the ET to verify whether the results were achieved with the expected costs.

Specifically, a reconstruction of the financial plan was undertaken (Table 5), indicating the estimated total cost for each item of expenditure, the total cost after variances and the actual expenditure in order to develop an analytical framework to identify any deviations or changes with respect to initial programming. Four budget modifications were approved in the course of the project; these were not significant with respect to the original financial plan 58 and did not result in changes in terms of the overall costs of the project, but did result in considerable variations requiring internal compensation between individual line items of expenditure. These modifications fall within the parameters permitted in the financial regulations established by the MFA-DGCD. 59 The changes affected personnel costs, infrastructure development (hygiene facilities and MCC), and purchase of goods for the MCC and supplies for schools. These variations highlight the need to review and refine project activities during the course of implementation based on changes in the context and needs of beneficiaries and partners, especially given the extensive lapse of time between the project design phase (2001) and the commencement of the project (2004). However, in certain instances, budget variations are linked to flaws in the initial project planning stage. In fact, when cross-checking the initial budget and cost analysis with the reformulated versions, several deviations are evident. Although these did not significantly affect the overall cost of the intervention, they nonetheless denoted the need to revise estimated costs to achieve the desired results. The most significant variations relate to “extracurricular activities” (item of expenditure in line 03:04), which revealed an increase of 45.05% compared to the budgeted cost as well as in relation to “Cleaning the drainage canal of Prilep” (item 03:08) which was 86.02% above the estimated budget.

58 2 modifications in 1st year (Prot. no. 337P/VII/0387001 of 02/09/04 and Prot. no. 337P/VII/0362116 of 14/09/05), 1 modification in 2nd year (Prot. no 337P/27/12/0509180 of 27/12/09) and 1 modification in 3rd year (Prot. no. 337P/VII/0655820 of 16/12/07) (Ref. Document “Reporting of Account Statements n.1 (III Final). 59 Ref. paragraph IV “Project management and reporting” sub-paragraph 5 “Changes in the financial plan” document “Projects promoted by NGOs reporting format and procedures” - MFA-DGCD Office VII (Approved by the executive committee by resolution no. 36 of 13.05.97). 32

Evaluation-Macedonia, Cities of Stip and Prilep

Tab. 5 DETAILED ANALYSIS OF THE SUBMITTED FINANCIAL PLAN, MODIFICATIONS AND EXPENDITURE in euro

Total expected Total cost after Actual Difference in expected costs after variance % actual Item of expenditure cost variance Variance % variance expenditure actual expenditure expenditure 1 Selection and training of volunteers and collaborators 01:01 Selection and training of volunteers 01:02 Selection and training of collaborators 2,066 .00 2,066 .00 0.00 0.00 % 2,066 .00 0. 00 0. 00% TOTAL 1 2,066 .00 2,066 .00 2,066 .00 2 Stipend and Management of Volunteers 02:01 Economic support of volunteers 02:02 Economic support for staff/collaborators Project Manager 10 ,4096 .00 91 ,937 .00 -12 ,159 .00 -11. 68% 84 ,241 .00 -7,696 .00 -9. 14% Construction technician 24,784.00 21,676.00 -3,108.00 -12.54% 24,230.00 2,554.00 10.54% 02:03 Expenses for management of volunteers/collaborators 8,240.00 7,100.00 -1,140.00 -13.83% 7,403.00 303.00 4.09% TOTAL 2 137 ,120 .00 120 ,713 .00 115 ,874 .00 3 Other items of expenditure Continuation of professional/technical volunteers and 03:01 collaborators 03:02 Other Italian personnel 33 ,302 .00 43 ,720 .00 10 ,418 .00 46 ,880 .00 3,160 .00 6. 74% Consulting in Italy Audit 0.00 1,500 .00 1,500 .00 new voice 3,600 .00 2,100 .00 58. 33% On -site consulting Experts in community dynamic 27 ,110 .00 38 ,266 .00 11 ,156 .00 41 .15% 40 ,472 .00 2,206 .00 5. 45% Trip to/from Italy 6,192 .00 3,954 .00 -2,238 .00 -36 .14% 2,808 .00 -1,146 .00 -40. 81% 03:03 Local personnel 479 ,904 .00 478 ,654 .00 -1,250 .00 45 ,6451 .00 -22 ,203 .00 -4. 86% Staff

Project Manager’s Assistant 37,152.00 35,022.00 -2,130.00 -5.73% 40,478.00 5,456.00 13.48% Chief Logistician 17,640.00 16,770.00 -870.00 -4.93% 26,125.00 9,355.00 35.81% Secretary 31,608.00 29,852.00 -1,756.00 -5.56% 26,157.00 -3,695.00 -14.13% Interpreters 35 ,280 .00 31 ,280 .00 -4.00 0.00 -11 .34% 36 ,039 .00 4,759 .00 13. 21%

Experts on dialogue networks 7,506.00 7,506.00 new voice 6,570.00 -936.00 -14.25% Entertainment /Organizers Stip 179 ,112 .00 179 ,112 .00 0.00 0.00 % 144 ,737 .00 -34 ,375 .00 -23. 75% Entertainers /Organizers Prilep 179 ,112 .00 179 ,112 .00 0.00 0.00 % 176 ,345 .00 -2,767 .00 -1. 57% Consultants 03:04 Increase in schooling 126 ,386 .00 122 ,786 -3,600 131 ,095 .00 8,309 .00 6.34% Clothing and school supplies for students 101,600.00 98,000.00 -3,600 -3,54% 96,006.00 -1,994.00 -2.08% 33

Evaluation-Macedonia, Cities of Stip and Prilep

Extra-curricular activities 15,492.00 15,492.00 0.00 0.00% 28,194.00 12,702.00 45.05% Classroom heating and electricity 9,294.00 9,294.00 0.00 0.00% 6,895.00 -2,399.00 -34.79% 03:05 Activities related to the Multicultural Centres 73,680.00 71,988.00 -1,692.00 62,621.00 -9,367.00 -14.96% Preliminary consultations with population 2,580.00 2,580.00 0.00 0.00% 4,290.00 1,710.00 39.86% Legal practices 2,582.00 2,582.00 0.00 0.00% 1.000.00 -1,582.00 -158.20%

Management costs for Multicultural Centre Stip 4,644.00 4,644.00 0.00 0.00% 4,366.00 -278.00 -6.37% Management costs for Multicultural Centre Prilep 1,548.00 1,548.00 0.00 0.00% 3,438.00 1,890.00 54.97% Cultural activities 12,380.00 12,380.00 0.00 0.00% 14,512.00 2,132.00 14.69% Afterschool - teachers 10,368.00 10,368.00 0.00 0.00% 10,316.00 -52.00 -0.50% Afterschool - teaching materials 3,098.00 3,098.00 0.00 0.00% 4,265.00 1,167.00 27.36% Health and hygiene education - teachers 5,184.00 5,184.00 0.00 0.00% 2,700.00 -2,484.00 -92.00% Health and hygiene education-sanitation kits 9,600.00 9,600.00 0.00 0.00% 7,131.00 -2,469.00 -34.62% Literacy - adult teachers 5,184.00 5,184.00 0.00 0.00% 3,120.00 -2,064.00 -66.15% Literacy - adult - materials 2,460.00 2,460.00 0.00 0.00% 1,334.00 -1,126.00 -84.41% Management of conflict dynamics - teachers 5,202.00 5,202.00 0.00 0.00% 2,160.00 -3,042.00 -140.83% Management of conflict dynamics - materials 1,440.00 1,440.00 0.00 0.00% 571.00 -869.00 -152.30% Corse in Management - teachers 3,456.00 2,728.00 -728.00 -21.06% 800.00 -1,928.00 -241.00% Corse in Management - materials 390.00 390.00 0.00 0.00% 19.00 -371.00 -1,908.24% Corse in Management - computers and printers 3,564.00 2,600.00 -964.00 -27.05% 2,600.00 0.00 0.00% 03:06 Facilities 114,141.00 109,454.00 -4,687.00 -4.11% 116,899.00 7,445.00 6.37% Purchase Washing machines for building hygiene facilities - Stip 2,850.00 2,850.00 0.00 0.00% 1,661.00 -1,189.00 -71.58% Boiler, radiators, building counters for hygiene facilities – Stip 3,615.00 3,615.00 0.00 0.00% 6,584.00 2,969.00 45.09% Equipment for 2 multicultural centres 15,492.00 10,805.00 -4,687.00 -30.25% 18,364.00 7,559.00 41.16% Transport and installation 1,807.00 1,807.00 0.00 0.00% 1,842.00 35.00 1.90% Cars 2X4 20,658.00 20,658.00 0.00 0.00% 20,809.00 151.00 0.73% Cars 4X4 41,316.00 41,316.00 0.00 0.00% 41,316.00 0.00 0.00% Spare parts 7,746.00 7,746.00 0.00 0.00% 8,016.00 270.00 3.37% Mini-bus 15,493.00 15,493.00 0.00 0.00% 15,227.00 -266.00 -1.75% Moped/scooter 5,164.00 5,164.00 0.00 0.00% 3,080.00 -2,084.00 -67.66%

34

Evaluation-Macedonia, Cities of Stip and Prilep

03:07 Construction 384,108.00 401,626.00 17,518.00 4.56% 415,900.00 14,274.00 3.43% Cost of construction Water adduction/distribution network Stip 77,500.00 68,112.00 -9,388.00 -12.11% 74,524.00 6,412.00 8.60% Building hygiene facilities in Stip - design 723.00 723.00 0.00 0.00% 2.000.00 1,277.00 63.85% Building hygiene facilities in Stip - construction 51,645.00 70,876.00 19,231.00 37.24% 78,239.00 7,363.00 9.41% Building hygiene facilities in Stip - toilets, sinks, showers, sinks 7,746.00 7,746.00 0.00 0.00% 1,053.00 -6,693.00 -635.61% Building hygiene facilities in Stip - operating costs 3,248.00 3,248.00 0.00 0.00% 3,208.00 -40.00 -1.25% Design of multi-cultural centres 2,582.00 2,582.00 0.00 0.00% 4,091.00 1,509.00 36.89% Construction of the Multicultural Centre Stip 72,200.00 84,275.00 12,075.00 16.72% 99,984.00 15,709.00 15.71% Construction of the Multicultural Centre of Prilep 108,300.00 103,900.00 -4,400.00 -4.06% 60,116.00 -43,784.00 -72.83% Cleaning the canal of Prilep 4,647.00 4,647.00 0.00 0.00% 33,232.00 28,585.00 86.02% Pavement of the road in Prilep 36,152.00 36,152.00 0.00 0.00% 50,798.00 14,646.00 28.83% Existing infrastructure Supervision of work 19,365.00 19,365.00 0.00 0.00% 8,655.00 -10,710.00 -123.74% 03:08 Land (use public land or private expropriation) 19,220.00 19,220.00 0.00 0.00% 19,778.00 558.00 2.82% Building hygiene facilities in Stip 3,720.00 3,720.00 0.00 0.00% 4,278.00 558.00 13.04% Multicultural Centre Stip 6,200.00 6,200.00 0.00 0.00% 6,200.00 0.00 0.00% Multicultural Centre Prilep 9,300.00 9,300.00 0.00 0.00% 9,300.00 0.00 0.00% 03:09 Additional financial resources Operating costs Fuel, insurance and maintenance 18,576.00 18,276.00 -300.00 -1.61% 20,335.00 2,059.00 10.13% 03:10 Monitoring and evaluation missions 27,110.00 27,110.00 0.00 0.00% 21,030.00 -6,080.00 -28.91% Monitoring visits 27,110.00 27,110.00 0.00 0.00% 21,030.00 -6,080.00 -28.91% SUB-TOTAL 3 (items from 3.01 to 3.10) 1,276,427.00 1,292,834.00 16,707.00 1,290,989.00 SUB-TOTAL (1,2,3.01-3.10) 1,415,613.00 1,415,613.00 16,707.00 1,408,929.00 Study of the project (3% of the sub total of the 3 03:11 years) 42,468.00 42,468.00 0.00 0.00% 42,468.00 0.00 0.00% 03:12 Contingencies (10% of sub-total) 03:13 General and administrative expenses Organizational structure in Italy (6% of sub-total 3) 76,586.00 76,586.00 0.00 0.00% 76,586.00 0.00 0.00%

35

Evaluation-Macedonia, Cities of Stip and Prilep

Organizational structure in country (2% of sub-total 3) 25,529.00 25,529.00 0.00 0.00% 25,529.00 0.00 0.00% TOTAL from 3.1 to 3.13 1,421,010.00 1,437,417.00 1,435,572.00 -1,845.00 TOTAL (1,2,3) 1,560,196.00 1,560,196.00 1,553,511.00 -6,685.00 Surplus 2nd year (amount deducted for expenses not accounted for) 3,246.00 Actual gross expenditure reported 1,550,265.00 -9,931.00

36

Evaluation-Macedonia, Cities of Stip and Prilep

2B To what extent were the inputs converted into results?

2C Was the alternative utilized the most efficient?

PILLAR A. IMPROVEMENT IN COMMUNITY HYGIENE

The actions undertaken to improve community hygiene in the target areas in Stip were characterized by the creation of a structure with bathrooms, showers and laundry facilities for the local population. In the analysis of the costs of the construction of the facilities, it should be highlighted that there was significant modification to the original financial plan, resulting in a sharp increase in spending for line item 03:07 sub- heading “Building hygiene facilities in Stip – construction”, equal to 37.24% (for a value of € 19,321.00). In addition, in the process of reporting, there were significant changes (both increases and decreases) in all items of expenditure.

The comparison between the costs incurred for the construction and proceeds on the part of the beneficiaries for utilizing the hygiene and sanitation facilities suggests that, although the costs to construct the facility for the beneficiaries may seem acceptable, the construction costs were much higher than the actual revenues. As evident in the analysis of the effectiveness and sustainability of the project, the estimated revenues amounted to € 22,875.64 per year, with approximately € 3,500.00 earned. This suggests the cost incurred per beneficiary was high in terms of results obtained.

The ET has not been able to determine, in the absence of any similar actions, whether the alternative used for the construction of the hygiene and sanitation facilities was the most efficient.

Tab. 6 COST ANALYSIS “SANITATION AND HYGIENE FACILITIES STIP” in euro Difference between Total Total cost expected costs after expected after % Actual variance and actual % difference in Item of expenditure cost variance Variance variance expenditure expenditure expenditure 03:06 Faci lity Washing machines for building hyg. serv. Stip 2,850.00 2,850.00 0.00 0.00% 1,661.00 -1,189.00 -71.58% Boiler, radiators, building counters for hyg. serv. Stip 3,615.00 3,615.00 0.00 0.00% 6,584.00 2,969.00 45.09% 03:07 Construction Bui lding hygiene facilities in Stip - design 723.00 723.00 0.00 0.00% 2,000.00 1,277.00 63.85% Building hygiene facilities in Stip - construction 51,645.00 70,876.00 19,231.00 37.24% 78,239.00 7,363.00 9.41% Building hygiene facilities in Stip - toilets, sinks, showers, sinks 7,746.00 7,746.00 0.00 0.00% 1,053.00 -6,693.00 -635.61% Building toilets in Stip - operating costs 3,248.00 3,248.00 0.00 0.00% 3,208.00 -40.00 -1.25% Land (public land use or 03:08 private expropriation) Building hygiene facilities in Stip 3,720.00 3,720.00 0.00 0.00% 4,278.00 558.00 13.04% Total 73 ,547 .00 92 ,778 .00 19 ,231 .00 97 ,023 .00 4,245 .00

37

Evaluation-Macedonia, Cities of Stip and Prilep

Tab. 7 AVERAGE COST PER BENEFICIARY “ SANITATION AND HYGIENE FACILITIES STIP” in euro

Activated 1st 2nd 3rd Total revenues for Total costs for Average cost per service year year (9 months) year (10 months) service provided activated service beneficiary Shower - 2,987 .00 2,066 .00 5,053 .00 95 ,362 .00 18 .87 Washing machine - 913.00 809.00 1,722.00 97,023.00 56.34

Other actions taken by the project intervention include : i) construction of the water adduction/distribution network in Stip; ii) cleaning of drainage canal in Prilep; and iii) the pavement of the road between Trizla 1 and Trizla 2 in Prilep .

The cost analysis (Table 8) shows for the first action, there was a reduction in the variance in the submission phase and a substantial increase, nearly reaching the original figure, in the accounting phase. For the second action (cleaning of the drainage canal), one of the most obvious deviations from the initial budget for the entire project is evident, with an increase by 86.02% (from € 4,647.00 to € 33,232.00 in expenditure). Finally, for the third action, there was a considerable increase in the actual expenditure as compared to the budgeted cost (28.83%).

However, these last complex interventions included the utilization of significant human and technical resources as a result of problems encountered in the field during the implementation phase – see the effectiveness section for further details. It is believed that the increase in costs can be considered acceptable given the large number of beneficiaries reached through the project activities: i) more than 100 domestic/household connections to water adduction/distribution network and a greater number of beneficiaries of the hygiene and sanitation facilities in Stip; and ii) the population of the districts of Trizla 1 and Trizla 2 in Prilep (about 12,000 people).

During the field mission, the ET interviewed representatives of the government agencies responsible for public works, who confirmed that the costs incurred for the construction of the facilities outlined in Table 8 represent the average.

Tab. 8 COST ANALYSIS “WATER ADDUCTION NETWORK STIP – CANAL CLEANING PRILEP – PAVEMENT OF ROAD PRILEP” in euro Difference in estimated costs after variance Total Total cost Actual and actual % difference Item of Expenditure expected cost after variance Variance % variance expenditure expenditure in expenditure 03:06 Facility Water adduction/distribution network Stip 77,500.00 68,112.00 -9,388.00 -12.11% 74,524.00 6,412.00 8.60% Cleaning of drainage canal - Prilep 4,647.00 4,647.00 0.00 0.00% 33,232.00 28,585.00 86.02% Pavement of road - Prilep 36,152.00 36,152.00 0.00 0.00% 50,798.00 14,646.00 28.83% Total 118,299.00 108,911.00 -9,388.00 158,554.00

38

Evaluation-Macedonia, Cities of Stip and Prilep

PILLAR B. INCREASE IN SCHOOLING

This pillar includes three main actions: i) the distribution of clothing and school supplies; ii) the development and implementation of extra-curricular activities; and ii) support for school operations through the payment of expenses for electricity and classroom heating. The action for which the budget increase is most obvious is that of extra-curricular activities: increase in 45.05% (from € 15,492.00 in the estimated budget to € 28,194.00 in actual expenditure).

Tab. 9 COST ANALYSIS “ INCREASE IN SCHOOLING” in euro Difference in costs after Total Total costs variance and expected after Actual actual % difference Item of expenditure costs variance Variance % variance expenditure expenditure in expenditure 03:04 Increase in schooling Clothing and school supplies for students 101,600.00 98,000.00 -3,600.00 -3.54% 96,006.00 -1,994.00 -2.08% Extracurricular Activities 15,492.00 15,492.00 0.00 0.00% 28,194.00 12,702.00 45.05% Heating and electricity for classrooms 9,294.00 9,294.00 0.00 0.00% 6,895.00 -2,399.00 -34.79% Total 126 ,386 .00 122 ,786 .00 -3,600 .00 131 ,095 .00 8,309 .00

Considering that the aim of the project was to increase primary school and education, it is interesting to investigate whether the average cost incurred by the project for each child entering school is in line with the average national costs. This comparison is only possible in light of data available on the subject, albeit partially. In fact, the InterSOS activities on this aspect of the project are unique in Macedonia and there are no similar projects for comparison.

Table 10 illustrates that the project spent an average of € 102.00 per child each year, considering the total cost for specific activities implemented to increase schooling. The project instead spent € 209.00 per child, considering costs associated project management (coordination, secretarial, administrators, etc.). It should be noted that this calculation does not reflect the costs of teachers' salaries, which were paid by the Ministry of Education and Science.

Tab. 10 AVERAGE COST PER BENEFICIARY “PARALLEL CLASSES AND INSERTION INTO REGULAR CLASSES” in euro Area 2004 -2005 2005 -2006 2006 -2007 Average cost per Children STIP 99 74 61 beneficiary (euro) – activities for Average cost per increased beneficiary (euro) – schooling and Total activities for project Children PRILEP 31 90 74 beneficiaries increased schooling management costs Children TOTAL 130 164 135 429 306 626 Costs per year per child 102 209

The following figures illustrate the average cost of schooling for children in Macedonia for primary and secondary school from 1998-2003 60 , as well as the data collected by the ET during the field mission at the school in Stip where parallel education classes were implemented and insertion into regular classes was promoted, relative to the costs incurred by the school for each child in the 2013-2014 school year.

60 Open Society Institute, Equal Access to Quality Education for Roma. Volume 2 - Croatia Macedonia Montenegro Slovakia , 2007, p. 250. 39

Evaluation-Macedonia, Cities of Stip and Prilep

Tab. 11 COSTS PER CHILD IN PRIMARY AND SECONDARY SCHOOL (1998 - 2003) Cost per child Primary Education Secondary Education Year MKD EURO MKD EURO Change 1998 1,7622.00 289.13 NA 0.01641

1999 1,7154.00 292.45 1,9270.00 328.52 0.01705 2000 1,7935.00 297.79 1,9753.00 327.98 0.01660 2001 1,7185.00 285.74 1,7025.00 283.08 0.01663 2002 2,0069.00 333.09 2,0914.00 347.11 0.01660 2003 NA 2,1319.00 353.13 0.01656

Tab. 12 COSTS PER CHILD AT DIMITAR VLAHOV PRIMARY SCHOOL – 2013 School Dimitar Vlahov Total costs for school year Number of 2013 – 2014 Cost per child students MKD EURO MKD EURO Change 1387 41,166,795.00 668,313.44 29,680.46 481.84 0.0162

Based on the national costs related to primary education for children, just over €300 in 2003 and €500 in 2013, the average cost incurred by the project, amounting to €209.00 (including the operating expenses of the project) indicates the decision to support the inclusion of children in school is in line with the corresponding national expenditure, bearing in mind that the project did not incur costs associated with teachers’ salaries.

PILLAR C. ORGANIZED MANAGEMENT OF ACTIVITIES AND CULTURAL EXCHANGES

Pillar C was been pivotal in that it included the construction of two multicultural centres (MCCs) in Stip and Prilep; and supported a series of activities to facilitate the launching, management and promotion of multicultural activities within the MCCs and in the surrounding community.

The cost verification for Pillar C activities highlights obvious differences between the expected costs and those actually incurred. In particular, the item of expenditure for “Entertainment/Organizers Stip” (03:03) consists of a reduction of 23.75% (amounting to €34,375.00) while the expenditure for "Construction of the Multicultural Centre Stip” (03:07) is characterized by a twofold increase, 16.72% in the variance phase and 15.71% in the reporting phase, attributable to the change in construction costs between the initial submission of the project (2001) and the beginning of work in Stip (2005). Likewise, for the “Building Multicultural Centre of Prilep” (03:07), there was a decrease of 72.83% in the reporting phase, attributable to the fact that the activity in Prilep included the renovation of a building that already existed, rather than construction from scratch.

The budget spent for this Pillar consists of more than one third of the total project costs; the Pillar has affected thousands of beneficiaries . Given that the intervention is the only one of its kind in Macedonia, the ET is unable to determine, based on the available data, the extent to which the inputs were converted into results, and if the alternative used was the most efficient. However, for complete information, please refer to detailed description of the expected budget and incurred costs for this Pillar.

40

Evaluation-Macedonia, Cities of Stip and Prilep

Tab. 13 COST ANALYSIS “MULTICULTURAL CENTRES” in euro Difference in total expected costs after Total Total costs variance % difference expected after Actual and actual in Item of Expenditure costs variance Variance % variance expenditure expenditure expenditure 03:02 Other Italian personnel Experts in community dynamics 27,110.00 38,266.00 11,156.00 41.15% 40,472.00 2,206.00 5.45% 03:03 Local staff

Expert dialogue and network 7,506.00 7,506.00 new voice 6,570.00 -936.00 -14.25% Entertainers/ Organizers Stip 179,112.00 179,112.00 0.00 0.00% 144,737.00 -34,375.00 -23.75% Entertainers / Organizers Prilep 179,112.00 179,112.00 0.00 0.00% 176,345.00 -2,767.00 -1.57% 03:05 Multicultural Centre Activities Preliminary meetings with the population 2,580.00 2,580.00 0.00 0.00% 4,290.00 1,710.00 39.86% Costs of running the Multicultural Centre Stip 4,644.00 4,644.00 0.00 0.00% 4,366.00 -278.00 -6.37% Costs of running the Multicultural Centre Prilep 1,548.00 1,548.00 0.00 0.00% 3,438.00 1,890.00 54.97% Cultural Activities 12,380.00 12,380.00 0.00 0.00% 14,512.00 2,132.00 14.69% Afterschool - teachers 10,368.00 10,368.00 0.00 0.00% 10,316.00 -52.00 -0.50% Afterschool - teaching materials 3,098.00 3,098.00 0.00 0.00% 4,265.00 1,167.00 27.36% Health and hygiene education teachers 5,184.00 5,184.00 0.00 0.00% 2,700.00 -2,484.00 -92.00% Education sanitary-hygiene kits 9,600.00 9,600.00 0.00 0.00% 7,131.00 -2,469.00 -34.62% Literacy - adult teachers 5,184.00 5,184.00 0.00 0.00% 3,120.00 -2,064.00 -66.15% Literacy - Adults didactic materials 2,460.00 2,460.00 0.00 0.00% 1,334.00 -1,126.00 -84.41% Conflict management - teachers 5,202.00 5,202.00 0.00 0.00% 2,160.00 -3,042.00 -140.83% Dynamics of conflict management - didactic materials 1,440.00 1,440.00 0.00 0.00% 571.00 -869.00 -152.30% Management course teachers 3,456.00 2,728.00 -728.00 -21.06% 800.00 -1,928.00 -241.00% Management course – didactic materials 390.00 390.00 0.00 0.00% 19.00 -371.00 -1,908.24% Management course – comp. Printers 3,564.00 2,600.00 -964.00 -27.05% 2,600.00 0.00 0.00% 03:06 Facility Equipment for the two MCCs 15,492.00 10,805.00 -4,687.00 -30.25% 18,364.00 7,559.00 41.16% 03:07 Construction Design of multicultural centres 2,582.00 2,582.00 0.00 0.00% 4,091.00 1,509.00 36.89% Construction of the Multicultural Centre in Stip 72,200.00 84,275.00 12,075.00 16.72% 99,984.00 15,709.00 15.71% Construction of the Multicultural Centre in Prilep 108,300.00 103,900.00 -4,400.00 -4.06% 60,116.00 -43,784.00 -72.83% Land (use of public land or private 03:08 expropriation) Multicultural Centre Stip 6,200.00 6,200.00 0.00 0.00% 6,200.00 0.00 0.00% Multicultural Centre Prilep 9,300.00 9,300.00 0.00 0.00% 9,300.00 0.00 0.00% Total 670,506.00 690,464.00 19,958.00 627,800.00 -62,664.00

41

Evaluation-Macedonia, Cities of Stip and Prilep

2D Were the results achieved within the anticipated timeframe?

In this section, a detailed analysis of the timing of project implementation is presented in order to verify whether the results were achieved in accordance with the anticipated schedule. A reproduction of the original timetable and the actual implementation timetable was developed (Table 14) in order provide an analytical framework to identify any deviations or changes from the initial programming. In Table 14, the original project workplan (indicated by “X”) and the actual timing of activities (highlighted in “grey”) have been combined in order to visualize the correspondence between the planned schedule of activities and actual implementation.

The analysis of the timing of the project highlights that implementation was maintained for three years, according with the overall duration of the project, as planned. At the same time, it is also evident that the implementation of individual activities substantially diverged from the original workplan developed for each grant disbursement phase.

Tab. 14 ACTIVITY WORKPLAN 61 Activity 1st year

Jun-04 Jul-04 Aug-04 Sep-04 Oct-04 Nov-04 Dec-04 Jan-05 Feb-05 Mar-05 Apr-05 May-05 A1 Construction of the water adduction network-Stip 62 A2 Sanitation/hygiene facilities in Stip A3 Cleaning of the canal in Prilep A4 Paving of the road in Prilep B1 Insertion in parallel and regular education classes 63 B1 Distribution of school supplies and clothing B2 Contribution to school operating costs B3 Extra-curricular activities C1 Meeting with Roma communities and their leaders C2 Creation of neighbourhood associations C3 Construction MCCs C4 Cultural activities C5 Afterschool activities C6 Health/san. courses for women C7 Literacy course of adults C8 Course in management of conflict dynamics C9 Course in admin. and management of MCC

61 The diagram shows the reconstruction of the original timetable according to expenditure items, combined with activities specified in the original project document. 62 The reports do not mention the period of realization. 63 This specific activity is not included in the original timetable. 42

Evaluation-Macedonia, Cities of Stip and Prilep

Activity 2nd year

Jun-05 Jul-05 Aug-05 Sep-05 Oct-05 Nov-05 Dec-05 Jan-06 Feb-06 Mar-06 Apr-06 May-06 A1 Construction of the water adduction/distribution network- Stip A2 Sanitation/hygiene facilities in Stip A3 Cleaning of the canal in Prilep A4 Paving of the road in Prilep B1 Insertion in parallel and regular education classes B1 Distribution of school supplies and clothing B2 Contribution to school operating costs B3 Extra-curricular activities C1 Meeting with Roma communities and their leaders C2 Creation of neighbourhood associations C3 Construction MCCs C4 Cultural activities C5 Afterschool activities C6 Health/san. courses for women C7 Literacy course of adults C8 Course in management of conflict dynamics C9 Course in admin. and management of MCC

43

Evaluation-Macedonia, Cities of Stip and Prilep

Activity 3rd year

Jun-06 Jul-06 Aug-06 Sep-06 Oct-06 Nov-06 Dec-06 Jan-07 Feb-07 Mar-07 Apr-07 May-07 Jun-07 A1 Construction of water adduction/distribution network-Stip A2 Sanitation/hygiene facilities in Stip A3 Cleaning of the canal in Prilep A4 Paving of the road in Prilep B1 Insertion in parallel and regular education classes B1 Distribution of school supplies and clothing B2 Contribution to school operating costs B3 Extra-curricular activities C1 Meeting with Roma communities and their leaders C2 Creation of neighbourhood associations C3 Construction MCCs C4 Cultural activities C5 Afterschool activities C6 Health/san. courses for women C7 Literacy course of adults C8 Course in management of conflict dynamics C9 Course in admin. and management of MCC

44

Evaluation-Macedonia, Cities of Stip and Prilep

Effectiveness

3. Is the project design valid?

3A Is there a logical and consistent connection between inputs, activities, outputs, outcomes and impact?

To verify the validity of the project design, the Sant’Anna School evaluation team used the Theory of Change technique, “an innovative tool to design and evaluate social programmes with an ultimate goal to generate change in the context in which they operate”. 64 The Theory of Change methodology can also be viewed as a representation of how an organization or initiative aims to achieve its desired results, indicating the logical sequence of the initiative according to the reasoning that “if y occurs, then x should occur”. In constructing the Theory of Change for the project “Macedonia, the cities of Stip and Prilep: Improving the quality of life of Roma and promoting integration”, the ET analysed the project documentation and, in particular, the 2003 expert report from the CTU and the logical framework included therein.

Annex 2 of the report presents a graphical representation of the intervention logic as reconstructed by the ET.

The structure of the Theory of Change follows that of the planned project activities, i.e. the three main pillars: Pillar A. Improvement in community hygiene Pillar B. Increase in schooling Pillar C: Organized management of activities and cultural exchanges.

For each pillar, the logical steps were analysed, beginning with the input level up until the impact level. This reconstruction revealed the project, as was thought during the planning stage, was consistent. Not only does the project demonstrate an internal logic, but many links exist uniting the three Pillars to achieve a common objective. In particular, the activities of the two multicultural centres (Pillar C) were carried out in line with activities to support increased education and schooling (Pillar B), for example through the development of extra-curricular activities, and with hygiene and sanitation activities (Pillar A), through health and sanitation courses targeting Roma women. Similarly, the education pillar (Pillar B) was designed to promote the integration of Roma and non-Roma children in attending the primary schools targeted by the project, supporting the general objective to strengthen integration of Roma communities (Pillar C).

From the analysis conducted, the only criticism in the project design relates to the paving of the road separating the two districts of Trizla 1 and Trizla 2 in the city of Prilep, and activities which, according to the project’s logical framework, aimed at improving community health. The progress report from the first year of implementation notes the road was “subject to a continuous accumulation of rubbish and organic material as a result of the absence of a sewage system. In addition, the collection tank/cesspool at the front of the road, was not functioning as it was repeatedly blocked by rubbish”. The activity to pave the road does not appear directly related to the improvement of health and sanitary conditions in the

64 ActKnowledge e Aspen Institute, Theory of Change. Roundtable on Community Change , 2003. https://www.theoryofchange.org 45

Evaluation-Macedonia, Cities of Stip and Prilep neighbourhood, but rather to the activity to clean the drainage canal due to the presence stagnant wastewater in the street. As such, the ET interpreted the asphalting of the road as a preparatory activity to improve the image of the surrounding environment, or as an incentive for the community to take greater care in reducing the accumulation of solid waste. In this context, the activity to pave the road appears to be more complementary with other project activities, such as cleaning the draining canal, which was achieved with the support of the local population, the establishment of a Citizens’ Committee for the Protection of Dabnichka and an educational health and hygiene campaign to raise awareness on the impacts on the environmental and personal health caused by the stagnation of wastewater. For these reasons, the ET added the following element between the impacts of the initiative, which was not included in the original logical framework: “the target population contributes to the care of the neighbourhood environment in Prilep in addition to other project activities” as a logical consequence of the awareness and involvement the population in the care of the health and sanitation facilities in the district, which is a crucial intervention. Likewise, the ET included more detailed outcomes for Pillar A, specifically related to the cleaning of the drainage canal in Prilep. As evident in the document prepared by the CTU in 2003, “it is clear that the realization of this task is relatively simple and connected with the progressive increase in the collective consciousness of damage to the environment and the health of citizens caused by the stagnation of waste water”. This awareness, in the opinion of the ET, needs to translate into the continuous participation of the population in the cleaning of the canal. Thus, in addition to the achievement of more immediate goals such as “improving the flow of wastewater into the canal” and “constant cleaning of the canal”, the ET has added a further outcome as follows: “the population participates in the cleaning of the canal”.

Also in relation to Pillar A, community hygiene and sanitation, it was decided to distinguish between the impacts resulting from the strategy implemented in Stip and that implemented Prilep. Given that in Stip, the project activated a strategy to improve personal hygiene, while in Prilep, the project focused on the improvement of environmental conditions and related health issues, the ET considered it appropriate to distinguish between the two impacts. For the city of Stip, the impact should reflect a “reduction in the % of diseases related to poor personal hygiene”, while in Prilep, the impact should reflect a "% reduction of diseases related to environmental pollution”. These elements were included among the impacts of the project and not - as indicated in original logical framework - as outcomes, by virtue of the time required to induce such changes.

In addition, the ET, in the reconstruction of the intervention noted the absence of appropriate targets to quantify the extent to which the project achieved its objectives. This absence could have been overcome if it had been possible to consult and compare with those who prepared the original project document and contributed to its realization. For the reasons mentioned above, related to the amount of time which has passed since project implementation, a comparison of this nature this was not possible. To overcome this difficulty, the ET referred, where possible, to targets at the national level provided by the competent ministries in Macedonia and/or by relevant international organizations.

46

Evaluation-Macedonia, Cities of Stip and Prilep

4. To what extent were the project objectives met?

4A What percentage of the population has access to and uses the hygiene/sanitation facilities?

During the first year of the project, Stip proceeded with the construction of the water adduction/distribution network (Activity A1 ) and the construction of community hygiene and sanitation facilities (Activity A2 ).

The activity to build a new water adduction/distribution network to replace the previous one resulted in the: i) construction of a pumping station (two water-pressure pumps with a capacity of 10 litres/second and the possible accommodation for a third additional pump); ii) laying of 355 metres of connecting pipeline for the water adduction/distribution network (a prior intervention was made by the Municipality of Stip to boost the water supply in Radanski Pat District); iii ) creation of an electric power network to operate the water system; iv) connection of the water system to the new community hygiene and sanitation facilities. The new water adduction/distribution network not only allowed for the community sanitation facilities to be connected to the water network, but also more than one hundred Roma and Macedonian families in Radanski Pat. The number of families, legally or illegally (i.e. having no regular contract with the municipal water management company), provided with water through the pumping station built by the project has grown over the years and now includes a total of 500 families.

The construction of the community hygiene and sanitation facilities, built on land owned by the local partner – Cerenja – and made available to the project (116.21 square metres) was equipped as follows: five bathrooms (two for women and two for men), 14 sinks (seven for women and seven for men), 12 showers (six for women and six for men), anti-lice shampoo, six washing machines, a room equipped with an oil heater, and a reception area. Compared to the original design, the construction of the sanitation facilities changed in terms of total square footage and with respect to the subdivision/composition of spaces within the building, as authorized by the MFA-DGCD. The construction of community hygiene services, although contracted to an outside firm, utilized local Roma labour.

The hygiene and sanitation facilities were operational starting from the second year of the project, for a total of 10 months (August 2005 to May 2006). During the third year, the services were only operational periodically: due to the low level of utilization in the summer months 65 , the facilities were closed temporarily, reopening in September 2006 and operational until May 2007. The following table presents data on the use of the facilities, in particular the number of times the showers and washing machines were utilized by beneficiaries during the months of operation. The table also shows the annual revenues from the use of facilities. It should be noted that the cost incurred for taking showers and for using washing machines also included use of the bathrooms.

65 In the summer months, given the climate, the population does their laundry outdoors and is used to taking open-air showers. In August 2005, the number of showers taken in the community hygiene facilities was 33, as compared with a monthly average of 298, and the number of times the washing machines were used was one, compared with a monthly average of 91.3 (data from the second year, 15 June 2005-14 June 2006). 47

Evaluation-Macedonia, Cities of Stip and Prilep

Tab. 15 USE OF THE “HYGIENE SERVICES” AND REVENUE RECORDED 2nd YEAR (10 MONTHS) 3rd YEAR (9 MONTHS)

NUMBER OF SHOWERS 2,987 2,066

TOTAL SHOWERS- ADULTS 2,470 1,479

TOTAL SHOWERS – CHILDREN 517 587

MONTHLY AVERAGE 298,7 229,5

REVENUE FROM SHOWERS Cost per shower- adults € 0.81 (50 dinar) Cost per shower - children € 0.48 (30 dinar) € 2,248.76 € 1,479.75

NUMBER OF WASHING MACHINE USES 913 809

MONTHLY AVERAGE 91.3 89.9

REVENUE FROM WASHING MACHINES € 739.53 € 655.29 Cost for one wash € 0.81 (50 dinar)

TOTAL REVENUE € 2,985.29 € 2,135.04

In the project document, it was estimated that at least 100 families (500 people, five people per family) per day would use the hygiene services and that the washing machines would be used at least 24 times per day. These average monthly estimates were significantly higher than those actually recorded by the project during the two years the facilities were operational (19 months). In fact, the project recorded an actual monthly average of 298.7 showers and 91.3 washing machine uses in the second year and 229.5 showers and 89.9 washes in the third year. Thus, in the project document, it was anticipated that 15,000 people per month would use the hygiene and sanitation facilities. In fact, during the operation period of the services (19 months), the monthly rate was 266 people, or 1.77% of the estimated population. This also affected the revenues the project expected to obtain from the facilities in order to support their self- sustainability . It should be noted that the figures listed in the project document include the number of times the services were used and the number of people who had access to the services. This implies that the 266 recorded uses were not necessarily made by 266 different people. It is also worth mentioning that during the project, the Municipality of Stip paved the previously dirt access road to the sanitation facilities and to the multicultural centre.

In conclusion, with respect to degree of effectiveness achieved by the hygiene and sanitation facilities established by InterSOS in Stip, the ET notes that although the services were built and utilized, the percentage of the population which accessed these sanitation services on a daily basis (0.2%) is much lower than initially estimated (13%)

4B Is the drainage canal consistently clean?

As anticipated, during the three years of the project, the drainage canal (450 metres) in the district of Trizla 1 in Prilep has been consistently clean (Activity A3 ). The project intervention conducted three cleanings, on per year , as described below. The Roma community participated in the cleaning each time, although to a lesser degree than initially hoped .

48

Evaluation-Macedonia, Cities of Stip and Prilep

The first cleaning commenced in November 2004 and was conducted by the Municipality of Prilep, thanks to funding from the Ministry of Ecology to clear the canal (“Dabnichka”) extending from the centre of Trizla 2 district, for a length of 1,800 metres. This operation was key to limit the risk of epidemics and flooding caused by the accumulation of rubbish in the canal. With regard to the section of the canal extending to the neighbourhood of Trizla 1, as outlined in the design phase project, the cleaning was conducted by InterSOS and began in January 2005. It was completed in April 2005 as a result of bad weather. The mechanized and manual cleaning removed a pile of rubbish equal to 1,460 cubic metres, distributed across the 450 metre length of the canal.

In the second year of the project, the cleaning took place from 7-9 June 2006. Those actively contributing to the cleaning, mostly through the manual collection of rubbish, included three members of the Roma Citizens’ Committee and four members of the company responsible for the collection of municipal waste “Komunalec”, which provided tools and trucks to transport the rubbish to the municipal landfill.

In the third year of the project, the cleaning took place in September 2006, in the same manner as before. The final cleaning operation was largely conducted by the local partner, AHP, which had also participated in previous cleanings, providing labour and involving about 20 Roma volunteers, as evident in the costs incurred by AHP in the third year, to support means of transport for volunteers.

During all three years of implementation, the canal was cleaned regularly, covering a distance greater than initially estimated, 450 metres rather than 200 metres.

4C Did the population participate in the cleaning of the canal?

During the first year of the project, it was decided to set up a Citizens’ Committee for the Protection of Dabnichka , consisting of five permanent members and supported by several volunteers. The project provided necessary equipment to the Committee, including wheelbarrows, shovels, rakes, gloves, hoes, rubber boots, bags and nets. Moreover, in the course of one year, 10 meetings on health and hygiene education were conducted with an aim to increase awareness on environmental and health risks caused by the stagnation of wastewater. The awareness campaign was preceded by a study, carried out through individual interviews, on the sanitary conditions of the population.

Three members of the Citizens’ Committee for the Protection of Dabnichka participated in the second and third cleaning of the drainage canal as did members from the municipality’s public waste collection company, Komunalec, which provided trucks to transport the rubbish.

Overall, the project did involve the Roma population from Trizla 1 in the cleaning activities, although to a lesser extent than anticipated, despite the awareness campaign conducted by the project during the first year. The project anticipated the involvement of 60 persons during the three years (20 per year), while only 29 persons actually participated (three in the first year; three in the second year; and 23 in the third year) 66 .

66 Three people who participate every year are the same, namely those belonging to the Citizens’ Committee. 49

Evaluation-Macedonia, Cities of Stip and Prilep

4D Has the presence of the paved road improved the image of the neighbourhood in terms of health and cleanliness?

4E Has paving the road facilitated movements between Trizla 1 and Trizla 2?

The paving of the central access road to the neighbourhoods of Trizla 1 and Trizla 2 (Activity A4 ) was completed in the third year of the project (October 2006). This activity was entirely supported the Municipality of Prilep, as planned .

The road should have been paved in the first year of the project; however, issues arose during the start-up phase of the initiative: the continuous accumulation of rubbish and organic matter due to the absence of a sewage system and the malfunctioning of the collection tank/cesspool at front of the street, delayed the completion of this activity until the third year. In fact, the Municipality had to intervene prior to the construction of the sewage system as wastewater was draining into the street, in a neighbourhood that, until the completion of the project, was not included in the master plan of the city.

The construction of the sewer system and the paving of the road undoubtedly improved the image of the neighbourhood in terms of health and cleanliness. It also facilitated movement between neighbourhoods Trizla 1 and Trizla 2 in Prilep. The paved road, as the ET observed in the course of the field mission, is the main access route between the two areas, which account for a large population. As such, the transition from a dirt road to the paved road has made circulation much easier.

4F To what extent there has been an increase in school attendance among children targeted by the project?

Pillar B of the project aimed to achieve two outcomes: i) in Stip and Prilep, 800 children been re/inserted in the primary school curriculum and attend class regularly; and ii) 100 children receive supplementary support.

These results are logically linked to the achievement of the specific objective to support an increase in schooling through a reduction in early dropout rates. The strategy employed to achieve these results and the specific objective included the re/insertion of 800 Roma children identified by the local partners (Cerenja and AHP) in the 2001/2002 school year. The re/insertion was encouraged through the distribution of clothing, books and other school supplies to those who regularly attended primary school. Three separate distributions were envisioned: the first at the beginning of the school year, the second at mid- term, and the third at the end of the year, in order for children to use the materials the following school year. The distributions were to be organized by the multicultural centres. School activities were to be carried out in six primary schools as follows: one school in Prilep and five schools Stip. The organization of the new classes and the inclusion of children in existing classes required the assistance and support of principals from the schools targeted by the intervention. Teachers already working in the selected schools were involved in the lessons. In addition, the project provided support for the electricity and heating costs for classrooms used in the context of the project and for the organization of extra-curricular activities .

Finally, activities organized by the multicultural centres in Stip and Prilep, included an after school programme for 100 Roma and non-Roma students with learning difficulties who, with the support of two teachers, a psychologist and a social worker, were able to benefit from additional educational support at least two- three times per week . 50

Evaluation-Macedonia, Cities of Stip and Prilep

As the start of the project was delayed for three years, InterSOS, in collaboration with local partners, conducted a new survey of Roma children needing additional assistance. In 2004, the number of children between 7 and 14 years of age who had never attended primary school was 400 (150 Stip, 250 Prilep). The Macedonian national law states that children over 8.5 years of age who have never attended school can no longer be enrolled in the public school system; therefore, these children are unable to complete primary education unless they attend private schools - which are expensive and limited - or, unless such children re- enter the system after the age of 15 and undertake study courses for adults. As such, the InterSOS project strategy included the piloting of parallel education classes in two of the primary schools targeted by the project (Dobre Jovanoski in Prilep and Dimitar Vlahov in Stip). The parallel classes are a means by which children excluded from the mainstream education system can attend public school from 1st to 3rd grade and then be reinserted into the ordinary school curriculum. This pilot/trial approach was approved by the Ministry of Education and Science.

Of the six primary schools involved, the project established parallel classes in only two, one in Stip and one in Prilep, as no other spaces were available to ensure the classes were conducted properly. However, the school supplies and clothing distribution activities were realized in all six schools.

With regard to the timing of the implementation of activities, is important to remember that, while the establishment of parallel classes took place in Stip from the first year of the project (2004/2005 school year), in Prilep, the parallel classes commenced in the second year of the project (2005/2006 school year). This shift meant that school children in Prilep continued parallel classes during the third year of the project, as the programme runs from 2nd to 3rd grade, with the approval of the Ministry of Education ( Activity B1 )

Finally, during the second year of the project, UNICEF also established parallel classes in the primary school in Stip, involving 25 children. However, during the project, there were no major synergies between the two operations, except for the distribution of clothing and school supplies which also benefited children involved in the UNICEF programme. To this end, it must be noted that a letter of support had been received from UNICEF at the time of the initial submission and presentation of the project.

Following approval from and with the support of the Macedonian Ministry of Education, teachers involved in the parallel classes included five persons: three in Stip and two in Prilep.

The table below illustrates relative data from the initial project document (2001) and from the 2004 census, as well as the total number of children benefiting from inclusion school through the parallel classes (children over 8.5 years of age) and regular classes (with children below 8.5 years of age).

51

Evaluation-Macedonia, Cities of Stip and Prilep

Tab. 16 EDUCATIONAL ACTIVITY DATA 2004 - 2007 Project document 2001 800 Census 2004 7-14 years 400 Total child beneficiaries 245 Total child beneficiaries 8,5 -14 years 106 Total child beneficiaries 7-8,5 139 1st project year 8,5-14 years 63 1st project year 7-8,5 years 67 2nd project year 8,5-14 years 35 67 2nd project year 7-8,5 years 80 3rd project year 8,5-14 years 0 3rd project year 7-8,5 years 0

In addition to those listed above, 60 children , aged six years, participate in “year zero”, in the last year of the project (2006-2007). “Year zero”, as already noted in the relevance section of the report, is a preparatory year of schooling which children must attend prior to primary school, as required by the state since 2005. In terms of the achievement of project objectives, as mentioned in Section 3 on the validity of project design, the project did not include quantifiable targets for success. For these reasons, the ET proposed to utilize, as a benchmark, the target indicated in to the Decade of Roma strategy, to which the project repeatedly refers. In 2004, the target indicated was an increase the schooling of Roma children of appropriate age by up to 90% .68 Based on the data from the 2004 census carried out by InterSOS, in collaboration with local partners, Cerenja and AHP, which included 400 Roma school-aged children in the school system, the target would be 360 children. Of the 400 children surveyed, the project reached 245, in addition to 60 children who were not included 2004 census. Thus, the success rate of the project in terms of increasing the schooling of Roma children from 2004-2007 was 61.25%.

With regard to the distribution of clothing and school supplies , the project completed five distributions in the first year and four in the second year; in the third year three distributions took place in Prilep and five in Stip (Activity B1 ). The distributions reached about 5,800 children; however, the precise number of children benefiting from this activity cannot be determined as each child participated in various distributions. Moreover, in contrast to the initial plan, the distributions aimed to benefit not only Roma children in the target schools, but also non-Roma children. Although this deviates from the initial project design, the ET considers this to be a positive approach. The decision not to direct distributions exclusively to Roma children was determined by the need to ensure non-discriminatory practices among children in the primary schools, to support non-Roma families in disadvantaged economic conditions and, finally, to avoid exacerbating ethnic differences among these communities in Macedonia. Table 17 illustrates the aggregated data on the distribution of clothing and school supplies to Roma children in Macedonia. It should be noted that disaggregated data are not available for this activity as they were not collected during the implementation of the project.

67 During the second year, 43 children were included in the parallel classes in the primary school in Prilep. Of these 43, during the school year, eight were placed in regular classes. For these reasons, the eight children placed in regular classes were included in the calculation of children between seven and eight and a half years. 68 Ministry of Labour and Social Policy - Republic of Macedonia, Strategy for Roma in the Republic of Macedonia , 2004, p. 43. 52

Evaluation-Macedonia, Cities of Stip and Prilep

Tab. 17 DISTRIBUTION OF CLOTHING AND SCHOOL SUPPLIES 2004 – 2007 Stip Prilep Number of distributions Number of beneficiaries Number of distributions Number of beneficiaries 2004-2005 5 757 5 862 2005-2006 4 675 4 1,197 2006-2007 3 632 5 1,678 Total 12 2,064 14 3,737

Numerous extra-curricular initiatives were organized during the three years of the project, in both Stip and in Prilep ( Activity B3 ). These activities involved school children participating in the project, regardless of ethnicity. The number of direct beneficiaries reached through extra-curricular activities was approximately 830. Activities organized included: dance classes, creative workshops, sports events, theatre shows, language courses.

Tab. 18 EXTRA-CURRICULAR ACTIVITIES 2004 - 2007 Stip Prilep Number of activities Number of beneficiaries Number of activities Number of beneficiaries 2004-2005 2 115 3 124 2005-2006 2 59 4 204 2006-2007 4 91 5 117 Total 12 265 14 445

Starting with the second year of the project, after-school activities were organized for children with learning difficulties hindering their academic performance. The children were assisted by Roma and non- Roma teachers and benefited from after-school programmes every day, except Saturday and Sunday, and the availability of suitable educational materials. The number of children who attended the after-school programme was 224 .

Finally, the project provided support to the target schools with respect to the payment of electricity and heating costs as well as for the realization of small repairs. Schools where the parallel classes were delivered in Stip and Prilep were those two schools benefiting the most from this project contribution during the three years of implementation (Activity B2 ).

4G To what extent have primary school dropout rates decreased for targeted children within the entire schooling path?

In reference to dropout rates in primary school, the project has taken steps to monitor the dropout rate which occurred in the parallel classes in Stip and, over the last two years, at the school in Prilep.

53

Evaluation-Macedonia, Cities of Stip and Prilep

The graph below highlights the trend in attendance rates over the course of the project.

Figure 1 - Attendance in parallel education classes 120%

100%

80%

60% Primary School - Dimitar Vlahov - Stip 40% Primary School Dobre 20% Jovanoski - Prilep 0% School Year 2004- School Year 2005- School Year 2006- 2005 2006 2007

Figure 2- School attendance in Stip and Prilep

95%

90%

85% Stip 80% Prilep 75%

70% Year 2004 -2005 Year 2005 -2006 Year 2006 -2007

As evident in Figure 1, the dropout rate in parallel education classes in Stip between 1st and 2nd grade was 22% and 24% between 2nd and 3rd grade. Between 1st and the 3rd grade, the dropout rate was 41%.

In the Dobre Jovanoski primary school in Prilep, the dropout rate between 1st and 3rd grade was 6% . The main reasons for leaving school early, as recorded by the project were: i) having reached an age greater than 12 years old; ii) problems with the law; iii) family and students moving abroad; iv) marriage; and v) undertaking domestic activities/work.

Figure 2 illustrates the data related to school attendance in the city of Stip (Goce Delcev, Dimitar Vlahov, Tosho Arsov and Vancho Prke primary schools) and Prilep (Dobre Jovanovski primary school) during the same year of the project initiative. The average school dropout during the three years in Stip plateaued around 10% , while in the city of Prilep at 16%.

54

Evaluation-Macedonia, Cities of Stip and Prilep

4H Are Roma and non-Roma organizing communal cultural activities? 4I Are Roma and non-Roma communities participating in the activities of the Multicultural Centres?

CREATION OF NEIGHBOURHOOD ASSOCIATIONS

As planned, in the first year of the project, approximately 50 meetings were organized with non-Roma and Roma communities in the district of Radanski Pat in Stip and Trizla 1 and Trizla 2 in Prilep. The meetings provided opportunities to discuss the objectives and activities of the intervention and served to mobilize existing leaders and families, initially through personal contacts and then through larger groups. Publicizing the initiative took place with the active involvement of partners and also by using the local radio. During the meetings, participants were motivated and interested in creating a self-formed/representative group. (Activity C1 )

At the end of the first year, a neighbourhood association had been formed in Stip, members of the Management Committee had been identified and meetings were held on a monthly basis; procedures for its official registration were also initiated. A neighbourhood association was also formed in Prilep, although it was in a more embryonic state and its registration was postponed until the second year. ( Activity C2 ) Both associations were officially registered in the second year of the project.

In Stip, CerenjaSOS , registered in November 2005, included 34 people, 19 Roma and 15 Macedonians . Its Management Committee was made up of six members, including a supervisory board of an additional three persons. Six of the nine members of the Management Committee were women. During the second year, the association met once per week and, in the third year, twice per week. As anticipated, the association managed and coordinated the multicultural centre (MCC), as well as the extra-curricular activities in the second and third years of the project (Pillar B). In the second year, CerenjaSOS, benefited from the course in administration and management ( Activity C9 ), as planned. The association , which the ET learned during its interview with the local partner (Cerenja) was composed of the same members of Cerenja, ceased to exist with the end of the project . At the time of interview, Cerenja informed the ET that both the NGO and the neighbourhood association, unlike what was indicated in the 3rd Progress Report, had collaborated actively in all activities promoted by the project.

In Prilep , the association, RomaSOS, was registered in February 2006, and was composed of eight members, four Roma and four Macedonians, including three women . All members constituted the Management Committee, with the addition of one regular member. The association met once per week . In November 2006, the association, due to internal problems, decided not to take part in the management of MCC, as expected . Although the ET was unable to interview members of RomaSOS, the association is still active. It was determined that internal disagreements linked to differences in perspectives among the members and the President on project objectives and the action strategy of the association led to its dissolution. According to information gathered during the field missions, RomaSOS is currently a Roma association which primarily works for the protection and promotion of the rights of Roma women. RomaSOS has never worked with the activities of the MCC in Prilep.

When RomaSOS withdrew from the project, InterSOS, in agreement with the Municipality of Prilep (which owned the MCC property, as outlined below), proceeded with an alternative strategy for the creation of the MCC Management Committee. It was decided to activate a network of local Roma and non-Roma

55

Evaluation-Macedonia, Cities of Stip and Prilep organizations in order to ensure the shared and collective management of the MCC which integrated contributions from different associations existing in the city. A process of consultation was thus initiated with local organizations (about 30) which, in February 2007, led to the signing of a preliminary agreement for the management of the MCC. The agreement involved six local NGOs operating in all districts of Trizla 1 and Trizla 2 (two Roma and four non-Roma, including the local partner AHP) and a representative of the city. In May 2007, with the signing of a memorandum of understanding with the Municipality of Prilep, and in June 2007, with an agreement between the NGOs, the ownership and management of the MCC was defined and a five-year activity programme adopted. From November 2006 to June 2007, the network benefited from courses to support its establishment, including a course in administration and management (Activity C9 ); the project was supported by collaboration from the Foundation for the Support and Development, a foundation which no longer exists but which, at that time, was central in providing support for NGOs in Prilep. In the third year of the project, the contribution made by the burgeoning local network to the activities of MC was obviously biased. Activities were largely implemented by InterSOS and the local partner AHP.

As anticipated, the network began to work with the MCC in 2008, and was officially registered under the name of “Network 5 to 12” in 2009. In order to lay the groundwork for the future sustainability of the centres during the third year of the project, and following the approval of the budget modification by the MFA-DGCD, two experts in intercultural dialogue and the management of local networks, who had worked at the InterSOS mission in Kosovo, provided four months of technical support to strengthen CerenjaSOS in Stip and the network that would manage the MCC in Prilep.

CONSTRUCTION OF THE MULTICULTURAL CENTRES (MCCs)

The project included the establishment of two multi-cultural centres, with the support of the Municipalities of Stip and Prilep in terms of design and construction ( Activity C3 ). The construction of the MCC in the district of Radanski Pat in Stip took place during the second year, as indicated in the initial timetable, on land provided by the local partner, Cerenja – different than the provisions outlined in the original project document and initial letters of partnership, but according to the assessment reports of the CTU. Following the budget modification approved by the MFA-DGCD, in order to meet the need for the community to have a common, public meeting space, the size of the MCC was increased from 200 square metres to 299.357 square metres. The new MCC included: an entrance and corridor, one room, two offices, four classrooms, a bathroom, kitchen and windbreak filter. It is important to note that the Municipality of Stip completed the paving of the access road to the MCC and to the sanitation and hygiene facilities in the second year of the project, which was not specified in the initial project agreement and which was recorded, in part, as a contribution by the Municipality to the realization of the MCC. The construction of the centre, although contracted to an outside firm, included Roma labour. The MCC was official inaugurated in February 2006.

During the second year of the project, a second-hand van was purchased for the MCC in Stip and, although the MCC in Prilep was yet to be built, given the planned activities for the third year of the project, it was decided to purchase a second-hand van for future use by the MCC in Prilep as well. As such, the van was already available for use by the local partner, AHP. The purchase of two vans was authorized by the MFA- DGCD, as the original project design only included the purchase of one vehicle.

56

Evaluation-Macedonia, Cities of Stip and Prilep

In Prilep, the renovation of a building provided by the Municipality for the MCC in Trizla 2 commenced during the third year. The construction of a new building in Trizla 1 was not included in the development plan of the city and would have created difficulties, hence the decision to establish the MCC in Trizla 1. The MCC, included two floors, with a total of area of 264.7 square metres, consisting of: an entrance corridor, two bathrooms, one office for the Management Committee, three classrooms and one kitchen. The MC renovation was completed in September 2006; however, given the aforementioned problems related to its management, the MCC was officially inaugurated in March 2007, in the presence of local authorities and Italian representatives (though it had been operational since October 2006). The construction of the centre, although contracted to an outside firm, included Roma labour.

ACTIVITIES IN THE MULTICULTURAL CENTRES

As anticipated, during the second year in Stip and during the third year in Prilep, initiatives for Roma communities and cultural, recreational, educational and social activities for Roma and non-Roma were organized by the MCCs (Activities C4, C5, C6, C7, C8 and C9) . MCCs activities were held in the two cities even in the period prior to their official inauguration/opening. It should be noted that, in Prilep, the adult literacy course ( Activity C7 ), following approval by the MFA- DGCD, was replaced by a German course for school-aged children and young people, in order to respond to the requests of the beneficiaries themselves. Table 19 outlines all initiatives carried out by the MCCs during the three years of project implementation.

In addition to the 32 initiatives conducted in/by the MCCs, seven activities were implemented, three in Stip in the second and third years of the project and four in Prilep during the third year. These initiatives were held at the MCCs, but in the context of extra-curricular activities.

It should be noted that exchanges between the Roma communities in Stip and Prilep and Roma communities in other countries where InterSOS was present were not conducted, as originally planned.

Altogether, a considerable number of Roma and non-Roma community members benefited from the many initiatives promoted by the MCCs (of which at least 20 were cultural, as proposed). In total, there were approximately 3,715 beneficiaries.

The ET was unable determine the number of Roma vs. non-Roma beneficiaries based on the data available. However, Table 19 illustrates that, in general, there was a balance between Roma and non-Roma beneficiaries. Nonetheless, in Stip, apart from the events attracting for the general public, the involvement of non-Roma beneficiaries was lower than that recorded in Prilep. When interviewed, the president of Cerenja confirmed the activities carried out by the MCC Stip were directed mainly at the Roma community.

57

Evaluation-Macedonia, Cities of Stip and Prilep

Tab. 19 MULTICULTURAL CENTRES INITIATIVES 2004 -2005 2005 -2006 2006 -2007 STIP Cultural Initiatives Carnival ROMA : 800 Cerenja singing and ballet exhibition at the Folk dance exhibition during an international spectators (8 April House of Culture of Stip: 800 spectators dance festival held in Bitola: 600 spectators 2005 – Roma Day) (December 2005) National festival of traditional dances organized by Excursion to Stobi : Cerenja: 300 spectators (8 April 2007 – Roma Day) 46 people including 40 children Puppet Show at the MC C, 50 spectators 2 theatre group performances at the city theatre : 700 (March 2006) spectators (one show in December 2006 and one in June 2007 at the conclusion of the project with the participation of all partners and various authorities)

Carnival ROMA : along the city streets and at city theatre: 600 spectators (April 2006 – Roma Day) Excursion to Stobi: 36 children (29 Roma Youth participation in a dance course at the Festival and 7 Macedonians, including 28 Roma of Kumanovo : 500 spectators children in the parallel classes) Afterschool Afterschool : 94 Roma children from 1st to Ecological excursion to Lake Berovo : 38 students (28 4th grade, including 30 from parallel classes. from the parallel classes and 10 from the dance Children from Dimitar Vlahov, Gocev Delcev course), 31 Roma and 7 non-Roma and Vancho Prke schools Hygiene and sanitation Hygiene and sanitation course for women course for women (6h) : 243 Roma beneficiaries of the course and hygiene-sanitation kits Adult literacy course Adult literacy course ( 6h) : 60 Roma participants (26 men, 34 women) Management of conflict Management of conflict dynamics course dynamics course (6h) : 126 Roma beneficiaries Administration and Course in administration and management management course (6h) : 11 Roma beneficiaries, mainly members of CerenjaSOS TOTAL BENEFICIARIES STIP 846 BENEFICIARIES 1.220 BENEFICIARIES 2.100 BENEFICIARIES

PRILEP Cultural initiatives Excursion to Skopje : Italian course : 42 children from 1st to Folk dancing : F estival of Beer in Prilep (August 2006) - 400 46 persons, including 8th grade (40 Roma and 2 Macedonians) spectators 40 children Art exhibition in the Course in modern and folk dance : 42 Dance exhibition in the C ity Theatre : 300 spectators - City Theatre : 700 Roma children from 1st to 8th grade exhibition by children participating in the extra-curricular participants – dance class activities presence of the dance corps of Cerenja School newspaper : 24 children from 4th Excursion to Krushevo : 65 children (40 Roma and non - to 8th grade (11Roma, 13 Macedonians) Roma 25) from the parallel classes, after-school, theatre course, creative workshop and dance groups Excursion to Ohrid : 32 children Exhibition by children from the theatre course : 300 (including children of parallel classes) spectators Football Tournament : 96 children from 5th to 6th grade (34 Roma, 62 Macedonians) Afterschool Afterschool : 130 children (96 Roma and 34 non Rom a) from 1st to 5th grade, from the Dobre Jovanoski School Hygiene and sanitation Hygiene and sanitation course for women (6h) : 240 Roma course for women beneficiaries (including 14 men) of the course and the hygiene and sanitation kits Adult literacy course German course (6h) : 60 person s (28 Roma and 32 non Roma; including 40 youth from 10 to 12 years of age and 20 youth from 16 to 21 years Management of conflict Management of conflict dynamic s course (6h) : 120 dynamics course participants (children from the Dobre Jovaniski school between 6th and 8th grade) , including 69 Roma and 51 non-Roma Administration and Administration and management course (6h) : Suspended management course for RomaSOS, but held for 3 months for the 6 NGO Network assigned to manage the MCC in Prilep, organized by the Foundation for the Support and Development (FSD) TOTAL BENEFICIARIES 1,615 BENEFICIARIES + some members of 6 NGO 746 BENEFICIARIES 204 BENEFICIARIES PRILEP Network MC Prilep 1,592 3,715 BENEFICIARIES + some members of the 6 TOTAL BENEFICIARIES BENEFICIARIES 1,424 BENEFICIARIES NGO Network MCC Prilep 58

Evaluation-Macedonia, Cities of Stip and Prilep

Impact

5. What was the impact of the project intervention?

5A Has the project contributed to the reduction of diseases linked to poor personal hygiene in Stip?

During the in-country evaluation mission, the ET learned that at the end of the project (14 June 2007), the operations of the hygiene and sanitary facilities in Stip ceased . At present, the facilities are used as family- run mini-market, the revenues of which support a family reporting to the Cerenja association. The hygiene and sanitation facilities were operational for two years, 2006 and 2007. The activity should have contributed to a reduction in diseases associated with poor personal hygiene in Stip, but it is the opinion of the ET that the duration of the activity and the number of people utilizing the facilities suggests the project was unable to contribute to level of impact initially estimated.

To complement activities related to hygiene and sanitation, the project supported the construction of a water adduction/distribution network , including the construction of a pumping station. This construction did not only conduct water to/from the hygiene facilities, but also allowed families in the surrounding area to connect to the water supply. The impact produced by this action was certainly positive; in the course of the project, these two services benefited 100 families, while benefiting a total of approximately 500 families 69 today, as determined by the ET during the field mission.

Despite this positive result, the Team was unable to determine the extent to which the project contributed to the reduction of diseases related to poor personal hygiene in Stip. Please note data on this issues was not collected by the project during any stage of implementation. The ET assumes there was a positive impact with regard to the water adduction/distribution network; however, this refers to the number of people connected to the municipal water system, rather than to improvement in community sanitation.

5B Has the project contributed to the reduction of diseases linked to poor personal hygiene in Prilep?

5C Has the population of Prilep contributed to environmental care in the neighbourhood, in addition to the other project activities?

At the end of the project, the section of the canal in Prilep running through the neighbourhood of Trizla 1 continued to be cleaned by the Municipality. The Roma community is no longer this activity. The road connecting Trizla 1 and Trizla 2 appears to be still be good condition after the paving of the road in 2006. However, the ET observed a low-level of cleanliness in the district and even the observed a fair amount of rubbish in the canal (Photography Report). In the district, there are very few rubbish bins – only two are located near the canal as reflected in the photograph. The ET was informed that waste collection services occur only once per week; considering that local residents do not pay for this service, this to be expected .

Seven years after the end of the project, the Roma population in Trizla 1 is poorly sensitized on the issue of pollution and environmental impact, in terms of the need for active and coordination participation in

69 The figure of 500 families is a rough estimate and is unofficial, as many families are connected to the system in an informal manner. 59

Evaluation-Macedonia, Cities of Stip and Prilep addressing these problems. This was confirmed by the fact that, in interviewing a group of Roma citizens living in the vicinity of the canal, the ET observed the community believed the task of cleaning the surrounding environment should be delegated to the Municipality, alone (see the Sustainability section).

In addition, the ET was unable to determine how the project contributed to the reduction of environmental pollution-related diseases in Prilep. At the municipal level, there is no source of information providing data on this issue. In addition, it must be remembered that no data on the topic was collected at any point during the project (from its initial planning stage and throughout implementation). However, the ET notes that data on diseases commonly linked to the contamination of water aquifers at the national level are available from the National Institute of Public Health for 2006-2011.

Tab. 20 DEATHS PER 100,000 Disease 2011 2010 2009 2008 2007 2006 Enterocolitus 716.9 586.2 616.2 833.6 532.4 444 Escherichia coli 6 0 0 n/a n/a n/a Salmonella 13.7 9 7.8 19.4 13.7 9.8 Hepatitis A 14.7 15.2 14.2 11.9 12.6 25.7 Hepatitis B 7.5 8.1 9.5 10.5 9.6 Giardia 0,1 0 0,05 n/a n/a n/a

5D Has there been a % increase in the education of Roma in Prilep and Stip as a direct consequence of the project?

The activities carried out under Pillar B of the project should have resulted, in the long-term perspective, in an increase in the percentage of primary education among Roma children in Stip and Prilep. While confirming what has already been reported regarding the validity of the project design, the ET expresses concern with regard to the function of the parallel classes as a useful tool to achieve the expected impact. In fact, the parallel classes provided education to children for only three years, including the 1st, 2nd and 3rd grades of primary school. To this end, it is worth remembering that no diploma or formal recognition may be issued in Macedonia before the completion of all primary school grades. The project failed to implement monitoring tools to document how many children participating in parallel classes successfully continued their studies until the 8th grade. Hence, it has not been possible to reconstruct this aspect of the project during the course of the evaluation.

However, without a doubt, the project’s parallel classes were incredibly valuable, providing an opportunity for education to children who would have otherwise had no such chance . In this regard, the ET notes the parallel classes, which were initiated in 2004 in Stip and in 2005 in Prilep, have continued to this day. Regardless of whether they obtained a primary school diploma, the children who had access to the parallel classes were able to learn to read, write and undertake simple calculations and mathematical equations. Unfortunately, the manner in which school records are managed and maintained in Macedonia did not permit the ET to accurately determine the number of students who, over the years, attended the parallel classes. Another positive aspect is that, in accordance with statements made respondents, the number of children excluded from the school system for being too old has been reduced considerably over the years, suggesting the parallel classes will no longer be necessary as they have fulfilled their function.

In light of the above and with respect to the parallel classes, the ET believes the comprehensive measures put in place by the project (field visits to raise awareness among parents, after-school activities, distribution 60

Evaluation-Macedonia, Cities of Stip and Prilep of books and clothing, parallel classes) were able to stimulate the enrolment of children in, but not necessarily the completion of, primary school. In order to verify the extent to which the project contributed to the achievement of the latter, relative data on the number of Roma students in Stip and Prilep from 2004, the year of project commencement, to 2012 must be analysed.

Tab. 21 NUMBER OF ROMA CHILDREN ENROLLED IN PRIMARY SCHOOL IN STIP AND PRILEP Stip Prilep 2004-2005 341 658 2005-2006 412 733 2006-2007 476 719 2007-2008 396 786 2008-2009 393 841 2009-2010 404 742 2010-2011 397 882 2011-2012 425 777

While the above data indicate that an increase in the schooling of Roma children occurred between 2004 and 2012 (24.36% in Stip and 18.08% in Prilep), it does not contribute to the identification of a cause-effect relationship between the project intervention and the observed improvement. This is for two reasons: • The number of Roma students is not indicated with respect to the total number of school-aged Roma children, due to the absence of a recent national population census and the presence of a percentage of Roma children whose births are not regularly registered. This implies that the increase in the number of children enrolled in school might also have been caused by a corresponding increase in the total number of school-aged Roma children in the project target areas. • At the national level a reform in the education system was introduced, which provided, among other measures, for the distribution of free books, the intervention of government inspectors in the enrolment of children and in cases of absences exceeding 10 days, penalties for parents of children who do not attend school regularly, use of psychologists and educators to support families and children most in need. According to evaluation respondents, these measures have had a significant impact on the school attendance of Roma children and the ET believes these reforms have affected the increased enrolment of Roma children in the areas of project intervention.

Given the above considerations, and in an attempt determine the role of the project in increasing the percentage of Roma children enrolled in primary school, the ET conducted a comparative analysis between the increase in the number of children in the areas of the project intervention and the corresponding national figure (Figure 3). This type of analysis allows one to isolate the influence of certain variables, such as the national reform and, in part, the increase in the number of school-aged Roma children during the years analysed.

61

Evaluation-Macedonia, Cities of Stip and Prilep

Figure 3 - Enrollment of Roma children - Stip, Prilep, National 2004-2012

1200 1000 800 600 Stip

Enrolled 400 200 Prilep 0 National

Figure 3: Enrolment of Roma children in the cities of Stip and Prilep (n° enrolled per year) and the national enrolment level (no enrolled *10 1 per year) from 2006 to 2012. The analysis is based on data from the Institute of Statistics of Macedonia (2012)

Figure 4 - Increase in Roma Children in Primary School 2005-2012

50

40 30 20 National 10 Percent Prilep 0 Stip -10

-20

As illustrated in Figure 4, the increase of Roma children enrolled in primary school in Prilep follows the national trend. This leads the ET to believe there was no significant impact of the project in the targeted areas of Prilep. With regard to the city of Stip, however, until the year 2008/2009 enrolment increases, following the national trend, while in 2009/2010 and 2011/2012 a greater increase is apparent, with small, intermediate decreased in the 2010/2011 year. While the extent to which the project has contributed to this positive impact cannot be precisely determined, given that many variables may have influence this trend in Stip, including assistance from other international donors, it is possible to identify a positive result of the project.

62

Evaluation-Macedonia, Cities of Stip and Prilep

5E Has the project contributed to greater integration between Roma and non-Roma in the target areas?

At the conclusion of the project, the MCC established in Stip ceased to function as a public gathering place for the Roma community and/or the population as a whole. The MCC, which underwent structural changes since its initial construction by the project (i.e. the extension and conversion of internal rooms) now serves as an entertainment room/banquet facility managed by Cerenja. On certain occasions, the MCC is available to the Municipality to undertake administrative procedures with respect to the Roma community (e.g. the renewal of identity documents). It should also be noted in this section that the neighbourhood association, CerenjaSOS, simultaneously ceased with the end of the project. Therefore, in Stip, actions taken by the project have not resulted in an impact with respect to greater integration of Roma and non-Roma communities. Regarding CerenjaSOS, as noted in earlier sections of this report, the association was composed of members of the local implementing partner, Cerenja. While support from Cerenja was required to achieve the project output related to the creation of an association to manage the MCC, the nature and composition of CerenjaSOS was not sufficient to ensure its survival at the end of the project – it existed without an identity in its own right and a specific mission independent of that of Cerenja. Consequently, this lead to no impact with respect to the project. Also, during the project, the MCC geared its activities more towards supporting the Roma community rather than towards integration between Roma and non-Roma. It should be noted that officials of the Municipality of Stip interviewed by the ET, particularly representatives the Roma community, complained about the absence of a centre to hold public gatherings in the city.

In Prilep, the MC established by the project is currently operating and has not undergone any structural changes. Today, the MCC houses some of Municipality offices, including the Roma Unit, the office of the NGO COSV, and the activities of the Network 5 to 12 formed in the last year of the project to manage the centre. The presence of Municipality offices (Office for Pensioners), in particular the Roma Unit, has made the MCC a central point of reference for the Roma and non-Roma communities in Trizla 1 and Trizla 2. In relation to the presence of the NGO COSV, it should be noted that from 2009 to 2012, COSV implemented a project funded by the MFA-DGCD on “Integration and support for minorities in the Southeast Balkans” 70 , continuing to work in Prilep in Macedonia for the integration of Roma, among others, in partnership with the Municipality and the Network 5 to 12 established by the evaluated project. Through the evaluated project, the Network benefited from specific courses on management and administration, the project cycle, monitoring and conflict management . The Network has continued to work at the MCC, organizing activities and events, although it did not follow the strategy developed by the InterSOS project. For the most part, the activities have been managed by each separate NGO, without real collaboration as a network. However, the Network became an entity with an independent legal personality in 2009. Activities have involved Roma and non-Roma but no precise data are available on the number of activities and beneficiaries involved, as the Network does not produce annual reports. As reported by representatives of the organizations interviewed, as well as by the president

70 The outline of the project, of which InterSOS was also a partner for activities in Kosovo, is available at the UTL Serbia, Kosovo and Montenegro (http://www.skmbalcani.cooperazione.esteri.it/utlskmbalcani/IT/Iniziative/SchedaIniziativa.asp?id_paese=114&id_temi=4 ).

63

Evaluation-Macedonia, Cities of Stip and Prilep and vice president of the Network, limited cooperation between organizations and lack of proven experience in working together and administering a unified Network budget constitute obstacles to the submission of joint funding applications which would allow the Network to implement projects promoting greater synergy and impact, as opposed to initiatives carried out individually. The Network did not expand during 2007-2013, retaining its initial composition including one Roma association and four non-Roma associations. On the contrary, the Network became smaller, as the AHP partner never actively participated in the MCC activities after the end of the project. In addition, the Network is now at a crucial point in terms of its survival: the collaboration agreement between the Network and Municipality for the use of the MCC will expire at the end of 2013 and the ET has been informed that the Municipality plans to separate from the Network and to manage the MCC independently.

64

Evaluation-Macedonia, Cities of Stip and Prilep

Sustainability

6. To what extent do the benefits of the project continue beyond the project’s life?

6A Has the project adopted a management system for the maintenance of the services established (hygiene and sanitation, cleaning the canal, pavement of the road, MCCs, parallel classes, extra-curricular activities)?

PILLAR A. IMPROVEMENT IN COMMUNITY HYGIENE

With regard to the hygiene and sanitation facilities in Stip, since these ceased to function in 2007, there was no sustainability in terms of economic (see the next section) benefits or in terms of their management. Similarly, with regard to the cleaning of the canal, the Citizens’ Committee for the Protection of Dabnichka no longer exists and the cleaning of the canal is currently managed by the Municipality of Prilep . During the field visit, the ET spoke with residents in the area surrounding the canal who noted that manual cleaning of the canal is not practicable given the dimensions and amount of rubbish, even if one undertakes an autonomous and informal attempt to conduct even minimal cleaning activities. It was also highlighted that during the rainy season, the area around the canal is dangerous as it is not enclosed and, in fact, a few months before the evaluation mission, a young girl drowned in the river. The ET observed there are no points for garbage collection in the area around the canal, which of course, contributes to the problem. The Municipality of Prilep has plans to cover the canal with grating in order to reduce the danger and to prevent the disposal of rubbish into the canal. With regard to the road, the ET observed the road paved by the project was no longer in optimal condition, however it was much better than it had been prior to project, when it was simply made of dirt (Photography Report). This may be attributable to a lack of adequate maintenance by the Municipality of Prilep, although the quality of the materials used in paving the road may have also contributed to its deterioration.

PILLAR B. INCREASE IN SCHOOLING

With regard to the sustainability of activities under Pillar B, as noted in the section on the impact, the management of the parallel education classes continued even after the conclusion of the project, generating benefits for more than six years. The continuity of the parallel classes has been guaranteed by the two primary schools in Stip and Prilep who have continued to provide classrooms and to coordinate the organization and management of classes. Teachers are paid by the Ministry of Education of Macedonia . However, it is important to emphasize that, beyond the contribution made by the Ministry of Education for the payment of teachers, no steps towards the institutionalization of parallel classes by the Ministry have been taken. Over the next few years, the parallel classes will most likely cease to exist for two reasons: on the one hand, there has been a gradual decrease in children who are too old to be included in the regular school system, a point which confirms the success of the initiative in conjunction with the positive impact of national policies; and, on the other, there exists a strategic political environment favouring the formation of mixed classes to the greatest extent possible. This prediction is in no way viewed as negative by the ET, as the sustainability of the activity is only sensible and relevant for as long as the intervention remains necessary.

65

Evaluation-Macedonia, Cities of Stip and Prilep

PILLAR C. ORGANIZED MANAGEMENT OF ACTIVITIES AND CULTURAL EXCHANGES

The management of the multicultural centres requires different analysis between the city of Stip and the city of Prilep. While in Stip, the MCC has ceased its original functions, the MCC in Prilep continues to implement some activities. In the case of Stip , one of the causes resulting in the change of use of the MCC is the structure of the neighbourhood association, CerenjaSOS, which was formed during the project to manage the centre. CerenjaSOS traced/followed the organization and structure of the local partner, Cerenja. The members of Cerenja include about 1,200 individuals - Macedonians (20%) and Roma (80%), of whom 20 are active. It is the opinion of the ET that there are critical issues within Cerenja and, consequently, these were also present in CerenjaSOS, in terms of the effective participation of members. This is, in part, evidenced by the fact that there was no reported opposition from either members of Cerenja or CerenjaSOS following the change in the intended use of the two structures (MCC and sanitation facilities), which have since ceased to serve their social functions in favour of more strictly individual purposes. Based on existing facts, it is the opinion of the ET that Cerenja has operated more as a “family association” and this has facilitated the transformation in the use of both the hygiene and sanitation facilities and the MCC.

As far as the MCC in Prilep is concerned, the ET learned that the centre continues to carry out activities today, though not without certain difficulties. From a managerial point of view, the MCC has the support of the Municipality, the presence of COSV and Network 5 to 12. The presence of COSV and activities funded by international donors have had a positive influence on the continued operation of the MCC. However, the ET also identified many difficulties in terms of the sustainable management of the MCC. Network 5 to 12 appears fragile, fragmented and not well-structured to ensure proper management of the MCC. There are many divisions within the Network, as highlighted by a key Network member, AHP, leaving the group, and the willingness to abandon the Network expressed by other members, the lack of jointly implemented activities and the absence of an annual or multi-year activity plan. This weakness in management can be easily traced to the late establishment of the Network, formed in the third year of the project in response to the dissolution of RomaSOS. The Network did not have time to consolidate and test the management of the MCC with the help of national and international experts. Furthermore, the composition of the Network has also caused problems. AHP was a member of the Network, an association which given its history and role as a project partner, contributed to upsetting the internal equilibrium of the Network. The hand-over of equipment and vehicles purchased by the project was conducted by AHP, given its role as project partner. Finally, the Municipality of Prilep recently opened an office in the MCC whose leader has expressed a willingness to independently manage the MCC. Consequently, the management of the MCC is fragmented, lacks a strategic plan, and is now oriented to function as a place for conducting a variety of activities, rather than as a centre with proactive strategies and guidelines aimed at facilitating integration and multiculturalism.

6B Are appropriate financial and human resources available for the established services?

PILLAR A: IMPROVEMENT IN COMMUNITY HYGIENE

As already explained in the previous sections of the evaluation report, the project intervention did not put in place measures to ensure the sustainability of project activities in terms of the benefits associated

66

Evaluation-Macedonia, Cities of Stip and Prilep with the community hygiene and sanitation facilities in Stip. These facilities and their services ceased to operate at the end of the project. The building constructed to house the facilities, as already mentioned in the discussion of effectiveness of the project, it is located on property owned by Cerenja, which provided the land to the project. The facilities are currently used as a family-run mini-market, whose revenues support a family reporting to Cerenja. No documents reviewed by the ET suggest the handover of the building for the hygiene and sanitation facilities to Cerenja

The fact that the provision of private land was viewed as a valuable contribution from the local partner and was permitted under ministerial procedures, and given that the MFA-DGCD does not customarily stipulate or require documentation on the intended use of buildings and vehicles during the handover process, has been somewhat problematic in this specific case. In fact, not only was no definite agreement reached on the sale of the goods, but no define path was outlined for the sustainability of the community hygiene facilities after the end of the project, neither in the initial drafting of the business plan (as already mention in the section on effectiveness) nor at any point during the implementation. With regard to the economic sustainability of the services, from the interviews, the ET learned that during the course of implementation, there were numerous problems related to the construction of the facilities . In particular, ruptured pipes and malfunctions with the plumbing and equipment occurred, according to some respondents, due to both the quality of the construction and the materials used, as well as a result of users of the facilities and the absence of a security service. It was not possible to solve these problems because the hygiene and sanitation facilities, different than initially planned, did not have genuine strategy for economic sustainability. Regardless of the quality of construction or the materials used, which was contradicted by other sources consulted during the desk study and field mission, the ET believes the critical aspect relates to the absence of a proper business plan which outlined in detail the management costs, including depreciation and maintenance costs. In the absence of a proper business plan, prices established for the use of the bathrooms do not appear to have been determined based on the revenue needed to ensure sustainable operations throughout the year. In examining how the prices for using the facilities were established, there is conflicting information. Some respondents argued the price had been determined on the basis of the cost of other municipal services present in Stip, while others stated prices were calculated on an indicative estimate of out-of-pocket costs including, electricity, water and gas, directly related to the daily operation of the services. As such, it is important to remember that the intention of the project was to establish a price which was – to the greatest extent possible – representative and accessible for Roma individuals utilizing the services, in particularly those living in poor economic conditions without a private bathroom in their homes. From an analysis conducted by the Open Society Institute in 2007, on the education and situation of the Roma community in Stip, it emerged that the costs of public services were too high: “To improve hygienic conditions the Italian government financed the establishment of a public bath that includes washing machines. However, this is not used frequently, as the 50 MKD price (about € 0.80) for the services is too high for the local Roma population ”.71 This consideration leads to the ET to find that the economic sustainability of the hygiene and sanitation services was compromised by the high cost associated with accessing the facilities, which reduced the number of users. As reported in the section on effectiveness, the number of users was lower than that which was intended in the design phase

The estimated monthly revenue expected from the hygiene and sanitation facilities was estimated at € 1,906.32. The cost to use each personal toilet and for one use of the washing machine was € 0.52,

71 Open Society Institute, Equal Access to Quality Education for Roma. Volume 2 - Croatia Macedonia Montenegro Slovakia , 2007, p 268. 67

Evaluation-Macedonia, Cities of Stip and Prilep inclusive of laundry detergents and soap; thus the monthly income was calculated as follows: € 1,560.00 for showers at a cost of € 0.52 per day for 100 families (daily entry was per family and not per person) for 30 days; € 346.32 for 666 uses of the washing machines per month at a cost of € 0.52. Therefore, the total estimated annual revenue (12 months) was anticipated to be approximately € 22,875.64. The comparison between the actual and estimated total annual revenues shows that the initial forecasts were highly overestimated (Table 15).

With respect to the costs of the hygiene and sanitation facilities, reports on the progress during the second and third years of the project note the cost to use the services had been recalibrated, differentiating between adults and children in order to allow for greater access. As can be seen from the above, the costs identified in the design phase were lower than the prevailing costs to maintain the facilities, and access was also provided per family rather than per individual user, as actually occurred during implementation. One can therefore conclude that costs applied in the implementation phase did not consider the economic circumstances of the beneficiaries.

During the second year of the project, the operating costs for the community hygiene facilities were supported by the project for a total of € 3,208.00; during the third year, these costs were incurred by Cerenja, which stated during the evaluation interview that the association had basically funded the activities of the facilities, in particularly with respect to repairs and electricity and heating costs. In addition, in order to reduce the cost of electricity, which is particularly expensive in Macedonia, Cerenja installed solar panels on the roof of the building at its own expense in 2006; due to vandalism, the solar panels have been destroyed.

With regard to the water adduction/distribution network and associated pumping station in Stip as well as the canal and road in Prilep , the respective Municipalities, in continuing the provisions of agreements formed during the course of project, have proceeded to maintain the established services, despite the fact that the agreements referred only to the period of project implementation. The two Municipalities have demonstrated their support for the design strategy, given that even in the absence of an agreement or official handover guaranteeing the economic and institutional sustainability of the services, the local authorities have taken over the task of maintaining them. The ET is unable to assess the extent to which the Municipalities maintain adequate resources to ensure the optimal management of these services, but nonetheless believes their commitment and work is worth noting.

PILLAR B. INCREASE IN SCHOOLING

With respect to Pillar B. Increase in schooling, costs were incurred by primary schools in Stip and Prilep, which have continued to provide classrooms and to coordinate the organization and management of classes. The teachers are paid by the Ministry of Education in Macedonia. The ET believes that the entities concerned maintain and have allocated adequate resources to ensure the optimal management of these activities, considering that the schools have been supported financially by other programmes .

PILLAR C. ORGANIZED MANAGEMENT OF ACTIVITIES AND CULTURAL EXCHANGES

As for Pillar C. Organized management of activities and cultural exchanges , the project was implemented in two different manners in Stip and Prilep.

68

Evaluation-Macedonia, Cities of Stip and Prilep

In Stip , as already mentioned in the section of this report on the project impact, the MCC now serves as an entertainment venue/banquet facility, the revenues of which support a family which reports to Cerenja. The ET notes that in the analysis conducted during the desk phase and during the field mission, it was evident that no hand-over of the structure was carried out at the end of the project . Therefore, in the absence of an agreement between the parties, the MCC, having been built land owned by Cerenja (and made available to the project), was returned and was at the complete the disposal of the local partner, which managed the MCC according to private interests, as deemed appropriate. It should be noted that in the design phase and in the letters of support for the project, the Municipality of Stip committed to providing available land on which to build the MCC. However, such arrangements changed from the initial submission phase (2001) to the approval phase of the project. As stated in the assessment of CTU, Cerenja instead became the actor offering to provide the land, although no explicit motivations are mentioned in the documentation.

For the realization of the MCC in Prilep , however, the project collaborated with the Municipality, as provided for in the initial design phase. At the end of project, agreements were drafted between InterSOS, the Municipality and Network 5 to 12, on the future management of the centre, the identification of property (Municipality), facilities and exemptions in the payment of taxes and utilities, land use, work programmes and the mode for shared management between the Municipality and Network. These type of agreements laid the groundwork for the sustainability of the centre, which was further facilitated by the presence of the offices of the NGO COSV in the MC from 2009 to 2012. COSV managed a project sponsored by the MFA-DGCD on “Integration and support for minorities in the Southeast Balkans”72 , continuing to work in Prilep in Macedonia with the Municipality and the Network 5 to 12 (created by the evaluated project to manage the MCC), among other partners.

In Prilep, the project developed a good system of institutionalized management for the MCC, in relation to its take-over by the Municipality. However, the actual administration of the centre was confusing, as outlined in the previous section, and the Network’s lack of maintenance of management and accounting records does not permit the ET to give a positive response on the existence of adequate financial and human resources for continued management of the centre. Currently, the Municipality supports a large portion of the centre’s costs.

Specific consideration must be devoted to the economic sustainability of the MCCs, as intended in the initial design of the project . As confirmed by the evaluation of CTU, the project design necessitated that members of the public who attended cultural activities pay a contribution, even a symbolic amount, for the operation of the MCCs. The funds were to gradually increase and lead to the economic independence of the MCCs over a period of time, estimated to be two years. The project document outlined precise figures for membership fees, event tickets, costs and revenues in the section on economic and financial sustainability. In relation this sustainability plan, there is no evidence of its actual implementation in the documents reviewed by the ET nor in the course of the interviews did it emerge that particular attention was given to this aspect. Again there is an absence of a clear strategy to ensure the sustainability of the actions taken.

72 The outline of the project, of which InterSOS was also a partner for activities in Kosovo, is available at the UTL Serbia, Kosovo and Montenegro (http://www.skmbalcani.cooperazione.esteri.it/utlskmbalcani/IT/Iniziative/SchedaIniziativa.asp?id_paese=114&id_temi=4 ). 69

Evaluation-Macedonia, Cities of Stip and Prilep

Conclusions

RELEVANCE

The ET concludes that there was a good degree of relevance of activities in relation to the environmental-hygiene issues in the target areas. However, it must be noted that the main weakness of the project remained the absence of data regarding the incidence of diseases linked to personal hygiene and environmental pollution in both Stip and Prilep, although the reduction of incidence was one of the indicators of the specific objective. The waste management activities along the Prilep drainage canal had limited relevance for three reasons: a. the rubbish typically accumulating in the canal is very difficult to remove with basic tools such as those supplied by the project; b. the weak structure of the Municipality’s waste management system constitutes one of the main reasons for the accumulation of waste along the canal; and c. the project’s activities to raise awareness among the population living along the canal were too short to be effective and generate behavioural change.

In relation to the second issue, i.e. the low level of education of Roma children , the project adopted a relevant strategy and was able to align with international and national guidelines issued during project implementation.

Finally, the ET concludes that the strategy and activities adopted to improve integration between Roma and non-Roma were relevant.

However, the related issue concerning the strong division between Roma groups in the target neighbourhoods, the lack of a common organization and the limited integration between Roma and non-Roma was not relevant given that similar types of organizations already existed in these locations. A demonstration of the limited relevance of this activity can be found in the nature of CerenjaSOS which is essentially a duplicate of Cerenja.

EFFICIENCY

The review of the financial reports indicates that adjustments were made in order to adapt project activities to the changed needs of beneficiaries. The adjustments were necessary due to the significant period of time between the development of the project (2001) and the actual implementation of activities (2004). The ET maintains that this is a positive indication of the ability of the project to adapt to changing conditions.

The review of individual activity costs reveals their appropriateness and general alignment with national costs.

The review of the project timeframe shows that the project respected the planned overall implementation timeframe of three years. Nevertheless, the timing of the implementation of activities within the course of each year was not in line with the original project workplan.

70

Evaluation-Macedonia, Cities of Stip and Prilep

EFFECTIVENESS

The reconstruction of the logical framework based on the Theory of Change tool revealed that the project, as it conceived at the beginning, was coherent and its three thematic pillars were logically connected.

In terms of hygiene/sanitation facilities , the ET observed good outcomes resulting from the construction of the water adduction/distribution network, which enabled many families, more than 100 during the course of the project, to connect to the water supply. On the other hand, the activities regarding communal hygiene/sanitation facilities were less effective. In fact, the ET found that although the facilities were constructed and utilized, the percentage of the population that actually accessed them on a daily basis was 0.2%, well below the estimated 13%.

The drainage canal in Prilep was thoroughly cleaned throughout the duration of the project, and for a section much longer than originally planned, i.e. 450m instead of 200m. This certainly resulted in less wastewater stagnating in the canal, consequently limiting the potential negative health effects on the population living in the suburb and especially along the canal. The project involved the Roma population in the neighbourhood of Trizla 1, although at a level which was lower than originally estimated, even though awareness campaigns were conducted during the first year of implementation.

In Prilep, the paving of the main access road to the neighbourhoods of Trizla 1 and Trizla 2 was completed. As planned, the intervention was entirely financed by the Municipality of Prilep, which also built the sewage system in Trizla 1. The construction of the sewage system beforehand and the paving of the road during the project certainly improves the image of the suburb in terms of urban cleanliness and decorum and facilitates traffic circulation between the Trizla 1 and Trizla 2.

In terms of assessing the achievement of the objectives in relation to the education activities, the project did not define any success targets. For this reason, the ET utilized benchmarks included in the Decade of Roma documents, to which the project often referred. In 2004, the set target was to increase schooling by up to 90% for Roma children of relevant age .73 Taking into account data from the 2004 census conducted by InterSOS in collaboration with its partners Cerenja and AHP, which revealed that 400 Roma school-aged children were out of school, the target should have been 360 children. Of the 400 children included in the census, the project reached 245 , in addition to another 60 who were not included in the 2004 census. The success percentage of the project in terms of schooling of Roma children in the years 2004-2007 was 61.25%.

The school dropout rate in the parallel classes in Stip between 1st and 2nd grade was 22% and 24% between 2nd and 3nd grade. Between 1st and 3rd grade, the school dropout rate was 41% . In Dobre Jovanoski school in Prilep, the dropout rate in parallel classes between 1st and 3rd grade was 6% . Considering that the average school dropout rate in all schools targeted by the project in Prilep and Stip was approximately 10% in Stip and 16% in Prilep, the results of the initiative prove to be more positive for Prilep and less positive for Stip.

73 Ministry of Labour and Social Policy - Republic of Macedonia, Strategy for Roma in the Republic of Macedonia , 2004, p. 43. 71

Evaluation-Macedonia, Cities of Stip and Prilep In terms of the objectives of Pillar C, there were some difficulties in the establishment of the two associations charged with running the multicultural centres. However, the high number of Roma and non-Roma beneficiaries reached by activities organized by the MCCs in Stip and Prilep reveal a good degree of effectiveness of such activities. It must be noted, however, that the involvement of the non- Roma community in Stip was achieved only through events with a large visibility organized outside the MCC, which, in effect, during the life of the project became more of a gathering location for the Roma community rather than a multicultural centre.

IMPACT

The closure of the hygiene/sanitation facilities in Stip at the end of the project, combined with the limited use during the project, indicate that their impact was nil. The ET found that the construction of the water adduction/distribution network and pumping system had a positive impact , given the relatively high number of people (500) who are currently connected to the municipal water system.

It was not possible to assess the impact of the project in reducing the incidence of diseases linked to environmental pollution in Prilep, given that the project did not collect relevant data at any stage of the project and considering the absence of data in the target area (the only available data are national projections). Given the limited dimension of the intervention, the ET excludes the possibility that the project had an impact at the national level.

In addition, in Prilep, the project aimed to: i) progressively increase the awareness of the population on the health effects of environmental degradation and the stagnation of wastewater in the drainage canal in order to encourage the participation of the Roma community in its cleaning; ii) generate impact in terms of cleanliness of the community in improvement in traffic with the paving of the road, which consequently intended to create greater environmental consciousness. The ET found that the expected impact was, in effect, limited, as demonstrated by the nearly total absence of participation of the local population in environmental cleaning and waste management activities.

There are doubts also regarding the impact of the project in relation to whether the parallel classes actually provided schooling opportunities to children who would have otherwise not attended school . The ET notes that the parallel classes continued from their creation in 2004 in Stip and 2005 in Prilep to the moment of drafting this report. Regardless of whether or not the children graduated, they did have the opportunity to learn to read, write and understand basic calculations. The increase in the number of Roma children enrolled in the primary school of Prilep followed the national trend. This indicates that project activities did not have a significant impact in the area of Prilep. On the other hand, in the city of Stip in 2008/2009 the increase in enrolments followed the national trend, while in 2009/2010 and in 20011/2012 there was a more pronounced increase (with a negative low in 2010/2011). Although it is not possible to determine to what extent the project contributed to the positive trend in Stip, given that many variables may have played a role (including the activities of other international donors), the ET nevertheless believes that the project may have had some positive effect.

In Stip, the project had no impact in terms of improved integration between Roma and non-Roma. During an interview, the municipality stated that there was no public meeting centre, although this was the purpose of the centres constructed by the project.

72

Evaluation-Macedonia, Cities of Stip and Prilep Based on information collected during the evaluation mission, which has not been confirmed by concrete data, the ET found that in Prilep there was a good level of impact from 2007-2013 in terms of contributing to the integration of the Roma and non-Roma communities thanks to the continuation of project activities conducted by COSV.

SUSTAINABILITY

The main issue in terms of sustainability of the project was the lack of a strategy to facilitate the continuation of activities after the end of the project. Sustainability was further compromised by the fact that InterSOS closed its mission in Macedonia at the conclusion of the project. Although, as previously mentioned, InterSOS handed over all activities under Pillar C in Prilep to the NGO COSV, the project was nevertheless negatively affected by the ad hoc sustainability planned that was ultimately developed.

The lack of a clear business plan for the sustainability of the hygiene/sanitation services is a clear example of this. In fact, the hygiene/sanitation services in Stip stopped functioning from 2007. Similarly, in relation to the cleaning of the drainage canal, the Citizens’ Committee for the Protection of Dabnichka was dissolved and currently the cleaning of the cana l is managed by the Municipality of Prilep. In regards to the road , the ET found that the road which was paved during the project is currently not in optimal conditions, although significantly better than when it was a dirt road before the project (attached photos). This could be the consequence of a lack of maintenance on behalf of the Municipality of Prilep, although the quality of materials utilized most likely further contributed to its degradation.

The running of the parallel classes continued after the conclusion of project activities, extending benefits for over six years. The continuation of the parallel classes was guaranteed by the primary schools of Prilep and Stip, which provided classrooms and organized and managed the classes. The teachers are paid by the Macedonian Ministry of Education.

In relation to the MCCs, the weak strategy adopted by the project in Stip led to the ending of their intended function at the conclusion of the project. The outcome was better in Prilep due to agreements reached between the parties and the collaboration with the Municipality. However, the MCC in Prilep is poorly managed, lacks a strategic plan, and is utilized to conduct a variety of activities rather than constituting a centre offering strategies and activities aimed at promoting integration and multiculturalism. For both MCCs, outcomes would have been significantly improved if the economic sustainability strategy defined in the project document had been implemented.

73

Evaluation-Macedonia, Cities of Stip and Prilep

Recommendations

RELEVANCE

1. Although the ET had a positive view of the relevance of the project, it is nevertheless recommended that the MFA-DGCD place more attention on the situational analysis included in proposals, ensuring that intervention strategies are relevant and in line with the findings of such analysis.

EFFICIENCY

2. When more than 12 months elapse between the time of submission and the time of approval of a project proposal, it is recommended that the MFA-DGCD request an updated financial proposal before formal approval. This would allow to align expected costs with current costs and minimize the need to make financial changes during project implementation. This is particularly important when the project involves the physical construction of structures. 3. The ET recommends that the MFA-DGCD request an update of the project workplan/timeframe when financial or activity changes are made necessary, as well as in all annual and final reports. This would allow the MFA-DGCD to maintain an overview of the progress of the project.

EFFECTIVENESS

4. In the case of revenue-generating activities, it is recommended that the MFA-DGCD request a business plan , which should be regularly updated and included in monitoring and evaluation reports. This would provide important information regarding the feasibility, progress and sustainability of the funded activities. 5. The ET advises the MFA-DGCD to ensure that implementing organizations collect relevant data for the evaluation of their projects, as stated in their strategies. It is recommended that implementing organizations prioritise the collection of relevant data for the evaluation of their projects, starting from baseline data for success indicators and targets for stated objectives. This allows one to assess the success of development interventions and, if conducted on an on-going basis, to re-align activities when the need arises. 6. The ET suggests that the MFA-DGCD adopt the Theory of Change tool in ex-ante evaluations for the reconstruction of logical frameworks and logical connections between input, activities, outcomes and impacts. 7. When projects involve complex processes, such as participation, awareness-raising and establishment of entities/associations, the ET recommends that the availability of financial, human and time resources is duly assessed and verified.

IMPACT

8. The ET suggests the MFA-DGCD to request, during the phase of project proposal, the definition of credible, measurable, time bound and cost-efficient impact indicators, and the identification

74

Evaluation-Macedonia, Cities of Stip and Prilep of related baseline data during project inception or implementation. The lack of such data compromises the possibility of assessing the expected and actual impact of projects. 9. In order to improve impact of projects, the ET recommends, where possible, to reduce the number of sectors and areas of intervention and instead seek synergies with similar projects.

SUSTAINABILITY

10. The ET recommends the MFA to consider developing clear guidelines for the sustainability of projects it funds, requesting implementing organizations to include pragmatic sustainability strategies in their project proposals, with a particular focus on institutional and economic sustainability. These strategies should be regularly reviewed during project implementation and include clear and concrete hand-over procedures agreed upon by the MFA, the leading NGO and partners. 11. The ET suggests, where possible, to involve local entities or other institutional bodies in sustainability strategies. 12. The ET advises not to rely on one local partner alone for the sustainability of projects, especially when the project involves the construction of infrastructures. In the event that this is not feasible, the ET suggests to thoroughly assess the partner’s relations within the region/area, especially with local institutions and missions, as well as past relations with the leading organization. 13. The ET notes that it is important to place due attention to the timing of activities, especially those that initiate processes and/or create entities. These should not be implemented towards the end of the project.

75

Evaluation-Macedonia, Cities of Stip and Prilep

Lessons Learned

RELEVANCE

a. The lack of a detailed problem analysis for some sectors negatively affected the level of relevance of the project. Specifically, the lack of an analysis of the additional causes of the pollution problem in the canal in Prilep – absence of a waste collection point – and of the paving of the road – absence of a sewage system – diminished the relevance of some of the activities and contributed to limiting the impact of the project. In fact, if on one hand the paving of the road did not ensure the elimination of stagnant wastewater, on the other hand the manual cleaning of the canal did not guarantee the improvement of the hygiene-sanitation conditions of the population residing along the canal (given that the volume of waste was too high for manual removal), nor a medium and long-term solution to the problem. b. The thorough consideration of the context in the area of intervention is an essential element for the development of a successful project and in order to ensure relevance to the needs of beneficiaries. The context analysis for Pillar C of the project did not provide a comprehensive picture of the relationship status among Roma in the areas of Prilep and Stip.

EFFICIENCY

c. When a significant period of time elapses between the development and implementation of a project, the context and problems in the intervention area may have significantly changed. In these circumstances, it is therefore necessary to dedicate further time, human and financial resources to reviewing the problem and context, as well the entire project strategy. d. The absence in loco of the leading NGO at the time of approval and launch of a project requires additional costs for the commencement of activities and may compromise the ability of the implementing organization to capitalize on existing relationships and contacts.

EFFECTIVENESS

e. The important contribution of the Municipalities of Stip and Prilep reveals a high degree of ownership of project activities, which contributed to enhancing its level of effectiveness. The project demonstrates that the genuine collaboration of local governments significantly improves the effectiveness of projects, contributing to virtuous cycles and multiplier effects. f. The activation of fee-based public services requires careful consideration in defining an appropriate fee in order to ensure that the target population is able to access the services.

IMPACT

g. The limited impact of the project reveals the need to reduce sectors and intervention areas in order to maximise the availability of resources and better focus on the initiation of complex processes, such as participation, integration and awareness-raising.

76

Evaluation-Macedonia, Cities of Stip and Prilep

SUSTAINABILITY

h. What happened to the communal hygiene-sanitation facilities and MC in Stip reveals the need to improve sustainability mechanisms and hand-over processes for facilities, goods and services financed by the project. In the case of income-generating activities, a project should ensure that a business plan is developed, including depreciation costs, as well as ordinary and extraordinary running costs, in order to ensure effective economic sustainability. Before assigning the property or the management of goods and services financed by a project to one private entity alone, a thorough assessment of the nature of the private entity should be conducted alongside the stipulation of specific agreements on the use of such goods and services. The analysis of the two different cases in Stip and Prilep and of the education activities, suggest that the involvement of a local entity or public institution contributes to improving project sustainability. i. The establishment of ad hoc bodies for the management of goods and services produced by the project did not ensure sustainability. In some cases, the associations which were established revealed to be similar to existing ones in terms of nature and composition – Cerenja and CerenjaSOS – in other case, the functioning of newly created associations was compromised by internal divisions and the lack of prior mutual understanding – RomaSOS and Network 5 to 12.

77

Annex 1 DESIGN MATRIX

TARGET OR DATA TYPE OF MEASURES OR BASELINE SAMPLE OR QUESTION SUB -QUESTION STANDARD IF DESIGN DATA SOURCES COLLECTION DATA ANALYSIS COMMENTS QUESTION INDICATORS DATA CENSUS NORMATIVE INSTRUMENTS RELEVANCE 1. Is the 1A Do the Descriptive Beneficiaries NA NA Non experimental Interviews Random Interviews Com par ative Veri fy sample project beneficiaries feel feel the project One shot Logical sample relevance strategy that the project was useful framework relevant? was useful? Theory of Change 1B Project Normative Degree of 100% Needs analysis Non experimental Needs analysis Documents Desk analysis Comparative The analysis strategy is consistency consistency of One shot Reports, WB, UNICEF needs to take consistent with between outcomes and Strategies of strategies and etc. into account the beneficiary needs project strategy outputs with relevant documents of different levels and the analysis beneficiary international international between of beneficiary needs organizations organizations outputs and needs. (UNICEF, (WB, UNICEF, outcomes WHO, WB, etc. ) and of etc.) relevant Strategies of Macedonian relevant ministries Macedonian Logical ministries framework Theory of Change EFFICIENCY 2 Was the 2A Were the Normative Expected and MFA standards Initial budget Non experimental MFA rules NA Desk analysis Comparative Project results achieved actual costs One shot MFA staff Interviews analysis efficient? with the expected costs? 2B To what extent Descriptive Average cost NA NA Non experimental Final project NA Desk analysis Financial report It is not possible were inputs (AC) x One shot financial report analysis to assess the AC converted into child attending Financial data x beneficiary x results? x school/extra from other afterschool school activities projects activities AC x beneficiary Statistical data because they x sanitation of Multicultural are integrated activities (in Centres into the costs of alternative to Statistical data the Centres. services) of sanitation AC drainage services canal cleaning Number of AC road children who construction regularly attend 2C Was the Normative Average Cost Implicit NA Non experimental Project report CENSUS Desk Analysis Comparative The alternative (AC) x similar standards One shot Financial data Interviews analysis comparative utilized the most services from other analysis with efficient (costs)? projects other projects (international will be in

78

organizations relation to costs and Ministry of Education) 2D Were the Normative Expected MFA stand ards Initial Non experimental MFA rules NA Desk analysis Comparative results achieved timeline and Timeframe/ One shot MFA staff Interviews analysis in the set time? actual timeline work plan EFFECTIVENESS 3 Is the project 3A Is there a Descriptive Redefinition of NA NA Non experimental Logical NA Desk analysis Redefinition of design valid? logical and the intervention One shot framework and the logic consistent logic initial project through Theory connection documents of Change between inputs, Theory of activities, outputs, Change outcomes and impact? 4 To what 4A What is the % Descriptive Percentage of NA NA CENSUS 2006 - Project CENSUS Desk analysis Document extent were of the population access and use 2007 documents Interviews review and the objectives that accesses and Cerenja data interview of the project uses the outcomes met? sanitation facilities? 4B Is the drainage Descriptive Number of NA NA Non experimental Project NA Desk analysis Document canal constantly annual cleaning One shot documents Interviews review and clean? sessions Interviews interview Onsite outcomes observation 4C Does the Descriptive Nu mber of NA NA CENSUS Project CENSUS Desk analysis Document population volunteers who documents Interviews review and participate in the participated in Interviews Onsite interview cleaning of the the cleaning of Onsite observation outcomes drainage canal? the drainage observation canal 4D Has the Cause - Objective NA NA Non experimental Project NA Desk analysis Document presence of the effect improvement in One shot documents Interviews review and paved road terms of Interviews Onsite interview improved the healthiness and Onsite observation outcomes image of the urban decorum observation neighbourhood through direct terms of health ET observation and cleanliness? 4E Has paving the Descri ptive Objective NA NA Non experimental Interviews NA Desk analysis Document Included in the roads facilitated improvement of One shot Onsite Interviews review and project but it movements movements observation interview appears not to between the between Trizla Document outcomes be coherent neighbourhoods 1 and Trizla 2 review of Trizla 1 and through ET Trizla 2? observation 4F To what extent Normative % dropouts in % not defined Non defined CENSUS Interviews CENSUS Interviews Interview have primary parallel classes by the project but extractable Document Documents and outcomes and school dropout and children review school statistics review of rates decreased integrated in School statistics documents and for the targeted normal school statistics

79

children within schooling the entire % dropouts schooling path? school aggregated data 4G Have Descriptive Number of NA NA Non experimental Interviews CENSUS Interviews Interview beneficiary children who One shot School statistics Primary school outcomes children have completed records Records review completed primary school primary school? 4H Are Roma and Descriptive Number of NA NA Non experimental Project CENSUS Desk analysis Document no-Roma initiatives One shot documents Interviews review and organizing organized Interviews outcome of communal Number of Data from the interviews cultural activities? Roma and non- Multicultural Roma who are Centres, if organizing available 4I Are Roma and Descriptive Number of NA NA Non experimental Project CENSUS Desk analysis Document non-Roma Roma and non- One shot documents Interviews review and communities Roma who Interviews outcome of participating in participate in Multicultural interviews the activities of the activities of Centres’ reports the Multicultural the Centres? Multicultural Centres IMPACT 5 What was 5A Has the Cause - % of diseases % not defined % of diseases Quasi Statistics from CENSUS 2003 Desk analysis Pre -post data the impact of project effect linked to poor in the project linked to poor Experimental municipal, and 2013 Interviews comparison and the Project? contributed to the personal personal regional and National non-equivalent reduction in Stip hygiene hygiene in Stip national health Regional comparison of diseases linked in 2003 centres Stip group to poor personal % of diseases hygiene? linked to poor Stip health personal centres hygiene at the regional and national level in 2003 5B Has the project Cause - % of diseases % not defined % of diseases Quasi Statistics f rom CENSUS 2003 Desk analysis Pre -post data contributed to the effect linked to in the project linked to experimental municipal, and 2013 Interviews comparison and reduction of environmental environmental regional and National non-equivalent diseases linked to pollution pollution in national health Regional comparison environmental Prilep in 2003 centres Prilep group pollution in % of diseases Prilep? linked to Prilep health environmental centres pollution at the regional and national level in 2003 5C Has the Descriptive Number of NA NA Non experimental Project Random Desk analysis Document population of activities One shot document sample Interviews review and Prilep contributed conducted for Interviews outcome of to environmental the care of the interviews

80

care in the environment neighbourhood, in additional to other activities? 5D Has there Cause - % of Roma Not defined by % of Roma Quasi Municipal CENSUS 2003 Desk Ana lysis Pre -post data been a % increase effect children with the project children with experimental statistics in and 2013 comparison and in the education primary primary Prilep and Stip non-equivalent of Roma in Prilep education in education in Regional comparison and Stip as a Prilep and Stip Prilep and Stip statistics group direct in 2003 Ministry of consequence of % of Roma Education the project? children with statistics primary education at the regional and national level in 2003 5E Has the projec t Cause - % increase of Not defined by Number of Quasi Interviews to CENSUS 2007 Desk analysis Document contributed to effect members of the the project members of experimental pre association and 2013 Interviews review greater Multicultural the and post members Outcome of integration Centres and of associations Review of interviews between Roma neighbourhood established by reports of the Data and non-Roma in association the 2007 Multicultural comparison the target areas? % increase in project Centres 2007 between 2007 the exchanges Number of and 2013, if and 2013 between Roma exchanges available and non-Roma between Roma associations and non-Roma associations 2007 SUSTAINABILITY 6 To what 6A Has the Descri ptive Existence of NA NA Interviews with Random Interviews Outcome of extent do the project adopted a management management sample interviews benefits of the management systems for the committees, project system for the maintenance of local continue maintenance of services counterparts, beyond the the services? (sanitation, municipalities project’s life? toilets, drainage and schools canal, road, Multicultural Centres, parallel education classes, etc.) 6B Are Descriptive Availability of NA NA Financial CENSUS Desk analysis Document appropriate sufficient statement of 2012-2013 Interviews review and financial and financial and sanitation outcome of human resources human services, interviews available for the resources for Multicultural services? the services Centres, schools Interviews

81

Annex 2 THEORY OF CHANGE

INPUT S ACTIVITIES OUTPUTS OUTCOMES IMP ACT

750m of pipes (500m project and Water a dduction/distribution A1 Construction of the water 250m municipality). network for sanitation services % population accessing and using % reduction in Stip of diseases adduction/distribution network in Excavation and laying (municipality) constructed and functional urban sanitation services linked to poor personal hygiene Stip (500m) (total 750) Population (volunteers) Access to water guaranteed

Exectuive palnning (municipality)

Land of Cerenja Sanitation services const ructed and

Construction materials and interiors functional % population accessing and using A2 Constriction of sanitation Executive planning urban sanitation services services block External construction company Population is aware of environmental

and health issues

Improvement of wastewater flow in % reduction in Prilep of diseases Population organized The drainage canal is cleaned 3 times the canal linked to environmental pollution Basic tools provided by the during the project community A3 Cleaning of drainage canal pipes The canal is constantly clean 2 teams 10 workers (60 days – 3 in Prilep (200 m) Trizla 1 Population is aware of environmental hours a year) and health issues The population participates in the

leaning of the canal

The targeted population in Prilep A4 Paving of the road in Prilep Improved the image of the contributes to taking care of the (1,000m2 = 200m length*5m environment in the Road in Prilep paved and usable neighborhood in terms of health and Machinery municipality width)/Trizla 1 and Trizla 2 – main urban cleanliness neighbourhood in addition to the road other project activities Improvement of trafic circulation

ASSUMPTION The Roma communities in Stip and Prilep share the project objectives PILLAR A. COMMUNITY HYGIENE MANAGEMENT 82

ASSUMPTION ASSUMPTION ASSUMPTION Support from Ministry of Education Parents support the idea of The improved mutual schooling for their children understanding and integration of Roma and non-Roma facilitates processes

INPUT S ACTIVITIES OUTPUTS OUTCOMES IMPA CT

5 primary schools (4 in Stip a nd 1 in Prilep) Classrooms for lessons 3 uniforms and stationary materials for each child times 800 children B1 Primary school attendance Targeted children regularly attend Reduction of dropout rates among Local suppliers uniform and stationary materials x school during the year targeted children throughout the Teachers 800 students x years entire schooling path % increase in primary education Electricity and heating costs among Roma children in Prilep School director Increase of schooling among the and Stip Costs for afterschool activities targeted population

Contribution to the costs of the B2 The school has heating and Targeted children have access to a school (electricity and heating) electricity also thanks to the suitable schooling space project’s contribution

B3 Afterschool activities x 500 Roma and non -Roma children in the students x 2 years targeted schools participate in the

afterschool activities implemented by the project

PILLAR B. INCREASE IN SCHOOLING

83

INPUTS ACTIVITIES OUTPUTS OUTCOMES IMPACT

Facilitators and means for C1 Meeting with Roma c ommunities communication dissemination and leaders

Neighborhood associations C2 Establishment of suburb established and registered Roma and non-Roma organize Improved integration among Commune lands in Prilep and Cerenja associations and management Management committee for the comunal cultural and formative Roma and non-Roma in the Executive planning Municipality committees associations established and activities targeted areas Construction materials functional Construction firms Roma volunteer manpower Interiors C3 Con struction of multi cultural Multicultural centers constructed Roma and non -Roma communities centers and functional participate in the activities organized by the cultural centers

Initial cultural costs C4 At least 20 cultural activities Educational material Trainers Teachers C5 Afterschool 100 students in Sanitary kits groups of 4 Computers and printers

C6 Training on hygiene -sanitation for 480 women + hygiene- environmental awareness raising

and educational campaign, for a total of 10 meetings, for the population in the suburb of Trizla 1

C7 Literacy training for at least 1 20

adults ASSUMPTION

The Roma communities in Stip and Prilep share the project objectives C8 Conflict management training

C9 Administration and management training for 30 people

PILLAR C. ORGANIZED MANAGEMENT OF ACTIVITIES AND CULTURAL EXCHANGES

84

Annex 3

COUNTRY CONTEXT MACEDONIA 1991-2013

Since 1991, the Republic of Macedonia has been undergoing a process of transition from socialism and a centralized economy towards the creation of a democratic society and a market economy. 1

The reform policies of the nineties generated some positive results, although progress was limited by continued regional instability as well as internal political and economic structural dynamics. In particular, the newly established Macedonian State had to deal with internal tensions generated by the various ethnic minorities present on its territory (especially the larger Albanian minority), as well as with deteriorating relations with Greece, which perceived the birth of Macedonia as a potential threat to the integrity of its borders. Between 1993 and 1994 Greece imposed heavy economic sanctions on Macedonia, which were removed only in 1995 following US mediation efforts. The 1998 parliamentary elections saw the victory of the Macedonian Internal Revolutionary Organization - Macedonian Democratic Party for National Unity (VMRO-DPMNE).

The new Government included the following at the forefront of its agenda: public administration reforms, increased economic growth, reduced unemployment, increased direct foreign investment, resolution of inter-ethnic tensions and improved relations with neighboring countries. In addition, at the center of the political agenda, was the aim to integrate Macedonia into the Euro-Atlantic structure. In 1999, however, the Republic of Macedonia found itself suffering from consequences related to the conflict in Kosovo, with the new Government having to manage the arrival of more than 250,000 refugees. The influx of refugees had important repercussions in Macedonia with respect to the ethnic make-up of the country, particularly with relation to the large Albanian minority. Tensions erupted in early 2001 when in the north of the country escalated at the hands of Albanian rebels who had formed the Army of National Liberation.

After several months of violent hostilities, in August 2001, the Ohrid Agreement was signed. The peace treaty established equality before the law for both the Orthodox and Muslim population, recognized Albanian as an official language and foresaw greater participation of the Albanian population in all state institutions. In addition, the benefits obtained by the major Albanian minority were extended to other minorities, including providing for fair presentation in the public administration and recognizing language rights. The Agreement also sanctioned the strengthening of local governments. In 2001, the Macedonian Constitution was amended by a large majority to introduce more safeguards to protect the rights of minorities as enshrined in the Ohrid Agreement. 2

In 2001, Macedonia was the first of the Balkan countries to sign the Stabilization and Association Agreement, which entered into force in 2004; in 2005, Macedonia became an official candidate country to join the European Union (EU). Since 2006, the programme of the Government of the Republic of Macedonia

1 Macedonia proclaimed its independence on 18 November 1991. The new State was recognized by the EU in December 1993 and the United States in Feburary 1994. In 1993, Macedonia became part of the United Nations. 2 European Commission-External Relations Directorate General-Directorate Western Balkans, Cards Assistance Programme: Former Yugoslav Republic of Macedonia 2002-2006 . Pirjevec J., Le guerre jugoslave 1991-1999 , Einaudi, 2011, pp. 105-108, 113, 137, 363, 512, 570, 615-617. Website Balkan Caucus Areas Observatory – Macedonia (http://www.balcanicaucaso.org/aree/Macedonia/(classes)/article). 86

– led by G. Ivanov, President of the coalition of the centre-right VMRO-DPMNE since 2009 – has aimed to strengthen and stimulate the economy and to respect the guidelines and roadmap for entry into the EU. 3

In 2009, the Commission noted that Macedonia was sufficiently respecting the political stability criteria set forth in the acquis communautaire of the EU and recommended, on several occasions, that the country continue to the second stage of negotiations provided for in the Stabilization and Association Agreement; however, to-date this has yet to commence.

In 2002, at the time of the dissolution of the Federal Republic of Yugoslavia, when Macedonia was the poorest of the Balkan states and contributed only 5% to the Federation’s GDP, the European Commission, identified the following priorities for the country: a) maintenance of peace and resolution of inter-ethnic tensions; b) improvement in the efficiency of the state; c) poverty reduction and human development; e) promotion and growth of the private sector and job creation; and e) implementation of the Stabilization and Association Agreement signed with the EU in 2001.

The analysis of the economic and social conditions of the country conducted by the European Commission in 2001 described Macedonia as heavily in debt, with a high rate of inflation and widespread poverty, especially in rural areas and among households with many unemployed members or those with low levels of education.

Tab. a MACROECONIC and SOCIAL DATA 2001 (Source: UE 2012) 4 Population 2,031,000 Territory 25,713 km 2 Rate of population growth 0,5% Life expectancy 73.2 years Gross domestic product (GDP) per capita € 1.890 GDP € 3.89 billion GDP growth rate -4.5% Unemployment 30.5 % Inflation 5.5% Public debt 48.8% of GDP External debt 44.2% of GDP Literacy 95% Population living below the poverty line 20%

In 2013, the European Commission highlighted the progress achieved by the Macedonian economy, which is quickly progressing as a functioning market economy. According to the Commission, the Macedonia should be able to cope with competitive pressure and European market forces in the medium-term, provided a series of reforms are adopted to reduce significant structural weaknesses which still persist.

Economic activity stagnated in 2012, with a recovery in growth in the first half of 2013. Despite the international crisis, macroeconomic stability was maintained. Unemployment remains very high, especially among young people. It should be noted that the level of unemployment from 2001-2011 totaled 30-34% of the population, with the rate of youth unemployment reaching 57.7% in 2007; in general the youth

3 For more information on Macedonian politics, please consult the website of the President of the Government of Macedonia (http://www.president.gov.mk/en) and the website of the Macedonian Parliament (http://www.sobranie.mk/en). 4 European Commission, The Former Yugoslav Republic of Macedonia 2012 Progress Report , SWD(2012) 332 final Statistical Annex pp. 68-70. 87

unemployment rate was never below 53.7% (2010). Increasing levels of public deficit and debt continue to increase the vulnerability of the country. 5

Tab. b MACROECONOMIC and SOCIAL DATA 2011 (Sources UE and UNDP* 2012) 6 Population 2,060,000 Territory 25.713 km 2 Population growth rate 0.17% Life expectancy 74.85 years GDP per capita € 3.651 GDP € 7.49 billion GDP growth rate 3.1% Unemployment 31% Inflation 3.3% Public debt 34.1% of GDP External debt 68.9% of GDP Literacy 97.3% * Population living below the poverty line 27.5% *

5 European Commission-External Relations Directorate General-Directorate Western Balkans, Cards Assistance Programme: Former Yugoslav Republic of Macedonia 2002-2006 . European Commission, The Former Yugoslav Republic of Macedonia 2011 Progress Report , SEC (2011) 1203 final. European Commission, The Former Yugoslav Republic of Macedonia 2012 Progress Report , SWD (2012) 332 final. European Commission, The Former Yugoslav Republic of Macedonia: Implementation of Reforms within the Framework of the High Level Accession Dialogue and Promotion of good neighborly relations , COM (2013) 205 final. 6 European Commission, The Former Yugoslav Republic of Macedonia 2012 Progress Report , SWD(2012) 332 final Statistical Annex pp. 68-70. Website UNDP in the former Yugoslav Republic of Macedonia (http://www.mk.undp.org/content/the_former_yugoslav_republic_of_macedonia/en/home/countryinfo/). 88

Annex 4 AGENDA OF THE EVALUATION FIELD MISSION

Mission in Macedonia SSSA EVALUATION TEAM - Serena Rossignoli and Annarosa Mezzasalma

20.10.2013 Sunday Time Event Place 09:55 Arrival Skopje airport

21.10.2013 Monday Time Event Place - PRILEP 09:00 - 10:00 Samija Ajdini - Roma advisor of Prilep Municipality MCC 10:00 - 10:15 Kirshan Mamudoski - Beneficiary MCC Activities MCC 10:15 - 10:30 Nuri Dervishovki - MCC psychologist MCC 10:30 - 11:00 Rubin Arizankoski - President of NGO Skorpion MCC Liljana Gjorgjioska - President of NGO Prerodba (as well vice president of “Network 5 11:00 - 12:00 to 12”) MCC Goce Pereski - President of NGO Videlina (as well president of the “Network 5 to 12”) 12:00 - 13:30 and Sashe Trenkoski - Member NGO Videlina MCC 13:30 - 14:30 LUNCH BREAK 14:00 - 16:00 Zarko GinovskiAHP - PARTNER in Prilep AHP office Beneficiary - 5 children afterschool, one of them beneficiary of the clothes 16:00 - 17:00 distribution AHP office

22.10.2013 Tuesday Time Event Place - PRILEP 09:00 - 11:00 Violeta Laskoska - Headmaster of Primary School “Dobre Jovanoski” School 11:00 - 12:00 Daniela and Blagica - Teachers of parallel classes and visit Street and Canal MCC 12:00 - 13:00 Elena Kostoska - Teacher of afterschool MCC 13:00 - 14:30 LUNCH BREAK 14:30 - 16:00 Marjan Matrakoski - Public works and education - Prilep Municipality Municipality 16:00 - 17:00 Kristina Bogojoska - Assistant of Intersos Project Manager from 2005 to 2007 MCC

23.10.2013 Wednesday Time Event Place - STIP 09:30 - 12:00 Enise Demirova - President of CERENJA - PARTNER in Stip MCC Visit of MCC (multicultural center), HSF (Hygienic service facilities) and water pump 12:00 - 13:00 station Radanski pat 13:00 - 14:30 LUNCH BREAK 15:00 - 16:00 Teacher of afterschool MCC

24.10.2013 Thursday Time Event Place - STIP 09:00 - 10:30 Ilco Zahariev - Mayor of Stip Municipality 10:30 - 12:00 Headmaster of Primary School Dimitar Vlahov School 88

12:00 - 13:30 Headmaster of Primary School Goce Delcev School 13:30 - 15:00 LUNCH BREAK 15:00 - 17:00 Vane Maneski - Responsible for Education sector in Stip Municipality Municipality 17:00 - 18:00 Erol Ademov - Roma Referent at the Municipality of Stip Municipality

25.10.2013 Friday Time Event Place - Skopje Min. of 10:00 - 11:30 Redzep Ali Cupi - Ministry of Education Skopje - Sector for Minorities Education 10:00 - 11:00 Italian Embassy in Skopje IT Embassy 15:00 - 17:30 Goce Bogoevski - Assistant of Intersos Project Manager from 2004 to 2007 Hotel

26.10.2013 Saturday Time Event Place 05:15 Departure Skopje airport

89

Annex 5 PHOTOGRAPHY REPORT

PILLAR A. IMPROVEMENT IN COMMUNITY HYGIENE

2013, STIP – HYGIENE AND SANITATION FACILITIES

MAIN ENTRANCE BATHROOMS, ANTI-LICE SHAMPOO AREA, RECEPTION

SHOWERS SHOWERS

90

2013, STIP – PUMPING STATION

2013, PRILEP – TRIZLA 1-TRIZLA 2 ROAD

91

2013, PRILEP – TRIZLA 1 CANAL

92

PILLAR B. INCREASE IN SCHOOLING

2004-2007, STIP – EXTRA-CURRICULAR ACTIVITIES

2006, STIP – CHRISTMAST EVENT 2007, STIP – CREATIVE WORKSHOP

2013, STIP – DIMITAR VLAHOV SCHOOL 2013, PRILEP –DOBRE JOVANOSKI SCHOOL

2013, PRILEP – FOCUS GROUP WITH BENEFICIARIES OF EXTRA-CURRICULAR INITIATIVES AND AFTER SCHOOL PROGRAMMES

93

PILLAR C. ORGANIZED MANAGEMENT OF ACTIVITIES AND CULTURAL EXCHANGES

2004-2007, STIP – MULTICULTURAL CENTER

2006 DISTIBUTION OF SCHOOL SUPPLIES

2006 DISTRIBUTION OF CLOTHING

2006 THEATRE SHOW AT THE MUNICIPALITY OF STIP

94

2013, STIP – MULTICULTURAL CENTER

MAIN ENTRANCE

95

MAIN HALL MAIN HALL

KITCHEN BAR

96

2004-2007, PRILEP – MULTICULTURAL CENTER

2006, PROPERTY RENOVATION

97

2007, OPENING CEREMONY

2004-2007, PRILEP – MULTICULTURAL CENTER ACTIVITIES

2006-2007, AFTER-SCHOOL PROGRAMME

98

2006-2007, THEATRE WORKSHOPS

2013, PRILEP – MULTICULTURAL CENTER

BUILDING PLATE ON THE MAIN DOOR

ROMA UNIT MAIN HALL

99

Annex 6 BIBLIOGRAPHY

ActKnowledge e Aspen Institute, Theory of Change. Roundtable on Community Change , 2003. https://www.theoryofchange.org

Consiglio d’Europa, Proteggere i diritti dei Rom , 2010. http://www.coe.int/AboutCoe/media/interface/publications/roms_it.pdf

Decade of Roma Inclusion, Progress Report 2012 - Republic of Macedonia. http://www.romadecade.org/cms/upload/file/9276_file10_progress_report_mk.pdf

Decade of Roma Inclusion, Progress Report 2012 - Republic of Macedonia. http://www.romadecade.org/cms/upload/file/9300_file7_macedonia_decade-progress-report-f.pdf

Decade of Roma Inclusion, Progress Report 2010 - Republic of Macedonia. http://www.romadecade.org/cms/upload/file/9301_file5_macedonia-decade-progress-report-2010.pdf

European Commission-External Relations Directorate General-Directorate Western Balkans, Cards Assistance Programme: Former Yugoslav Republic of Macedonia 2002-2006 . http://ec.europa.eu/enlargement/pdf/financial_assistance/cards/publications/fyrom_strategy_paper_en.pdf

European Commission, The Former Yugoslav Republic of Macedonia 2011 Progress Report , SEC(2011) 1203 final. http://ec.europa.eu/enlargement/pdf/key_documents/2011/package/mk_rapport_2011_en.pdf

European Commission , The Former Yugoslav Republic Of Macedonia 2012 Progress Report , SWD(2012) 332 final. http://ec.europa.eu/enlargement/pdf/key_documents/2012/package/mk_rapport_2012_en.pdf

European Commission, The Former Yugoslav Republic of Macedonia: Implementation of Reforms within the Framework of the High Level Accession Dialogue and Promotion of Good Neighbourly Relations , COM(2013) 205 final. http://ec.europa.eu/enlargement/pdf/key_documents/2013/mk_spring_report_2013_en.pdf

European Commission, The Former Yugoslav Republic of Macedonia 2013 Progress Report , SWD(2013) 413 final. http://ec.europa.eu/enlargement/pdf/key_documents/2013/package/brochures/the_former_yugoslav_republic_of_macedonia_2 013.pdf

European Roma Rights Centre, Macedonia Country Profile 2011-2012 , 2013. http://www.osce.org/odihr/105348

Ministry of Education and Science - Republic of Macedonia, Development and State of the adult learning and education - National Report of the Republic of Macedonia , 2008. http://www.unesco.org/fileadmin/MULTIMEDIA/INSTITUTES/UIL/confintea/pdf/National_Reports/Europe%20- %20North%20America/The_former_Yugoslav_Republic_of_Macedonia.pdf

100

Ministry of Education and Science - Republic of Macedonia, National Programme for the Development of Education in the Republic of Macedonia 2005-2015 . http://www.npro.edu.mk/dokumenti/strategija-en.pdf

Ministry of Labour and Social Policy - Republic of Macedonia, Strategy for intensifying of the social inclusion of Roma in the social protection system in the Republic of Macedonia for the period 2012-2014 , 2011. http://www.google.it/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CDEQFjAA&url=http%3A%2F%2Fwww.mtsp.gov.mk%2 FWBStorage%2FFiles%2Froma_incl_en.doc&ei=lOeMUsCLOdCB7Qbq0oHgDA&usg=AFQjCNG1P5boO8W3hAodhKlFndhKGjKtjw&bv m=bv.56643336,d.ZGU

Ministry of Labor and Social Policy - Republic of Macedonia, Strategy for Roma in the Republic of Macedonia , 2004. http://www.mtsp.gov.mk/WBStorage/Files/roma_strategy.pdf

Open Society Institute Report, No Data - No Progress, Data Collection in Countries Participating to the Decade of Roma Inclusion 2005-2015 , 2010. http://www.opensocietyfoundations.org/sites/default/files/no-data-no-progress-country-reports-20100628_0.pdf

Open Society Institute, Equal Access To Quality Education For Roma. Volume 2 - Croatia Macedonia Montenegro Slovakia , 2007, pp. 167-275. http://www.opensocietyfoundations.org/sites/default/files/equal_20071218.pdf

Roma Education Fund, Country Assessment Macedonia , 2011. http://www.romaeducationfund.hu/sites/default/files/publications/ref_ca_2011_mac_english_screen.pdf

Roma Education Fund, Needs Assessment Study for the Roma Education Fund. Background paper. Republic of Macedonia , 2004. http://www- wds.worldbank.org/external/default/WDSContentServer/WDSP/IB/2006/02/28/000090341_20060228112346/Rendered/PDF/352 070MK0NA0Report.pdf

State Statistical Office, Republic of Macedonia-Multiple Indicator Cluster Survey 2011 . http://www.childinfo.org/files/MICS4_Macedonia_FinalReport_Eng.pdf

State Statistical Office, Republic of Macedonia-Multiple Indicator Cluster Survey 2005-2006 . http://www.unicef.org/tfyrmacedonia/MICS-ENG-1%281%29.pdf

State Statistical Office - Republic of Macedonia, Census of Population, Households and Dwellings in the Republic of Macedonia , 2002. http://www.stat.gov.mk/pdf/kniga_13.pdf

UNDP, National Human Development Report 2001. Social Exclusion and Human Insecurity in FYR Macedonia , 2001. http://hdr.undp.org/en/reports/national/europethecis/macedonia/macedonia_2001_en.pdf

UNDP, National Vulnerability Report for Macedonia - Focus on Roma , 2006. http://europeandcis.undp.org/uploads/public/file/Report2006-angl-web2.pdf

101

UNDP-UNDESA, Macedonia National Human Development Report – 2001. Social Exclusion And Human Insecurity in Macedonia . http://hdr.undp.org/en/reports/national/europethecis/macedonia/macedonia_2001_en.pdf

UNDP-WB-European Commission, Regional Roma Survey 2011 . http://europeandcis.undp.org/data/show/D69F01FE-F203-1EE9-B45121B12A557E1B

UNESCO, World data 7th Edition 2010/2011- The Former Yugoslav Republic of Macedonia , December 2011. http://www.ibe.unesco.org/fileadmin/user_upload/Publications/WDE/2010/pdf- versions/The_Former_Yugoslav_Rep_of_Macedonia.pdf

UNICEF, Breaking the cycle of exclusion. Roma children in South East Europe , 2007. http://www.unicef.org/ceecis/070305-Subregional_Study_Roma_Children.pdf

UNICEF-WB, Study project - Vulnerability of Roma children in the dispersed Roma communities in Skopje , 2000. http://siteresources.worldbank.org/EXTGLDEVLEARN/Resources/Roma_Report_Combined.pdf

WHO, The former Yugoslav Republic of Macedonia: Health profile , 2013. http://www.who.int/gho/countries/mkd.pdf

102

WEBSITES

Council of Europe – Roma Portal http://hub.coe.int/web/coe-portal/roma

Constitution of the Republic of Macedonia http://www.sobranie.mk/en/default.asp?ItemID=9F7452BF44EE814B8DB897C1858B71FF

Decade of Roma Inclusion 2005-2015 http://www.romadecade.org

Eu and Roma http://ec.europa.eu/justice/discrimination/roma/index_en.htm

Assembly of the Republic of Macedonia http://www.sobranie.mk/en

President of the Republic of Macedonia http://www.president.gov.mk/en

Osservatorio Balcani Caucaso – Aree – Macedonia http://www.balcanicaucaso.org/aree/Macedonia/(classes)/article

UNDP in the former Yugoslav Republic of Macedonia http://www.mk.undp.org/content/the_former_yugoslav_republic_of_macedonia/en/home/countryinfo

103

Annex 7 LIST OF PERSONS INTERVIEWED

MACEDONIA

PRILEP

INTERVIEWED PERSONS Samija Ajdini Roma A dvisor of Prilep Municipality Kirshan Mamudoski Beneficiary MCC Activities Nuri Dervishovki MCC P sychologist Rubin Arizankoski President of NGO Skorpion Liljana Gjorgjioska President of NGO Prerodba (as well vice president of “Network 5 to 12”) Goce Pereski President of NGO Videlina (as well president of the “Network "5 to 12”) Sashe Trenkoski Member of NGO Videlina Zarko Ginovski President of AHP - PARTNER IN PRILEP 5 adolescents Beneficiaries of A fterschool, one of them beneficiary of the clothes distribution Violeta Laskoska Headmaster of Primary School “Dobre Jovanoski” 2 teachers Teachers of parallel classes 5 persons Roma inhabitants of Trizla 1 Elena Kostoska Teacher of afterschool Marjan Matrakoski Superintendent of P ublic Issues Department of Prilep Municipality Kristina Bogojoska Assistant of Intersos Project Manager from 2005 to 2007

VISITED PLACES Multicultural Center of Prilep in Trizla 2 Dabnichka Canal in Trizla 1 Street linking Trizla 1 and Trizla 2

104

STIP

INTERVIEWED PERSONS Enise Demirova President of CERENJA - PARTNER IN STIP Gjuletmana Mustafova Teacher of afterschool Ilco Zahariev Mayor of Stip Municipality Headmaster of Primary School “ Dimitar Vlahov ” Headmaster of Primary School “ Goce Delcev ” Vane Maneski Head of Sector for public services and local economic development Erol Ademov Roma referent at the Stip Municipality

VISITED PLACES Multicultural Center of Stip in Radanski Pat Hygienic service facilities and water pump station

SKOPJE

INTERVIEWED PERSONS Redzep Ali Cupi Ministry of Education and Science – Directorate for Promotion and Development of the Languages in the education for the Ethnic Minorities Ernesto Massimino Italian Ambassador in Macedonia Bellelli Goce Bogoevski Assistant of Intersos Project Manager from 2004 to 2007

ITALY

INTERVIEWED PERSONS Valentina Stivanello Intersos Macedonia – Community expert 2005 / Project Manager 2005 - 2006 Maddalena Maiuro Intersos Macedonia – Community expert 2006 -2007 Magda Bellù Intersos Italy – Project Evaluator in 2007 / Regional Director Europe and Haiti Simona Verrusio MAE -DGCS/ VII th Office

105

ANNEX 8 TERMS OF REFERENCE

MINISTERO DEGLI AFFARI ESTERI

DIREZIONE GENERALE PER LA COOPERAZIONE ALLO SVILUPPO Ufficio IX Sezione Valutazione

TERMINI DI RIFERIMENTO PER LA VALUTAZIONE INDIPENDENTE “Macedonia, città di Stip e Prilep: Miglioramento della qualità della vita dei Rom e avvio dell’integrazione”

AID N. 7478

106

MINISTERO DEGLI AFFARI ESTERI

DIREZIONE GENERALE PER LA COOPERAZIONE ALLO SVILUPPO

TITOLO DEL PROGETTO : “Macedonia, città di Stip e di Prilep: Miglioramento della qualità della vita dei rom e avvio dell’integrazione” – AID N. 7478

LUOGO DEL PROGETTO : MACEDONIA

LINGUA DEL PROGETTO : Italiano

ORGANISMO ESECUTORE : INTERSOS

DURATA : 36 mesi

BUDGET : EURO 1.560.196,00

A CARICO DELLA DGCS: EURO 774.009,00 (Art. 15 - L.49/87)

A CARICO DELL’INTERSOS: EURO 237.831,00

A CARICO DELLE CONTROPARTI LOCALI: EURO 433.570,00

A CARICO DI ALTRI CO-FINANZIATORI: EURO 114.786,00

DONATORE : Governo italiano

107

1. Obiettivi del progetto

Il progetto “Macedonia, città di Stip e di Prelep: Miglioramento della qualità della vita dei rom e avvio dell’integrazione” è stato implementato dalla ONG “Intersos”, con la partecipazione delle controparti locali: per la città di Stip l’associazione “Cerenja, per la città di Prilep l’associazione “Aid for Handicapped and poor”. L’obiettivo principale del progetto è di contribuire al miglioramento delle condizioni di vita e di relazione delle comunità Rom di Stip e di Prilep. Gli obiettivi specifici del progetto sono: a) migliorare le condizioni igieniche delle comunità, b) sostenere l’incremento della scolarizzazione, c) contribuire alla gestione organizzata delle attività e degli scambi culturali, tutto ciò attraverso azioni integrate tendenti a raggiungere: - Con riferimento all’obiettivo a) costruzione della rete di adduzione e deflusso acque, nonché di un edificio dei servizi igienici con relativo equipaggiamento di bagni, docce, lavatoi, lavandini, macchine lavatrici elettriche ed una caldaia a nafta nella città di Stip; pulizia del canale di scarico ed asfaltatura di 200 m di strada centrale che separa i quartieri di Trizla 1 e Trizla 2 nella città di Prilep; - Con riferimento all’obiettivo b) la reintegrazione al percorso scolastico e formativo di tipo primario di ottocento bambini, nonché la partecipazione ed il sostegno di cento bambini con maggiori difficoltà alle attività di doposcuola e parascolastiche; - Con riferimento all’obiettivo c) la costruzione e l’equipaggiamento di due Centri Multiculturali, i quali saranno usufruiti per la realizzazione di venti iniziative culturali tra le quali: spettacoli organizzati con gruppi musicali del quartiere, attività di doposcuola per almeno cento bambini e monitoraggio degli stessi, educazione socio-sanitaria all’igiene personale e familiare, informazione circa le sintomatologie delle malattie legate al degrado ambientale e relative norme di prevenzione, alfabetizzazione di almeno centoventi adulti, attivazione di corsi di amministrazione e contabilità; oltre alla costituzione di Associazioni di Quartieri atte all’organizzazione di attività di scambio culturale e alla gestione di dinamiche conflittuali, formando i duecentoquaranta partecipanti di tale iniziativa alla comprensione di differenti opinioni e comportamenti, attraverso la simulazione di situazioni frequenti e possibili soluzioni. Il progetto ha preso avvio nel 2004 e si è concluso nel 2007.

108

2. Utilità della valutazione L’utilità della valutazione è la seguente: - rendere conto ai vari interlocutori esterni (Parlamento e opinione pubblica) sulle attività svolte attraverso un quadro conoscitivo degli esiti conseguiti; - condividere le esperienze acquisite al fine di poter indirizzare i futuri finanziamenti nel settore in Macedonia.

3. Scopo della valutazione

La valutazione dovrà: • esprimere un giudizio sulla rilevanza degli obiettivi e sul loro grado di raggiungimento, • esprimere un giudizio su efficienza, efficacia, impatto e sostenibilità del progetto; • esaminare il Progetto nella sua completezza, per identificare le buone pratiche e le lezioni apprese, in modo da usarle come base conoscitiva per sviluppare i futuri pacchetti d'assistenza tecnica; • analizzare le strategie e le modalità d’implementazione, come fornire raccomandazioni da integrare nel programma di decentramento culturale e dell’istruzione e della formazione Repubblica di Macedonia; • tenere in considerazione i fattori di sostenibilità e l’impatto che l’implementazione di tale programma avrà sulle condizioni educative, igieniche, sanitarie e culturali del Paese; • stimare i risultati e l’effettività dei programmi pilota a livello distrettuale, la loro discutibilità, nonché l’effettiva capacità di gestione da parte degli enti locali; Infine, la valutazione terrà in considerazione: 1. efficacia ed impatto dell’assistenza tecnica e formazione fornita e degli strumenti utilizzati a tal fine; 2. funzionamento del Comitato di Gestione e dei Centri Multiculturali; 3. analisi delle capacità gestionali degli enti coinvolti nel programma.

109

4. Quadro analitico suggerito

Il team di valutazione può includere altri aspetti in conformità con lo scopo della valutazione. La chiave dei criteri di valutazione ruota attorno ai seguenti aspetti: • Rilevanza : Il valutatore dovrà verificare il grado in cui il Progetto tiene conto del contesto e dei problemi amministrativi del Paese. La valutazione riesaminerà la misura con la quale gli obiettivi del Progetto sono coerenti con i requisiti e le esigenze del beneficiario. La valutazione stimerà se l’approccio è strategico e in che misura INTERSOS ha usato le risorse per l’attuazione del Progetto. Nel valutare la rilevanza dell’iniziativa bisognerà tenere conto: i) in che misura gli obiettivi dell’iniziativa sono validi, ii) In che misura gli obiettivi dell’iniziativa sono coerenti, iii) percezione dell’utilità dell’iniziativa da parte del destinatario. • Validità del design del progetto: La valutazione riesaminerà la misura con la quale il design del progetto è stato logico e coerente. • Efficienza: Analisi dell’ottimizzazione nell’utilizzo delle risorse per conseguire i risultati del Progetto. Nel valutare l’efficienza sarà utile considerare i) Se i risultati sono stati raggiunti con i costi previsti, ii) Se i risultati sono stati raggiunti nel tempo previsto, iii) Se l’alternativa utilizzata era la più efficiente (minori costi o minori tempi) rispetto alle altre. La valutazione indicherà come le risorse e gli inputs sono stati convertiti in risultati. • Efficacia: La valutazione misurerà il grado e l’entità di raggiungimento degli obiettivi del programma. Nel valutare l’efficacia del progetto sarà utile considerare i) Se l’obiettivo generale e specifico del progetto sono stati chiaramente identificati e quantificati, ii) Verificare se le caratteristiche progettuali del progetto sono coerenti con l’obiettivo generale e l’obiettivo specifico, iii) verificare in che misura gli obiettivi generali sono stati raggiunti, iv) Analizzare i principali fattori che hanno influenzato il raggiungimento degli obiettivi. • Impatto: La valutazione misurerà gli effetti diretti ed indiretti provocati dal progetto nel contesto di riferimento. Nel valutare l’impatto si dovrà tenere conto di quali reali cambiamenti l’iniziativa ha provocato nella collettività. La valutazione stimerà l’orientamento strategico del progetto in relazione al contributo apportato. • Sostenibilità: Si valuterà la capacità del progetto di produrre e riprodurre benefici nel tempo. Nel valutare la sostenibilità del progetto sarà utile considerare in che misura i benefici del progetto continuano anche dopo che è cessato l’aiuto della DGCS, ii) verificare

110

i principali fattori che hanno influenzato il raggiungimento o il non raggiungimento della sostenibilità del progetto.

5. Outputs

Gli outputs dell’esercizio saranno: • Un rapporto finale in inglese ed italiano con i risultati e le raccomandazioni per indirizzare i citati criteri chiave di valutazione. • Quattro pagine di sommario del Rapporto di Valutazione del Progetto in inglese e in italiano.

6. Metodologia

L’attività di valutazione ex post dovrà appurare che le previsioni di impatto si siano effettivamente verificate ed identificare le cause degli eventuali scostamenti. La valutazione sarà effettuata attraverso analisi di varie fonti informative, indagini di dati derivanti dalle attività di monitoraggio dei vari settori quali quello della produzione agricola ed artigianale, della commercializzazione e della formazione. I principi metodologici saranno indicati dal valutatore.

7. Disposizioni gestionali, piano di lavoro e quadro temporale

1. Desk Analysis Revisione della documentazione relativa 7 giorni lavorativi al progetto.

2. Field Visit Il team di valutazione visita i luoghi del 10 giorni lavorativi progetto, intervista le parti coinvolte, i

beneficiari e raccoglie informazioni supplementari. 3. Rapporto di valutazione Bozza del rapporto di valutazione. 5 giorni lavorativi 4. Commenti delle parti La prima stesura del rapporto di 5 giorni lavorativi interessate e feedback valutazione circola tra le parti interessate per commenti e feedback . Questi vengono consolidati ed inviati al team di valutazione. 5. Workshop Workshop sulla presentazione della 2 giorni lavorativi bozza del rapporto di valutazione con il

coinvolgimento delle parti interessate presso la DGCS

111

6. Relazione finale Il team di valutazione mette a punto il 5 giorni lavorativi rapporto di valutazione incorporando i

commenti.

E’ previsto che il valutatore dovrà condurre consultazioni ed incontri con i rappresentanti delle seguenti istituzioni:

- Municipalità di Stip - Municipalità di Prilep - Dipartimento dei Lavori Pubblici e dell’approvvigionamento idrico - ONG INTERSOS - Associazione “Cerenja” - Associazione “Aid for Handicapped and the Poor”

Profilo del valutatore • Ottima conoscenza nel campo della valutazione di progetti di cooperazione allo sviluppo • Laurea magistrale • Esperienza in interviste, ricerche documentate, redazione e scrittura di relazioni • Eccellenti capacità analitiche e di sintesi • Eccellenti capacità comunicative e di scrittura • Eccellente padronanza della lingua Inglese

112

FORMATO SUGGERITO DEL RAPPORTO DI VALUTAZIONE

Il file relativo alla prima pagina sarà fornito Copertina dall’Ufficio IX della DGCS. 1. Sintesi Quadro generale che mette in rilievo i punti di forza e di debolezza del progetto . Max 4 pagine, focalizzandosi sulle lezioni apprese e raccomandazioni. 2. Contesto del progetto - Situazione paese - Breve descrizione delle necessità che il progetto ha inteso soddisfare - Analisi del quadro logico - Stato di realizzazione delle attività e stima dei tempi di completamento del progetto

3. Obiettivo - Tipo di valutazione - Descrizione dello scopo e dell’utilità della valutazione

4. Quadro teorico e metodologico - Gli obiettivi della valutazione - I criteri della valutazione - L’approccio e i principi metodologici adottati - Fonti informative: interviste, focus groups, site visit - -Le difficoltà metodologiche incontrate

5. Verifica della realizzazione Verifica dei principali stadi di realizzazione del progetto. 6. Presentazione dei risultati 7. Conclusioni Concludere la valutazione facendola derivare dai risultati e dalle comunicazioni principali. 8. Raccomandazioni Le raccomandazioni dovrebbero essere volte al miglioramento dei progetti futuri e delle strategie della DGCS. 9. Lezioni apprese Osservazioni, intuizioni e riflessioni generate dalla valutazione, non esclusivamente relative all’ambito del progetto, ma originate dai findings e dalle raccomandazioni. 10. Annexes Devono includere i TORs, la lista delle persone contattate e ogni altra informazione/documentazione rilevante.

113