ESF Ex-post Evaluation
Synthesis 2007-2013
Country Report – Romania
Contract VC/2015/0098
Written by Metis GmbH, Fondazione Brodolini and Panteia October – 2016
Social Europe Country Synthesis Report - Romania
April 2016 2
EUROPEAN COMMISSION Directorate-General for Employment, Social Affairs and Inclusion Directorate G — Audit and Evaluation Unit EMPL. G4 — Evaluation & Impact Assessment E-mail: [email protected]
European Commission B-1049 Brussels
EUROPEAN COMMISSION
ESF Ex-post Evaluation Synthesis 2007-2013 Country Report – Romania
Directorate-General for Employment, Social Affairs and Inclusion European Social Fund 2016
Europe Direct is a service to help you find answers to your questions about the European Union.
Freephone number (*):
00 800 6 7 8 9 10 11
(*) The information given is free, as are most calls (though some operators, phone boxes or hotels may charge you).
LEGAL NOTICE This document has been prepared for the European Commission; however, it reflects the views only of the authors, and the Commission cannot be held responsible for any use which may be made of the information contained therein.
More information on the European Union is available on the Internet (http://www.europa.eu).
Luxembourg: Publications Office of the European Union, 2016
Catalogue number: KE-04-16-757-EN-N ISBN: 978-92-79-61697-6 doi: 10.2767/14263
© European Union, 2016 Reproduction is authorised provided the source is acknowledged. For any use of photo which are not under the European Union copyright, permission must be sought directly from the copyright holder(s) indicated.
Country Synthesis Report – Romania
Glossary of key concepts A2E Access and sustainable integration into employment – one of the Priorities of Article 3 ‘Scope of assistance’ of the ESF Regulation No 1081/2006 Action The second level in the OP architecture, usually the Priority Axis (see below) consists of several actions Adaptability A key policy area in the ESF, consisting of activities to increase the adaptation of workers and enterprises to
the changing economic circumstances and labour market demands - one of the Priorities of Article 3 ‘Scope of assistance’ of the ESF Regulation No 1081/2006 Annex XXIII categories The socio-economic characteristics of ESF participants reported in the ESF monitoring systems, relating to the participant gender, labour market status (employed (of which self-employed), unemployed (of which long-term unemployed), inactive of which in education and training), age (young people aged 15-24 and older people aged 55-64), disadvantaged status (migrants, minorities, disabled, other disadvantaged) and educational attainment status (by ISCED levels) Category of expenditure Categorisation of the Structural Fund expenditure; cf (CoE) ANNEX IV of COUNCIL REGULATION (EC) No 1083/2006 of 11 July 2006. ESF expenditure relates to Categories 62 to 74. Certified expenditure Expenditure incurred in the implementation of the ESF activities which has been approved by the Managing Authority of the Operational Programme and the European Commission Cluster A group of actions or interventions with common objectives and activities Community added value The extent to which the ESF activities provided effects (CAV) additional to the national / regional activities Convergence objective Speeding up the convergence of the least-developed Member States and regions: NUTS level 2 regions in the EU Member States whose gross domestic product (GDP) per capita was less than 75 % of the average GDP of the EU-25 for the same reference period Effectiveness The extent to which the set aims and objectives have been reached Efficiency The comparison between the achieved outputs and results and the costs incurred Gender sensitivity The extent to which the planning, design, implementation and monitoring reflects the gender issues ISCED International Standard Classification of Education, an international standard classification used to classify the education levels: 1 Primary education 2 Lower secondary education 3 Upper secondary education 4 Post-secondary non-tertiary education 5 Short-cycle tertiary education
6 Country Synthesis Report – Romania
6 Bachelor or equivalent Intermediate Body (IB) The organisation charged by the Managing Authority to implement the ESF funds in the OP Intervention The third level in the OP architecture, usually the Actions in the OP consist of several interventions Human capital A key policy area in the ESF, consisting of activities to develop the skills and knowledge of human resources across the different stages of the education and training system cycle, this theme is subject to another thematic evaluation - one of the Priorities of Article 3 ‘Scope of assistance’ of the ESF Regulation No 1081/2006 Managing Authority The institution in each Member State OP responsible for (MA) the strategic direction and financial management of the OP Operational Programme The means through which the ESF support was (OP) implemented in the Member States, as agreed between the European Commission and the Member States. Each OP consists of several Priority Axes, which in turn consist of several actions, which in turn consist of several interventions Output The immediate reach of the ESF activity (e.g. number of participants reached, number of schools or enterprises supported) Participant The person who participated in the ESF funded activity. Priority Axis (PA) The first level in the OP architecture, usually the OP consists of several Priority Axes (concepts of priorities, areas and others are also used in the OPs), which in turn consist of several actions and each action of several interventions Project promoter The organisation in charge of implementing specific ESF funded projects
Result The change achieved through the activity upon leaving to long term achievements of ESF activities (e.g. number of qualifications acquired by participants, number of enterprises providing training) Strengthening A key priority focussing on the efficiency of public Institutional Capacity administrations and public services at national, regional and local level by promoting mechanisms to improve good policy and programme design, monitoring and evaluation, and capacity building in the delivery of policies and programmes in the relevant fields. SIC is one of the Priorities of Article 3 “Scope of assistance” of the ESF Regulation No 1081/2006. Sustainability The extent to which the achieved results last
7 Country Synthesis Report – Romania
Table of Contents EXECUTIVE SUMMARY ...... 10 INTRODUCTION ...... 15 1 BACKGROUND AND CONTEXT ...... 16 1.1 Challenges and role/position of ESF ...... 16 1.1.1 Challenges faced by the country in the different policy fields at the time of programming and during the programming period ...... 16 1.1.2 CSR addressed by the ESF ...... 17 1.1.3 ESF response to the crisis ...... 18 1.2 Support and Contribution to the Lisbon Strategy and Europe 2020 strategy and targets ...... 18 2 SCALE OF ESF INVESTMENT ...... 20 3 EFFECTIVENESS BY PRIORITY ...... 22 3.1 Outputs and results ...... 22 3.1.1 Outputs ...... 22 3.1.2 Results ...... 27 3.2 Evidence of effectiveness ...... 28 4 EFFICIENCY BY THEME ...... 31 5 SUSTAINABILITY ...... 33 6 GENDER SENSITIVITY ...... 34 7 COMMUNITY ADDED VALUE ...... 35 7.1 Volume effects ...... 35 7.2 Scope effects ...... 35 7.3 Role effects ...... 35 7.4 Process effects ...... 35 8 SOCIO-ECONOMIC IMPACT ...... 37 9 KEY LESSONS LEARNED ...... 39 9.1 What are the key lessons in terms of policy choices? ...... 39 9.2 What are the key lessons in terms of target groups? ...... 39 9.3 What are the key lessons in terms of appropriate programming? ...... 40 9.4 What are the key lessons in terms of effective implementation? ...... 40 9.5 What are the key lessons in terms of monitoring? ...... 41 9.6 What are the key lessons in terms of evaluation? ...... 41 10 ANNEXES ...... 42
8 Country Synthesis Report – Romania
List of tables and figures
Table 1. ESF Ex post evaluation, main figures by policy theme ...... 14 Table 2. Financial allocation and implementation rate of the ESF programming period 2007-2013 ...... 21 Table 3. Cost per participant/participation of some interventions ...... 31 Table 4. Cost per participation by PA / theme ...... 32 Table 5. Share of women by Priority ...... 34 Table 6. Changes in Socio-economic indicators ...... 37 Table 7. Financial allocation and implementation rate of the ESF programming period 2007-2013 ...... 42 Table 8. Data on participants of the SOP HRD and OP ACD ...... 43 Table 9. Output indicators by policy field ...... 44 Table 10. Result indicators by policy field ...... 47 Table 11. Inventory of evaluations ...... 50 Table 12. CSRs and ESF contribution ...... 51 Table 13. Mapping of SIC ...... 54
Figure 1. Labour market status of participants in ESF 2007-2013 (entering) ...... 22 Figure 2. Education status of participants in ESF 2007-2013 (entering) ...... 23 Figure 3. Vulnerable target groups participating in ESF 2007-2013 (entering) ..... 23 Figure 4. Participants by policy field ...... 24
9 Country Synthesis Report – Romania
Acronyms A2E Access to employment and sustainable integration into employment AIR Annual Implementation Report ALMP Active labour market policy CAV Community Added Value CIE Counterfactual Impact Evaluation CoE Category of expenditure CF Cohesion Fund CSR Country Specific Recommendation CVT Continuous Vocational Training EC European Commission EEN Expert Evaluation Network EES European Employment Strategy ESF European Social Fund EU European Union HC Human Capital IB Intermediary Body ISCED International Standards Classification of Education IT Information Technology KAI Key Area of Intervention LMP Labour market policy MA Managing Authority MS Member State NEET Not in Employment, Education or Training NGO Non-governmental organisation NSRF National Strategic Reference Framework OP Operational Programme OP ACD Operational Programme Administrative Capacity Development PA Priority Axis PES Public Employment Service R&D Research and Development SFC Structural Funds Common Database SI Social Inclusion SOP HRD Sectoral Operational Programme Human Resources Development
10 Country Synthesis Report – Romania
Executive summary
Headline figures and key findings at the end of 2014
Activities were implemented under two Operational Programmes: Human Resources Development and the Administrative Capacity Development Total financial allocation: EUR 4.3 billion (ESF contribution: 85%) 44.5% or EUR 1.9 billion was incurred by the end of 2014 More than 45% of the total ESF investment was attributed to the Human Capital (HC) theme, 18% to Access to Employment (A2E). 1.4 million participations were registered by the end of 2014, of which 50% participated under the HC theme and 27% under A2E. 38% of the participants were employed, 32% with tertiary education, 30% with lower education (ISCED 1 and 2), 24% were young people (15-24) and 8% were minorities. Aggregation of results People in employment: 43,725 People gaining qualification: 344,893 Other positive results: - Products: 182 Entities: 2,643 Main findings: The policy field Increasing Social Inclusion, was one of the most effective and successful measures according to its evaluations. The activities targeted underprivileged and vulnerable groups and regarded their training and integration in the labour market. Volume effects were the most significant Community Added Value of the ESF, mainly concerning Social Inclusion, where the scale of the projects was considerable. Sustainability was achieved mainly when efficiency and effectiveness were coupled with investment in particular infrastructure projects or when the institutions implementing the activities were able to make the project action long-term and turn it into a mainstream practice.
Background The country synthesis report covers 2007-2013 programming period up until the end of 2014 and includes the analysis of expenditure, outputs and results in Romania across all policy themes. It builds on the Human Capital, Social Inclusion and Access to Employment ex post evaluations, Annual Implementation Reports and on other available ESF evaluations carried out in Romania. 1. Policy challenges addressed by the ESF The Romanian labour market remained fairly stable throughout the economic and financial crisis. The general unemployment rate grew slightly until 2013 but continues to be below the EU-28 average. Some of the major challenges that the labour market still has to overcome are undeclared work, labour migration and employment in the agricultural sector. Under the ESF Operational Programmes, the planned interventions addressed mainly job creation, in particular for vulnerable groups, participation in education and training, the competence and accessibility of the Romanian social services and the administrative capacity. Entering the EU in 2007 Romania faced challenges in all areas covered by EU2020 indicators (low employment (64%), educational attainment (14%) and high rates of early school leave (17%) and share of the population at risk of poverty (46%). The main challenges facing Romania, presented in the National Strategic Reference Framework, refer to employment stimulation and particularly to employment opportunities for certain at-risk groups, such as the Roma community, women, the young and the disabled, through active labour market policies and by overcoming informal work. Thus the priorities are to confront social exclusion by enhancing job security and labour market integration, especially for vulnerable groups.
11 Country Synthesis Report – Romania
The Country Specific Recommendations for Romania address these concerns. They especially repeatedly encourage labour market participation, and the employability and increase in productivity of the labour force. They also focus very strongly on the education and training system in order to improve skills and qualifications for a better integration in the labour market, particularly of young people and vulnerable groups such as the Roma community. Furthermore, they address the amelioration of social services. The Romanian ESF Operational Programmes are in line with the reform agenda of the Lisbon Strategy, the Europe 2020 Strategy and the Country Specific Recommendations. Social inclusion of disadvantaged groups was encouraged, also through training and incentives, while there were activities to integrate young people and women in the employment sector by improving their competences. 2. The scale of ESF investment Funding Romania was allocated about EUR 4.3 billion in total funding for the programming period 2007-2013 of which about EUR 3.7 billion was ESF funding. The funding was divided between two OPs. EUR 4.1 billion were allocated to the Sector Operational Programme Human Resources Development (SOP HRD) and EUR 0.2 billion to the Operational Programme Administrative Capacity Development (OP ACD). The implementation rate of the resources used by the end of 2014 was only 44.5%. Participation In respect to outputs, there were 1,423,443 participations in the interventions funded by the Romanian ESF OPs in the 2007-2013 programming period: 1,345,713 in the SOP HRD and 70,440 in the OP ACD. The largest share of participants was supported under the policy field Access to Employment (28.2%). Among the total participants the majority were employed, had mainly a tertiary education (ISCED 5 and 6) or had a primary or lower secondary education (ISCED 1 & 2) and were mostly young. The share of migrants and disabled people is very low, while the minorities make up 7.7%. 3 The effectiveness of interventions The OP results were achieved to some extent (39% of the indicators exceeded the target set), with the field of Social Inclusion standing out as having accomplished all but one of the set targets. However, in terms of participations this policy field did not reach the initially planned amount. Fairly low implementation rates were noted in the fields of Increasing Adaptability and Enhancing Human Capital (23%), while Access to Employment (33%) and SIC were more successful with partially achieved indicators. The policy field Strengthening Institutional Capacity presented problems with the indicators not being clearly defined at the beginning and with targets either over or under- estimated. Concerning effectiveness, the policy field Increasing Social Inclusion, was one of the most effective and successful measures according to its evaluations. The activities targeted underprivileged and vulnerable groups and regarded their training and integration in the labour market. Aggregation of results People in employment: 43,725 People gaining qualification: 344,893 Other positive results: - Products: 182 Entities: 2,643 4 Efficiency With reference to efficiency, the projects that involved a large number of participants presented lower cost per participant per month, while those with few participants had higher costs. In some cases the efficiency was undermined by the long duration of
12 Country Synthesis Report – Romania particularly expensive interventions. Projects assessed as efficient were significant in terms of the considerable share of underprivileged groups that they supported. 5 Sustainability Sustainability was measured only for some of the interventions, thus it is difficult to draw general conclusions. It has been reported that sustainability has been achieved in particular cases that refer to interventions concerning vulnerable groups and where investment in special infrastructure was possible. Projects presented as being especially sustainable are the ones targeting alcohol addicts, the blind, prisoners, as well as a project involving the training of lifeguards and divers. 6 The gender sensitivity of interventions The Romanian SOP HRD contains specific objectives on gender issues and most of the projects addressed in various ways the gender dimension. This issue is particularly relevant as the Gender Equality Index in Romania is the lowest in EU-28. One of the measures was entirely devoted to women’s employment (Developing skills and qualifications for women). Women were generally over represented and, in projects concerning social workers and volunteers, women constituted the largest share among the target groups. 7 Community added value of ESF investment Volume effects were the most significant Community Added Value of the ESF, mainly concerning Social Inclusion, where the scale of the projects was considerable. The SOP HRD interventions supported some actions that would not have been possible otherwise due to lack of qualified staff and poor funding. There is little information on role effects, but the process effects of the ESF funding can be seen in some projects targeting women’s qualifications and training, where the networks between actors in rural communities were strengthened. Moreover, process effects are found in the OP ACD, as the new procedures introduced by ESF helped to improve the institutional capacity of the public administration. 8 The socio-economic impact of interventions 8.1 Micro level The ad-hoc evaluation of SOP HRD intervention on youth highlights an interesting impact of ESF on the intervention, especially related to measures aiming to reduce school dropouts and to support students in the transition from school to work. Young people (15-24) were also very interested and participated in large numbers in measures supporting development of entrepreneurial skills. 90% of young persons that opened a new business supported by ESF declare that these are still operational. 8.2 Meso As per the Partnership Agreement of Romania, in order to support the challenges in terms of poverty and social exclusion, ‘…a higher involvement of the local authorities in addressing these challenges, which should be correlated with an appropriate investment in building their administrative capacity in applying the appropriate solutions’ is needed. 8.3 Macro Overall the ESF interventions, accompanied by national resources, helped job creation and participation in training activities, with an encouraging impact on productivity, development of skills, participation on the labour market, institutional development and the formation of additional know-how. An interesting result indicator proposed by SOP HRD refers to Share of enterprises providing CVT to their personnel, which registered 120.6% accomplishment. 9 Overall assessment of all evaluation criteria Overall, the effectiveness of the Romanian OPs can be considered as mixed. Some policy fields, such as Social Inclusion, performed better than others. The certified expenditure of most of the policy fields was lagging behind by the end of 2014.
13 Country Synthesis Report – Romania
14 Country Synthesis Report – Romania
10 Key lessons The second interim evaluation report of SOP HRD identifies the need to put a sharper focus on interventions for certain categories of target groups, like people with disabilities, children, youth, victims of domestic violence and other vulnerable groups. Both ESF OPs identified a common cause of ineffectiveness, which was also highlighted in a previous system evaluation1: the low capacity of project beneficiaries in project formulation, implementation and the lack of general project management skills. The OP ACD evaluation identified some key lessons in terms of programming, related to identification of a set of strategic projects for each category of beneficiaries, to cover all specific objectives of the OP. The need to better calculate the target values for the programme indicators was also identified. Table 1. ESF Ex post evaluation, main figures by policy theme
Human Access to Social Human Human Capital Employ- Inclusi Capital Capital SIC Total & PP ment on (PA 1) (PA 2) (PA 3) (PA 4 & 5) (PA 6) Allocated in million EUR (EU + national 992 991 520 779 644 245 4,334 funding)
of which EU (%) ------85 Participations 207 240 303 393 210 70 1,423 (thousands)
Women % 63 54 57 56 71 61 58
Youth (16-24)% 33 44 15 28 22 5 24
Unemployed % 0.3 0.4 1.3 64 16.5 - 20
Inactive % 74 69 10 27 42 - 36
Disadvantaged% 49 43 26 54 66 5 42 Low skilled (ISCED 1- 28 43 11 46 40 - 30 2)%
Results Participants in 22,612 8,736 12,377 43,725 employment Participants gaining 114,842 1,746 119,169 18,885 38,023 52,228 344,893 qualification Products 182 182
Entities 1,137 1,228 278 2,643
Source: SFC2007, Author’s calculation
1 Challenges in the Capacity of Public and Private Structural Instruments Beneficiaries, March 2011
15 Country Synthesis Report – Romania
Introduction This country synthesis report for Romania is part of Evaluation Task 2B: National level analysis of the ESF 2007-2013 Ex post Evaluation Synthesis. The purpose of the present report is to summarise the most recent and most robust evaluation evidence available, analyse the most recent data available and provide a succinct overview of the outputs, results and impacts of the ESF under the main ESF Priorities as defined in the ESF Regulation 2. These are generally: increasing adaptability of workers, enterprises and entrepreneurs with a view to improving the anticipation and positive management of economic change enhancing access to employment and the sustainable inclusion in the labour market of job seekers and inactive people, preventing unemployment, in particular long-term and youth unemployment, encouraging active ageing and longer working lives, and increasing participation in the labour market reinforcing the social inclusion of disadvantaged people with a view to their sustainable integration in employment and combating all forms of discrimination in the labour market enhancing human capital promoting partnerships, pacts and initiatives through networking of relevant stakeholders, such as the social partners and non-governmental organisations, at the transnational, national, regional and local levels in order to mobilise for reforms in the field of employment and labour market inclusiveness And, additionally in Convergence regions: expanding and improving investment in human capital strengthening institutional capacity and the efficiency of public administrations and public services at national, regional and local level and, where relevant, of the social partners and non-governmental organisations, with a view to reforms, better regulation and good governance especially in the economic, employment, education, social, environmental and judicial fields The ESF achievements under each Priority have been evaluated in the following sections: Extent to which resources were used, Effectiveness, Efficiency/cost-effectiveness, Socio- economic impact, Community Added Value (CAV), Gender sensitivity and Sustainability. For this country synthesis report, existing reports have been used emanating from EU level evaluations, in particular the recent thematic ex-post evaluations on Human Capital, Social Inclusion and Access to Employment and the reports of the ESF, the expert evaluation network (EEN) as well as national evaluation material and other relevant reports. This information has been updated by the AIRs 2014 in the case of this report. Given the reliance of the country synthesis reports on existing material for the most part, there is more detailed information on some policy fields than others. This is the case for the chapters on Human Capital, Access to Employment and Social Inclusion. Given the financial volume, however, of the Priorities covered thoroughly by the thematic ex post evaluations, this difference seems acceptable. For example, 72.6% of the total OP resources are used in the Priority Axes (PA) concerning Adaptability, Human Capital (HC), and Social Inclusion (SI).
2 REGULATION (EC) No 1081/2006 OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL of 5 July 2006 Article 3, Scope of assistance
16 Country Synthesis Report – Romania
1 Background and Context 1.1 Challenges and role/position of ESF 1.1.1 Challenges faced by the country in the different policy fields at the time of programming and during the programming period The Romanian labour market was not affected by the crisis as much as other European countries as it managed to keep certain stability throughout the economic recession. 3 In 2008 Romania’s GDP growth rate reached 7.3%, and dropped the following year to 6.6%. The unemployment rate remained roughly the same from 2006 to 2014, slightly growing during the crisis from 5.6% in 2008 to 7.1% in 2013, then falling to 6.8% in 2014, which is considerably lower than the EU-28 average of 10.2%. Also long-term unemployment increased to some extent during the crisis, although staying below the rate of 2006 (4.1%), while in 2014 it was down to 2.8%.4 The youth unemployment rate continued to be high throughout the programming period 2007-2013, with the employment and unemployment issues remaining unchanged.5 Some of the main challenges in Romanian employment are informal work (there are several million people involved in undeclared work) and labour migration, especially for the youth. Moreover, about 30% of the Romanian workers are engaged in the agricultural sector, with low paid jobs, while the industrial sector is in decline and the service sector is not particularly significant. The low skilled employment rates are satisfactory throughout the period (55% compared to EU-28 51.8% in 2014), while many people are engaged in posts at a lower level than their education, indicating the abundance of jobs that require low qualifications and a shortage of higher qualification job opportunities. This is reflected in the consistently low levels of employment for higher educated people – the employment rate of recent graduates is 66.2% (2014), which is much lower than the EU-28 average of 76.1%. During the crisis the number of people at risk of poverty or social exclusion decreased from 45.9 in 2007, although the rate is still the second highest in EU (40.2% compared to the EU-28 average of 24.5% in 2014).6 SOP HRD did not design specific measures for Roma persons, but included them as target group members belonging to vulnerable persons. PA6 – Promoting social inclusion is the only priority axis that addressed Roma directly, through KAI 6.1 – social economy and 6.2 – social inclusion, while Roma children at risk of school abandon were targeted by KAI 2.2, as part of all children at risk of school drop or early leaving school. Roma women, although supporting multiple discrimination, were not directly addressed by the OP, but included in the general group of disadvantaged women targeted by KAI 6.3. The ad-hoc evaluation of SOP HRD intervention on Roma persons highlighted the need to design integrated measures to address all issues Roma persons are facing in Romania. Regarding the objective regarding the social inclusion of Roma, due to the scarcity of public resources characterizing the whole period between 2009 and 2013 (with 2009 being the first year when the OP actually started significant operations) one may consider that ESF had a major contribution to the social inclusion of Roma persons. The same may be said with regard to the training of social assistance staff which would not have been possible on the same scale without ESF funding. According to the thematic Ex-post evaluation on Social Inclusion, the OP addressed adequately the needs of the Roma population, identified as the most disadvantaged group in terms of access to the labour market, education and training, etc. Out of the overall PA 6 target of achieving 150,000 beneficiaries through actions/interventions, the
3 ESF 2007-2013 Ex-post evaluation: investment in access and sustainable integration into employment – National Template Romania 4 Eurostat 5 ESF 2007-2013 Ex-post evaluation: investment in access and sustainable integration into employment National Template Romania 6 Eurostat
17 Country Synthesis Report – Romania planned target for Roma was of 65,000 persons. This represented approximately 15% of the Roma population, which at the time was estimated roughly at around 450,000. This was also in line with commitments made under the NRPs for the period which stated as a priority the betterment of the social and especially labour market condition of the Roma, their access to education (with special regard to Roma children) etc. 7 The rate of young people not in employment, education or training (NEET) steadily increased from 13.2% in 2008 to 19.6% in 2014, which is substantially higher than the EU-28 average of 12.4%. Before the crisis, the early school leaving rate was very high and diminished progressively up to 15.9% in 2008. During the crisis this rate rose to 19.3% in 2010, and remained high (18.1% in 2014). The difference in gender is not very significant, in fact male students have a slightly higher early school leaving rate than female. 8 Not only newly graduates and the higher educated struggle to find a job. The same applies also to older job seekers who lost their job and struggle to find new employment, in addition to weaker groups such as the disabled and the Roma community, which account for some 2.5 million people. 9 Tertiary education attainment gradually increased since 2005, reaching 10.4% in 2007 and 13.5% in 2014. However, it remains the lowest in EU-28 (average 25.6% in 2014). The rate of tertiary education attainment of the age group 30-34 (22.9% in 2013) is still below the national target of the Europe 2020 strategy of 26.7%, and significantly lower than the Europe 2020 target of 40%, although it rose by 9 percentage points in this programming period. The share of women is somewhat higher than men, 24.2% compared to 21.6% (age group 30-34, 2013).10 1.1.2 CSR addressed by the ESF In the context of the Semester, since 2011 the Commission undertakes a detailed analysis of EU Member States' plans of budgetary, macroeconomic and structural reforms and provides them with country-specific recommendations for the next 12-18 months. These recommendations also contribute to the objectives of the EU's long-term strategy for jobs and growth, the Europe 2020 strategy. Before 2011 Member States received country-specific recommendations in the context of the Lisbon Strategy for growth and jobs. Before the beginning of this programming period Romania did not have any Country Specific Recommendations (CSR) made under European Semester. In 2008 they focused on implementing an integrated approach to develop employment, principally by speeding up educational reform procedures, reducing early school leaving, enhancing participation of adults in education and fostering employment in the agricultural sector. Furthermore, they suggested tackling undeclared work and ameliorating the effectiveness and geographical scope of public employment services (PES), with a particular reference to vulnerable groups. Between 2009 and 2013 Romania did not have any CRS therefore the commitments under the Memorandum of Understanding were considered as recommendations.11 In 2009 these addressed the education and training systems in terms of improving their quality and labour market importance, again by reducing early school leavening, and fostering the employability of young graduates through training. In 2011 and 2012 there were no specific recommendations. In 2013, the CSR advised further encouragement for labour market participation, and the employability and productivity of the labour force. In
7 ESF 2007-2013 Ex-post evaluation: supporting the integration of disadvantaged groups into the labour market and society – Country report Romania 8 Ibid. 9 ESF 2007-2013 Ex-post evaluation: investment in access and sustainable integration into employment National Template Romania 10 Eurostat, ESF 2007-2013 Ex-post evaluation: investment into human capital 11 ESF 2007-2013 Ex-post evaluation: supporting the integration of disadvantaged groups into the labour market and society
18 Country Synthesis Report – Romania order to achieve this, labour market policies (LMP) should be revised and intensified. Moreover lifelong learning and training services should be provided. Additionally, recommendations covered: improving the quality and coverage of the National Employment Agency’s services; tackling youth unemployment through the National Plan for Youth Employment; helping vulnerable groups, and especially children, by ameliorating social transfers; strengthening the social assistance reform; addressing the Roma community with the National Roma integration strategy. 12 1.1.3 ESF response to the crisis The design of the programme has remained unchanged throughout the crisis period. The Romanian government believed that the previous design of the programme was meeting the country’s needs for medium term development and was more than adequate for a period of severe economic downturn. The Romanian Government did not alter the general objectives of the SOP HRD in response to the crisis as it considered that they were responding not to short-term objectives (i.e.: reducing budget deficit, improving external financial equilibriums, reining into the salary costs of the public sector and mostly keeping ‘salary developments in line with productivity’) but rather to long-term development needs.13 1.2 Support and Contribution to the Lisbon Strategy and Europe 2020 strategy and targets The main priorities identified in the National Strategic Reference Framework (NSRF) for the 2007-2013 period were stimulating employment, creating good-quality opportunities through active LMPs, tackling informal work in the black market, improving health and safety in the work place, address employment opportunities for the vulnerable groups and preventing poverty and social exclusion.14 After being approved under the Lisbon Strategy, the Sector Operational Programme Human Resources Development (SOP HRD) and Operational Programme Administrative Capacity Development (OP ACD) did not undergo any modification under the EU 2020. 15 The measures implemented by these two OPs reflected adequately the Joint Inclusion Memorandum completed in 2006. This document is contributing to the EU Lisbon Strategy principally according to the objectives on employment, but also on integration into the labour market of vulnerable groups and the inactive through training and vocational training. The OPs were in line also with the Integrated Guidelines for Jobs and Growth (2005-08), for example PA 6 mostly followed Guidelines 17-24 (Guideline 17: attaining full employment, improving the quality of employment and territorial cohesion; Guideline 18: promoting a lifetime approach to work; Guideline 19: ensuring an inclusive labour market and making work pay for job seekers, the inactive and the disadvantaged). The OPs are also involved in the implementation of the Youth Guarantee plan in Romania, promoted by the Ministry of Labour and social partners, they are co- financing two pilot projects targeting the NEETs. 16 Within the Europe 2020 strategy, the national target is to reach 70% employment of the population aged 20-64 by 2020, and a 2% gross domestic expenditure on research and development (R&D). The national target of the Europe 2020 strategy with regard to early school leaving is to reduce the share of early school leavers to 11.3%, and reach 26.7% in tertiary educational attainment (aged 30-34). The ESF OP, as seen before, did not make any amendments under the EU 2020 strategy, thus responding less to the priorities
12 ESF 2007-2013 Ex-post evaluation: investment in access and sustainable integration into employment National Template Romania 13 ESF 2007-2013 Ex-post evaluation: supporting the integration of disadvantaged groups into the labour market and society – Country report Romania 14 ESF 2007-2013 Ex-post evaluation: investment in access and sustainable integration into employment 15 ESF 2007-2013 Ex-post evaluation: supporting the integration of disadvantaged groups into the labour market and society – Country report Romania 16 ESF 2007-2013 Ex-post evaluation: investment into human capital
19 Country Synthesis Report – Romania of EU 2020. According to the HC ex-post evaluation, it was expected from the SOP HRD to be redirected into the EU 2020 priorities and CSR. The ESF Access to employment and sustainable integration into employment (A2E) funding reflected both national and EU themes, concerning all subjects except for some of the NSRF challenges, for example the creation of new job opportunities. Also, not much attention was given to workforce health and safety regulation and undeclared work was tackled only by indirect measures. The ESF HC and SI funding were in line with all national and EU policy directions. 17
17 ESF 2007-2013 Ex-post evaluation: investment in access and sustainable integration into employment
20 Country Synthesis Report – Romania
2 Scale of ESF investment Romania was allocated EUR 3,684 million ESF funding for the programming period 2007- 2013. Including national funding (EUR 650 million) this gave a total of EUR 4,334 million as illustrated in Table 2. Of this total EUR 4,089 million were allocated to the SOP HRD and EUR 244 million to the OP ACD. Table 2 is ordered by PA as it has not been possible to structure it by policy field reflecting the availability of the data. 18 The majority of the funding (about 58%) was allocated to SOP HRD PA 1 Education and training in support for growth and development of knowledge based society, SOP HRD PA 2 Linking life-long learning and labour market and SOP HRD PA 3 Increasing adaptability of workers and enterprises. Public expenditure on Active Labour Market Policy (ALMP) was EUR 345 million (Cat. 2-7) 2007-2013, underlining the significance of the ESF funding in the Access to Employment (A2E) theme that was were allocated EUR 779 million or 18% of the total ESF funds ). Overall, the implementation rate of the two ESF OPs combined was rather low. Of the two OPs, the SOP HRD had a much lower rate (42.8%19) than the OP ACD. The latter was reported to have achieved an implementation rate of 65.8%20 and according to the latest data from the Ministry of European Funds, 81.2%21. OP ACD was a small OP, some 250 million Euro and the main beneficiaries were public institutions. SOP HRD was confronted with numerous problems, the European Commission interrupted payments a couple of times, which made some beneficiaries to cancel their projects. SOP HRD has the lowest absorption rate of all OPs in Romania. Among the main causes of its poor performances identified by the Second Interim Evaluation are: low capacity in design and project management of project beneficiaries; overcharge of MA SOP HRD personnel; a rather complex system for reporting on indicators and project expenditure; general economic situation influenced by the financial and economic crisis. The highest implementation rate was achieved by PA 1 of the OP ACD Improving structure and processes of public policies cycle management, at 71.4% in 2014. OP ACD PA 2 Improving quality and efficiency of public services, with a focus on decentralisation had an implementation rate of 62.8% in 2014, while the highest implementation rate in SOP HRD was PA 3 at 60.3% in 2014. The field with the lowest implementation rate was SOP HRD PA 4 Modernisation of Public Employment Service at 27.4% by the end of 2014. PA4 was addressed to Public Employment Services and certain legal provisions related to the possibility of civil servants to work with EU funded projects prevented PES employees to participate in projects and caused delays in implementation. According to the SCF 2007 data, about 44% of the allocated financial volume was spent by the end of 2014.
18 Policy field Adaptability regards sub-priorities 2.3, 3.1 and 3.2; policy field Human Capital concerns priority axes 1, 2.1 and 2.2; policy field Access to Employment regards priority axes 4 and 5; Social Inclusion regards priority axis 6.1, 6.2 and 6.3; Promoting Partnerships concerns priority axes 3.3 and 6.4. OP ACD entirely regards Strengthening Institutional Capacity. 19 SOP HRD AIR 2014 20 OP ACD AIR 2014 21 http://old.fonduri-ue.ro/res/filepicker_users/cd25a597fd- 62/rezultate/std_abs/Anexa%201%2031%20iulie%202015%20cu%20DLC%20trimise%20la%20CE%20si%20p lati%20CE%20.pdf – accessed at September 10, 2015
21 Country Synthesis Report – Romania
Table 2. Financial allocation and implementation rate of the ESF programming period 2007-2013
Financial Share of total Implementation OP Policy field Allocation funding the OPs rate (in %)
SOP HRD 1 Education and training in support for 991,788,814 22.9% 49.7 growth and development of knowledge based society* SOP HRD 2 Linking life-long learning and labour 991,397,184 22.9% 28.6 market* SOP HRD 3 Increasing adaptability of workers and 519,656,411 12.0% 60.3 enterprises* SOP HRD Access to Employment 778,897,763 18.0% 49.9 (PA 4 and 5 of the SOP HRD) SOP HRD Reinforcing Social Inclusion (PA 6) 644,007,986 14.9% 40.3
OP ACD Strengthening institutional capacity 244,708,967 5.6% 65.8 (all PAs of the OP ACD) Total SOP HRD 4,089,358,714 94.4% 42.8 Total OP ACD 244,708,967 5.6% 72.0 Total SOP HRD & OP ACD 4,334,067,681 - 44.1
*PA 1, 2 and 3 (except sub-priority 3.3 which is related to PP) are related to the Human Capital theme, but no data on sub-priority level was available). Source: SFC2007, AIR 2014
22 Country Synthesis Report – Romania
3 Effectiveness by Priority 3.1 Outputs and results 3.1.1 Outputs In total, there were 1,423,443 participations22 in the two ESF OPs from 2007-2013: 1,345,713 in the SOP HRD and 70,440 in the OP ACD. The largest share of participants (28.2%) were supported under the policy field of Access to Employment. Overall, 48% of the output indicators exceeded the final target set. The policy fields Access to Employment and Strengthening institutional capacity (SIC) were above the average (71% and 60% respectively), while in Social Inclusion only 1 out of seven indicators and in the Human Capital theme 40% reached the target. The following figures provide an overview of the employment and educational status of the participants entering the SOP HRD activities over the programming period. The largest group (37.6%) of all participants were employed, and the self-employed the smallest group at about 3%. The second largest group were the inactive, with 21%, which together with the inactive in education and training (16.6%) reached 37.6%. The percentage for the unemployed was 13.7% which together with the long term unemployed reached 21.4%. Figure 1. Labour market status of participants in ESF 2007-2013 (entering)
Employed 16,6% Self-employed 37,6% Unemployed 21,0% LTU
Inactive 13,7% 3,3% 7,7% Inactive in education/training Source: Author’s figures on the basis of SFC 2007 (AIR 2014) As illustrated in Figure 2, the largest group of participants had a tertiary education, 32.1%. Even the share of people with primary or lower secondary education was considerable, 30.3%, while 26.8% of the participants had a primary or lower secondary education.
22 As one person could participate more than once, the term `participations` is used here. In the rest of the document, `participants` is used, but may include double counting of individuals.
23 Country Synthesis Report – Romania
Figure 2. Education status of participants in ESF 2007-2013 (entering)
Primary or lower secondary education (ISDEC 1 and 2) 32,1% 30,3% Upper secondary education (ISCED 3) Post-secondary non tertiary education (ISCED 4) Tertiary education (ISCED 5 4,2% 26,8% and 6)
Source: Author’s figures on the basis of SFC 2007 (AIR 2014) The Romanian SOP HRD, regarding the vulnerable groups, targeted mainly young people, with a significant share of 24%, although the largest group of participants were listed under the ‘other’ category, such as offenders and ex-offenders, homeless people, refugees, participants facing problems with addictions, etc. (see Figure 3). Among the vulnerable groups participating, the share of minorities is 7.7%, in fact the OP targeted the Roma community, while the share of older people is low (5.4%), and the disabled at 1.7% are the smallest group. Roma were especially targeted under PA2 where about 66,000 participated in the measures related to ESL. Furthermore, about 5,000 Roma people participated in programs of qualification / retraining for vulnerable groups of the Social inclusion theme. Figure 3. Vulnerable target groups participating in ESF 2007-2013 (entering)
35% 32,2% 30% 24,1% 25% 20% 15% 10% 7,7% 5,4% 5% 1,7% 0,0% 0% Young people Older people Migrants Minorities Disabled Others (15-24 years) (55-64 years)
Source: Author’s figure on the basis of SFC 2007 (AIR 2014) Looking at the share of participants by policy field in Figure 4 below, we find that the participants are well distributed across the policy fields, as about a quarter of the participants was supported either by the policy Increasing adaptability, Enhancing human capital or Access to Employment. Fewer people were supported within the policy fields Promoting partnerships and Strengthening institutional capacity.
24 Country Synthesis Report – Romania
Figure 4. Participants by policy field
Increasing adaptability 2% 5% Enhancing human capital 23% 15% Access to Employment
Reinforcing Social Inclusion 28% Promoting partnerships 27%
Strengthening institutional capacity Source: Author’s figure on the basis of SFC 2007 (AIR 2014) Looking at the outputs by policy field, with reference to Table 8 and 9 in the Annex, the following must be noted: Enhancing human capital This is the largest theme of the ESF 2007-2013 in Romania. Overall, 724,929 people were supported under this theme. In terms of target groups PA 1 and two sub-priorities (named Key Areas of Interventions (KAI) in Romania) covered mostly inactive people who are in education/training (e.g. pupils, students), of which about 33.4% are between 15 and 24 years, and teachers. With regard to Early School Leaving (ESL), 82,590 students at risk of ESL and 3,063 early school leavers participated in the KAI 2.2. According to the AIR 2014, about 66,000 Roma people participated in the measures of PA 2, out of which 98% participated in the projects related to ESL. The two most successful interventions regarded support to universities (with an extraordinarily high 3,100% success rate, as it seems that participations or final recipients were counted) and training staff of the educational system (250% of the set target). The final values for the indicators of the interventions KAI 2.1 and 2.2 related to the decentralization and the improvement of the quality of efficiency of public services are expected to increase by the end of 2015, due to finalisation of projects targeting these specific indicators, contracted in 2014. 23 The participants of the action 2.3 ‘Access and participation in continuous vocational training (CVT)’, 3.1 ‘Promoting entrepreneurial culture’ and 3.2 ‘Training and support for enterprises and employees in order to promote adaptability’ are mostly employed (90%) and have tertiary education (48%). Young people represent 15% of the participants, while older people only (6.8%). Overall, according to the needs analysis regarding education and training in Romania published by the Ministry of Education in March 2013, ‘Participation of adults age 15-64 to vocational training in Romania is still at minimum level according to the European standards – 1.6% in 2011, compared to EU average of 8.9%’. Overall, implementation rates showed mixed results in this policy field. While the number of students in the management and organisation of work showed good results in terms of participation (more than 34,000 out of the planned 15,000 participated), other interventions like ‘second chance’ education (16.7%) and CVT (15.3%) programmes were far from reaching the targets set, as it is illustrated in Table 9 in the Annex. These
23 SOP HRD 2014 AIR
25 Country Synthesis Report – Romania figures are expected to rise by the end of 2015, due to finalisation of projects targeting these specific indicators, contracted in 2014. Enhancing access and sustainable integration into employment This was the second largest policy field of the ESF 2007-2013 in Romania, with 393,111 participations recorded. By way of comparison the total number of 389,000 participants had been recorded under ALMP measures within same period. The activities undertaken were intended to be twofold: first, to support employment agencies to train their staff and to improve services and offer; second, to support the unemployed (63.7%) and inactive people (26.6%) to improve their chances to get a job. The majority of participants had finished only primary or lower secondary education (46.2%), but there was also a large number of people with upper secondary education (33.9%). Almost all the output indicators were exceeded for PA 4 Modernisation of PES, with 65024 supported employment services (instead of targeted 200), 6,017 people trained (planned 3,000). A large number of long-term unemployed (63,480 vs. targeted 65,000) participated in the integrated programmes of PA 5 Promoting Active Labour Measures. The output indicator regarding the participation of people from rural areas had a low (64.7%) implementation rate, with chances of improvement by the end of 2015. 25
Reinforcing Social Inclusion The activities under this policy field were about training, qualification and requalification of vulnerable groups, together with activities directed at the development of social economy structures and their support. During this programming period about 210,444 people participated in the interventions of this policy field, out of which 42% were employed and 42% were inactive people.(41.8%) Of the participants 20.5% are minorities, including particularly Roma communities that have been the target group for this policy field. More than 40% of participants have only primary or lower secondary education (ISCED 1 and 2). All output indicators are under-achieved, with the exception of Number of participants in training programmes for professionals in the field of social inclusion where 16,476 instead of 10,000 persons were trained.26 The situation for some underperforming indicators is noteworthy. The indicator Number of participants in programs of qualification / requalification for vulnerable groups which has only reached 34% with poor chances of improvement. The situation for the sub-indicator counting Number of participants (Roma) in qualification programmes / retraining for vulnerable groups, which has reached only 8% of the established target, is even more acute. One of the main reasons for this heavy under-achievement is the unrealistic setting of the targets27. Promoting partnerships The activities undertaken within this policy field are directed at enhancing partnerships between various entities, thus the majority of the participants are employed and have a tertiary education. This is also the reason why the participation of young and older people is very low. There are two output indicators for this intervention, Number of social partners and NGOs supported, that has been achieved with an implementation rate of 285% of the set target, and Number of trans-national initiatives and partnerships
24 The supported agencies were 85 – 42 at county level, 42 and local leve land 1 at national level 25 SOP HRD AIR 2014 26 SOP HRD AIR 2014 27 Ad-hoc evaluation of SOP HRD intervention for Roma people
26 Country Synthesis Report – Romania supported, which had a very low achievement rate, with nine initiatives supported instead of the targeted 120. 28 SOP HRD Programme indicators were not compulsory for the projects submitted during the first calls for proposals, which affected the achievement of many of the programme indicators. This was tackled starting with 2014 calls for proposals, when there were set minimal targets for projects programme indicators. One of the reasons for the high under achievement related to the output indicator Number of trans-national initiatives and partnerships supported is also due to the low number of calls for proposals for this specific KAI (4 out 180+ until the end of 2014). According to the 2014 AIR, there are chances to improve this indicator, when all projects will be finalised. Strengthening Institutional Capacity This policy field is covered by the OP ACD, making the point that this policy field is considered a high priority for Romania. A total number of 70,440 people were involved in this intervention which targeted the employees of central and local administration. Output indicators illustrate diverging results regarding the achievement of the final targets as they count number of training hours per participant and not directly participants.29 The output indicator Number of training hours per participant has been defined for each intervention, with different targets and various degrees of accomplishment. As regards entities, all structures in public administrations are targeted by OP ACD, but there is no indicator counting number of public administration structures participating or involved in OP ACD activities.
28 SOP HRD AIR 2014 29 EEN, 2014, Final country report Romania & OP ACD AIR 2014
27 Country Synthesis Report – Romania
3.1.2 Results Overall, 36% of the result indicators reached or exceeded the targets set. The share of result indicators achieved was highest for Social Inclusion (83%), and lowest for SIC (22%). Aggregation of results People in employment: 43,725 People gaining qualification: 344,893 Other positive results: - Products: 182 Entities: 2,643 Enhancing human capital Table 10 in the Annex shows that the result indicators of this policy field illustrate a low implementation rate of the interventions regarding results. The only result indicator which has been overachieved was Share of enterprises providing CVT to their personnel with 24.1% against the targeted 20%. The percentage of students certified either in the management and organisation of work or for updating and improving skills was very close to the target with an implementation rate of about 95%. However, the share of trainees that set up a new business is very low, 12% compared with the targeted 35%, due to the economic situation that does not favour the setting up of new businesses, and the increase of the base number of participants, with poor chances of improvement. The share of schools that received accreditation according to the new quality assurance standards was only 36% of the target and the share of PhD students which graduated was 33% compared to the targeted 90%. Also the share of student support on transition from school to work (36.9%) and the graduates of the ‘second chance’ programme were highly under-achieved – (22.7%). According to SOP HRD AIR 2014 and based on projects under implementation, it is likely that a higher degree of achievement for these indicators by the end of 2015 will be achieved, but remaining still below target. The only over-achieved result regarded the universities having received accreditation according to the new quality assurance standards, with 111% as against the targeted 90%. Enhancing access and sustainable integration into employment According to the result indicators only 33% of the indicators (for which a target was set) reached or overachieved the final target. An important indicator was the Share of certified participants from the rural areas who obtained employment, as the crisis significantly reduced the available employment opportunities in these areas. The indicator reached 9%, while the target was set at 15% with 68% of the long-term unemployed which participated in training obtained a certification. The indicator Share of employment agencies certified in quality management will reach 95% of the set target by the end of 2015, based on projects under implementation. The result indicators regarding agencies offering self-services and the staff who got a certificate were overachieved (115% and 113% respectively). Reinforcing Social Inclusion The result indicators of this policy field show very good results. The number of jobs created by social economy entities were 12,377 (targeted 5,000), and the share of participants in training/retraining programmes for vulnerable groups who obtained certification was 58% (out of planned 55%). The share of Roma people who obtained a certification in the training/retraining programmes was 86% (targeted 60%) and the share of disabled who obtained a certification was 63% (targeted 60%); while the indicator regarding specialists in SI was the only one that underachieved.
28 Country Synthesis Report – Romania
Promoting partnerships In this policy field there was only one result indicator, share of social partners and NGOs support that provide community services, and it over-achieved with an implementation rate of 172%. Strengthening Institutional Capacity Regarding this policy field, it appears that the indicators were not clearly defined at the beginning, with targets either over or under-estimated.30 The indicator Share of public policies and legislative acts rejected as unsatisfactory by the Secretariat General of the Government was based on external measurement by the Secretariat. The ministries do not collect data related to this indicator, but overall there is a tendency of reducing the number of rejected acts as it overburdens the administration and produces delays. This indicator proved to be irrelevant and difficult to be measured accurately31. The administrative costs were reduced by 15.3% (targeted 20%), measured following the implementation of five projects aimed at simplifying administrative costs.32 The share of public institutions that produce an annual report on transparency is 88.6% (targeted 80%). Regarding the Number of monitoring and evaluation reports prepared/required by central public institutions, the indicator is under-achieved. According to targets set in the projects under implementation, this indicator will reach 92,6% of the 2015 target, achieving better correlation between the number of guides and methodological documents and the number of monitoring and evaluation reports. An indicator that had a high achievement level was Number of new set-up and operational reorganised structures, with 3,246 structures compared to the target of 287. This was due mainly to one project which created 3,223 structures33 all over the country. The indicator Number of decentralised operational (reorganised and new set-up) structures (55.6% out of the set target at present) will reach 67% by the end of the programming period34. 3.2 Evidence of effectiveness Generally, the effectiveness of the ESF intervention in Romania may be assessed as low. SOP HRD did not provide impact indicators, which makes it difficult to assess the longer term results. However, currently, the OPs are still under implementation, with high expectations regarding the achievement of programme indicators and financial execution based on last year’s efforts of the Managing Authority (MA). According to SOP HRD second interim evaluation, this is due to several factors that influenced the OP implementation and achievement of programme indicators – effectiveness. Among these factors, the evaluation mentions – the low capacity of project beneficiaries to design and implement projects; the rather complicated system of SOP HRD indicators, which proposed three types of indicators – programme indicators, supplementary indicators and additional indicators, without setting clear rules on which indicators were mandatory. Moreover, the indicators were not well explained, a Guide on SOP HRD indicators was produced only in January 2012, when the great majority of projects were under implementation. Another factor impeding the effectiveness of the OP was the rather poor rhythm of financial reimbursements, which affected the project beneficiaries cash-flow and ultimately the quality of projects results. Nonetheless, given the moment of writing of this report and of the evaluations taken into consideration, as well as the cut-off date of the last AIR, the situation of the SOP HRD indicators may improve, with an impact on the overall effectiveness of the OP. Increasing adaptability The implemented activities and measures of this policy field targeted mainly employees and management staff. The employees underwent training in order to develop their skills
30 EEN, 2014, Final country report Romania 31 OP ACD AIR 2014 32 Idem 33 Child protection and public social protection public services (OP ACD 2014 AIR) 34 OP ACD AIR 2014
29 Country Synthesis Report – Romania and general competences. Judging by the high participation into this policy field and by the contracted amounts (145%)35, this proves to be one of the effective measures. There is no additional information on the performance of the employees who participated in the training and improved their qualification, or on the people who started a new business. Enhancing human capital This policy field, through PA 1, supported long life learning and access to education and training, targeting students of different levels, teachers and staff connected to the educational system. Through the key areas of intervention 2.1 and 2.2 it also supported students entering the labour market and students at risk of abandoning school. The prioritised targeted groups were women and minorities, such as the Roma community, and the activities included innovative approaches in education and lifelong learning. Based on the general achievement of indicators, although its contracted value is 137.8%36 out of the allocated funds, this policy field was not very effective. PA 1 is a very complex area of intervention, addressing as target groups kindergartens, schools, universities, children, students, teachers, educators and PhD students. As a consequence, the degree of effectiveness, measured by the degree of accomplishment of programme indicators varies from 3112% universities supported to 4% training providers supported in introducing quality standards. In this latter case, the legal framework in the field is not yet in place. A special mention is to be made here for KAI 2.2 – Preventing and correcting early school dropping which registered good results in relation to Roma children participation to education. The related programme indicators do not count Roma persons, but the Ad-hoc evaluation of SOP HRD intervention on Roma has found out that KAI 2.2 has been an effective measure for Roma children. Access to Employment This policy field targeted PES employees and structures under PA 4 and the unemployed (with a focus on the rural areas) under PA 5. One of the evaluations37 assessed that the activities in this policy field were too broad, and did not focus enough on actively addressing the problems of the vulnerable groups. The activities were then reduced and divided in order to better focus on the expected results. The final report on ESF achievements 2007-2013 of the Expert Evaluation Network (EEN) states that one of the weaknesses in assessing the effectiveness of the SOP HRD was the design of the indicator system, which was not well understood by the beneficiaries. 38 According to the forecast on indicators achievement made by the second interim evaluation of SOP HRD, based on the latest reporting and on all projects under implementation, there is a good prospect of the overall effectiveness of Access to Employment policy field improving, especially regarding the long-term unemployed and the participants from the rural areas. PA 5 is contracted 154.2%39 of the allocated amount for 2007-2015. Reinforcing Social Inclusion When looking at the result indicators of this policy field, it is clear that most of them were achieved, and some even over-achieved. Altogether, regarding the achievement of indicators, PA 6 of the SOP HRD 2007-2013 was effective. However regarding the activities concerning social economy structures and creating of jobs, it can be found that they are not supported by a parallel legislative framework, causing delays in the implementation and under-achievement of related targets. 40 PA 6 has been contracted 164% of the total allocated amount41.
35 Second intermediate evaluation OP HRD 36 SOP HRD 2014 AIR 37 First Interim Evaluation of SOP HRD, final report, June 2011. 38 EEN, 2014, Final country report Romania 39 SOP HRD 2014 AIR 40 EEN, 2014, Final country report Romania 41 SOP HRD 2014 AIR
30 Country Synthesis Report – Romania
Regarding the effectiveness of the interventions assessed in-depths, the following can be stated: Projects displaying a high share of vulnerable groups as part of their overall target groups have been the most effective. They have reached the individuals in need of help while making maximal use of their resources. Thus such projects tend also to have monthly cost per participant below the average. This is the case for projects as: ‘Jobs: Investment in people’, ‘the Centre for Alcohol addicts’, ‘Adaptability and access on the labour market’. In such cases what paid off was a certain type of economy of scope achieved by the projects and that is an economy of scope that concentrated on the vulnerable, or in other words effectiveness has been the product of a ‘focused economy of scope’ achieved by the projects. In some cases, these are the same projects that achieved economies of scale (large numbers of participants) which consequently made them also the most efficient of projects in the sample.42 Strengthening institutional capacity Regarding this policy field, the main target groups were the central and local public administration and its staff. The output and results indicators were achieved unevenly. Also, as a result of the initial design of the indicators system the targets were set too high. In the second interim evaluation of the OP ACD it is stated that the level of effectiveness cannot be clearly defined as the OP covers interventions with intangible effects and the complexity of assessing them through quantitative data is excessive.43 The MA of the OP ACD performed an evaluation of performance of the OP’s management and implementation, focused on the effectiveness of the OP interventions. The conclusions were that PA 1 of the OP registered a higher effectiveness than PA 2. All the interventions contributed to the achievement of the specific objectives, although the projects focused more on immediate needs than at strategic approaches. OP ACD had a contracting rate of 123.6%.
42 ESF Ex-post evaluation on Social Inclusion - Country Report Romania 43 EEN, 2014, Final country report Romania
31 Country Synthesis Report – Romania
4 Efficiency by Theme With reference to efficiency, it appears that the projects with the larger number of participants had lower cost per participant per month, while in the activities with fewer participants the cost was much higher. This is also due to the absence of standards for costs per activity and per participant. The ‘Ex-Post Social Inclusion Country Report of Romania’ stresses the need to shorten the duration of interventions, in addition it suggests the need to better conformity to the estimation of cost, duration and number of participants. Therefore, efficiency was attained best in those projects where economies of scale were achieved, hence in the activities having a large number of participants (in general above 500 participants). In these projects the costs were kept low and resources were put to better use. Projects that obtained good efficiency are for example FEMINA, the Centre for Alcohol addicts, Job-Investment in People or Adaptability and Access on the Labour Market. In all these projects (except for FEMINA) the number of participants was above 1,000 and the costs per participant per month were below the average. 44 This comment is based on a reduced sample of projects and does not stand as a general efficiency rule for ESF funded projects. The general conclusion of the last SOP HRD interim evaluations related to efficiency was that the efficiency of measures was difficult to be established without a project budget designed on activities. This was also identified by the Managing Authority who introduced a more detailed budget starting with 2013. Information about efficiency and cost effectiveness was not available for each priority. In PA 6 data about efficiency was available only with regard to some implemented initiatives. Table 3 shows the actual expenditure on each of the analysed interventions in the Ex-Post Evaluation (2007-2013) Supporting the integration of disadvantaged groups into the labour market and society. Table 3. Cost per participant/participation of some interventions
Actual expenditure OP Project Name Cost per participant (EUR) (EUR) SOP HRD Developing skills and 1,488,632 2,102 qualifications for women SOP HRD Counselling and training for the 1,409,512 3,748 unemployed SOP HRD ‘Lifeguards and divers’ for the RO 3,718,218 Individuals trained 1,731 economy Trainers trained 4,816 SOP HRD Improving access of youngsters 1,571,193 11,222 in detention to vocational training and Labour market integration SOP HRD Centre for HR development in 1,344,233 2,036 the rural areas SOP HRD Infomation and professional 504,147 Participants trained 666 counselling for the roma Dependants supported 315 SOP HRD Innovative activities aimed at 1,388,039 33,048 facilitating labour market integration for groups at a disadvantage (COMPETENT) SOP HRD Adaptability and access to the 675,680 401 labour market through training and active job search SOP HRD Co-operation for the facilitation 1,297,641 7,415 of employment amongst the blind via ICT skills
44 Ex-Post Evaluation of 2007-2013 Social Inclusion Country Report Romania
32 Country Synthesis Report – Romania
Source: Ex-post evaluation of 2007-2013 Supporting the integration of disadvantaged groups into the labour market and society The ad-hoc evaluations of SOP HRD interventions on youth and Roma have also assessed the efficiency of interventions and the cost effectiveness. Thus, it is stated that SOP HRD did not set reasonable costs per activity and/or participant, according to the market, nor imposed standard packages of activities per intervention dedicated to youth. The proposed costs were entirely at the discretion of project beneficiaries, which led to a significant variation in unit costs. The situation is similar in SOP HRD interventions dedicated to Roma people. It was not possible to clearly determine which measures have been efficient/cost effective, as there have not been clear indicators set for each measure and there was not an explicit and transparent system to associate costs with indicators. Moreover, overall costs are burdened by heavy administrative costs due to excessive bureaucracy in reporting. With reference to OP ACD and policy field Strengthening Institutional Capacity, the second interim evaluation stated that ‘The programme is, to a great extent, highly efficient, from the cost point of view, considering the fact that with a similar level of investment, generated more immediate effects than initially envisaged’. The table below illustrates the cost per output. As mentioned in chapter 2, data are only available on PA level, which means that cost per output by theme is only available for A2E and SIC. The theme Access to Employment has the lowest costs per participant and its unit cost is also lower than the cost per ALMP participant in the same period (EUR 1,137). Table 4. Cost per participation by PA / theme
Cost per output PA / theme (in EUR) 1 Education and training in support for growth and development of knowledge based society 2,386 2 Linking life-long learning and labour market 1,182 3 Increasing adaptability of workers and enterprises 1,034 Access to Employment (PA 4 and 5 of the SOP HRD) 989 6 Promotion of Social Inclusion 1,234 (all PAs of the OP ACD) Strengthening institutional capacity (all PAs of the OP ACD) 2,285 Total SOP HRD 1,285 Total OP ACD 2,285 Total SOP HRD & OP ACD 1,334
Source: SFC2007
33 Country Synthesis Report – Romania
5 Sustainability Sustainability of result for individuals and entities None of result indicators were related to the sustainability for individuals and entities. Sustainability of interventions According to the Romanian SI report, in terms of sustainability, all projects, and all beneficiaries are committed by contract to continue their services for at least three years after the ending of the financial agreement. Moreover, sustainability is the subject of a separate chapter in the project content, covering three dimensions: financial sustainability of activities and results after the closing of financing; institutional sustainability, in terms of maintenance of structures created through the project activities and partnerships; policy sustainability, in terms of supporting development and implementation of internal/external policies in the field. Sustainability was achieved mainly when efficiency and effectiveness were coupled with investment in particular infrastructure projects or when the institutions implementing the activities were able to make the project action long-term and turn it into a mainstream practice. 45 A similar conclusion was reached by the second evaluation of SOP HRD, which emphasises that the sustainability of achieved results depends on the capacity and availability of social partners – at local, regional and national level – to support the interventions through resource mobilisation and promotion of policies and corresponding legislative framework. As regards the interventions funded by OP ACD, the second interim evaluation assessment was that ‘the results obtained create positive effects and, in the long run, are sustainable’.
45 Ex-Post Evaluation of 2007-2013 Social Inclusion Country Report Romania
34 Country Synthesis Report – Romania
6 Gender sensitivity SOP HRD contains some specific objectives on gender issues and most of the projects addressed the gender dimension. There were, however, only two programmes of actions in the SOP HRD explicitly devoted to women in the labour market, while the only one completely dedicated to women is the programme Developing skills and qualifications for women. In the programmes targeting the social sector, such as social workers, volunteers and civil society organisations’ personnel, the target groups the women included formed the largest share. In some of the programmes there was a distinct male domination, especially in the activities regarding the training of lifeguards and divers and prisoners. On the other hand in the projects targeting the Roma community genders were equally balanced, suggesting that in such vulnerable communities, men and women find themselves in the same situation. Furthermore, in certain groups, such as the unemployed and the long term unemployed, men participants were targeted mainly, as in the Romanian labour market more men are unemployed than women. Overall, women were represented on average between 45-55% of the share of participants in all programmes and categories (young, older, long-term unemployed, disabled, etc.). There were no thematic evaluations based on the effectiveness of interventions on women, youth etc. The projects had to observe the equal opportunities theme and to report accordingly. SOP HRD had one intervention (KAI 6.3) whose target group was represented by women and other disadvantaged persons. This was an intervention which was very much accessed by the project promoters and, as a consequence, the financial allocation for this respective KAI has been increased during the OP implementation. This KAI was considered effective especially for Roma women who participated in large numbers to projects activities. Table 5 illustrates the gender balance of the respective policy field. Women are over- represented in all policy fields of the ESF 2007-2013 OPs (only Promoting Partnerships has a share slightly below 50%). The share of women out of the total number of participants is the highest within the policy field Reinforcing Social Inclusion (70.8%). Table 5. Share of women by Priority
Share of women (in % of total OP Priority participants) SOP HRD Increasing adaptability 54.8 SOP HRD Enhancing human capital 59.3 Enhancing access and sustainable integration SOP HRD 53.2 into employment SOP HRD Reinforcing social inclusion 70.8 SOP HRD Promoting partnerships 49.8 OP ACD Strengthening institutional capacity 60.7 Total (SOP HRD & OP ACD) 58.2
Source: SCF 2007; Author´s calculation According to the Gender Equality Index (2012) 46, Romania is significantly below the EU average. Romania has an overall index of 33.7, and is therefore on the last place in gender equality amongst the EU-28 countries. Romania is under-achieving in all of the six components, particularly in ‘money’, ‘knowledge’, ‘power’ and ‘time’, while on the categories of ‘work’ and ‘health’ the indexes are very close to the EU average.
46 http://eige.europa.eu/gender-statistics/gender-equality-index
35 Country Synthesis Report – Romania
7 Community Added Value 7.1 Volume effects In relation to Community Added Value, SOP HRD interventions supported actions which otherwise would not have been possible, mainly due to the lack of appropriate funding and qualified staff. For example, due to the crisis, companies had to cut down their budget allocation to training and funds supporting the County Employment Agencies for Active Employment Measures had been reduced. 47 Regarding volume effects, they were particularly established in the field of Social Inclusion, where the scale of the projects was significant, in fact activities involved from around 200-250 participants to more than 1,000. They regarded numerous different services, as training, counselling, job placement, information provision, information technology (IT) training and access to IT infrastructure, exchanges of experiences etc. and included various disadvantaged groups. 48 One of the projects that presented volume effects is the development of skills and qualification for women, which resulted in the growth of the services offered in rural communities. Another good example concerns the project ‘Counselling and training for unemployed’ where relevant volume effects were registered, also due to the low national funding in these activities. Likewise the project ‘Adaptability and access to the labour market through training and active job search’ produced some volume effects by extending the support to vulnerable groups living in rural areas where the job opportunities and service provisions are limited. Finally the project on lifeguards and divers had a considerable scale that allowed it to acquire particular accomplishments, such as the investment in specialised infrastructure (training centre for lifeguards and divers). 49 7.2 Scope effects As in the case of volume effects, scope effects of the CAV were established mostly for Reinforcing Social Inclusion. It is to be noted that some of the projects targeted only one target group, while others targeted many groups, up to seven. These kinds of projects, which can be defined as ‘scope oriented’, had various target groups in order to be sure to reach their target numbers. As a consequence they met the needs of diverse stakeholders and created partnerships, and by doing this they improved the community capacity to cooperate, also involving disadvantaged groups in the discussion. Particularly, in projects regarding the blind and the mentally ill, innovative services have been introduced, mainly concerning IT systems. 7.3 Role effects There is not enough information available concerning role effects. However, one of the innovative projects regarding counselling and training for the unemployed was successfully mainstreamed for various communities. In fact the proposed practice of integrated service provision for young unemployed could be replicated for the delivery of the Youth Guarantee. 50 7.4 Process effects Process effects were visible in the project regarding the development of skills and qualification for women, where the networks between actors in the rural communities improved considerably. Also the project ‘Adaptability and access to the labour market through training and active job search’ accounted for better partnerships between actors in rural communities. In the projects concerning training for lifeguards and divers process
47 EEN, 2014, Final country report Romania 48 Ex-Post Evaluation of 2007-2013 Social Inclusion Country Report Romania 49 Idem. 50 Idem.
36 Country Synthesis Report – Romania effects were registered as partnership activities between municipalities and expanded remarkably. 51 The OP ACD concerned the reorganisation and training of the public administration, and by doing so the activities addressed those areas where there was not enough government funding or there was none. The action undertaken by OP ACD and the government action developed under the Reform of Public Administration. In the second interim evaluation of OP ACD it is underlined that the created processes ‘aimed at consolidating the institutional capacity of the public administration’. 52
51 Ex-Post Evaluation of 2007-2013 Disadvantaged groups 52 EEN, 2014, Final country report Romania
37 Country Synthesis Report – Romania
8 Socio-economic impact Macro-level Overall the ESF interventions, accompanied by national resources, helped job creation and participation in training activities, with an encouraging impact on productivity, development of skills, participation on the labour market, institutional development and the formation of additional know-how. Looking at the socio-economic trends in Romania over the programming period, it is evident that Romania, although hit by the economic and financial crisis, was not as heavily affected as other European countries. Very little change is notable from the pre- crisis phase, the overall employment rate remained almost the same, rising by 0.3%. Female employment rates remained steady, at the same time the share of NEET increased considerably from 11.6% in 2008 to 17% in 2013. The unemployment rate did not grow significantly. Table 6. Changes in Socio-economic indicators
2007 2008 2009 2010 2011 2012 2013 2014 Trendline Access to Employment Employment rate 58.8 59.0 58.6 60.2 59.3 60.2 60.1 61.0 Unemployment rate 6.4 5.6 6.5 7.0 7.2 6.8 7.1 6.8 Young people NEET, 15-24y. 13.3 11.6 13.9 16.6 17.5 16.8 17 17 Employment rates, Females, 15-64y. 52.8 52.5 52.0 52.5 52.3 52.8 52.6 53.3 Human capital Early school leaving 17.3 15.9 16.6 19.3 18.1 17.8 17.3 18.1 Educational attainment 13.9 16 16.8 18.3 20.3 21.7 22.9 25 Social Inclusion Population at risk of poverty 45.9 44.2 43.1 41.4 40.3 41.7 40.4 40.2
Source: Eurostat The Romanian ESF programme 2007 – 2013 is still on its way of achieving its goals. The main priorities for SOP HRD were also the ones of the European Strategy for Employment. These focus on attracting and maintaining people in the labour market, developing adaptability of workers and enterprises, matching education and CVT systems to new qualifications requested by the market, reducing poverty rate and social exclusion. Meso-level PES had an entire Priority Axis allocated for projects designed to improve the services and their delivery. The National Agency had no coordination of the proposed activities, and given the competitive system of allocation, it was difficult to finance integrated reforms. One ESF funded project developed by the Ministry of Labour developed the law on social economy based on an extensive research. The Ministry of Education also developed several projects intended to support education reforms, but was also facing problems, as some key projects have been rejected during the competitive selection process. Towards the end of the programming period, the MA for SOP HRD proposed systemic projects dedicated to ministries in order to implement social reforms. An interesting result indicator proposed by SOP HRD refers to Share of enterprises providing CVT to their personnel, which registered 120.6% accomplishment. The baseline for this indicator was 11% and the end result 24.1%. SOP HRD has financed 171 companies to provide CVT. According to 2014 AIR, SOP HRD contribution to this indicator (24%) has been 6.55%. This is important, and although this indicator was calculated by the National Institute for Statistics, given the economic situation and the general understanding that companies often do not invest in the improvement of their staff
38 Country Synthesis Report – Romania knowledge and competencies, one can state that this increase is due mainly to SOP HRD funding in the field. As per the Partnership Agreement of Romania, in order to support the challenges in terms of poverty and social exclusion, ‘…a higher involvement of the local authorities in addressing these challenges, which should be correlated with an appropriate investment in building their administrative capacity in applying the appropriate solutions’ is needed. Micro-level The ad-hoc evaluation of SOP HRD intervention on youth highlights an interesting impact of ESF the intervention, especially related to measures aiming to reduce school-drop outs and to support students in the transition from school to work. Young people (15-24) were also very interested and participated in large numbers in measures supporting development of entrepreneurial skills, 90% of young persons that opened a new business supported by ESF declare these are still operational. As regards the impact of SOP HRD measures for the improvement of the Roma situation, the ad-hoc evaluation report concludes a low impact on the short term, with the possibility of improving results based on continuation of similar measures, with a more targeted focus on Roma communities and their needs. As SOP HRD did not propose impact indicators, in order to further understand the impact of the Romanian ESF programmes for the period 2007 – 2013, the National Labour Research Institute was commissioned by the Ministry of Labour to carry out a wide- ranging study that will concentrate on the assessment of all active labour market measures (including the activities co-funded by ESF). This study begun on September 2014 and is expected to be published in 2016. 53
53 Ex-Post Evaluation of 2007-2013 Social Inclusion Country Report Romania
39 Country Synthesis Report – Romania
9 Key lessons learned 9.1 What are the key lessons in terms of policy choices? According to the second interim evaluation of SOP HRD, the needs of the target groups as they have been identified before 2007 and the strategic guidelines of SOP HRD are still relevant. One lesson learned refers to regional approach, as the SOP HRD did not set general objectives at the regional level, nor financial allocations by region. The differences between Romania’s regions are noticeable in terms of socio-economic development and employment, but there were no up-to-date sectoral studies and/or analysis to base any regionally focused interventions in the SOP HRD. This is also valid in terms of SOP HRD interventions related to youth and to Roma people. Moreover, ‘…disparities in the employment rate in Romania's regions have a range of 11.5 [percentage points] between the region with the highest employment rate (North-East: 64.9% in 2012) and that with the lowest employment rate (Centre: 53.4% in 2012)’54. The strategy of the OP ACD continued to be relevant through the 2010-2012 period, considering the fact that the interventions financed reflect existing needs, as identified by current national strategic documents, and this is a strong point of the programme’s logic of intervention. The analysis, developed in order to support the OP ACD, correctly identified the horizontal issues affecting the Romanian public administration, but its lack of focus on types of beneficiaries and target groups led to the elaboration of a programme with objectives which are difficult to quantify, with a low level of prioritization and with an untargeted implementation strategy. The lack of depth of the analysis underpinning OP DAC is the weakest point in the logic of intervention of the programme55. 9.2 What are the key lessons in terms of target groups? The second interim evaluation report of SOP HRD identifies the need to put a sharper focus on interventions for certain categories of target groups, like people with disabilities, children, youth, victims of domestic violence and other vulnerable groups. Although operations targeted a diversity of target groups with a variety of positions in the labour market – employees, managers, people looking for a job, unemployed, long term unemployed, youth, vulnerable persons – the programme activities and indicators did not prioritise the most needy members of the addressed target groups. Youth were not directly targeted by SOP HRD measures, neither were the Roma, and this becomes even more clear with the formulation of the first evaluation question of the ad-hoc evaluations of youth and Roma interventions: Identification of Priority Axes and Key areas of intervention (implicit and explicit) dedicated to youth and to Roma people. In this respect, the Ad-hoc evaluation of SOP HRD intervention on youth recommends to design measures for youth based on a better needs analysis, with a special focus on youth unemployment (unemployment rates for youth varies between 20.9% in Bucharest- Ilfov56 and 25.8% in South Muntenia), institutionalised youth and youth participation in higher education. The Ad-hoc evaluation of SOP HRD Roma intervention also recommends integrated measures addressed to Roma persons, based on sectoral analysis, including access to labour market, support for children at risk of early school leaving and support for Roma women. Roma participation in the SOP HRD measures addressed to disadvantaged groups was low, especially in the regions with a high level of Roma persons (8% in Centru region and 15% in North-West, where there is the higher concentration of Roma persons in the total population of the region). The evaluation of the OP ACD states that ‘The target groups of OP [ACD] are generally relevant for the programme intervention logic, but the lack of individualization, coming out of the supporting analysis, led to their identification in a rather general manner’.
54 Partnership Agreement România 2014RO16M8PA001.1.2 55 Second Interim Evaluation OP ACD 56 Bucharest-Ilfov is a developed region according to EU standards
40 Country Synthesis Report – Romania
9.3 What are the key lessons in terms of appropriate programming? The Second SOP HRD interim evaluation identified several key lessons for appropriate programming: establishing a smaller number of objectives, in order to create focus on specific needs identified based on regional and national sectoral studies ensuring correlation between specific objectives by priority axis and the general objective of the OP formulating operations and eligible activities to clearly reflect the specific objectives and to facilitate attainment formulating programme indicators to cover all specific objectives and to ensure appropriate monitoring of their achievement. The OP ACD evaluation identified some key lessons in terms of programming, related to identification of a set of strategic projects for each category of beneficiaries, to cover all specific objectives of the OP. The need to better calculate the target values for the programme indicators was also identified. 9.4 What are the key lessons in terms of effective implementation? Project beneficiaries had to choose from a rather large set of eligible activities indicated in the Applicant guide; any other activities would result in project rejection or ineligible costs. Following the first interim SOP HRD evaluation, the number of eligible activities was reduced and they were placed in two categories: principal activities and secondary ones. The latter were allowed only to accompany and support activities from the first category. The first calls for proposals did not introduce any prioritisation for target groups or requirement to focus on any specific category. As a result of a big shortfall in accomplishing programme indicators, minimal compulsory values for certain categories of target groups were introduced in 2013 (e.g. long term unemployed, youth, Roma, women etc.). Although the Romanian labour market was not affected by the crisis as much as other European countries, the effects of the crisis were felt in a higher demand in projects dealing with the unemployed. A major effect was felt in the availability of jobs, which rendered ineffective some interventions in terms of counselling and training for the unemployed. There were also other limitations, notably in the implementation mechanism, which affected the effectiveness of the SOP HRD interventions, summarised in the recommendations of the OP, as follows: ‘…persistent deficit of human resources, amplified by high turnover, including at top levels; high complexity and volatility of project implementation regulatory framework; very complicated and unclear indicators system; big gaps between launching calls for proposals’. Although the project beneficiaries declare that they have not encountered difficulties in preparing the project proposals and in the project implementation and general management following the award of grants, the high number of projects rejected (for SOP HRD, the percentage of rejected projects by the end of 2013 was 63%57, with a similar rate for OP ACD), the projects cancelled by the MAs due to bad management and the low absorption rate contradict their opinion. Some of the reasons identified by the SOP HRD Second interim evaluation refer to the large selection of types of beneficiaries addressed by the OP, which did not possess the expertise needed to develop such a demanding project, and to the beneficiaries’ limited (financial and managerial) capacity of implementation.
57 Second SOP HRD interim evaluation
41 Country Synthesis Report – Romania
The low capacity of OP ACD beneficiaries has also been identified by the OP ACD Performance evaluation. Although in the case of this OP the project beneficiaries are mainly public authorities, these have admitted to encountering difficulties in defining the projects objectives, indicators and in preparing a supporting budget. 12.9-16.7% admitted in the online survey that they have seen the OP more as a funding opportunity and less of a means to sort institutional issues. Some of the OP ACD project beneficiaries used external consultants for development of projects (both for proposal writing and for the implementation), which is seen as an opportunity of know-how transfer for the project beneficiaries staff. Both ESF OPs identified a common cause of ineffectiveness, which was also highlighted in a previous system evaluation58: the low capacity of project beneficiaries in project formulation and implementation and the lack of general project management skills. 9.5 What are the key lessons in terms of monitoring? Managing authorities have tended to focus on rather crude statistics for the projects. In most of the cases these have been substantiated by output indicators expressed in numbers of participants from different target groups. The achievement of volume targets qualified a project as successful or unsuccessful. Project managers and management teams were constantly trying to involve additional individuals matching the characteristics of their target groups in their projects. The poor match between the activities performed by the projects, which display a fairly homogeneous structure, and the array of target groups addressed by the sample of projects, which is highly heterogeneous, is an indication of that. This tendency also helps explain why in almost all of the cases volume targets have been reported as attained. The most important aim therefore was to attain, by the end of the project, these volume targets and the main effort was not necessarily directed towards the quality of the processes or even towards the outcomes, but towards reaching the volume targets. This clearly shows that the choice of indicators available for measuring the performance of the projects was inappropriate and inadequate. Reporting and monitoring of activities focused on the attainment of simple volume targets. There was no indication of project effectiveness, efficiency or impact.59 The evaluations of the OP ACD in terms of monitoring system and indicators, identified the need to introduce a monitoring system based on projects’ risk analysis and the introduction of qualitative indicators to allow for better assessing of effects and impact. 9.6 What are the key lessons in terms of evaluation? The Second Interim Evaluation included an assessment of the efficiency of the IT monitoring system, which highlighted the need to integrate the internal SOP HRD management information system (Action Web) with the general information system for the Structural Funds – SMIS (Single Management Information System). There was no impact evaluation planned or conducted. With regard to the OP ACD, the multi-annual evaluation plan was also updated, in 2012, when two management evaluations and an impact evaluation were replaced by a performance evaluation that took place in 2014.
58 Challenges in the Capacity of Public and Private Structural Instruments Beneficiaries, March 2011 59 Ex-Post Evaluation of 2007-2013 Social Inclusion Country Report Romania
42 Country Synthesis Report - Romania
10 Annexes Table 7. Financial allocation and implementation rate of the ESF programming period 2007-2013
Financial Corresponding public Implementation rate OP Priority Allocation contribution (in %)
SOP HRD 1 Education and training in support for growth and development of 991,788,814 492,905,338 49.7 knowledge based society SOP HRD 2 Linking life-long learning and labour market 991,397,184 283,294,596 28.6
SOP HRD 3 Increasing adaptability of workers and enterprises 519,656,411 313,298,311 60.3
SOP HRD 4 Modernisation of Public Employment Service 235,541,719 64,568,864 27.4
SOP HRD 5 Promoting active employment measures 543,356,044 324,243,214 59.7
SOP HRD 6 Promotion of Social Inclusion 644,007,986 259,756,963 40.3
SOP HRD 7 Technical assistance 163,610,556 13,730,657 8.4 OP ACD 1 Improving structure and processes of public policies cycle management 137,037,022 97,824,099 71.4
OP ACD 2 Improving quality and efficiency of public services, with a focus on 97,883,587 61,494,704 62.8 decentralization OP ACD 3 Technical assistance 9,788,358 1,653,448 16.9 Total SOP HRD 4,089,358,714 1,751,797,946 42.8 Total OP ACD 244,708,967 176,239,398 72.0 Total SOP HRD & OP ACD 4,334,067,681 1,928,037,344 44.5
Source: SCF 2007; Ex-post evaluation of 2007-2013 ESF investment into human capital National template – Romania
43 Country Synthesis Report - Romania
Table 8. Data on participants of the SOP HRD and OP ACD
Enhancing access Strengthening Increasing Enhancing and sustainable Reinforcing Promoting Share of institutional Total adaptability human capital integration into social inclusion Partnerships women capacity employment Total number of participants 335,935 388,994 393,111 210,444 24,519 70,440 1,345,713 813,415 Employed 90.2% 25.9% 9.7% 41.8% 97.7% 100.0% 43.9% 61.2% of which self-employed 7.5% 1.3% 30.4% 7.4% 10.0% 0.0% 3.1% 47.4% Unemployed 1.1% 0.3% 63.7% 16.5% 0.6% 0.0% 20.4% 52.3% of which LTU 33.9% 61.4% 34.9% 42.9% 40.9% 0.0% 7.2% 52.7% Inactive 8.6% 73.8% 26.6% 41.8% 1.7% 0.0% 35.8% 57.5% of which in education/training 62.7% 64.7% 7.7% 14.5% 34.1% 0.0% 15.8% 55.8% Young people (15-24 years) 15.1% 33.4% 28.0% 22.4% 7.1% 5.0% 24.1% 56.3% Older people (55-64 years) 6.8% 2.7% 5.5% 5.6% 11.1% 10.3% 5.4% 46.7% Migrants 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 51.8% Minorities 1.8% 7.3% 7.1% 20.5% 2.5% 4.7% 7.7% 56.6% Disabled 0.1% 0.4% 0.2% 9.8% 0.2% 0.2% 1.7% 47.5% Others 24.3% 41.2% 46.2% 35.8% 30.0% 0.0% 32.2% 84.9% Primary or lower secondary 16.2% 27.7% 46.2% 40.3% 13.2% 0.0% 30.3% 50.6% education (ISCED 1 and 2) Upper secondary education 26.9% 24.3% 33.9% 23.4% 18.4% 13.2% 26.8% 57.6% (ISCED 3) Post-secondary non tertiary 9.2% 1.1% 2.9% 4.5% 8.1% 3.2% 4.2% 71.3% education (ISCED 4) Tertiary education (ISCED 5 36.3% 17.7% 15.7% 71.0% 25.4% 34.5% 32.1% 65.4% and 6)
Source: SCF 2007 and AIR 2014
44 Country Synthesis Report - Romania
Table 9. Output indicators by policy field
Final Type Indicator 2007-2014 Implementation rate Target Increasing adaptability Output 14 - Number of participants in programs such as education ‘second chance’ M 23,000 4,026 17.5% Output 14 - Number of participants in programs such as education ‘second chance’ W 12,000 1,838 15.3% Output 15 - Number of participants in CVT programmes (qualification and re -qualification ) M 200,000 29,545 14.8% Output 15 - Number of participants in CVT programmes (qualification and re -qualification ) W 160,000 25,484 15.9% Output 20 - Number of students ready to start a business M 15,000 15,406 102.7% Output 20 - Number of students ready for business start- W 15,000 30,995 206.6% Output 21 - Number of students in the management and organisation of work - M 8,500 14.893 175.2% Output 21 - Number of students in the management and organisation of work - W 6,500 19,777 304.3% Output 22 - Number of students for updating and improving skills 280,000 118,811 42.4% Output 22.1 - The number of medical doctors for updating and improving skills 10,000 32,158 321.6% Output 22.2 - The number of nurses for updating and improving skills 60,000 29,954 49.9% Enhancing Human Capital Output 01 - Number of schools supported 6,500 3,921 60.3% Output 02 - Number of universities supported 70 2,179 3,112.9% Output 03 - Staff of the educational systems trained 75,000 187,659 250.2% Output 04 - Number of PhD supported 15,000 6,808 45.4% Output 05 - Number of CVT providers supported for the introduction of quality standards 5,000 187 3.7% Output 06 - Number of professional qualifications developed in CVT 700 111 15.9% Output 12 - Number of students supported in the transition from school to work 150,000 94,218 62.8% Output 13 - Number of beneficiaries of career counselling 400,000 65,247 16.3% Enhancing access and sustainable integration into employment Output 28 - Number of supported employment agencies 200 650 325.0% Output 29 - Labour market analysis and forecasts 36 59 163.9% Output 30 - Number of staff included in training 3,000 6,017 200.6% Output 35 - Number of long-term unemployed participating in integrated programmes 65,000 63,480 97.7%
45 Country Synthesis Report - Romania
Final Type Indicator 2007-2014 Implementation rate Target Output 35.1 - Number of long-term unemployed (women) participating in integrated programmes 30,000 32,706 109.0% Output 35.2 - Number of long-term unemployed (young) participants in integrated programmes 16,000 20,251 126.6% Output 36 - Number of participants from rural areas to integrated programmes 150,000 97,062 64.7% Reinforcing social inclusion Output 40 - Number of structures of the social economy established 830 346 41.7% 41 - Number of participants in programs of qualification / requalification for vulnerable Output 150,000 51,069 34.0% groups 41.1 - Number of participants (Roma) in programs of qualification / retraining for Output 65,000 5,230 8.0% vulnerable groups 41.2 - Number of participants (people with disabilities) programs qualification / Output 20,500 3,846 18.8% requalification for vulnerable groups 41.3 - Number of participants (young people leaving the state system of protection) to Output 5,400 731 13.5% programs of qualification / requalification for vulnerable groups Output 42 - Number of dependents supported 40,000 8,814 22.0% 43 - Number of participants in training programs for professionals in the field of social Output 10,000 16,476 164.8% inclusion Promoting Partnerships Output 23 - Number of social partners and NGOs supported 500 1428 285.6% Output 44 - Number of trans-national initiatives and partnerships supported 120 9 7.5% Strengthening institutional capacity Output 01 - Guides and other methodological documents 11 151 1,372.7% Output 02 - Number of training hours per participant 130,250 60,867 46.7% Output 05 - Guides and other methodological documents 8 50 625.0% Output 06 - Developed formation module 3 33 1,100.0% Output 07 - Number of training hours per participant 45,000 30,108 66.9% Output 10 - Number of structural revisions 133 182 136.8% Output 11 - Number of training hours per participant 207,200 616,374 297.5% Output 12 - Formation module 7 426 6,085.7%
46 Country Synthesis Report - Romania
Final Type Indicator 2007-2014 Implementation rate Target Output 15 - Number of structural revisions 142 110 77.5% Output 16 - Guides and other methodological documents 25 47 188.0% Output 17 - Number of training hours per participant 31,300 28,932 92.4% Output 19 - Number of sectors that develop cost and quality standards 3 8 266.7% Output 20 - Guidelines to improve revenue collection 2 0 0.0% Output 21 - Number of training hours per participant 10,500 71,222 678.3% Output 22 - Administrative simplification analysis 250 184 73.6% Output 25 - Number of training hours per participant 700 3,031 433.0% Output 26 - Meetings of committees and relevant working groups 18 25 138.9% Output 28 - Communication and promotion events 30 14 46.7%
Source: SCF 2007
47 Country Synthesis Report - Romania
Table 10. Result indicators by policy field
Final 2007- Implementation Result Type Indicator Target 2014 rate aggregation Increasing adaptability Result 18 - Share of enterprises providing CVT to their personnel 20 24.1 120.6% Result 19 - The percentage of certified CVT programme participants 90 34 37.8% Result 24 - The percentage of training recipients initiating new business 35 12 34.3% 25 - The percentage of students certified in the management and organization of Result 80 73 91.3% work qualification Result 26 - The percentage of students certified for updating and improving skills 80 79 98.8% qualification Enhancing human capital 07 - Share of supported schools having received accreditation according to the Result 80 29 36.25% new quality assurance standards. entities 08 - Share of supported universities having received accreditation according to Result 90 100 111.1% the new quality assurance standards. Result 09 - Share of supported doctoral candidates having graduated as PhDs 90 33 36.67% qualification Result 10 - Percentage of certified staff in education and training 97 60 61.9% qualification 11 - Percentage of CVT providers authorised/certified according to quality Result 80 2.7 3.31% assurance standards 16 - The proportion of pupils / students assisted in the transition from school to Result 65 24 36.9% work who have obtained a job or actively participated in subsequent courses employment Result 17 - The percentage of graduates of education programs ‘second chance’ 75 17 22.67% qualification Enhancing access and sustainable integration into employment Result 31 - Number of agencies offering self-service 100 115 115.0% entities Result 32 - Share of employment agencies certified in quality management 95 25 26.3% entities Result 33 - Share of trained staff having obtained a certificate 80 91 113.8% qualification 34 - Share of unemployed enrolled in training programmes from the total of Result 10 7.1 70.9% unemployed people supported via at least one active employment measure 37 - Share of long-term unemployed who participated in integrated programmes Result 60 41 68.3% and were certified qualification Result 37.1 - Share of long-term unemployed (women) who participated in integrated no final 44 -
48 Country Synthesis Report - Romania
Final 2007- Implementation Result Type Indicator Target 2014 rate aggregation programmes and were target set no final 37.2 - Share of long-term unemployed (youth) who participated in integrated Result target 42 - programmes and were certified set no final 38 - Share of participants from the rural area who participated in the integrated Result target 53 programmes and were certified set - Share of participants from the rural area (women) who participated in the Result 50 57 114% integrated programmes and were certified employment 39 - Share of certified participants from the rural areas who obtained Result 15 9 60.0% employment entities Reinforcing social inclusion Result 45 - Number of jobs created by the entities of the social economy 5,000 12,377 247.5% employment 46 - Share of participants to the training/retraining programmes for vulnerable Result 55 58 105.5% groups who obtain certification qualification 46.1 - Share of (Roma) participants to the training/retraining programmes for Result 60 86 143.3% vulnerable groups who obtain certification 46.2 - Share of (disabled) participants to the training/retraining programmes for Result 60 63 105.0% vulnerable groups who obtain certification 46.3 - Share of participants to the training/retraining programmes for vulnerable Result 50 62 124.0% groups who obtain certification - children who leave the State childcare system 47 - Share of participants in training programmes for professionals in the field of Result 85 51 60.0% social inclusion having obtained certification qualification Promoting Partnerships 27 - Share of social partners and NGOs support that provides community entities Result 50 86 172.0% services Strengthening institutional capacity 03 - Share of public policies and legislative acts rejected as unsatisfactory by the Result 20 0 0.0% Secretariat General of the Government (SGG) Result 04 - Share of administrative costs reduced 20 15.3 76.7%
49 Country Synthesis Report - Romania
Final 2007- Implementation Result Type Indicator Target 2014 rate aggregation 08 - Share of public institutions producing an annual report of activity according Result 95 88.6 93.3% to Law 544/200 on transparency 09 - Number of monitoring and evaluation reports prepared/required by central Result 150 95 63.3% public institutions product Result 13 - Number of operational (reorganised and new set-up) structures 287 3,246 1,131.0% Result 14 - Number of participants to training who got a certification 75,000 52,228 69.6% qualification 18 - Number of decentralised operational (reorganised and new set-up) Result 142 79 55.6% structures product Result 23 - Number of sectors using cost and quality standards 3 8 266.7% product Result 24 - Share of tax collecting by local public administrations 95 91.38 96.2%
Source: SCF 2007
50 Country Synthesis Report - Romania
Table 11. Inventory of evaluations60
Title of evaluation Author Type of evaluation Date Scope
Ad-hoc evaluation of SOP HRD ARCHIDATA (ITALIA), in association Process evaluation 2015 Contribution to the improvement of SOP intervention for Roma people with NTSN CONECT (ROMÂNIA), HRD implementation and provide CROISSANCE CONSULTING recommendations for both 2007-2013 and (ROMÂNIA), IRS (ITALIA) and next programming period for the specific GRUPPO CLASS (ITALIA)TBD field under consideration - Roma population. Second intermediate evaluation ARCHIDATA (ITALIA), in assiciation Process evaluation 2015 Second interim evaluation has 2 OP HRD with NTSN CONECT (ROMÂNIA), Components: Component 1 - Interim CROISSANCE CONSULTING evaluation SOP HRD; Component 2 - (ROMÂNIA), IRS (ITALIA) and Administrative capacity development in GRUPPO CLASS (ITALIA) the field of programme evaluation
Interim evaluation of Operational Ernst&Young SRL Process evaluation 2013 Relevance, Effectiveness, Efficiency, Programme Administrative Analysis of Indicators Capacity Development for the period 2010 - 2012
Challenges in the Capacity of KPMG Romania in consortium with Impact evaluation 2011 Review of the OPs and implementation Public and Private Structural GEA Strategy&Consultimg and framework documents; defining the Instruments Beneficiaries Pluriconsult problems at the level of beneficieries'capacity; analysing and setting the weight of the problems in relation with each capacity level
60 Evaluation reports conducted by ESF managing authorities not covered by the ESF Expert Evaluation Network national inventories (last updated on 16/12/2013) available up until 30 June 2015.
51 Country Synthesis Report - Romania
Table 12. CSRs and ESF contribution
ESF ESF ESF Year Human Capital Access to Employment Social Inclusion contribution contribution contribution 2004 CSR 1 making permanent contracts Not the case more attractive for employers and for RO, not a discouraging the use of fixed-term MS, no access contracts so as to counter the to ESF segmentation of the labour market; increasing the attractiveness of temporary agency work for workers; removing obstacles to part-time work CSR 3 raising incentives for women to participate in the labour market by bringing down the cost of care facilities for children and other dependants CSR 4 improving active labour market measures for disadvantaged people, in particular young people, people with disabilities, immigrants and the long- term unemployed; completing the modernisation of the public employment services; strengthening the coordination between regional employment services; and removing remaining obstacles to geographical mobility 2005
2006 CSR 2 modernises employment protection, building on recent measures, including legislation to foster flexibility and security in the labour market; to counter segmentation; and to increase the attractiveness of part- time work 2007 CSR 3 continues its efforts to implement the new training model, which integrates occupational and vocational training, to provide a better response to labour market needs; and ensure the effective implementation of education reforms, also at regional level, to reduce early school leaving 2008 CSR 2 ensure the effective implement an integrated approach to implement an integrated approach to implementation of education reforms, increasing employment, activity rates increasing employment, activity rates also at regional level, to reduce early and and productivity levels, especially by
52 Country Synthesis Report - Romania
ESF ESF ESF Year Human Capital Access to Employment Social Inclusion contribution contribution contribution school leaving productivity levels, especially by accelerating reforms of the education accelerating reforms of the education system to respond better to labour system to respond better to labour market needs, by reducing early school market needs, by reducing early school leaving, by significantly increasing adult leaving, by significantly increasing adult participation in education and training; participation in education and training; and by transforming subsistence/semi- and by transforming subsistence/semi- subsistence farming into sustainable subsistence farming into sustainable employment. employment. intensifying efforts to tackle undeclared work; improving the effectiveness and geographical scope of public employment services, particularly to assist vulnerable groups. 2009 CSR 1 promote a swift transition into improves the quality and labour market improve the quality and labour market employment relevance of the education and training relevance of the education and training CSR 1 further encouraging mobility, systems, including lifelong learning, systems, including lifelong learning, upgrading skills and countering reduces early school leaving, and reduce early school leaving, and segmentation in the labour market facilitates the transition of young people facilitate the transition of young people CSR 2 ensure the effective into employment, including through into employment, including through implementation of education reforms, work-based training. work-based training. including at regional level, with the main objectives of reducing early school leaving and increasing the graduation rate in upper secondary education, and ensure that universities adapt to the Bologna process swiftly 2010
2011 No real recommendation; text about √ unemployment 2012
2013 CSR no. 4 and 5 √ Improve labour market participation etc √ 2. […] Continue the pension reform […] by promoting the employability of older workers. 4. Improve labour market participation, as well as employability and productivity of the labour force, by reviewing and strengthening active labour market policies, to provide training and individualised services and promoting lifelong learning. Enhance the capacity of the National
53 Country Synthesis Report - Romania
ESF ESF ESF Year Human Capital Access to Employment Social Inclusion contribution contribution contribution Employment Agency to increase the quality and coverage of its services. To fight youth unemployment, implement rapidly the National Plan for Youth Employment, including for example through a Youth Guarantee. To alleviate poverty, improve the effectiveness and efficiency of social transfers with a particular focus on children. Complete the social assistance reform by adopting the relevant legislation and strengthening its link with activation measures. Ensure concrete delivery of the National Roma integration strategy.
54 Country Synthesis Report - Romania
Table 13. Mapping of SIC
Main dimension(s) of the Type of support intervention (see explanatory appendix) Beneficiaries (see explanatory appendix) Indicate the applicable List the beneficiaries Level of intervention Indicate the applicable category(ies) of support and distinguishing between the Indicate the level(s) of dimension(s) indicate their relative following three categories: intervention and their importance: OP/PA/Action - Structures and processes - public entities relative importance: - Human resources - judicial - social partners - local - Service delivery -e-governance - NGOs - regional - Judiciary - territorial planning Please also indicate their - national - Social partners - PES relative importance. - NGOs - education - Other, please specify - other, please specify Operational Programme Administrative Capacity Development Priority Axis 1 Improving structure and processes of public policies cycle management - Structures and processes General administration public entities - national (1st) 1.1 - Improving decision making - Human resources support processes at administrative and - local political level local and municipal governance - Structures and processes General administration public entities - national (1st) 1.2 – Increasing public - Human resources support - local administration responsibility - Service delivery local and municipal governance - Structures and processes General administration public entities - national (1st) 1.3 - Improving organizational - Human resources support - local effectiveness local and municipal governance Priority Axis 2 Improving quality and efficiency of public services, with a focus on decentralization
55 Country Synthesis Report - Romania
Main dimension(s) of the Type of support intervention (see explanatory appendix) Beneficiaries (see explanatory appendix) Indicate the applicable List the beneficiaries Level of intervention Indicate the applicable category(ies) of support and distinguishing between the Indicate the level(s) of dimension(s) indicate their relative following three categories: intervention and their importance: OP/PA/Action - Structures and processes - public entities relative importance: - Human resources - judicial - social partners - local - Service delivery -e-governance - NGOs - regional - Judiciary - territorial planning Please also indicate their - national - Social partners - PES relative importance. - NGOs - education - Other, please specify - other, please specify - Structures and processes General administration public entities - national (1st) 2.1 – Support for sectoral - Human resources support - local decentralization of services - Service delivery local and municipal governance - Structures and processes General administration public entities - national (1st) 2.2 - Improving quality and - Human resources support - local efficiency of public services - Service delivery local and municipal governance
56
HOW TO OBTAIN EU PUBLICATIONS
Free publications:
• one copy: via EU Bookshop (http://bookshop.europa.eu);
• more than one copy or posters/maps: from the European Union’s representations (http://ec.europa.eu/represent_en.htm); from the delegations in non-EU countries (http://eeas.europa.eu/delegations/index_en.htm); by contacting the Europe Direct service (http://europa.eu/europedirect/index_en.htm) or calling 00 800 6 7 8 9 10 11 (freephone number from anywhere in the EU) (*).
(*) The information given is free, as are most calls (though some operators, phone boxes or hotels may charge you).
Priced publications:
• via EU Bookshop (http://bookshop.europa.eu).
Priced subscriptions:
• via one of the sales agents of the Publications Office of the European Union (http://publications.europa.eu/others/agents/index_en.htm).
KE
-
04
[Catalogue number
-
16
-
757
-
EN
-
N
]
doi: 10.2767/14263