STUDY Requestedtabf by the REGI committee The Role of Evaluation in Cohesion Policy Policy Department for Structural and Cohesion Policies Directorate-General for Internal Policies PE 629.219 - April 2020 RESEARCH FOR REGI COMMITTEE The Role of Evaluation in Cohesion Policy Abstract This study analyses the role of evaluation in Cohesion Policy, with a focus on the 2014–2020 programming period. It presents and assesses those EU rules which shape evaluations and their implementation at both the EU and Member State levels. Based on this evidence, it discusses possible options for the post–2020 period. This document was requested by the European Parliament's Committee on Regional Development. AUTHORS CSIL: Julie PELLEGRIN, Louis COLNOT, with support from Matteo PEDRALLI Country experts: University of Warsaw Diana IONESCU (RO), Tomasz KUPIEC (PL) Agnieszka OLECHNICKA (PL) CSIL: Matteo PEDRALLI (IT) ESTEP: Neringa VIRŠILIENĖ (LT) Scientific Advisers: FREE UNIVERSITY OF BRUSSELS-VUB: Nicola FRANCESCO DOTTI and THE UNIVERSITY OF MILAN: Massimo FLORIO Research Manager: Stephan Dietzen Project and publication assistance: Jeanette Bell Policy Department for Structural and Cohesion Policies, European Parliament LINGUISTIC VERSIONS Original: EN ABOUT THE PUBLISHER To contact the Policy Department or to subscribe to updates on our work for the REGI Committee please write to:[email protected] Manuscript completed in April 2020 © European Union, 2020 This document is available on the internet in summary with option to download the full text at: https://bit.ly/2xBMj5o This document is available on the internet at: http://www.europarl.europa.eu/thinktank/en/document.html?reference=629219 Further information on research for REGI by the Policy Department is available at: https://research4committees.blog/regi/ Follow us on Twitter: @PolicyREGI Please use the following reference to cite this study: Pellegrin, J & Colnot, L 2020, Research for REGI Committee – The Role of Evaluation In Cohesion Policy, European Parliament, Policy Department for Structural and Cohesion Policies, Brussels Please use the following reference for in-text citations: Pellegrin and Colnot (2020) DISCLAIMER The opinions expressed in this document are the sole responsibility of the author and do not necessarily represent the official position of the European Parliament. Reproduction and translation for non-commercial purposes are authorised, provided the source is acknowledged and the publisher is given prior notice and sent a copy. © Cover image used under licence from Shutterstock.com. The Role of Evaluation in Cohesion Policy CONTENTS LIST OF ABBREVIATIONS 5 LIST OF BOXES 7 LIST OF FIGURES 7 LIST OF TABLES 8 LIST OF MAPS 8 EXECUTIVE SUMMARY 9 Objectives and background 9 Key findings 9 Recommendations 10 INTRODUCTION 1 Study background 1 Objectives of the study 1 Methodological approach and outline of the report 2 1. DEFINITIONS AND CONCEPTUAL FRAMEWORK OF THE REPORT 3 1.1. Defining evaluation: key theoretical and practical aspects 3 1.2. Analytical framework adapted to Cohesion Policy specificities 6 2. EU REGULATORY FRAMEWORK GOVERNING COHESION POLICY EVALUATION 9 2.1. Historical evolution of the EU rules for evaluation of Cohesion Policy 9 2.2. Analysis of the regulatory framework governing Cohesion Policy evaluation in 2014– 2020 13 3. EVALUATION OF COHESION POLICY AT THE EU LEVEL 23 3.1. Involvement of EU stakeholders in Cohesion Policy evaluation at the EU level 23 3.2. Overview of evaluation of Cohesion Policy by the European Commission 26 3.3. Complementary activities of the European Commission for evaluation 30 4. EVALUATION OF COHESION POLICY AT THE MEMBER STATE AND REGIONAL LEVELS 37 4.1. Importance of national and regional evaluation of Cohesion Policy 37 4.2. Organisation of Cohesion Policy evaluation in the Member States 40 4.3. Overview of evaluations carried out in the Member States 46 4.4. Evidence on the use of evaluations by the Member States and regions 50 5. ANALYTICAL ASSESSMENT OF THE CURRENT FRAMEWORK AND CHALLENGES FOR THE POST-2020 PERIOD 57 5.1. Effectiveness and efficiency of the 2014–2020 framework 57 3 IPOL | Policy Department for Structural and Cohesion Policies 5.2. Analysis of the proposed framework for 2021–2027 62 5.3. Alternative scenarios for the post-2020 period 71 6. CONCLUSIONS AND RECOMMENDATIONS 75 6.1. Main findings 75 6.2. Recommendations 78 REFERENCES 81 ANNEXES 89 A.1 METHODOLOGICAL ANNEX 89 A.2 REGULATORY PROVISIONS GOVERNING COHESION POLICY EVALUATION 92 A.3 EXAMPLES OF FINDINGS FROM COHESION POLICY EVALUATIONS 101 A.3.1 Examples of findings from evaluations at EU level 101 A.3.2 Examples of findings from evaluations at the Member State level 102 A.4 EXAMPLES OF EVALUATION ARRANGEMENTS IN OTHER EU POLICIES 105 A.4.1 Horizon 2020 and Horizon Europe 105 A.4.2 The European Investment Bank and the EFSI 106 A.5 CASE STUDIES 109 A.5.1 France 109 A.5.2 Ireland 130 A.5.3 Italy 145 A.5.4 Lithuania 164 A.5.5 Poland 180 A.5.6 Romania 201 4 The Role of Evaluation in Cohesion Policy LIST OF ABBREVIATIONS AIR Annual Implementation Reports AMIF Asylum Migration and Integration Fund BMVI Border Management and VIsa CBA Cost-Benefit Analysis CF Cohesion Fund CP Cohesion Policy CPR Common Provisions Regulation (2014-2020 regulation) CRIE Centre for Research on Impact Evaluation DG EMPL Directorate-General for Employment, Social Affairs and Inclusion DG REGIO Directorate-General for Regional and Urban Policy DG RTD Directorate-General for Research and Innovation EC European Commission ECA European Court of Auditors EFSI European Fund for Strategic Investments EIB European Investment Bank EP European Parliament ERDF European Regional Development Fund ESF European Social Fund ESI(F) European Structural and Investment Funds EU European Union EUR Euro (currency) EV EValution of Operations (EIB) GDPR General Data Protection Regulation 5 IPOL | Policy Department for Structural and Cohesion Policies GPR General Provisions Regulation (2007-2013 regulation) IB Intermediate Bodies H2020 Horizon 2020 ICT Information and Communication Technologies ISF Internal Security Fund JRC Joint Research Centre MA Managing Authority MC Monitoring Committee MS Member State OECD Organisation for Economic Co-operation and Development OP Operational Programme R&D Research and Development ROP Regional OP SME Small and Medium Enterprises TB(I)E Theory-Based (Impact) Evaluation TEN Trans-European Network YEI Youth Employment Initiative 6 The Role of Evaluation in Cohesion Policy LIST OF BOXES Box 1 Open Data Platform of ESI Fund 24 Box 2 Work packages of the 2007–2013 ex-post evaluations (ERDF and CF) 28 Box 3 Examples of guidance documents provided by the European Commission for Cohesion Policy evaluation 32 Box 4 Evaluation Network 33 Box 5 Examples of ESF-related guidance 33 Box 6 CRIE 34 Box 7 Synthesis of evaluations at the EU level 35 Box 8 The OpenCoesione initiative (Italy) 45 Box 9 Tracking recommendations in Hauts-de-France (France) 46 Box 10 Evaluation capacity-building measures in Poland, Lithuania and Italy 49 Box 11 European Parliament’s main amendments on monitoring and evaluation in first reading position 71 LIST OF FIGURES Figure 1 Overview of different paradigms of evaluation 5 Figure 2 Distinction between evaluation, monitoring and audit 5 Figure 3 Conceptual framework of evaluation in the context of Cohesion Policy 7 Figure 4 Summary of the historical evolution of EU rules framing Cohesion Policy evaluation 11 Figure 5 Long-term trends related to EU rules governing the evaluation of Cohesion Policy 12 Figure 6 Main arrangements for the evaluation of Cohesion Policy in 2014–2020 in the EU legal rules 15 Figure 7 Presentation of monitoring indicators in 2014–2020 17 Figure 8 Main changes in EU rules governing Cohesion Policy evaluation between the periods 2007–2013 and 2014–2020 19 Figure 9 Number of main ex-post evaluation reports of Cohesion Policy by the European Commission 27 Figure 10 Comparison between the number of evaluations at the Member State level for 2007–2013 and 2014–2020 47 Figure 11 Themes of the evaluations conducted by Member States (2014–2020) by December 2019 48 Figure 12 Methods used by the Member States in their evaluations (2014–2020) by December 2019 49 Figure 13 Factors affecting the use of Cohesion Policy evaluation at the Member State and regional levels 52 Figure 14 Main legal arrangements proposed by the European Commission for Cohesion Policy evaluation for the 2021-2027 period 64 Figure 15 Main changes in EU rules governing Cohesion Policy evaluation between the 2014-2020 period and the proposal for 2021-2027 65 Figure 16 Alternative scenarios for Cohesion Policy evaluation post-2020 72 7 IPOL | Policy Department for Structural and Cohesion Policies LIST OF TABLES Table A.1. List of interviewed stakeholders (anonymised) 90 Table A.2. Overview of respondents to the online survey 91 Table A.3. Main EU rules governing Cohesion Policy evaluation during the 2014-2020 programming period 92 Table A.4. Comparison of the main evolutions of EU rules governing the evaluation of Cohesion Policy between 2007-13 and 2014-20 95 Table A.5. Analysis of the EU rules proposed by the European Commission for evaluation of Cohesion Policy post-2020 98 Table A.6. Governance of Cohesion Policy evaluation in France for 2014-2020 112 Table A.7. Checklist summarising the French case study 127 Table A.8. Governance of Cohesion Policy evaluation in Ireland for 2014-2020 134 Table A.9. Checklist summarising the Irish case study 143 Table A.10. Governance of Cohesion Policy evaluation in Italy for 2014-2020 149 Table A.11. Checklist summarising the Italian case study 162 Table A.12. Governance of Cohesion Policy evaluation in Lithuania for 2014-2020 166 Table A.13. Checklist summarising the Lithuanian case study 177 Table A.14. Governance of Cohesion Policy evaluation in Poland for 2014-2020 182 Table A.15.