ADE – ENTERPRISE PLC - ZENIT 2. EVALUATION METHODOLOGY 2.1 OBJECTIVES AND SCOPE OF THE EVALUATION Objective of the evaluation: the purpose of this study is to provide a thorough and systematic analysis of the contribution and impact of the Structural Funds RTD and innovation actions in the Member States and regions from an economic development perspective. The study should also contribute to establishing guidelines for future investments by Structural Funds in this field in the assisted regions, based on the experience of the past and current intervention. In defining the objectives of the evaluation, the Commission services noted that in developing RTDI policies for Objective areas, “it is imperative to have a detailed understanding of the current regional situation in order to determine what policies might be most appropriate. A region- specific analysis, according to the characteristics of the Objectives, enables current innovation and technology trends to be assessed and gaps to be identified in the policy framework”. This observation was taken as the starting point by the evaluation team in developing the methodology since it both underlines that there may be need to consider more fully the specific economic and innovation profiles of the Objective 2 zones when devising policie s; and the need for a thematic evaluation of interventions in favour of RTDI to take due account of the wider system of support agencies and the culture of innovation in the regions concerned. As such, the basic building blocks for the evaluation are the 15 regional case studies through which the evaluation attempted to provide a richer assessment of the different factors which influence the means of delivery and the effectiveness of Objective 2 RTDI measures. The Terms of Reference for the evaluation set out seven key issues and five key aspects for the comparative analysis which are resumed in the following tables. TABLE 2.1: The Seven key issues identified by the TOR Core issues identified by the Commission Ancillary Questions 1. How effective were the Structural Funds in promoting RTDI Pertinence of actions funded; capabilities Coherence with regional strategy 2. To what extent and through which mechanisms have the Private sector participation in RTDI Structural Fund programmes contributed to reinforcing links to local measure design; business and industry? Good practice examples and transferability 3. To what extent have the Structural Funds programmes ERDF/ESF synergies; contributed to overcome skill shortages and improve qualifications Graduate retention / integration; in the field of RTDI? Equal opportunities 4. To what extent has the new RTDI capacity provided with the Impact on business performance and assistance of Structural Funds, enhanced competitiveness and employment economic performance in the areas concerned? Sustainable development Diversification of economy 5. What has been the impact of CSFs and SPDs on national and Change in regional innovation regional RTDI policies and funding, especially on regional systems distribution of RTDI capacity? Network and linkage effects 6. Has there been synergy and co-operation or rather duplication Exchange of good practice; with other relevant Community policies, in particular the RTD Complementarity of actions. framework programmes? 7. What are the main policy issues for the future in regard to the Programme design development needs of the Objective 2 regions Strategy & implementation mechanisms Thematic Evaluation RTDI in Objective 2 – Synthesis Report 10 ADE – ENTERPRISE PLC - ZENIT Type of measures Thematic Evaluation RTDI in Objective 2 – Synthesis Report 11 ADE – ENTERPRISE PLC - ZENIT TABLE 2.2: Five Key Aspects of the Comparative Analysis Theme Focus of analysis RTDI funding Amount and proportion of funds allocated to RTDI The importance assigned to different Breakdown of expenditure on RTDI related actions basis of categorisation of types of projects Quality of RTDI organisation and Evaluation and management structures structures The effectiveness of instruments and Appraisal of key types of mechanisms instruments through case studies The impact on human resources and Analysis of ESF measures in training certain case study regions In order to make these issues more explicit, the evaluation team developed a full set of research questions which were presented in the inception report. Both the original issues and the research questions are recalled at appropriate occasions during the remainder of this report. 2.2 METHODOLOGICAL APPROACH In order to carry out the evaluation, the evaluation team proceeded in three main stages in line with the terms of reference of the study: · definition and submission of a detailed methodological framework and work-programme to the Commission in the form of an Inception Report; · a comparative analysis of trends in funding, objectives and programme design (e.g. single priority against horizontal distribution of RTDI measures across programmes) in all 86 Objective 2 zones carried out on the basis of desk-research; · an in-depth review of RTDI measures, projects, partnership and programme management procedures in 15 case study regions. These case studies were used to both confirm the trends identified in the initial desk research and to analyse key thematic issues based on an extensive series of interviews with the main programme participants and questionnaire surveys. An impact survey on companies as the final beneficiaries of RTDI measures was implemented in certain Member States. A series of methods and tools were developed in order to gather and analyse the data and qualitative information at each stage of the evaluation: · a categorisation of RTDI measures with a view to facilitating the analysis of funding trends; · a logical framework type table which allowed the structuring of key information on RTDI measures in each of the SPDs; · a template for the country reports with synthetic financial and analytical tables including regional innovation profiles; · an SME impact survey questionnaire; · good practice templates used to summarise measures and projects providing a novel approach to supporting the innovation activities of regional firms or with an identifiable impact on regional competitiveness. · an interview guide and questionnaire for the case studies. Thematic Evaluation RTDI in Objective 2 – Synthesis Report 12 ADE – ENTERPRISE PLC - ZENIT Rather than producing stand-alone regional case study reports, the information obtained from the field work in the case study regions enriched the final versions of the Country Reports. Thematic Evaluation RTDI in Objective 2 – Synthesis Report 13 ADE – ENTERPRISE PLC - ZENIT TABLE 2.3: Sources of information COUNTRY CSFs/OPs/ CSFs/OPs/RP SPDs SPDs Evaluations Evaluations Evaluations Annual Reg. Other RPDs/SPDs Ds/SPDs 94-96 97-99 89-93 94-96 97-99 Reports RTD Sources 89-91 91-93 Reports (RIS, RTP..) Austria N/E N/E 4/4 4/4 N/E 4/4 4/4 4/4 2/2 4/4 Belg./Lux. N/R N/R 5/5 5/5 4/5 5/5 5/5 N/A N/E Denmark 2/2 2/2 2/2 2/2 2/2 2/2 N/E 2/2 Finland N/E N/E 7/7 5/7 N/E N/E N/E N/E N/A N/A France 1/19 N/A 19/19 19/19 9/19 15/19 19/19 19/19 3 12/21 Germany 6/6 6/6 9/9 9/9 6/6 9/9 9/9 9/9 Italy N/R N/R 11/11 11/11 9/9 5/5 N/E 7 1 3/3 Netherlands N/R N/R 5/5 5/5 3/3 5/5 5/5 N/A 2/2 Spain 4/6 6/6 7/7 7/7 6/6 5/5 6/7 N/A N/E Sweden N/E N/E 5/5 5/5 N/E N/E N/E S S United N/A 1/9 12/12 12/12 6/12 3/12 9/12 5/12 Kingdom N/E Note: For detailed information on available data please see the respective country reports. N/E: document non existent N/R: document not requested N/A: document not available Thematic Evaluation RTDI in Objective 2 – Synthesis Report 14 ADE – ENTERPRISE PLC - ZENIT Comparative analysis As a first step, desk research was carried out in each of the eleven1 Member States covered. The documentation used at this stage consisted of the available SPDs and CSFs, the annual reports, ex- ante, interim and ex-post evaluations, etc. Table 2.3 gives a general overview of the data that has been employed for the desk research. In most of the countries, this material was accessible either through the Commission or direct contacts with regional authorities. Data availability varied across the Member States, e.g. for Germany and Spain, previous programming documents and annual reports were fully available. However, some national evaluation teams suffered from a lack of source material, mainly due to difficulties in obtaining requested information through official channels. The documentation available was used to analyse each Objective 2 region in detail concerning baseline data, the RTDI strategy and measures in the SPDs and the actual and planned expenditures on RTDI measures from 1989 (where applicable) to 1999. A key emphasis was placed on completing the logical-framework and financial tables for the 1994-96 period on the basis of a categorisation of RTDI measures. Due to the increasingly all encompassing definition of the term innovation, such a categorisation is complex since there is inevitably overlap with traditional policy domains, notably business support. Moreover, each Member State tends, for historical or institutional reasons, to define the scope and remit of RTDI measures/actions in a different manner (e.g. in Germany, investment grants are considered to promote "innovation" in the recipient companies through the acquisition of new equipment, etc.). In the Terms of Reference for the evaluation, the Commission suggested three broad categories of measures: scientific potential; technological potential and support to innovation.