The Added Value of the European Map of Excellence and Specialization (EMES) for R&I Policy Making

Home , Art Academy of Latvia, Blaise Pascal University, Education in Warsaw, Jaume I University, Jean Monnet University, Marconi University

Cinzia Daraio July – 2015 EUR 27389 EN

EUROPEAN COMMISSION Directorate-General for Research and Innovation Directorate A – Policy Development and coordination Unit A6 – Science Policy, foresight and data Contact: Emanuele Barbarossa, Katarzyna Bitka E-mail: [email protected] [email protected] [email protected] [email protected] European Commission B-1049 Brussels

EUROPEAN COMMISSION

The Added Value of the European Map of Excellence and Specialization (EMES) for R&I Policy Making

Cinzia Daraio

The document is based on projects carried out by the ONTORES research group at Sapienza University of Rome and on the Smart.CI.EU (Sapienza microdata architecture for education, research and technology studies. A Competence-based data Infrastructure on European Universities). The contributions of Marco Angelini, Alessandro Daraio, Flavia di Costa, Maurizio Lenzerini, Claudio Leporelli, Henk F. Moed, Gabriele Petrotta, and Giuseppe Santucci are gratefully acknowledged.

Directorate-General for Research and Innovation 2015 Research, Innovation, and Science Policy Experts High Level Group EUR 27389 EN

EUROPE DIRECT is a service to help you find answers to your questions about the European Union Freephone number (*): 00 800 6 7 8 9 10 11

(*) The information given is free, as are most calls (though some operators, phone boxes or hotels may charge you)

LEGAL NOTICE This document has been prepared for the European Commission however it reflects the views only of the authors, and the Commission cannot be held responsible for any use which may be made of the information contained therein.

More information on the European Union is available on the internet (http://europa.eu).

Luxembourg: Publications Office of the European Union, 2015.

ISBN 978-92-79-50354-2 doi 10.2777/553985

ISSN 1831-9424

Table of contents

EXECUTIVE SUMMARY ...... 5 RÉSUMÉ ...... 9 TABLE OF CONTENTS ...... 3 1. INTRODUCTION AND CONTENT OF THE STUDY ...... 12 Introduction ...... 12 Content of the study ...... 12 2. POLICY RELEVANCE OF AN EMES FOR R&I POLICY MAKING ...... 14 3. DEFINING CRITERIA FOR EMES ...... 17 4. GEO-REFERENCING INFORMATION ON EUROPEAN UNIVERSITIES ...... 17 Drawbacks and limitations: multi-site institutions ...... 19 5. INTEGRATING BIBLIOMETRIC DATA AT THE LEVEL OF INDIVIDUAL UNIVERSITIES ...... 19 State of the art ...... 19 Scimago Institutions Rankings ...... 20 Global Research Benchmarking System ...... 20 Leiden Ranking ...... 21 Altmetrics, webometrics and other complementary information ...... 22 Coverage of the European university landscape ...... 24 6. LOCATING PUBLICATIONS OF UNIVERSITIES AND PROS ON A GEOGRAPHIC MAP ...... 25 Towards an authority file for PROs ...... 26 Breakdown by discipline...... 26 7. TOWARDS A EUROPEAN MAP OF EXCELLENCE AND SPECIALIZATION...... 28 Geo-referencing data on publications...... 28 Integrating information from other projects: the case of U-Multirank ...... 31 Integrating other socio-economic indicators ...... 32 Feasibility of selected indicators ...... 34 8. CONCORDANCE TABLES OF DIFFERENT SUBJECT CLASSIFICATION SYSTEMS ...... 40 Introduction ...... 40 Results from a survey ...... 41 Approaches and systems developed in the past. Correspondence tables between Intellectual Patent Classification (IPC) and Fields of Science (FoS); and between IPC and industrial classification...... 43 Correspondence tables between Fields of Education (FoE) and Fields of Science (FoS) .. 43 Correspondence tables from the Eumida project ...... 43 Correspondence tables from the ETER Project ...... 45 Conclusions and recommendations ...... 47 9. VISUAL ANALYTICS FOR A PILOT EMES ...... 47 General Design ...... 48 Proof-of-concept prototypal application ...... 51 10. ASSESSMENT ...... 55 11. EXPLORATION OF POSSIBLE BUSINESS MODELS AND BUDGET ...... 56 1. Model Supported by the European Commission ...... 56 2. Public-private sponsorship Model...... 57 3. Science 2.0 Model ...... 57 Linking data in an open platform ...... 57 Automation and maintenance of the infrastructural data system ...... 58 A real options approach to estimate the investment in an OBDM approach ...... 60 An estimate of the needed budget...... 61

12. RECOMMENDATIONS ...... 62 REFERENCES ...... 65 APPENDICES ...... 68 Appendix 1: Authority file of European universities ...... 68 Appendix 2: Concordance tables ...... 101 Appendix 3: Possible User Groups ...... 107

EXECUTIVE SUMMARY

This study examines the feasibility of constructing a European Map of Excellence and Specialization (EMES) by offering a proof of the concept and illustrating the potential for policy making.

The term 'Map of Excellence and Specialization' refers to a geographical information system (GIS) that combines and georeferences information from various sources at different geographic scales (Nomenclature of Units for Territorial Statistics (NUTS) levels 2 and possibly 3) and provides indicators intended for policy use.

Drawing a Map of Excellence and Specialization entails many challenges: Actors in the European Science and Technology (S&T) field are heterogeneous (for example, universities and Public Research Organisations (PROs)), their output is composite (for example, education, publications, and patents), the location of their activities is not fully disclosed, etc.

A number of S&T policy decisions depend on assumptions about the impact of public expenditure on national, regional or local variables such as employment, productivity, and growth. These assumptions are rarely based on sound empirical evidence, however. Furthermore, they tend to ignore the magnitude of knowledge spillovers and tend to assume a simplistic view of agglomeration.

What is needed is a robust empirical base at geographic level in which data on knowledge production are firmly established. Such a base would then offer the opportunity to integrate other data using the same unit of reference at geographic level.

The current study explores also the policy implications of using such a Map of Excellence and Specialization.

The main criteria to assess the successful implementation of the EMES have been identified in:

 Availability of data on publications (adequate economic and legal framework for the use of commercial data on publications including sources, commercial conditions and update for a medium long period of time);  Standardization (consideration of existing standards in the science and technology higher education funding fields and adequate solutions to import relevant standards, e.g. ORCID, CERIF, EUROCRIS,CASRAI etc.) openness and interoperability with other sources of data available;  Compliance with state of the art data quality techniques;  Continuity (adequate organizational solutions for the continuous update, maintenance and improvement of the map);  Extensions and scalability (explicit solutions to make the map suitable for future integration of new actors, new/updated sources of data, indicators);  Expertise in the access and analysis of publicly available data; interactivity (ability of the map to allow the automatic generation of new indicators, explicit solutions for controlling the statistical properties of new indicators and potential misuse) and usability (the reader has to play with it without entering in the technical details);  Existence of concordance tables among different subject classifications. On the base of the study carried out, and taking into account the established criteria for the assessment of the successful feasibility of the EMES, we suggest to the Commission to further proceed with a full scale study for the realization of the European Map of Excellence and Specialization.

The EMES should be designed by following an Ontology-Based Data Management (OBDM) approach to ensure a sustainable and up-to-date map, interoperable and extendable over time. The Map should integrate in a GIS-format at least the following groups of indicators:  structural indicators for higher education institutions (HEIs)

 structural indicators for Public Research Organisations (PROs)  publications of HEIs and PROs  patents assigned to HEIs and PROs  academic staff at HEIs  undergraduate students  PhD students  undergraduate degrees  PhD degrees. All these information should be geo-referentiated and supported by extensive metadata.

The data should have a breakdown by discipline (Field of Science, or Subject categories) and by Field of Education.

In addition, data should be integrated with relevant indicators at regional (NUTS2 and, where possible, NUTS3) level. These should include industrial, employment, GDP, social and demographic data.

 The proposed Map should specify the procedures for the updating of data, offering solutions for the automatic update as frequently as possible.  The proposed Map should also demonstrate the sustainability of the organization or business model in the future, by addressing issues such as provision of commercially available data, cost of update, IPRs, standardization and robustness issues.  With respect to Higher Education institutions, the census established by the ETER project, funded by DG Education and Culture in collaboration with DG Research and EUROSTAT, should be assumed as the official source. Consequently, the ID system proposed by the ETER project should be used as a reference in all documents. The project should provide the list of all affiliation names and possible variations found in publications affiliations.  With respect to Public Research Organisations, the project should issue a similar ID system, organized in a hierarchical way as suggested by the ontology model. For instance:  organization name at the country level (e.g. Max Planck Society, CNRS, or CSIC)  first-tier sub-organization name (e.g. institute, or department)  second-tier organization name (e.g. institute within a department, laboratory within an institute)  bottom level organization (e.g. research group, research team, laboratory). The system should aim at maintaining stability at organization name, establishing a permanent list as a standard reference list. This would be an Authority File to be maintained at official level.

With respect to first-tier and second-tier organizations, the system should provide a reliable mapping structure, which can be managed automatically. This means that the system should provide a full list of all possible names and abbreviations, in all possible combinations, so that they can be matched to publication data in an automatic way. Each occurrence should be unambiguously related to the Authority File.

The system must specify the procedures by which the lists of first-tier and second-tier names are updated, corrected, and cleaned over time.1 The system must specify under which

1 One should distinguish two approaches in which the Authority File can be used in the affiliation-de-duplication process. In the first the Authority file is used to assign author affiliation strings in scientific publications to organization names. For instance, if one finds articles with the affiliation ‘Dept Astronomy’ in a paper from Leiden while the name “University of Leiden” is missing in those strings, a rich Authority File containing information on first and second tier sub- organizations indicates that University of Leiden actually contains a Department of Astronomy, so that these affilation strings should be assigned to the University of Leiden. But this does not necessarily mean that one can obtain a reliable estimate of the publication output of Dept Astronomy at Univ Leiden by counting only articles containing ‘Dept Astronomy’ in their affiliation data, since there may be many papers from this department that do not contain Dept Astronomy in their affiliation data. The reason is that the affiliation information in scientific publications is often too

conditions the list of first-tier names could eventually become an Authority File in its own, becoming an official and stable source. The same should be examined for the second-tier list of names.

The Steering Group might meet regularly in order to review the development of the project and to define and refine requirements for the construction of indicators, on the basis of their own needs.

The Commission might also consider whether to invite separately ERC and EIIT representatives, as well as members of national governments.

In parallel, a Committee representing the PROs should be created. It should include at least one representative for the largest PROs in Europe (i.e. Max Planck, Leibniz, Helmholtz in Germany, CNRS, INSERM, INRA, INRIA, CEA in France, CSIC in Spain, CNR, INFN in Italy) and a number of representatives from other PROs.

This Committee should regularly meet to supervise the development of the analysis aimed at the geo-referentiation of scientific publications of PROs. The Committee should validate the allocation of specific research outputs to teams, laboratories or institutes that could be located geographically. Provisions for fractional allocation should be examined and validated.

Meanwhile, other issues should be discussed (e.g. collection of data on patents) for future activities of indicator construction.

 In all future calls for research projects of the Commission there should be a mandatory provision for submission to provide full coverage of ORCID numbers for all researchers involved.  In all future documentation the standards established in the project and/or available at international level should be adopted:  ID numbers of HEIs  ID numbers of PROs  CERIF ID numbers of funding agencies  ORCID ID numbers for researchers

 In future studies commissioned by the Commission there should be a provision for establishing linkages with the platform produced under the project. In particular, data should be delivered to the Commission in such a way to be integrated in a seamless way into the platform. This requires that the ontology model developed as the base of the data integration process become a standard reference point.  In addition the ontology suggested by the project should be published. An interactive consultation with producers and users of indicators should be opened. After a given period, the ontology should be published in an official version and should become a standard reference point. Future releases should be published in due time.

incomplete or inaccurate to achieve de-duplication at lower levels. The recall of such a process tends to be low, even though the precision may be high. This problem is for PROs even worse than for academic institutions.

 The feasibility study has shown the enormous potential for reliable, effective and efficient construction of indicators provided by the creation of Authority Files. This is an authoritative source established in the form of a list, associated to complete definitions and rules for inclusion and exclusion, and to explicit rules for updating over time. These processes could be, in principle, executed automatically.

Once an Authority File is established, the same information is propagated in the entire information system without ambiguity.

This means that the same piece of information gets more value, since it is appropriately used in many contexts.

In the context of this study the following Authority Files should be established:

 official list of Higher Education Institutions  official list of Public Research Organisations  author ID  publication ID  funding agency ID

The official list of HEIs is available under ETER: we recommend the adoption as a standard.

The official list of PROs is to be constructed under a dedicated project.

With respect to the IDs, we recommend the Commission to support actively all international efforts to establish and maintain standards, such as ORCID and CERIF. The Commission might issue a document asking Member States to adopt these standards in their administrative activities.

RÉSUMÉ

Cette étude examine la faisabilité de la construction d'une carte européenne d'excellence (EMES) en offrant une preuve du concept et illustrant le potentiel pour l'élaboration des politiques.

Le terme «carte de l'excellence» se réfère à un système d'information géographique (SIG) qui combine et géoréférences informations provenant de diverses sources, à différentes échelles géographiques (nomenclature des unités territoriales statistiques (NUTS) niveaux 2 et 3) et fournit des indicateurs destinés à l'usage de la politique.

Un certain nombre de décisions politiques dépendent d'hypothèses sur l'impact des dépenses publiques sur les variables nationales, régionales ou locales telles que l'emploi, la productivité et la croissance. Ces hypothèses sont rarement fondées sur des preuves empiriques solides.

Ce qui est nécessaire est une base de données solide au niveau géographique dans laquelle les données sur la production de connaissances sont fermement établies. Une telle base serait alors offrir la possibilité d'intégrer d'autres données à l'aide de la même unité de référence au niveau géographique.

L'étude actuelle explore également les implications politiques d'une telle carte de l'excellence et de spécialisation.

Les principaux critères pour évaluer la mise en œuvre réussie d'une carte européenne d'excellence ont été identifiées dans:

 la disponibilité de données sur les publications (cadre économique et juridique adéquat pour l'utilisation de données commerciales sur les publications y compris les sources, les conditions commerciales et mise à jour pour une longue période de temps moyenne);  Normalisation (standardisation, par exemple ORCID, CERIF, euroCRIS, CASRAI etc.) ouverture et interopérabilité avec d'autres sources de données disponibles;  la qualité des données (application des solutions méthodologiques au jour pour assurer la qualité des données selon les normes internationales);  la continuité (solutions organisationnelles adéquates pour le maintien de la mise à jour continue et l'amélioration de la carte);  Extensions et l'évolutivité (solutions explicites pour faire la carte adapté à l'intégration future de nouveaux acteurs, de nouvelles sources / mises à jour de données, indicateurs);  Expertise dans l'accès et l'analyse des données accessibles au public; interactivité (capacité de la carte pour permettre la génération automatique de nouveaux indicateurs de solutions explicites pour contrôler les propriétés statistiques de nouveaux indicateurs et abus potentiel) et la facilité d'utilisation (le lecteur a à jouer avec elle sans entrer dans les détails techniques);  L'existence de tables de concordance entre les différents systémes de classifications. Sur la base de l'étude réalisée, et en tenant compte des critères établis pour l'évaluation de la faisabilité de succès de l'EMES, nous suggérons à la Commission d'aller plus loin avec une étude à grande échelle pour la réalisation de l’EMES.

 L’ EMES devraient être conçus en suivant une approche de gestion des données Ontology- Based (OBDM) afin d'assurer la mise à jour de la carte, l’interopérabilité et l’extensibilité au fil du temps.  L’EMES devrait intégrer dans un systéme SIG au moins les groupes d'indicateurs suivants:  les indicateurs structurels pour les institutions d'enseignement supérieur (IES)  indicateurs structurels pour organismes publics de recherche (OPR)  Publications des IES et des OPR  brevets cédés aux IES et aux OPR  Le personnel académique de IES

 les étudiants  doctorants  diplômes  Doctorat. Toutes ces informations doivent être géo-referentiated et soutenu par une vaste documentation (métadonnées).

Les données devraient être detaillées par discipline et par domaine de l'éducation.

En outre, les données doivent être intégrées avec des indicateurs pertinents au niveau régional (NUTS 2 et, si possible, NUTS3). Ceux-ci devraient inclure industrielle, l'emploi, le PIB, social et données démographiques.

 Le Plan proposé devrait préciser les modalités de la mise à jour des données, proposant des solutions pour la mise à jour automatique aussi souvent que possible.  La Plan proposée devrait également démontrer la viabilité de l'organisation ou de modèle d'affaires à l'avenir, en abordant des questions telles que la fourniture de données disponibles dans le commerce, le coût des problèmes de mise à jour, de normalisation et de robustesse.  En ce qui concerne les institutions d'enseignement supérieur (IES), recensement établi par le projet ETER, financé par la DG Education et Culture, en collaboration avec la DG Recherche et EUROSTAT, doit être assumé comme une source officielle. Par conséquent, le système d'identification proposé par le projet de ETER doit être utilisé comme référence dans l'ensemble des documents. Le projet devrait fournir la liste de tous les noms d'affiliation et les variations possibles trouvés dans les affiliations des publications.  En ce qui concerne les ORP, le projet devrait émettre un système d'identification similaire, organisée de façon hiérarchique comme suggéré par le modèle de l'ontologie. Par exemple:  Nom de l'organisation au niveau des pays (par exemple, le Max Planck, CNRS, ou CSIC)  Premier niveau sous-nom de l'organisation (par exemple, un institut ou département)  De second rang nom de l'organisation (par exemple institut au sein d'un département, laboratoire dans un institut)  Organisation de niveau inférieur (par exemple un groupe de recherche, l'équipe de recherche, laboratoire). Le système devrait viser à maintenir la stabilité au nom de l'organisation, l'établissement d'une liste permanente comme une liste de référence standard. Ce serait un Authority File à être maintenu au niveau officiel.

En ce qui concerne les organisations de premier rang et de second rang, le système devrait fournir une structure de cartographie fiable, qui peut être géré automatiquement. Cela signifie que le système doit fournir une liste complète de tous les noms et les abréviations possibles, dans toutes les combinaisons possibles, de sorte qu'ils peuvent être adaptés aux données de publication d'une manière automatique. Chaque occurrence devrait être clairement lié au Authority File.

Le système doit préciser les modalités selon lesquelles les listes de noms de premier rang et de second rang sont mis à jour, corrigées et nettoyées au fil du temps.

En outre, le système doit indiquer dans quelles conditions la liste des noms de premier rang pourrait éventuellement devenir un fichier d'autorité dans son propre, devenant une source officielle et stable. La même chose devrait être examiné pour la liste de second niveau.

Afin de superviser le développement du projet, un groupe de pilotage devrait être formé, impliquant  DG Recherche  DG Education

 DG Région  Eurostat  OCDE  Parlement Européenne - STOA. Le groupe de pilotage pourrait se réunir régulièrement afin d'examiner le développement du projet et de définir et d'affiner les exigences pour la construction d'indicateurs, sur la base de leurs propres besoins. La Commission pourrait également examiner si d'inviter séparément les représentants de ERC et EIIT, ainsi que les membres des gouvernements nationaux.  En parallèle, un comité représentant les ORP devraient être créés. Il devrait inclure au moins un représentant pour les plus grandes ORP en Europe (i.e.,Max Planck, en Allemagne, CNRS, INSERM, INRA, INRIA, CEA en France, le CSIC en Espagne, le CNR, l'INFN en Italie) et un certain nombre de représentants d'autres ORP. Ce comité devrait se réunir régulièrement pour superviser le développement du projet.  Dans tous les futurs appels à projets de recherche de la Commission, il devrait y avoir une disposition pour la présentation à fournir une couverture complète des numéros de ORCID pour tous les chercheurs impliqués.  Dans toute la documentation avenir, les normes établies dans le projet et / ou disponibles au niveau international devraient être adoptées:  numéros d'identification des IES d'enseignement supérieur  Numéros d'identification des OPR  Numéros d'identification CERIF des organismes de financement  Numéros ORCID d'identification pour les chercheurs.  Dans les futures études commandées par la Commission il devrait y avoir une disposition pour établir des liens avec la plateforme produite dans le cadre du projet. En particulier, les données doivent être transmises à la Commission de manière à être intégré de manière transparente dans la plateforme crée. Cela nécessite que le modèle de l'ontologie développé comme la base du processus d'intégration de données devient un point de référence standard.  En outre l'ontologie proposée par le projet devrait être publié. Une consultation interactive avec les producteurs et utilisateurs d'indicateurs doit être ouvert. Après une période donnée, l'ontologie devrait être publié dans une version officielle et devrait devenir un point de référence standard.  L'étude de faisabilité a montré le potentiel énorme pour la construction fiable, efficace et efficiente des indicateurs fournis par la création de Authority Files. Ceci est une liste, associé à compléter les définitions et les règles d'inclusion et d'exclusion, et de règles explicites pour mettre à jour au fil du temps. Ces processus pourraient être, en principe, exécutées automatiquement. Une fois un Authority File est établie, la même information se propage dans le système d'information sans ambiguïté. Cela signifie que le même élément d'information est de plus de valeur, car il est utilisé de manière appropriée dans de nombreux contextes. Dans le cadre de cette étude, les Authority File suivante devrait être établi:  Liste officielle des IES  Liste officielle des OPR  ID Auteur  ID Publication  ID Agence de financement La liste officielle des IES est disponible sous ETER: nous recommandons l'adoption en tant que norme.

La liste officielle des OPR doit être construite par un projet dédié. En ce qui concerne les ID, nous recommandons à la Commission de soutenir activement tous les efforts internationaux visant à établir et maintenir des standard, telles que ORCID et CERIF. La Commission pourrait émettre un document demandant aux États membres d'adopter ces standard dans leurs activités administratives.

1. INTRODUCTION AND CONTENT OF THE STUDY

Introduction

This study examines the feasibility of constructing a European Map of Excellence and Specialization (EMES) by offering a proof of the concept and illustrating the potential for policy making.

The term 'Map of Excellence and Specialization' refers to a geographical information system (GIS) that combines and georeferences information from various sources at different geographic scales (Nomenclature of Units for Territorial Statistics (NUTS) levels 2 and 31) and provides indicators intended for policy use.

Drawing a EMES entails many challenges: Actors in the European Science and Technology (S&T) field are heterogeneous (for example, universities and Public Research Organisations (PROs)), their output is composite (for example, education, publications, and patents), the location of their activities is not fully disclosed, etc.

The current study explores also the policy implications of using such a Map of Excellence and Specialization.

Content of the study

The main analyses carried out in this study are reported in the following. Firstly, the study examines the feasibility of geo-referencing information pertaining to excellence in S&T at NUTS 2 and NUTS 3 level of European universities based on the results of the ETER (European Tertiary Education Register) study. If data are missing, the last EUMIDA data should be used. The task will be based on those higher education institutions that deliver the PhD degree (i.e. universities). Secondly, the study examines the feasibility of integrating data on scientific publications at the level of individual universities with a breakdown by disciplines employing the most recent data using the following indicators as examples: number of publications; number of citations; percentage of publications in top journals; percentage of citations from top journals; percentage of publications with international collaboration. Thirdly, the study examines the feasibility of locating publications of universities on a geographic map of Europe, by integrating all data coming from various universities at NUTS 2 and NUTS 3 level. The feasibility is based on data on all European universities, as defined by the Eumida and ETER studies. With respect to Public Research Organizations (PROs) the study discusses the list extracted from affiliations of publications in commercial databases and its comparison with the list currently maintained by DG RDT in light of the experiences gathered from previous research projects carried out at Sapienza. The study also offers a proof of the concept by building up a sample of regions and/or universities and locating them on a GIS computer platform.

Fourthly, the study analyses how other socio-economic indicators could be integrated in the GIS in order to build up a Map of Excellence. It explores the feasibility of integrating official statistics, including population, economic, industrial and infrastructure statistics, in the GIS at the appropriate level of granularity. Fifthly, in order to prepare for future integration of data on the specialisation of European countries and regions, the study reports about the state of the art in the literature regarding the correspondence between different classifications in S&T and industrial statistics. In particular, it extensively examines the correspondence between Fields of Education (FoE) and Fields of Science (FoS), between the latter and the International Patent Classification (IPC), between the latter and industrial classifications. In discussing the feasibility of a European Map of Excellence and Specialization (EMES), the indicators reported in Box 1 have been taken into account.

BOX 1: Indicators considered in the feasibility study on the EMES.

Specialization indicators - Revealed Scientific Advantage of regions (NUTS 2), normalized both at EU and country level; - Position of NUTS 2 or possible NUTS 3 territory in European ranking by discipline, based on Number of publications both as an absolute and normalized against socioeconomic indicators, e.g. population; - Position of NUTS 2 or possible NUTS 3 territory in European ranking by discipline, based on Number of citations (including derived indicators, such as share of publications among the 1% most cited).

Excellence indicators - Composite indicator including Number of publications, Number of citations, and Percentage of publications and citations in top journals.

Critical mass indicators Indicators stating whether the territory (NUTS 2 or possibly NUTS 3) have or have not reached a given threshold of publications in given disciplines using the indicators developed by the EC as test cases.

Research productivity indicators - Number of publications per unit of academic staff - Number of excellent publications per unit of academic staff - Number of citations per unit of academic staff - Number of excellent citations per unit of academic staff

Research intensity indicators - Number of publications per 000 inhabitants - Number of publications per million euro GDP - Number of citations per 000 inhabitants - Number of citations per million euro GDP

Research Excellence indicators - Number of excellent publications per 000 inhabitants - Number of excellent publications per million euro GDP - Number of excellent citations per 000 inhabitants - Number of excellent citations per million euro GDP

The study unfolds as follows. Section 2 illustrates the relevance of EMES for policy making in research and innovation sectors in Europe. Section 3 introduces criteria for feasibility which are assessed in Section 10. Section 4 describes the feasibility and limitations of geo-refering universities’ activities on a European map. Section 5 deals with the localization of bibliometric data taking into account three examples of projects and discussing problems and further issues for a full scale exercise. Sections 6 and 7 further analyse the steps towards a multidimensional EMES for which a proof-of-concept prototypal application is described in Section 9. Section 8 reviews the literature on concordance tables for research and innovation analysis. Section 10 summarizes the assessment of the feasibility of an EMES, while Section 11 explores possible business models for a sustainable EMES over time and outlines a budget for a full scale project on the realization of the EMES. The study is closed by a set of recommendations for future developments reported in Section 12. Three appendices complete the study.

2. POLICY RELEVANCE OF AN EMES FOR R&I POLICY MAKING

The main goal of this feasibility study is to illustrate the potential of a platform that allows advanced levels of integration, analysis, automation, and scalability of indicators in the field of science, technology and growth at European level.

The needs underlying the concept of such a platform can be described as follows:

 integration of data from heterogeneous sources;  levels of aggregation (geographic, institutional, disciplinary);  updating and scalability. From a policy point of view there are a number of relevant issues that require a novel approach to data management. Let us consider, for example, the following ones.

Research investment, innovation and growth in laggard regions

There is a growing evidence that the investment in R&D, funded by public resources, is not a sufficient condition for the catching up effort of laggard regions in Europe, both in Southern and Eastern countries. The key concept here is one of complementarity: investment in R&D only produces effects on growth if the specialization in science matches the identification of niches of technological specialization, and if both of them are supported by complementary investment into education and human capital. While the theoretical arguments underlying this point are clear, there is a lack of data that allows the integration at regional, or even urban, scale, of indicators on research, technology, education, and growth, with a breakdown by discipline/ field of technology/ industry. Therefore an important pillar of European policies rests on weak grounds from an empirical point of view.

Excellence of European universities

There is a recurring debate in European countries, fuelled year after year by the publication of university rankings. Why the overall volume of European publications is roughly comparable, or even larger, than the US one, and at the same time there are so few European universities able

to compete at the top? And does it really matter to be in the top league, or it is only a matter of visibility and prestige? Although, thanks to recent projects on the matter, microdata at the level of individual universities can be obtained from public sources (i.e. ETER for Higher Education Institutions), they are not integrated with other data on inputs and complementary outputs (publications, patents, third mission indicators). Thus, the debate rests on aggregate notions of excellence, or lack thereof, without helping progress in policy making. We argue that the current debate is incorrectly set. First of all, the meaning of excellence should be clearly defined and formally conceptualized in order to derive a coherent set of indicators able to monitor its evolution and its dynamic changes over time. Secondly, excellence should be “contextualized”, that is, it depends on the missions of the institutions (teaching, research, third mission), often complementary and/or rivalry, on the external environmental context as well as on the comparison set, and again can change and evolve over time. In order to address the issue of the excellence of European universities on a sound empirical base there is a need for developing a long lasting data infrastructure, interoperable with existing available sources of data, able to be updated and extended over time without having to start from the scratch each time a new policy need appears.

Role of Public Research Organisations

The European landscape benefits from a large actor, which is missing in other countries or has a different mission, i.e. large PROs. Their aggregate role in the production of science and technology is well known, yet there is a lack of empirical evidence regarding the complementarity between PROs and universities, and their impact on regional and national competitiveness. In all these examples (but it might be possible to add other issues) there is a distinct need for an integration of heterogeneous data from various sources, without the need to start a new study, and a new database, each time a policy issue appears.

General relevance of data integration and platform for science of science policy Clearly, the information needs and the analytic goals of the research community interested in science of science are of historical nature. The ability to compare research institutions in their evolution over time is of paramount importance. This is best obtained, and most economically, if some institutions, or network of institutions, take charge to maintain a certified repository of relevant data. In other words, science of science should be included among the domains national and regional statistical offices include in their regular surveys and census activities, using methodologies that maximize the ability to understand, use, and compare data. Moreover, these institutions should publish not only statistical summaries, but also microdata sources. In particular, administrative databases of research institutions could become a privileged source of research oriented information. The social benefits of the building of such an infrastructure greatly exceeds their social costs. In this study we argue that in order to design an EMES as a long lasting data infrastructure an Ontology-Based Data Management (OBDM) approach (Poggi et al., 2008; Lenzerini, 2011, 2015a; Daraio, Lenzerini et al. 2015) should be followed. The main idea of the OBDM approach is the organization of a three-level architecture, constituted by:

 The ontology: is a conceptual, formal description of the domain of interest (expressed in terms of relevant concepts, attributes of concepts, relationships between concepts, and logical assertions characterizing the domain knowledge).  The sources: are the repositories accessible by the organization where data concerning the domain are stored. In the general case, such repositories are numerous, heterogeneous, each one managed and maintained independently from the others.

 The mapping: is a precise specification of the correspondence between the data contained in the data sources and the elements of the ontology. An illustration follows in the next figure.

Figure 1. An illustration of the basic idea of the OBDM approach. Source: Daraio (2015).

The main purpose of an OBDM approach is to allow information consumers to query the data using the elements in the ontology as predicates. It can be seen as a form of information integration, where the usual global schema is replaced by the conceptual model of the application domain, formulated as an ontology expressed in a logic-based language. The main advantages of an OBDM approach are:  Users can access the data by using the elements of the ontology.  By making the representation of the domain explicit, we gain re-usability of the acquired knowledge.  The mapping layer explicitly specify the relationships between the domain concepts and the data sources. It is useful also for documentation and standardization purposes.  Flexibility of the system: you do not have to merge and integrate all the data sources at once which could be extremely costly.  Extensibility of the system: you can incrementally add new data sources or new elements (ability to follow the incremental understanding of the domain) when they become available.

3. DEFINING CRITERIA FOR EMES

The study carried out and reported in this report has been based on the competences, previous experiences and current research projects carried out at the University of Rome La Sapienza, as well as on the existing literature on the related topics.

The current study can be seen as a proof-of-concept, needed in order to verify the feasibility and provide further insights for a full scale exercise.

In the following, we report the proposed criteria to assess the feasibility of the EMES.

 Availability of data on publications (adequate economic and legal framework for the use of commercial data on publications including sources, commercial conditions and update for a medium long period of time);  Standardization (consideration of existing standards in the science and technology higher education funding fields and adequate solutions to import relevant standards, e.g. ORCID, CERIF, EUROCRIS,CASRAI etc.) openness and interoperability with other sources of data available;  Compliance with state of the art data quality techniques;  Continuity (adequate organizational solutions for the continuous update maintenance and improvement of the map);  Extensions and scalability (explicit solutions to make the map suitable for future integration of new actors, new/updated sources of data, indicators);  Expertise in the access and analysis of publicly available data; interactivity (ability of the map to allow the automatic generation of new indicators explicit solutions for controlling the statistical properties of new indicators and potential misuse) and usability (the reader has to play with it without entering in the technical details);  Existence of concordance tables among different subject classifications.

4. GEO-REFERENCING INFORMATION ON EUROPEAN UNIVERSITIES

The first analytical step for the evaluation of the feasibility of geo-referencing information pertaining excellence in S&T consists in locating activities of the universities in European regions. Recent efforts toward building a European tertiary education register promoted by the European Commission have ensured big advancement towards the possibility of geo-referencing information pertaining to excellence in S&T of European universities. We specifically refer to the results of two projects promoted by the European Commission, respectively:  EUMIDA - Feasibility Study for Creating a European University Data Collection (Contract No. RTD/C/C4/2009/0233402) completed in 2010;  ETER - European Tertiary Education Register (Contract No. EAC‐2013‐038) completed in July 2015. It will be continued up to 2017 with the project “Implement and Disseminate the European Tertiary Education Register (Contract No. EAC-2015-0280). EUMIDA demonstrated the feasibility of collecting data at Higher Education Institutions (HEIs) level, covering both input (staff and finance) and output (education, research) dimensions, along with a set of descriptors allowing for a more precise profiling at European level (institution category, legal status, foundation, region of establishment, etc.). ETER further extended the data collection coverage to all ERA countries, adding at the same time more details and variables breakdowns. Focusing our attention to the information useful for a systematic process of geo-referencing, the situation is as follows:  ETER published the results of the first data collection (reference year 2011) in 2014 (eter.joanneum.at/imdas-eter). The dataset includes the perimeter (list of HEIs with ID code and name) for all 36 ERA countries and data for 29 out of 36 countries. Within the European

Union, information on Hungary, Romania and Slovenia are missing. Among other variables, ETER includes the following geographical information:  Country of establishment: the country where the institution is established (ie. where the institution develops most of its activities, for example where the largest part of the staff is located, even if this is not the legal seat of the institution). Official ISO 3166 country codes are used to identify the country;  Region of establishment (NUTS2 and NUTS3): the region where the institution’s main seat is located. The official NUTS classification (Nomenclature of Territorial Units for Statistics) is adopted. This information is not reported when HEI’s activities are distributed in more than one NUTS3 region so that no main seat can be identified;  City: the name in English of the city/town where the main seat and most of the activities are located;  Postal code: the postal code of the official address of the HEI (postcode system is not in force in Ireland);  Geographic coordinates: longitudes and latitudes, based on the postcode of the official address. EUMIDA published the results of DC1 collection in 2011 (reference year 2008) and covered all European Union (except Denmark and France) + Norway and Switzerland. The only geographical information collected was the region of establishment at NUTS2 level. In order to test the feasibility of the geo-referencing process we created an inventory list of higher education institutions that deliver the PhD degree in the European Union and in EFTA countries (Iceland, Liechtenstein, Norway, Switzerland) by integrating information provided by the two projects and filling in the few remaining information gaps. In details:

 retrieving of the most recent information contained in ETER (reference year 2011) for the majority of countries concerned by this study. Hungary, Romania and Slovenia are missing;  retrieving of the missing data for Hungary and Romania from the EUMIDA DC1 (reference year 2008) integrated with ad hoc investigation online. The list of institutions has been retrieved by ETER in order to be updated and consistent with other countries;  retrieving of the missing data for Slovenia (reference year 2011) from Sapienza internal resource and ad hoc investigation online, on the basis of the list of institutions contained in ETER;  revising and updating of NUTS codes to the last version in order to allow for full interoperability with the Eurostat regional database (see below). The file lists 1,131 Institutions which represent slightly 50% of all institutions contained in the European tertiary education register (the remaining ones are institutions not awarding doctoral - ISCED8- degree). For each institution the file reports the following information:

 ETER code  the institution name in national language and in English  the NUTS codes at level 2 and 3  the name of the city

Additional geographical information (postcode, GIS coordinates) are available in the ETER database and can be easily integrated when relevant, thanks to the interoperability of the datasets ensured by the use of a unique ID code. With the information contained in ETER it is possible to locate each HEI within the city in a map of Europe. GIS coordinates have been calculated from postal codes automatically from Google Maps through the website http://www.doogal.co.uk/BatchGeocoding.php. It should be considered that postal code and address refer to an institution’s main seat and that usually activities are spread in a number of different buildings with different addresses. The information reported therefore cannot be used for micro-localization within a single city/neighbourhood, but rather for a broader localization at a regional, national and European level.

Drawbacks and limitations: multi-site institutions

In principle, the location of universities within NUTS2 regions is quite straightforward given that multi-site institutions with activities spreading across regional borders are not very diffused. Nevertheless almost 10% of HEIs comprised in the Authority file have secondary branches located in two or more regions (very rarely abroad).

This share increases up to 24% if we look at the NUTS3 level. The share of multi-site institutions is actually increasing around Europe as a consequence of two opposite phenomena: spread of universities activities from the original seat in the surrounding region (decentralisation and wider regional coverage); institutional concentration through the merger of small institutions and the creation of larger and more comprehensive HEIs which usually maintain at least partially the original seats and locations in different cities and regions.

At this stage it is not possible to disentangle information for multi-site institutions, and the whole university activities are located in the region of the main seat. This could create a slight distortion especially when data are analysed by disciplinary area. It could happen that all function in a disciplinary area are located in a secondary campus outside the region (e.g. the medical department of Università Cattolica del Sacro Cuore is located in Rome (ITI43) while the main seat is in Milan (ITC4): in the map of excellence the figures of the medical department will be attributed to ITC4 region instead of ITI43).

In ETER multi-site institutions are treated in two different ways depending on their typology:

 multi-site HEIs at national level (usually different establishments different regions within one country): are treated as a unique HEI and no disaggregated data are collected. A dummy variable for multisite institutions is collected and NUTS3 codes of local branches is requested;  international multi-site HEIs (secondary branches abroad): foreign campuses, consistently with UOE guidelines, are treated as self-standing HEIs in the country where they are established. In this case data are disaggregated at the level of the campus (although the inclusion/exclusion of figures in the parent country is not always easy to verify). Only two cases are in ETER: Webster University Vienna - Private University in Austria and The Branch of the University of Bialystok 'Faculty of Economics and Informatics’ Lithuania.

This step of analysis confirms the possibility to locate universities’ activities at regional level across Europe, with the limitations related to multi-site institutions recalled above.

The Authority File is attached to this report (Appendix 1).

5. INTEGRATING BIBLIOMETRIC DATA AT THE LEVEL OF INDIVIDUAL UNIVERSITIES

State of the art

In recent years several projects related to the use of bibliometric output of universities (and other research performing organisations) have been launched. Generally speaking they have analysed the research output (i.e. publications) of a sample of institutions around the world in order to publish a benchmark and/or a rank of their performance.

We briefly describe below three examples that we consider to be the most representative for the European context - Scimago Institutions Rankings (SI Ranking); Global Research Benchmarking System (GRBS); Leiden Ranking- in order to gain some insights on the feasibility of a full scale exercise of integration of data on scientific publications at the level of individual universities.

In addition, we describe also altmetrics and webometric information as possible additional information to consider.

Scimago Institutions Rankings

Scimago Institutions Rankings (SI Ranking) is published yearly by the SCImago Research Group and takes into account those organizations from any country, with at least 100 documents published in the last year of a five-year period. Data come from Scopus database for scientific literature, containing mainly scholarly journals and conference proceedings. The research group ensured the identification and disambiguation of institutions through the institutional affiliation of documents included in Scopus through the definition and unique identification of institutions (addressing issues related to institution's merge or segregation and denomination changes) and the attribution of publications and citations to each institution (manual and automatic system for the attribution of affiliations to one or more institutions).

For the purpose of this feasibility, we refer to 2013 Scimago world report, based on scientific production of the period 2007-2011. The following indicators are available:

 Output: total number of documents published in scholarly journals indexed in Scopus;  International Collaboration: Institution's output ratio produced in collaboration with foreign institutions. The values are computed by analyzing an institution's output whose affiliations include more than one country address;  Normalized Impact: normalized Impact is computed using the methodology established by the Karolinska Intitutet in Sweden where it is named "Item oriented field normalized citation score average". The normalization of the citation values is done on an individual article level. The values (in %) show the relationship between an institution's average scientific impact and the world average set to a score of 1, --i.e. a NI score of 0.8 means the institution is cited 20% less than the world average and 1.3 means the institution is cited 30% more than world average;  High Quality Publications: ratio of publications that an institution publishes in the most influential scholarly journals of the world, those ranked in the first quartile (25%) in their categories as ordered by SCImago Journal Rank (SJRII) indicator;  Specialization Index: it indicates the extent of thematic concentration /dispersion of an institution’s scientific output. Values range between 0 and 1, indicating generalist vs. specialized institutions respectively. This indicator is computed according to the Gini Index used in economics;  Excellence Rate: it indicates the amount (in %) of an institution’s scientific output that is included into the set of the 10% of the most cited papers in their respective scientific fields. It is a measure of high quality output of research institutions;  Scientific Leadership: leadership indicates an institution’s “output as main contributor”, that is the number of papers in which the corresponding author belongs to the institution;  Excellence with Leadership: it indicates the amount of documents in the Excellence rate in which the institution is the main contributor. The 2013 report ranks 2,744 institutions worldwide, in five sectors: government, higher education, health, private, others. 680 of them are higher education institutions located in Europe, within the perimeter of the present feasibility.

Global Research Benchmarking System

The Global Research Benchmarking System (GRBS) was intended to provide objective data and analyses to benchmark research performance in traditional disciplinary subject areas and in interdisciplinary areas for the purpose of strengthening the quality and impact of research. The GRBS is an open collaborative effort of the academic community. Data come from Elsevier's Scopus database including titles of types Journal, Conference Proceedings, and Book Series.

The scope of the first release of the GRBS (with reference year 2011) has been limited to three world macro-regions: US and Canada, Asia-Pacific, European Union (plus Norway and Switzerland). Universities are selected for inclusion in GRBS by examining research output in the 4-year window (2007-2010) at two levels:

 First in each of the third level subject areas the universities with the highest number of publications are identified. For Asia-Pacific the top 50 are taken and for US & Canada the top 40 are taken. The different depths are due to the differences in the size of the regions.  Then in each category a minimum cut-off of 50 publications is applied so that universities with fewer than 50 publications in the 4-year window in that subject area are not included in the list.  In addition, the 200 universities with the highest number of publications are identified in broader 2nd level categories. The reason is to include universities that have significant publication output in a broad category but not in any of its subareas. This results in adding a few additional universities since most are captured by searching the third level areas. The set of universities included in GRBS is then the union of all these resulting subject area lists. Any university that appears in at least one list is included in GRBS and analysed in all subject areas. In this way the GRBS is able to recognize universities that have particular niche strengths.

From the disciplinary area point of view, GRBS covers 23 of the 27 top level subject areas and 251 of the sub-areas of the All Science Journal Classification (ASJC) focusing on Science and Technology.

For the purpose of this feasibility we refer to the 2011 release that uses Scopus publication and citation data in the time window 2007 – 2010. The following indicators are available:

 Total Pubs: Total number of publications during a 4-year time window.  %Pubs in Top 10% SNIP: Percentage of Total Pubs published in source titles that are within top 10% of that subject area, based on the SNIP value [Ranked Outlets] of the last year in the time window. For the window 2007 - 2010, the SNIP values for 2010 are used.  %Pubs in Top 25% SNIP: Percentage of Total Pubs published in source titles that are within top 25 % of that subject area, based on the SNIP value of the last year in the time window.  Total Cites: Total number of citations within a 4-year time window to papers published in that time window. All citation counts used in GRBS exclude self citations.  %Cites from Top 10% SNIP: Percentage of Total Cites received from publications in journals that are within top 10% based on SNIP value.  %Cites from Top 25% SNIP: Percentage of Total Cites received from publications in journals that are within top 25 % based on SNIP value.  4-year H-Index: A university having 4-year H-index of X means that at least X of its publications (during that 4-year window ) have no less than X publications citing them (during that window). A 4-year H-index of a university is computed for a particular subject area.

The 2011 release includes 1,355 universities worldwide, and 606 affiliations are located in Europe within the perimeter of the present feasibility (there are cases of multiple affiliation corresponding to the same university).

Leiden Ranking

The Leiden Ranking is produced by the Centre for Science and Technology Studies (CWTS) and measures the scientific performance of major universities worldwide, based on data from the Web of Science bibliographic database produced by Thomson Reuters.

A sophisticated methodology has been developed for the identification of the perimeter of the universities, the evolution of their configuration over time and the presence of affiliated institutions (i.e. hospitals).

The universities that appear in the Leiden Ranking have been selected based on their contribution to articles and review articles published in international scientific journals in a 4- year period (only publications in core journals are included). A minimum threshold of the equivalent of 1,000 papers was required for a university to be ranked.

The CWTS Leiden Ranking provides statistics at the level of science as a whole and at the level of the following seven broad fields of science:

 Cognitive and health sciences  Earth and environmental sciences  Life sciences  Mathematics, computer science, and engineering  Medical sciences  Natural sciences  Social sciences

The above fields have been defined using a unique bottom-up approach at the level of individual publications rather than at the journal level. Using a computer algorithm, each publication in the Web of Science database has been assigned to one of these seven fields. This has been done based on a large-scale analysis of hundreds of millions of citation relations between publications.

For the purpose of this feasibility, we refer to the 2014 (most recent) edition based on publications in Thomson Reuters' Web of Science database (Science Citation Index Expanded, Social Sciences Citation Index, and Arts & Humanities Citation Index) in the period 2009–2012. The following indicators are available, in addition to absolute number of publications and citations:

 MCS (mean citation score): The average number of citations of the publications of a university.  MNCS (mean normalized citation score): The average number of citations of the publications of a university, normalized for field differences and publication year. An MNCS value of two for instance means that the publications of a university have been cited twice with respect to the world average.  PP(top 10%) (proportion of top 10% publications): The proportion of the publications of a university that, compared with other publications in the same field and in the same year, belongs to the top 10% most frequently cited.  PP(collab) (proportion of interinstitutional collaborative publications): The proportion of the publications of a university that have been co-authored with one or more other organizations.  PP(int collab) (proportion of international collaborative publications): The proportion of the publications of a university that have been co-authored by two or more countries.  PP(UI collab) (proportion of collaborative publications with industry): The proportion of the publications of a university that have been co-authored with one or more industrial partners.  PP(<100 km) (proportion of short distance collaborative publications): The proportion of the publications of a university with a geographical collaboration distance of less than 100 km, where the geographical collaboration distance of a publication equals the largest geographical distance between two addresses mentioned in the publication's address list.  PP(>1000 km) (proportion of long distance collaborative publications): The proportion of the publications of a university with a geographical collaboration distance of more than 1000 km.

The 2014 edition ranks 750 universities in the world of which 275 are located in Europe, within the perimeter of the present feasibility.

Altmetrics, webometrics and other complementary information

Additional data and information on research and higher education systems in Europe could be integrated in the EMES.

Interesting sources to be explored are the so called “altmetrics” (e.g. Cronin & Sugimoto, 2014), which include information on download of research papers, as reflected in social media

such as Twitter and Facebook, reader libraries such as Mendeley and ResearchGate, scholarly blogs and mass media. A dedicated effort is required to consider their inclusion taking into account their specificities, and particularly, data quality aspects.

Another source to be explored is represented by the webometrics ranking of universities (for more information see Box n. 2) whose data could be integrated and reported in the European Map of Excellence and Specialization.

BOX 2: Alternatives to bibliometric data: the case of Webometrics Ranking of World Universities The Webometrics Ranking of World Universities is an initiative of the Cybermetrics Lab, a research group belonging to the Consejo Superior de Investigaciones Científicas (CSIC), the largest public research body in Spain. The Cybermetrics Lab using quantitative methods has designed and applied indicators that measure the scientific activity on the Web. The cybermetric indicators are useful to evaluate science and technology as alternative/complement to the traditional bibliometric indicators. The Webometrics Ranking is the largest academic ranking of Higher Education Institutions worldwide (more than 5,000 institutions covered) published every six month since 2004 with the aim of providing reliable, multidimensional, updated and useful information about the performance of universities based on their web presence and impact. The original aim of the Ranking was to promote Web publication (Open Access initiatives, electronic access to scientific publications and to other academic material). However web indicators are useful for ranking purposes too as they are not based on number of visits or page design but on the global performance and visibility of the universities. Webometrics advocates a number of strengths with respect to bibliometric based rankings (reduced field of science distortion and better (indirect) inclusion of other missions like teaching or the so-called third mission; link analysis as a more powerful tool than citation analysis for quality evaluation; wider perimeter of research output including also informal scholarly communication). In Webometrics Ranking the unit for analysis is the institutional domain, so only universities and research centres with an independent web domain are considered. The first web indicator, Web Impact Factor (WIF), was based on link analysis that combines the number of external inlinks and the number of pages of the website, a ratio of 1:1 between visibility and size. This ratio is used for the ranking, adding two new indicators to the size component: number of documents, measured from the number of rich files in a web domain, and number of publications collected by Google Scholar database. Four indicators were obtained from the quantitative results provided by the main search engines as follows: Size (S). Number of pages recovered from four engines: Google, Yahoo, and Bing Search. Visibility (V). The total number of unique external links received (inlinks) by a site, according to Yahoo Site Explorer. Rich Files (R). After evaluation of their relevance to academic and publication activities and considering the volume of the different file formats, the following were selected: Adobe Acrobat (.pdf), Adobe PostScript (.ps), Microsoft Word (.doc) and Microsoft Powerpoint (.ppt). These data were extracted using Google, Yahoo and Bing. Scholar (Sc). The data is a combination of items published between 2006 and 2010 included in Google Scholar and the global output (2004-2008) obtained from Scimago SIR. The four ranks were combined according to a formula where each one has a different weight but maintaining the ratio 1:1. For more information: www.webometrics.info

Coverage of the European university landscape

All the three bibliometric databases analysed (and ETER itself) have a perimeter defined by a minimum size threshold which impacts their coverage of the European university landscape. In this respect, Scimago has the widest coverage (58%), followed by GRBS (49%) and finally Leiden ranking with only 24%. Given the uneven distribution of research output among HEIs and its concentration in larger and more research-oriented institutions, we can assume that the three databases have a better coverage in terms of total publications. The problem of coverage is well known in the literature, but still not solved. Three kinds of interpretational problems have to be taken into account:

 To the extent that indicators are derived from bibliographical databases such as Web of Science or Scopus, the outcomes depend upon the adequacy of coverage of such databases, and differences therein between countries and subject fields. Although in the past years both databases dedicate large efforts in expanding with conference proceedings and scholarly books, they are predominantly journal indexes. As a consequence, their coverage of fields in which sources other than journals play an important role in the scientific communication may be limited. Also, technologically oriented institutions aimed to develop new products and processes tend to produce outputs that are not well reflected in scientific-scholarly publications in the serial, refereed literature.  Results for an individual institution can only be interpreted properly when one takes into account the structure of the national academic system in which it is embedded, and the particular role of the university therein. Historical, political and cultural factors – including national or regional rivalry, different religious traditions or different concepts of academic education – may account for structural differences across national research systems (Moed 2006; Lambert and Butler 2006). Also, an institution’s performance should be viewed from the perspective of its position in the national and international research collaboration network.  In order to obtain a meaningful interpretation of bibliometric and other types of output indicators of research institutions, other types of information are needed. Three important types are the following: information on the degree of variability within institutions –as a complement to indicators at the level of institutions as a whole–, such as their degree of disciplinary specialization (e.g., Lopez-Illescas et al 2011; Calero et al. 2008); insight into the relation between ‘output’ and ‘input’, based on econometric models of research efficiency (Daraio, Bonaccorsi and Simar, 2015a,b); and, last but not least, information on the mission of research institutions.

Table 1 summarises problems of coverage listed above and reports some suggested solutions.

Problem Examples Solution Bibliographic databases Limited coverage of research Collect information from may reveal differences in publications in social sciences, institutions themselves and add adequacy of coverage of humanities; journal articles are this to the research output among not the primary output in annotation/background file research fields technological institutions Interpretation should take Historical, political and cultural Make available results at into account the structure factors may account for distinct aggregation levels: per of the national academic structural differences across research discipline within an system in which an national research systems institution; per institution; and institution is embedded per country; and scientific collaboration patterns Proper interpretation Indicator of the degree of Include such indicators in a requires info on the degree disciplinary specialization; of comprehensive information of variability within research efficiency; system on academic institutions institutions, on the relation information on mission and and public research between ‘output’ and regional function organizations ‘input’, and on their mission Table 1: Major problems in coverage and their solutions. Source: Moed, 2015.

Nevertheless, in order to have a full geographical coverage at European level an additional effort would be necessary in order to reach a higher coverage of the ETER perimeter. This would allow for a less biased comparison between national and regional higher education systems - with different average institution size, different level of concentration of activities and different subject specialization patterns.

6. LOCATING PUBLICATIONS OF UNIVERSITIES AND PROS ON A GEOGRAPHIC MAP

A full scale exercise to build a map of research require a preliminary process of disambiguation of affiliations and clear definition of the perimeter of HEIs and PROs. The three example of projects described above have faced similar problems and solved them in different ways. Different methodological choices (full count vs. fractional count of publications, different data sources, different time coverage, different coverage of publication typologies, etc.) anyway hamper the possibility to compare final results and consequently to assess the impact of the disambiguation process. Generally speaking the problem of disambiguation of affiliations in bibliometric exercises can be detailed in the following issues:  Researchers do not indicate their institutional affiliations in a uniform manner (different names, acronyms, abbreviation and truncation, evolving naming conventions);  Bibliographic database producers apply data capturing rules that re-format and modify the original affiliation data in the scientific articles they index. This activity might be sources of errors and distortions;  Additional information sources are needed to comprehensively identify research institutions or organizations in a particular country, especially large countries with complex and articulated research systems;  The definition of the institutional perimeter of a research institutions might be problematic (affiliated institutions, university hospitals, presence of umbrella organisations;  Author affiliations not reflecting the whole research process (i.e. the affiliation usually refers to the organization where the author works, but the research might be coordinated and/or funded by different institutions.

Table 2 presents a summary of these problems with examples and their solutions.

Problem Examples Solution a. Researchers do not Leads to variations in Use a validated thesaurus or indicate their institutional institutional names, incomplete authority file; consult national affiliations in a uniform names experts; use advanced manner. disambiguation software b. Data capturing by Database producers may Start from the raw affiliation bibliographic database change the order of data; do not use ill-understood producers is useful but may components of an affiliation features implemented by contain errors structure indexers c. Additional information Analysts of national affiliation Use a validated thesaurus or sources are needed to data need to “know what they authority file; consult national comprehensively identify are looking for” experts; research institutions d. Problems may arise with Position of academic or Consult national and the definition of research teaching hospitals and of institutional experts; create an institutions umbrella organizations such as annotation file with background university systems info on how a institution is defined; e. Traditional author Responsibility for research Initiate further research into affiliations may not program and for research this issue. Examine use of properly express the infrastructure may be split information from funding increasing complexities of amongst different organizations acknowledgements the research system Table 2: Major problems in affiliation disambiguation and their solutions. Source: Moed, 2015.

Towards an authority file for PROs

Academic institutions constitute in most countries by far the most important type of research entities. A second important group of (mainly) publicly funded research institutions is labelled as Public Research Organizations (PROs). Public Research Organizations can be divided into 4 categories or “ideal types” (OECD Innovation Policy Platform 2011):

 Mission Oriented Centres (MOCs), owned by government departments or ministries at a national level (e.g., INSERM in France; CIEMAT in Spain).  Public Research Centres (PRCs), publicly funded overarching research institutions such as CNRS in France, CNR in Italy, Max Planck Gesellschaft in Germany.  Research Technology Organizations (RTOs), often in the public sphere, private but not-for- profit, such as Fraunhofer Gesellschaft in Germany and TNO in the Netherlands.  Independent Research Institutes (IRIs), often at the boundary between the public and the private sector, denoted as centre of excellence, and recently founded.

While the scientific output of universities is the object of a large effort in the dedicated literature and several exercises have been carried out, much less is known, on a systematic and comparable basis, on the scientific output of PROs. This is because there are some issues that are still open: (a) establishing a comprehensive list of PROs; (b) disambiguating the innumerable ways in which PROs show up in bibliometric information and normalizing affiliations; (c) locating the scientific production of PROs at geographic level.

The creation of an Authority file for the European PROs requires a dedicated research effort, aimed at identifying and characterizing the perimeter of the institutions to be involved which is beyond the objective of this feasibility study.

For the purpose of this study, and on the base of a bibliometric study carried out and in progress at Sapienza, the “Elsevier Bibliometric Research Project: Assessing the Scientific Performance of Regions and Countries at Disciplinary level by means of Robust Nonparametric Methods: new indicators to measure regional and national Scientific Competitiveness”, we consider as a first “rough” starting point the list resulting from the combination of institutions included in the RPOs (Research Performing Organizations) Inventory provided by the European Commission, the list of the RPOs and RFOs (Research Funding Organizations) to which the ERA Surveys were submitted, and finally the lists of participants at the Framework Programmes initiatives and at the Horizon 2020 programme.

Breakdown by discipline

The breakdown of bibliometric data at disciplinary level indeed is crucial for a sound analysis of S&T activities, but requires an additional and dedicated effort.

Two alternative approaches have been proposed, respectively top-down focusing on journal classification or bottom-up working at the level of the individual publication.

Following the first approach journals are assigned to one or more research areas, bringing along all the publications appeared in the journal. GRBS follow this approach and makes reference to the Elsevier’s Scopus All Science Journal Classification (ASJC) which maps source titles in a structured hierarchy of disciplines and sub-disciplines allowing research activity to be categorized according to the field of research. ASJC classifies about 30,000 source titles into a two-level hierarchy. The top level contains 27 subject areas including a Multidisciplinary category and second level contains 309 subject areas.

The second approach works at the level of individual publications, assigning each publication to a research area through large-scale bibliometric analysis. The Leiden Ranking follows this approach, using a computer algorithm to assign each publication in the Web of Science database to a research area on the base of citation relations. Lowest level research areas are organized in a hierarchical structure which lead to seven top level fields (Waltman and van Eck 2012).

An alternative bottom-up approach has been experimented in GRBS to deal with interdisciplinary and emerging areas, which are not fully captured by a top-down approach. The

GRBS pilot effort with a focus on areas of sustainable development was based on keywords: candidate keyword lists were automatically generated and then checked by domain experts; resulting keyword lists were matched against keywords in the title, abstract, and list of author defined keywords of each publication.

A second order of issues arises when aiming at developing indicators combining bibliometric data with data on other dimensions of university’s profile. The European tertiary education register contains a breakdown of variables related to education (enrolled students and graduates), research activity (doctoral students and graduates) and resources (academic staff) by field of education. Data are broken down in the following eleven broad fields (plus one ‘unclassified’ category) according to the 2011 ISCED-F classification system:

 00 Generic programmes and qualifications  01 Education  02 Arts and humanities  03 Social sciences, journalism and information  04 Business, administration and law  05 Natural sciences, mathematics and statistics  06 Information and Communication Technologies (ICTs)  07 Engineering, manufacturing and construction  08 Agriculture, forestry, fisheries and veterinary  09 Health and welfare  10 Services

Although at the broadest level all the classification systems explored here converge to large disciplinary fields, there is no perfect match between the categories used in bibliometric databases and ISCED field of education. This is partially due to different scopes (ISCED-F is aimed at classifying educational activities rather than research ones) and partially to different classification choices.

See Section 8 for further discussion on concordance tables of different subject classification systems.

7. TOWARDS A EUROPEAN MAP OF EXCELLENCE AND SPECIALIZATION

Geo-referencing data on publications

Given the availability of an authority file of European universities containing their localization based on ETER, the easiest way for geo-referencing bibliometric data consists in matching affiliations reported in publication indexed databases and ETER IDs.

An alternative approach could start from the information on affiliations contained in each publication (usually affiliation contains the name of the institution and the address with the name of the city). This approach might lead to more accurate localizations, making reference to the actual address of the branch where the author work (instead of the institutional main seat). At the same time, it requires a substantial larger efforts for disambiguation of affiliation’s address and cities and interpreting the proper placement of the affiliation within its institutional framework. These issues have been addressed by Sapienza though a pilot application of a new alghoritm for disambiguation of RPOs affiliation (see Lenzerini, 2015b).

Although a classification of research institutions through their url is difficult to carry out, the information provided by the url that is reported in the webometric ranking of universities (see Box n. 2), could be extremely useful to map and disambiguate institutional affiliations from the publications’ databases.

For the scope of this feasibility we explored the first option and matched the list of universities in our authority file with the list of institutions reported in Scimago, GRBS, Leiden. In most cases, the same university appears in the three databases with a different name and these denominations only in a minority of cases perfectly correspond to the ETER institution name (either in national language or in English). Nevertheless, a non-ambiguous correspondence can be found between names variants with a dedicated effort.

The results are really satisfactory:

 100% of universities in the Leiden ranking found a correspondence with an ETER ID;  Over 99% of GBRS affiliations found a match with an ETER ID (all except Flanders Interuniversity Institute for Biotechnology in Belgium). In few cases an ETER ID found correspondence with two or three affiliation name variants in GRBS. The presence of university hospital in GRBS rose a specific issue for their correspondence with a HEI (which was checked manually case by case);  98.5% of higher education sector organizations in Scimago found a correspondence with ETER ID. In several cases, mostly located in France, the corresponding HEI in ETER is classified as ‘non phd awarding’. The Authority file has been integrated adding institutions that in 2011 were not awarding doctoral degrees according to ETER, but appear in one of the three bibliometric databases analysed. The final list is composed by 1,164 HEIs (33 more than the original list).

Table 3 shows a sample of cases of successful correspondence. The full version is available at Sapienza on request.

Table 3: University name correspondence table (sample) ID ETER: Institution Name ETER: English Name Leiden: University GRBS: University Scimago: Organization Leopold Franzens AT0003 Universität Innsbruck University of Innsbruck University of Innsbruck Universitat Innsbruck University of Innsbruck Katholieke Universiteit Katholieke Universiteit Katholieke Universiteit Catholic University of BE0056 KU Leuven Leuven Leuven Leuven Leuven Eidgenössische Technische Federal Institute of Federal Institute of Swiss Federal Institute of CH0012 ETH Zurich Hoschule Zürich Technology Zurich Technology Zurich Technology CZ0007 Masarykova univerzita Masaryk University Masaryk University, Brno Masaryk University Masaryk University Julius-Maximilians- Julius-Maximilians- Julius-Maximilians- DE0027 University of Würzburg University of Würzburg Universität Würzburg Universität Würzburg Universitat Wurzburg Justus-Liebig-Universität Justus Liebig University Justus Liebig University Justus-Liebig-Universitat DE0049 Giessen University Gießen Giessen Giessen Giessen Danmarks Tekniske Technical University of Technical University of Technical University of Technical University of DK0006 Universitet Denmark Denmark Denmark Denmark Universitat Autònoma de Autonomous University of Universidad Autonoma de Universidad Autónoma Universitat Autonoma de ES0016 Barcelona Barcelona Barcelona Barcelona Barcelona FI0006 Jyväskylän yliopisto University of Jyväskylä University of Jyväskylä University of Jyväskylä University of Jyvaskyla Université Bordeaux 2 Bordeaux Segalen University of Bordeaux Universite Victor FR0045 Universite BORDEAUX II Segalen University Segalen Segalen, Bordeaux 2 NATIONAL AND ΕΘΝΙΚΟ ΚΑΠΟΔΙΣΤΡΙΑΚΟ National and Kapodistrian National and Kapodistrian GR0020 KAPODISTRIAN University of Athens ΠΑΝΕΠΙΣΤΗΜΙΟ ΑΘΗΝΩΝ University of Athens University of Athens UNIVERSITY OF ATHENS Eötvös Loránd Eötvös Loránd University, Eötvös Loránd University, HU004 Eötvös Loránd University Eotvos Lorand University Tudományegyetem (ELTE) Budapest Budapest IE0004 Trinity College Dublin Trinity College Dublin Trinity College, Dublin Trinity College Dublin Trinity College Dublin Polytechnic University of IT0082 Politecnico di TORINO Politecnico di Torino Politecnique of Torino Politecnico di Torino Turin Vrije Universiteit NL0002 VU University Amsterdam VU University Amsterdam VU University Amsterdam VU University Amsterdam Amsterdam University of Tromsø - Universitetet i Tromsø - NO0003 The arctic university of University of Tromsø University of Tromsø University of Tromso Norges arktiske universitet Norway Uniwersytet Jagielloński w Jagiellonian University in Jagiellonian University, Jagiellonian University in PL0191 Jagiellonian University Krakowie Cracow Krakow Cracow

Universidade Nova de Universidade Nova de Universidade Nova de PT0008 New University of Lisbon New University of Lisbon Lisboa Lisboa Lisboa SE0003 Göteborgs universitet Göteborg University University of Gothenburg Göteborg University University of Gothenburg SI0001 Univerza v Ljubljani University of Ljubljana University of Ljubljana University of Ljubljana University of Ljubljana Univerzita Komenského V Comenius University in Comenius University, Comenius University in Comenius University in SK0001 Bratislave Bratislava Bratislava Bratislava Bratislava Imperial College of Imperial College of Science, UK0053 Science, Technology and Imperial College London Imperial College Imperial College London Technology and Medicine Medicine Source: our elaboration on ETER, GRBS, Leiden Ranking, Scimago IR data

Integrating information from other projects: the case of U-Multirank

Additional data and information on research and higher education systems in Europe might be integrated in the EMES.

An interesting source to be explored is U-Multirank, promoted by the European Commission. It is a multi-dimensional, user-driven tool to international ranking of higher education institutions. The dimensions it includes are teaching and learning, research, knowledge transfer, international orientation and regional engagement. Based on empirical data U-Multirank compares institutions with similar institutional profiles and allows users to develop personalised rankings by selecting performance measures/indicators in terms of their own preferences.

The first ranking in 2014 covered more than 850 higher education institutions (offering at least Bachelor’s degrees -ISCED 5 level) from 74 countries around the world; almost 600 of them are located in countries covered by the present feasibility study. It provides an institutional ranking of whole institutions as well as field-based rankings for electrical and mechanical engineering, business studies and physics. The coverage of institutions and fields has been extended in the 2015 ranking.

U-Multirank relies on a number of data sources in order to produce a multi-perspective ranking. Data sources include:

 self-reported data provided by the participating universities for the institution as a whole, as well as for the departments offering degree programmes (if any) related to the selected academic fields covered in the 2014 and 2015 editions;  student survey on their learning experience (addressed to a random sample of up to 500 students per field), in order to provide a “peer perspective” for prospective students;  bibliometric and patent data included in international databases. Indicator scores derived from bibliometric analysis are based on information extracted from publications that are indexed in the CWTS-licensed edition of the Web of Science (WoS) database (Science Citation Index Expanded, Social Sciences Citation Index, and Arts & Humanities Citation Index). Note that lower thresholds have been imposed compared with the CWTS Leiden Ranking: 50 WoS publications over the period 2008-2011 for the institution as a whole; 20 WoS publications for individual fields of science in the field-based rankings. The data about publications cited in patents as well as other patent related indicators are based on the EPO Worldwide Patent Statistical Database (PATSTAT). For the field-based patent indicators, the number of patent families was broken down into three sub-fields (electrical engineering, mechanical engineering, physics), based on existing technology classification schemes. U-Multirank makes use of three different types of indicators: Ranking indicators (institutional-level and field-based), Mapping indicators and Descriptives. The complete and updated list and description of the different measures and features assessed by U-Multirank’s research can be found at: http://www.umultirank.org/#!/measures?trackType=home&sightMode=undefined§io n=

The project has therefore collected and analysed a number of interesting data and indicators but they are not publicly available and cannot be re-used. In fact, the only information available is a classification in 4 categories: A (Very good), B (Good), C (Average), D (Below average), E (Weak).

Although the U-MultiRank system and the EMES information system proposed in the current Feasibility Study have different objectives and user groups – the former focusing on information on the various aspects of academic institutions primarily for (potential) students, researchers, and other interested user groups, and the latter on picturing research performance of publicly funded research from a regional perspective – there is a partial overlap in the data and technical infrastructures to be created in two projects, especially the creation of a large, flexible bibliometric database based on Web of Science and/or Scopus data, the de-duplication of academic institutions in bibliometric databases, the integration of bibliometric, educational, demographic and economic datasets and indicators, and the creation of a concordance between the various types of subject classification systems in these datasets.

Integrating other socio-economic indicators

The location of individual HEIs in their region at NUTS level 2 and 3 (see the Authority File above) leads to increased interoperability with other socio-economic indicators available at sub-national level.

The table below shows an overview or regional statistics made available by Eurostat at European level. A large number of socio-economic dimensions are covered in some cases with a good level of details and breakdown.

Table 4: Eurostat’s regional statistics indicators Regional data NUTS Indicators Coverage collection levels Population: age, age groups, sex, density, annual average, 1 January, demographic balance and crude rates

Area Demographic NUTS 3 / EU-28, statistics ( demo / Vital events: Births by age of the mother, NUTS 2 EFTA, CC reg_dem) Deaths by sex, age, Infant mortality, Infant mortality rates, Fertility rates by age

Life tables: Age specific death rate, Number dying, Probability of dying and surviving, Life expectancy at

Annual national GDP at current market prices, Real growth rate NUTS 2 EU-28, accounts (nama / of regional gross value added (GVA) at basic NUTS 3 EFTA, CC reg_eco) - GDP prices, Dispersion of regional GDP (%).

Annual national Gross fixed capital formation, Compensation of accounts (nama / employees, Employment (in hours worked), NUTS 2 / EU-28, reg_eco) - Branch Employment (in persons, NUTS 3), Gross value NUTS 3 EFTA, CC accounts added at basic prices (NUTS 3).

Annual national Allocation of primary income account of accounts (nama / households, Secondary distribution of income NUTS 1 / EU-28, reg_eco) - Household account of households, disposable Income of NUTS 2 EFTA, CC accounts households

People at risk of poverty or social exclusion: People living in households with very low work Income and living EU-28, IS, intensity, Severe material deprivation rate, At- NUTS 1 or conditions (ilc / reg- TR, NO, CH risk-of-poverty rate. 2 ilc) Housing conditions: Average number of rooms per person

Earnings (earn / reg- Mean annual earnings and annual bonuses as % EU-28, MK, earn-ses-06 / reg- of annual earnings by region and economic NUTS 1 TR, IS, NO, earn-ses-10) activity, and mean hourly earnings CH

Labour costs and their main components Labour costs (Wages and salaries, direct renumeration, EU-28, MK, statistics ( lc / bonuses and allowances) NUTS 1 TR, IS, NO reg_lcs) Number of employees, hours worked and hours paid.

Labour market Employment, Unemployment, Economically NUTS 1 / EU-28, statistics (lfst / active population, Population, Employment rate, NUTS 2 EFTA, CC reg_lmk) Unemployment rate, labour market disparities.

Persons aged 25-64 with lower/upper/tertiary education attainment: by sex (%).

Persons aged 30-34 with tertiary education attainment: by sex (%) Educational attainment, Early leavers from education and training: by EU-28, IS, outcomes and sex NUTS 2 NO, CH, returns of education Young people aged 18-24 not in employment MK, TR (edat / reg_educ) and not in any education and training: by sex (NUTS 2).

Participation of adults aged 25-64 in education and training (from 2000 onwards, %).

Indicators: Pupils/Students at ISCED level (0-6) by NUTS 2 regions - as % of pupils/students/regional population; ratio by Education (educ / EU-28, NUTS 1 and NUTS 2; participation rates NUTS 2 reg_educ) (educ_regind). EFTA, CC Number of students: by level of education, orientation; by age and sex.

Human Resources in Different indicators and breakdowns of HRST NUTS 1 / EU-28, Science & Technology stocks and flows (the latter being divided NUTS 2 EFTA, CC (hrst / hrst_st_reg) into job-to-job mobility and education inflows).

Research & Total intramural R&D expenditure by sectors of EU-28, Development (rd_e / performance, Total R&D personnel and NUTS 2 EFTA, CC re_p) researchers by sectors of performance and sex.

Patent applications to the European Patent Patents (pat / Office - EPO, patents granted by the USPTO and NUTS 3 EU-28 pat_epo_reg) triadic patent families.

High-tech industry Different high-tech indicators on business and knowledge- statistics, employment, trade, knowledge NUTS 1 / EU-28, intensive services intensive activities, innovation, R&D, Venture NUTS 2 EFTA, CC (htec / htec_emp_r) Capital and patents.

Annual regional statistics for services, industry, Structural business construction and distributive trades (Number of NUTS 2 EU-28, NO (sbs / reg_sbs) local units, Wages and salaries, Number of persons employed)

BG, CZ, DK, number of enterprises, births, deaths, and ES, FR, HU, survivals up to three years, and also the related Business demography NUTS 2 / AT, RO, SI, employment figures (persons employed and (reg_bd) NUTS 3 SK, FI, NL, employees). Breakdowns available by size class IT, LT, PL, and NACE Rev. 2. EE, PT

1. Households with Internet access, 2. Households with Broadband access, 3. Regular Information society use of internet by individuals, 4. Individuals NUTS 1 / EU-28, IS, (isoc / reg_isoc) who have never used a computer, 5. Individuals NUTS 2 NO who bought or ordered goods or services over the internet for private use.

Transport (tran / EU-28, IS, Road, rail, inland waterways, maritime and air NUTS 2 / reg_tran / reg_road / LI, NO, CH, transport of goods and passengers. NUTS 3 reg_otran) TR Source: http://ec.europa.eu/eurostat/web/regions/overview

The interoperability, on the one hand allows for analysis of the higher education system taking into account the broader socio-economic context where the institution is operating; on the other hand permits an easier inclusion of detailed information on the higher education system in territorial analysis for broader policy purposes (i.e. local development, industrial cluster, regional well-being, etc.)

Feasibility of selected indicators

The match based on ETER ID ensures the interoperability of HE survey/administrative microdata with two sources of data –bibliometric and socioeconomic one- which have been developed in quite separate ways. This match produces a great value added in terms of:

 (through the ETER ID match) Integration of bibliometric data (research output) with other information pertaining the university’s input (staff, financial resources), output (education) and organisational (institution category, legal status, etc.) structure, which are now available at European level with a good average coverage;  (through NUTS code) Integration with statistical indicators regarding the socio-economic context including population, economic, industrial and infrastructure statistics at regional level. In the following, we present a description of several indicators (including those of Box 1) for which we report the assessment of their feasibility.

Indicator Description and sources Feasible Notes BASIC INDICATORS A count of publications (journals, conference proceedings and book series from Scopus in Number of Scimago and GRBS; articles and reviews from YES publications WOS in Leiden) over a defined time period Geographical data: summing up all HEIs of the region Citation data are derived from citation indexes (available in GRBS and Leiden) Number of citations YES Geographical data: summing up all HEIs of the region The percentage of journal articles published in the top-ranked, high impact journals for the % of publications fields of research. YES from top journals Available in Scimago (top 25%), GRBS (top 10% and top 25%)

% of citations from Percentage of Total Cites received from top journals publications in top-ranked journals. Available YES only in GRBS (top 10% and top 25%)

Share of publications produced in collaboration % of publications with foreign institutions (according to authors’ with international YES affiliations). collaboration Available in Scimago and Leiden

SPECIALIZATION INDICATORS Revealed Scientific RSA calculated as the Balassa index (share of Advantage (RSA) of region’s products in a specific field over the regions (NUTS 2), YES share of EU/country in the same field, see Box normalized both at 3) EU and country level Summing-up publications of all HEIs in each region a rank of European regions can be Position of NUTS 2 / derived (only regions hosting main seat of NUTS 3 regions in a HEIs). European ranking by YES To reduce size effect, the rank can be discipline based on normalised against region’s total population. total publications Row data by discipline available in GRBS and Leiden

Summing-up citations received by publications of all HEIs in each region a rank of European Position of NUTS 2 / regions can be derived (only regions hosting NUTS 3 regions in a main seat of HEIs). European ranking by YES To reduce size effect, the rank can be discipline, based on normalised against region’s total publication total output or total population. Row data available in GRBS and Leiden EXCELLENCE INDICATORS Composite indicator including Number of publications, Number Several analytical methods are available in the of citations, and literature. YES Percentage of Row data available in Scimago and GRBS publications and citations from top journals CRITICAL MASS INDICATORS Flag (red/yellow/green) With available data in Scimago / GRBS / Leiden indicators stating this indicator can tell something on the number whether the region of HEIs (and the amount of their output) above (NUTS 2 / NUTS 3) a critical mass located in the region. With YES have or have not It could also be calculated as the share of HEIs limitations reached a given with critical mass on total HEIs in the region. threshold of The level of threshold is set in different ways in publications in given the three available datasets. disciplines RESEARCH PRODUCTIVITY INDICATORS

Number of publications per academic staff (in head count HC or full time equivalent FTE). Number of Supports cross‐institutional comparisons, Derived publications per unit adjusted for scale of institution. indicator of academic staff Limited coverage and comparability of academic staff data in ETER Excellent publications may be defined in two ways: - the number of products published in the top- ranked, high impact journals for the fields of Number of excellent research; Derived publications per unit - the amount of an institution’s scientific output YES indicator of academic staff that is included into the set of the XX% of the most cited papers in their respective scientific fields (highly cited publications). The latter is available in Scimago and Leiden (10% most cited) Number of total citations over a time period per academic staff (in head count HC or full time equivalent FTE). Number of citations Supports cross‐institutional comparisons, Derived per unit of academic adjusted for scale of institution and output YES indicator staff quality. Limited coverage and comparability of academic staff data in ETER

Excellent citations are defined as: Number of excellent - the number citations received from Derived citations per unit of publications in the top journals for the fields of YES indicator academic staff research. Available in GRBS only RESEARCH INTENSITY INDICATORS Feasible both at NUTS level 2 and 3. Number of Problems with multi-site institutions. Possible Derived publications per 000 YES refinement by broad age classes (ie. only indicator inhabitants population in working age 15-64)

Feasible both at NUTS level 2 and 3. Number of Problems with multi-site institutions. Derived publications per YES Available also in purchasing power standard indicator million euro GDP (PPS) for cross country comparisons Number of citations Derived As for publications YES per 000 inhabitants indicator

Number of citations Derived As for publications YES per million euro GDP indicator RESEARCH EXCELLENCE INDICATORS Two alternative versions (publications in top Number of excellent journals /highly cited pubs). Derived publications per 000 Feasible both at NUTS level 2 and 3, by broad YES indicator inhabitants age classes. Problems with multi-site institutions. Two alternative versions (publications in top Number of excellent journals /highly cited pubs). Derived publications per Feasible both at NUTS level 2 and 3, also in YES indicator million euro GDP PPS. Problems with multi-site institutions. Number of excellent Derived citations per 000 As for publications YES indicator inhabitants Number of excellent Derived citations per million As for publications YES indicator euro GDP

General note and limitations applying to all indicators

Publicly available datasets and rankings published online have a partial coverage of the European university landscape (from 25% to 60% of phd awarding institutions) and are not suitable for building a complete European map. They are examples used to demonstrate and test the feasibility of proposed indicators.

For the full scale realization of the EMES it is important to consider the following general limitations which apply to all the indicators discussed above:

 Availability of clean (disambiguated) and complete bibliometric data for a sufficient period of time and for all European countries;  Definition of the correct geographical location (the match with ETER ID is not perfect for multi- site institutions, possibility to integrate with the address reported by authors);  Availability of detailed information from the raw publication data files in addition to the simple count of publications and citations;  Development of a coherent system for research field breakdown.

BOX 3: An indicator of specialization: the Balassa Index

Subject mix is an important feature of the higher education institutions activities. A number of activities show specific disciplinary patterns (i.e. the ratio of students per academic staff, unitary cost of students, research orientations, etc.). Also the publication intensity (number of publications per researcher) largely varies by disciplinary fields, partly because of specific publication practices by discipline, partly because of the different coverage of disciplines in the publication databases.

Therefore, the institution’s specialization is an important indicator per se (contribute to institutional profilings), but it is also a necessary information for the proper interpretation and reading of other indicators. Comparing the total scientific production of two institutions (or two regions or countries) without knowing anything about their

field of specialization, could lead to misleading interpretation and incorrect assessment of their performance.

An easy and common way to measure the specialization is the calculation of the so called Revealed Scientific Advantage (RSA) indicator, originally developed within the international trade field of study, but currently applied in a wide number of fields. The RSA can be expressed by the following Balassa Index (BI):

푋 푋 푋 푐,푠 푐,푠 푐,푠 푋푐 푋퐸,푠 푋퐸,푠 BI[CS|E] = ≡ ≡ 푋퐸,푠 푆 푋푐,푠 푋퐸,푠 푋푐 ∑푠=1 푋퐸 푋퐸,푠 푋퐸 푋퐸

where Xc,s indicates the number of publications (citations)of country/region c in sector s, c denotes a specific country/region, E denotes a European reference, s a specific sector and S the total number of sectors considered. The formula therefore indicates the ratio between the country/region’s share of publications (citations) in a specific field over the EU’s share in the same field. A value of the indicator above 1 means a relative specialization of the territorial unit in the considered field compared to the reference EU level, while a value below 1 points to de-specialization.

Each territorial unit will necessarily be relatively specialized (de-specialized) in at least one field, unless it has exactly the same distribution than the European aggregate.

To give an example of how this simple indicator can be used within the context of a European map of excellence, we report an application using bibliometric data derived by the GRBS database. GRBS has a fair coverage of the European university landscape and a detailed breakdown of data by field (see description above for details).

To make the example easier to read we aggregated fields of research in 6 macro-areas broadly corresponding to the top level fields of education: economic and business sciences (ecbus), natural sciences (nat), information and communication technologies (ict), engineering and architecture (eng), agricultural sciences (agr), medical and pharmaceutical sciences (med).

The table below shows the relative specialization of European countries in terms of scientific publication, compared to the European aggregate (as defined in the GRBS perimeter).

Specialization of the scientific production (publications) of European countries

Country Pub_agr Pub_ecbus Pub_eng Pub_ict Pub_med Pub_nat

Austria 1.020 0.771 0.787 1.032 1.081 1.002

Belgium 1.427 1.166 0.946 1.241 1.084 0.900

Bulgaria 0 0 0.515 0.578 0.260 1.477

Switzerland 0.976 1.127 1.055 1.233 0.822 1.054

Cyprus 0 3.643 1.901 3.840 0 1.207

Czech Rep. 2.222 0.420 1.011 0.974 0.449 1.169

Germany 0.799 0.508 0.788 0.703 1.147 1.018

Denmark 1.996 1.232 1.048 0.749 0.890 0.949

Estonia 3.531 0 0.980 0.455 0.345 1.130

Greece 0.847 1.019 1.279 1.945 1.037 0.883

Spain 1.731 1.175 1.147 1.731 0.413 1.143

Finland 1.340 0.534 0.807 1.187 1.115 0.937

France 0.590 0.510 0.894 0.662 0.814 1.196

Hungary 1.390 0 0.327 0.289 1.055 1.133

Ireland 1.521 1.043 1.353 1.332 0.662 1.034

Italy 0.782 0.505 1.094 1.061 1.112 0.959

Lithuania 0.590 4.924 2.625 1.216 0 1.175

Luxembourg 0 0 0 0 0 1.920

Latvia 0 0 1.827 0 0 1.580

Netherlands 0.841 1.862 0.754 0.933 1.618 0.730

Norway 2.141 1.058 1.013 0.910 1.047 0.854

Poland 1.034 0 1.121 0.846 0.623 1.209

Portugal 1.596 0.801 1.732 1.062 0.323 1.156

Romania 0.188 0 1.983 0.735 0.137 1.404

Sweden 1.122 0.628 0.845 0.498 1.337 0.881

Croatia 1.186 0.661 1.961 2.052 0.348 1.076

Slovakia 1.587 0 1.502 0.934 0.207 1.286

United 0.772 1.921 1.112 1.084 1.011 0.967 Kingdom

Source: our elaboration on GRBS data

The same information is shown by the flower charts below which report the European median value in the central circle of the flower for comparison.

The analysis could be applied, mutatis mutandis, with similar results to the regional level to show regional patterns of specialization.

The results of specialization might vary using different variables, i.e. citations instead of publications. Given an adequate availability of comparable data, it is possible to compare specialization in terms of publications and citations. The chart below shows an illustration on the Italian regions, in the field of engineering and architecture.

Specialization of Italian regions in engineering and architecture

Source: our elaboration on GRBS data

From the comparison of the two distributions some interesting insights on the specialization of the scientific production of a region can be derived (although results have to be interpreted with caution). Regions like Trento, Friuli V. Giulia, Calabria and Emilia-Romagna are more specialized in terms of citations than in terms of publications, pointing to a higher “average quality” of their scientific output in the considered field.

8. CONCORDANCE TABLES OF DIFFERENT SUBJECT CLASSIFICATION SYSTEMS

Introduction

The emergence of “big data” scientometrics, the increasing emphasis on multi-dimensional assessment, and the increasing interest of research institutions and their funding organizations in valid, reliable and useful indicators, creates the need to analyse, further develop and – if possible- align a series of relevant classification systems. Table 5 presents a summary of the major trends in the field of quantitative research assessment.

Table 5: Major trends in scientometrics Biblio/sciento/informetrics as big data science

More (Large) datasets electronically available

Combination of large datasets

More interest in research assessment, metrics

Multi-dimensional approaches , integral views

Linking components within a system

Comparing, benchmarking

Emphasis on accountability, productivity, societal impact

Quantitative research assessment is more and more becoming a “big data” science in which large datasets on different aspects of the science, technology and innovation (STI) system are being combined. Combining different datasets is especially important in studies analysing the relationship between the various parts of the STI system or between the various components within each part. At the same time, international organisations such as the OECD, UNESCO, and EUROSTAT generate standardised statistics on R&D activities, both in terms of input and output.

For instance, studies on the science system seek to capture the relationship between funding and scientific-scholarly output in the various domains of science and human scholarship. But statistics from funding organizations may use subject classifications that are different from publication- or journal-based scientific subject classifications. The description and evaluation of teaching and research activities of staff members in academic departments or institutions is confronted with the problem that teaching and research subject classifications do not coincide.

Luwel (2004) noted that attempts to calculate per scholarly field productivity measures relating these input measures to output indicators are hampered by the two types of statistics giving aggregate measures based on different subject classification systems. Studies on the science- technology-industry interface are confronted with the need to create concordance tables between technology and industry subject classifications (e.g., Schmoch et al., 2003) and between patent (IPC) and technology classifications (Schmoch, 2008; Lybbert and Zolas (2014). Table 6 gives an overview of the various science-related concordance tables which need to be created.

Table 6: Science-related concordance tables Type of analysis Concordances needed

1 Output - Input Journal (WoS, Scopus) – input stats (OECD); funding data (NSF)

2 Research – Teaching Research – Teaching Subject concordance tables

3 Science – Technology Journal – Patent classification

4 Science – Industry Journal – industrial Sector

It must be noted that also within a particular domain different classification systems may exist for which concordance tables are needed. For instance, the two multi-disciplinary, bibliographical databases Web of Science (produced by Thomson Reuters) and Elsevier’s Scopus have different journal-based subject classification systems, both at the top level (“disciplines”) and on a more granular level (“subfields” or “categories”). As a result, bibliometric indicators derived from these two databases cannot always be easily combined or directly compared.

In the next section we report the outcomes of a literature search on the topic. This search aims to systematically analyse which attempts have been made to develop concordance tables between different subject classifications; which concordance tables have actually been created; which methods were used to create these, and how successful these methods were, in terms of the degree of validity of the proposed concordance. Further below it is illustrated that a pragmatic approach may indeed lead to satisfactory solutions, by discussing concordance systems created in the past years. Finally, the last paragraph of this section draws conclusions and makes recommendations.

Results from a survey2

As a first step a manual search was carried out to identify a core of relevant “seed” articles with essential keyword like “classification”, “taxonomy”, “concordance table”. Subsequently, from their titles and keywords, relevant keywords were extracted to build up a query with which an automatic search was conducted in Scopus. A wide-ranging analysis was performed using TITLE-ABS-KEY

2 This section is based on Daraio, Di Costa and Moed (2014, 2015).

search in the combined field that searches abstracts, keywords, and article titles. Table 7 presents the terms from the “seed articles” that were used as argument in the query.

Table 7: List of terms used in the search

"industr* classification" "patents AND paper*" MeSH AND classification "statistical classification*" "Classification Systems" IPC "Classification of Industries" "Subject clustering" AND "ISI AND Technology category classification" subfields AND publications hierarchical AND taxonom* Hierarchical AND classification keyword AND classification Taxonomy AND Classification "Research literature" AND maps clustering AND "scientific texts" "patent classification*" "Patent Categorization" AND IPC "document categorization" "classification scheme" AND "subject-classification schemes" "science fields" categorisation AND patent "Technology classification" AND indexing” "Context*aware systems" "map of science" "automatic classification" AND "Hybrid Clustering" AND "medical data classification" "scientific literature" classification Classification AND articles "publication*classification" Patent AND Categorization Taxonomy AND Mapping " hybrid mapping" Patent AND Science patent AND "classification "terminology mapping" "classification* AND journals" system" coding and "classification "Technology Concordance" "Structure AND literature" systems" "cross*classification table*" Manufacturing AND classification "concordance table*"

Source: Daraio, Di Costa and Moed (2014).

435.855 records were obtained. In this set the following five additional selections were made:

 Articles, reviews and conference papers only (number of records was reduced to 415.485);  Documents written in English only (reduction to 338.593 records);  Documents included in the following Subject Area: Engineering, Computer Science, Social Sciences, Mathematics, Economics, Econometrics and Finance, Business, Management and Accounting, Decision Sciences (reduction to 50.656 recs);  Given a large amount of records remaining in the set, its number was further reduced by selecting those that contained at least one of the following keywords: “patent”, “classification”, “science”, “field”, “map*”, “taxonom*”. The remaining set contained around 900 records.  These records were then manually analyzed, on the basis of their title and abstract. After removing duplicates they were added to the list of those found in the initial manual search.

The final dataset consisted of 167 records indexed in Scopus.

The software tool VosViewer (www.vosviewer.com) was used to analyse the contents of the set of 167 articles. A map was created based upon a text analysis of the abstracts using the model “Create a map based on a text corpus”. The text analysis resulted in 3,470 terms. Setting a frequency threshold at 10, 62 terms were selected. Figure 1 shows a map of the 37 most relevant ones. The graphical representation of the keyword structure reveals basically two worlds of work: “technology” and “science”. The technology world consists of two clusters, one related to technology-industry concordance and another cluster on patent- technology classifications. The science world relates to subject classifications based on scientific articles or journals.

Figure 2: VosViewer map of 37 most relevant keywords

Source: Daraio, Di Costa and Moed (2014).

A detailed, paper-by-paper analysis reveals from the science cluster a series of papers by Glanzel and co-authors on citations- and text-based subject classifications of scientific journals indexed in Web of Science, and the development of hybrid classifications, including Janssens et al. (2009). A second series, in the technology domain, relates to the concordance between technology and industry classifications (e.g., Schmoch et al., 2003) and between patent (IPC) and industry classification (ISIC) systems, including Schmoch (2008), and Verspagen, Van Moergestel and Slabbers (1994).

Approaches and systems developed in the past. Correspondence tables between Intellectual Patent Classification (IPC) and Fields of Science (FoS); and between IPC and industrial classification See Appendix 2 were these two relevant types of concordance tables are reported in details together with an overview of previous concordance tables reported in other earlier studies. A rich bibliographic list completes the Appendix.

Correspondence tables between Fields of Education (FoE) and Fields of Science (FoS) Correspondence tables from the Eumida project3 The publication intensity –defined as the number of publications per researcher– largely varies by disciplinary fields. This observation is partly linked to specific publication practices by discipline, partly to the different coverage of disciplines in the publication databases. The two aspects cannot be clearly separated. Therefore it is not appropriate to exclusively look at the total publication numbers of HEIs, as this would favour HEIs with a focus on fields with high publication intensity. As more citations to other publications are possible in fields with a high number of publications, the fields with high publication intensity are also those with high citation intensity. In the EUMIDA study, the fields of science were classified as indicated in Table 8. The classification by fields of

3 This paragraph is taken from the EUMIDA (2010) final study report.

science and technology (FOS) was introduced in the Frascati Manual in the 1960s and slightly revised since then. The last revision of FOS classification was conducted by the OECD in 2006 (OECD DSTI 2007).

For the bibliometric searches, a concordance of the classification of the documents in both databases and the 6 fields of science according to Table 8 had to be constructed. In most cases this is not problematic, but some specific issues have to be commented on. In classifying the Fields of Education for the EUMIDA survey, Social sciences and Education were separated. In the classification of the research-oriented Fields of Science, the Educational sciences are part of the Social sciences. Computer and information sciences belong to the Natural sciences according to the FoS classification, in contrast, Information engineering to Engineering and technology. However, the classifications in publication databases are based on journals and not on single articles. In journals a strict separation between computer sciences and information engineering (software and hardware) cannot be realised. In these cases the journals have multiple classifications, so that fractional counting was applied. Also in other fields, a clear distinction of Engineering and Natural sciences is difficult, for instance in Environmental or geological engineering. In this case, the publications were classified in Engineering. A further borderline case is Radiology. This discipline is classified in the Medical sciences, although a relevant part of the publications deals with technical hardware. Nevertheless, also these articles are associated to Medical sciences, as a finer classification is not available. Finally, the concordance scheme of Table 8 was constructed. In the case of WoS, each field is identified by a code of two characters, in the case of Scopus by a code of 4 digits.

Table 8: Concordance Scheme for the Fields of Science Classification for Bibliometric searches in WoS and Scopus

Source: EUMIDA Final Study Report, 2010

Comparability of fields of education and fields of science

One of the areas where comparability across countries is not perfect is the classification of students into fields of education. How severe is this problem? Luckily enough, the classification of fields of

education is one the areas where the international statistical systems have been working more intensely for years, so that classifications are quite robust and updated regularly.

Another problem, which must be mentioned here, is the relation between fields of education, as classified in the UOE Manual, and fields of science, as classified in the Frascati Manual.

We noticed that there is no single scheme of classification for all EUMIDA variables, since these represent different realities and have to be classified using different principles; hence, the EUMIDA approach was to keep the original classification schemes and to build afterwards correspondence tables where this is required. Broadly speaking, EUMIDA has adopted the following classification schemes:

 the classification by fields of education of the UOE manual (UOE manual 4.2), which is based on the contents of the educational programmes; it thus used to classify programs, students and degrees (based on the program in which they are enrolled);  the fields of science and technology (FOS) classification of the Frascati manual (Frascati manual 3.6.2), which is based on fields of science where R&D activities are performed; EUMIDA has extensively checked its usability for classification of academic personnel (as alternative to FOE).

FOS and FOE classification are based on different principles (research fields vs. content of educational programs) and thus classify different objects. Nevertheless, at the 1st level of classification the two classifications are broadly compatible, with some rather minor differences. The following table provides an overview.

Table 9: Correspondence between Fields of education and Fields of science

Source: EUMIDA Final study report, 2010

Correspondence tables from the ETER project Classifying HEI activities by field is of primary importance for different reasons: first, HEIs display very different mixes of subjects and this represents an important dimension of HEI diversity, which needs to be investigated carefully. Second, different subject domains might display different forms of organization and mixes of activities, owing to underlying differences in disciplinary structures, and hence using data disaggregated by field is of primary importance.

In principle, all HEI activities covered in ETER could and should be disaggregated by subject. However, on the one hand availability of disaggregated data is considered as problematic for a number of variables. On the other hand, different activities are classified using different principles – for example student’s based on the topic of the curriculum to which they are enrolled and scientific publications based on the scientific domain of the journal where they are published.

The general principles of ETER are the same as Eumida, i.e. to keep the original subject classification of the specific type of data; correspondence tables can then be constructed for analytical purposes afterwards.

Isced fields of education and training classification For the classification of students and degrees, the classification of Fields of Education and Training 2013 (ISCED-F) should be used at the first level (broad fields). This classification is consistent with

previous Fields of Education classification 1997 and 2011, as well as with Eurostat Fields of Education and Training 1999 (FOE 1999) except for including a specific code for Business, Administration and Law (included in social sciences in the previous schemes) and for ICT (included in natural sciences in FOE-1997).

Table 10: Fields of Education and Training correspondence table Code Name Subfields ISCED 1997 FOE 00 General 001 Basic programmes and 01 Basic programmes programmes qualifications 08 Literacy and numeracy and 002 Literacy and numeracy 09 Personal development qualifications 003 Personal skills 01 Education 011 Education 14 Teacher training and education science 02 Humanities and 021 Arts 21 Arts Arts 022 Humanities 22 Humanities 023 Languages 03 Social sciences 031 Social and behavioral 31 Social and behavioral science science 32 Journalism and information 032 Journalism and information 04 Business and 041 Business and 34 Business and administration law administration 38 Law 042 Law 05 Natural Science, 051 Biological and related 42 Life sciences mathematics sciences Part of 62 (natural parks and and statistics 052 Environment wildlife) 053Physical sciences 44 Physical sciences 054Mathematics and statistics 46 Mathematics and statistics 06 Information and 061 Information & 48 Computing communication Communication Technologies technologies 07 Engineering, 071 Engineering and 52 Engineering and engineering manufacturing engineering trades trades (plus most of 85 and construction 072 Manufacturing and environmental protection) processing 54 Manufacturing and 073 Architecture and processing construction 58 Architecture and building 08 Agriculture, 081 Agriculture 62 Agriculture, forestry and forestry, 082 Forestry fishery (minus natural parks and fisheries and 083 Fisheries wildlife) veterinary 084 Veterinary 64Veterinary 09 Health and 091 Health 72 Health welfare 092 Welfare 76 Social services 10 Services 101Personal services 81 Personal services 102 Safety services Part of 85 environmental 103 Security services protection (community 104 Transport services sanitation and labor protection and security) 86 Security services 84 Transport services Source: ETER project. Handbook for Data Collection, 2014

The correspondence between FOE and FOS was done also in the Eumida project. However the ETER project has detailed and updated the previous Eumida correspondence and hence it is the most recent correspondence table available (see Table 11).

Table 11: Correspondence table FOE-FOS

ISCED-F 2013 Fields of Science FOS - 2007 00 General programmes and qualifications - 01 Education 5.3 Educational sciences 02 Humanities and Arts 6. Humanities 03 Social sciences 5. Social sciences without 5.2, 5.3 and 5.5 04 Business and law 5.2 Economics and Business 5.5 Law

05 Natural Science, mathematics and 1. Natural sciences without 1.2 statistics 06 Information and communication 1.2 Computer and information sciences technologies 07 Engineering, manufacturing and 2. Engineering and technology construction* 08 Agriculture, forestry, fisheries and 4. Agricultural sciences veterinary 09 Health and welfare 3. Medical sciences 10 Services - * Includes urban planning which is in the FOS classification is included in 5. Social sciences

Source: ETER project. Handbook for Data Collection, 2014

Conclusions and recommendations

As outlined above, the development of concordance tables of the various types of subject classification systems is a very important topic. Our conclusion is that several studies have illustrated how a pragmatic solution may generate useful results; however, much more work is needed. We believe that the issue of concordances of various types of subject classification systems needs much more attention than it has received thus far.

We propose to the European Commission to consider taking well structured actions aimed to further develop concordance systems, in a joint effort between biblimetric/ informetric experts in the field, EU key organizations such as EUROSTAT, but also relevant departments of external organizations, especially OECD and UNESCO.

Also, given the crucial importance of bibliometric databases Web of Science, Scopus and also Google Scholar4 in the production of science and technology indicators, the European Commission should consider to take initiatives for developing a standard science classification system that is implemented in each of the three databases, involving the producers of these databases.

Subject classification systems can be developed at various levels of aggregation. Generally speaking, the top level (groups of disciplines, e.g., “natural sciences” or “social sciences”) contains a limited number of classes, typically 5. A next, intermediary level relates to disciplines, typically 20-50 in number (e.g., “physics”) ; last but not least, a granular system of several hundreds of subfields or categories exists (e.g., “linguistics; “nano science”; “cardiology”). It is proposed to focus at the top level. Our working hypothesis is that at this level the creation of concordance tables between the various systems is feasible. In a next step, it can be examined to which extent a concordance at the intermediary level of 20-50 disciplines is possible.

9. VISUAL ANALYTICS FOR A PILOT EMES

In this section, the principles and possible design choices of the Visual Analytics Environment (Cook abd Thomas, 2005, Keim et al. 2010) for exploring excellence in research data will be explained. A different approach in data exploration is necessary to cope with multidimensional (e.g. a lot of different metrics for the evaluation of research results) and abundant data (e.g. a lot of different research institutions spread around Europe; connections among research institutions).

Explorative analysis does not follow a precise goal a priori, and the user is asked to formulate not only the solution but also the questions during this kind of analysis. Moreover, the exploration of the data can generate in the user new insights not foreseen at the beginning of the analysis.

In order to support this kind of tasks -particularly suited when talking about evaluation of excellence in research- a powerful visual environment (capable of allowing the user to quickly inspect the data, refine the analysis using visual instruments and then prepare the data for reporting in an easy and full descriptive way) is a strong added value.

The visual environment will have its design principles based on the discipline of Visual Analytics. Visual Analytics is ”the science of analytical reasoning facilitated by interactive visual interfaces”. It is particularly suited for problems that cannot be treated otherwise because of their size, complexity, and need for closely coupled human and machine analysis. Visual analytics brings

4 Several studies have compared Google Scholar with traditional bibliometric databases. Just to cite a few, see Jacso (2005), Bakkalbasi et al (2006), Bar-Ilan (2008), Delgado-Lopez-Cozar et al (2013), Falagas et al (2008), Franceschet (2010), Meho and Yang (2007).

together advancements in analytical reasoning, interaction, data transformations and representations, data visualization and analytic reporting (see Figure 3). Visual Analytics ties together several scientific and technical communities from computer science, information visualization, cognitive and perceptual sciences, interactive design, graphic design, and social sciences (Angelini et al. 2014 a,b).

For an interesting analysis of the implications of an Ontology Based Data Management approach for Visual Analytics, see Santucci (2015).

Figure 3: Visual Analytics Cycle

The approach of using Visual Analytics for exploratory analysis of research data has the following advantages over the previous solutions adopted:

 It allows to present the data in an easier way, allowing to visually highlight areas of interest and lead the exploratory process using visual cues.  It scales with respect to the data dimensions, allowing to aggregate/refine data and present a compact yet descriptive visual overview of the data, with the possibility to obtain fine details on demand.  It is a dynamic environment (Odijk et al 2012): new data can be uploaded at any time while the framework is still in place. New analyses can be conducted, in visual form on the already available datasets without the need to restart each time the whole computational workflow.  Using visual elements it helps in creating/sharing report of discovered insights about the data, without the need to create ad-hoc report solution.  Multi-coordinated views approach makes easier for the user to navigate and explore the dataset. Moreover, it makes the framework easily expandable, with the possibility to add new visualizations on top of existing ones.

General Design

The system of visual analysis of data on excellence and specialization of research in Europe will be based on a series of multiple views, coordinated and interactive, following the dictates of the discipline of Visual Analytics. Coordination among visualizations will help the user in the analysis of the refinement: beginning from a visualization, the user will select particular areas of interest, which values will be translated to the other visualizations in real-time, allowing to inspect different dimensions of the analysis. In this way the user will always have the data under evaluation presented in the most appropriate way, making the discovery of new insights as a quick and effective process.

Among different visualization paradigms, the geographic representation of different research institutions seems the more natural, as an entry point of analysis, for the evaluation of research

excellence. The user will be presented with a geographic representation of Europe, where the colour will be associated with a selected evaluation metric. Areas with an high score on the metric will be represented with a stronger tone of colour, allowing a quick discovery of the better performing areas. In order to refine the analysis on the geographic layer, the data will be presented accordingly to the NUTS classification. In this way all the data will result aggregated on different level of details, and will allow to precisely identify areas of strong interest, both for excellent results or anomalous performance.

Moreover, in order to identify into the same zone the various research institutions involved, the geographic layer will be geo-referenced and will allow to display the various research institutions with their exact position. Aggregated indicators will be showed for each area selected (e.g. number of research institutions contained) in order to keep the user always aware of important data.

The analyst will be able to further refine the analysis in terms of geography: from the NUTS level 1, she can refine the analysis up to the level NUTS 3; the exploration will take place according to the principle of direct manipulation: selecting an area with the right button (drill-down), it will be expanded into the next level of the hierarchy; a second click instead will allow to go up one level (roll-up). The transition among the various geographical levels is visible in figure 4.

Figure 4: Navigation between different geographical levels.

The user will be able to navigate the data from the most general level (NUTS1 classification) to the most specific (single research institution or set of research institutions) passing through intermediate levels (NUTS2 and NUTS3). The resulting data will be aggregated accordingly.

Starting from the geographic layer, the user will produce a first refinement of the data, to be analyzed in greater detail in the coordinated visualizations, more focused towards a domain specific analysis.

Given the high multidimensionality of the data, a good visualization to use in order to characterize each area on its dimensions is the radar view. Radar view allows to plot multiple dimensions on the same visualization and to compare resulting shapes among different research institutions. By direct comparison the user will quickly identify in which dimensions the particular set of research institutions is performing well and in which not. Moreover, the user could also set particular reference level (by simply connecting the values on the various axes constituting the radar view) and to compare multiple areas or research institutions against this reference level. A mock-up of this solution is visible in Figure 5.

This refinement will more easily identify areas where the European “excellence” in scientific production is localized.

Figure 5: Breakdown of comparison, using radar display, seven reference metrics for 3 selected universities in the geographic map

Furthermore, by clicking on the legend of one of the dimensions presented in the radar view, a second radar view will be instantiated which will automatically show a set of metrics homogeneous to the selected one: in this way the first radar characterizes the zones of interest in a global manner, while the additional radar view will focus on specific topics (e.g. bibliometric indicators, income indicators, and so on).

Finally, selecting the analysis task related to international collaborations, the map will show the selected area and the connections that it has, in terms of scientific production, with the other areas, both in Europe and worldwide.

Both the thickness and colour of the connecting links can map additional information, like the amount of collaborations between two or more research institutions and the weights within these collaborations (e.g. how many authors from the different institutions) or the timeliness of these collaborations (how recent the last publications/collaborations are).

The analysis can be refined, using clustering mechanisms and highlighting, operating on three main types of connections.

National Connections: degree of collaboration between research institutions belonging to the same country;

European Connections: degree of collaboration between research institutions belonging to any of the countries belonging to the European Union;

Worldwide Connections: degree of collaboration between research institutions belonging to any of the EU countries and external research bodies (that is belonging to any country outside Europe).

These connections can be represented either on the geographical layer (in order to maintain the information at local level) or in a coordinated visualization, for example based on the chords diagram reported in Figure 6.

Figure 6: chord diagram representing the connections among different worldwide countries. The user can select one or more countries on the border of the big circle; their interconnections, represented as links with a thickness proportional to their numbers, will be highlighted (red color)

It is out of the scope of this feasibility study to design the whole visual framework, and the reader is encouraged to think in perspective about the high versatility and efficacy of the proposed conceptual framework. What is important to show here is the applicability of this framework to the proposed scenario of representation of excellence and specialization of research, and how it brings added values to the explorative analysis. Nevertheless, in order to additionally support this thesis, a proof-of-concept prototype has been designed and implemented, and it is described in the next section.

Proof-of-concept prototypal application

In this section a description of a proof-of-concept prototype is presented in order to validate the suitability and efficacy of the previously described approach. It is possible to visit and interact with this prototype at: http://151.100.59.83:11768/V-Eureka/.

The implemented system presents the user with a display that maps metrics specific evaluation on geographical representation of the European countries, according to the NUTS classification (NUTS2 level, but the solution is general and in the final design can transition seamlessly among the different NUTS levels). The metrics displayed can be selected from a set of metrics (e.g, number of publications, number of students enrolled, total income) by a drop-down menu. Each area of the map will have a color corresponding to the value on the selected metric, as visible in Figure 7.

Figure 7: example of mapping between the geographic representation and selected evaluation metric, based on the number of publications. In green well performing regions, in white poorly performing ones (for regions in grey data is missing).

The analyst will also be able, by clicking on an area of interest, to enable a second display, based on the paradigm of "radar display", highlighting the behavior of the area compared to a more general set of metrics (and not only the one mapped to geography) characterizing the analysis.

This view allows an easy comparison between diverse areas. The analyst can select multiple areas at the same time, allowing an easy comparison between the forms that the scores on the various metrics will produce in the radar view. In addition, the analyst can select the set of dimensions on which instantiate the radar view, allowing to exclude and/or include further dimensions based on her specific analysis requirements. An example of this behavior is shown in Figure 8.

Figure 8: the user is able to select multiple NUTS2 areas from the geographic layer by simply clicking on them (left). The overall description, based on the selected metrics, will be presented as a radar view (right) maintaining the same color code.

By Passing the mouse over a shape it would be possible to highlight it in order to make clear comparative analysis among different research institutions/areas selected (Figure 9). Moreover, passing the mouse over the vertex of the shapes will reveal their score with respect to the selected evaluation metric (Figure 10).

Figure 9: highlight of the radar polygon of a research institution; having the others polygon put in the background allows for easily spotting the differences on various evaluation metric scores with respect to the selected one.

Figure 10: numerical information is still present and shown by simply hover the mouse on the polygon’s vertices.

Figure 11: Screenshot with the menu of selection of the metrics from the Pilot Visual EUREKA (European Research Excellence and Knowledge Analysis) system developed by Sapienza.

Figure 12: Screenshot with the implemented tooltip.

The pilot tool Visual EUREKA (European Research Excellence and Knowledge Analysis) system developed by Sapienza is available at the following link: http://151.100.59.83:11768/V- Eureka/.

10. ASSESSMENT

The current study can be seen as a proof-of-concept, needed in order to verify the feasibility and provide further insights for a full scale exercise.

The assessment of the feasibility of the proposed approach is reported below in Table 12 and is based on the criteria established in Section 3, namely:

 Availability of data on publications  Standardization, openness and interoperability with other sources of data available;  Data quality;  Continuity;  Extensions and scalability;  Interactivity and usability;  Existence of concordance tables among different subject classifications.

Description Assessment according to the criteria 1 Feasibility of geo-referencing Information available in ETER. Good data quality but: information pertaining to incomplete coverage of countries, update to last NUTS excellence in S&T at NUTS 2 version. Integrated and updated by Sapienza (Authority and NUTS 3 level of European file in appendix). Open issue: multi-site institutions. universities Continuity ensured (new tender for collecting other two years). Bibliometric data and third mission data are not included. 2 Feasibility of integrating data This activity is feasible. Data in indexed bibliometric on scientific publications at the databases are available but subject to licence terms and level of individual universities condition and commercial fees. Strong affiliation disambiguation and standardization activities are

with a breakdown by discipline necessary. Concordance with ETER ID is necessary to ensure interoperability with data on other dimensions. Breakdown by discipline feasible developing a proper methodology with a view to interoperability with other sectoral classification systems (no standard available) 3 Feasibility of locating This activity is feasible. Two possible alternatives: publications of universities on a bottom-up exploiting individual publication affiliation geographic map of Europe, at address; top-down matching the university with NUTS 2 and NUTS 3 level Authority file geographic information. At present stage the second option is easier and less costly, but less precise. It could be improved working on the quality of the Authority file. 3.1 Discussion of coverage and Pilot test on one large RPO. Necessity to improve the adequacy of RPOs inventory list inventory list and integrate information on organisation structure in a semantic way. Large standardization and disambiguation effort is required. Regionalisation of RPOs research output data is feasible according to the Sapienza pilot test carried out (Lenzerini, 2015b). 3.2 Proof of the concept by building A proof-of concept prototypal application based on up a sample of regions and/or visual analytics principles and tools has been developed universities and locating them (EUREKA). Multidimensional data from several sources on a GIS computer platform (ETER, Scimago, GRBS, Eurostat) have been integrated for the sample at European wide scale. The tool is extensible and scalable and ensures interactivity and very good usability. 4 Integration of other socio- The integration is feasible and relatively easy. economic indicators the GIS in Integration through ETER_ID for individual institution’s order to build up a Map of data and through NUTS code or city name for regional Excellence statistics. Extensions, scalability, interactivity and usability ensured with EUREKA tool and magnified in the framework of an ontology based model as proposed in this study. 5 Discussion about the state of Crucial point for interoperability and integration of the art in the literature multi-dimensional data. Sectoral information cannot be regarding the correspondence ignored for proper performance evaluation and between different comparisons. Several types of concordance Tables have classifications in S&T and been analysed (IPC-Industry; IPC-Scientific Fields; industrial statistics Scientific Fields-Fields of Education). Further effort is required, however at the top level (5 broad classes) our working hypothesis is that the creation of concordance tables between the various classification systems is feasible. Table 12: Summary table on the feasibility assessment

11. EXPLORATION OF POSSIBLE BUSINESS MODELS AND BUDGET

The definition of “the” business model for a sustainable infrastructure to support the realization of the European Map of Excellence and Specialization (EMES) is a strategic choice that should be taken by the Commission at the end of the large scale project. In this paragraph we explore three alternative options which range from a model almost fully supported by the European Commission to a model completely open to the community of users. For a comprehensive list of possible users, see Appendix 3.

1. Model Supported by the European Commission

This option foresees a direct involvement of the European Commission, through, for instance, the endorsement by the European Parliament following its actions on pan-European research performance information systems (ESTA).

In this framework, the creation of a scientific board should be provided to coordinate the activities of the EMES and its maintenance.

About the management of the system, a decision has to be taken if to proceed with an in house management (for instance through the Joint Research Center, JRC) or contracting out the service through the issue of a Tender for contract service providing all the technical specifications of the underlying infrastructural data.

2. Public-private sponsorship Model

This option is based on a mixed public-private sponsorship, based on a collection of private foundations and public funding agencies.

In this framework, the creation of a scientific board should be provided to coordinate the activities of the EMES and its maintenance as well.

About the management of the system, it could be carried out by a non-profit organization (see for instance the management of ORCID) on the base of the technical specifications of the underlying infrastructural data. These technical specifications should be provided as an output by the full scale project (see more details below about Budget).

An example: the European Science Foundation (ESF) signed in January 2014 a memorandum of understanding with the Norwegian Social Science Data Services (NSD) to transfer the maintenance and operations of the European Reference Index for the Humanities (ERIH) to NSD. The ERIH database operated by NSD is called ERIH PLUS.

In the public-private sponsorship model, a crucial task is to define and properly organize the part of the activities that must be publicly funded, and the part funded from the private domain. Activities related to the conceptualization of the information system, the development of its scientific-scholarly basis, the testing of validity of the methodologies, and the practical usefulness of the system, should be conducted independently on the basis of scientific-scholarly quality criteria. Such activities are in our view best conducted in an entity (possibly a network of academic groups) which is mainly publicly funded. The technical realization of the system, its maintenance, its introduction to the market as a commercially sustainable product, is to be primarily privately funded. Both parties sign a collaboration agreement specifying the various roles and their responsabilities.

3. Science 2.0 Model

This model is based on a completely decentralized-individual based sponsorship, based on a collection of donations from individuals, but also private foundations and public funding agencies.

A legal framework should be established with respect to the treatment of individual data.

A mechanism of self correction of metadata should be put in place.

This model relies on the mobilisation of researchers.

The funding should be based on crowd-funding.

Some examples of this model include: Research Gate and Wikipedia.

It should be foreseen, as an explicit output of the full scale exercise the examination and comparison of the advantages and disadvantages of each option.

Linking data in an open platform

Daraio and Bonaccorsi (2015) call for a shift of the existing paradigm:

 from proprietary sources to open linked data sources;  from ad hoc indicators to indicators that can be generated reliably by the combination of existing data in a platform;  from indicators mainly based on research to a comprehensive set of indicators including several outputs and taking into account the structure of input.

They propose the generation of new indicators from an open platform as a reference.5

The development of new technologies in database design, data analysis, data quality, data integration, data visualization and related fields- all areas in computer science that have seen impressive achievements in recent years- allow a conceptual jump to support the shift of the existing paradigm as discussed above. What follows is an attempt to build new indicators in higher education and research that is consistent with the literature on Ontology-based data management (Lenzerini, 2011; Calvanese et al. 2010; Poggi et al. 2008). We advocate the exploration and exploitation of this approach in the field of design of indicators, as well as in the production of evidence-based support to complex policy making.

The first step is conceptualizing research and innovation systems as complex systems formed by more elementary entities. There are several possible ways of doing this, as witnessed in the literature. One distinction is between an actor-centered perspective, which defines elementary action units in the system, and a function-based perspective, which on the contrary works more on structural relations and flows within the system. We will try to reconcile these two perspectives. A starting point for conceptualizing this issue is to think in terms of actors, e.g. universities, non- university higher education institutions, PROs, firms, governments, funding agencies, government agencies, Non Governmental Organisations (NGOs) and several others. The large literature on systems of innovation and the systematic mapping of European landscapes carried out in recent years (Lundvall, 2010) allow us to build up almost complete maps of actors.

The second step is to develop concepts associated to entities. The development of concepts may be helped by the users requirements discussed above. Rather than following strict definitions, actors should be conceptualized in terms of a list of associated concepts. For example a university is associated to concepts such as students, professors, disciplines, location, publications etc. Each of these concepts is, in turn, susceptible of further conceptualization. For example students are associated to concepts such as nationality, country of the last degree, gender, field of education, etc. For each concept we should carry out an extensive abstract analysis of the underlying definitions and of the relations with the other concepts. It is not important that definitions are formally similar, since higher level concepts will be generated by variable degrees of membership of lower levels concepts. This is a crucial point. While in the integration of databases what is needed is a perfect matching between the concept and the definition, in order to carry out a query, in this approach this issue does not prevent integration. The similarity of definitions can formally be defined on the basis of the articulation of concepts. If two definitions are equal, their concepts overlap perfectly. If two definitions are different (for example, a “foreign student” might be considered a student of foreign nationality, or a student who has attended the secondary school in a different country), then the associated concepts will reveal the difference. Given this approach, it is always possible to identify the subset of concepts that make two definitions similar. This abstract definition gives flexibility to the information system.

The third step is to look for lists and standards, or more generally in the database language, Authority Files. The existence of a list is enormously important, because it allows exhaustive search. Lists are flexible tools, in the sense that are associated to procedures for update and correction. For example, having an Authority File of companies investing into R&D, or a list of Higher Education Institutions (HEIs), or of Public Research Organisations (PROs) is an important step. As an example, having built the official list of HEIs in Eumida, associated with basic indicators, has permitted the detailed geo-referentiation and the attribution of further information to the same entity.

Standards are also extremely important. For example, one would take on board standard definitions adopted at international level as to what constitute an author of a scientific publication (ORCID, see www.orcid.org), or a research organisation (CERIF, see www.eurocris.org).

The fourth step is to establish links between concepts.

Automation and maintenance of the infrastructural data system

In Section 2 of this report we have briefly described the main ideas of an Ontology-Based Data Management (OBDM) Approach.

In this section we briefly present the main advantages of an OBDM approach compared to conventional data-base integration approaches.6

5 This section is taken from Daraio and Bonaccorsi (2015).

While the amount of data stored in current information systems and the processes making use of such data continuously grow, turning these data into information, and governing both data and processes are still tremendously challenging tasks for Information Technology. The problem is complicated due to the proliferation of data sources and services both within a single organization, and in cooperating environments. The following factors explain why such a proliferation constitutes a major problem with respect to the goal of carrying out effective data governance tasks:

- Although the initial design of a collection of data sources and services might be adequate, corrective maintenance actions tend to re-shape them into a form that often diverges from the original conceptual structure.

- It is common practice to change a data source (e.g., a database) so as to adapt it both to specific application-dependent needs, and to new requirements. The result is that data sources often become data structures coupled to a specific application (or, a class of applications), rather than application-independent databases.

- The data stored in different sources and the processes operating over them tend to be redundant, and mutually inconsistent, mainly because of the lack of central, coherent and unified coordination of data management tasks.

The result is that information systems of medium and large organizations are typically structured according to a “sylos”-based architecture, constituted by several, independent, and distributed data sources, each one serving a specific application. This poses great difficulties with respect to the goal of accessing data in a unified and coherent way. Analogously, processes relevant to the organizations are often hidden in software applications, and a formal, up-to-date description of what they do on the data and how they are related with other processes is often missing. The introduction of service-oriented architectures is not a solution to this problem per se, because the fact that data and processes are packed into services is not sufficient for making the meaning of data and processes explicit. Indeed, services become other artifacts to document and maintain, adding complexity to the governance problem. Analogously, data warehousing techniques and the separation they advocate between the management of data for the operation level, and data for the decision level, do not provide solutions to this challenge. On the contrary, they also add complexity to the system, by replicating data in different layers of the system, and introducing synchronization processes across layers. All the above observations show that a unified access to data and an effective governance of processes and services are extremely difficult goals to achieve in modern information systems. Yet, both are crucial objectives for getting useful information out of the information system, as well as for taking decisions based on them. This explains why organizations spend a great deal of time and money for the understanding, the governance, the curation, and the integration of data stored in different sources, and of the processes/services that operate on them, and why this problem is often cited as a key and costly Information Technology challenge faced by medium and large organizations today (Bernstein & Haas, 2008). We argue that ontology-based data management (OBDM, Lenzerini, 2011) is a promising direction for addressing the above challenges. The key idea of OBDM is to resort to a three-level architecture, constituted by the ontology, the sources, and the mapping between the two. The ontology is a conceptual, formal description of the domain of interest to a given organization (or, a community of users), expressed in terms of relevant concepts, attributes of concepts, relationships between concepts, and logical assertions characterizing the domain knowledge. The data sources are the repositories accessible by the organization where data concerning the domain are stored. In the general case, such repositories are numerous, heterogeneous, each one managed and maintained independently from the others. The mapping is a precise specification of the correspondence between the data contained in the data sources and the elements of the ontology. The main purpose of an OBDM system is to allow information consumers to query the data using the elements in the ontology as predicates. In this sense, OBDM can be seen as a form of information integration, where the usual global schema is replaced by the conceptual model of the application domain, formulated as an ontology expressed in a logic-based language. With this approach, the integrated view that the system provides to information consumers is not merely a data structure accommodating the various data at the sources, but a semantically rich description of the relevant concepts in the domain of interest, as well as the relationships between such concepts. The distinction between the ontology and the data sources reflects the separation between the conceptual level, the one presented to the client, and the logical/physical level of the information system, the one stored in the sources, with the mapping acting as the reconciling structure between the two levels. This separation brings several potential advantages:

6 Most of this section is taken from Daraio, Lenzerini, et al. (2015) to which the reader is referred to for the references cited therein.

 The ontology layer in the architecture is the obvious mean for pursuing a declarative approach to information integration, and, more generally, to data governance. By making the representation of the domain explicit, we gain re-usability of the acquired knowledge, which is not achieved when the global schema is simply a unified description of the underlying data sources.  The mapping layer explicitly specifies the relationships between the domain concepts on the one hand and the data sources on the other hand. Such a mapping is not only used for the operation of the information system, but also for documentation purposes. The importance of this aspect clearly emerges when looking at large organisations where the information about data is widespread into separate pieces of documentation that are often difficult to access and rarely conforming to common standards. The ontology and the corresponding mappings to the data sources provide a common ground for the documentation of all the data in the organisation, with obvious advantages for the governance and the management of the information system.  A third advantage has to do with the extensibility of the system. One criticism that is often raised to data integration is that it requires merging and integrating the source data in advance, and this merging process can be very costly. However, the ontology-based approach we advocate does not impose to fully integrate the data sources at once. Rather, after building even a rough skeleton of the domain model, one can incrementally add new data sources or new elements therein, when they become available, or when needed, thus amortising the cost of integration. Therefore, the overall design can be regarded as the incremental process of understanding and representing the domain, the available data sources, and the relationships between them. The goal is to support the evolution of both the ontology and the mappings in such a way that the system continues to operate while evolving, along the lines of "pay-as-you- go" data integration pursued in the research on data-spaces (Sarma et al., 2008).

A real options approach to estimate the investment in an OBDM approach

Following a real options approach in investment theory (Li & Johnson, 2002), we can conceive systems of indicators as investment activity of government and public administration. The data platforms are then considered as assets allowing repeated use. In this context, investment costs are made by front-up costs for the platform, maintenance costs and recurring costs for projects. The revenues instead are the gains from better decisions in policy making; the possible use for performance-based allocation of public resources; the possible use for strategic priorities in S&T; or to set up public subsidies to firms for industrial R&D.

The traditional investment appraisal approach based on the comparison of front-up costs at the initial time period with the discounted cash flow from revenues at subsequent periods of time could be overcome by adopting a real options approach (Li and Johnson, 2002) which includes in the appraisal of investment the options which have been made possible by the investment. Interesting within this framework are learning options and transfer options.

A real options analysis in this context should follow a modular engineering design perspective (Baldwin and Clark, 2000).

The concept of modularity in design was introduced in business economics by Baldwin and Clark (2000). They propose a quantitative model to describe the economic forces that push a design towards modularization and the consequences of modularity on the business environment. Value creation is the goal of the modularization process and real options theory offers a natural framework to evaluate a modular design.

Baldwin and Clark (2000) pointed out six operators describing the structure of a modular system, or alternatively its evolution from a non–modular (or interconnected) design to a modular design: splitting, substitution, augmenting, excluding, inversion, and porting. These operators can be thought of as options in the designer’s palette and Baldwin and Clark propose to link the six operators to real options theory.

Baldwin and Clark (2000) describe three types of modularity: modularity in design, modularity in production and modularity in use. The first refers to the creation of a modular system, the second is related to the simplification of the production process (i.e. dividing complex production tasks into smaller processes); the third concerns the possibility for the consumer to arrange elements in order to obtain a design configuration that reflects his needs. In this work we focus on the modularization of a design and on the related modularization of the production process.

Modularity is a property of quasi-decomposition of hierarchical systems, based on the minimization of the interdependence of sub-systems (see Simon, 1962).

The modification of sub-systems does not require the re-design of the entire system. Making the design of products modular requires a large front up investment in conceptual design.

The standardization of interfaces is necessary.

However, the design of successive versions of the product and/or re-design becomes cheaper.

An OBDM approach for the design and implementation of S&T indicators (Daraio et al., 2015) requires a large scale investment into the formal definition of the main relevant concepts (and relationships among them) of the domain of interest.

The baseline ontology must be subject to a large scale community-based debugging and testing.

The formal ontology must be tested against the real-world availability of data at European level.

A business model for the provision of bibliometric data should be discussed and agreed between the Commission (assisted by experts in the field) and the main commercial providers of data (Thomson Reuters and Scopus), but also some experiments should be done with alternative data sources, like Google Scholar.

Disambiguation of affiliations will require some work with test and development of a semi- automatic algorithmic approach, combined with final manual refinements and checks.

An estimate of the needed budget

On the base of the elaborations carried out within this study, we provide an estimate of the needed resources to carry out a full scale study to realize the European Map of Excellence and Specialization (EMES).

The full scale exercise should be realized in at least two years, in the form of a research project, with an overall budget of at least 1.8 millions of Euro (see also Table 13). The final output of the project should be the development of a working prototype of the EMES, with the detailed technical specifications for its subsequent implementation.

The alternative scenarios, to compare, are a scenario based on separate databases of S&T indicators, whose integration is built through ad hoc projects, versus a scenario which includes a front-up investment in the development of the ontological description of the underlying domain of interest, then a mapping of the ontology with the heterogeneous data sources; in this way, the subsequent integration of data for the construction of new S&T indicators is much easier and less costly. Moreover, this second scenario allows for the on demand construction of indicators which is not possible instead in the case of the first scenario.

OBDM Approach: Investment view- two Traditional sylos-based approach: years project Different parallel projects based on different databases

Platform building and testing– 1.8-2 mn euro Ad hoc S&T indicators project- 100k euro

Running costs (licence fees)- 200-300k euro Running costs (licence fees)- 50k euro

S&T indicator construction- in the order of 10k euro

Table 13 Comparison of costs in the two alternative scenarios

A general conclusion that could be reached is that at the European level the investment in ontology development would be repayed in the first 1.5/2 years of operations (assuming 10 projects per year).

12. RECOMMENDATIONS

On the base of the study carried out, and taking into account the established criteria for the assessment of the successful feasibility of the EMES, we suggest to the Commission to further proceed with a full scale study for the realization of the European Map of Excellence and Specialization.

 The EMES should be designed by following an Ontology-Based Data Management (OBDM) approach to ensure a sustainable and up-to-date map, interoperable and extendable over time.  The Map should integrate in a GIS-format at least the following groups of indicators:  structural indicators for higher education institutions (HEIs)  structural indicators for Public Research Organisations (PROs)  publications of HEIs and PROs  patents assigned to HEIs and PROs  academic staff at HEIs  undergraduate students  PhD students  undergraduate degrees  PhD degrees.

All these information should be geo-referentiated and supported by extensive metadata.

The data should have a breakdown by discipline (Field of Science, or Subject categories) and by Field of Education.

In addition, data should be integrated with relevant indicators at regional (NUTS2 and, where possible, NUTS3) level. These should include industrial, employment, GDP, social and demographic data.

 The proposed Map should specify the procedures for the updating of data, offering solutions for the automatic update as frequently as possible.  The proposed Map should also demonstrate the sustainability of the organization or business model in the future, by addressing issues such as provision of commercially available data, cost of update, IPRs, standardization and robustness issues.  With respect to higher education institutions, the census established by the ETER project, funded by DG Education and Culture in collaboration with DG Research and EUROSTAT, should be assumed as the official source. Consequently, the ID system proposed by the ETER project should be used as a reference in all documents. The project should provide the list of all affiliation names and possible variations found in publications affiliations.  With respect to Public Reserch Organisations, the project should issue a similar ID system, organized in a hierarchical way as suggested by the ontology model. For instance:  organization name at the country level (e.g. Max Planck Society, CNRS, or CSIC)  first-tier sub-organization name (e.g. institute, or department)  second-tier organization name (e.g. institute within a department, laboratory within an institute)  bottom level organization (e.g. research group, research team, laboratory). The system should aim at maintaining stability at organization name, establishing a permanent list as a standard reference list. This would be an Authority File to be maintained at official level.

The system must specify the procedures by which the lists of first-tier and second-tier names are updated, corrected, and cleaned over time.7 The system must specify under which conditions the list of first-tier names could eventually become an Authority File in its own, becoming an official and stable source. The same should be examined for the second-tier list of names.

 In order to supervise the development of the project, a Steering group should be formed, involving  DG Research  DG Education  DG Region  Eurostat  OECD  European Parliament- Science and Technology Options Assessment (STOA). The Steering Group might meet regularly in order to review the development of the project and to define and refine requirements for the construction of indicators, on the basis of their own needs.

The Commission might also consider whether to invite separately ERC and EIIT representatives, as well as members of national governments.

 In parallel, a Committee representing the PROs should be created. It should include at least one representative for the largest PROs in Europe (i.e. Max Planck, Leibniz, Helmholtz in Germany, CNRS, INSERM, INRA, INRIA, CEA in France, CSIC in Spain, CNR, INFN in Italy) and a number of representatives from other PROs. This Committee should regularly meet to supervise the development of the analysis aimed at the geo-referentiation of scientific publications of PROs. The Committe should validate the allocation of specific research outputs to teams, laboratories or institutes that could be located geographically. Provisions for fractional allocation should be examined and validated. Meanwhile, other issues should be discussed (e.g. collection of data on patents) for future acivities of indicator construction.  In all future calls for research projects of the Commission there should be a mandatory provision for submission to provide full coverage of ORCID numbers for all researchers involved.  In all future documentation the standards established in the project and/or available at international level should be adopted:  ID numbers of HEIs  ID numbers of PROs  CERIF ID numbers of funding agencies  ORCID ID numbers for researchers  In future studies commissioned by the Commission there should be a provision for establishing linkages with the platform produced under the project. In particular, data should be delivered to the Commission in such a way to be integrated in a seamless way into the platform. This requires that the ontology model developed as the base of the data integration process become a standard reference point.  In addition the ontology suggested by the project should be published. An interactive consultation with producers and users of indicators should be opened. After a given period, the ontology should be published in an official version and should become a standard reference point. Future releases should be published in due time.  The feasibility study has shown the enormous potential for reliable, effective and efficient construction of indicators provided by the creation of Authority Files. This is an authoritative source established in the form of a list, associated to complete definitions and rules for inclusion and exclusion, and to explicit rules for updating over time. These processes could be, in principle, executed automatically. Once an Authority File is established, the same information is

7 One should distinguish two approaches in which the Authority File can be used in the affiliation-de-duplication process. In the first the Authority file is used to assign author affiliation strings in scientific publications to organization names. For instance, if one finds articles with the affiliation ‘Dept Astronomy’ in a paper from Leiden while the name “University of Leiden” is missing in those strings, a rich Authority File containing information on first and second tier sub-organizations indicates that University of Leiden actually contains a Department of Astronomy, so that these affilation strings should be assigned to the University of Leiden. But this does not necessarily mean that one can obtain a reliable estimate of the publication output of Dept Astronomy at Univ Leiden by counting only articles containing ‘Dept Astronomy’ in their affiliation data, since there may be many papers from this department that do not contain Dept Astronomy in their affiliation data. The reason is that the affiliation information in scientific publications is often too incomplete or inaccurate to achieve de-duplication at lower levels. The recall of such a process tends to be low, even though the precision may be high. This problem is for PROs even worse than for academic institutions.

propagated in the entire information system without ambiguity. This means that the same piece of information gets more value, since it is appropriately used in many contexts. In the context of this study the following Authority Files should be established:

 official list of Higher Education Institutions  official list of Public Research Organisations  author ID  publication ID  funding agency ID The official list of HEIs is available under ETER: we recommend the adoption as a standard.

The official list of PROs is to be constructed under a dedicated project.

REFERENCES

Altenberg, L. (2005), “Modularity in evolution: some low-level questions”, in Callebaut, W., Rasskin-Gutman, D. (Eds), Modularity: Understanding the Development and Evolution of Natural Complex Systems, The MIT Press, pp. 99-128.

Angelini M., Ferro N., Santucci G., & Silvello G. (2014a), VIRTUE: A visual tool for information retrieval performance evaluation and failure analysis. Journal of Visual Languages & Computing, 25(4), 394-413.

Angelini M., Ferro N., Larsen B., Müller H., Santucci G., Silvello G., & Tsikrika T. (2014b), Measuring and Analyzing the Scholarly Impact of Experimental Evaluation Initiatives. Procedia Computer Science, 38, 133-137.

Bakkalbasi N., Bauer K., Glover J., Wang L. (2006), Three options for citation tracking: Google Scholar, Scopus and Web of Science. Biomedical Digital Libraries, 3, art. no. 7.

Baldwin, C. Y., Clark, K. (2000), Design Rules - The Power of Modularity, MIT Press, Cambridge.

Baldwin, C. Y., Woodard, C. J. (2009), “The architecture of platforms: a unified view”, in Gawer, A. (Ed.), Platforms, Markets and Innovation, Edward Elgar, Cheltenham, pp. 19-44.

Bar-Ilan J. (2008), Which h-index? - A comparison of WoS, Scopus and Google Scholar, Scientometrics, 74 (2), pp. 257-271.

Bonaccorsi A. (2015), The economics of S&T indicators, in Daraio C., eds, Efficiency, Effectiveness and Impact of Research and Innovation, Proceedings of the Workshop of the 20th February 2015, DIAG Sapienza University of Rome, Efesto Edizioni, Rome, ISBN 9788899104306.

Calero-Medina C., López-Illescas C., Visser M.S., Moed H.F. (2008), Important factors in the interpretation of bibliometric rankings of world universities. Research Evaluation 17, 71-81.

Cook K. A., & Thomas J. J. (2005), Illuminating the path: The research and development agenda for visual analytics (No. PNNL-SA-45230). Pacific Northwest National Laboratory (PNNL), Richland, WA (US).

Cronin B. & Sugimoto C. (ed) (2014), Beyond bibliometrics. Harnessing multidimensional indicators of scholarly impact. MIT Press, Cambridge Mass.

Cuxac P., Lamirel J.-C. & Bonvallot V. (2013), Efficient supervised and semi-supervised approaches for affiliations disambiguation. Scientometrics, 97 (1), 47-58.

Daraio C., (2015) (Eds), Efficiency, Effectiveness and Impact of Research and Innovation, Proceedings of the Workshop of the 20th February 2015, DIAG Sapienza University of Rome, Efesto Edizioni, Rome, ISBN 9788899104306.

Daraio C., Bonaccorsi A. (2015), Beyond university rankings? Generating new indicators on universities by linking data in open platforms, under review for JASIST.

Daraio C., Bonaccorsi A., Simar L., (2015a), Rankings and university performance: A conditional multidimensional approach, European Journal of Operational Research, 244, 918-930.

Daraio C., Bonaccorsi A., Simar L. (2015b), Efficiency and economies of scale and specialization in European universities. A directional distance approach, Journal of Informetrics, 9, 430-448.

Daraio C., Di Costa F., Moed H.F. (2014), Towards Concordance Tables of Different Subject Classification Systems. A literature review with policy implications, in Proceedings of the Science and Technology Indicators Conference 2014 “Context Counts: Pathways to Master Big and Little Data”, 3-5 September 2014, edited by E. Noyons, published by Universiteit Leiden CWTS 2014, pp. 132-135, ISBN 978-90-817527-1-8.

Daraio C., Di Costa F., Moed H.F. (2015), Concordance tables of subject classification systems: a systematic review, DIAG Technical Report.

Daraio C., Lenzerini M., Leporelli C., Moed H.F., Naggar P., Bonaccorsi A. & Bartolucci A. (2015). Sapientia: the Ontology of Multi-Dimensional Research Assessment, in Salah, A.A., Y. Tonta, A.A. Akdag Salah, C. Sugimoto, U. Al (Eds.), Proceedings of ISSI 2015 Istanbul: 15th International Society of Scientometrics and Informetrics Conference, Istanbul, Turkey, 29 June to 3 July, 2015, Bogaziçi University Printhouse, pp. 965-977.

Delgado-López-Cózar E., Cabezas-Clavijo Á. (2013), Ranking journals: Could Google Scholar Metrics be an alternative to journal citation reports and Scimago journal rank? Learned Publishing, 26 (2), pp. 101-114.

ETER (n.d.), http://eter.joanneum.at/imdas-eter/.

EUMIDA (2010) Final Study Report, Feasibility Study for Creating a European University Data Collection [Contract No. RTD/C/C4/2009/0233402], European Commission.

European Commission (2010). Assessing Europe’s university based research. Expert Group on Assessment of University-Based Research (AUBR). Retrieved July 30, 2014, from http://ec.europa.eu/research/science-society/document_library/pdf_06/ assessing- europe-university-based-research_en.pdf

Falagas M.E., Pitsouni E.I., Malietzis G.A., Pappas G. (2008), Comparison of PubMed, Scopus, Web of Science, and Google Scholar: Strengths and weaknesses, FASEB Journal, 22 (2), pp. 338-342.

Fellegi I. & Sunter A. (1969), A theory for record linkage. Journal of the American Statistical Association, 64(328), 1183–1210.

Franceschet M. (2010), A comparison of bibliometric indicators for computer science scholars and journals on Web of Science and Google Scholar. Scientometrics, 83 (1), pp. 243-258.

Jacso P. (2005), As we may search - Comparison of major features of the Web of Science, Scopus, and Google Scholar citation-based and citation-enhanced databases. Current Science, 89 (9), pp. 1537-1547.

Keim D., Kohlhammer J., Ellis G., & Mansmann F. (2010), Mastering the information age: Solving problems with visual analytics. VisMaster.

Lambert L. and Butler N. (2006), The Future of European Universities: Renaissance or Decay? London (UK): Centre for European Reform (CER), ISBN 1 901 229 67 X.

Lenzerini M. (2015a), Sapientia: the Ontology of Multidimentional Research Assessment, in "Efficiency, Effectiveness and Impact of Research and Innovation", Proceedings of the Workshop of the 20th February 2015, DIAG Sapienza University of Rome, edited by C.Daraio, Efesto Edizioni, Rome, ISBN 9788899104306, pagg. 26-53.

Lenzerini M. (2015b), Corroboration of the algorithm and the technical specifications for the European Map of Excellence and Specialization (EMES), European Commission, Brussels.

Lenzerini M. (2011), Ontology-based data management, CIKM 2011: 5-6.

Lepori, B., Daraio, C., Bonaccorsi, A., Daraio, A., Scannapieco, M., Gunnes, H., Hovdhaugen, E., Ploder, M. and D. Wagner-Schuster (2014), ‘ETER Project, Handbook for Data Collection’, Brussels, June.

Li, X., Johnson, J. D. (2002). Evaluate IT investment opportunities using real options theory. Information Resources Management Journal, 15(3), 32-47.

López-Illescas. C., de Moya-Anegón F., Moed H.F. (2011), A ranking of universities should account for differences in their disciplinary specialization. Scientometrics 88, 563-574.

Luwel, M. (2004). The use of input data in the performance analysis of R&D systems. In Handbook of Quantitative Science and Technology Research (pp. 315-338). Kluwer, the Netherlands.

Meho L.I., Yang K. (2007), Impact of data sources on citation counts and rankings of LIS faculty: Web of science versus scopus and google scholar. Journal of the American Society for Information Science and Technology, 58 (13), pp. 2105-2125.

Moed H. F. (2015), Metrics-Based Research Assessment as a Big Data Science, Lecture Notes, PhD in Engineering in Computer Science, DIAG, Sapienza University of Rome.

Moed H.F. (2006), Bibliometric Rankings of World Universities. Leiden (The Netherlands): Centre fr Science and Technology Studies, CWTS Report 2006-01.

Odijk D., Santucci G., de Rijke M., Angelini M., & Granato G. L. (2012), Time-aware exploratory search: Exploring word meaning through time. In SIGIR 2012 Workshop on Time-aware Information Access, Portland, OR, USA.

OECD Innovation Policy Platform (2011). Actor Brief on Public Research Organizations (PRO). http://www.oecd.org/innovation/policyplatform/48136051.pdf.

Poggi A., D. Lembo, D. Calvanese, G. De Giacomo, M. Lenzerini, & R. Rosati (2008), Linking data to ontologies. J. on Data Semantics, X:133–173.

Rahm E., & Do H. H. (2000), Data cleaning: Problems and current approaches. IEEE Data Engineering Bulletin, 23(4), 3–13.

Santucci G. (2015), Implications of an OBDM Approach for Data visualization, in "Efficiency, Effectiveness and Impact of Research and Innovation", Proceedings of the Workshop of the 20th February 2015, DIAG Sapienza University of Rome, edited by C.Daraio, Efesto Edizioni, Rome, ISBN 9788899104306, pagg. 62-78.

UNESCO, International Standard Classification of Education: Fields of Education and Training 2013, draft, May 2013.

Van Raan, A. F. J. (2005). Fatal attraction: Conceptual and methodological problems in the ranking of universities by bibliometric methods. Scientometrics, 62(1), 133–143.

Waltman L., & van Eck N.J. (2012), A new methodology for constructing a publication-level classification system of science, Journal of the American Society for Information Science and Technology, Volume 63 Issue 12, December 2012 Pages 2378-2392.

APPENDICES

Appendix 1: Authority file of European universities

NUTS NUTS Name of the ID Institution Name English Institution Name Count. 2 3 city AT0001 Universität Wien University of Vienna AT AT13 AT130 Vienna AT0002 Universität Graz University of Graz AT AT22 AT221 Graz AT0003 Universität Innsbruck University of Innsbruck AT AT33 AT332 Innsbruck AT0004 Universität Salzburg University of Salzburg AT AT32 AT323 Salzburg Vienna University of AT0005 Technische Universität Wien AT AT13 AT130 Vienna Technology Graz University of AT0006 Technische Universität Graz AT AT22 AT221 Graz Technology University of Mining AT0007 Montanuniversität Leoben AT AT22 AT223 Leoben Leoben University of Natural Universität für Bodenkultur AT0008 Resources and Applied AT AT13 AT130 Vienna Wien Life Sciences, Vienna Veterinärmedizinische University of Veterinary AT0009 AT AT13 AT130 Vienna Universität Wien Medicine Vienna Vienna University of AT0010 Wirtschaftsuniversität Wien Economics and Business AT AT13 AT130 Vienna Administration AT0011 Universität Linz University of Linz AT AT31 AT312 Linz AT0012 Universität Klagenfurt University of Klagenfurt AT AT21 AT211 Klagenfurt Medizinische Universität Vienna University of AT0014 AT AT13 AT130 Vienna Wien Medicine Medizinische Universität Graz University of AT0015 AT AT22 AT221 Graz Graz Medicine Medizinische Universität Innsbruck University of AT0016 AT AT33 AT332 Innsbruck Innsbruck Medicine Akademie der bildenden Academy of Fine Arts AT0017 AT AT13 AT130 Vienna Künste Wien Vienna Universität für angewandte University of Applied Arts AT0018 AT AT13 AT130 Vienna Kunst Wien Vienna Universität für Musik und University of Music and AT0019 AT AT13 AT130 Vienna darstellende Kunst Wien Performing Arts in Vienna University of Music and Universität Mozarteum AT0020 Dramatic Arts Mozarteum AT AT32 AT323 Salzburg Salzburg Salzburg Universität für Musik und University of Music and AT0021 AT AT22 AT221 Graz darstellende Kunst Graz Dramatic Arts Graz

Universität für künstlerische University of Art and AT0022 und industrielle Gestaltung AT AT31 AT312 Linz Industrial Design Linz Linz

Katholische Theologische Catholic Theological AT0023 AT AT31 AT312 Linz Privatuniversität Linz Private University Linz Private Universität für Private University for Gesundheitswissenschaften, AT0025 Health Informatics and AT AT33 AT332 Hall in Tirol Medizinische Informatik und Technology Tyrol Technik (UMIT) Paracelsus Medizinische AT0026 Paracelsus University AT AT32 AT323 Salzburg Privatuniversität Salzburg Sigmund Freud Sigmund Freud Private AT0032 AT AT13 AT130 Vienna Privatuniversität Wien University Vienna BE0005 Université de Namur Université de Namur BE BE35 BE351 Namur Facultés Universitaires Facultés Universitaires BE0006 BE BE10 BE100 Brussels Saint-Louis Saint-Louis Université Catholique de Université Catholique de Louvain-La- BE0007 BE BE31 BE310 Louvain (UCL) Louvain (UCL) Neuve BE0008 Université de Liège (ULG) Université de Liège (ULG) BE BE33 BE332 Liège Université de Mons Université de Mons BE0009 BE BE32 BE323 Mons (UMmons) (UMmons) Université Libre de Université Libre de BE0010 BE BE10 BE100 Brussels Bruxelles (ULB) Bruxelles (ULB)

Hogeschool - Universiteit BE0055 HUB-KUBrussel BE BE10 BE100 Brussels Brussel Katholieke Universiteit BE0056 KU Leuven BE BE24 BE242 Leuven Leuven Transnationale Universiteit Transnational University BE0057 BE BE22 BE221 Diepenbeek Limburg Limburg BE0058 Universiteit Hasselt Hasselt University BE BE22 BE221 Diepenbeek BE0059 Universiteit Antwerpen University of Antwerp BE BE21 BE211 Antwerpen BE0060 Universiteit Gent Ghent University BE BE23 BE234 Gent The Vrije Universiteit BE0061 Vrije Universiteit Brussel BE BE10 BE100 Elsene Brussel South-West University Югозападен университет BG0001 'Neofit Rilski', BG BG41 BG413 Blagoevgrad 'Неофит Рилски' Blagoevgrad. Университет 'Проф. д-р Burgas Professor Assen BG0004 BG BG34 BG341 Burgas Асен Златаров' Zlatarov University Икономически University of Economics - BG0006 BG BG33 BG331 Varna университет - Варна Varna Технически университет - Varna Technical BG0007 BG BG33 BG331 Varna Варна University Медицински университет Varna medical university BG0008 'проф. д-р Параскев 'prof. dr. Paraskev BG BG33 BG331 Varna Стоянов' Stoyanov' Варненски свободен Varna Free University BG0009 университет 'Черноризец BG BG33 BG331 Varna ‘Chernorizets Hrabar’ Храбър'

Висшето военноморско Nikola Vaptsarov Naval BG0010 BG BG33 BG331 Varna училище 'Н. Й. Вапцаров' Academy’

Великотърновски St. Cyril and St. Veliko BG0011 университет 'Св. св. Кирил Methodius University of BG BG32 BG321 Tarnovo и Методий' Veliko Tarnovo Националeн военен Vasil Levski National Veliko BG0012 университет 'Васил BG BG32 BG321 military university Tarnovo Левски Стопанска академия 'Д. А. D.A.Tsenov Academy of BG0013 BG BG32 BG321 Svishtov Ценов' - Свищов Economics Технически университет - Technical University of BG0014 BG BG32 BG322 Gabrovo Габрово Gabrovo Медицински университет - Medical University - BG0016 BG BG31 BG314 Pleven Плевен Pleven Пловдивски университет Paisii Hilendarski BG0017 BG BG42 BG421 Plovdiv 'Паисий Хилендарски' University of Plovdiv Университетът по University of Food BG0018 BG BG42 BG421 Plovdiv хранителни технологии Technologies – Plovdiv Аграрен университет – Agricultural University – BG0019 BG BG42 BG421 Plvodiv Пловдив Plovdiv Медицински университет - Medical University – BG0020 BG BG42 BG421 Plovdiv Пловдив Plovdiv Академия за музикално, Academy of Music, Dance BG0021 танцово и изобразително BG BG42 BG421 Plovdiv and Fine Arts изкуство Русенски университет Angel Kanchev University BG0025 BG BG32 BG323 Ruse 'Ангел Кънчев' of Ruse Софийски университет Sofia University St. BG0026 BG BG41 BG411 Sofia 'Св. Климент Охридски' Kliment Ohridski Университет за University of National and BG0027 национално и световно BG BG41 BG411 Sofia World Economy стопанство Университет по University of Architecture, BG0028 архитектура, строителство Civil Engineering and BG BG41 BG411 Sofia и геодезия Geodesy Технически Университет - Technical University of BG0029 BG BG41 BG411 Sofia София Sofia University of Chemical Химико Технологичен и BG0030 Technology and BG BG41 BG411 Sofia Металургичен Университет Metallurgy Минно-геоложкият St. Ivan Rilski University BG0031 университет 'Св. Иван BG BG41 BG411 Sofia of mining and geology Рилски'

Лесотехнически BG0032 University of Forestry BG BG41 BG411 Sofia университет Медицински университет - BG0033 Medical university - Sofia BG BG41 BG411 Sofia София Национална спортна National sports academy BG0034 BG BG41 BG411 Sofia академия 'Васил Левски' Vasil Levski Национална Академия за Krastyo Sarafov National Театрално и Филмово BG0035 Academy for Theatre and BG BG41 BG411 Sofia Изкуство 'Кръстьо Film Arts Сарафов' Национална художествена BG0036 National Academy of Art BG BG41 BG411 Sofia академия Национална музикална Pantcho Vladigerov BG0037 академия 'Проф. П. National Academy of BG BG41 BG411 Sofia Владигеров' Music Нов Български BG0038 New Bulgarian University BG BG41 BG411 Sofia Университет Висше строително Higher School of Civil BG0039 училище 'Любен Engineering 'Lyuben BG BG41 BG411 Sofia Каравелов' Karavelov' Висше транспортно Todor Kableshkov Higher BG0040 училище 'Тодор BG BG41 BG411 Sofia School of Transport Каблешков' Academy of Ministry of BG0041 Академия на МВР BG BG41 BG411 Sofia internal affairs Военна академия 'Георги 'G. S. Rakovski' National BG0042 BG BG41 BG411 Sofia Стойков Раковски' Defence Academy Университет по State University of Library библиотекознание и BG0043 Studies and Information BG BG41 BG411 Sofia информационни Technologies технологии

BG0049 Тракийски Университет Trakian University BG BG34 BG344 Stara Zagora

Шуменски университет Konstantin Preslavsky BG0051 'Епископ Константин BG BG33 BG333 Shumen University of Shumen Преславски' Българска академия на Bulgarian academy of BG0054 BG BG41 BG411 Sofia науките sciences CH0001 Universität Basel University of Basle CH CH03 CH031 Basel CH0002 Universität Bern University of Bern CH CH02 CH021 Bern CH0003 Université de Fribourg University of Fribourg CH CH02 CH022 Fribourg CH0004 Université de Genève University of Geneva CH CH01 CH013 Genève CH0005 Université de Lausanne University of Lausanne CH CH01 CH011 Lausanne CH0006 Universität Luzern University of Lucerne CH CH06 CH061 Luzern CH0007 Université de Neuchâtel University of Neuchâtel CH CH02 CH024 Neuchâtel CH0008 Universität Sankt Gallen University of Sankt Gallen CH CH05 CH055 Sankt Gallen CH0009 Universität Zürich University of Zurich CH CH04 CH040 Zürich Università della Svizzera Università della Svizzera CH0010 CH CH07 CH070 Lugano italiana italiana Ecole Polytechnique Fédéral Federal Institute of CH0011 CH CH01 CH011 Lausanne de Lausanne Technology Lausanne

Eidgenössische Technische Federal Institute of CH0012 CH CH04 CH040 Zürich Hoschule Zürich Technology Zurich

CY0001 ΠΑΝΕΠΙΣΤΗΜΙΟ ΚΥΠΡΟΥ University of Cyprus CY CY00 CY000 Lefkosia ΑΝΟΙΚΤΟ ΠΑΝΕΠΙΣΤΗΜΙΟ CY0002 Open University of Cyprus CY CY00 CY000 Latsia ΚΥΠΡΟΥ ΤΕΧΝΟΛΟΓΙΚΟ Cyprus University of CY0003 CY CY00 CY000 Lemesos ΠΑΝΕΠΙΣΤΗΜΙΟ ΚΥΠΡΟΥ Technology ΠΑΝΕΠΙΣΤΗΜΙΟ European Univeristy - CY0004 CY CY00 CY000 Lefkosia ΛΕΥΚΩΣΙΑΣ Cyprus ΠΑΝΕΠΙΣΤΗΜΙΟ Frederick University - CY0005 CY CY00 CY000 Lefkosia FREDERICK Cyprus ΕΥΡΩΠΑΪΚΟ CY0006 University of Nicosia CY CY00 CY000 Lefkosia ΠΑΝΕΠΙΣΤΗΜΙΟ ΚΥΠΡΟΥ CY0039 The Cyprus Institute The Cyprus Institute CY CY00 CY000 Lefkosia Cyprus School of Molecular Cyprus School of Medicine-The Cyprus Molecular Medicine-The CY0041 CY CY00 CY000 Lefkosia Institute of Neurology and Cyprus Institute of Genetics Neurology and Genetics

Akademie múzických umení Academy of Performing CZ0001 CZ CZ01 CZ010 Praha v Praze Arts in Prague Akademie výtvarných Academy of Fine Arts in CZ0002 CZ CZ01 CZ010 Praha umení v Praze Prague Ceská zemedelská Czech University of Life CZ0003 CZ CZ01 CZ010 Praha univerzita v Praze Sciences, Prague Ceské vysoké ucení Czech Technical CZ0004 CZ CZ01 CZ010 Praha technické v Praze University in Prague Janácek Academy of Janáckova akademie CZ0005 Music and Performing Arts CZ CZ06 CZ064 Brno múzických umení v Brne in Brno University of South Jihoceská univerzita v Ceské CZ0006 Bohemia in Ceské CZ CZ03 CZ031 Ceských Budejovicích Budejovice Budejovice CZ0007 Masarykova univerzita Masaryk University CZ CZ06 CZ064 Brno Mendel University of Mendelova zemedelská a CZ0008 Agriculture and Forestry, CZ CZ06 CZ064 Brno lesnická univerzita v Brne Brno Ostravská univerzita v CZ0009 University of Ostrava CZ CZ08 CZ080 Ostrava I Ostrave Silesian University in CZ0010 Slezská univerzita v Opave CZ CZ08 CZ080 Opava Opava Technická univerzita v Technical University in CZ0011 CZ CZ05 CZ051 Liberec Liberci Liberec University of Hradec Hradec CZ0012 Univerzita Hradec Králové CZ CZ05 CZ052 Králové Králové

Univerzita J. E. Purkyne v University of J. E. Purkyne CZ0013 CZ CZ04 CZ042 Ústí n.L. Ústí nad Labem in Ústí nad Labem

Charles University in CZ0014 Univerzita Karlova v Praze CZ CZ01 CZ010 Praha Prague Univerzita Palackého v Palacký University, CZ0015 CZ CZ07 CZ071 Olomouc Olomouci Olomouc CZ0016 Univerzita Pardubice University of Pardubice CZ CZ05 CZ053 Pardubice Univerzita Tomáše Bati ve Tomas Bata University in CZ0017 CZ CZ07 CZ072 Zlín Zlíne Zlín University of Veterinary Veterinární a farmaceutická CZ0018 and Pharmaceutical CZ CZ06 CZ064 Brno univerzita Brno Sciences, Brno Vysoká škola bánská - VŠB – Technical CZ0019 Technická univerzita CZ CZ08 CZ080 Ostrava I University of Ostrava Ostrava Vysoká škola ekonomická v University of Economics, CZ0020 CZ CZ01 CZ010 Praha Praze Prague Vysoká škola chemicko- Institute of Chemical CZ0021 CZ CZ01 CZ010 Praha technologická v Praze Technology, Prague Academy of Arts, Vysoká škola umelecko- CZ0024 Architecture and Design CZ CZ01 CZ010 Praha prumyslová v Praze in Prague Vysoké ucení technické v Brno University of CZ0025 CZ CZ06 CZ064 Brno Brne Technology Západoceská univerzita v University of West CZ0026 CZ CZ03 CZ032 Plzen Plzni Bohemia in Pilsen Vysoká škola financní a Institute of Finance and CZ0030 CZ CZ01 CZ010 Praha správní, o.p.s. - Praha Administration Metropolitní univerzita v Metropolitan University, CZ0039 CZ CZ01 CZ010 Praha Praze, o.p.s. Prague Univerzita Jana Amose Jan Amos Komensky CZ0040 CZ CZ01 CZ010 Praha Komenského Praha, s.r.o. University, Prague Policejní akademie Ceské Police Academy of the CZ0072 CZ CZ01 CZ010 Praha republiky v Praze Czech Republic in Prague CZ0073 Univerzita obrany v Brne University of Defence CZ CZ06 CZ064 Brno Albert-Ludwigs-Universität DE0001 University of Freiburg DE DE13 DE131 Freiburg i. Br. Freiburg Ruprecht-Karls-Universität DE0002 Heidelberg University DE DE12 DE125 Heidelberg Heidelberg DE0003 Universität Hohenheim Universität Hohenheim DE DE11 DE111 Stuttgart Karlsruher Institut für Karlsruhe Institute of DE0004 Technologie (KIT) - Bereich DE DE12 DE122 Karlsruhe Technology Hochschule

DE0005 Universität Konstanz University of Konstanz DE DE13 DE138 Konstanz DE0006 Universität Mannheim University of Mannheim DE DE12 DE126 Mannheim DE0007 Universität Stuttgart University of Stuttgart DE DE11 DE111 Stuttgart Eberhard Karls Universität DE0008 University of Tübingen DE DE14 DE142 Tübingen Tübingen DE0009 Universität Ulm Ulm University DE DE14 DE144 Ulm Zeppelin Universität Zeppelin University Friedrichshafe DE0012 DE DE14 DE147 Friedrichshafen (Priv. H) Friedrichshafen (Priv. H) n Hochschule für jüdische Hochschule für jüdische DE0015 DE DE12 DE125 Heidelberg Studien Heidelberg Studien Heidelberg DE0016 Universität Augsburg Augsburg University DE DE27 DE271 Augsburg Otto-Friedrich-Universität DE0017 University of Bamberg DE DE24 DE241 Bamberg Bamberg DE0018 Universität Bayreuth University of Bayreuth DE DE24 DE242 Bayreuth Katholische Universität Catholic University Eichstätt, DE0019 DE DE21 DE219 Eichstätt-Ingolstadt (KU) Eichstätt-Ingolstadt Ingolstadt Friedrich-Alexander- University of Erlangen- Erlangen, DE0020 Universität Erlangen- DE DE25 DE252 Nürnberg Nürnberg Nürnberg Ludwig-Maximilians- Ludwig-Maximilians- DE0021 DE DE21 DE212 München Universität München Universität München München, Technische Universität Technische Universität DE0022 DE DE21 DE212 Weihenstepha München München n Universität der Bundeswehr Universität der DE0023 DE DE21 DE212 München München Bundeswehr München Hochschule für Politik The Munich School of DE0024 DE DE21 DE212 München München Political Science DE0025 Universität Passau University of Passau DE DE22 DE222 Passau DE0026 Universität Regensburg University of Regensburg DE DE23 DE232 Regensburg Julius-Maximilians- DE0027 University of Würzburg DE DE26 DE263 Würzburg Universität Würzburg DE0028 Freie Universität Berlin Freie Universität Berlin DE DE30 DE300 Berlin Technische Universität Technische Universität DE0029 DE DE30 DE300 Berlin Berlin Berlin Charité - Charité - DE0030 DE DE30 DE300 Berlin Universitätsmedizin Berlin Universitätsmedizin Berlin Humboldt-Universität zu Humboldt-Universität zu DE0031 DE DE30 DE300 Berlin Berlin Berlin Hertie School of Hertie School of DE0034 Governance Berlin (Priv. Governance Berlin (Priv. DE DE30 DE300 Berlin wiss. H) wiss. H) Steinbeis-Hochschule Berlin Steinbeis-Hochschule DE0035 DE DE30 DE300 Berlin (Priv. H) Berlin (Priv. H) Brandenburgische Brandenburg University of DE0037 Technische Universität DE DE40 DE402 Cottbus Technology Cottbus Europa-Universität Viadrina Europa-Universität Frankfurt an DE0038 DE DE40 DE403 Frankfurt (Oder) Viadrina Frankfurt (Oder) der Oder DE0039 Universität Potsdam University of Potsdam DE DE40 DE404 Potsdam DE0040 Universität Bremen University of Bremen DE DE50 DE501 Bremen Jacobs University Bremen DE0041 Jacobs University Bremen DE DE50 DE501 Bremen (Priv. H) DE0042 Universität Hamburg University of Hamburg DE DE60 DE600 Hamburg Technische Universität Hamburg University of DE0043 DE DE60 DE600 Hamburg Hamburg-Harburg Technology Hafencity Universität HafenCity University DE0044 DE DE60 DE600 Hamburg Hamburg Hamburg Helmut Schmidt Helmut-Schmidt-Universität DE0045 University of the Federal DE DE60 DE600 Hamburg Hamburg Armed Forces Hamburg Bucerius Law School Bucerius Law School DE0046 DE DE60 DE600 Hamburg Hamburg (Priv. H) Hamburg (Priv. H) Technische Universität Technische Universität DE0047 DE DE71 DE711 Darmstadt Darmstadt Darmstadt Johann Wolfgang Goethe- Frankfurt am DE0048 Universität Frankfurt am Frankfurt University DE DE71 DE712 Main Main Justus-Liebig-Universität Justus Liebig University DE0049 DE DE72 DE721 Gießen Gießen Giessen

Universitätsklinikum Gießen Universitätsklinikum Gießen, DE0050 DE DE72 DE721 und Marburg Gießen und Marburg Marburg DE0051 Universität Kassel University of Kassel DE DE73 DE731 Kassel Philipps-Universität Philipps-Universität DE0052 DE DE72 DE724 Marburg Marburg Marburg Frankfurt School of Finance Frankfurt School of Frankfurt am DE0053 & Management-HfB (Priv. Finance & Management- DE DE71 DE712 Main H) HfB (Priv. H) EBS Universität für Wiesbaden, EBS Universität für DE0054 Wirtschaft und Recht DE DE71 DE714 Oestrich- Wirtschaft und Recht (Priv.) (Priv.) Winkel Ernst-Moritz-Arndt- Ernst Moritz Arndt DE0055 DE DE80 DE801 Greifswald Universität Greifswald University of Greifswald DE0056 Universität Rostock University of Rostock DE DE80 DE803 Rostock Technische Universität Technische Universität DE0058 Carolo-Wilhelmina zu Carolo-Wilhelmina zu DE DE91 DE911 Braunschweig Braunschweig Braunschweig Technische Universität Clausthal University of DE0059 DE DE91 DE916 Clausthal Clausthal Technology Georg-August-Universität Georg August Göttingen DE0060 DE DE91 DE915 Göttingen Göttingen University Leibniz Universität Leibniz Universität DE0061 DE DE92 DE929 Hannover Hannover Hannover Medizinische Hochschule Hannover Medical School DE0062 DE DE92 DE929 Hannover Hannover (MHH) Tierärztliche Hochschule Tierärztliche Hochschule DE0063 DE DE92 DE929 Hannover Hannover Hannover Stiftung Universität DE0064 University of Hildesheim DE DE92 DE925 Hildesheim Hildesheim Leuphana Universität Leuphana University of DE0065 DE DE93 DE935 Lüneburg Lüneburg Lüneburg Carl von Ossietzky Carl von Ossietzky the DE0066 DE DE94 DE943 Oldenburg Universität Oldenburg University of Oldenburg DE0067 Universität Osnabrück Osnabrück University DE DE94 DE944 Osnabrück DE0068 Universität Vechta University of Vechta DE DE94 DE94F Vechta DE0069 RWTH Aachen Aachen University DE DEA2 DEA2D Aachen DE0070 Universität Bielefeld Bielefeld University DE DEA4 DEA41 Bielefeld DE0071 Ruhr-Universität Bochum Ruhr-Universität Bochum DE DEA5 DEA51 Bochum

Rheinische Friedrich- DE0072 University of Bonn DE DEA2 DEA22 Bonn Wilchelms-Universität Bonn

Technischen Universität DE0073 TU Dortmund University DE DEA5 DEA52 Dortmund Dortmund Heinrich-Heine-Universität Heinrich Heine University DE0074 DE DEA1 DEA11 Düsseldorf Düsseldorf Düsseldorf University of Duisburg- DE0075 Universität Duisburg-Essen DE DEA1 DEA13 Essen Essen DE0076 Fernuniversität in Hagen University of Hagen DE DEA5 DEA53 Hagen DE0077 Universität zu Köln University of Cologne DE DEA2 DEA23 Köln Deutsche Sporthochschule German Sport University DE0078 DE DEA2 DEA23 Köln Köln Cologne Westfälische Wilhelms- DE0079 University of Münster DE DEA3 DEA33 Münster Universität Münster DE0080 Universität Paderborn University of Paderborn DE DEA4 DEA47 Paderborn DE0081 Universität Siegen University of Siegen DE DEA5 DEA5A Siegen Bergische Universität DE0082 University of Wuppertal DE DEA1 DEA1A Wuppertal Wuppertal Universität University Witten- DE0083 DE DEA5 DEA56 Witten/Herdecke Witten/Herdecke Herdecke Deutsche Hochschule der DE0084 German Police University DE DEA3 DEA33 Münster Polizei, Münster Technische Universität University of Kaiserslauter DE0085 DE DEB3 DEB32 Kaiserslautern Kaiserslautern n Koblenz, University of Koblenz- DE0086 Universität Koblenz-Landau DE DEB1 DEB11 Landau, Landau Mainz Johannes Gutenberg- Johannes Gutenberg DE0087 DE DEB3 DEB35 Mainz Universität Mainz University Mainz

Universitätsmedizin der Universitätsmedizin der DE0088 Johannes Gutenberg- Johannes Gutenberg- DE DEB3 DEB35 Mainz Universität Mainz Universität Mainz Deutsche Universität für German University of DE0089 Verwaltungswissenschaften Administrative Sciences DE DEB3 DEB38 Speyer Speyer Speyer DE0090 Universität Trier University of Trier DE DEB2 DEB21 Trier WHU – Otto Beisheim WHU – Otto Beisheim DE0091 DE DEB1 DEB17 Vallendar School of Management School of Management

Saarbrücken, Universität des Saarlandes DE0092 Saarland University DE DEC0 DEC01 Homburg/Saa Saarbrücken r

Technische Universität Chemnitz University of DE0093 DE DED4 DED41 Chemnitz Chemnitz Technology Technische Universität Technische Universität DE0094 DE DED2 DED21 Dresden Dresden Dresden Technische Universität Technische Universität DE0095 DE DED2 DED21 Dresden Dresden (Med. Fakultät) Dresden (Med. Fakultät) TU Bergakademie Technische Universität DE0096 Freiberg - University of DE DED4 DED43 Freiberg Bergakademie Freiberg Resources DE0097 Universität Leipzig Universität Leipzig DE DED5 DED51 Leipzig Universität Leipzig (Med. Universität Leipzig (Med. DE0098 DE DED5 DED51 Leipzig Fakultät) Fakultät) Internationales International Graduate DE0099 DE DED2 DED2D Zittau Hochschulinstitut Zittau School (IHI) Zittau Handelshochschule Leipzig Leipzig Graduate School DE0101 DE DED5 DED51 Leipzig (Priv. H) of Management Martin-Luther-Universität Halle, DE0102 Martin Luther University DE DEE0 DEE02 Halle-Wittenberg Merseburg Otto-von-Guericke- Otto von Guericke DE0103 DE DEE0 DEE03 Magdeburg Universität Magdeburg University of Magdeburg DE0104 Universität Flensburg Universität Flensburg DE DEF0 DEF01 Flensburg DE0105 Universität Kiel Kiel University DE DEF0 DEF02 Kiel DE0106 Universität Lübeck Universität Lübeck DE DEF0 DEF03 Lübeck Universitätsklinikum Universitätsklinikum DE0107 DE DEF0 DEF02 Kiel, Lübeck Schleswig-Holstein Schleswig-Holstein DE0108 Universität Erfurt Universität Erfurt DE DEG0 DEG01 Erfurt Technische Universität Ilmenau University of DE0109 DE DEG0 DEG0F Illmenau Ilmenau Technology Friedrich-Schiller- Friedrich-Schiller- DE0110 DE DEG0 DEG03 Jena Universität Jena University of Jena Bauhaus-Universität Bauhaus-Universität DE0111 DE DEG0 DEG05 Weimar Weimar Weimar Pädagogische Hochschule University of Education DE0112 DE DE13 DE131 Freiburg i. Br. Freiburg i.Br. Freiburg Pädagogische Hochschule Heidelberg University of DE0113 DE DE12 DE125 Heidelberg Heidelberg Education Pädagogische Hochschule University of Education DE0114 DE DE12 DE122 Karlsruhe Karlsruhe Karlsruhe Pädagogische Hochschule Ludwigsburg University of Ludwigsburg, DE0115 DE DE11 DE115 Ludwigsburg Education Reutlingen Pädagogische Hochschule University of Education Schwäbisch DE0116 DE DE11 DE11D Schwäbisch Gmünd Schwäbisch Gmünd Gmünd Pädagogische Hochschule University of Education DE0117 DE DE14 DE148 Weingarten Weingarten Weingarten Philosophisch-Theologische Philosophisch- Benediktbeue DE0118 Hochschule Benediktbeuern Theologische Hochschule DE DE21 DE216 rn (rk) Benediktbeuern (rk) Hochschule für Philosophie Munich School of DE0119 DE DE21 DE212 München München (rk) Philosophy Philosophisch-Theologische Philosophisch- Frankfurt am DE0121 Hochschule Frankfurt a.M. Theologische Hochschule DE DE71 DE712 Main (rk) Frankfurt a.M. (rk) Theologische Fakultät Fulda DE0122 Fulda Theology Faculty DE DE73 DE732 Fulda (rk) Theologische Fakultät Faculty of Theology in DE0127 DE DEA4 DEA47 Paderborn Paderborn (rk) Paderborn

Philosophisch-Theologische Philosophisch- DE0128 Hochschule St. Augustin Theologische Hochschule DE DEA2 DEA2C St. Augustin (rk) St. Augustin (rk) Theologische Fakultät DE0130 Theologische Fakultät Trier DE DEB2 DEB21 Trier Trier Theologische Hochschule Theologische Hochschule DE0131 DE DEB1 DEB17 Vallendar Vallendar Vallendar Staatliche Hochschule für University of Music DE0133 DE DE13 DE131 Freiburg i. Br. Musik Freiburg i.Br. Freiburg Staatliche Hochschule für Staatliche Hochschule für DE0135 DE DE12 DE122 Karlsruhe Gestaltung Karlsruhe Gestaltung Karlsruhe Staatliche Hochschule für University of Music DE0136 DE DE12 DE122 Karlsruhe Musik Karlsruhe Karlsruhe Staatliche Hochschule für Mannheim University of DE0137 Musik und Darstellende DE DE12 DE126 Mannheim Music and Performing Arts Kunst Mannheim

Staatliche Akademie der Stuttgart State Academy DE0138 DE DE11 DE111 Stuttgart Bildenden Künste Stuttgart of Art and Design

Staatliche Hochschule für Staatliche Hochschule für DE0139 Musik und Darstellende Musik und Darstellende DE DE11 DE111 Stuttgart Kunst Stuttgart Kunst Stuttgart Staatliche Hochschule für Trossingen School of DE0140 DE DE13 DE137 Trossingen Musik Trossingen Music Hochschule für Fernsehen University of Television DE0142 DE DE21 DE212 München und Film München and Film Munich Hochschule für Musik und Hochschule für Musik und DE0143 DE DE21 DE212 München Theater München Theater München Hochschule für Musik University of Music DE0146 DE DE26 DE263 Würzburg Würzburg Würzburg Universität der Künste Berlin University of the DE0149 DE DE30 DE300 Berlin Berlin Arts Hochschule für Film und Hochschule für Film und DE0153 Fernsehen in Potsdam- Fernsehen in Potsdam- DE DE40 DE404 Potsdam Babelsberg Babelsberg Hochschule für Bildende University of Fine Arts of DE0155 DE DE60 DE600 Hamburg Künste Hamburg Hamburg Hochschule für Musik und Hochschule für Musik und DE0156 DE DE60 DE600 Hamburg Theater Hamburg Theater Hamburg Hochschule für Musik und Frankfurt University of Frankfurt am DE0158 Darstellende Kunst DE DE71 DE712 Music and Performing Arts Main Frankfurt a.M. Hochschule für Gestaltung Offenbach Academy of Art DE0159 DE DE71 DE713 Offenbach Offenbach and Design Hochschule für Musik und Rostock University of DE0160 DE DE80 DE803 Rostock Theater, Rostock Music and Drama Hochschule für Bildende Braunschweig University DE0161 DE DE91 DE911 Braunschweig Künste Braunschweig of Art Hochschule für Musik, The University of Music, DE0162 Theater und Medien Drama und Media DE DE92 DE929 Hannover Hannover Hanover Hochschule für Musik Hochschule für Musik DE0163 DE DEA4 DEA45 Detmold Detmold Detmold Kunstakademie DE0164 Kunstakademie Düsseldorf DE DEA1 DEA11 Düsseldorf Düsseldorf Robert-Schumann- Robert Schumann School DE0165 DE DEA1 DEA11 Düsseldorf Hochschule Düsseldorf of Music and Media Bochum, Folkwang Universität der Folkwang University of Dortmund, DE0166 DE DEA5 DEA51 Künste the Arts Duisburg, Essen Kunsthochschule für Medien DE0167 Academy of Media Arts DE DEA2 DEA23 Köln Köln

Cologne University of Aachen, Köln, DE0168 Hochschule für Musik Köln DE DEA2 DEA2D Music Wuppertal

Academy of Fine Arts DE0169 Kunstakademie Münster DE DEA3 DEA33 Münster Münster Alanus Hochschule Alfter DE0170 Alanus University DE DEA2 DEA2C Alfter (Priv. H)

Hochschule der Bildenden Hochschule der Bildenden DE0171 DE DEC0 DEC01 Saarbrücken Künste Saarbrücken Künste Saarbrücken Hochschule für Bildende Dresden Academy of Fine DE0173 DE DED2 DED21 Dresden Künste Dresden Arts Hochschule für Musik Hochschule für Musik Carla DE0175 Carla Maria von Weben DE DED2 DED21 Dresden Maria von Weben Dresden Dresden Hochschule für Graphik und Academy of Visual Arts DE0177 DE DED5 DED51 Leipzig Buchkunst Leipzig Leipzig Hochschule für Musik und University of Music & DE0178 DE DED5 DED51 Leipzig Theater Leipzig Theatre Leipzig Burg Giebichenstein Burg Giebichenstein DE0180 University of Art and DE DEE0 DEE02 Halle Kunsthochschule Halle Design Halle Muthesius Kunsthochschule Muthesius Academy of DE0182 DE DEF0 DEF02 Kiel Kiel Fine Arts and Design University of Music DE0183 Musikhochschule Lübeck DE DEF0 DEF03 Lübeck Lübeck Hochschule für Musik Liszt School of Music DE0184 DE DEG0 DEG05 Weimar Weimar Weimar

Hochschule für Angewandte Hamburg University of DE0274 DE DE60 DE600 Hamburg Wissenschaften Hamburg Applied Sciences

Hannover University of DE0299 Hochschule Hannover (FH) DE DE92 DE929 Hannover Applied Sciences and Arts Evangelische Hochschule Evangelische Hochschule DE0356 für Soziale Arbeit, Dresden für Soziale Arbeit, DE DED2 DED21 Dresden (FH) Dresden (FH) DK0001 Københavns Universitet University of Copenhagen DK DK01 DK011 København DK0002 Aarhus Universitet Aarhus University DK DK04 DK042 Aarhus University of Southern DK0003 Syddansk Universitet DK DK03 DK031 Odense Denmark DK0004 Roskilde Universitet Roskilde University DK DK02 DK021 Roskilde DK0005 Aalborg Universitet Aalborg University DK DK05 DK050 Aalborg Danmarks Tekniske Technical University of DK0006 DK DK01 DK012 Lyngby Universitet Denmark Copenhagen Business DK0007 Handelshøjskolen DK DK01 DK011 Fredriksberg School IT-Universitetet i IT University of DK0008 DK DK01 DK011 København København Copenhagen Aarhus School of DK0011 Arkitektskolen Aarhus DK DK04 DK042 Aarhus Architecture Det The Royal School of DK0014 informasjonsvidenskapelige Library and Information DK DK01 DK011 København Akademi Science Det Kongelige Danske The Royal Danish Kunstakademis Skoler for Academy of Fine Arts, DK0015 DK DK01 DK011 København Arkitektur, Design og Schools of Architecture, Konservering Design and Conservation EE0001 Tartu Ülikool University of Tartu EE EE00 EE008 Tartu Tallinn University of EE0002 Tallinna Tehnikaülikool EE EE00 EE001 Tallinn Technology EE0003 Tallinna Ülikool Tallinn University EE EE00 EE001 Tallinn Estonian University of Life EE0004 Eesti Maaülikool EE EE00 EE008 Tartu Sciences EE0007 Estonian Business School Estonian Business School EE EE00 EE001 Tallinn EE0008 Euroakadeemia Euroacademy EE EE00 EE001 Tallinn EE0011 Eesti Kunstiakadeemia Estonian Academy of Arts EE EE00 EE001 Tallinn Eesti Muusika- ja Estonian Academy of EE0022 EE EE00 EE001 Tallinn Teatriakadeemia Music and Theatre Institute of Theology of EE0032 EELK Usuteaduse Instituut the Estonian Evagelical EE EE00 EE001 Tallinn Lutheran Church ES0001 Universidad de Almeria University of Almeria ES ES61 ES611 ALMERÍA ES0002 Universidad de Cádiz University of Cádiz ES ES61 ES612 CÁDIZ ES0003 Universidad de Cordoba University of Cordoba ES ES61 ES613 CÓRDOBA ES0004 Universidad de Granada University of Granada ES ES61 ES614 GRANADA ES0005 Universidad de Huelva University of Huelva ES ES61 ES615 HUELVA ES0006 Universidad de Jaén University of Jaén ES ES61 ES616 JAÉN ES0007 Universidad de Málaga University of Málaga ES ES61 ES300 MÁLAGA

Universidad Pablo de Pablo de Olavide ES0008 ES ES61 ES618 SEVILLA Olavide University ES0009 Universidad de Sevilla University of Sevilla ES ES61 ES618 SEVILLA ES0010 Universidad de Zaragoza University of Zaragoza ES ES24 ES243 ZARAGOZA ES0011 Universidad de Oviedo University of Oviedo ES ES12 ES120 ASTURIAS Universitat de les Illes University of the Balearic ILLES ES0012 ES ES53 ES532 Balears Islands BALEARS

SANTA CRUZ ES0013 Universidad de La Laguna University of La Laguna ES ES70 ES709 DE TENERIFE

Universidad de Las Palmas University of Las Palmas ES0014 ES ES70 ES705 LAS PALMAS de Gran Canaria de Gran Canaria ES0015 Universidad de Cantabria University of Cantabria ES ES13 ES130 CANTABRIA Universitat Autònoma de Autonomous University of ES0016 ES ES51 ES511 BARCELONA Barcelona Barcelona ES0017 Universitat de Barcelona University of Barcelona ES ES51 ES511 BARCELONA ES0018 Universitat de Girona University of Girona ES ES51 ES512 GIRONA ES0019 Universitat de Lleida University of Lleida ES ES51 ES513 LLEIDA Universitat Politécnica de Technical University of ES0020 ES ES51 ES511 BARCELONA Catalunya Catalonia ES0021 Universitat Pompeu Fabra Pompeu Fabra University ES ES51 ES511 BARCELONA ES0022 Universitat Rovira i Virgili Rovira i Virgili University ES ES51 ES514 TARRAGONA Universidad de Castilla-La University of Castilla-La ES0023 ES ES42 ES422 CIUDAD REAL Mancha Mancha ES0024 Universidad de Alicante University of Alicante ES ES52 ES521 ALICANTE Universidad Jaume I de ES0025 Jaume I University ES ES52 ES522 CASTELLÓN Castellón Universidad Miguel Miguel Hernández ES0026 ES ES52 ES521 ALICANTE Hernández de Elche University of Elche Universidad Politécnica de Technical University of ES0027 ES ES52 ES523 VALENCIA Valencia Valencia Universitat de València ES0028 University of Valencia ES ES52 ES523 VALENCIA (Estudi General) ES0029 Universidad de Burgos University of Burgos ES ES41 ES412 BURGOS ES0030 Universidad de León University of León ES ES41 ES413 LEÓN

ES0031 Universidad de Salamanca University of Salamanca ES ES41 ES415 SALAMANCA

ES0032 Universidad de Valladolid University of Valladolid ES ES41 ES418 VALLADOLID Universidad de ES0033 University of Extremadura ES ES43 ES431 BADAJOZ Extremadura ES0034 Universidad de A Coruña University of A Coruña ES ES11 ES111 A CORUÑA Universidad de Santiago de University of Santiago de ES0035 ES ES11 ES111 A CORUÑA Compostela Compostela

ES0036 Universidad de Vigo University of Vigo ES ES11 ES114 PONTEVEDRA

ES0037 Universidad de Alcalá University of Alcalá ES ES30 ES300 MADRID Universidad Autónoma de Autonomous University of ES0038 ES ES30 ES300 MADRID Madrid Madrid Universidad Carlos III de Carlos III University of ES0039 ES ES30 ES300 MADRID Madrid Madrid Universidad Complutense Complutense University of ES0040 ES ES30 ES300 MADRID de Madrid Madrid Universidad Politécnica de Technical University of ES0041 ES ES30 ES300 MADRID Madrid Madrid Universidad Rey Juan Rey Juan Carlos ES0042 ES ES30 ES300 MADRID Carlos University ES0043 Universidad de Murcia University of Murcia ES ES62 ES620 MURCIA Universidad Politécnica de Technical University of ES0044 ES ES62 ES620 MURCIA Cartagena Cartagena Universidad Pública de Public University of ES0045 ES ES22 ES220 NAVARRA Navarra Navarra Universidad del País Vasco / University of the Basque ES0046 Euskal Herriko ES ES21 ES213 BIZKAIA Country Unibertsitatea ES0047 Universidad de La Rioja University of La Rioja ES ES23 ES230 LA RIOJA ES0048 IE. Universidad IE University ES ES41 ES416 SEGOVIA Universidad a Distancia de Open University of Madrid ES0049 ES ES30 ES300 MADRID Madrid (UDIMA) (UDIMA) Universidad Alfonso X El Alfonso X El Sabio ES0050 ES ES30 ES300 MADRID Sabio University

Universidad Antonio de ES0051 Nebrija University ES ES30 ES300 MADRID Nebrija Universidad Camilo José Camilo José Cela ES0052 ES ES30 ES300 MADRID Cela University Universidad Cardenal Cardenal Herrera-CEU ES0053 ES ES52 ES523 VALENCIA Herrera-CEU University Universidad Católica de Catholic University of ES0054 ES ES41 ES411 ÁVILA Ávila Avila Universidad Católica de Catholic University of ES0055 ES ES52 ES523 VALENCIA Valencia Valencia Universidad Católica San San Antonio Catholic ES0056 ES ES62 ES620 MURCIA Antonio de Murcia University of Murcia ES0057 Universidad de Deusto University of Deusto ES ES21 ES213 BIZKAIA ES0058 Universidad de Navarra University of Navarra ES ES22 ES220 NAVARRA Universidad Europea de European University of ES0059 ES ES30 ES300 MADRID Madrid Madrid Universidad Europea Miguel European University ES0060 ES ES41 ES418 VALLADOLID de Cervantes Miguel de Cervantes Universidad Francisco de Francisco de Vitoria ES0061 ES ES30 ES300 MADRID Vitoria University Universidad Internacional International University of ES0062 ES ES61 ES618 SEVILLA de Andalucía Andalucía Universidad Internacional International University ES0063 ES ES41 ES412 BURGOS Isabel I de Castilla Isabel I de Castilla Universidad Internacional International University of ES0064 ES ES23 ES230 LA RIOJA de la Rioja La Rioja Universidad Internacional Menéndez Pelayo ES0065 ES ES30 ES300 MADRID Menéndez Pelayo International University Universidad Mondragón ES0066 Mondragon University ES ES21 ES212 GIPUZKOA Unibertsitatea Universidad Nacional de National University of ES0067 ES ES30 ES300 MADRID Educación a Distancia Distance Education Universidad Pontificia Comillas Pontifical ES0068 ES ES30 ES300 MADRID Comillas University Universidad Pontificia de Pontifical University of ES0069 ES ES41 ES415 SALAMANCA Salamanca Salamanca ES0070 Universidad San Jorge San Jorge University ES ES24 ES243 ZARAGOZA

ES0071 Universidad San Pablo-CEU San Pablo-CEU University ES ES30 ES300 MADRID

ES0072 Universitat Abat Oliba CEU Abat Oliva-CEU university ES ES51 ES511 BARCELONA

ES0073 Universitat de Vic University of Vic ES ES51 ES511 BARCELONA Universitat Internacional de International University of ES0074 ES ES51 ES511 BARCELONA Catalunya Catalonia Universitat Internacional Valencian International ES0075 ES ES52 ES523 VALENCIA Valenciana (VIU) University Universitat Oberta de Open University of ES0076 ES ES51 ES511 BARCELONA Catalunya Catalonia ES0077 Universitat Ramón Llull Ramon Llull University ES ES51 ES511 BARCELONA FI0001 Helsingin yliopisto University of Helsinki FI FI1B FI1B1 Helsinki FI0002 Turun yliopisto University of Turku FI FI1C FI1C1 Turku FI0003 Åbo Akademi Åbo Akademi University FI FI1C FI1C1 Turku FI0004 Oulun yliopisto University of Oulu FI FI1D FI1D6 Oulu FI0005 Tampereen yliopisto University of Tampere FI FI19 FI197 Tampere FI0006 Jyväskylän yliopisto University of Jyväskylä FI FI19 FI193 Jyväskylä Hanken School of FI0010 Svenska handelshögskolan FI FI1B FI1B1 Helsinki Economics FI0013 Vaasan yliopisto University of Vaasa FI FI19 FI195 Vaasa Lappeenrannan teknillinen Lappeenranta University FI0014 FI FI1C FI1C5 Lappeenranta yliopisto of Technology Tampereen teknillinen Tampere University of FI0015 FI FI19 FI197 Tampere yliopisto Technology FI0019 Sibelius-Akatemia Sibelius Academy FI FI1B FI1B1 Helsinki FI0021 Lapin yliopisto University of Lapland FI FI1D FI1D7 Rovaniemi FI0022 Teatterikorkeakoulu Theatre Academy FI FI1B FI1B1 Helsinki Finnish Academy of Fine FI0023 Kuvataideakatemia FI FI1B FI1B1 Helsinki Arts Finnish National Defence FI0024 Maanpuolustuskorkeakoulu FI FI1B FI1B1 Helsinki University FI0025 Aalto-yliopisto Aalto University FI FI1B FI1B1 Espoo

University of Eastern FI0026 Itä-Suomen yliopisto FI FI1D FI1D2 Kuopio Finland Université Nice - Sophia- University of Nice Sophia FR0001 FR FR82 FR823 Nice Antipolis Antipolis Université de technologie University of technology FR0002 FR FR21 FR212 Troyes de Troyes of Troyes École centrale de FR0006 École centrale de Marseille FR FR82 FR824 Marseille Marseille FR0007 Aix-Marseille université Aix-Marseille University FR FR82 FR824 Marseille Université de Caen Basse- University of Caen Lower FR0008 FR FR25 FR251 Caen Normandie Normandy

École nationale supérieure École nationale supérieure FR0010 FR FR25 FR251 Caen d'ingénieurs de Caen d'ingénieurs de Caen

FR0014 Université de La Rochelle University of La Rochelle FR FR53 FR532 La Rochelle FR0018 Université de Bourgogne University of Burgundy FR FR26 FR261 Dijon Institut national supérieur Institut national supérieur des sciences des sciences agronomiques, FR0020 agronomiques, de FR FR26 FR261 Dijon de l'alimentation et de l'alimentation et de l'environnement l'environnement École nationale supérieure École nationale supérieure FR0021 de mécanique et des de mécanique et des FR FR43 FR431 Besançon microtechniques microtechniques Université de Franche- University of Franche- FR0022 FR FR43 FR431 Besançon Comté Comté Université de Bretagne University of Western FR0026 FR FR52 FR522 Brest Occidentale Brittany FR0027 Télécom Bretagne Télécom Bretagne FR FR52 FR522 Brest Institut national des Institut national des FR0033 sciences appliquées de sciences appliquées de FR FR62 FR623 Toulouse Toulouse Toulouse Institut catholique de Catholic University of FR0035 FR FR62 FR623 Toulouse Toulouse Toulouse

Institut national National polytechnic FR0037 FR FR62 FR623 Toulouse polytechnique de Toulouse institute of Toulouse

Université Toulouse 1 - Toulouse 1 University FR0038 FR FR62 FR623 Toulouse Capitole Capitole Université de Toulouse 2 - University of Toulouse II FR0039 FR FR62 FR623 Toulouse Le Mirail – Le Mirail Université de Toulouse 3 - FR0040 Paul Sabatier University FR FR62 FR623 Toulouse Paul Sabatier École nationale supérieure École nationale supérieure des sciences FR0043 des sciences agronomiques FR FR61 FR612 Gradignan agronomiques de de Bordeaux Aquitaine Bordeaux Aquitaine Université Bordeaux 1 FR0044 University of Bordeaux 1 FR FR61 FR612 Talence Sciences et Technologies Université Bordeaux 2 Bordeaux Segalen FR0045 FR FR61 FR612 Bordeaux Segalen University Université Bordeaux 3 Michel de Montaigne FR0046 FR FR61 FR612 Pessac Michel de Montaigne University Bordeaux 3 Université Bordeaux 4 Montesquieu University – FR0049 FR FR61 FR612 Pessac Montesquieu Bordeaux 4 Institut polytechnique de Institut polytechnique de FR0050 FR FR61 FR612 Talence Bordeaux Bordeaux

École nationale supérieure École nationale supérieure FR0051 FR FR81 FR813 Montpellier de chimie de Montpellier de chimie de Montpellier

FR0053 Université Montpellier 1 University of Montpellier 1 FR FR81 FR813 Montpellier Université Montpellier 2 - FR0054 Montpellier 2 University FR FR81 FR813 Montpellier Sciences et Techniques Université Montpellier 3 - Paul Valéry University, FR0055 FR FR81 FR813 Montpellier Paul-Valéry Montpellier 3 Centre international Centre international FR0056 d'études supérieures en d'études supérieures en FR FR81 FR813 Montpellier sciences agronomiques sciences agronomiques

École nationale supérieure École nationale supérieure FR0057 FR FR52 FR523 Rennes de chimie de Rennes de chimie de Rennes

Institut national des Institut national des FR0061 sciences appliquées de sciences appliquées de FR FR52 FR523 Rennes Rennes Rennes FR0062 Université de Rennes 1 University of Rennes 1 FR FR52 FR523 Rennes University of Rennes 2 – FR0063 Université Rennes 2 FR FR52 FR523 Rennes Upper Brittany Institut supérieur des Institut supérieur des sciences agronomiques, sciences agronomiques, FR0069 FR FR52 FR523 Rennes agroalimentaires, horticoles agroalimentaires, et du paysage horticoles et du paysage Université François- François Rabelais FR0070 FR FR24 FR244 Tours Rabelais University FR0073 Université Joseph Fourier Joseph Fourier University FR FR71 FR714 Grenoble Université Pierre Mendès Pierre Mendès-France FR0074 FR FR71 FR714 Grenoble France University Saint-Martin- FR0075 Université Stendhal Stendhal University FR FR71 FR714 d'Hères Grenoble institute of FR0076 Grenoble INP FR FR71 FR714 Grenoble technology Saint-Martin- FR0077 Université de Grenoble Université de Grenoble FR FR71 FR714 d'Hères

FR0081 Telecom Saint-Étienne Telecom Saint-Étienne FR FR71 FR715 Saint-Étienne

FR0082 Université Jean Monnet Jean Monnet University FR FR71 FR715 Saint-Étienne

FR0084 Ecole centrale de Nantes Ecole centrale de Nantes FR FR51 FR511 Nantes FR0087 Université de Nantes University of Nantes FR FR51 FR511 Nantes École nationale d'ingénieurs École nationale des techniques des d'ingénieurs des FR0089 FR FR51 FR511 Nantes industries agricoles et techniques des industries alimentaires agricoles et alimentaires

École nationale supérieure École nationale supérieure FR0090 FR FR51 FR511 Nantes des mines de Nantes des mines de Nantes

FR0092 Université d'Orléans University of Orléans FR FR24 FR246 Orléans Université catholique de Catholic University of the FR0098 FR FR51 FR512 Angers l'Ouest West FR0099 Université d'Angers University of Angers FR FR51 FR512 Angers Université de Reims University of Reims FR0102 FR FR21 FR213 Reims Champagne-Ardenne Champagne-Ardenne FR0105 Université de Lorraine University of Lorraine FR FR41 FR411 Nancy University of Southern FR0108 Université de Bretagne-Sud FR FR52 FR524 Lorient Brittany École nationale d'ingénieurs École nationale FR0109 FR FR41 FR413 Metz de Metz d'ingénieurs de Metz

École nationale supérieure École nationale supérieure Villeneuve- FR0112 FR FR30 FR301 de chimie de Lille de chimie de Lille d'Ascq

FR0118 Institut catholique de Lille Institut catholique de Lille FR FR30 FR301 Lille Villeneuve- FR0121 École centrale de Lille École centrale de Lille FR FR30 FR301 d'Ascq

Université de Valenciennes University of Valenciennes Aulnoy-lez- FR0122 FR FR30 FR301 et du Hainaut-Cambrésis and Hainaut-Cambresis Valenciennes

Université Lille 1 - Sciences Lille University of Science Villeneuve- FR0123 FR FR30 FR301 technologies and Technology d'Ascq Université Lille 2 - Droit et Lille 2 University of Health FR0124 FR FR30 FR301 Lille Santé and Law Université Lille 3 - Charles- Lille 3 - Charles de Gaulle Villeneuve- FR0125 FR FR30 FR301 de-Gaulle University d'Ascq Université du Littoral Côte University of the Littoral FR0127 FR FR30 FR301 Dunkerque d'Opale Opal Coast Université de technologie University of technology FR0130 FR FR22 FR222 Compiègne de Compiègne of compiègne FR0132 Université d'Artois Artois University FR FR30 FR302 Arras Clermont- FR0135 Université d'Auvergne University of Auvergne FR FR72 FR724 Ferrand

Clermont- FR0137 Université Blaise Pascal Blaise Pascal University FR FR72 FR724 Ferrand FR0138 VetAgro Sup VetAgro Sup FR FR72 FR724 Lempdes Université de Pau et des University of Pau and FR0140 FR FR61 FR615 Pau Pays de l'Adour Pays de l'Adour Université de Perpignan - FR0144 University of Perpignan FR FR81 FR815 Perpignan Via Domitia Institut national des Institut national des FR0148 sciences appliquées de sciences appliquées de FR FR42 FR421 Strasbourg Strasbourg Strasbourg FR0149 Université de Strasbourg University of Strasbourg FR FR42 FR421 Strasbourg

FR0150 Université de Haute-Alsace University of Upper Alsace FR FR42 FR422 Mulhouse

FR0153 École centrale de Lyon École centrale de Lyon FR FR71 FR716 Écully Institut national des Institut national des FR0154 sciences appliquées de sciences appliquées de FR FR71 FR716 Villeurbanne Lyon Lyon Université catholique de Catholic University of FR0156 FR FR71 FR716 Lyon Lyon Lyon Université Claude Bernard - Université Claude Bernard FR0157 FR FR71 FR716 Villeurbanne Lyon 1 Lyon 1

FR0158 Université Lumière - Lyon 2 Lumière University Lyon 2 FR FR71 FR716 Lyon

Université Jean Moulin - Jean Moulin University FR0161 FR FR71 FR716 Lyon Lyon 3 Lyon 3 École normale supérieure École normale supérieure FR0168 FR FR71 FR716 Lyon de Lyon de Lyon FR0169 Université du Maine University of Maine FR FR51 FR514 Le Mans FR0170 Université de Savoie University of Savoy FR FR71 FR717 Chambéry FR0173 Université Paris-Dauphine Paris Dauphine University FR FR10 FR101 Paris Université Paris 1 - Pantheon-Sorbonne FR0174 FR FR10 FR101 Paris Panthéon Sorbonne University Pantheon-Assas FR0175 Université Panthéon-Assas FR FR10 FR101 Paris University Université Sorbonne University of Paris 3 - FR0176 FR FR10 FR101 Paris Nouvelle - Paris 3 Sorbonne Nouvelle FR0177 Université Paris-Sorbonne Paris-Sorbonne University FR FR10 FR101 Paris FR0178 Université Paris Descartes Paris Descartes University FR FR10 FR101 Paris Université Pierre et Marie Pierre and Marie Curie FR0179 FR FR10 FR101 Paris Curie University FR0180 Université Paris Diderot Paris Diderot University FR FR10 FR101 Paris

École nationale supérieure École nationale supérieure FR0185 FR FR10 FR101 Paris d'arts et métiers d'arts et métiers

École nationale supérieure École nationale supérieure FR0186 FR FR10 FR101 Paris de chimie de Paris de chimie de Paris

Institut de physique du Institut de physique du FR0187 FR FR10 FR101 Paris globe globe École supérieure de École supérieure de physique et de chimie physique et de chimie FR0188 FR FR10 FR101 Paris industrielles de la ville de industrielles de la ville de Paris Paris Institut d'études politiques Paris Institute of political FR0189 FR FR10 FR101 Paris de Paris studies École normale supérieure École normale supérieure FR0190 FR FR10 FR101 Paris de Paris de Paris Institut des sciences et Institut des sciences et FR0191 industries du vivant et de industries du vivant et de FR FR10 FR101 Paris l'environnement l'environnement Conservatoire national des Conservatoire national FR0193 FR FR10 FR101 Paris arts et métiers des arts et métiers École nationale des FR0194 École nationale des Chartes FR FR10 FR101 Paris Chartes FR0195 Collège de France Collège de France FR FR10 FR101 Paris École pratique des hautes École pratique des hautes FR0196 FR FR10 FR101 Paris études études Institut national des Institut national des FR0197 langues et civilisations langues et civilisations FR FR10 FR101 Paris orientales orientales

École nationale supérieure École nationale supérieure FR0199 FR FR10 FR101 Paris des mines de Paris des mines de Paris

Muséum national d'histoire National museum of FR0200 FR FR10 FR101 Paris naturelle natural history FR0202 Observatoire de Paris Observatoire de Paris FR FR10 FR101 Paris FR0203 Télécom ParisTech Télécom ParisTech FR FR10 FR101 Paris Institut Catholique de FR0204 Institut Catholique de Paris FR FR10 FR101 Paris Paris School for advanced École des hautes études en FR0208 studies in the social FR FR10 FR101 Paris sciences sociales sciences Institut national des Institut national des Saint- FR0217 sciences appliquées de sciences appliquées de FR FR23 FR232 Étienne-du- Rouen Rouen Rouvray Mont-Saint- FR0218 Université de Rouen University of Rouen FR FR23 FR232 Aignan FR0220 Université du Havre University of Le Havre FR FR23 FR232 Le Havre Université Paris-Est Marne- University of Marne la Champs-sur- FR0223 FR FR10 FR102 la-Vallée Vallée Marne École nationale des ponts et École nationale des ponts Champs-sur- FR0224 FR FR10 FR102 chaussées et chaussées Marne Champs-sur- FR0226 Université Paris-Est Université Paris-Est FR FR10 FR102 Marne Université de Versailles Versailles Saint-Quentin- FR0229 FR FR10 FR103 Versailles Saint-Quentin-en-Yvelines en-Yvelines University Université de Picardie Jules- University of Picardie FR0230 FR FR22 FR223 Amiens Verne Jules Verne École Supérieure École Supérieure d'Ingénieurs en d'Ingénieurs en FR0231 FR FR22 FR223 Amiens Électronique et Électronique et Électrotechnique Électrotechnique Université du Sud Toulon - University of the South, FR0234 FR FR82 FR825 La Garde Var Toulon-Var Université d'Avignon et des FR0236 University of Avignon FR FR82 FR826 Avignon Pays de Vaucluse École nationale supérieure École nationale supérieure de mécanique et Chasseneuil- FR0238 de mécanique et FR FR53 FR534 d'aérotechnique de du-Poitou d'aérotechnique de Poitiers Poitiers FR0239 Université de Poitiers University of Poitiers FR FR53 FR534 Poitiers FR0240 Université de Limoges University of Limoges FR FR63 FR633 Limoges École nationale supérieure École nationale supérieure FR0241 de céramique industrielle de céramique industrielle FR FR63 FR633 Limoges de Limoges de Limoges Université de technologie University of technology FR0243 FR FR43 FR434 Sevenans de Belfort-Montbéliard of Belfort-Montbéliard Institut d'Optique Graduate FR0244 Higher school of optics FR FR10 FR104 Palaiseau School FR0245 Université Paris-Sud University of Paris-Sud FR FR10 FR104 Orsay École supérieure École supérieure FR0246 FR FR10 FR104 Gif-sur-Yvette d'électricité d'électricité FR0247 École polytechnique École polytechnique FR FR10 FR104 Palaiseau FR0248 Télécom SudParis Télécom SudParis FR FR10 FR104 Évry Université d'Évry-Val University of Évry Val FR0249 FR FR10 FR104 Évry d'Essonne d'Essonne Université Paris Ouest Paris West University FR0255 FR FR10 FR105 Nanterre Nanterre La Défense Nanterre La Défense Châtenay- FR0256 Ecole centrale de Paris Ecole centrale de Paris FR FR10 FR105 Malabry Université Paris 13 - Paris FR0262 Paris 13 University FR FR10 FR106 Villetaneuse Nord Université Paris 8 - FR0263 Paris 8 University FR FR10 FR106 Saint-Denis Vincennes - Saint-Denis École normale supérieure École normale supérieure FR0265 FR FR10 FR107 Cachan de Cachan de Cachan Université Paris-Est Créteil Paris 12 Val de Marne FR0266 FR FR10 FR107 Créteil Val-de-Marne University Université de Cergy- FR0275 Cergy-Pontoise University FR FR10 FR108 Cergy Pontoise

Université de Corse University of Corsica FR0281 FR FR83 FR832 Corte Pasquale Paoli Pascal Paoli Université des Antilles et de University of the French FR0282 FR FR91 FR910 Pointe-à-Pitre la Guyane West Indies and Guiana FR0283 Université de La Réunion University of La Réunion FR FR94 FR940 Saint-Denis Université de la Nouvelle- University of New FR0285 FR a a Nouméa Calédonie Caledonia Université de la Polynésie University of French FR0286 FR a a Punaauia Française Polynesia ΕΛΛΗΝΙΚΟ ΑΝΟΙΚΤΟ HELLENIC OPEN GR0001 GR EL23 EL232 Patra ΠΑΝΕΠΙΣΤΗΜΙΟ UNIVERSITY ΠΑΝΕΠΙΣΤΗΜΙΟ ΔΥΤΙΚΗΣ University of Western GR0003 GR EL23 EL231 Agrinio ΕΛΛΑΔΑΣ Greece ΠΑΝΕΠΙΣΤΗΜΙΟ ΣΤΕΡΕΑΣ UNIVERSITY OF CENTRAL GR0004 GR EL24 EL244 Lamia ΕΛΛΑΔΟΣ GREECE ΠΑΝΕΠΙΣΤΗΜΙΟ ΔΥΤΙΚΗΣ UNIVERSITY OF GR0005 GR EL13 EL133 Kozani ΜΑΚΕΔΟΝΙΑΣ WESTERN MACEDONIA ΠΑΝΕΠΙΣΤΗΜΙΟ UNIVERSITY OF GR0006 GR EL25 EL252 Tripoli ΠΕΛΟΠΟΝΝΗΣΟΥ PELOPONNESE ΧΑΡΟΚΟΠΕΙΟ HAROKOPION GR0007 GR EL30 EL300 Athina ΠΑΝΕΠΙΣΤΗΜΙΟ UNIVERSITY ΑΝΩΤΑΤΗ ΣΧΟΛΗ ΚΑΛΩΝ THE ATHENS SCHOOL OF GR0008 GR EL30 EL300 Athina ΤΕΧΝΩΝ FINE ARTS ΓΕΩΠΟΝΙΚΟ AGRICULTURAL GR0009 GR EL30 EL300 Athina ΠΑΝΕΠΙΣΤΗΜΙΟ ΑΘΗΝΩΝ UNIVERSITY OF ATHENS ΠΑΝΕΠΙΣΤΗΜΙΟ UNIVERSITY OF ΜΑΚΕΔΟΝΙΑΣ GR0010 MACEDONIA ECONOMIC GR EL12 EL122 Thessaloniki ΟΙΚΟΝΟΜΙΚΩΝ ΚΑΙ AND SOCIAL SCIENCES ΚΟΙΝΩΝΙΚΩΝ ΕΠΙΣΤΗΜΩΝ GR0011 ΠΑΝΕΠΙΣΤΗΜΙΟ ΠΕΙΡΑΙΩΣ UNIVERSITY OF PIRAEUS GR EL30 EL300 Athina ΠΑΝΤΕΙΟ ΠΑΝΕΠΙΣΤΗΜΙΟ PANTEION UΝIVERSITY GR0012 ΚΟΙΝΩΝΙΚΩΝ ΚΑΙ OF SOCIAL AND GR EL30 EL300 Athina ΠΟΛΙΤΙΚΩΝ ΕΠΙΣΤΗΜΩΝ POLITICAL SCIENCES ATHENS UNIVERSITY OF ΟΙΚΟΝΟΜΙΚΟ GR0013 ECONOMICS AND GR EL30 EL300 Athina ΠΑΝΕΠΙΣΤΗΜΙΟ ΑΘΗΝΩΝ BUSINESS TECHNICAL UNIVERSITY GR0014 ΠΟΛΥΤΕΧΝΕΙΟ ΚΡΗΤΗΣ GR EL43 EL434 Chania OF CRETE ΕΘΝΙΚΟ ΜΕΤΣΟΒΙΟ NATIONAL TECHNICAL GR0015 GR EL30 EL300 Athina ΠΟΛΥΤΕΧΝΕΙΟ UNIVERSITY OF ATHENS GR0016 ΠΑΝΕΠΙΣΤΗΜΙΟ ΠΑΤΡΩΝ UNIVERSITY OF PATRAS GR EL23 EL232 Patra GR0017 ΠΑΝΕΠΙΣΤΗΜΙΟ ΚΡΗΤΗΣ UNIVERSITY OF CRETE GR EL43 EL433 Rethymno ΠΑΝΕΠΙΣΤΗΜΙΟ UNIVERSITY OF GR0018 GR EL21 EL213 Ioannina ΙΩΑΝΝΙΝΩΝ IOANNINA GR0019 ΙΟΝΙΟ ΠΑΝΕΠΙΣΤΗΜΙΟ IONIAN UNIVERSITY GR EL22 EL223 Kerkyra NATIONAL AND ΕΘΝΙΚΟ ΚΑΠΟΔΙΣΤΡΙΑΚΟ GR0020 KAPODISTRIAN GR EL30 EL300 Athina ΠΑΝΕΠΙΣΤΗΜΙΟ ΑΘΗΝΩΝ UNIVERSITY OF ATHENS ΑΡΙΣΤΟΤΕΛΕΙΟ ARISTOTLE UNIVERSITY GR0021 ΠΑΝΕΠΙΣΤΗΜΙΟ GR EL12 EL122 Thessaloniki OF THESSALONIKI ΘΕΣΣΑΛΟΝΙΚΗΣ ΔΗΜΟΚΡΙΤΕΙΟ DEMOCRITUS GR0022 GR EL11 EL113 Komotini ΠΑΝΕΠΙΣΤΗΜΙΟ ΘΡΑΚΗΣ UNIVERSITY OF THRACE ΠΑΝΕΠΙΣΤΗΜΙΟ UNIVERSITY OF GR0023 GR EL14 EL143 Volos ΘΕΣΣΑΛΙΑΣ THESSALY UNIVERSITY OF THE GR0024 ΠΑΝΕΠΙΣΤΗΜΙΟ ΑΙΓΑΙΟΥ GR EL41 EL411 Lesvos AEGEAN HR0001 Sveucilište u Dubrovniku University of Dubrovnik HR HR03 HR037 Dubrovnik Sveucilište Jurja Dobrile U Juraj Dobrila University HR0002 HR HR03 HR036 Pula-Pola Puli-Pola of Pula-Pola Josip Juraj Sveucilište J.J. HR0003 Strossmayer University of HR HR04 HR04B Osijek Strossmayera u Osijeku Osijek HR0004 Sveucilište u Rijeci University of Rijeka HR HR03 HR031 Rijeka HR0005 Sveucilište u Splitu University of Split HR HR03 HR035 Split HR0006 Sveucilište u Zadru University of Zadar HR HR03 HR033 Zadar HR0007 Sveucilište u Zagrebu University of Zagreb HR HR04 HR041 Zagreb Budapesti Corvinus Corvinus University of HU001 HU HU10 HU101 Budapest Egyetem (BCE) Budapest

Budapesti Műszaki és Budapest University of HU002 Gazdaságtudományi Technology and HU HU10 HU101 Budapest Egyetem (BME) Economics HU003 Debreceni Egyetem (DE) University of Debrecen HU HU32 HU321 Debrecen

Eötvös Loránd Eötvös Loránd University, HU004 HU HU10 HU101 Budapest Tudományegyetem (ELTE) Budapest

HU005 Kaposvári Egyetem (KE) University of Kaposvár HU HU23 HU232 Kaposvár Liszt Ferenc Academy of Liszt Ferenc Zeneművészeti HU006 Music (University), HU HU10 HU101 Budapest Egyetem (LFZE) Budapest Magyar Képzőművészeti Hungarian Academy of HU007 HU HU10 HU101 Budapest Egyetem (MKE) Fine Arts, Budapest HU008 Miskolci Egyetem (ME) University of Miskolc HU HU31 HU311 Miskolc Moholy-Nagy Művészeti Moholy-Nagy University of HU009 HU HU10 HU101 Budapest Egyetem (MOME) Art and Design, Budapest Nyugat-magyarországi University of West HU010 HU HU22 HU221 Sopron Egyetem (NYME) Hungary, Sopron Unviersity of Pannonia, HU011 Pannon Egyetem (PE) HU HU21 HU213 Veszprém Veszprém Pécsi Tudományegyetem HU012 University of Pécs HU HU23 HU231 Pécs (PTE) Semmelweis University, HU013 Semmelweis Egyetem (SE) HU HU10 HU101 Budapest Budapest Széchenyi István Egyetem Széchenyi István HU014 HU HU22 HU221 Győr (SZE) University, Győr

Szegedi Tudományegyetem HU015 University of Szeged HU HU33 HU333 Szeged (SZTE)

Szent István Egyetem Szent István University, HU016 HU HU10 HU102 Gödöllő (SZIE) Gödöllő

Színház- és Filmművészeti University of Drama and HU017 HU HU10 HU101 Budapest Egyetem (SZFE) Film, Budapest

Zrínyi Miklós Zrínyi Miklós University of HU018 Nemzetvédelmi Egyetem National Defence, HU HU10 HU101 Budapest (ZNME) Budapest Andrássy Gyula Budapesti Andrássy Gyula HU019 HU HU10 HU101 Budapest Német Nyelvű Egyetem University, Budapest Debreceni Református Debrecen University of HU020 Hittudományi Egyetem HU HU32 HU321 Debrecen Reformed Theology (DRHE) Evangelical-Lutheran Evangélikus Hittudományi HU021 Theological University, HU HU10 HU101 Budapest Egyetem (EHE) Budapest Károli Gáspár Universiy of Károli Gáspár Református HU022 the Reformed Church, HU HU10 HU101 Budapest Egyetem (KGRE) Budapest Közép-európai Egyetem Central European HU023 HU HU10 HU101 Budapest (CEU) University, Budapest Jewish Theological Országos Rabbiképző - HU024 Seminary - University of HU HU10 HU101 Budapest Zsidó Egyetem (OR-ZSE) Jewish Studies, Budapest Pázmány Péter Katolikus Pázmány Péter Catholic HU025 HU HU10 HU101 Budapest Egyetem (PPKE) University, Budapest IE0001 University College Dublin University College Dublin IE IE02 IE021 Dublin IE0002 University College Cork University College Cork IE IE02 IE025 Cork National University of National University of IE0003 IE IE01 IE013 Galway Ireland, Galway Ireland, Galway IE0004 Trinity College Dublin Trinity College Dublin IE IE02 IE021 Dublin National University of National University of IE0005 IE IE02 IE021 Maynooth Ireland, Maynooth Ireland, Maynooth IE0006 Dublin City University Dublin City University IE IE02 IE021 Dublin IE0007 University of Limerick University of Limerick IE IE02 IE023 Limerick Athlone Institute of Athlone Institute of IE0008 IE IE01 IE012 Athlone Technology Technology Cork Institute of Cork Institute of IE0009 IE IE02 IE025 Cork Technology Technology Dublin Institute of Dublin Institute of IE0010 IE IE02 IE021 Dublin Technology Technology

Dundalk Institute of Dundalk Institute of IE0012 IE IE01 IE011 Dundalk Technology Technology Galway-Mayo Institute of Galway-Mayo Institute of IE0013 IE IE01 IE013 Galway Technology Technology Institute of Technology, Institute of Technology, IE0014 IE IE02 IE021 Dublin Blanchardstown Blanchardstown Institute of Technology, Institute of Technology, IE0015 IE IE02 IE024 Carlow Carlow Carlow Institute of Technology, Institute of Technology, IE0016 IE IE01 IE011 Sligo Sligo Sligo Institute of Technology, Institute of Technology, IE0017 IE IE02 IE021 Dublin Tallaght Tallaght Institute of Technology, Institute of Technology, IE0018 IE IE02 IE025 Tralee Tralee Tralee Letterkenny Institute of Letterkenny Institute of IE0019 IE IE01 IE011 Letterkenny Technology Technology Limerick Institute of Limerick Institute of IE0020 IE IE02 IE023 Limerick Technology Technology Waterford Institute of Waterford Institute of IE0021 IE IE02 IE024 Waterford Technology Technology Mary Immaculate College of Mary Immaculate College IE0022 IE IE02 IE023 Limerick Education, Limerick of Education, Limerick Mater Dei Institute, Mater Dei Institute, IE0023 IE IE02 IE021 Dublin Clonliffe Road, Dublin Clonliffe Road, Dublin National College of Art & National College of Art & IE0024 IE IE02 IE021 Dublin Design, Dublin Design, Dublin Royal College of Surgeons Royal College of Surgeons IE0025 IE IE02 IE021 Dublin in Ireland in Ireland St. Patrick's Teacher St. Patrick's Teacher IE0027 Training College, Training College, IE IE02 IE021 Dublin Drumcondra, Dublin Drumcondra, Dublin Agricultural University of IS0001 Landbúnaðarháskóli íslands IS IS00 IS002 Borgarnes Iceland IS0005 Háskólinn í Reykjavík Reykjavik University IS IS00 IS001 Reykjavík IS0007 Háskóli Íslands University of Iceland IS IS00 IS001 Reykjavík Università Politecnica delle Università Politecnica IT0001 IT ITI3 ITI32 ANCONA MARCHE delle Marche ARCAVACATA IT0003 Università della CALABRIA University of Calabria IT ITF6 ITF61 DI RENDE IT0004 Politecnico di BARI Polytechnic of Bari IT ITF4 ITF47 BARI Università degli Studi di University of Bari Aldo IT0005 IT ITF4 ITF47 BARI BARI ALDO MORO Moro Università degli Studi del IT0006 University of Sannio IT ITF3 ITF32 BENEVENTO SANNIO di BENEVENTO Università degli Studi di IT0008 University of Bergamo IT ITC4 ITC46 BERGAMO BERGAMO Università degli Studi di IT0009 University of Bologna IT ITH5 ITH55 BOLOGNA BOLOGNA Libera Università di Free University of Bozen- IT0010 IT ITH1 ITH10 BOLZANO BOLZANO Bolzano Università degli Studi di IT0012 University of Brescia IT ITC4 ITC47 BRESCIA BRESCIA Università degli Studi di IT0013 University of Cagliari IT ITG2 ITG27 CAGLIARI CAGLIARI Università degli Studi di IT0014 University of Camerino IT ITI3 ITI33 CAMERINO CAMERINO Università degli Studi del CAMPOBASS IT0015 University of Molise IT ITF2 ITF22 MOLISE O Libera Università University LUM 'Jean CASAMASSIM IT0016 Mediterranea 'Jean Monnet' IT ITF4 ITF47 Monnet' A – LUM Università degli Studi di University of Cassino and IT0017 CASSINO e del LAZIO IT ITI4 ITI45 CASSINO Lazio Meridionale MERIDIONALE Università 'Carlo Cattaneo' LIUC – Università CASTELLANZ IT0018 IT ITC4 ITC41 - LIUC Cattaneo A Università degli Studi di IT0019 University of Catania IT ITG1 ITG17 CATANIA CATANIA

Università degli Studi University 'Magna IT0020 'Magna Graecia' di IT ITF6 ITF63 CATANZARO Graecia' CATANZARO Università degli Studi 'G. University G. d’Annunzío IT0021 d'Annunzio' CHIETI- IT ITF1 ITF14 CHIETI in Chieti-Pescara PESCARA UKE - Università Kore di Kore University of Enna IT0022 IT ITG1 ITG16 ENNA ENNA (UKE) Università degli Studi di IT0023 University of Ferrara IT ITH5 ITH56 FERRARA FERRARA SUM - Istituto Italiano di Istituto Italiano di Scienze IT0024 SCIENZE UMANE di IT ITI1 ITI14 FIRENZE Umane (SUM) FIRENZE Università degli Studi di IT0025 University of Florence IT ITI1 ITI14 FIRENZE FIRENZE Università degli Studi di IT0027 University of Foggia IT ITF4 ITF46 FOGGIA FOGGIA Università degli Studi di IT0028 University of Genova IT ITC3 ITC33 GENOVA GENOVA Università degli Studi de IT0029 University of L'Aquila IT ITF1 ITF11 L'AQUILA L'AQUILA Università degli Studi del IT0030 Università del Salento IT ITF4 ITF45 LECCE SALENTO Scuola IMT - Istituzioni, IMT Institute for IT0031 Mercati, Tecnologie - Alti IT ITI1 ITI12 LUCCA Advanced Studies Lucca Studi - LUCCA Università degli Studi di IT0032 University of Macerata IT ITI3 ITI33 MACERATA MACERATA Università degli Studi di IT0033 University of Messina IT ITG1 ITG13 MESSINA MESSINA Free University of Libera Università di lingue e IT0034 Languages and IT ITC4 ITC4C MILANO comunicazione IULM-MI Communication IT0035 Politecnico di MILANO Politecnico di Milano IT ITC4 ITC4C MILANO Università Cattolica del Università Cattolica del IT0036 IT ITC4 ITC4C MILANO Sacro Cuore Sacro Cuore Università Commerciale IT0037 Università Bocconi IT ITC4 ITC4C MILANO 'Luigi Bocconi' MILANO Università degli Studi di IT0038 University of Milano IT ITC4 ITC4C MILANO MILANO

Libera Università 'Vita Vita-Salute San Raffaele IT0039 IT ITC4 ITC4C MILANO Salute S.Raffaele' MILANO University

Università degli Studi di University of Milano- IT0040 IT ITC4 ITC4C MILANO MILANO-BICOCCA Bicocca

Università degli Studi di University of Modena and IT0042 IT ITH5 ITH54 MODENA MODENA e REGGIO EMILIA Reggio Emilia (UNIMORE)

Università degli Studi Suor Suor Orsola Benincasa IT0043 IT ITF3 ITF33 NAPOLI Orsola Benincasa - NAPOLI University

Seconda Università degli Second University of IT0044 IT ITF3 ITF31 CASERTA Studi di NAPOLI Naples Università degli Studi di University of Naples IT0045 IT ITF3 ITF33 NAPOLI NAPOLI 'Federico II' Federico II Università degli Studi di 'Orientale' University of IT0046 IT ITF3 ITF33 NAPOLI NAPOLI 'L'Orientale' Naples Università degli Studi di University of Naples IT0047 IT ITF3 ITF33 NAPOLI NAPOLI 'Parthenope' 'Parthenope' Università degli Studi di IT0050 University of Padova IT ITH3 ITH36 PADOVA PADOVA Università degli Studi di IT0051 University of Palermo IT ITG1 ITG12 PALERMO PALERMO Università degli Studi di IT0052 University of Parma IT ITH5 ITH52 PARMA PARMA I.U.S.S. - Istituto Institute for Advanced IT0053 Universitario di Studi IT ITC4 ITC48 PAVIA Study - IUSS of Pavia Superiori - PAVIA Università degli Studi di IT0054 University of Pavia IT ITC4 ITC48 PAVIA PAVIA

Università degli Studi di IT0055 University of Perugia IT ITI2 ITI21 PERUGIA PERUGIA Università per Stranieri di University for Foreigners IT0056 IT ITI2 ITI21 PERUGIA PERUGIA Perugia Scuola Normale Superiore IT0057 Scuola Normale Superiore IT ITI1 ITI17 PISA di PISA

Scuola Superiore di Studi Sant'Anna School for IT0058 Universitari e IT ITI1 ITI17 PISA Advanced Studies Perfezionamento Sant'Anna

IT0059 Università di PISA University of Pisa IT ITI1 ITI17 PISA Università degli Studi della University of the IT0060 IT ITF5 ITF51 POTENZA BASILICATA Basilicata Università degli Studi Università Mediterranea REGGIO IT0061 'Mediterranea' di REGGIO IT ITF6 ITF65 of Reggio Calabria CALABRIA CALABRIA Università degli Studi ROMA IT0063 Roma Tre University IT ITI4 ITI43 ROMA TRE Libera Università degli Studi IT0064 Per l'Innovazione e le Free University LUSPIO IT ITI4 ITI43 ROMA Organizzazioni LUSPIO Libera Univ. Inter.le Studi IT0065 Sociali 'Guido Carli' LUISS- LUISS Guido Carli IT ITI4 ITI43 ROMA ROMA Libera Univ. degli Studi Free University Maria IT0066 'Maria SS.Assunta' - LUMSA IT ITI4 ITI43 ROMA SS.Assunta (LUMSA) - Roma Università 'Campus Bio- Campus Bio-Medico IT0067 IT ITI4 ITI43 ROMA Medico' di ROMA University Università degli Studi di Sapienza University of IT0068 IT ITI4 ITI43 ROMA ROMA 'La Sapienza' Rome Università degli Studi di University of Rome 'Foro IT0069 IT ITI4 ITI43 ROMA ROMA 'Foro Italico' Italico' Università degli Studi di University of Rome Tor IT0070 IT ITI4 ITI43 ROMA ROMA 'Tor Vergata' Vergata Università degli Studi European University of IT0071 IT ITI4 ITI43 ROMA EUROPEA di ROMA Rome Università Telematica IT0072 Marconi University IT ITI4 ITI43 ROMA GUGLIELMO MARCONI Università degli Studi di IT0077 University of Salerno IT ITF3 ITF35 FISCIANO SALERNO Università degli Studi di IT0078 University of Sassari IT ITG2 ITG25 SASSARI SASSARI Università degli Studi di IT0079 University of Siena IT ITI1 ITI19 SIENA SIENA Università per Stranieri di University for Foreigners IT0080 IT ITI1 ITI19 SIENA SIENA of Siena Università degli Studi di IT0081 University of Teramo IT ITF1 ITF12 TERAMO TERAMO IT0082 Politecnico di TORINO Politecnico di Torino IT ITC1 ITC11 TORINO Università degli Studi di IT0083 University of Turin IT ITC1 ITC11 TORINO TORINO Università degli Studi di IT0085 University of Trento IT ITH2 ITH20 TRENTO TRENTO Scuola Internazionale International School for IT0086 Superiore di Studi Avanzati Advanced Studies IT ITH4 ITH44 TRIESTE di TRIESTE (SISSA) Università degli Studi di IT0087 University of Trieste IT ITH4 ITH44 TRIESTE TRIESTE Università degli Studi di IT0088 University of Udine IT ITH4 ITH42 UDINE UDINE Università degli Studi di University of Urbino Carlo IT0089 IT ITI3 ITI31 URBINO URBINO 'Carlo BO' Bo Università degli Studi IT0090 University of Insubria IT ITC4 ITC41 VARESE INSUBRIA Varese-Como Università 'Cà Foscari' Ca’ Foscari University of IT0091 IT ITH3 ITH35 VENEZIA VENEZIA Venice

IT0092 Università IUAV di VENEZIA University IUAV of Venice IT ITH3 ITH35 VENEZIA

Università degli Studi del University of Piemonte IT0093 PIEMONTE ORIENTALE Orientale 'Amedeo IT ITC1 ITC12 VERCELLI 'Amedeo Avogadro'-Vercelli Avogadro'

Università degli Studi di IT0094 University of Verona IT ITH3 ITH31 VERONA VERONA Università degli Studi della IT0095 University of Tuscia IT ITI4 ITI41 VITERBO TUSCIA University of LI0001 Universität Liechtenstein LI LI00 LI000 Vaduz Liechtenstein LT0001 Vilniaus universitetas Vilnius University LT LT00 LT00A Vilnius Vilniaus Gedimino technikos Vilnius Gediminas LT0002 LT LT00 LT00A Vilnius universitetas Technical University Lietuvos edukologijos Lithuanian University of LT0003 LT LT00 LT00A Vilnius universitetas Education Vilnius Academy of Fine LT0004 Vilniaus dailes akademija LT LT00 LT00A Vilnius Arts Lietuvos muzikos ir teatro Lithuanian Academy of LT0005 LT LT00 LT00A Vilnius akademija Music and Theatre Mykolo Romerio Mykolas Romeris LT0006 LT LT00 LT00A Vilnius universitetas University The General Jonas Generolo Jono Žemaicio LT0007 Zemaitis Military Academy LT LT00 LT00A Vilnius Lietuvos karo akademija of Lithuania Kauno technologijos Kaunas University of LT0008 LT LT00 LT002 Kaunas universitetas Technology Vytauto Didžiojo Vytautas Magnus LT0009 LT LT00 LT002 Kaunas universitetas University Aleksandro Stulginskio Aleksandras Stulginskis Akademijos LT0012 LT LT00 LT002 universitetas University mstl. Lietuvos sporto Lithuanian Sports LT0013 LT LT00 LT002 Kaunas universitetas University LT0014 Klaipedos universitetas Klaipeda University LT LT00 LT003 Klaipeda LT0015 Šiauliu universitetas Siauliai University LT LT00 LT006 Siauliai ISM University of ISM Vadybos ir ekonomikos LT0018 Management and LT LT00 LT00A Vilnius universitetas, UAB Economics, JSC European Humanities Viešoji istaiga 'Europos LT0042 University, Public LT LT00 LT00A Vilnius Humanitarinis Universitetas' Institution Lietuvos sveikatos mokslu Lithuanian University of LT0047 LT LT00 LT002 Kaunas universitetas Health Sciences LU0001 Université du Luxembourg University of Luxembourg LU LU00 LU000 Luxembourg LV0001 Latvijas Universitate University of Latvia LV LV00 LV006 Riga

LV0002 Rigas Tehniska universitate Riga Technical University LV LV00 LV006 Riga

LV0003 Rigas Stradina universitate Riga Stradinš University LV LV00 LV006 Riga

Latvijas Lauksaimniecibas Latvia University of LV0004 LV LV00 LV009 Jelgava universitate Agriculture LV0005 Daugavpils Universitate Daugavpils University LV LV00 LV005 Daugavpils LV0006 Liepajas Universitate Liepaja University LV LV00 LV003 Liepaja Latvian Academy of LV0007 Latvijas Kulturas akademija LV LV00 LV006 Riga Culture

LV0008 Latvijas Makslas akademija Art Academy of Latvia LV LV00 LV006 Riga

Jazepa Vitola Latvijas Jazeps Vitols Latvian LV0009 LV LV00 LV006 Riga Muzikas akademija Academy of Music Latvijas Sporta Pedagogijas Latvian Academy of Sport LV0010 LV LV00 LV006 Riga akademija Education Rigas Pedagogijas un Riga Teacher Training and LV0012 izglitibas vadibas Educational Management LV LV00 LV006 Riga augstskola Academy Rezekne Higher Education LV0013 Rezeknes augstskola LV LV00 LV005 Rezekne Institution Ventspils University LV0015 Ventspils augstskola LV LV00 LV003 Ventspils College Vidzeme University LV0016 Vidzemes augstskola LV LV00 LV008 Valmiera College

BA School of Business LV0017 Banku augstskola LV LV00 LV006 Riga and Finance

Rigas Starptautiska Riga International School LV0020 ekonomikas un biznesa of Economics and LV LV00 LV006 Riga administracijas augstskola Business Administration

School of Business LV0021 Biznesa augstskola Turiba LV LV00 LV006 Riga Administration Turiba Baltijas Starptautiska Baltic International LV0025 LV LV00 LV006 Riga akademija Academy Information Systems Informacijas sistemu LV0028 Management Institute LV LV00 LV006 Riga menedžmenta augstskola (ISMA) Transport and Transporta un sakaru LV0031 Telecommunication LV LV00 LV006 Riga instituts Institute University of Malta MT0001 University of Malta MT MT00 MT001 MSIDA (L'Universita` ta` Malta)

NL0001 Universiteit van Amsterdam University of Amsterdam NL NL32 NL326 Amsterdam

Vrije Universiteit NL0002 VU University Amsterdam NL NL32 NL326 Amsterdam Amsterdam Technische Universiteit Delft University of NL0003 NL NL33 NL333 Delft Delft Technology Technische Universiteit Eindhoven University of NL0004 NL NL41 NL414 Eindhoven Eindhoven Technology NL0005 Universiteit Twente University of Twente NL NL21 NL213 Enschede

NL0006 Rijksuniversiteit Groningen University of Groningen NL NL11 NL113 Groningen

NL0007 Universiteit Leiden Leiden University NL NL33 NL337 Leiden NL0008 Universiteit Maastricht Maastricht University NL NL42 NL423 Maastricht Radboud Universiteit Radboud University NL0009 NL NL22 NL226 Nijmegen Nijmegen Nijmegen Erasmus Universiteit Erasmus University NL0010 NL NL33 NL339 Rotterdam Rotterdam Rotterdam NL0011 Universiteit van Tilburg Tilburg University NL NL41 NL412 Tilburg NL0012 Universiteit Utrecht Utrecht University NL NL31 NL310 Utrecht Wageningen University NL0013 Wageningen Universiteit NL NL22 NL221 Wageningen and Research Centre Open University of the NL0014 Open Universiteit NL NL42 NL423 Heerlen Netherlands Universiteit voor University of Humanistic NL0015 NL NL31 NL310 Utrecht Humanistiek Studies Protestantse Theologische Protestant Theological NL0017 NL NL21 NL211 Kampen Universiteit University Theologische Universiteit Theological University of NL0018 van de Gereformeerde NL NL21 NL211 Kampen the Reformed Churches Kerken Theologische Universiteit Theological University NL0019 NL NL22 NL221 Apeldoorn Apeldoorn Apeldoorn NO0001 Universitetet i Oslo University of Oslo NO NO01 NO011 Oslo NO0002 Universitetet i Bergen University of Bergen NO NO05 NO051 Bergen University of Tromsø - Universitetet i Tromsø - NO0003 The arctic university of NO NO07 NO072 Tromsø Norges arktiske universitet Norway Norges teknisk- The Norwegian University NO0004 naturvitenskapelige of Science and NO NO06 NO061 Trondheim universitet Technology Universitetet for miljø- og Norwegian University of NO0005 NO NO01 NO012 Ås biovitenskap Life Sciences NO0006 Universitetet i Stavanger University of Stavanger NO NO04 NO043 Stavanger NO0007 Universitetet i Agder University of Agder NO NO04 NO042 Kristiansand Arkitektur- og The Oslo School of NO0008 NO NO01 NO011 Oslo designhøgskolen i Oslo Architecture and Design BI - Norwegian School of NO0009 Handelshøyskolen BI NO NO01 NO011 Oslo Management School of Mission and NO0010 Misjonshøgskolen NO NO04 NO043 Stavanger Theology

Norwegian School of NO0011 Norges Handelshøyskole Economics and Business NO NO05 NO051 Bergen administration Norwegian School of NO0012 Norges idrettshøgskole NO NO01 NO011 Oslo Sport Sciences Norwegian School of NO0013 Norges veterinærhøgskole NO NO01 NO011 Oslo Veterinary Science Norwegian Academy of NO0014 Norges musikkhøgskole NO NO01 NO011 Oslo Music Det teologiske Norwegian School of NO0015 NO NO01 NO011 Oslo menighetsfakultet Theology NO0018 Universitetet i Nordland University of Nordland NO NO07 NO071 Bodø NO0021 Høgskolen i Gjøvik Gjøvik University College NO NO02 NO022 Gjøvik Hedmark University NO0023 Høgskolen i Hedmark NO NO02 NO021 Elverum College Lillehammer University NO0024 Høgskolen i Lillehammer NO NO02 NO022 Lillehammer College Molde University College, NO0025 Høgskolen i Molde Specialized University in NO NO05 NO053 Molde Logistics Telemark University NO0033 Høgskolen i Telemark NO NO03 NO034 Porsgrunn College Vestfold University NO0034 Høgskolen i Vestfold NO NO03 NO033 Tønsberg College Oslo and Akershus Høgskolen i Oslo og NO0069 university college of NO NO01 NO011 Oslo Akershus applied sciences Uniwersytet Ekonomiczny w University of Economics in PL0001 PL PL22 PL22A KATOWICE Katowicach Katowice Akademia Górniczo- AGH University of Science PL0003 Hutnicza im. St. Staszica w PL PL21 PL213 KRAKÓW and Technology in Cracow Krakowie Akademia Humanistyczna The Pultusk School of PL0004 im. A. Gieysztora w PL PL12 PL122 PULTUSK Humanities Pułtusku Akademia im. Jana Jan Długosz University in CZESTOCHO PL0005 PL PL22 PL224 Długosza w Częstochowie Częstochowa WA Leon Kozminski Academy Akademia Leona PL0006 of Entrepreneurship and PL PL12 PL127 WARSZAWA Koźmińskiego w Warszawie Management in Warsaw Akademia Marynarki PL0007 Naval Academy in Gdynia PL PL63 PL633 GDYNIA Wojennej w Gdyni Akademia Medyczna im. Wrocław Medical PL0008 Piastów Śląskich we PL PL51 PL514 WROCLAW University Wrocławiu Uniwersytet Medyczny w Medical University of PL0009 PL PL31 PL314 LUBLIN Lublinie Lublin Gdański Uniwersytet Medical University of PL0010 PL PL63 PL633 GDANSK Medyczny Gdańsk Gdynia Maritime PL0011 Akademia Morska w Gdyni PL PL63 PL633 GDYNIA University Akademia Morska w Maritime University in PL0012 PL PL42 PL424 SZCZECIN Szczecinie Szczecin Akademia Muzyczna im. Academy of Music in PL0013 Feliksa Nowowiejskiego w PL PL61 PL613 BYDGOSZCZ Bydgoszcz Bydgoszczy Akademia Muzyczna im. The Grazyna and Kiejstut PL0014 Grażyny i Kiejstuta Bacewicz Academy of PL PL11 PL113 LÓDZ Bacewiczów w Łodzi Music in Łódź

Akademia Muzyczna im. The Ignacy Jan PL0015 Ignacego Jana Paderewski Academy of PL PL41 PL415 POZNAN Paderewskiego w Poznaniu Music in Poznań

Akademia Muzyczna im. The Karol Lipiński PL0016 Karola Lipińskiego we University of Music in PL PL51 PL514 WROCLAW Wrocławiu Wrocław Akademia Muzyczna im. The Karol Szymanowski PL0017 Karola Szymanowskiego w Academy of Music in PL PL22 PL22A KATOWICE Katowicach Katowice

Akademia Muzyczna im. Academy of Music in PL0018 Stanisława Moniuszki w PL PL63 PL633 GDANSK Gdańsk Gdańsku Akademia Muzyczna w Academy of Music in PL0019 PL PL21 PL213 KRAKÓW Krakowie Cracow Akademia Obrony National Defence PL0020 PL PL12 PL127 WARSZAWA Narodowej University in Warsaw Akademia Pedagogiki Specjalnej im. M. Academy of Special PL0021 PL PL12 PL127 WARSZAWA Grzegorzewskiej w Education in Warsaw Warszawie

Uniwersytet Przyrodniczo- University of Podlasie in PL0022 PL PL12 PL122 SIEDLCE Humanistyczny w Siedlcach Siedlce

Akademia Pomorska w Pomeranian Pedagogical PL0024 PL PL63 PL613 SLUPSK Słupsku Academy in Słupsk Akademia Sztuk Pięknych Jan Matejko Academy of PL0026 im. Jana Matejki w PL PL21 PL213 KRAKÓW Fine Arts in Cracow Krakowie Akademia Sztuk Pięknych Academy of Fine Arts in PL0027 im. Władysława PL PL11 PL113 LÓDZ Łódź Strzemińskiego w Łodzi Akademia Sztuk Pięknych w Academy of Fine Arts in PL0028 PL PL63 PL633 GDANSK Gdańsku Gdańsk Akademia Sztuk Pięknych w Academy of Fine Arts in PL0029 PL PL22 PL22A KATOWICE Katowicach Katowice Uniwersytet Artystyczny w University of Arts in PL0030 PL PL41 PL415 POZNAN Poznaniu Poznań Akademia Sztuk Pięknych w Academy of Fine Arts in PL0031 PL PL12 PL127 WARSZAWA Warszawie Warsaw Akademia Sztuk Pięknych Academy of Fine Arts in PL0032 PL PL51 PL514 WROCLAW we Wrocławiu Wrocław Akademia Teatralna im. Theatre Academy in PL0033 Aleksandra Zelwerowicza w PL PL12 PL127 WARSZAWA Warsaw Warszawie Akademia Techniczno- BIELSKO- PL0034 Humanistyczna w Bielsku- Unversity of Bielsko-Biała PL PL22 PL225 BIALA Białej Akademia Wychowania Academy of Physical Fizycznego i Sportu im. PL0035 Education and Sport in PL PL63 PL633 GDANSK Jędrzeja Śniadeckiego w Gdańsk Gdańsku Akademia Wychowania Academy of Physical PL0036 Fizycznego im. Bronisława PL PL21 PL213 KRAKÓW Education in Cracow Czecha w Krakowie The Eugeniusz Piasecki Akademia Wychowania University School of PL0037 Fizycznego im. Eugeniusza PL PL41 PL415 POZNAN Physical Education in Piaseckiego w Poznaniu Poznań Akademia Wychowania Academy of Physical PL0038 Fizycznego im. Jerzego PL PL22 PL22A KATOWICE Education in Katowice Kukuczki w Katowicach Akademia Wychowania Academy of Physical PL0039 Fizycznego im. Józefa PL PL12 PL127 WARSZAWA Education in Warsaw Piłsudskiego w Warszawie Akademia Wychowania Academy of Physical PL0040 PL PL51 PL514 WROCLAW Fizycznego we Wrocławiu Education in Wrocław Chrześcijańska Akademia Christian Theological PL0046 PL PL12 PL127 WARSZAWA Teologiczna w Warszawie Academy in Warsaw Collegium Civitas w Collegium Civitas in PL0047 PL PL12 PL127 WARSZAWA Warszawie Warsaw Dolnośląska Szkoła Wyższa PL0049 University of Lower Silesia PL PL51 PL514 WROCLAW we Wrocławiu

Górnośląska Wyższa Szkoła Katowice School of PL0062 Handlowa im. Wojciecha PL PL22 PL22A KATOWICE Economics Korfantego w Katowicach

Katolicki Uniwersytet Catolic University in PL0069 PL PL31 PL314 LUBLIN Lubelski Jana Pawła II Lublin

Krakowska Szkoła Wyższa Andrzej Frycz Modrzewski PL0072 im. A. Frycza PL PL21 PL213 KRAKÓW Cracow College Modrzewskiego w Krakowie

Państwowa Wyższa Szkoła The National Film, Filmowa, Telewizyjna i PL0091 Television and Theatre PL PL11 PL113 LÓDZ Teatralna im. Leona School in Łódź Schillera w Łodzi

Państwowa Wyższa Szkoła The National Theatre High PL0093 Teatralna im. Ludwika PL PL21 PL213 KRAKÓW School in Cracow Solskiego w Krakowie

Uniwersytet Papieski Jana The Pontifical University PL0125 PL PL21 PL213 KRAKÓW Pawła II w Krakowie of John Paul II in Cracow Papieski Wydział Jan Chrzciciel Pontifical PL0126 Teologiczny w Warszawie Faculty of Theology in PL PL12 PL127 WARSZAWA im. św. Jana Chrzciciela Warsaw Papieski Wydział Pontifical Faculty of PL0128 PL PL51 PL514 WROCLAW Teologiczny we Wrocławiu Theology in Wrocław Białystok Technical PL0133 Politechnika Białostocka PL PL34 PL343 BIALYSTOK University Technical University of CZESTOCHO PL0134 Politechnika Częstochowska PL PL22 PL224 Częstochowa WA Gdańsk University of PL0135 Politechnika Gdańska PL PL63 PL633 GDANSK Technology Politechnika Koszalińska w Technical University of PL0136 PL PL42 PL422 KOSZALIN Koszalinie Koszalin Politechnika Krakowska im. University of Technology PL0137 PL PL21 PL213 KRAKÓW T. Kościuszki in Cracow Lublin University of PL0138 Politechnika Lubelska PL PL31 PL314 LUBLIN Technology Technical University of PL0139 Politechnika Łódzka PL PL11 PL113 LÓDZ Łódź Politechnika Opolska w Opole University of PL0140 PL PL52 PL522 OPOLE Opolu Technology Poznań University of PL0141 Politechnika Poznańska PL PL41 PL415 POZNAN Technology

Politechnika Radomska im. University of Technology PL0142 PL PL12 PL128 RADOM K. Pułaskiego w Radomiu in Radom

Politechnika Rzeszowska Rzeszów University of PL0143 PL PL32 PL325 RZESZÓW im. I. Łukasiewicza Technology Politechnika Śląska w Silesian University of PL0145 PL PL22 PL229 GLIWICE Gliwicach Technology in Gliwic Politechnika Świętokrzyska Kielce University of PL0146 PL PL33 PL331 KIELCE w Kielcach Technology Warsaw University of PL0147 Politechnika Warszawska PL PL12 PL127 WARSZAWA Technology Wrocław University of PL0148 Politechnika Wrocławska PL PL51 PL514 WROCLAW Technology Polsko-Japońska Wyższa Polish-Japanese Institute Szkoła Nowych Technik PL0150 of Information Technology PL PL12 PL127 WARSZAWA Komputerowych w in Warsaw Warszawie Pomorski Uniwersytet Pomeranian Medical PL0151 PL PL42 PL424 SZCZECIN Medyczny University in Szczecin Społeczna Wyższa Szkoła Academy of Management PL0166 Przedsiębiorczości i PL PL11 PL113 LÓDZ in Łódź Zarządzania w Łodzi Szkoła Główna Gospodarstwa Wiejskiego - Warsaw Agricultural PL0169 PL PL12 PL127 WARSZAWA Akademia Rolnicza w University Warszawie Szkoła Główna Handlowa w Warsaw School of PL0170 PL PL12 PL127 WARSZAWA Warszawie Economics

Szkoła Wyższa Psychologii Warsaw School of Social PL0175 PL PL12 PL127 WARSZAWA Społecznej w Warszawie Psychology

Śląski Uniwersytet Medical University of PL0180 PL PL22 PL22A KATOWICE Medyczny w Katowicach Silesia in Katowice Uniwersytet Ekonomiczny w Cracow University of PL0185 PL PL21 PL213 KRAKÓW Krakowie Economics Uniwersytet Ekonomiczny w The Poznań University of PL0186 PL PL41 PL415 POZNAN Poznaniu Economics Uniwersytet Ekonomiczny Wrocław University of PL0187 PL PL51 PL514 WROCLAW we Wrocławiu Economics PL0188 Uniwersytet Gdański University of Gdańsk PL PL63 PL633 GDANSK

Uniwersytet Jana Jan Kochanowski PL0189 PL PL33 PL331 KIELCE Kochanowskiego w Kielcach University in Kielce

Uniwersytet im. Adama Adam Mickiewicz PL0190 PL PL41 PL415 POZNAN Mickiewicza w Poznaniu University in Poznań Uniwersytet Jagielloński w Jagiellonian University in PL0191 PL PL21 PL213 KRAKÓW Krakowie Cracow Uniwersytet Kardynała Cardinal Stefan PL0192 Stefana Wyszyńskiego w Wyszyński University in PL PL12 PL127 WARSZAWA Warszawie Warsaw Uniwersytet Kazimierza Kazimierz Wielki PL0193 PL PL61 PL613 BYDGOSZCZ Wielkiego w Bydgoszczy University in Bydgoszcz PL0194 Uniwersytet Łódzki University of Łódź PL PL11 PL113 LÓDZ Uniwersytet Marii Curie- Maria Curie-Skłodowska PL0195 PL PL31 PL314 LUBLIN Skłodowskiej w Lublinie University in Lublin Uniwersytet Medyczny im. Poznań University of PL0196 K. Marcinkowskiego w PL PL41 PL415 POZNAN Medical Sciences Poznaniu Uniwersytet Medyczny w Medical University of PL0197 PL PL34 PL343 BIALYSTOK Białymstoku Białystok Uniwersytet Medyczny w PL0198 Medical University of Łódź PL PL11 PL113 LÓDZ Łodzi Uniwersytet Mikołaja Nicolaus Copernicus PL0199 PL PL61 PL613 TORUN Kopernika w Toruniu University in Toruń Uniwersytet Muzyczny im. The Frederic Chopin PL0200 Fryderyka Chopina w Academy of Music in PL PL12 PL127 WARSZAWA Warszawie Warsaw PL0201 Uniwersytet Opolski University of Opole PL PL52 PL522 OPOLE Uniwersytet Pedagogiczny Pedagogical University in PL0202 im. Komisji Edukacji PL PL21 PL213 KRAKÓW Cracow Narodowej w Krakowie Uniwersytet Przyrodniczy w Agricultural University of PL0203 PL PL31 PL314 LUBLIN Lublinie Lublin Uniwersytet Przyrodniczy w University of Agricultural PL0204 PL PL41 PL415 POZNAN Poznaniu in Poznań Uniwersytet Przyrodniczy Agricultural University of PL0205 PL PL51 PL514 WROCLAW we Wrocławiu Wrocław Uniwersytet Rolniczy im. H. Agricultural University of PL0206 PL PL21 PL213 KRAKÓW Kołłątaja w Krakowie Cracow PL0207 Uniwersytet Rzeszowski University of Rzeszów PL PL32 PL325 RZESZÓW PL0208 Uniwersytet Szczeciński University of Szczecin PL PL42 PL424 SZCZECIN Uniwersytet Śląski w University of Silesia in PL0209 PL PL22 PL22A KATOWICE Katowicach Katowice

Uniwersytet University of Technology Technologiczno- PL0210 and Agriculture in PL PL61 PL613 BYDGOSZCZ Przyrodniczy im. J. J. Bydgoszcz Śniadeckich w Bydgoszczy

PL0211 Uniwersytet w Białymstoku University of Białystok PL PL34 PL343 BIALYSTOK

Uniwersytet Warmińsko- University of Warmia and PL0212 PL PL62 PL622 OLSZTYN Mazurski w Olsztynie Mazury in Olsztyn PL0213 Uniwersytet Warszawski Warsaw University PL PL12 PL127 WARSZAWA PL0214 Uniwersytet Wrocławski Wrocław University PL PL51 PL514 WROCLAW ZIELONA PL0215 Uniwersytet Zielonogórski University of Zielona Góra PL PL43 PL432 GÓRA Warszawski Uniwersytet Warsaw Medical PL0221 PL PL12 PL127 WARSZAWA Medyczny University Wojskowa Akademia Military University of PL0226 PL PL12 PL127 WARSZAWA Techniczna Technology in Warsaw

Warsaw School of Akademia Finansów i PL0276 Tourism and Hospitality PL PL12 PL127 WARSZAWA Biznesu Vistula Management Jesuit University of Akademia Ignatianum w PL0287 Philosophy and Education PL PL21 PL213 KRAKÓW Krakowie Ignatianum in Krakow Uczelnia Łazarskiego w PL0307 Łazarski University PL PL12 PL127 WARSZAWA Warszawie Akademia Humanistyczno- University of Humanities PL0320 PL PL11 PL113 LÓDZ Ekonomiczna w Łodzi and Economics in Lodz

Wyższa Szkoła Oficerska Sił Air Forces Miilitary PL0375 PL PL31 PL315 DEBLIN Powietrznych w Dęblinie Academy in Dęblin

General Tadeusz Wyższa Szkoła Oficerska Kościuszko Military PL0376 Wojsk Lądowych im. T. PL PL51 PL514 WROCLAW Academy of Land Forces Kościuszki we Wrocławiu in Wrocław School of Public Wyższa Szkoła Prawa i PL0385 Administration and PL PL32 PL324 PRZEMYSL Administracji w Przemyślu Management in Przemyśl

Zachodniopomorski West Pomeranian PL0459 Uniwersytet Technologiczny University of Technology, PL PL42 PL424 SZCZECIN w Szczecinie Szczecin

Ponta PT0001 Universidade dos Açores University of Azores PT PT20 PT200 Delgada PT0002 Universidade do Algarve University of Algarve PT PT15 PT150 Faro PT0003 Universidade de Aveiro University of Aveiro PT PT16 PT161 Aveiro Universidade da Beira University of Beira PT0004 PT PT16 PT16A Covilhã Interior Interior PT0005 Universidade de Coimbra University of Coimbra PT PT16 PT162 Coimbra PT0006 Universidade de Évora University of Évora PT PT18 PT183 Évora PT0007 Universidade de Lisboa University of Lisbon PT PT17 PT171 Lisboa Universidade Nova de PT0008 New University of Lisbon PT PT17 PT171 Lisboa Lisboa Universidade Técnica de Technical University of PT0009 PT PT17 PT171 Lisboa Lisboa Lisbon PT0010 Universidade da Madeira University of Madeira PT PT30 PT300 Funchal PT0011 Universidade do Minho University of Minho PT PT11 PT112 Braga PT0012 Universidade do Porto University of Porto PT PT11 PT114 Porto Universidade de Trás-os- University of Trás-os- PT0013 PT PT11 PT117 Vila Real Montes e Alto Douro Montes and Alto Douro ISCTE - Instituto PT0014 Lisbon University Institute PT PT17 PT171 Lisboa Universitário de Lisboa PT0015 Universidade Aberta Universidade Aberta PT PT17 PT171 Lisboa Instituto Politécnico de Polytechnic Institute of PT0017 PT PT11 PT118 Bragança Bragança Bragança Instituto Politécnico de Polytechnic Institute of PT0020 PT PT16 PT162 Coimbra Coimbra Coimbra Instituto Politécnico de Polytechnic Institute of PT0022 PT PT16 PT163 Leiria Leiria Leiria Instituto Politécnico de Lisbon Polytechnic PT0023 PT PT17 PT171 Lisboa Lisboa Institute Instituto Politécnico do PT0025 Porto Polytechnic Institute PT PT11 PT114 Porto Porto Instituto Politécnico de Polytechnic Institute of PT0027 PT PT17 PT172 Setúbal Setúbal Setúbal Universidade Católica Catholic University of PT0041 PT PT17 PT171 Lisboa Portuguesa Portugal ISPA-Instituto ISPA-Instituto Universitário Universitário de Ciências PT0067 de Ciências Psicológicas, PT PT17 PT171 Lisboa Psicológicas, Sociais e da Sociais e da Vida Vida Universidade Autónoma de Universidade Autónoma PT0073 PT PT17 PT171 Lisboa Lisboa Luís de Camões de Lisboa Luís de Camões Universidade Fernando Fernando Pessoa PT0076 PT PT11 PT114 Porto Pessoa University PT0077 Universidade Lusíada University Lusíada PT PT17 PT171 Lisboa Universidade Lusíada PT0078 University Lusíada - Porto PT PT11 PT114 Porto (Porto)

Lusophone University of Universidade Lusófona de PT0080 Humanities and PT PT17 PT171 Lisboa Humanidades e Tecnologias Technologies Universidade Lusófona do Lusophone University of PT0081 PT PT11 PT114 Porto Porto Porto

Universidade Portucalense Universidade Portucalense PT0082 PT PT11 PT114 Porto Infante D. Henrique Infante D. Henrique

UNIVERSITATEA " 1 1 DECEMBRIE 1918 RO001 DECEMBRIE 1918 " DIN UNIVERSITY OF ALBA RO RO12 RO121 Alba Iulia ALBA IULIA IULIA UNIVERSITATEA DIN RO003 University of Pitesti RO RO31 RO311 Pitesti PITESTI - ARGES

UNIVERSITATEA "VASILE RO004 University of Bacau RO RO21 RO211 Bacau ALECSANDRI" DIN BACAU

UNIVERSITATEA DIN RO005 University of Oradea RO RO11 RO111 Oradea ORADEA - BIHOR UNIVERSITATEA " Transilvania University of RO006 TRANSILVANIA " DIN RO RO12 RO122 Brasov Brasov BRASOV UNIVERSITATEA TEHNICA Tehnical University of RO009 RO RO11 RO113 Cluj-Napoca DIN CLUJ-NAPOCA Cluj Napoca UNIVERSITATEA " BABES - RO010 BOLYAI " DIN CLUJ- Babes-Bolyai University RO RO11 RO113 Cluj-Napoca NAPOCA UNIVERSITATEA DE University of Agricultural STIINTE AGRICOLE SI MED. RO011 Sciences and Veterinary RO RO11 RO113 Cluj-Napoca VETERINARA DIN CLUJ- Medicine of Cluj-Napoca NAPOCA UNIVERSITATEA DE MED. University of Medicine and SI FARM. " IULIU RO012 Pharmacy Iuliu Hateganu RO RO11 RO113 Cluj-Napoca HATIEGANU " DIN CLUJ- of Cluj Napoca NAPOCA UNIVERSITATEA DE ARTA University of Art and SI DESIGN " ION RO013 Design Ion Andreescu of RO RO11 RO113 Cluj-Napoca ANDREESCU " DIN CLUJ- Cluj Napoca NAPOCA ACADEMIA DE MUZICA " "Gheorghe Dima" Music RO014 GHEORGHE DIMA " DIN RO RO11 RO113 Cluj-Napoca Academy of Cluj-Napoca CLUJ-NAPOCA

UNIVERSITATEA " OVIDIUS Ovidius University of RO016 RO RO22 RO223 Constanta " DIN CONSTANTA Constanta

UNIVERSITATEA " VALAHIA Valahia University of RO018 RO RO31 RO313 Targoviste " DIN TARGOVISTE Targoviste

UNIVERSITATEA DIN RO019 University of Craiova RO RO41 RO411 Craiova CRAIOVA - DOLJ UNIVERSITATEA DE University of Medicine and RO020 MEDICINA SI FARMACIE RO RO41 RO411 Craiova Pharmacy of Craiova CRAIOVA - DOLJ UNIVERSITATEA " "Dunarea de Jos" RO021 DUNAREA DE JOS " DIN RO RO22 RO224 Galati University of Galati GALATI UNIVERSITATEA TEHNICA The "Gheorghe Asachi" RO024 "GHEORGHE ASACHI" DIN Technical University of RO RO21 RO213 Iași IASI Iasi UNIVERSITATEA " The Alexandru Ioan Cuza RO025 ALEXANDRU IOAN CUZA " RO RO21 RO213 Iași University of Iași DIN IASI UNIVERSITATEA DE “Grigore. T. Popa” MEDICINA SI FARMACIE " RO027 University of Medicine and RO RO21 RO213 Iași GRIGORE T. POPA " DIN Pharmacy Iasi IASI UNIVERSITATEA DE ARTE " UNIVERSITY OF ARTS RO028 GEORGE ENESCU " DIN RO RO21 RO213 Iași "GEORGE ENESCU" IAŞI IASI UNIVERSITATEA DE NORD North University of Baia Baia Mare- RO029 DIN BAIA MARE - RO RO11 RO114 Mare-Maramures Maramures MARAMURES

UNIVERSITATEA "PETRU “Petru Maior” University RO030 RO RO12 RO125 Târgu-Mures MAIOR" DIN TARGU-MURES of Targu Mures

UNIVERSITATEA DE University of Medicine and RO031 MEDICINA SI FARMACIE RO RO12 RO125 Târgu-Mures Pharmacy of Targu Mures DIN TARGU MURES UNIVERSITATEA DE ARTA Târgu-Mures University of RO032 TEATRALA DIN TARGU RO RO12 RO125 Târgu-Mures Theatre MURES UNIVERSITATEA " PETROL Petroleum-Gas University RO033 SI GAZE " DIN PLOIESTI - RO RO31 RO316 Ploieşti of Ploieşti PRAHOVA

UNIVERSITATEA " LUCIAN "Lucian Blaga" University RO034 RO RO12 RO126 Sibiu BLAGA " DIN SIBIU of Sibiu

UNIVERSITATEA " STEFAN "Stefan Cel Mare" RO036 RO RO21 RO215 Suceava CEL MARE " DIN SUCEAVA University of Suceava

UNIVERSITATEA " “Politehnica” University of RO037 POLITEHNICA " DIN RO RO42 RO424 Timisoara Timisoara TIMISOARA UNIVERSITATEA DE VEST West University of RO038 RO RO42 RO424 Timisoara DIN TIMISOARA Timişoara BANAT UNIVERSITY OF UNIV. DE STIINTE AGRICULTURAL AGRICOLE SI MED. VETER. RO039 SCIENCES AND RO RO42 RO424 Timisoara A BANATULUI DIN VETERINARY MEDICINE TIMISOARA TIMISOARA UNIVERSITATEA DE University of Medicine and MEDICINA SI FARMACIE RO040 Pharmacy "Victor Babes" RO RO42 RO424 Timisoara "VICTOR BABES" DIN Timisoara TIMISOARA UNIVERSITATEA " University "Politehnica" of RO041 POLITEHNICA " DIN RO RO32 RO321 Bucharest Bucharest BUCURESTI INSTITUTUL DE Ion Mincu” University of RO042 ARHITECTURA " ION MINCU RO RO32 RO321 Bucharest Architecture & Urbanism " DIN BUCURESTI UNIVERSITATEA TEHNICA Technical University of RO043 DE CONSTRUCTII DIN Civil Engineering RO RO32 RO321 Bucharest BUCURESTI Bucharest UNIVERSITATEA RO044 University of Bucharest RO RO32 RO321 Bucharest BUCURESTI ACADEMIA NATIONALA DE National Academy of RO045 EDUCATIE FIZICA SI Physical Education and RO RO32 RO321 Bucharest SPORT DIN BUCURESTI Sport of Bucharest UNIVERSITATEA DE Academy of Economic RO046 STIINTE AGRON. SI MED. RO RO32 RO321 Bucharest Studies of Bucharest VETER. DIN BUCURESTI ACADEMIA DE STUDII The Bucharest Academy RO047 ECONOMICE DIN RO RO32 RO321 Bucharest of Economic Studies BUCURESTI UNIVERSITATEA DE University of Medicine and MEDICINA SI FARMACIE " RO048 Pharmacy "Carol Davila" RO RO32 RO321 Bucharest CAROL DAVILA " DIN of Bucharest BUCURESTI UNIVERSITATEA RO050 NATIONALA DE ARTE DIN National University of Arts RO RO32 RO321 Bucharest BUCURESTI ACADEMIA DE POLITIE " "Alexandru Ioan Cuza" RO053 ALEXANDRU IOAN CUZA " - Police Academy of RO RO32 RO321 Bucharest BUCURESTI Bucharest ACADEMIA TEHNICA Military Technical RO054 RO RO32 RO321 Bucharest MILITARA BUCURESTI Academy of Bucharest UNIVERSITATEA National Defence RO055 NATIONALA DE APARARE RO RO32 RO321 Bucharest University "Carol I" "CAROL I" BUCURESTI “VASILE GOLDIŞ” UNIVERSITATEA DE VEST " RO056 WESTERN UNIVERSITY RO RO42 RO421 Arad VASILE GOLDIS " ARAD OF ARAD

UNIVERSITATEA "NICOLAE "Nicolae Titulescu" RO071 RO RO32 RO321 Bucharest TITULESCU" BUCURESTI University of Bucharest

Universitatea Crestina ' 'Dimitrie Cantemir' RO0073 Dimitrie Cantemir ' Christian University of RO RO32 RO321 Bucharest Bucuresti Bucharest SE0001 Uppsala universitet Uppsala University SE SE12 SE121 Uppsala SE0002 Lunds universitet Lund University SE SE22 SE224 Lund SE0003 Göteborgs universitet Göteborg University SE SE23 SE232 Göteborg SE0004 Stockholms universitet Stockholm University SE SE11 SE110 Stockholm SE0005 Umeå universitet Umeå university SE SE33 SE331 Umeå SE0006 Linköpings universitet Linköping University SE SE12 SE123 Lindköping SE0007 Karolinska institutet Karolinska Institute SE SE11 SE110 Stockholm Kungliga Tekniska Royal Institute of SE0008 SE SE11 SE110 Stockholm högskolan Technology Chalmers University of SE0009 Chalmers tekniska högskola SE SE23 SE232 Göteborg Technology Luleå University of SE0010 Luleå tekniska universitet SE SE33 SE332 Luleå Technology Handelshögskolan i Stockholm School of SE0011 SE SE11 SE110 Stockholm Stockholm Economics Sveriges Swedish University of SE0012 SE SE12 SE121 Uppsala lantbruksuniversitet Agricultural Sciences SE0013 Karlstads universitet Karlstad University SE SE31 SE311 Karlstad SE0015 Örebro universitet Örebro University SE SE12 SE124 Örebro SE0016 Mittuniversitetet Mid-Sweden University SE SE32 SE321 Östersund Blekinge Institute of SE0017 Blekinge tekniska högskola SE SE22 SE221 Karlskrona Technology Jönköping University SE0018 Högskolan i Jönköping SE SE21 SE211 Jönköping College SE0020 Malmö högskola Malmö University SE SE22 SE224 Malmö Mälardalen University SE0021 Mälardalens högskola SE SE12 SE125 Eskilstuna College Gymnastik- och Swedish School of Sport SE0022 SE SE11 SE110 Stockholm idrottshögskolan and Health Sciences SE0023 Högskolan i Borås University of Borås SE SE23 SE232 Borås SE0024 Högskolan Dalarna Dalarna University SE SE31 SE312 Falun SE0026 Högskolan i Gävle University of Gävle SE SE31 SE313 Gävle SE0027 Högskolan i Halmstad Halmstad University SE SE23 SE231 Halmstad SE0029 Högskolan i Skövde University of Skövde SE SE23 SE232 Skövde SE0030 Högskolan Väst University West SE SE23 SE232 Trollhättan SE0031 Södertörns högskola Södertörn University SE SE11 SE110 Huddinge SE0050 Linnéuniversitetet Linnaeus University SE SE21 SE212 Växjö SI0001 Univerza v Ljubljani University of Ljubljana SI SI02 SI021 Ljubiana SI0002 Univerza v Mariboru University of Maribor SI SI01 SI012 Maribor SI0003 Univerza na Primorskem University of Primorska SI SI02 SI024 Koper SI0004 Univerza Nova Gorica University of Nova Gorica SI SI02 SI023 Nova Gorica Univerzita Komenského V Comenius University in SK0001 SK SK01 SK010 Bratislava Bratislave Bratislava Univerzita Pavla Jozefa Pavol Jozef Šafárik SK0002 SK SK04 SK042 Košice Šafárika V Košiciach University in Košice Prešovska Univerzita V University of Prešov in SK0003 SK SK04 SK041 Prešov Prešove Prešov Univerzita Sv. Cyrila A University of St. Cyril and SK0004 SK SK02 SK021 Trnava Metoda V Trnave Methodius in Trnava Katolická Univerzita V Catholic University in SK0005 SK SK03 SK032 Ružomerok Ružomberku Ružomberok Univerzita J. Selyeho V Selye János University in SK0007 SK SK02 SK023 Komárno Komárne Komárno Vysoká Škola V SK0008 College in Sládkovičovo SK SK02 SK021 Sládkovičovo Sládkovičove

SK0009 Paneurópska vysoká škola Pan-European University SK SK01 SK010 Bratislava

Slovenská Zdravotnícka Slovak Medical University SK0010 SK SK01 SK010 Bratislava Univerzita V Bratislave in Bratislava

Vysoká Škola Zdravotníctva St. Elizabeth College of SK0011 A Sociálnej Práce Sv. Health and Social Work in SK SK01 SK010 Bratislava Alžbety V Bratislave, N.O. Bratislava, n. o.

Univerzita Veterinárskeho University of Veterinary SK0012 SK SK04 SK042 Košice Lekárstva V Košiciach Medicine in Košice Univerzita Konštantína University of Constantinus SK0013 SK SK02 SK023 Nitra Filozofa V Nitre the Philosopher in Nitra Univerzita Mateja Bela V Matej Bel University in Banská SK0014 SK SK03 SK032 Banskej Bystrici Banská Bystrica Bystrica Trnavská Univerzita V University of Trnava in SK0015 SK SK02 SK021 Trnava Trnave Trnava Slovenská Technická Slovak University of SK0016 SK SK01 SK010 Bratislava Univerzita V Bratislave Technology in Bratislava Technická Univerzita V Technical University of SK0017 SK SK04 SK042 Košice Košiciach Košice University of Žilina in SK0019 Žilinská Univerzita V Žiline SK SK03 SK031 Žilina Žilina Trenčianska Univerzita Alexander Dubček SK0020 Alexandra Dubčeka V University of Trenčín in SK SK02 SK022 Trenčín Trenčíne Trenčín Ekonomická Univerzita V University of Economics in SK0021 SK SK01 SK010 Bratislava Bratislave Bratislava Vysoká Škola Manažmentu College of Management in SK0023 SK SK02 SK022 Trenčín V Trenčíne Trenčín Stredoeurópska Vysoká Central European College SK0024 SK SK02 SK021 Skalica Škola V Skalici in Skalica Slovenská Slovak University of SK0026 Poĺnohospodárska SK SK02 SK023 Nitra Agriculture in Nitra Univerzita V Nitre Technická Univerzita Vo University of Technology SK0027 SK SK03 SK032 Zvolen Zvolene in Zvolen Vysoká Škola Múzických Academy of Performing SK0028 SK SK01 SK010 Bratislava Umení V Bratislave Arts in Bratislava Vysoká Škola Výtvarných Academy of Fine Arts and SK0029 SK SK01 SK010 Bratislava Umení V Bratislave Design in Bratislava Akadémia Umení V Banskej Academy of Arts in Banská SK0030 SK SK03 SK032 Bystrici Banská Bystrica Bystrica Akadémia Ozbrojených Síl General Milan Rastislav Generála Milana Rastislava Štefánik Armed Forces Liptovský SK0031 SK SK03 SK031 Štefánika V Liptovskom Academy in Liptovský Mikuláš Mikuláši Mikuláš Akadémia Policajného Police Academy in SK0032 SK SK01 SK010 Bratislava Zboru V Bratislave Bratislava UK0001 Anglia Ruskin University Anglia Ruskin University UK UKH3 UKH33 Chelmsford UK0002 Aston University Aston University UK UKG3 UKG31 Birmingham UK0003 Bath Spa University Bath Spa University UK UKK1 UKK12 Bath UK0004 The University of Bath The University of Bath UK UKK1 UKK12 Bath UK0005 University of Bedfordshire University of Bedfordshire UK UKH2 UKH21 Luton UK0006 Birkbeck College Birkbeck College UK UKI1 UKI11 London Birmingham City UK0007 Birmingham City University UK UKG3 UKG31 Birmingham University The University of The University of UK0008 UK UKG3 UKG31 Birmingham Birmingham Birmingham UK0011 The University of Bolton The University of Bolton UK UKD3 UKD32 Bolton UK0013 Bournemouth University Bournemouth University UK UKK2 UKK21 Poole

UK0014 The University of Bradford The University of Bradford UK UKE4 UKE41 Bradford

UK0015 The University of Brighton The University of Brighton UK UKJ2 UKJ21 Brighton UK0016 The University of Bristol The University of Bristol UK UKK1 UKK11 Bristol UK0017 Brunel University Brunel University UK UKI2 UKI23 Uxbridge Buckinghamshire New Buckinghamshire New High UK0018 UK UKJ1 UKJ13 University University Wycombe The University of The University of UK0019 UK UKJ1 UKJ13 Buckingham Buckingham Buckingham The University of The University of UK0020 UK UKH1 UKH12 Cambridge Cambridge Cambridge The Institute of Cancer The Institute of Cancer UK0021 UK UKI1 UKI11 London Research Research Canterbury Christ Church Canterbury Christ Church UK0022 UK UKJ4 UKJ42 Canterbury University University The University of Central The University of Central UK0023 UK UKD4 UKD43 Preston Lancashire Lancashire UK0025 University of Chester University of Chester UK UKD6 UKD63 Chester

The University of The University of UK0026 UK UKJ2 UKJ24 Chichester Chichester Chichester UK0027 The City University The City University UK UKI1 UKI11 London UK0029 Courtauld Institute of Art Courtauld Institute of Art UK UKI1 UKI11 London UK0030 Coventry University Coventry University UK UKG3 UKG33 Coventry UK0031 Cranfield University Cranfield University UK UKH2 UKH25 Cranfield University for the Creative University for the Creative UK0032 UK UKJ2 UKJ23 Farnham Arts Arts UK0033 University of Cumbria University of Cumbria UK UKD1 UKD12 Carlisle UK0035 De Montfort University De Montfort University UK UKF2 UKF21 Leicester UK0036 University of Derby University of Derby UK UKF1 UKF11 Derby UK0037 University of Durham University of Durham UK UKC1 UKC14 Durham The University of East The University of East UK0038 UK UKH1 UKH13 Norwich Anglia Anglia The University of East The University of East UK0039 UK UKI1 UKI12 London London London UK0040 Edge Hill University Edge Hill University UK UKD4 UKD43 Ormskirk UK0041 The University of Essex The University of Essex UK UKH3 UKH33 Colchester UK0042 The University of Exeter The University of Exeter UK UKK4 UKK43 Exeter University of University of UK0044 UK UKK1 UKK13 Cheltenham Gloucestershire Gloucestershire UK0045 Goldsmiths College Goldsmiths College UK UKI1 UKI12 London The University of UK0046 The University of Greenwich UK UKI2 UKI21 Greenwich Greenwich UK0048 Harper Adams University Harper Adams University UK UKG2 UKG21 Newport University of UK0049 University of Hertfordshire UK UKH2 UKH23 Hatfield Hertfordshire UK0050 Heythrop College Heythrop College UK UKI1 UKI11 London The University of The University of UK0051 UK UKE4 UKE44 Huddersfield Huddersfield Huddersfield UK0052 The University of Hull The University of Hull UK UKE1 UKE11 Hull Imperial College of Imperial College of Science, UK0053 Science, Technology and UK UKI1 UKI11 London Technology and Medicine Medicine UK0054 Institute of Education Institute of Education UK UKI1 UKI11 London UK0055 The University of Keele The University of Keele UK UKG2 UKG24 Keele UK0056 The University of Kent The University of Kent UK UKJ4 UKJ42 Canterbury UK0057 King's College London King's College London UK UKI1 UKI11 London

Kingston UK0058 Kingston University Kingston University UK UKI2 UKI22 upon Thames

The University of Bailrigg, UK0059 The University of Lancaster UK UKD4 UKD43 Lancaster Lancaster Leeds Metropolitan Leeds Metropolitan UK0061 UK UKE4 UKE42 Leeds University University UK0062 The University of Leeds The University of Leeds UK UKE4 UKE42 Leeds The University of UK0064 The University of Leicester UK UKF2 UKF21 Leicester Leicester UK0065 The University of Lincoln The University of Lincoln UK UKF3 UKF30 Lincoln UK0066 Liverpool Hope University Liverpool Hope University UK UKD7 UKD72 Liverpool Liverpool John Moores Liverpool John Moores UK0067 UK UKD7 UKD72 Liverpool University University The University of UK0069 The University of Liverpool UK UKD7 UKD72 Liverpool Liverpool University of the Arts, University of the Arts, UK0070 UK UKI1 UKI11 London London London UK0071 London Business School London Business School UK UKI1 UKI11 London London Metropolitan London Metropolitan UK0073 UK UKI1 UKI12 London University University London South Bank London South Bank UK0074 UK UKI1 UKI12 London University University London School of London School of UK0075 Economics and Political Economics and Political UK UKI1 UKI11 London Science Science London School of Hygiene London School of Hygiene UK0076 UK UKI1 UKI11 London and Tropical Medicine and Tropical Medicine Loughboroug UK0077 Loughborough University Loughborough University UK UKF2 UKF22 h The Manchester The Manchester UK0078 UK UKD3 UKD31 Manchester Metropolitan University Metropolitan University

The University of The University of UK0079 UK UKD3 UKD31 Manchester Manchester Manchester UK0080 Middlesex University Middlesex University UK UKI2 UKI23 London The University of The University of UK0081 UK UKC2 UKC22 Newcastle Newcastle-upon-Tyne Newcastle-upon-Tyne The University of The University of UK0083 UK UKF2 UKF24 Northampton Northampton Northampton The University of The University of UK0084 UK UKC2 UKC22 Newcastle Northumbria at Newcastle Northumbria at Newcastle The University of The University of UK0086 UK UKF1 UKF14 Nottingham Nottingham Nottingham The Nottingham Trent The Nottingham Trent UK0087 UK UKF1 UKF14 Nottingham University University

UK0088 The Open University The Open University UK UKJ1 UKJ12 Milton Keynes

UK0089 Oxford Brookes University Oxford Brookes University UK UKJ1 UKJ14 Oxford

UK0090 The University of Oxford The University of Oxford UK UKJ1 UKJ14 Oxford The University of UK0092 The University of Plymouth UK UKK4 UKK41 Plymouth Plymouth The University of The University of UK0093 UK UKJ3 UKJ31 Portsmouth Portsmouth Portsmouth Queen Mary University of Queen Mary University of UK0094 UK UKI1 UKI12 London London London UK0096 The University of Reading The University of Reading UK UKJ1 UKJ11 Reading UK0097 Roehampton University Roehampton University UK UKI1 UKI11 London UK0099 Royal Academy of Music Royal Academy of Music UK UKI1 UKI11 London UK0101 Royal College of Art Royal College of Art UK UKI1 UKI11 London UK0102 Royal College of Music Royal College of Music UK UKI1 UKI11 London Royal Holloway and Bedford Royal Holloway and UK0103 UK UKJ2 UKJ23 Egham New College Bedford New College The Royal Veterinary The Royal Veterinary UK0105 UK UKI1 UKI11 London College College St George's Hospital St George's Hospital UK0106 UK UKI1 UKI12 London Medical School Medical School UK0108 The University of Salford The University of Salford UK UKD3 UKD31 Salford The School of Oriental and The School of Oriental UK0109 UK UKI1 UKI11 London African Studies and African Studies UK0110 The School of Pharmacy The School of Pharmacy UK UKI1 UKI11 London Sheffield Hallam UK0111 Sheffield Hallam University UK UKE3 UKE32 Sheffield University

UK0112 The University of Sheffield The University of Sheffield UK UKE3 UKE32 Sheffield

Southampton Solent Southampton Solent UK0113 UK UKJ3 UKJ32 Southampton University University The University of The University of UK0114 UK UKJ3 UKJ32 Southampton Southampton Southampton Stoke-on- UK0115 Staffordshire University Staffordshire University UK UKG2 UKG24 Trent The University of The University of UK0117 UK UKC2 UKC23 Sunderland Sunderland Sunderland UK0118 The University of Surrey The University of Surrey UK UKJ2 UKJ23 Guildford UK0119 The University of Sussex The University of Sussex UK UKJ2 UKJ21 Brighton Middlesbroug UK0120 Teesside University Teesside University UK UKC1 UKC12 h The University of West The University of West UK0121 UK UKl2 UKI23 Ealing London London UK0123 University College London University College London UK UKI1 UKI11 London UK0124 The University of Warwick The University of Warwick UK UKG3 UKG33 Coventry University of the West of University of the West of UK0125 UK UKK1 UKK11 Bristol England, Bristol England, Bristol The University of The University of UK0126 UK UKI1 UKI11 London Westminster Westminster The University of The University of UK0127 UK UKJ3 UKJ33 Winchester Winchester Winchester The University of The University of Wolverhampt UK0128 UK UKG3 UKG39 Wolverhampton Wolverhampton on The University of UK0129 The University of Worcester UK UKG1 UKG12 Worcester Worcester UK0131 York St John University York St John University UK UKE2 UKE21 York

100

UK0132 The University of York The University of York UK UKE2 UKE21 York UK0133 Aberystwyth University Aberystwyth University UK UKL1 UKL14 Aberystwyth UK0134 Bangor University Bangor University UK UKL1 UKL12 Bangor UK0135 Cardiff University Cardiff University UK UKL2 UKL22 Cardiff Cardiff Metropolitan Cardiff Metropolitan UK0136 UK UKL2 UKL22 Cardiff University University UK0137 University of Glamorgan University of Glamorgan UK UKL1 UKL15 Pontypridd UK0138 Glyndwr University Glyndwr University UK UKL2 UKL23 Wrexham The University of Wales, The University of Wales, UK0140 UK UKL2 UKL21 Newport Newport Newport Swansea Metropolitan Swansea Metropolitan UK0141 UK UKL1 UKL18 Swansea University University UK0142 Swansea University Swansea University UK UKL1 UKL18 Swansea The University of UK0144 The University of Aberdeen UK UKM5 UKM50 Aberdeen Aberdeen University of Abertay University of Abertay UK0145 UK UKM2 UKM21 Dundee Dundee Dundee UK0146 The University of Dundee The University of Dundee UK UKM2 UKM21 Dundee The University of UK0148 The University of Edinburgh UK UKM2 UKM25 Edinburgh Edinburgh Glasgow Caledonian Glasgow Caledonian UK0149 UK UKM3 UKM34 Glasgow University University UK0151 The University of Glasgow The University of Glasgow UK UKM3 UKM34 Glasgow UK0152 Heriot-Watt University Heriot-Watt University UK UKM2 UKM25 Edinburgh Edinburgh Napier UK0153 Edinburgh Napier University UK UKM2 UKM25 Edinburgh University Queen Margaret University, Queen Margaret UK0154 UK UKM2 UKM25 Edinburgh Edinburgh University, Edinburgh The Robert Gordon The Robert Gordon UK0155 UK UKM5 UKM50 Aberdeen University University The University of St The University of St UK0157 UK UKM2 UKM22 St Andrews Andrews Andrews UK0159 The University of Stirling The University of Stirling UK UKM2 UKM27 Stirling The University of The University of UK0160 UK UKM3 UKM34 Glasgow Strathclyde Strathclyde The University of the West The University of the UK0162 UK UKM3 UKM34 Paisley of Scotland West of Scotland The Queen's University of The Queen's University of UK0163 UK UKN0 UKN01 Belfast Belfast Belfast UK0166 University of Ulster University of Ulster UK UKN0 UKN04 Coleraine Trinity Laban Trinity Laban Conservatoire UK0169 Conservatoire of Music UK UKI1 UKI12 London of Music and Dance and Dance University of Wales Trinity University of Wales Trinity UK0170 UK UKL1 UKL14 Carmarthen Saint David Saint David

Appendix 2: Concordance tables

Source of this Appendix: Daraio, C., Di Costa, F., Moed, H. F. (2015).Towards Concordance Tables of Different Subject Classification Systems. A literature review with policy implications. DIAG Technical Report, Sapienza University of Rome.

Establishing a correspondence between IPC (International Patent Classification) classes and industry classification and integrating this into a concordance table is a complex task because patent classification and economic classifications are made with different aims and criteria. The IPC provides a classification for novel technical characteristics of an invention according to hierarchical criteria. Industry classification, instead, disaggregate goods into commercially and economically meaningful categories.

There have been several attempts to build concordance tables between patents and industries. The classification made by Schmookler (1966) assigns US patents to the industry where they would find the most appropriate use. The Yale Technology Concordance (Evenson et al. 1991, Putnam and Evenson 1994 Kortum and Putnam 1997) connects the IPC code to the ISIC (Canadian Standardized Industrial Classification System). The UNU-MERIT Concordance assigns IPC subclasses in 22 industrial classes based on a mix of 2- and 3-digit ISIC (International Standard Industrial Classification) codes (Verspagen, van Moergastel, and Slabbers 1994). The Organisation for Economic Cooperation and Development (OECD) Concordance (Johnston 2002), is based on the

101

correspondence of the IPC code to the ISIC. The DG ISI-SPRU-OST-concordance (Schmoch et al. 2003) assigns 625 IPC subclasses to 44 manufacturing sectors characterized by ISIC codes. The most recent proposal is Lybbert and Zolas (2014) who proposed an 'algorithmic links with probabilities' approach for joint analyses of patenting and economic activity.

The scientific non-patent references (SNPR) are references to the scientific literature, cited as prior art in patent documents. The link between cited papers and citing patent is used as an indicator of the relationship between Science and Technology.

Meyer and Persson (1998) find out that most of cited papers are not the source of the patented invention. Tacit knowledge seems to play a more decisive role than the knowledge encoded by scientific articles in the inventive process. Accordingly scientific literature plays an indirect role as a source of relevant background information.

Tijssen et al. (2000) state that rationale underlying the selection of citations remains unclear but the SNPR-based indicators represents the direct, observable links between research and technical inventions. Sternitzke (2009) claims that patent references serve as a source for qualifying novelty, while the scientific references rather qualify the inventive step such as the non- obviousness of the invention. According to Callaert et al. (2014) "scientific references in patent documents signal relatedness with the necessarily implied inventions without implying a direct, inspirational, knowledge flow between both activity realms".

In order to identify previous works containing any concordance tables linking the IPC patent classification to the scientific field, it has been carried out an analysis of the literature. The starting point was the report EU called LINKST_2007 (S & T Linkage Indicators). This report was the primary source to initiate the literature review.

The analysis was performed according to the following steps:

 in the list of references were identified 15 articles as potentially relevant;  it was checked for the presence or absence of a concordance table;  for each of 15 articles was compiled a list containing the articles in the references and the articles that cited it. It was identified a new set of publications as potentially relevant.  The list of the new publications was inspected to check for the presence of concordance tables.

The list of items containing a concordance table between IPC- scientific field is shown in Table 1A and 2A.

Table 1A: References selected from LINKST_2007_report references (Source: Daraio, Di Costa and Moed, 2015) Cited by Conc. Article/report Description (Google Data Table-Fig/page table Scholar)

OECD field of science and technology FOS Table 5-pag LINKST report and broad fields of 17|Figure 1-pag 18| seed article nd Y (2007) ISCED 1997 | Figure 2-pag Matching ISI subject 23|Table 6-pag24 categories to FOS cited by Acosta and LINKST_200 71 N / / Coronado (2003) 7_report EPO patents with cited by Callaert et al application year LINKST_200 92 Y Table 6. --pag 17 (2006) between 1991 and 7_report 2001 24 selected cited by Carpenter and technology in 3 areas LINKST_200 100 Y Table1-pag 181 Narin (1983) (mechanical, 7_report chemistry, electrical) cited by 24 selected Carpenter et al. 132 Y Table2-pag 34 LINKST_200 technology in 3 areas

102

Cited by Conc. Article/report Description (Google Data Table-Fig/page table Scholar)

(1980) 7_report (mechanical, chemistry, electrical) cited by Grupp and LINKST_200 82 N / / Schmoch (1992) 7_report cited by Table6-pag 170 | All patent citing Meyer (2000a) LINKST_200 89 Y Table 7-pag 171 scientific literature 7_report |Appendix1-pag 177 cited by Meyer (2000b) LINKST_200 349 N / / 7_report cited by All patent citing Narin and Noma LINKST_200 270 Y scientific literature Table2-pag 375 (1985) 7_report (1978-1980) cited by Narin et al. LINKST_200 1010 N / / (1997) 7_report 30 technology areas cited by at the European Schmoch (1997) LINKST_200 120 Y Fig2-pag 108 Patent Office for 7_report 1989-1992 Dutch paper cited at cited by Tijssen et al. least by one USPTO Table1-pag 401 | LINKST_200 79 Y (2000). patent issued in Table2-pag 402 7_report period 1993-1996 cited by Van Looy et al. LINKST_200 23 N / / (2003a) 7_report cited by Van Looy LINKST_200 48 N / / (2003b) 7_report cited by Verbeek et al. LINKST_200 46 N / / (2003) 7_report cited by Verbeek et al. LINKST_200 100 N / / (2002a) 7_report

103

Table 2: Articles containing a concordance table (Articles coming from the references or the list of citing articles of the report LINKST_2007_report). na= not available, Y=yes, N=no. (Source: Daraio, Di Costa and Moed, 2015) Short references Pag_Table/Figure

Bassecoulard and Zitt (2005) pag686_Figure 30.3 Bhattacharya et al. (2003) pag681_Table 2 Callaert et al. (2006) pag17_Table6 Carpenter and Narin (1983) pag 181_Table1 Carpenter et al. (1980) pag 34_Table2 Coronado et al. (2004) pag1086_Table 5 Glänzel and Meyer (2003) pag423_Table6|Appendix_Table 8 Grupp and Schmoch (1992) pag 85_Table 4.1 Han (2007) pag13_Table1 Hullmann and Meyer (2003) pag521_Table 5 Meyer (2000) pag 170_Table6 Narin and Olivastro (1992) pag232_Table 2 Noyons et al. (1994) pag449_Table3 Park and Suh (2013) pag3_Fig1 | pag6_Fig3 Ribeiro et al. (2014) pag62_Fig2|pag63_Fig3 Ribeiro et al. (2010) pag60_Table2 Schmoch (1997) pag108_Fig2 Tijssen et al. (2000) pag401_Table1 Van Looy et al. (2003) pag361_Table 1 Verbeek et al. (2002a) pag418_Table 2 Vol1_pag34_Tab3.A_Tab3.B_Table III.4A_Table Verbeek et al. (2002b) III.4B_APPENDIX 1.A_APPENDIX 1.B_APPENDIX 2.A_APPENDIX 2.B|Vol2 pag41_Fig10 pag417_Figure D6.1.3|pag418_Figure D6.1.4_Table Verbeek et al (2002c) D6.1.1|pag419_Figure D6.1.5_Table D6.1.2|pag420_Table D6.1.3 Wang et al. (2014) pag11_Fig. 6

104

REFERENCES

Acosta, M., and Coronado, D. (2003). "Science-technology flows in Spanish regions: An analysis of scientific citations in patents", Research Policy, 32 (10), 1783 -1803.

Bassecoulard, E., Zitt, M. (2005). Patents and publications. In Handbook of quantitative science and technology research (pp. 665-694). Springer Netherlands.

Bhattacharya, S., Kretschmer, H., Meyer, M. (2003). Characterizing intellectual spaces between science and technology. Scientometrics, 58(2), 369-390.

Callaert, J., Looy, B. V., Verbeek, A., Debackere, K., Thijs, B. (2006). Traces of Prior Art: An analysis of non-patent references found in patent documents. Scientometrics, 69(1), 3-20.

Carpenter, M., Narin, F. (1983). Validation study: patent citations as indicators of science and foreign dependence. World Patent Information , 5(3), 180-185.

Carpenter, M.P., M. Cooper and F. Narin (1980). Linkage between basic research and patents. Research Management , 23, 30-35.

Coronado, D., Acosta, M., León 1, D. (2004). Regional planning of R&D and science–technology interactions in Andalucia: a bibliometric analysis of patent documents. European Planning Studies, 12(8), 1075-1095.

Evenson, R. E., J. Putnam and S. Kortum (1991): Estimating Patent Counts by Industry using the Yale-Canada Concordance. Final Report to the National Science Foundation, Yale University

Glänzel, W., Meyer, M. (2003). Patents cited in the scientific literature: An exploratory study of 'reverse' citation relations. Scientometrics, 58(2), 415-428.

Grupp, H., Schmoch, U. (1992). Perceptions of scientification of innovation as measured by referring between patents and papers: dynamics in science -based fields of technology. In: Grupp, H. (Ed.) Dynamics of science -based innovation. Berlin: Springer-Verlag.

Han, Y. J. (2007). Linking Science and Technology with Industry. Journal of Technology Management & Innovation, 2(2), 7-17.

Hullmann, A., Meyer, M. (2003). Publications and patents in nanotechnology. Scientometrics, 58(3), 507-527.

Johnson, D. K. (2002). Unclassified DSTI/DOC (2002)5.

Kortum, S., Putnam, J. (1997). Assigning patents to industries: tests of the Yale Technology Concordance. Economic Systems Research, 9(2), 161-176.

Lybbert, T. J., Zolas, N. J. (2014). Getting patents and economic data to speak to each other: An 'algorithmic links with probabilities' approach for joint analyses of patenting and economic activity. Research Policy, 43(3), 530-542.

MERIT (2007). S&T Linkage Indicators (LINKST), Final Report to the European Commission, DG Research

Meyer, M. (2000a). Patent citations in a novel field of technology - what can the y tell about interactions between emerging communities of science and technology? Scientometrics, 48(2), 151-178.

Meyer, M. (2000b). Does science push technology? Patents citing scientific literature, Research Policy, 29(3), 409 -434.

Meyer, M., Persson, O. (1998). Nanotechnology-interdisciplinarity, patterns of collaboration and differences in application. Scientometrics, 42, 195–205.

Narin, F., and Noma, E. (1985). Is technology becoming science? Scientometrics, 7(3-6), 369-381.

Narin, F., Hamilton, K., Olivastro, D. (1997). The increasing linkage between US technology and public science. Research Policy, 26(3), 317 -330.

105

Narin, F., Olivastro, D. (1992). Status report: linkage between technology and science. Research policy, 21(3), 237-249.

Noyons, E. C. M., Van Raan, A. F. J., Grupp, H., Schmoch, U. (1994). Exploring the science and technology interface: Inventor-author relations in laser medicine research. Research Policy, 23(4), 443-457.

Park, H. W., Suh, S. H. (2013). Scientific and technological knowledge flow and technological innovation: quantitative approach using patent citation. Asian Journal of Technology Innovation, 21(1), 153-169.

Putnam, J. & Evenson, R. E. (1994) Inter-sectoral technology flows: estimates from a patent concordance with an application to Italy, mimeo, Yale University.

Ribeiro, L. C., Kruss, G., Britto, G., Bernardes, A. T., Albuquerque, E. D. M. (2014). A methodology for unveiling global innovation networks: patent citations as clues to cross border knowledge flows. Scientometrics, 101(1), 61-83.

Ribeiro, L. C., Ruiz, R. M., Bernardes, A. T., Albuquerque, E. M. (2010). Matrices of science and technology interactions and patterns of structured growth: implications for development. Scientometrics, 83(1), 55-75.

Schmoch, U. (1997). Indicators and the relations between science and technology. Scientometrics, 38(1), 103 -116.

Schmoch, U., Laville, F., Patel, P., Frietsch, R. (2003). Linking technology areas to industrial sectors. Final Report to the European Commission, DG Research.

Schmookler, J. (1966). Invention and Economic Growth. Cambridge: Harvard University Press.

Sternitzke, C. (2009). Patents and publications as sources of novel and inventive knowledge. Scientometrics, 79, 551–561.

Tijssen, R.J.W., Buter, R.K., Van Leeuwen, T.N. (2000). Technological relevance of science: Validation and analysis of citation linkages between patents and research papers, Scientometrics, 47(2), 389 -412.

Van Looy, B., Callaert, J., Debackere, K., Verbeek, A. (2003a). Patent -related indicators for assessing knowledge-generating institutions: Towards a contextualised approach. Journal of Technology Transfer, 28(1), 53 -61.

Van Looy, B., Zimmermann, E., Veugelers, R., Verbeek, A., Mello, J., and Debackere, K. (2003b). Do science - technology interactions pay off when developing technology? Scientometrics, 57(3), 355 -367.

Verbeek et al (2002c). 3° report EU on science & technology Indicators

Verbeek, A., Debackere, K., Luwel, M. (2003). Science cited in patents: a geographic 'flow' analysis of bibliographic citation patterns in patents. Scientometrics, 58(2), 241-263.

Verbeek, A., Debackere, K., Luwel, M., Andries, P., Zimmermann, E., Deleus, F. (2002a). Linking science to technology: Using bibliographic references in patents to build linkage schemes, Scientometrics, 54(3), 399 -420.

Verbeek, A., P. Andries, J. Callaert, K. Debackere, M. Luwel, and R. Veugelers, (2002b). Linking Science to Technology: Bibliographic References in Patents_Report to the European Commission.

Verspagen, B., Morgastel, T. V., Slabbers, M. (1994): MERIT concordance table: IPC - ISIC (rev. 2), Maastricht: MERIT Research Memorandum 2,94-004.

Wang, M. Y., Fang, S. C., Chang, Y. H. (2014). Exploring technological opportunities by mining the gaps between science and technology: Microalgal biofuels. Technological Forecasting and Social Change.

106

Appendix 3: Possible User Groups

Source: AUBR (European Commission, 2010)

HE MANAGEMENT AND GOVERNANCE Governing Bodies/Councils � Policy and planning � Strategic positioning � Research strategy development/management � Investor confidence/value‐for‐money and efficiency � Quality assurance � Institutional and discipline/field data re. level of intensity, expertise, quality and competence � Benchmarking against peer institutions, nationally and worldwide � Efficiency level: how much output vis‐a‐vis funding � Quality of academic staff and PhD students � Attraction capacity: recruitment of students, academics and researchers from outside region and internationally HE Executives/Management � Policy and planning � Strategic positioning � Research strategy development/management � Investor confidence/value‐for‐money and efficiency � Quality assurance � Publicity � Student and academic recruitment � Improve and benchmark performance and quality � Institutional and discipline/field data re. level of intensity, expertise, quality and competence � Benchmarking against peer institutions, nationally and worldwide � Efficiency level: how much output vis‐a‐vis funding � Quality of academic staff and PhD students � Attraction capacity: recruitment of students, academics and researchers from outside region and internationally � Identification of Partnerships (academic, public/private sector, NGOs, research organisations, etc.) HE Research Groups � Strategic positioning � Research strategy development/management � Investor confidence/value‐for‐money and efficiency � Student and academic recruitment � Discipline data re. level of intensity, expertise, quality and competence benchmarked against peer institutions � Quality of academic staff and PhD students � Attraction capacity: recruitment of students, academics and researchers from outside region and internationally � Identification of Partnerships (academic, public/private sector, NGOs, research organisations, etc.) GOVERNMENTS AND GOVERNMENT AGENCIES EU and National Governments � Define policy and inform decisions about HE system and HEIs � Determine national/international competitiveness � Quality, sustainability, relevance and impact of research activity � System and institutional data re level of intensity, expertise, quality and competence � Performance of HE system and individual institutions � Benchmarking between nationally and worldwide � Indicator of national competitiveness � Investor confidence/value‐for‐money and efficiency � Improve performance and quality � Improve system functionality � Attraction capacity: recruitment of students, academics and researchers from outside region and internationally � Quality of academic staff and PhD students � Efficiency level: how much output vis‐a‐vis funding � Research infrastructure: level of use and efficiency Ministries of Education/Higher Education or Enterprise and Employment � Policy and planning � Strategic positioning of HE institutions � Quality, sustainability, relevance and impact of research activity � Research strategy development/management � Investor confidence/value‐for‐money and efficiency � Quality assurance

107

� Institutional and discipline/field data re. level of intensity, expertise, quality and competence � Benchmarking against peer institutions, nationally and worldwide � Indicator of national competitiveness � Performance of HE system and individual institutions � Attraction capacity: recruitment of students, academics and researchers from outside region and internationally � Efficiency level: how much output vis‐a‐vis funding � Research infrastructure: level of use and efficiency Local and Regional Governments � Define local/regional policy and competitiveness � Quality, sustainability, relevance and impact of research activity � Improve integration/collaboration between universities, government and private sector � Improve attraction capacity � Benchmarking performance and quality of HE system/institutions nationally and worldwide � Indicator of national competitiveness � Attraction capacity: recruitment of students, academics and researchers from outside region and internationally � Efficiency level: how much output vis‐a‐vis funding HE Agencies � Define policy and inform decisions about HE system and HEIs � Quality, sustainability, relevance and impact of research activity � Determine national/international competitiveness � Investor confidence/value‐for‐money and efficiency � Improve performance and quality � Improve system functionality � System and institutional data re level of intensity, expertise, quality and competence � Performance of HE system and individual institutions � Benchmarking between nationally and worldwide � Indicator of national competitiveness � Attraction capacity: recruitment of students, academics and researchers from outside region and internationally � Quality of academic staff and PhD students � Efficiency level: how much output vis‐a‐vis funding � Research infrastructure: level of use and efficiency Other Government Agencies � Improve and benchmark performance and quality � Aid resource allocation � Investor confidence/value‐for‐money and efficiency � Benchmarking performance and quality of HE system institutions nationally and worldwide ACADEMIC ORGANISATIONS AND ACADEMIES � Benchmark professional and academic performance and quality � Academic and discipline/field data re. level of intensity, expertise, quality and competence � Student and Academic Recruitment � Benchmarking against peer institutions, nationally and worldwide � Quality of academic staff and PhD students INDIVIDUALS Academics and Researchers � Identify career opportunities � Identify research partners � Identify best research infrastructure and support for research � Institutional and field data re level of intensity, expertise, quality, competence and sustainability � Performance of individual institution benchmarked against peers in field of interest � Employment conditions � Impact of research on teaching, Staff/student ratio � Institutional research support Students � Inform choice of HEI � Identify career opportunities � Institutional and field data re level of intensity, expertise, quality, competence and sustainability � Performance of individual institution benchmarked against peers in field of interest � Research capacity of institution and research team, e.g. graduate students/academic ratio, age of PhD students, time to completion, structure/characteristics of PhD programme and support � Graduate career and employment trends � Quality of the research infrastructure � Staff/student ratio PEER HEIS � Identify peer HEIs and best research partners � Institutional and field data re level of intensity, expertise, quality, competence and sustainability � Performance of individual institutions and researchers benchmarked against peers in field of interest

108

� Research capacity of institution and research team � Potential for partnership INDUSTRY PARTNER ORGANISATIONS Private firms and entrepreneurs � Quality, sustainability, relevance and impact of research activity � Identify potential partners and expertise � Identify consultancy, technology transfer and knowledge transfer partners and expertise � Identify potential employees � Institutional and field data re level of intensity, expertise, quality, competence and sustainability � Performance of individual institution benchmarked against peers in field of interest � Competitive positioning of institution and researchers � Trends in graduate employment and competence � Quality of HE programme, and link between research and teaching Public Organisations � Quality, sustainability, relevance and impact of research activity � Identify potential partners and expertise � Identify consultancy, technology transfer and knowledge transfer partners and expertise � Identify potential employees � Institutional and field data re level of intensity, expertise, quality, competence and sustainability � Performance of individual institution benchmarked against peers in field of interest � Competitive positioning of institution and researchers � Trends in graduate employment and competence � Quality of HE programme, and link between research and teaching Employers � Quality, sustainability, relevance and impact of research activity � Identify potential partners and expertise � Identify consultancy, technology transfer and knowledge transfer partners and expertise � Identify potential employees � Institutional and field data re level of intensity, expertise, quality, competence and sustainability � Performance of individual institution benchmarked against peers in field of interest � Competitive positioning of institution and researchers � Trends in graduate employment and competence � Quality of HE programme, and link between research and teaching CIVIC SOCIETY AND CIVIC ORGANIZATIONS � Identify specific expertise and information � Identify potential collaborator � Identify consultancy, technology transfer and knowledge transfer partners � Institutional and field data re expertise, quality and competence � Peer esteem indicators MINISTRIES OF HIGHER EDUCATION IN DEVELOPING COUNTRIES � To help determine which foreign higher education institutions are applicable for overseas scholarships studies. � To help determine research partnerships for knowledge and technology transfer � Institutional and discipline/field data re. level of intensity, expertise, quality and competence � Competitive positioning of institution and researchers � Trends in graduate employment and competence � Quality of academic staff and PhD students SPONSORS AND PRIVATE INVESTORS Benefactors/Philanthropists � Determine institutional performance vis‐a‐vis national and international competitors � Investor confidence/value‐for‐money and efficiency � Quality, sustainability, relevance and impact of research activity � Quality of academic staff and PhD student � Contributor to own brand image � Institutional data re level of quality and international competitiveness � Benchmarking between nationally and worldwide � Quality of academic staff and PhD students Alumni � Determine institutional performance vis‐a‐vis national and international competitors � Institutional data re level of quality and international competitiveness � Investor confidence/value‐for‐money and efficiency � Quality, sustainability, relevance and impact of research activity � Quality of academic staff and PhD student � Reflect pride and career aspirations/reputation � Benchmarking between nationally and worldwide � Quality of academic staff and PhD students PUBLIC OPINION � Determine institutional performance vis‐a‐vis national and international competitors � Quality, sustainability, relevance and impact of research activity

109

� Student choice and career opportunities � Investor/parental confidence and value‐for‐money � Institutional data re. level of intensity, expertise, quality and competence � Benchmarking against peer institutions, nationally and worldwide � Indicator of national competitiveness � Performance of HE system and individual institutions � Efficiency level: how much output vis‐a‐vis funding

110

How to obtain EU publications

Free publications: • one copy: via EU Bookshop (http://bookshop.europa.eu); • more than one copy or posters/maps: from the European Union’s representations (http://ec.europa.eu/represent_en.htm); from the delegations in non-EU countries (http://eeas.europa.eu/delegations/index_en.htm); by contacting the Europe Direct service (http://europa.eu/europedirect/index_en.htm) or calling 00 800 6 7 8 9 10 11 (freephone number from anywhere in the EU) (*).

(*) The information given is free, as are most calls (though some operators, phone boxes or hotels may charge you).

Priced publications: • via EU Bookshop (http://bookshop.europa.eu).

[Catalogue number [Catalogue

- 27

389

]

This study examines the feasibility of constructing a European Map of Excellence and Specialization (EMES). The term 'Map of Excellence and Specialization' refers to a geographical information system that combines and georeferences information from various sources at different geographic scales and provides indicators intended for policy use. The study explores also the policy implications of using such a Map of Excellence and Specialization. However, drawing an EMES entails many challenges: Actors in the European S&T field are heterogeneous, their output is composite, the location of their activities is not fully disclosed, and so on. The main criteria to assess the successful implementation of the EMES have been identified in: Availability of data on publications; Standardization, openness and interoperability with other sources of data available; Compliance with state of the art data quality techniques; Continuity; Extensions and scalability; Expertise in the access and analysis of publicly available data; Interactivity and usability; Existence of concordance tables among different subject classifications.

Studies and reports

ISBN 978-92-79-50354-2