Comparative scientometric assessment of the results of ERC funded projects Peer-review evaluation of highly ranked publications from scientometric assessment (D8) EUR [number] EN EUROPEAN COMMISSION Directorate-General for Research and Innovation ERC EA – The European Research Council Executive Agency Unit A1 — Support to the Scientific Council E-mail: [email protected] European Commission, ERC Executive Agency B-1049 Brussels EUROPEAN COMMISSION Comparative scientometric assessment of the results of ERC funded projects Peer-review evaluation of highly ranked publications from scientometric assessment (D8) Directorate-General for Research and Innovation 2015 European Research Council Executive Agency EUR [number] EN EUROPE DIRECT is a service to help you find answers to your questions about the European Union Freephone number (*): 00 800 6 7 8 9 10 11 (*) Certain mobile telephone operators do not allow access to 00 800 numbers or these calls may be billed LEGAL NOTICE Neither the European Commission nor any person acting on behalf of the Commission is responsible for the use which might be made of the following information. The views expressed in this publication are the sole responsibility of the author and do not necessarily reflect the views of the European Commission. More information on the European Union is available on the Internet (http://europa.eu). Cataloguing data can be found at the end of this publication. Luxembourg: Publications Office of the European Union, 2015. ISBN [number] doi:[number] © European Union, 2015 Reproduction is authorised provided the source is acknowledged. Comparative scientometric assessment of the results of ERC funded projects Peer-review evaluation of highly ranked publications from scientometric assessment (D8) Alexandra Pollitt, Salil Gunashekar, Gavin Cochrane, Jean-Pierre Robitaille, Vincent Larivière Prepared for EUROPEAN COMMISSION, RESEARCH & INNOVATION DIRECTORATE-GENERAL Unit A1 – Support to the Scientific Council ERCEA/A1/2014/20 Under the Framework Contract: ‘Provision of Services in the Fields of Research Evaluation and Research Policy Analysis’ [No 2010/S 172-262618] Lot 2: Data collection and performance indicators to monitor European research policy RR-1179-ERC July 2015 RAND's publications do not necessarily reflect the opinions of its research clients and sponsors. is a registered trademark Preface This document, prepared by RAND Europe and Observatoire des sciences et des technologies (OST), serves as the final deliverable ‘Peer-review evaluation of highly ranked publications from scientometric assessment’ (deliverable: D8) for the study “Comparative scientometric assessment of the results of ERC funded projects” for the European Research Council Executive Agency (ERCEA). In addition to this report, other analysis and findings from this study are reported in: D3: Field classification report D4: Data coverage report D5: Bibliometric assessment report D6: Patent analysis report D7: Alternative metrics report D11: Final synthesis report The authors would like to acknowledge the support of ERCEA, in particular Boris Kragelj, members of the ERC Scientific Council and other ERC staff members who have provided useful feedback, data and advice throughout the study. We would also like to thank our quality assurance reviewers, Yves Gingras, Susan Guthrie and David Kryl, for their constructive comments and timely review. This document has been peer reviewed in accordance with RAND Europe’s quality assurance standards. i Table of Contents Preface .............................................................................................................................. i Table of Contents ............................................................................................................. iii Figures ............................................................................................................................. v Tables ............................................................................................................................. vii 1. Summary ..................................................................................................................... 1 2. Introduction and background ...................................................................................... 5 2.1. About the study ................................................................................................ 5 2.2. Introduction ..................................................................................................... 5 3. Methods ...................................................................................................................... 7 3.1. The survey ....................................................................................................... 7 3.2. Selecting the papers .......................................................................................... 8 3.3. Selecting reviewers ........................................................................................... 9 4. Results ...................................................................................................................... 11 4.1. Overview of all responses ..................................................................................11 4.2. Life sciences ....................................................................................................23 4.3. Physical sciences and engineering ......................................................................27 5. Discussion ................................................................................................................. 31 5.1. Some methodological caveats ............................................................................31 5.2. What kinds of contributions has ERC-funded research made? .................................32 5.3. What does highly cited ERC research look like? ....................................................32 5.4. Why were papers highly cited? ...........................................................................33 5.5. What leads to social media attention? .................................................................34 5.6. In conclusion ...................................................................................................34 References ...................................................................................................................... 37 Appendices ..................................................................................................................... 39 Appendix A: List of questions used for the online peer-review evaluation of highly cited papers ....41 Appendix B: List of highly cited papers selected for peer-review evaluation ................................45 Appendix C: Respondents and response rate ..........................................................................49 Appendix D: Reviewer ratings for the article’s overall contribution to the advancement of science / knowledge .........................................................................................................................51 Appendix E: Breakdown of papers’ characteristics by overarching characteristic group and research domain ..............................................................................................................................55 iii Appendix F: ‘Landmark’ papers vs ‘non-landmark’ papers: comparing scientometric indicators .....59 Appendix G: Breakdown of papers’ characteristics by level of social media attention ...................61 Appendix H: Reasons suggested by reviewers for citation and social media attention – life sciences63 Appendix I: Reasons suggested by reviewers for citation and social media attention – physical sciences and engineering .....................................................................................................65 iv Figures Figure 1-1. Overall contribution of papers to the advancement of science or knowledge .............. 1 Figure 1-2. Reasons selected by reviewers for papers being highly cited .................................... 2 Figure 4-1. Word cloud highlighting the most frequently occurring words in the titles of all the highly cited articles used in the peer review task ............................................................ 11 Figure 4-2. Overall contribution of papers across each of the three domains ............................. 13 Figure 4-3. Top ten reasons for papers being highly cited (across all three research domains), as suggested by reviewers............................................................................................... 19 Figure 4-4. Top reasons suggested by reviewers that might lead to a paper receiving social media attention (across all three research domains) ................................................................ 21 Figure 4-5. Overall contribution of LS papers......................................................................... 24 Figure 4-6. Overall contribution of PE papers......................................................................... 27 Figure A-1. Questions for expert reviewers undertaking the qualitative assessment of highly ranked publications from the scientometric assessment ............................................................. 41 Figure C-1. Response rates by research domain .................................................................... 49 Figure E-1. Number of characteristics by paper: The type of content is… .................................