The Metric Tide Literature Review Supplementary Report I to the Independent Review of the Role of Metrics in Research Assessment
Total Page:16
File Type:pdf, Size:1020Kb
The Metric Tide Literature Review Supplementary Report I to the Independent Review of the Role of Metrics in Research Assessment and Management July 2015 Authors Paul Wouters*, Mike Thelwall**, Kayvan Kousha**, Ludo Waltman*, Sarah de Rijcke*, Alex Rushforth*, and Thomas Franssen* *Centre for Science and Technology Studies (CWTS), Leiden University, Netherlands. **Statistical Cybermetrics Research Group, University of Wolverhampton, UK. Cite as Wouters, P. et al. (2015). The Metric Tide: Literature Review (Supplementary Report I to the Independent Review of the Role of Metrics in Research Assessment and Management). HEFCE. DOI: 10.13140/RG.2.1.5066.3520 © HEFCE 2015, except where indicated. Cover image © JL-Pfeifer / Shutterstock.com. The parts of this work that are © HEFCE are available under the Open Government Licence 2.0: www.nationalarchives.gov.uk/doc/open-government-licence/version/2 ISBN: 1902369280 DOI: 10.13140/RG.2.1.5066.3520 Twitter: #HEFCEmetrics Contents Executive Summary ............................................................................................................................... iv Bibliometrics and the use of indicators .............................................................................................. iv Bibliographic databases ............................................................................................................ iv Basic citation impact indicators ................................................................................................ iv Exclusion of specific types of publications and citations .......................................................... v Normalisation of citation impact indicators ............................................................................... v Credit allocation in the case of multi-author publications ......................................................... v Indicators of the citation impact of journals .............................................................................. v Main research strands on indicator effects ................................................................................ vi Peer review and bibliometrics ........................................................................................................... vii Alternative metrics for research evaluation ....................................................................................... ix 1. Bibliometrics and the use of citation indicators in research assessment .................................... 1 1.1. Bibliographic databases ......................................................................................................... 3 1.1.1. Comparing WoS and Scopus ........................................................................................ 4 1.1.2. Comparing GS with WoS and Scopus .......................................................................... 5 1.1.3. Social sciences and humanities ..................................................................................... 6 1.1.4. Conference proceedings ................................................................................................ 7 1.2. Citation impact indicators ...................................................................................................... 8 1.2.1. Basic citation impact indicators .................................................................................... 8 1.2.2. Exclusion of specific types of publications and citations ........................................... 11 1.2.3. Normalisation of citation impact indicators ................................................................ 13 1.2.4. Credit allocation in the case of multi-author publications .......................................... 20 1.2.5. Indicators of the citation impact of journals ............................................................... 24 1.3. Evaluation practices and effects of indicator use ................................................................ 29 1.3.1. Introduction: main research strands ............................................................................ 29 1.3.2. Known effects ............................................................................................................. 30 1.4. Bibliometrics and the use of indicators summary ................................................................ 38 1.4.1. Bibliographic databases .............................................................................................. 38 i 1.4.2. Basic citation impact indicators .................................................................................. 38 1.4.3. Exclusion of specific types of publications and citations ........................................... 39 1.4.4. Normalisation of citation impact indicators ................................................................ 39 1.4.5. Credit allocation in the case of multi-author publications .......................................... 40 1.4.6. Indicators of the citation impact of journals ............................................................... 41 1.4.7. Main research strands on indicator effects .................................................................. 41 1.4.8. Strategic behaviour and goal displacement ................................................................. 42 1.4.9. Effects on interdisciplinarity ....................................................................................... 42 1.4.10. Task reduction ........................................................................................................ 42 1.4.11. Effects on institutions ............................................................................................. 42 1.4.12. Effects on knowledge production ........................................................................... 43 2. Peer review and bibliometrics .................................................................................................. 44 2.1. Forms of peer review ........................................................................................................... 44 2.2. Correlating bibliometrics with peer review ......................................................................... 49 2.2.1. Journal peer review ..................................................................................................... 49 2.2.2. Grant peer review ........................................................................................................ 49 2.2.3. Correlating bibliometrics and peer judgment of research groups .............................. 52 2.2.4. Bibliometrics and national research evaluation exercises ........................................... 53 2.2.5. Predicting the outcomes of the UK RAE and REF by bibliometrics .......................... 55 2.3. Informed peer review........................................................................................................... 61 2.4. Peer review and bibliometrics conclusions and summary ................................................... 65 3. Alternative metrics for research evaluation ............................................................................. 68 3.1. Web-based open access scholarly databases ....................................................................... 69 3.1.1. Limitations of traditional citation databases: A brief overview .................................. 70 3.1.2. GS ............................................................................................................................... 71 3.1.3. Patents and Google Patents ......................................................................................... 73 3.1.4. Usage indicators from scholarly databases ................................................................. 74 3.2. Citations and links from the general web ............................................................................ 76 3.2.1. Link analysis ............................................................................................................... 77 3.2.2. Web and URL citations ............................................................................................... 77 ii 3.3. Citations from specific parts of the general web ................................................................. 78 3.3.1. Online presentations ................................................................................................... 79 3.3.2. Online course syllabi for educational impact .............................................................. 80 3.3.3. Science blogs .............................................................................................................. 81 3.3.4. Other sources of online impact ................................................................................... 83 3.4. Citations, links, downloads and recommendations from social websites ............................ 83 3.4.1. Faculty of 1000 web recommendations ...................................................................... 84 3.4.2. Mendeley and other online reference managers ......................................................... 86 3.4.3. Twitter and microblog citations .................................................................................. 89 3.4.4. Facebook and Google+ citations ................................................................................. 92 3.4.5. Academic social network sites: Usage and follower counts ....................................... 92 3.5. Indicators for book impact assessment