Bibliometrics As a Tool for Supporting Prospective R&D Decision-Making in the Health Sciences: Strengths, Weaknesses And
Total Page:16
File Type:pdf, Size:1020Kb
THE ARTS This PDF document was made available from www.rand.org as a public CHILD POLICY service of the RAND Corporation. CIVIL JUSTICE EDUCATION ENERGY AND ENVIRONMENT Jump down to document6 HEALTH AND HEALTH CARE INTERNATIONAL AFFAIRS The RAND Corporation is a nonprofit research NATIONAL SECURITY POPULATION AND AGING organization providing objective analysis and effective PUBLIC SAFETY solutions that address the challenges facing the public SCIENCE AND TECHNOLOGY and private sectors around the world. SUBSTANCE ABUSE TERRORISM AND HOMELAND SECURITY TRANSPORTATION AND INFRASTRUCTURE Support RAND WORKFORCE AND WORKPLACE Browse Books & Publications Make a charitable contribution For More Information Visit RAND at www.rand.org Explore RAND Europe View document details Limited Electronic Distribution Rights This document and trademark(s) contained herein are protected by law as indicated in a notice appearing later in this work. This electronic representation of RAND intellectual property is provided for non-commercial use only. Unauthorized posting of RAND PDFs to a non-RAND Web site is prohibited. RAND PDFs are protected under copyright law. Permission is required from RAND to reproduce, or reuse in another form, any of our research documents for commercial use. For information on reprint and linking permissions, please see RAND Permissions. This product is part of the RAND Corporation technical report series. Reports may include research findings on a specific topic that is limited in scope; present discus- sions of the methodology employed in research; provide literature reviews, survey instruments, modeling exercises, guidelines for practitioners and research profes- sionals, and supporting documentation; or deliver preliminary findings. All RAND reports undergo rigorous peer review to ensure that they meet high standards for re- search quality and objectivity. Bibliometrics as a tool for supporting prospective R&D decision-making in the health sciences Strengths, weaknesses and options for future development Sharif Ismail, Edward Nason, Sonja Marjanovic, Jonathan Grant Prepared with funding support from the Health R&D Policy Research Unit of the Department of Health (England) EUROPE The research described in this report was prepared for the Department of Health England. The RAND Corporation is a nonprofit research organization providing objective analysis and effective solutions that address the challenges facing the public and private sectors around the world. RAND’s publications do not necessarily reflect the opinions of its research clients and sponsors. R® is a registered trademark. © Copyright 2009 RAND Corporation Permission is given to duplicate this document for personal use only, as long as it is unaltered and complete. Copies may not be duplicated for commercial purposes. Unauthorized posting of RAND documents to a non-RAND Web site is prohibited. RAND documents are protected under copyright law. For information on reprint and linking permissions, please visit the RAND permissions page (http://www.rand.org/publications/ permissions.html). Published 2009 by the RAND Corporation 1776 Main Street, P.O. Box 2138, Santa Monica, CA 90407-2138 1200 South Hayes Street, Arlington, VA 22202-5050 4570 Fifth Avenue, Suite 600, Pittsburgh, PA 15213-2665 Westbrook Centre, Milton Road, Cambridge CB4 1YG, United Kingdom RAND URL: http://www.rand.org RAND Europe URL: http://www.rand.org/randeurope To order RAND documents or to obtain additional information, contact Distribution Services: Telephone: (310) 451-7002; Fax: (310) 451-6915; Email: [email protected] Preface Bibliometric analysis is an increasingly important part of a broader ‘toolbox’ of evaluation methods available to R&D policymakers to support decision-making. In the US, UK and Australia, for example, there is evidence of gradual convergence over the past ten years towards a model of university research assessment and ranking incorporating the use of bibliometric measures. In Britain, the Department of Health (England) has shown growing interest in using bibliometric analysis to support prospective R&D decision-making, and has engaged RAND Europe’s expertise in this area through a number of exercises since 2005. These range from the macro-level selection of potentially high impact institutions, to micro-level selection of high impact individuals for the National Institute for Health Research’s faculty of researchers. The aim of this document is to be an accessible, ‘beginner’s guide’ to bibliometric theory and application in the area of health research and development (R&D) decision-making. The report also aims to identify future directions and possible next steps in this area, based on RAND Europe’s work with the Department of Health to date. It is targeted at a range of audiences, and will be of interest to health and biomedical researchers, as well as R&D decision-makers in the UK and elsewhere. The report was produced with funding support from RAND Europe’s Health R&D Policy Research Unit with the Department of Health. RAND Europe is an independent, not-for-profit policy research institute based in Cambridge. For the past three years, RAND has been a designated research unit of the Department of Health, which supported the research for and publication of this document. For more information about RAND Europe or this document, please contact: Dr Jonathan Grant President RAND Europe Westbrook Centre Milton Road Cambridge CB4 1YG Tel: +44 1223 353329 Email: [email protected] iii Contents Preface ........................................................................................................................ iii List of figures ............................................................................................................... ix List of tables ................................................................................................................ xi List of abbreviations ...................................................................................................xiii Summary .................................................................................................................... xv Acknowledgements ................................................................................................... xvii CHAPTER 1 Introduction .................................................................................... 1 1.1 Setting the scene: why evaluate? ......................................................................... 1 1.2 Doing evaluation: the research evaluator’s toolkit .............................................. 3 1.3 Organisation and structure of the report ............................................................ 4 CHAPTER 2 Structuring bibliometric analyses ..................................................... 5 2.1 What is bibliometrics? ....................................................................................... 5 2.1.1 Bibliometrics is part of a family of methods for assessing scientific communication .................................................................................... 5 2.1.2 Publication and citation data have important characteristics ................. 6 2.2 Why do we need bibliometrics? ......................................................................... 7 2.2.1 Well-recognised problems with other evaluation methodologies ........... 7 2.2.2 Bibliometric analysis offers some clear advantages over alternative approaches ............................................................................................ 7 2.3 Some key issues in evaluation: what can bibliometrics measure? ......................... 9 2.3.1 Measuring scientific collaboration ......................................................... 9 2.3.2 Assessing interdisciplinary research ....................................................... 9 2.3.3 The quest for ‘quality’ and ‘research excellence’ .................................. 10 2.4 Integrating bibliometric analysis with other performance analysis tools ............ 10 2.4.1 Australian Research Quality Framework ............................................. 10 2.4.2 Moving towards Excellence in Research for Australia .......................... 11 2.5 Using bibliometrics to map scientific research ................................................. 11 2.6 Bibliometrics for research evaluation: measures and indicators ......................... 12 2.6.1 Publication volume measures .............................................................. 15 2.6.2 Journal-linked performance measures ................................................. 15 2.6.3 Citation analyses as measures of research impact ................................. 16 2.6.4 Composite measures ........................................................................... 19 v Contents RAND Europe 2.6.5 Distribution-based approaches ............................................................ 19 CHAPTER 3 Using bibliometrics for selection procedures ................................. 21 3.1 Background ..................................................................................................... 21 3.2 Selecting centres of excellence (biomedical research centres) ............................. 22 3.2.1 Pre-application.................................................................................... 22 3.2.2 Post-application .................................................................................. 25 3.3 Selecting biomedical research units .................................................................