For Peer Review Only
Total Page:16
File Type:pdf, Size:1020Kb
BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from Bibliometrics of NIHR HTA monographs and their related journal articles ForJournal: peerBMJ Open review only Manuscript ID: bmjopen-2014-006595 Article Type: Research Date Submitted by the Author: 10-Sep-2014 Complete List of Authors: Royle, Pamela; Warwick Medical School, Division of Health Sciences Waugh, Norman; university of Warwick, Division of Health Sciences <b>Primary Subject Health services research Heading</b>: Secondary Subject Heading: Research methods, Medical publishing and peer review bibliometrics, health technology assessment, bibliographic databases, Keywords: citations http://bmjopen.bmj.com/ on September 23, 2021 by guest. Protected copyright. For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 1 of 12 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from 1 2 3 Bibliometrics of NIHR HTA monographs and their related journal articles. 4 5 Corresponding author; 6 7 Dr Pamela Royle 8 9 Division of Health Sciences 10 11 Warwick Medical School 12 13 University of Warwick, Coventry CV4 7AL , UK [email protected] tel: 02476 151762 14 15 For peer review only 16 17 Authors: Pamela Royle and Norman Waugh 18 19 Postal addresses as above. Email: [email protected] 20 21 22 23 MeSH terms: bibliometrics, bibliographic databases, health technology assessment 24 25 Word count: 3944 26 27 28 29 30 31 32 33 34 http://bmjopen.bmj.com/ 35 36 37 38 39 40 41 42 on September 23, 2021 by guest. Protected copyright. 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 1 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 2 of 12 BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from 1 ABSTRACT 2 3 Objectives: To undertake a bibliometric analysis of the UK National Institute for Health Research (NIHR) 4 Health Technology Assessment (HTA) monographs and their related journal articles by: 1) exploring the 5 6 differences in citations to the HTA monographs in Google Scholar (GS), Scopus, and Web of Science (WoS) 7 8 and, 2) comparing Scopus citations to the monographs with their related journal articles. 9 Setting: A retrospective cohort study of 111 HTA monographs published in 2010 and 2011, and their external 10 11 journal articles identified from the NIHR Journals Library website 12 Main outcome measures: Citations to the monographs in GS, Scopus and WoS and to their external journal 13 14 articles in Scopus 15 Results. The numberFor of citations peer varied amongst review the three databases, with only GS having the highest and WoS the 16 17 lowest, but the citation based rankings between the three databases were highly correlated. Overall, 56% of 18 monographs had a related publication, with the highest proportion for primary research (76%) and lowest for 19 20 evidence syntheses (43%). When comparing the differences in the number of citations in Scopus between 21 pairs of monograph publications with each of their related journal articles from the same project, we found 22 23 that monographs received more citations than their journal article versions for evidence syntheses and 24 methodology projects, whereas for primary research, the journal article versions were more highly cited than 25 26 their monograph version. Only 24% of authors citing the monographs and 21% citing the journals were from 27 the UK, implying a large international readership. 28 29 Conclusions: Those using bibliometric data for decisions regarding academic career progression and research 30 funding need to be aware of how using different databases will affect their results. Publishing in an external 31 32 journal as well as a monograph can aid dissemination of the work, but may have less impact than a single 33 34 publication, and could therefore adversely affect an author’s h-index. http://bmjopen.bmj.com/ 35 36 37 Strengths and limitations of this study 38 • First study to perform a detailed bibliometric analysis of a cohort of HTA monographs and their 39 40 related outputs published in external journals. 41 • We compared citations to the HTA monographs, using three citation databases, and compared 42 on September 23, 2021 by guest. Protected copyright. 43 citations by type of evidence. 44 • We chose monographs published in 2010 and 2011 in order to allow adequate time for citations to 45 46 accrue and for the related journal articles to be published. 47 48 • One limitation is that we obtained details of related publications from the HTA website, where the 49 number of related journal publications may be underestimated by 15.8%. 50 51 • A possible limitation is that we used the date that a publication was first added to PubMed as a 52 surrogate for the publication date because the actual date of journal publication is difficult to 53 54 determine. 55 56 57 58 59 60 2 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 3 of 12 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from 1 2 3 INTRODUCTION 4 5 The NIHR HTA monograph series 6 In the UK, the Health Technology Assessment (HTA) Programme of the National Institute for Health 7 8 Research (NIHR) is a major funder of research that evaluates health care interventions.[1] HTA research 9 takes three main forms: primary research (normally trials), evidence synthesis (systematic reviews, which 10 11 usually include economic modelling), and methodological studies. The research is published in the NIHR 12 HTA Programme’s peer reviewed open access journal, Health Technology Assessment, also known as the 13 14 monograph series. Each monograph contains the full details of a single study, and as the suggested word limit 15 on the monographsFor is 50,000, thispeer enables more detailsreview of the work to be onlyincluded compared to a typical peer 16 17 reviewed journal.[2] Authors are also encouraged to write up the monographs in shorter forms as journal 18 articles to increase dissemination.[3] 19 20 21 In 2013 the HTA monograph series had a journal impact factor (JIF) of 5.116 and was ranked second of 85 22 23 journals in its subject category Health Care Sciences and Services. Research projects are commissioned by the 24 HTA Programme after a prioritisation process to select only the most important. This might be expected to 25 26 lead to the results of the research being well cited. 27 28 29 The role of bibliometrics in assessing research performance in the UK 30 Authors of HTA monographs are usually based in academic units, where citations to published research are 31 32 important as indicators of quality and impact. Citation rates are used for comparing research performance of 33 34 medical schools or individual departments, assessing individual researchers during appointment or promotions http://bmjopen.bmj.com/ 35 processes (including their H-indices), and judging past performance of researchers applying to grant-awarding 36 37 bodies for research funds. 38 39 40 RAND Europe has undertaken substantial bibliometric work for the Department of Health in England, and 41 bibliometrics has increasingly come to be used, in combination with peer review, to help inform the 42 on September 23, 2021 by guest. Protected copyright. 43 Department’s funding decisions. For example, the first stage in the NIHR senior investigators award process is 44 a bibliometric assessment by RAND.[4] 45 46 47 The use of bibliometric data for assessing the performance of universities was considered in a pilot exercise in 48 49 2008 and 2009 for the 2013 Research Excellence Framework. The report concluded that “bibliometrics was 50 not sufficiently robust at this stage to be used formulaically or to replace expert review in the REF. However 51 52 there is considerable scope for citation information to be used to inform expert review”[5]. Subsequently, in 53 April 2014, HEFCE announced a new review of metrics which will build on the pilot exercise and consider 54 55 the role that metrics-based assessment could play in determining quality, impact and other key characteristics 56 of research undertaken in universities.[6] 57 58 59 60 3 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 4 of 12 BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from 1 It is known that there is variation in the robustness of bibliometric indictors between disciplines and the 2 3 comparison of rankings can vary depending on the databases used.[5 7] Therefore, given the increasing use of 4 bibliometric indicators in higher education in the UK, it is important that the methods used and the accuracy 5 6 and coverage of these citation databases are understood. 7 8 9 Citation databases 10 11 The three databases currently used for citation analysis are the Web of Science (WoS), Scopus, and Google 12 Scholar (GS). The oldest of these is the Web of Science™ Core Collection database, produced by Thomson 13 14 Reuters. It indexes approximately 12,000 of the highest impact journals and conference proceedings, covering 15 all disciplines. TheFor Scopus database peer was introduced review by Elsevier in 2004, onlyand covers nearly 21,000 peer- 16 17 reviewed journals titles from a wide range of disciplines, as well as books and conference proceedings. 18 Google Scholar, also introduced in 2004, is freely accessible and covers journals and other scholarly literature 19 20 from a wide range of research areas and formats.