BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from
Bibliometrics of NIHR HTA monographs and their related journal articles
ForJournal: peerBMJ Open review only Manuscript ID: bmjopen-2014-006595
Article Type: Research
Date Submitted by the Author: 10-Sep-2014
Complete List of Authors: Royle, Pamela; Warwick Medical School, Division of Health Sciences Waugh, Norman; university of Warwick, Division of Health Sciences
Primary Subject Health services research Heading:
Secondary Subject Heading: Research methods, Medical publishing and peer review
bibliometrics, health technology assessment, bibliographic databases, Keywords: citations
http://bmjopen.bmj.com/
on September 23, 2021 by guest. Protected copyright.
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 1 of 12 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from
1
2 3 Bibliometrics of NIHR HTA monographs and their related journal articles. 4 5 Corresponding author; 6 7 Dr Pamela Royle 8 9 Division of Health Sciences 10 11 Warwick Medical School 12 13 University of Warwick, Coventry CV4 7AL , UK [email protected] tel: 02476 151762 14 15 For peer review only 16 17 Authors: Pamela Royle and Norman Waugh 18 19 Postal addresses as above. Email: [email protected] 20 21 22 23 MeSH terms: bibliometrics, bibliographic databases, health technology assessment 24 25 Word count: 3944 26 27 28 29 30 31 32 33
34 http://bmjopen.bmj.com/ 35 36 37 38 39 40 41
42 on September 23, 2021 by guest. Protected copyright. 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 1 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 2 of 12 BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from
1 ABSTRACT 2 3 Objectives: To undertake a bibliometric analysis of the UK National Institute for Health Research (NIHR) 4 Health Technology Assessment (HTA) monographs and their related journal articles by: 1) exploring the 5 6 differences in citations to the HTA monographs in Google Scholar (GS), Scopus, and Web of Science (WoS) 7 8 and, 2) comparing Scopus citations to the monographs with their related journal articles. 9 Setting: A retrospective cohort study of 111 HTA monographs published in 2010 and 2011, and their external 10 11 journal articles identified from the NIHR Journals Library website 12 Main outcome measures: Citations to the monographs in GS, Scopus and WoS and to their external journal 13 14 articles in Scopus 15 Results. The numberFor of citations peer varied amongst review the three databases, with only GS having the highest and WoS the 16 17 lowest, but the citation based rankings between the three databases were highly correlated. Overall, 56% of 18 monographs had a related publication, with the highest proportion for primary research (76%) and lowest for 19 20 evidence syntheses (43%). When comparing the differences in the number of citations in Scopus between 21 pairs of monograph publications with each of their related journal articles from the same project, we found 22 23 that monographs received more citations than their journal article versions for evidence syntheses and 24 methodology projects, whereas for primary research, the journal article versions were more highly cited than 25 26 their monograph version. Only 24% of authors citing the monographs and 21% citing the journals were from 27 the UK, implying a large international readership. 28 29 Conclusions: Those using bibliometric data for decisions regarding academic career progression and research 30 funding need to be aware of how using different databases will affect their results. Publishing in an external 31 32 journal as well as a monograph can aid dissemination of the work, but may have less impact than a single 33
34 publication, and could therefore adversely affect an author’s h-index. http://bmjopen.bmj.com/ 35 36 37 Strengths and limitations of this study 38 • First study to perform a detailed bibliometric analysis of a cohort of HTA monographs and their 39 40 related outputs published in external journals. 41 • We compared citations to the HTA monographs, using three citation databases, and compared
42 on September 23, 2021 by guest. Protected copyright. 43 citations by type of evidence. 44 • We chose monographs published in 2010 and 2011 in order to allow adequate time for citations to 45 46 accrue and for the related journal articles to be published. 47 48 • One limitation is that we obtained details of related publications from the HTA website, where the 49 number of related journal publications may be underestimated by 15.8%. 50 51 • A possible limitation is that we used the date that a publication was first added to PubMed as a 52 surrogate for the publication date because the actual date of journal publication is difficult to 53 54 determine. 55 56 57 58
59 60 2 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 3 of 12 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from
1
2 3 INTRODUCTION 4 5 The NIHR HTA monograph series 6 In the UK, the Health Technology Assessment (HTA) Programme of the National Institute for Health 7 8 Research (NIHR) is a major funder of research that evaluates health care interventions.[1] HTA research 9 takes three main forms: primary research (normally trials), evidence synthesis (systematic reviews, which 10 11 usually include economic modelling), and methodological studies. The research is published in the NIHR 12 HTA Programme’s peer reviewed open access journal, Health Technology Assessment, also known as the 13 14 monograph series. Each monograph contains the full details of a single study, and as the suggested word limit 15 on the monographsFor is 50,000, thispeer enables more detailsreview of the work to be onlyincluded compared to a typical peer 16 17 reviewed journal.[2] Authors are also encouraged to write up the monographs in shorter forms as journal 18 articles to increase dissemination.[3] 19 20 21 In 2013 the HTA monograph series had a journal impact factor (JIF) of 5.116 and was ranked second of 85 22 23 journals in its subject category Health Care Sciences and Services. Research projects are commissioned by the 24 HTA Programme after a prioritisation process to select only the most important. This might be expected to 25 26 lead to the results of the research being well cited. 27
28 29 The role of bibliometrics in assessing research performance in the UK 30 Authors of HTA monographs are usually based in academic units, where citations to published research are 31 32 important as indicators of quality and impact. Citation rates are used for comparing research performance of 33
34 medical schools or individual departments, assessing individual researchers during appointment or promotions http://bmjopen.bmj.com/ 35 processes (including their H-indices), and judging past performance of researchers applying to grant-awarding 36 37 bodies for research funds. 38 39 40 RAND Europe has undertaken substantial bibliometric work for the Department of Health in England, and 41 bibliometrics has increasingly come to be used, in combination with peer review, to help inform the
42 on September 23, 2021 by guest. Protected copyright. 43 Department’s funding decisions. For example, the first stage in the NIHR senior investigators award process is 44 a bibliometric assessment by RAND.[4] 45 46 47 The use of bibliometric data for assessing the performance of universities was considered in a pilot exercise in 48 49 2008 and 2009 for the 2013 Research Excellence Framework. The report concluded that “bibliometrics was 50 not sufficiently robust at this stage to be used formulaically or to replace expert review in the REF. However 51 52 there is considerable scope for citation information to be used to inform expert review”[5]. Subsequently, in 53 April 2014, HEFCE announced a new review of metrics which will build on the pilot exercise and consider 54 55 the role that metrics-based assessment could play in determining quality, impact and other key characteristics 56 of research undertaken in universities.[6] 57 58 59 60 3 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 4 of 12 BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from
1 It is known that there is variation in the robustness of bibliometric indictors between disciplines and the 2 3 comparison of rankings can vary depending on the databases used.[5 7] Therefore, given the increasing use of 4 bibliometric indicators in higher education in the UK, it is important that the methods used and the accuracy 5 6 and coverage of these citation databases are understood. 7 8 9 Citation databases 10 11 The three databases currently used for citation analysis are the Web of Science (WoS), Scopus, and Google 12 Scholar (GS). The oldest of these is the Web of Science™ Core Collection database, produced by Thomson 13 14 Reuters. It indexes approximately 12,000 of the highest impact journals and conference proceedings, covering 15 all disciplines. TheFor Scopus database peer was introduced review by Elsevier in 2004, onlyand covers nearly 21,000 peer- 16 17 reviewed journals titles from a wide range of disciplines, as well as books and conference proceedings. 18 Google Scholar, also introduced in 2004, is freely accessible and covers journals and other scholarly literature 19 20 from a wide range of research areas and formats. Its "Cited by" feature provides access to abstracts of articles 21 that have cited the article being viewed. WoS and Scopus are only available via subscription, and both index 22 23 the NIHR HTA monograph series. GS is freely available, but unlike the other two databases, it does not 24 provide a list of sources that it indexes. However, it appears to cover all the NIHR HTA monographs, which 25 26 are also indexed in PubMed. 27
28 29 Aims 30 Our aim was to undertake a bibliometric analysis of the NIHR HTA monographs and their related journal 31 32 articles. The steps were to: 33
34 1. Explore the differences in citations to the HTA monographs in GS, Scopus, and WoS http://bmjopen.bmj.com/ 35 2. Compare citations to the HTA monographs with their related journal articles. 36 37 38 METHODS 39 40 Downloading the HTA monographs. PubMed was searched to identify all of the articles in 2010 (volume 41 14) and 2011 (volume 15) published in the Health Technology Assessment (HTA) monograph series. These
42 on September 23, 2021 by guest. Protected copyright. 43 were downloaded into Endnote and then exported into Excel. We excluded supplements to the monographs 44 that contained summaries of the Evidence Review Groups reports based upon the evidence submission to the 45 46 National Institute for Health and Clinical Excellence (NICE) as part of the single technology appraisal 47 process. The NIHR Journals Library HTA website[8] was used to obtain the research type and details of 48 49 related journal publications for each HTA monograph. 50 51 52 Date added to PubMed. The Entrez Date (EDAT) field indicates the date the article was added to PubMed. 53 The EDAT was noted for each HTA monograph and its related publications. The bibliographic record of each 54 55 related publication was then downloaded from PubMed into Endnote and to Excel. The number of related 56 publications was noted for each HTA report. 57 58 59 60 4 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 5 of 12 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from
1 JIFS and subject categories. The JIFs and subject categories of the journals of the related publication were 2 3 obtained from the ISI Web of Knowledge Journal Citation Reports (JCR) 2012. Subjects were classified as 4 either general (Medicine, General & Internal or Multidisciplinary Sciences) or subject specific (all other 5 6 subject categories). 7 8 9 Citation data. The citations to all HTA monographs and their related publications were obtained from the 10 11 Web of Science Core Collection (WoS), Scopus and Google Scholar (GS); all citation data was collected 12 within the same week (18th July 2014 to 25th July). 13 14 • Citations in WoS. The Cited Reference Search facility in WoS was used to determine the total number 15 of citationsFor to each HTA peer monograph. The review number of citation variants, only the total citations associated 16 17 with the excluded variants, and the number of citations attached to the Full Record in WoS was 18 recorded for each HTA monograph. 19 20 • Citations in Scopus. A search for each HTA monograph and its related journal publication was 21 performed in Scopus. The HTA monograph reports often appeared under two different forms of the 22 23 journal name; i.e. Health Technology Assessment or Health Technology Assessment (Winchester 24 25 England).When this occurred, citations to both forms were combined to ensure they were unique 26 citations. The ‘View Cited by’ and ‘Analyze results’ features were used to obtain detailed data on the 27 28 citing documents for each article. Data on the Source Titles, Author Names, Affiliation names, 29 Countries and Document types were downloaded into Excel for analysis. 30 31 • Citations in GS. GS was searched using the title of the monograph article, and the number of citations 32 from the Cited by number beneath each reference was recorded. 33
34 http://bmjopen.bmj.com/ 35 Data analysis. Descriptive analyses were performed for all variables. Due to the skewed non-normal 36 37 distribution of the citation data, non-parametric statistical tests were performed and median and interquartile 38 range values were reported. The Wilcoxon Mann-Whitney Test was used to analyse the difference in citations 39 40 between two groups, the Kruskal-Wallis rank test was used to analyse the differences in citations between 41 three or more groups, and Spearman’s correlation coefficient was used to test the correlation between citations
42 on September 23, 2021 by guest. Protected copyright. 43 and JIF. All analyses were undertaken in Stata 12.1 (Stata Corporation, College Station, TX, USA) 44
45 46 RESULTS 47 48 1. Bibliometric analysis of the HTA monographs 49 There were 111 monographs in Health Technology Assessment which fulfilled our eligibility criteria: 66 from 50 51 Volume 14 (2010) and 45 from Volume 15 (2011); 58 (52.3%) were evidence syntheses, 42 (37.8%) primary 52 research and 11 (9.9%) methodology. 53 54 55 Comparison of citations using GS, Scopus and WoS. 56 57 The results of the comparison of citations to the monographs in GS, Scopus and WoS by research type are 58 shown in Table 1. 59 60 5 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 6 of 12 BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from
1
2 3 Table 1 Citations to HTA monographs by database and research type 4 5 6 Total Median (IQR) Range 7 8 Google Scholar 9 10 All Research Types 3459 22 (14 to 39) 2 to 206 11 12 Evidence Synthesis 2000 27 (17 to 42) 4 to 121 13 14 Primary 976 16.5 (9 to 26) 2 to 83 15 Methodology For peer483 20review (18 to 48) 9 to 206 only 16 17 Scopus 18 19 All Research Types 2094 13 (7 to 24) 2 to 140 20 21 Evidence Synthesis 1203 16 (10 to 25) 2 to 70 22 23 Primary 608 10 (6 to 17) 3 to 62 24 25 Methodology 283 11 (5 to 25) 4 to 140 26 Web of Science 27 28 All Research Types 1948 13 (7 to 22) 2 to 142 29 30 Evidence Synthesis 1054 14.5 (9 to 22) 3 to 57 31 32 Primary 614 10.5 (2 to 57) 2 to 57 33 Methodology 280 14 (4 to 22) 3 to 142 34 http://bmjopen.bmj.com/ 35 36 Table 1 shows the numbers of citations found using GS was 78% (3459/1948) more than WoS and 65% 37 38 (3459/1948) more than Scopus. Also Scopus found 7% (2094/1948) more than WoS. The median number of 39 citations in each database was highest for evidence syntheses, followed by methodology, and fewest for 40 41 primary research. The differences in citations between research types within each database were statistically
42 on September 23, 2021 by guest. Protected copyright. 43 significant for GS and Scopus (p=0.0216 and p=0.0268 respectively), but not for WoS (p=0.1312). 44 The correlation between the numbers of citations in the three databases was high. The Spearman’s rho for the 45 46 three comparisons was: GS versus Scopus=0.8993, GS versus WoS=0.8804 and Scopus versus WoS =0.8717; 47 p<0.00001 for all comparisons. Therefore the ranking of monographs by number of citations is similar across 48 49 the three databases even though absolute numbers vary. 50 51 52 Cited reference variants to the monographs in the Web of Science. 53 When using the WoS for citation analysis it was noted that 84 (76%) of the monographs had one or more 54 55 citations from cited reference variants that were not linked to the Times Cited number on the ‘Full Record’ 56 page, and were only detected when the ‘Cited Reference Search’ was used. Thirty monographs had citations to 57 58 one unlinked cited reference variant, and the remaining 54 records had between 2 and 12 variants with 59 60 6 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 7 of 12 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from
1 unlinked citations. Overall, 491 citations were unlinked to their Full Record, resulting in 25.5% of the 2 3 citations to the monographs not being attached to their Full Record in the Web of Science Core Collection. 4
5 6 As Scopus does not have an equivalent ‘Cited Reference Search’ function it was not possible to determine the 7 8 unlinked citations due to cited reference variants in this database. The remaining citation analyses were 9 performed using Scopus, unless otherwise stated. 10 11 12 Further results from citation analysis in Scopus. 13 14 An analysis using Scopus of citations to the HTA monographs showed the following: 15 i) The authors citingFor the monographs peer came from review the UK (24.1%), then onlythe USA (17.5%), Germany (5.7%), 16 17 Canada (4.8%), and Australia (4.4%). The remaining 43.5% were from 75 different countries. 18 ii) The journal that most commonly cited the HTA monographs was PLOS ONE (6.8%), followed by the 19 20 citations for the HTA journal (3.5%). The remaining citations came from 78 different journals. 21 iii) The document types that cited the HTA monographs were: 63.4% from articles, 26.5% from reviews, 22 23 2.9% from editorials, with 7.2% from other document types. 24 25 26 2. Bibliometric analysis of journal publications related to the HTA monographs 27 Number of related publications. 28 29 Overall 56% (62) of the monographs had published one or more related journal publication; of these 30 monographs, half (31) had one journal publication, and half had between 2 and 8 related publications. Of the 31 32 monographs that had at least one related journal article, their research types were: 76% primary research, 43% 33
34 evidence synthesis, and 56% methodology. There was no significant difference between citations to the http://bmjopen.bmj.com/ 35 monographs that had a related journal publication and those that did not (p=0.942). 36 37 38 Cited reference variants to the journals in WoS. 39 40 Fewer related journal articles had cited reference variants unlinked to the source items in WoS than was seen 41 for the monographs (see above): i.e. 29% of the journal articles had at least one unlinked citation (only
42 on September 23, 2021 by guest. Protected copyright. 43 viewable in a Cited Reference Search) and the number of variants per article ranged from 1 to 5, also fewer 44 than for the monographs. 45 46 47 Citations to journals by research type. 48 49 Citations to the journals by research type are shown in Table 2. Of the 128 journal publications in Scopus, 45 50 (35%) were from evidence synthesis monographs, 69 (54%) from primary research and 14 (11%) from 51 52 methodology. 53
54 55 56
57 58 59 60 7 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 8 of 12 BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from
1
2 3 Table 2 Scopus citations to the journal publications 4 5 Total 6 citations Medians IQR Range 7 8 All Research Types 9 (n=128) 4850 11 4 to 25 0 to 707 10 11 Evidence 12 Synthesis 13 14 (n=45) 604 6 4 to 22 0 to 73 15 Primary For peer review only 16 17 (n=69) 3358 12 4 to 27 0 to 707 18 Methodology 19 20 (n=14) 888 12 4 to 82 1 to 308 21 22 23 Although primary research and methodology had a higher median number of citations (12 for each) than 24 25 evidence synthesis (6), there was no statistically significant difference (p=0.3516) in citations between the 26 research types. However, the ranges of citations were much wider for primary research and methodology (707 27 28 and 307 respectively) than evidence syntheses (73). 29 30 31 Citations and JIFs. 32 The related publications were most frequently published in the BMJ (16 articles) and the Lancet (6). The 33
34 remaining 112 articles were published in 79 different journals. For the 127 articles that were in a journal with http://bmjopen.bmj.com/ 35 a JIF, the mean JIF was 7.202, and the range was from 0.768 to 51.658. The mean JIFs for each research type 36 37 was: evidence synthesis = 4.698, primary research=9.957 and methodology = 5.040. There was a moderate 38 correlation between the citations to the related journal publications and their JIF; Spearman’s rho = 0.5561 39 40 (p<0.00001). 41
42 on September 23, 2021 by guest. Protected copyright. 43 Countries of authors citing the journal publications. 44 Of the citing authors, 21% came from the United Kingdom, 20% from the United States, then Germany (6%), 45 46 Canada (5%) and Australia (5%) The remaining citations were from 91 different countries. 47 48 49 Difference in the citations between the monograph and journal versions. 50 51 In most cases, monographs were added to PubMed at about the same time (a median of 0.03 months 52 difference, IQR -10.71 to + 0.33 months), but the range was wide i.e. with one journal article appearing 9 53 54 years before its monograph, and one monograph appearing 4 years before the journal article. Such large 55 differences in publication dates mean that some monographs and articles will vary substantially in the time 56 57 they have had to accrue citations. Therefore, we compared numbers of citations to monographs with their 58 59 60 8 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 9 of 12 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from
1 related journal articles in a subset, containing only those 80 pairs of monograph and journal article that were 2 3 published within 12 months of each other. 4
5 6 Overall, there were slightly fewer (3) citations to the journal articles than to their monograph versions, but the 7 8 differences in citations between monographs and their journal versions varied significantly by research types 9 (p=0.0001). For evidence syntheses and methodology the monograph version was more highly cited (by a 10 11 median of 6 and 29 citations respectively) than its journal version(s), whereas for primary research the journal 12 articles were more highly cited than their monograph versions by a median of 4 citations. The difference in 13 14 citations between the monograph and its journal version(s) was highly skewed, but mostly driven by some 15 highly cited primaryFor research articlespeer published inreview general medical journals, only with the citations to the journal 16 17 version greatly exceeding those to their related monographs. The largest difference was to the Lancet article 18 by Peek et al [9] which received 544 citations, compared to its monograph version [10] which received 32 19 20 citations. The top 10 percent (8 articles) of the differences favouring citations to the journal articles all came 21 from primary research published in general medical journals (BMJ =2, Lancet =4, New England Journal of 22 23 Medicine=1, JAMA=1). 24 25 26 DISCUSSION 27 This analysis of 111 HTA monographs and comparison of three citation databases showed that overall GS 28 29 gave 78% more citations than WoS and 65% more than WoS, and Scopus gave 7% more than WoS; however 30 the citations between the databases were highly correlated. When citations to the monographs were analysed 31 32 by research type, all databases showed the highest number of citations to evidence syntheses, then 33
34 methodology, and fewest for primary research. The lower proportion of citations for primary research may be http://bmjopen.bmj.com/ 35 because there were more journal articles published from the primary research projects, with at least one 36 37 journal article from 76% of primary research, 56% from methodology and 43% from evidence syntheses. 38 When overall citations to the journal articles were analysed by research type, with primary research received 39 40 the highest number of citations and evidence syntheses the fewest. A comparison of differences in the number 41 of citations in Scopus between pairs of monograph publications with each of their related journal articles from
42 on September 23, 2021 by guest. Protected copyright. 43 the same project found that the monographs received more citations than their journal article versions for 44 evidence syntheses and methodology projects, but for primary research the journal articles were more highly 45 46 cited than their monograph version. The monographs and journals were both shown to have a large 47 international readership, as 76% and 79% respectively of the citing authors came from outside the UK, with 48 49 the largest percentage of these coming from the USA. Also, the journal that most cited the monographs, PLOS 50 ONE, is based in the USA. 51 52 53 The strengths of this study were that it was the first study to perform a detailed bibliometric analysis of a 54 55 cohort of HTA monographs and their related outputs published in external journals. We compared citations to 56 the HTA monographs, using three citation databases, and compared citations by type of evidence. A limitation 57 58 is that we obtained details of the related publications from the HTA website[8], where the number of related 59 60 9 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 10 of 12 BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from
1 journal publications may be underestimated by 15.8%.[11]. However, this should not affect our results unless 2 3 there is some systematic bias in reporting of external publications. Also, we used the date that a publication 4 was first added to PubMed as a surrogate for the publication date; this was because the actual date of journal 5 6 publications are difficult to determine, as some journals are electronic only and many print journals now make 7 8 their articles available online before appearing in print[12] (these epub ahead of print articles are included in 9 PubMed). However, there may be variation amongst journals with respect to the time taken to be included in 10 11 PubMed. 12 13 14 Although the median number of citations to the monographs was 69% higher for GS compared to Scopus and 15 and WoS, there wasFor a high correlation peer between thereview three databases for the only number of citations, indicating that 16 17 even though numbers of citations differ, the relative rankings are very similar. Higher numbers of citations 18 using GS compared to Scopus and WoS was also found by Kulkarni et al [13] The higher citations using GS is 19 20 understandable, due to the greater range of types of literature it covers, such as non-peer reviewed and grey 21 literature. The high correlation between the ranking of citations using GS and WoS, as observed in our study, 22 23 has also been observed by Kousha et al.[14] Also, in a comparison of the number of citations using WoS and 24 GS for articles by 20 Nobel Prize Winners in four disciplines, Harzing et al found their ranking for Medicine 25 26 to be nearly identical for GS and WoS data, but for the other disciplines the rankings differed between the two 27 databases.[7]. Our data showed that 56% of the monographs had published at least one external journal article, 28 29 but the proportions by research type varied; i.e. 76% primary research, 56% methodology and 43% evidence 30 syntheses. The higher proportion of HTA primary research being published in an external journal was also 31 32 found by Chinnery et al[11] who reported that 62% of primary research and 44% of evidence syntheses 33
34 monographs were published in an external journal. The higher proportion of journal articles coming from http://bmjopen.bmj.com/ 35 primary research may be because journals consider primary research more newsworthy than reviews of 36 37 already published evidence. The monograph versions of evidence syntheses and methodology were more 38 highly cited than the journal versions, whereas journal versions of primary research (especially those in 39 40 general medical journals) were more highly cited than the monographs. Primary research also appeared in 41 journals with higher impact factors.
42 on September 23, 2021 by guest. Protected copyright. 43 44 The monographs have the advantage over the journal versions in that they allow for more complete 45 46 description of the study. However we found that monograph authors could be potentially disadvantaged in a 47 bibliometric analysis if those using the WoS Core Collection do not perform a Cited Reference Search to 48 49 capture all citations. This was because 76% of the monographs in the WoS had at least one cited reference 50 variant unlinked to their full record (this was 2.6 times more frequent than for the journal article), and this 51 52 resulted in 26% of monograph citations being missed unless the much more time consuming method of a 53 Cited Reference Search was used. The major source of unlinked citations to the monographs were due the 54 55 variations in how the page numbers were cited, as the monographs are paginated using both Roman and 56 Arabic numerals. Conversely, publishing in an external journal can improve dissemination of findings. 57 58 Clinicians are likely to prefer a short journal article compared to a 170-page monograph. However, academics 59 60 10 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 11 of 12 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from
1 who publish journal papers from their monograph may spread their citations across more papers and hence 2 3 reduce their h-index; i.e. an academic with an h-index of 30 will increase that by publishing one paper that 4 receives 32 citations, but not by publishing two papers that are cited 16 times each. 5 6 7 8 Future research could involve investigating if the relative citations and rankings between the different citations 9 databases found in this study is generalizable to other disciplines, and which citation database gives the most 10 11 reliable and consistent results. Also, it has been reported that a higher proportion of statistically significant 12 findings are reported in journal articles when compared with the outcomes reported in HTA monographs.[15] 13 14 Therefore, it would be interesting to see whether monographs and journal publications with significant 15 findings are cited Formore often thanpeer those with non-significant review outcomes. only 16 17 18 19 20 CONCLUSIONS 21 Those using bibliometric data for decisions regarding academic career progression and research funding need 22 23 to be aware of how differences amongst databases will affect their results. Publishing in an external journal as 24 well as a monograph can aid dissemination of the work, but may have less impact than a single publication, 25 26 and could therefore adversely affect an author’s h-index. However there does seem to be a clear citation 27 advantage in publishing primary research articles in general medical journals, as well as via the monograph 28 29 series. 30
31 32 33
34 Contribution of authors: PR and NW conceived the study; PR designed the study, extracted the data and http://bmjopen.bmj.com/ 35 carried out all analyses; PR and NW drafted the manuscript and approved the final version. 36 37 Funding: Internal 38 Competing interests: The authors publish reports in the Health Technology Assessment monograph series and 39 40 sometimes as related journal articles. 41
42 on September 23, 2021 by guest. Protected copyright. 43 REFERENCES 44 1. Wright D, Young A, Iserman E, et al. The clinical relevance and newsworthiness of NIHR HTA-funded 45 research: a cohort study. BMJ open 2014;4(5). 46 2. Douet L, Milne R, Anstee S, Habens F, Young A, Wright D. The completeness of intervention descriptions 47 in published National Institute of Health Research HTA-funded trials: a cross-sectional study. BMJ 48 open 2014;4(1):e003713. 49 3. Turner S, Wright D, Maeso R, Cook A, Milne R. Publication rate for funded studies from a major UK 50 health research funder: a cohort study. BMJ open 2013;3(5). 51 4. Ismail S, Nason E, Marjanovic S, Grant J. Bibliometrics as a tool for supporting prospective R&D decision- 52 making in the health sciences: strengths, weaknesses and options for future development: RAND 53 Technical Report Secondary Bibliometrics as a tool for supporting prospective R&D decision-making 54 in the health sciences: strengths, weaknesses and options for future development: RAND Technical 55 Report 2009. http://www.rand.org/pubs/technical_reports/TR685.html (accessed 10/09/2014). 56 5. Higher Education Funding Council for England (HEFCE). Report on the pilot exercise to develop 57 bibliometric indicators for the Research Excellence Framework. Secondary Report on the pilot 58 59 60 11 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 12 of 12 BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from
1 exercise to develop bibliometric indicators for the Research Excellence Framework 2009. 2 http://www.hefce.ac.uk/pubs/year/2009/200939/ (accessed 10/09/2014). 3 6. Higher Education Funding Council for England (HEFCE). Independent review of the role of metrics in 4 research assessment. Secondary Independent review of the role of metrics in research assessment 5 6 2014. http://www.hefce.ac.uk/whatwedo/rsrch/howfundr/metrics/ (accessed 10/09/2014). 7 7. Harzing AW. A preliminary test of Google Scholar as a source for citation data: A longitudinal study of 8 Nobel prize winners. Scientometrics 2013;94(3):1057-75. 9 8. National Institute for Health Research. NIHR Journals Library: Health Technology Assessment. Secondary 10 NIHR Journals Library: Health Technology Assessment 2014. 11 http://www.journalslibrary.nihr.ac.uk/hta (accessed 10/09/2014). 12 9. Peek GJ, Mugford M, Tiruvoipati R, et al. Efficacy and economic assessment of conventional ventilatory 13 support versus extracorporeal membrane oxygenation for severe adult respiratory failure (CESAR): a 14 multicentre randomised controlled trial. Lancet 2009;374(9698):1351-63. 15 10. Peek GJ, ElbourneFor D, Mugford peer M, et al. Randomised review controlled trial onlyand parallel economic evaluation of 16 conventional ventilatory support versus extracorporeal membrane oxygenation for severe adult 17 respiratory failure (CESAR). Health Technol. Assess. 2010;14(35):1-+. 18 11. Chinnery F, Young A, Goodman J, Ashton-Key M, Milne R. Time to publication for NIHR HTA 19 programme-funded research: a cohort study. BMJ open 2013;3(11):e004121. 20 12. Tort AB, Targino ZH, Amaral OB. Rising publication delays inflate journal impact factors. PLoS One 21 2012;7(12):e53374. 22 13. Kulkarni AV, Aziz B, Shams I, Busse JW. Comparisons of citations in web of science, Scopus, and 23 Google Scholar for articles published in general medical journals. JAMA - Journal of the American 24 Medical Association 2009;302(10):1092-96. 25 14. Kousha K, Thelwall M. Google scholar citations and google Web/URL citations: A multi-discipline 26 exploratory analysis. Journal of the American Society for Information Science and Technology 27 2007;58(7):1055-65. 28 15. Matthews GA, Dumville JC, Hewitt CE, Torgerson DJ. Retrospective cohort study highlighted outcome 29 reporting bias in UK publicly funded trials. Journal of clinical epidemiology 2011;64(12):1317-24. 30 31 32 33
34 http://bmjopen.bmj.com/ 35 36 37 38 39 40 41
42 on September 23, 2021 by guest. Protected copyright. 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 12 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from
Bibliometrics of NIHR HTA monographs and their related journal articles
ForJournal: peerBMJ Open review only Manuscript ID: bmjopen-2014-006595.R1
Article Type: Research
Date Submitted by the Author: 23-Dec-2014
Complete List of Authors: Royle, Pamela; Warwick Medical School, Division of Health Sciences Waugh, Norman; university of Warwick, Division of Health Sciences
Primary Subject Health services research Heading:
Secondary Subject Heading: Research methods, Medical publishing and peer review
bibliometrics, health technology assessment, bibliographic databases, Keywords: citations
http://bmjopen.bmj.com/
on September 23, 2021 by guest. Protected copyright.
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 1 of 37 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from
1 2 3 4 5 6 7 8 9 10 Bibliometrics of NIHR HTA monographs and their related 11 journal articles 12 13 14 15 ForJournal: peerBMJ Open review only 16 Manuscript ID: bmjopen-2014-006595 17 18 Article Type: Research 19 20 Date Submitted by the Author: 10-Sep-2014 21 Complete List of Authors: Royle, Pamela; Warwick Medical School, University of Warwick, 22 Waugh, Norman; Warwick Medical School, University of Warwick, 23 24 25 Primary Subject Health services research 26 Heading: 27 28 Secondary Subject Heading: Research methods, Medical publishing and peer review 29 bibliometrics, health technology assessment, bibliographic databases, Keywords: 30 citations 31 32 33
34 http://bmjopen.bmj.com/ 35 36 37 38 39 40 41
42 on September 23, 2021 by guest. Protected copyright. 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 For peer review only -- http://bmjopen.bmj.com/site/about/guidelines.xhtmlhttp://bmjopen.bmj.com/site/about/guidelines.xhtml Page 1 of 12 BMJBMJ Open Open Page 2 of 37 BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from
11 22 33 44 Bibliometrics of NIHR HTA monographs and their related journal articles. 55 Corresponding author; 66 77 Dr Pamela Royle 88 9 9 Division of Health Sciences 110 0 1 1 11 Warwick Medical School 112 2 1 3 13 University of Warwick, Coventry CV4 7AL , UK [email protected] tel: 02476 151762 114 4 115 5 For peer review only 116 6 1 7 17 Authors: Pamela Royle and Norman Waugh 118 8 1 9 19 Postal addresses as above. Email: [email protected] 220 0 221 1 222 2 223 3 MeSH terms: bibliometrics, bibliographic databases, health technology 224 4 225 5 assessment Word count: 3944 226 6 227 7 228 8 229 9 330 0 331 1 332 2 333 3
334 4 http://bmjopen.bmj.com/ 335 5 336 6 337 7 338 8 339 9 440 0 441 1
442 2 on September 23, 2021 by guest. Protected copyright. 443 3 444 4 445 5 446 6 447 7 448 8 449 9 550 0 551 1 552 2 553 3 554 4 555 5 556 6 557 7 558 8 559 9 660 0 1 For peer review only -- http://bmjopen.bmj.com/site/about/guidelines.xhtmlhttp://bmjopen.bmj.com/site/about/guidelines.xhtml Page 3 of 37 ABSTRACT BMJ Open BMJ
Objectives: To undertake a bibliometric analysis of the UK National Institute for Health Research (NIHR) Health Open BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from 11 Technology Assessment (HTA) monographs and their related journal articles by: 1) exploring the differences in 2 2 citations to the HTA monographs in Google Scholar (GS), Scopus, and Web of Science (WoS) and, 2) comparing 33 44 Scopus citations to the monographs with their related journal articles. 5 5 Setting: A study of 111 HTA monographs published in 2010 and 2011, and their external journal articles. 66 77 Main outcome measures: Citations to the monographs in GS, Scopus and WoS and to their external journal 88 articles in Scopus. 99 1010 Results: The number of citations varied amongst the three databases, with GS having the highest and WoS the 111 1 1212 lowest; however the citation based rankings amongst the three databases were highly correlated. Overall, 56% of 1313 monographs had a related publication, with the highest proportion for primary research (76%) and lowest for 1414 1515 evidence synthesesFor (43%). There peer was a large variationreview in how the monographs only were cited, compared to journal 1616 articles, and this resulted in more frequent problems, with unlinked citations in Scopus and WoS. When using 1717 1818 Scopus to compare differences in the number of citations between monograph publications with their related 1919 journal articles from the same project, we found that monographs received more citations than their journal 220 0 221 1 articles for evidence syntheses and methodology projects; by contrast, journal articles related to primary research 222 2 monographs were more highly cited than their monograph. 223 3 224 4 Conclusions: The numbers of citations to the HTA monographs differed considerably between the databases, 2 5 25 but were highly correlated. When a HTA monograph had a journal article from the same study, there were 226 6 227 7 more citations to the journal article for primary research, but more to the monographs for evidence 2 8 28 syntheses. Citations to the related journal articles were more reliably recorded than citations to the HTA 229 9 330 0 monograph series. 331 1
332 2 333 3
334 4 http://bmjopen.bmj.com/ 335 5 336 6 337 7 Strengths and limitations of this study 3 8 38 • First study to perform a detailed bibliometric analysis of a cohort of HTA monographs and 339 9 440 0 their related outputs published in external journals. 441 1 • We compared citations to the HTA monographs, using three citation databases, and
442 2 on September 23, 2021 by guest. Protected copyright. 443 3 compared citations by type of evidence. 444 4 445 5 • We chose monographs published in 2010 and 2011 in order to allow adequate time for citations 446 6 to accrue and for the related journal articles to be published. 447 7 448 8 • One limitation is that we obtained details of related publications from the HTA website, where 4 9 49 the number of related journal publications may be underestimated by 15.8%. 550 0 551 1 • A possible limitation is that we used the date that a publication was first added to PubMed as 552 2 a surrogate for the publication date because the actual date of journal publication is difficult 553 3 554 4 to determine. 555 5 556 6 557 7 558 8 559 9 660 0 2 For peer review only -- http://bmjopen.bmj.com/site/about/guidelines.xhtmlhttp://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ OpenBMJ Open Page 4 of 37 BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from
1 1 22 33 INTRODUCTION 44 55 The NIHR HTA monograph series 66 In the UK, the Health Technology Assessment (HTA) Programme of the National Institute for Health Research (NIHR) is a 77 88 major funder of research that evaluates health care interventions.[1] Set up in 1993, it is the largest and longest running of the 99 NIHR research programmes, and has invested over £647 million over the last 20 years. The Chief Medical Officer for 110 0 111 1 England, Professor Sally Davies, noted in 2013 that there were then 316 active projects, with an expected spend in 2013/14 of 112 2 around £75 million.[2] The HTA Programme supports policy-makers such as the National Institute for Health and Care 113 3 114 4 Excellence, the National Screening Committee and the Department of Health. 115 5 For peer review only 116 6 117 7 HTA research takes three main forms: primary research (normally trials), evidence synthesis (systematic reviews, which 118 8 usually include economic modelling), and methodological studies. The research is published in the NIHR HTA Programme’s 119 9 220 0 peer reviewed open access journal, Health Technology Assessment, also known as the monograph series. Each monograph 221 1 contains the full details of a single study, and as the suggested word limit on the monographs is 50,000, this enables more 222 2 223 3 details of the work to be included compared to a typical peer reviewed journal.[3] Authors are also encouraged to publish 2 4 24 journal articles from the projects described in the monographs to increase dissemination.[4] There may be more than one 225 5 226 6 article from each project; for example, with one on clinical effectiveness and one on the economics. 227 7 228 8 229 9 In 2013 the HTA monograph series had a journal impact factor (JIF) of 5.116 and was ranked second of 85 journals in its 3 0 30 subject category Health Care Sciences and Services. Research projects are commissioned by the HTA Programme after a 331 1 332 2 prioritisation process to select only the most important. This might be expected to lead to the results of the research being 333 3 well cited. 334 4 http://bmjopen.bmj.com/ 335 5 336 6 337 7 The role of bibliometrics in assessing research performance in the UK 338 8 Authors of HTA monographs are usually based in academic units, where citations to published research are important as 339 9 440 0 indicators of quality and impact. Citation rates are used for comparing research performance of medical schools or individual 441 1 departments, assessing individual researchers during appointment or promotions processes (including their h-indices), and
442 2 on September 23, 2021 by guest. Protected copyright. 443 3 judging past performance of researchers applying to grant-awarding bodies for research funds. 444 4 445 5 446 6 RAND Europe has undertaken substantial bibliometric work for the Department of Health in England, and bibliometrics has 447 7 increasingly come to be used, in combination with peer review, to help inform the Department’s funding decisions. For 448 8 449 9 example, the first stage in the NIHR senior investigators award process is a bibliometric assessment by RAND.[5] 550 0 551 1 552 2 The use of bibliometric data for assessing the performance of universities was considered in a pilot exercise in 2008 and 2009 for 5 3 53 the 2013 Research Excellence Framework. The report concluded that “bibliometrics was not sufficiently robust at this stage to be 554 4 555 5 used formulaically or to replace expert review in the REF. However there is considerable scope for citation information to be 5 6 56 used to inform expert review”.[6] Subsequently, in April 2014, HEFCE announced a new review of metrics which will build on 557 7 558 8 the pilot exercise and consider the role that metrics-based assessment could play in determining quality, impact and other key 559 9 characteristics of research undertaken in universities.[7] 660 0 BMJ Open For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 5It of is 37known that there is variation in the robustness of bibliometricBMJ Open indictors between disciplines and the comparison BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from of rankings can vary depending on the databases used.[6 8] Therefore, given the increasing use of bibliometric 1 1 indicators in higher education in the UK, it is important that the methods used and the accuracy and coverage of these 22 33 citation databases are understood. 44 55 66 Citation databases 77 88 The three databases currently used for citation analysis are the Web of Science (WoS), Scopus, and Google Scholar 99 (GS). The oldest of these is the Web of ScienceTM Core Collection database, produced by Thomson Reuters. It indexes 1010 111 1 approximately 12,000 of the highest impact journals and conference proceedings, covering all disciplines. The Scopus 1212 database was introduced by Elsevier in 2004, and covers nearly 21,000 peer-reviewed journals titles from a wide range 1313 1414 of disciplines, as well as books and conference proceedings. Google Scholar, also introduced in 2004, is freely 1515 accessible and covers journalsFor and other peerscholarly literature review from a wide range of researchonly areas and formats. Its "Cited 1616 1717 by" feature provides access to abstracts of articles that have cited the article being viewed. WoS and Scopus are only 1818 available via subscription, and both index the NIHR HTA monograph series. GS is freely available, but unlike the 1919 220 0 other two databases, it does not provide a list of sources that it indexes. However, it appears to cover all the NIHR 221 1 HTA monographs, which are also indexed in PubMed. 222 2 223 3 2 4 24 Aims 225 5 226 6 Our aim was to undertake a bibliometric analysis of the NIHR HTA monographs and their related journal 2 7 27 articles. The steps were to: 228 8 229 9 1. Explore the differences in citations to the HTA monographs in GS, Scopus, and WoS 330 0 2. Compare citations to the HTA monographs with journal articles from the same project. 331 1 332 2 333 3 METHODS 334 4 http://bmjopen.bmj.com/ 335 5 Downloading the HTA monographs 3 6 36 PubMed was searched to identify all of the articles in 2010 (volume 14) and 2011 (volume 15) published in the 337 7 338 8 Health Technology Assessment (HTA) monograph series. These were downloaded into Endnote and then exported 339 9 into Excel. We excluded supplements to the monographs that contained summaries of the Evidence Review Groups 440 0 441 1 reports based upon the evidence submission to the National Institute for Health and Clinical Excellence (NICE) as
442 2 on September 23, 2021 by guest. Protected copyright. 443 3 part of the single technology appraisal process. The NIHR Journals Library HTA website [9] was used to obtain the 444 4 research type and details of related journal publications for each HTA monograph. 445 5 446 6 447 7 We used PubMed (as well as identifying monographs from the citation databases and HTA website) in order to 448 8 449 9 obtain the Entrez Date (EDAT) field, which indicates the date the article was first added to PubMed. The EDAT was 550 0 noted for each HTA monograph and its related publications. The bibliographic record of each related publication was 551 1 552 2 then downloaded from PubMed into Endnote and to Excel. The number of related publications was noted for each 553 3 HTA report. 554 4 555 5 556 6 557 7 4 5 8 58 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 559 9 660 0 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJBMJ Open Open Page 6 of 37 BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from
1 1 JIFS and subject categories 22 33 44 55 The JIFs and subject categories of the journals of the related publication were obtained from the ISI Web of 66 ® Knowledge Journal Citation Reports (JCR) 2012. Subjects were classified as either general (Medicine, 77 88 General & Internal or Multidisciplinary Sciences) or subject specific (all other subject categories). 99 1010 1111 1212 Citation data 1313 1414 The citations to all HTA monographs and their related publications were obtained from the Web of 1515 For peer review only 1616 Science Core Collection (WoS), Scopus and Google Scholar (GS); all citation data was collected within 1717 the same week (18th July 2014 to 25th July). 1818 1919 • Citations in WoS. The Cited Reference Search facility in WoS was used to determine the total 20 20 number of citations to each HTA monograph. The number of citation variants, the total citations associated 2121 2222 with the excluded variants, and the number of citations attached to the Full Record in WoS were recorded for 23 23 each HTA monograph. 2424 2525 • Citations in Scopus. A search for each HTA monograph and its related journal publication was 2626 performed in Scopus. The HTA monograph reports often appeared under two different forms of the journal 2727 2828 name; i.e. Health Technology Assessment or Health Technology Assessment (Winchester England).When this 2929 3030 occurred, citations to both forms were combined to ensure they were unique citations. The ‘View Cited by’ 3131 and ‘Analyze results’ features were used to obtain detailed data on the citing documents for each article. Data 3232 3333 on the Source Titles, Author Names, Affiliation names, Countries and Document types were downloaded
3434 into Excel for analysis. http://bmjopen.bmj.com/ 3535 3636 • Citations in GS. GS was searched using the title of the monograph article, and the number of citations 3737 from the Cited by number beneath each reference was recorded. 3838 3939 4040 41 41 Data analysis
4242 on September 23, 2021 by guest. Protected copyright. 43 43 Descriptive analyses were performed for all variables. Due to the skewed distribution of the citation data, non- 4444 4545 parametric statistical tests were performed and median and interquartile range values were reported. The 46 46 Wilcoxon Mann-Whitney Test was used to analyse the difference in citations between two groups, the Kruskal- 4747 4848 Wallis rank test was used to analyse the differences in citations between three or more groups, and Spearman’s 4949 correlation coefficient was used to compute the correlation between citations and JIF. All analyses were 5050 5151 undertaken in Stata 12.1 (Stata Corporation, College Station, TX, USA) 5252 5353 5454 RESULTS 5555 5656 1. Bibliometric analysis of the HTA monographs 5757 There were 111 monographs in Health Technology Assessment which fulfilled our eligibility criteria: 66 5858 5959 from Volume 14 (2010) and 45 from Volume 15 (2011); 58 (52.3%) were evidence syntheses, 42 (37.8%) 6060 primary research and 11 (9.9%) methodology. For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 7 of 37 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from
11 22 33 44 55 Comparison of citations using GS, Scopus and WoS 66 77 The results of the comparison of citations to the monographs in GS, Scopus and WoS by research type are shown in 88 99 Table 1. 1010 111 1 1212 Table 1 Citations to HTA monographs by database and research type
1313 1414 Total Median (IQR) Range 1515 For peer review only 1616 Google Scholar 17 17 All Research Types 3459 22 (14 to 39) 2 to 206 1818 1919 Evidence Synthesis 2000 27 (17 to 42) 4 to 121 220 0 221 1 Primary 976 16.5 (9 to 26) 2 to 83 222 2 223 3 Methodology 483 20 (18 to 48) 9 to 206 224 4 225 5 Scopus 226 6 227 7 All Research Types 2094 13 (7 to 24) 2 to 140 228 8 229 9 Evidence Synthesis 1203 16 (10 to 25) 2 to 70 330 0 Primary 608 10 (6 to 17) 3 to 62 331 1 332 2 Methodology 283 11 (5 to 25) 4 to 140 333 3
334 4 Web of Science http://bmjopen.bmj.com/ 3 5 35 All Research Types 1948 13 (7 to 22) 2 to 142 336 6 337 7 Evidence Synthesis 1054 14.5 (9 to 22) 3 to 57 338 8 339 9 Primary 614 10.5 (2 to 57) 2 to 57 440 0 441 1 Methodology 280 14 (4 to 22) 3 to 142
442 2 on September 23, 2021 by guest. Protected copyright. 443 3 444 4 Table 1 shows the number of citations found using GS was 78% (3459/1948) more than WoS and 65% (3459/1948) 445 5 446 6 more than Scopus. Also Scopus found 7% (2094/1948) more than WoS. 447 7 448 8 449 9 However we noted an apparent anomaly in that the number of citations for primary research was higher in Web of 550 0 Science (608) than Scopus (614). Although this difference was small, we would expect the number of citations from 551 1 552 2 Scopus to be at least equal to those from Web of Science, as the former covers more journals than the latter. On 553 3 554 4 investigation we found that 12 of the 42 primary research articles had more citations in WoS than Scopus. The 5 5 55 largest difference was for citations to Shaw et al[10], with 25 more citations in WoS (34 in WoS and 9 in Scopus). 556 6 557 7 We checked the full Scopus bibliographic records (which lists the Cited References), of each of these 25 citing 558 8 559 9 articles, and observed that in all cases, in contrast to the other cited references, the title of the Shaw article was 660 0 missing, and there was no hyperlink on the reference. This miscoding meant that these 25 citations were not linked to the Shaw et al monograph.For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 8 of 37 BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from In response to one of the referees for this paper, we include below Table 2 showing some examples of differences in 1 numbers of citations to the five 2010 monographs with the largest numbers of citations. Table 2 shows that the biggest 2 3 difference is between GS and the other two with sometimes little difference between WoS and Scopus. 4
5 6 Table 2. Comparisons of citations to the most frequently cited HTA monographs published in 2010 7 Publication Google Scopus Web of 8 scholar Science 9 10 Song et al.[12] 206 140 142 11 12 Waugh et al.[13] 121 67 40 13 14 15 Clar et al.[14] For peer review101 70 only 52 16 17 Cooper et al.[15] 92 67 54 18 19 Mowatt et al.[16] 86 63 57 20 21 22 23 24Correlation between citations between the databases 25 26The correlation between the rankings of the monographs on the basis of number of citations in the three 27databases was high, even though the absolute numbers differed. The Spearman’s rho for the three 28 29comparisons was: GS versus Scopus=0.8993, GS versus WoS=0.8804 and Scopus versus WoS =0.8717; 30p<0.00001 for all comparisons. 31 32 33Differences in citations between research types 34 http://bmjopen.bmj.com/ 35We analyzed the differences in the median number of citations in each database by research type, and 36found in all databases it was highest for evidence syntheses, followed by methodology, and fewest for 37 38primary research. These differences in citations between research types within each database were 39 40statistically significant for GS and Scopus (p=0.0216 and p=0.0268 respectively), but not for WoS 41(p=0.1312).
42 on September 23, 2021 by guest. Protected copyright. 43 44Cited reference variants to the monographs in the Web of Science 45 46A Basic Search to access articles in the Web of Science opens a screen within the WoS Core Collection, which has a 47 48field called Times Cited, and shows the number of times the article has been cited. However citations can also be 49investigated using the more time consuming method of a Cited Reference Search. When using the WoS for citation 50 51analysis it was noted that 84 (76%) of the monographs had one or more citations from cited reference variants that 52 53were not linked to the Times Cited number on the Full Record page, and were only detected when the Cited 54 Reference Search was used. Table 3, using the example of citations to the HTA monograph by Ashfaq et al[11], 55 56shows the variant citations typically seen in the WoS when using a WoS Cited Reference Search. 57 58 59 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 9 of 37 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from
1 Table 3. Example of the results for a Web of Science Cited Reference Search for a HTA monograph 2 Authors Journal Year Volume Issue Page Citing View 3 number numbers Articles Record 4 5 Ashfaq, K HEALTH TECHNOL 2010 14 54 1 7 View 6 ASSES Record in 7 Web of 8 Science 9 Core 10 Collection 11 12Ashfaq, K HLTH TECHNOL 2010 14 54 1 13 ASSESS 14Ashfaq, K HEALTH TECHNOL 2010 14 54 11 1 15 ASSES For peer review only 16Ashfaq, K Health Technol Assess 2010 14 54 ii 3 View 17 Record in 18 19 Medline 20Ashfaq, K HLTH TECHNOL ASSES 2010 14 54 ix 2 21 22 23The HTA monograph recommends the report to be cited as: “Ashfaq K, Yahaya I, Hyde C, Andronis L, Barton P, 24Bayliss S, et al. Clinical effectiveness and cost-effectiveness of stem cell transplantation in the management of acute 25 26leukaemia: a systematic review. Health Technol Assess 2010;14(54)”. This form is used in row 2 in Table 2. 27 However PubMed requires page numbers when indexing journals (the HTA monographs are indexed as journal 28 29articles in the bibliographic databases). The situation is complicated by the monographs having the first pages with 30 31Roman numerals followed by the rest in Arabic numbering. So this monograph is referenced in PubMed as; “Health 32Technol Assess. 2010 Dec;14(54):iii-iv, ix-xi, 1-141”. 33
34 http://bmjopen.bmj.com/ 35 36The first row of data lists the number of citing articles from the WoS Core Collection and records only 7 citations that 37are linked to this record: therefore if a Basic Search is done, it will appear that this article has only been cited 7 times. 38 39The second to fifth rows are additional citations found using the Cited Reference Search, and show variants in how 40 41the monograph has been cited, with variations in abbreviations of the title (with Health and Hlth, and Asses and
42Assess), and in page numbering. These rows show that half the citations are missed if only a Basic Search is used, on September 23, 2021 by guest. Protected copyright. 43 44because these citations are not linked to the record in the Core Collection. 45 46 47Overall, thirty monographs had citations to one unlinked cited reference variant, and the remaining 54 48 49records had between 2 and 12 variants with unlinked citations. This resulted in a total of 25.5% (491) citations 50 to the monographs being unlinked to their Full Record in the Web of Science Core Collection. 51 52 53 As Scopus does not have an equivalent Cited Reference Search function it was not possible to determine the unlinked 54 55citations due to cited reference variants in this database. The remaining citation analyses were performed using Scopus, 56 57unless otherwise stated. 58 59 60Further results from citation analysis in Scopus An analysis of citations to theFor HTA peer monographs review only showed - http://bmjopen.bmj.com/site/about/guidelines.xhtml the following: i) The authors citing the monographs came from the UK (24.1%),BMJ thenOpen the USA (17.5%), Germany (5.7%), Canada Page 10 of 37 BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from (4.8%), and Australia (4.4%). The remaining 43.5% were from 75 different countries. 1 1 ii) The journal that most commonly cited the HTA monographs was PLOS ONE (6.8%), followed by the citations for 22 33 the HTA journal (3.5%). The remaining citations came from 78 different journals. 44 iii) The document types that cited the HTA monographs were: 63.4% from articles, 26.5% from reviews, 2.9% from 55 66 editorials, with 7.2% from other document types. 77
88 99 1010 111 1 1212 2. Bibliometric analysis of journal publications related to the HTA monographs 1313 1414 1515 Number of related publicationsFor peer review only 1616 1717 Overall 56% (62) of the monographs had published one or more related journal publication; of these monographs, half 1818 (31) had one journal publication, and half had between 2 and 8 related publications. Of the monographs that had at least 1919 220 0 one related journal article, their research types were: 76% primary research, 43% evidence synthesis, and 56% 221 1 methodology. There was no significant difference between citations to the monographs that had a related journal 222 2 223 3 publication and those that did not (p=0.942). 224 4 225 5 226 6 Cited reference variants to the journals in WoS 2 7 27 Fewer of the related journal articles had cited reference variants unlinked to the source items in WoS than was seen for 228 8 229 9 the monographs (see above): i.e. 29% of the journal articles had at least one unlinked citation (only viewable in a Cited 3 0 30 Reference Search) and the number of variants per article ranged from 1 to 5, also fewer than for the monographs. 331 1 332 2 333 3 Citations to the related journal articles by research type 334 4 http://bmjopen.bmj.com/ 335 5 The number of citations to the related journal articles, broken down by research type, are shown in Table 4. Of the 128 336 6 journal publications in Scopus, 45 (35%) were from evidence synthesis monographs, 69 (54%) from primary research and 337 7 338 8 14 (11%) from methodology. 339 9 440 0 441 1
442 2 on September 23, 2021 by guest. Protected copyright. 443 3 444 4 Table 4 Scopus citations to the related ournal publications
445 5 446 6 Total 4 7 47 citations Medians IQR Range 448 8 449 9 All Research Types 550 0 (n=128) 4850 11 4 to 25 0 to 707 551 1 5 2 52 Evidence 553 3 554 4 Synthesis 555 5 (n=45) 604 6 4 to 22 0 to 73 556 6 557 7 Primary 558 8 (n=69) 3358 12 4 to 27 0 to 707 559 9 660 0 Methodology (n=14) For 888peer review 12 only 4 to- http://bmjopen.bmj.com/site/about/guidelines.xhtml 82 1 to 308
PageAlthough 11 of 37 primary research and methodology journal articles hadBMJ a Openhigher median number of citations BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from (12 for each) than evidence synthesis (6), there was no statistically significant difference (p=0.3516) in 1 citations between the research types. However, the ranges of citations were much wider for primary 2 research and methodology (707 and 307 respectively) than evidence syntheses (73). 3 4 5 Citations and JIFs 6 The related publications were most frequently published in the BMJ (16 articles) and the Lancet (6). The 7 8 remaining 112 articles were published in 79 different journals. For the 127 articles that were in a journal 9 with a JIF, the mean JIF was 7.202, and the range was from 0.768 to 51.658. The mean JIFs for each 10 11 research type were: evidence synthesis = 4.698, primary research=9.957 and methodology = 5.040. There 12 was a moderate correlation between the citations to the related journal publications and their JIF; 13 14 Spearman’s rho = 0.5561 (p<0.00001). 15 For peer review only 16 17 Countries of authors citing the journal publications 18 Of the citing authors, 21% came from the United Kingdom, 20% from the United States, then Germany 19 20 (6%), Canada (5%) and Australia (5%) The remaining citations were from 91 different countries. 21 22 23 24Difference in the citations between the monograph and journal versions 25 26In most cases, monographs were added to PubMed at about the same time (a median of 0.03 months difference, IQR - 27 2810.71 to + 0.33 months), but the range was wide i.e. with one journal article appearing 9 years before its monograph, and 29one monograph appearing 4 years before the journal article. Such large differences in publication dates mean that some 30 31monographs and articles will vary substantially in the time they have had to accrue citations. Therefore, we compared 32numbers of citations to monographs with their related journal articles in a subset, containing only those 80 pairs of 33
34monograph and journal article that were published within 12 months of each other. http://bmjopen.bmj.com/ 35 36 37Overall, there were slightly fewer (3) citations to the related journal articles than to their monographs, but these differences 38in citations varied significantly by research types (p=0.0001). For evidence syntheses and methodology the monograph was 39 40more frequently cited (by a median of 6 and 29 citations respectively) than its related journal article; this is in contrast to 41primary research, where the journal articles were more frequently cited than their monographs by a median of 4 citations.
42 on September 23, 2021 by guest. Protected copyright. 43The differences in citations between the monograph and its related journal articles were highly skewed, but mostly driven by 44some highly cited primary research articles published in general medical journals, with the citations to the journal article 45 46greatly exceeding those to their related monographs. The largest difference was to the Lancet article by Peek et al[17]which 47 received 544 citations, compared to its monograph[18] which received 32 citations. The top 10 percent (8 articles) of the 48 49differences favouring citations to the journal articles all came from primary research published in general medical journals 50 (BMJ =2, Lancet =4, New England Journal of Medicine=1, JAMA=1). 51 52 53 54DISCUSSION 55This comparison of three citation databases showed that overall GS gave 78% more citations than WoS and 65% more 56 57than WoS, and Scopus gave 7% more than WoS; however the rankings of numbers of citations amongst the databases 58were highly correlated. When citations to the 111 HTA monographs were analysed by research type, all databases showed 59 60the highest number of citations to evidence syntheses, then methodology, and fewest for primary research. The lower proportion of citations for primaryFor peer research review monographs only - http://bmjopen.bmj.com/site/about/guidelines.xhtml may be because there were more journal articles published from the primary research projects, with at least one journal article fromBMJ 76% Open of primary research, compared to 56% from Page 12 of 37 BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from methodology and 43% from evidence syntheses. 1 1 22 When overall citations to the related journal articles were analysed by research type, primary research received the 33 44 highest number of citations, and evidence syntheses the fewest. A comparison of differences in the number of citations in 55 66 Scopus between monograph publications paired with each of their related journal articles from the same project, found 77 that the monographs received more citations than their related journal article(s) for evidence syntheses and methodology 88 99 projects, but for primary research the journal articles were more highly cited than their monograph version. The 1010 monographs and journals were both shown to have a large international readership, as 76% and 79% respectively of the 111 1 1212 citing authors came from outside the UK, with the largest percentage of these coming from the USA. Also, the journal 13 13 that most cited the monographs, PLOS ONE, is based in the USA. 1414 1515 For peer review only 1616 1717 The strengths of this study were that it was the first study to perform a detailed bibliometric analysis of a cohort of HTA 18 18 monographs and their related outputs published in external journals. We compared citations to the HTA monographs, 1919 220 0 using three citation databases, and compared citations by type of evidence. A limitation is that we obtained details of the 221 1 222 2 related journal publications from the HTA website[9], where the number of publications may be underestimated by 223 3 15.8%.[19] However, this should not affect our results unless there is some systematic bias in reporting of external 224 4 225 5 publications. 226 6 227 7 228 8 229 9 Also, we used the date that a publication was first added to PubMed as a surrogate for the publication 3 0 30 date; this was because the actual date of journal publications are difficult to determine, as some journals 331 1 332 2 are electronic only and many print journals now make their articles available online before appearing in 3 3 33 print[20] (these epub ahead of print articles are included in PubMed). However, there may be variation 334 4 http://bmjopen.bmj.com/ 335 5 amongst journals with respect to the time taken to be included in PubMed. 336 6 337 7 338 8 Although the absolute number of citations to the individual monographs usually differed significantly 339 9 between GS, Scopus and WoS (e.g. the Waugh et al monograph[13] received 121 citations in GS, but 440 0 441 1 67 in Scopus, and 40 in WoS), the rankings of the monographs based on the citations were highly 4 2 42 on September 23, 2021 by guest. Protected copyright. correlated between the three databases. In the recent responses to the call for evidence for the review 443 3 444 4 of the role of metrics in the assessment of research[7] it was reported that 17 universities were using 445 5 446 6 Scopus, 15 using Google Scholar, and 18 using WoS. Hence it is important that individuals quoting 447 7 citation numbers for their papers (e.g. on applications for jobs, promotion or grants) specify the 448 8 449 9 database they used, to ensure that there is a fair comparison between researchers from different 550 0 institutions 551 1 552 2 553 3 Higher numbers of citations using GS compared to Scopus and WoS was also found by Kulkarni et al 554 4 555 5 [21] This is understandable, due to the greater range of types of literature GS covers, such as non-peer 556 6 reviewed and grey literature. The high correlation between the ranking of citations using GS and WoS, as 557 7 558 8 observed in our study, has also been observed by Kousha et al.[21] Also, in a comparison of the number 5 9 59 of citations using WoS and GS for articles by 20 Nobel Prize Winners in four disciplines, Harzing et al[8] 660 0 found their ranking for Medicine to be nearly identical for GS and WoS data, but for the other disciplines For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml the rankings differed between the two databases. PageOur 13 data of 37showed that 56% of the monographs had published atBMJ least Open one external journal article, but the BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from proportions by research type varied; i.e. 76% primary research, 56% methodology and 43% evidence 1 syntheses. The higher proportion of HTA primary research being published in an external journal was also 2 3 found by Chinnery et al[19] who reported that 62% of primary research and 44% of evidence syntheses 4 monographs were also published in an external journal. This higher proportion of journal articles coming 5 6 from primary research may be because journals consider primary research more newsworthy than reviews 7 of already published evidence. In our study, the monographs of evidence syntheses and methodology were 8 9 more highly cited than their related journal articles, whereas journal articles of primary research 10 11 (especially those in general medical journals) were more highly cited than the monographs. Primary 12 research also appeared in journals with higher impact factors. 13 14 15 The monographs have the advantageFor over their peer related journal reviewarticles in that they allow only for more complete 16 17 description of the study. However we found that monograph authors could be potentially disadvantaged in 18 a bibliometric analysis if those using the WoS Core Collection do not perform a Cited Reference Search to 19 20 capture all citations. This was because 76% of the monographs in the WoS had at least one cited reference 21 variant unlinked to their full record (this was 2.6 times more frequent than for the related journal articles), 22 23 and this resulted in 26% of monograph citations being missed unless the much more time consuming 24 method of a Cited Reference Search was used. The major source of unlinked citations to the monographs in 25 26 WoS appeared to be due to variations in how the journal name, issue and page numbers (the monographs 27 are paginated using both Roman and Arabic numerals) were cited. 28
29 30We also noted inaccuracies in citations to monographs in Scopus. Our detailed analysis of one monograph for the 31 32reasons for the missing citations (i.e. those that were in WoS but not Scopus) showed incomplete entries of the cited 33monographs in Scopus and missing citations links to the monograph. Such errors were not noted for the other articles 34 http://bmjopen.bmj.com/ 35cited alongside the monographs. We speculated that the number of variations in the form in which HTA journal is 36 37cited contributed to the much higher rate of unlinked citations to the monographs in Scopus. 38 39 40 41Publishing in an external journal can improve dissemination of findings. Clinicians are likely to prefer a short journal article
42 on September 23, 2021 by guest. Protected copyright. compared to a 170-page monograph. However, academics who publish journal papers from their monograph may spread 43 44their citations across more papers and hence reduce their h-index; i.e. an academic with an h-index of 30 will increase that 45 46by publishing one paper that receives 32 citations, but not by publishing two papers that are cited 16 times each. The h- 47index is of course only one of a number of citation indicators, and has advantages and disadvantages (summarised in the 48 49RAND report [5]), but the h-index is easily available, and we know it is being used in medical schools. 50 51 52 53The decision on publication strategy will depend on several factors: the key audience, dissemination and citations. 54Publishing primary research in general medical journals appears to be a good strategy for maximising citations The 55 56HTA monograph series is entirely open access which might be thought to increase numbers of citations. However 57Davis[22] reported that in a randomised trial, open access articles were cited no more often than subscription-access 58 59ones. 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Problems with indexing in WoS have been reported by a CanadianBMJ study Open exploring the use of bibliometric indicators for Page 1214 of of 12 37 rating researchers.[23] The Universities UK report also noted that not all citations are collated, partly because authors use BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from 11 a variety of abbreviations for journal titles.[24] Meanwhile, people using citation databases for bibliometric 22 33 assessments should know about possible sources of error in citation databases, and how best to use them, or should 4 4 seek advice from experts. 55 66 77 Future research could investigate if the relative citations and rankings between the different citations databases found in 88 99 this study are generalizable to other disciplines, and which citation database gives the most reliable and consistent results. 1010 1111 Also, it has been reported that a higher proportion of statistically significant findings are reported in journal articles when 1212 compared with the outcomes reported in HTA monographs.[25] Therefore, it would be interesting to see whether 1313 1414 monographs and journal publications with significant findings are cited more often than those with non-significant 1515 outcomes. For peer review only 1616 1717 1818 1919 20 20CONCLUSIONS 2121 2222 The numbers of citations to the HTA monographs differed considerably between the databases, but rankings were highly 23 23correlated. When a HTA monograph had a journal article published from the same study, there were more citations 2424 2525 to the related journal article for primary research, but more to the monographs than their related journal articles for 26 26evidence syntheses. Citations to related journal articles were more reliably recorded than citations to the HTA 2727 2828 monograph series, which showed many variant citation forms, leading to unlinked citations. Those using bibliometric 29 29data for decisions regarding academic career progression and research funding need to be aware of how differences 3030 3131 amongst databases will affect their results. Publishing in an external journal as well as a monograph can aid 3232 dissemination of the work and increase overall citations to the work; alternatively, if the related journal article is cited 3333
3434 instead of the monograph, it could reduce the citations to the monograph. However there does seem to be a clear citation http://bmjopen.bmj.com/ 3535 advantage in publishing primary research articles in general medical journals, as well as via the monograph series. 3636 3737 3838 3939 4040 4141 Contribution of authors: PR and NW conceived the study; PR designed the study, extracted the data and carried out 4242 on September 23, 2021 by guest. Protected copyright. 4343 all analyses; PR and NW drafted the manuscript and approved the final version. 4444 4545 Funding: Internal
4646 Competing interests: The authors publish reports in the Health Technology Assessment monograph series and sometimes 4747 4848 as related journal articles. 4949 5050 5151 REFERENCES 52 52 5353 5454 5555 1. Wright D, Young A, Iserman E, et al. The clinical relevance and newsworthiness of NIHR HTA-funded research: a 5656 5757 cohort study. BMJ open 2014;4(5). 5858 2. Davies S. NIHR and the HTA Programme. Secondary NIHR and the HTA Programme 2013. 5959 6060 http://www.profbriefings.co.uk/nihrhta2013/presentations/Plenary%20Day%201%20Sally%20Davies.pdf (accessed 21/12/2014).For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page3. Douet 15 of L,37 Milne R, Anstee S, Habens F, Young A, WrightBMJ D. The Open completeness of intervention descriptions in BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from published National Institute of Health Research HTA-funded trials: a cross-sectional study. BMJ open 1 2014;4(1):e003713. 2 3 4. Turner S, Wright D, Maeso R, Cook A, Milne R. Publication rate for funded studies from a major UK health 4 research funder: a cohort study. BMJ open 2013;3(5). 5 6 5. Ismail S, Nason E, Marjanovic S, Grant J. Bibliometrics as a tool for supporting prospective R&D decision-making 7 8 in the health sciences: strengths, weaknesses and options for future development: RAND Technical Report 9 Secondary Bibliometrics as a tool for supporting prospective R&D decision-making in the health sciences: 10 11 strengths, weaknesses and options for future development: RAND Technical Report 2009. 12 13 http://www.rand.org/pubs/technical_reports/TR685.html (accessed 10/09/2014). 146. Higher Education Funding Council for England (HEFCE). Report on the pilot exercise to develop bibliometric 15 For peer review only 16 indicators for the Research Excellence Framework. Secondary Report on the pilot exercise to develop 17 bibliometric indicators for the Research Excellence Framework 2009. 18 19 http://www.hefce.ac.uk/pubs/year/2009/200939/ (accessed 10/09/2014). 20 217. Higher Education Funding Council for England (HEFCE). Independent review of the role of metrics in research 22 assessment. Secondary Independent review of the role of metrics in research assessment 2014. 23 24 http://www.hefce.ac.uk/whatwedo/rsrch/howfundr/metrics/ (accessed 10/09/2014). 25 268. Harzing AW. A preliminary test of Google Scholar as a source for citation data: A longitudinal study of Nobel 27 prize winners. Scientometrics 2013;94(3):1057-75. 28 299. National Institute for Health Research. NIHR Journals Library: Health Technology Assessment. Secondary NIHR 30 Journals Library: Health Technology Assessment 2014. http://www.journalslibrary.nihr.ac.uk/hta (accessed 31 32 10/09/2014). 33
3410. Shaw L, Rodgers H, Price C, et al. BoTULS: a multicentre randomised controlled trial to evaluate the clinical http://bmjopen.bmj.com/ 35 effectiveness and cost-effectiveness of treating upper limb spasticity due to stroke with botulinum toxin type 36 37 A. Health Technol. Assess. 2010;14(26):1-+. 38 3911. Ashfaq K, Yahaya I, Hyde C, et al. Clinical effectiveness and cost-effectiveness of stem cell transplantation in the 40 management of acute leukaemia: a systematic review. Health Technol. Assess. 2010;14(54):1-+. 41
4212. Song F, Parekh S, Hooper L, et al. Dissemination and publication of research findings: an updated review of on September 23, 2021 by guest. Protected copyright. 43 related biases. Health Technol. Assess. 2010;14(8):1-+. 44 4513. Waugh N, Cummins E, Royle P, et al. Newer agents for blood glucose control in type 2 diabetes: systematic 46 47 review and economic evaluation. Health Technol. Assess. 2010;14(36):1-+. 4814. Measuring the impact of knowledge: A comparison of Web of Science and google scholar; 2009. 49 5015. Cooper K, Squires H, Carroll C, et al. Chemoprevention of colorectal cancer: systematic review and economic 51 52 evaluation. Health Technol. Assess. 2010;14(32):1-+. 5316. Mowatt G, Zhu S, Kilonzo M, et al. Systematic review of the clinical effectiveness and cost-effectiveness of 54 55 photodynamic diagnosis and urine biomarkers (FISH, ImmunoCyt, NMP22) and cytology for the detection 56 and follow-up of bladder cancer. Health Technol. Assess. 2010;14(4):1-+. 57 5817. Peek GJ, Mugford M, Tiruvoipati R, et al. Efficacy and economic assessment of conventional ventilatory support 59 60 versus extracorporeal membrane oxygenation for severe adult respiratory failure (CESAR): a multicentre randomised controlledFor peer trial. review Lancet only 2009; - http://bmjopen.bmj.com/site/about/guidelines.xhtml374(9698):1351-63. 18. Peek GJ, Elbourne D, Mugford M, et al. Randomised controlledBMJ Open trial and parallel economic evaluation of Page 16 of 37 BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from conventional ventilatory support versus extracorporeal membrane oxygenation for severe adult respiratory 1 failure (CESAR). Health Technol. Assess. 2010;14(35):1-+. 2 3 19. Chinnery F, Young A, Goodman J, Ashton-Key M, Milne R. Time to publication for NIHR HTA programme- 4 funded research: a cohort study. BMJ open 2013;3(11):e004121. 5 6 20. Tort AB, Targino ZH, Amaral OB. Rising publication delays inflate journal impact factors. PLoS One 7 8 2012;7(12):e53374. 9 21. Kulkarni AV, Aziz B, Shams I, Busse JW. Comparisons of citations in web of science, Scopus, and Google 10 11 Scholar for articles published in general medical journals. JAMA - Journal of the American Medical 12 13 Association 2009;302(10):1092-96. 1422. Davis PM. Open access, readership, citations: a randomized controlled trial of scientific journal publishing. 15 For peer review only 16 FASEB journal : official publication of the Federation of American Societies for Experimental Biology 17 2011;25(7):2129-34. 18 1923. Campbell D, Picard-Aitken M, Côté G, et al. Bibliometrics as a performance measurement tool for research 20 21 evaluation: The case of research funded by the national cancer institute of Canada. American Journal of 22 Evaluation 2010;31(1):66-83. 23 2424. Universities UK. The use of bibliometrics to measure research quality in UK higher education institutions. 25 26 Secondary The use of bibliometrics to measure research quality in UK higher education institutions 27 15/12/2014 2007. http://www.universitiesuk.ac.uk/highereducation/Documents/2007/Bibliometrics.pdf 28 29 (accessed 29/08/2014). 30 25. Matthews GA, Dumville JC, Hewitt CE, Torgerson DJ. Retrospective cohort study highlighted outcome reporting 31 32 bias in UK publicly funded trials. Journal of clinical epidemiology 2011;64(12):1317-24. 33
34 http://bmjopen.bmj.com/ 35 36 37 38 39 40 41
42 on September 23, 2021 by guest. Protected copyright. 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 17 of 37 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from
1 2 3 4 5 6 7 8 9 10 Bibliometrics of NIHR HTA monographs and their related 11 journal articles 12 13 14 15 ForJournal: peerBMJ Open review only 16 Manuscript ID: bmjopen-2014-006595 17 18 Article Type: Research 19 20 Date Submitted by the Author: 10-Sep-2014 21 Complete List of Authors: Royle, Pamela; Warwick Medical School, University of Warwick 22 Medical School, Division of Health Sciences Waugh, Norman; 23 Warwick Medical School, Uuniversity of Warwick, Division of Health 24 Sciences 25 Primary Subject Health services research 26 Heading: 27 28 Secondary Subject Heading: Research methods, Medical publishing and peer review 29 bibliometrics, health technology assessment, bibliographic databases, Keywords: 30 citations 31 32 33
34 http://bmjopen.bmj.com/ 35 36 37 38 39 40 41
42 on September 23, 2021 by guest. Protected copyright. 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 For peer review only -- http://bmjopen.bmj.com/site/about/guidelines.xhtmlhttp://bmjopen.bmj.com/site/about/guidelines.xhtml Page 1 of 12 BMJBMJ Open Open Page 18 of 37 BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from
11 22 33 44 Bibliometrics of NIHR HTA monographs and their related journal articles. 55 Corresponding author; 66 77 Dr Pamela Royle 88 9 9 Division of Health Sciences 110 0 1 1 11 Warwick Medical School 112 2 1 3 13 University of Warwick, Coventry CV4 7AL , UK [email protected] tel: 02476 151762 114 4 115 5 For peer review only 116 6 1 7 17 Authors: Pamela Royle and Norman Waugh 118 8 1 9 19 Postal addresses as above. Email: [email protected] 220 0 221 1 222 2 223 3 MeSH terms: bibliometrics, bibliographic databases, health technology 224 4 225 5 assessment Word count: 3944 226 6 227 7 228 8 229 9 330 0 331 1 332 2 333 3
334 4 http://bmjopen.bmj.com/ 335 5 336 6 337 7 338 8 339 9 440 0 441 1
442 2 on September 23, 2021 by guest. Protected copyright. 443 3 444 4 445 5 446 6 447 7 448 8 449 9 550 0 551 1 552 2 553 3 554 4 555 5 556 6 557 7 558 8 559 9 660 0 1 For peer review only -- http://bmjopen.bmj.com/site/about/guidelines.xhtmlhttp://bmjopen.bmj.com/site/about/guidelines.xhtml Page 19 of 37 ABSTRACT BMJ Open BMJ
Objectives: To undertake a bibliometric analysis of the UK National Institute for Health Research (NIHR) Health Open BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from 11 Technology Assessment (HTA) monographs and their related journal articles by: 1) exploring the differences in 2 2 citations to the HTA monographs in Google Scholar (GS), Scopus, and Web of Science (WoS) and, 2) comparing 33 44 Scopus citations to the monographs with their related journal articles. 5 5 Setting: A retrospective cohort study of 111 HTA monographs published in 2010 and 2011, and their external 66 77 journal articles identified from the NIHR Journals Library website. 88 Main outcome measures: Citations to the monographs in GS, Scopus and WoS and to their external journal 99 1010 articles in Scopus. 111 1 1212 Results.: The number of citations varied amongst the three databases, with GS having the highest and WoS the 1313 lowest;, however but the citation based rankings amongstbetween the three databases were highly correlated. . 1414 1515 Overall, 56% ofFor monographs peerhad a related publication, review with the highest only proportion for primary research (76%) 1616 and lowest for evidence syntheses (43%). There was a large variation in how the monographs were cited, 1717 1818 compared to journal articles, and this resulted in more frequent problems, with unlinked citations in Scopus and 1919 WoS compared to journal articles. When using Scopus to compare ing the differences in the number of citations 220 0 221 1 in Scopus between pairs of monograph publications with with each of their related journal articles from the same 222 2 project, we found that monographs received more citations than their journal article versionsarticles for evidence 223 3 224 4 syntheses and methodology projects;, whereas by contrast, journal articles related to primary research 2 5 25 monographs were more highly cited than their monograph. 226 6 227 7 . .Conclusions: The numbers of citations to the HTA monographs differed considerably between the databases, 2 8 28 but were highly correlated. When a HTA monograph had a journal article from the same study, there were 229 9 330 0 more citations to the journal versions article for primary research, but more to the monographs for 3 1 31 evidence syntheses. Citations to the related journal versions articles were more reliably recorded than 332 2 333 3 citations to the HTA monograph series., which showed many variant citation forms, and hence unlinked
334 4 http://bmjopen.bmj.com/ 335 5 336 6 337 7 Strengths and limitations of this study 3 8 38 • First study to perform a detailed bibliometric analysis of a cohort of HTA monographs and 339 9 440 0 their related outputs published in external journals. 441 1 • We compared citations to the HTA monographs, using three citation databases, and
442 2 on September 23, 2021 by guest. Protected copyright. 443 3 compared citations by type of evidence. 444 4 445 5 • We chose monographs published in 2010 and 2011 in order to allow adequate time for citations 446 6 to accrue and for the related journal articles to be published. 447 7 448 8 • One limitation is that we obtained details of related publications from the HTA website, where 4 9 49 the number of related journal publications may be underestimated by 15.8%. 550 0 551 1 • A possible limitation is that we used the date that a publication was first added to PubMed as 552 2 a surrogate for the publication date because the actual date of journal publication is difficult 553 3 554 4 to determine. 555 5 556 6 557 7 558 8 559 9 660 0 2 For peer review only -- http://bmjopen.bmj.com/site/about/guidelines.xhtmlhttp://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ OpenBMJ Open Page 20 of 37 BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from
1 1 22 33 INTRODUCTION 44 55 The NIHR HTA monograph series 66 In the UK, the Health Technology Assessment (HTA) Programme of the National Institute for Health Research (NIHR) is a 77 88 major funder of research that evaluates health care interventions.[1] Set up in 1993, it is the largest and longest running of the 99 NIHR research programmes, and has invested over £647 million over the last 20 years. The Chief Medical Officer for 110 0 111 1 England, Professor Sally Davies, noted in 2013 that there were then 316 active projects, with an expected spend in 2013/14 of 112 2 around £75 million.[2]. The HTA Programme supports policy-makers such as the National Institute for Health and Care 113 3 114 4 Excellence, the National Screening Committee and the Department of Health. 115 5 For peer review only 116 6 117 7 HTA research takes three main forms: primary research (normally trials), evidence synthesis (systematic reviews, which 118 8 usually include economic modelling), and methodological studies. The research is published in the NIHR HTA Programme’s 119 9 220 0 peer reviewed open access journal, Health Technology Assessment, also known as the monograph series. Each monograph 221 1 contains the full details of a single study, and as the suggested word limit on the monographs is 50,000, this enables more 222 2 223 3 details of the work to be included compared to a typical peer reviewed journal.[3] Authors are also encouraged to 2 4 24 publishwrite up the monographs in shorter forms as journal articles from the projects described in the monographs to increase 225 5 226 6 dissemination.[4] There may be more than one article from each project;, for example, with one on clinical effectiveness and 2 7 27 one on the economics. 228 8 229 9 3 0 30 In 2013 the HTA monograph series had a journal impact factor (JIF) of 5.116 and was ranked second of 85 journals in its 331 1 332 2 subject category Health Care Sciences and Services. Research projects are commissioned by the HTA Programme after a 333 3 prioritisation process to select only the most important. This might be expected to lead to the results of the research being 334 4 http://bmjopen.bmj.com/ 335 5 well cited. 336 6 337 7 338 8 The role of bibliometrics in assessing research performance in the UK 339 9 440 0 Authors of HTA monographs are usually based in academic units, where citations to published research are important as 441 1 indicators of quality and impact. Citation rates are used for comparing research performance of medical schools or individual
442 2 on September 23, 2021 by guest. Protected copyright. 443 3 departments, assessing individual researchers during appointment or promotions processes (including their Hh-indices), and 444 4 judging past performance of researchers applying to grant-awarding bodies for research funds. 445 5 446 6 447 7 RAND Europe has undertaken substantial bibliometric work for the Department of Health in England, and bibliometrics has 448 8 449 9 increasingly come to be used, in combination with peer review, to help inform the Department’s funding decisions. For 550 0 example, the first stage in the NIHR senior investigators award process is a bibliometric assessment by RAND.[5] 551 1 552 2 5 3 53 The use of bibliometric data for assessing the performance of universities was considered in a pilot exercise in 2008 and 2009 for 554 4 555 5 the 2013 Research Excellence Framework. The report concluded that “bibliometrics was not sufficiently robust at this stage to be 5 6 56 used formulaically or to replace expert review in the REF. However there is considerable scope for citation information to be 557 7 558 8 used to inform expert review”.[6] Subsequently, in April 2014, HEFCE announced a new review of metrics which will build on 559 9 the pilot exercise and consider the role that metrics-based assessment could play in determining quality, impact and other key 660 0 characteristics of research undertaken in universities.[7] For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 21 of 37 3 BMJ Open For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 For peer review only 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33
34 http://bmjopen.bmj.com/ 35 36 37 38 39 40 41
42 on September 23, 2021 by guest. Protected copyright. 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ OpenBMJ Open Page 22 of 37 BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from
1 1 It is known that there is variation in the robustness of bibliometric indictors between disciplines and the comparison 22 33 of rankings can vary depending on the databases used.[6 8] Therefore, given the increasing use of bibliometric 44 indicators in higher education in the UK, it is important that the methods used and the accuracy and coverage of these 55 66 citation databases are understood. 77 88 99 Citation databases 1010 111 1 The three databases currently used for citation analysis are the Web of Science (WoS), Scopus, and Google Scholar 1212 (GS). The oldest of these is the Web of ScienceTM Core Collection database, produced by Thomson Reuters. It indexes 1313 1414 approximately 12,000 of the highest impact journals and conference proceedings, covering all disciplines. The Scopus 1515 database was introduced byFor Elsevier in 2004,peer and covers nearlyreview 21,000 peer-reviewed only journals titles from a wide range 1616 1717 of disciplines, as well as books and conference proceedings. Google Scholar, also introduced in 2004, is freely 1818 accessible and covers journals and other scholarly literature from a wide range of research areas and formats. Its "Cited 1919 220 0 by" feature provides access to abstracts of articles that have cited the article being viewed. WoS and Scopus are only 221 1 available via subscription, and both index the NIHR HTA monograph series. GS is freely available, but unlike the 222 2 223 3 other two databases, it does not provide a list of sources that it indexes. However, it appears to cover all the NIHR 2 4 24 HTA monographs, which are also indexed in PubMed. 225 5 226 6 227 7 Aims 228 8 229 9 Our aim was to undertake a bibliometric analysis of the NIHR HTA monographs and their related journal 3 0 30 articles. The steps were to: 331 1 332 2 1. Explore the differences in citations to the HTA monographs in GS, Scopus, and WoS 3 3 33 334 4 2. Compare citations to the HTA monographs with their related journal articles from the same project.. http://bmjopen.bmj.com/ 335 5 336 6 337 7 METHODS 338 8 Downloading the HTA monographs 339 9 PubMed was searched to identify all of the articles in 2010 (volume 14) and 2011 (volume 15) published in the 440 0 441 1 Health Technology Assessment (HTA) monograph series. These were downloaded into Endnote and then exported
442 2 on September 23, 2021 by guest. Protected copyright. 443 3 into Excel. We excluded supplements to the monographs that contained summaries of the Evidence Review Groups 444 4 reports based upon the evidence submission to the National Institute for Health and Clinical Excellence (NICE) as 445 5 446 6 part of the single technology appraisal process. The NIHR Journals Library HTA website [9] was used to obtain the 447 7 research type and details of related journal publications for each HTA monograph. 448 8 449 9 550 0 We usedDate added to PubMed (as well as identifying monographs from the citation databases and HTA 551 1 552 2 website) in order to obtain. T the Entrez Date (EDAT) field, which indicates the date the article was first added to 5 3 53 PubMed. . The EDAT was noted for each HTA monograph and its related publications. The bibliographic record of 554 4 555 5 each related publication was then downloaded from PubMed into Endnote and to Excel. The number of related 5 6 56 publications was noted for each HTA report. 557 7 558 8 559 9 660 0 4 For peerFor review peer only review - http://bmjopen.bmj.com/site/about/guidelines.xhtml only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 23 of 37 BMJBMJ Open Open BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from
1 1 JIFS and subject categories. 22 33 44 55 The JIFs and subject categories of the journals of the related publication were obtained from the ISI Web of 66 ® Knowledge Journal Citation Reports (JCR) 2012. Subjects were classified as either general (Medicine, 77 88 General & Internal or Multidisciplinary Sciences) or subject specific (all other subject categories). 99 1010 1111 1212 Citation data 1313 1414 The citations to all HTA monographs and their related publications were obtained from the Web of 1515 For peer review only 1616 Science Core Collection (WoS), Scopus and Google Scholar (GS); all citation data was collected within 1717 the same week (18th July 2014 to 25th July). 1818 1919 • Citations in WoS. The Cited Reference Search facility in WoS was used to determine the total 20 20 number of citations to each HTA monograph. The number of citation variants, the total citations associated 2121 2222 with the excluded variants, and the number of citations attached to the Full Record in WoS wereas recorded 23 23 for each HTA monograph. 2424 2525 • Citations in Scopus. A search for each HTA monograph and its related journal publication was 2626 performed in Scopus. The HTA monograph reports often appeared under two different forms of the journal 2727 2828 name; i.e. Health Technology Assessment or Health Technology Assessment (Winchester England).When this 2929 3030 occurred, citations to both forms were combined to ensure they were unique citations. The ‘View Cited by’ 3131 and ‘Analyze results’ features were used to obtain detailed data on the citing documents for each article. Data 3232 3333 on the Source Titles, Author Names, Affiliation names, Countries and Document types were downloaded
3434 into Excel for analysis. http://bmjopen.bmj.com/ 3535 3636 • Citations in GS. GS was searched using the title of the monograph article, and the number of citations 3737 from the Cited by number beneath each reference was recorded. 3838 3939 4040 41 41 Data analysis.
4242 on September 23, 2021 by guest. Protected copyright. 43 43 Descriptive analyses were performed for all variables. Due to the skewed non-normal distribution of the 4444 4545 citation data, non-parametric statistical tests were performed and median and interquartile range values were 46 46 reported. The Wilcoxon Mann-Whitney Test was used to analyse the difference in citations between two 4747 4848 groups, the Kruskal-Wallis rank test was used to analyse the differences in citations between three or more 4949 groups, and Spearman’s correlation coefficient was used to computetest the correlation between citations and 5050 5151 JIF. All analyses were undertaken in Stata 12.1 (Stata Corporation, College Station, TX, USA) 5252 5353 5454 RESULTS 5555 5656 1. Bibliometric analysis of the HTA monographs 5757 There were 111 monographs in Health Technology Assessment which fulfilled our eligibility criteria: 66 5858 5959 from Volume 14 (2010) and 45 from Volume 15 (2011); 58 (52.3%) were evidence syntheses, 42 (37.8%) 6060 primary research and 11 (9.9%) methodology. For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 24 of 37 BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from
1 2 3
4 5 Comparison of citations using GS, Scopus and WoS. 6 7 The results of the comparison of citations to the monographs in GS, Scopus and WoS by research type are shown in 8 9 Table 1. 10 11 125 13For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 14 15 For peer review only 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33
34 http://bmjopen.bmj.com/ 35 36 37 38 39 40 41
42 on September 23, 2021 by guest. Protected copyright. 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 25 of 37 BMJ OpenPage 6 of 12Table 1 Citations to HTABMJ monographs Open by database and research type
Total Median (IQR) Range BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from 11 Google Scholar 22 33 All Research Types 3459 22 (14 to 39) 2 to 206 44 55 Evidence Synthesis 2000 27 (17 to 42) 4 to 121 66 77 Primary 976 16.5 (9 to 26) 2 to 83 8 8 Methodology 483 20 (18 to 48) 9 to 206 99 1010 Scopus 111 1 12 12 All Research Types 2094 13 (7 to 24) 2 to 140 1313 1414 Evidence Synthesis 1203 16 (10 to 25) 2 to 70 1515 For peer review only 1616 Primary 608 10 (6 to 17) 3 to 62 1717 1818 Methodology 283 11 (5 to 25) 4 to 140 1919 Web of Science 220 0 221 1 All Research Types 1948 13 (7 to 22) 2 to 142 222 2 223 3 Evidence Synthesis 1054 14.5 (9 to 22) 3 to 57 224 4 225 5 Primary 614 10.5 (2 to 57) 2 to 57 226 6 Methodology 280 14 (4 to 22) 3 to 142 227 7 228 8 229 9 330 0 Table 1 shows the numbers of citations found using GS was 78% (3459/1948) more than WoS and 65% (3459/1948) more 3 1 31 than Scopus. Also Scopus found 7% (2094/1948) more than WoS. The median number of citations in each database was 332 2 333 3 highest for evidence syntheses, followed by methodology, and fewest for primary research.
334 4 http://bmjopen.bmj.com/ 335 5 336 6 However we noted an apparent anomaly in that the number of citations for primary research was higher in Web of 337 7 338 8 Science (608) than Scopus (614). Although this difference was small, we would expect the number of citations from 3 9 39 Scopus to be at least equal to those from Web of Science, as the former covers more journals than the latter. On 440 0 441 1 investigation we found that 12 of the 42 primary research articles had more citations in WoS than Scopus. The largest
442 2 on September 23, 2021 by guest. Protected copyright. 443 3 difference was for citations to Shaw et al[10], with 25 more citations in WoS (34 in WoS and 9 in Scopus). We checked 4 4 44 the full Scopus bibliographic records (which lists the Ccited Rreferences), of each of these 25 citing articles, and 445 5 446 6 observed that in all cases, in contrast to the other cited references, the title of the Shaw article was missing, and there 447 7 448 8 was no hyperlink on the reference. This miscoding meant that these 25 citations were not linked to the Shaw et al 449 9 monograph. 550 0 551 1 552 2 In response to one of the referees for this paper, we include below Table 2 showing some examples of differences in 553 3 554 4 numbers of citations to the five 2010 monographs with the largest numbers of citations. Table 2 shows that the biggest 555 5 556 6 difference is between GS and the other two with sometimes little difference between WoS and Scopus. 5 7 57 5 8 58 Table 2. Comparisons of citations to the most frequently cited HTA monographs published in 2010 559 9 660 0 Publication Google Scopus Web of scholar Science For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Song et al.[12] BMJ Open 206 140 142 Page 26 of 37 BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from Waugh et al.[13] 121 67 40 1 2 3 Clar et al.[14] 101 70 52 4 5 Cooper et al.[15] 92 67 54 6 7 Mowatt et al.[16] 86 63 57 8 9 10 11 12 Correlation between citations between the databases. 13 14 The differences in citations between research types within each database were statistically significant for GS 15 and Scopus (p=0.0216For and p=0.0268 peer respectively), review but not for WoS (p=0.1312). only 16 The correlation between the rankings of the monographs byon the basis of number of citations in the three 17 18 databases was high, even though the absolute numbers differed. The Spearman’s rho for the three 19 comparisons was: GS versus Scopus=0.8993, GS versus WoS=0.8804 and Scopus versus WoS =0.8717; 20 21 p<0.00001 for all comparisons. 22 23 24 Differences in citations between research types 25 26 We analyzed the differences in Tthe median number of citations in each database by research type, and 27 found in all databases it was highest for evidence syntheses, followed by methodology, and fewest for 28 29 primary research. TThese differences in citations between research types within each database were 30 31 statistically significant for GS and Scopus (p=0.0216 and p=0.0268 respectively), but not for WoS 32 (p=0.1312). 33
34 http://bmjopen.bmj.com/ 35 36 37 38 39 Cited reference variants to the monographs in the Web of Science. 40 41 A Basic Search to access articles in the Web of Science opens a screen within the WoS Core Collection,
42 which has a field called Times Cited, and shows the number of times the article has been cited. on September 23, 2021 by guest. Protected copyright. 43 44 However citations can also be investigated using the more time consuming method of a Cited Reference 45 Search. .When using the WoS for citation analysis it was noted that 84 (76%) of the monographs had 46 47 one or more citations from cited reference variants that were not linked to the Times Cited number on 48 the Full Record page, and were only detected when the ‘Cited Reference Search’ was used. shows an 49 50 example. Thirty monographs had citations to one unlinked cited reference variant, and the remaining 54 51 records had between 2 and 12 variants with 52 53 6 54 55 For peer review only - 56 57 58 59 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 27 of 37 Page 7 of 12 BMJ Open BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from . Overall, 491 citations were unlinked to their Full Record, resulting in 25.5% of the citations 11 22 to the monographs not being attached to their Full Record in the Web of Science Core 3 3 Collection. 44 55 Table 3, using the example of citations to the HTA monograph by Ashfaq et al[11], shows the variant 66 77 citations typically seen in the WoS when using a WoS Cited Reference Search. 88 99 1010 1111 1212 1313 1414 1515 For peer review only 1616 1717 1818 1919 2020 2121 2222 2323 2424 2525 2626 2727 2828 2929 3030 3131 3232 3333
3434 http://bmjopen.bmj.com/ 3535 3636 3737 3838 3939 4040 4141 42 42 on September 23, 2021 by guest. Protected copyright. 4343 4444 4545 4646 4747 4848 4949 5050 5151 5252 5353 5454 5555 5656 5757 5858 5959 6060 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 28 of 37 BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from
1 Table 3. Example of the results for a Web of Science Cited Reference Search for a HTA monograph 2 3 4 5 6 Authors Journal Year Volume Issue Page Citing View 7 number numbers Articles Record 8 Ashfaq, K HEALTH TECHNOL 2010 14 54 1 7 View 9 ASSES Record in 10 Web of 11 12 Science 13 Core 14 Collection 15Ashfaq, K HLTH TECHNOL For2010 peer 14 54review 1 only 16 ASSESS 17 Ashfaq, K HEALTH TECHNOL 2010 14 54 11 1 18 19 ASSES 20Ashfaq, K Health Technol Assess 2010 14 54 ii 3 View 21 Record in 22 Medline 23Ashfaq, K HLTH TECHNOL ASSES 2010 14 54 ix 2 24 25 26The HTA monograph recommends the report to be cited as: “Ashfaq K, Yahaya I, Hyde C, Andronis L, Barton P, 27 28Bayliss S, et al. Clinical effectiveness and cost-effectiveness of stem cell transplantation in the management of acute 29leukaemia: a systematic review. Health Technol Assess 2010;14(54)”. This form is used in row 2 in Table 2. 30 31However PubMed requires page numbers when indexing journals (the HTA monographs are indexed as journal 32 33articles in the bibliographic databases). The situation is complicated by the monographs having the first pages with 34Roman numerals followed by the rest in Arabic numbering. So this monograph is referenced in PubMed as; http://bmjopen.bmj.com/ 35 36 “Health Technol Assess. 2010 Dec;14(54):iii-iv, ix-xi, 1-141”. 37 38 39The first row of data islists the number of citing articles from the WoS Core Collection and records only 7 citations 40 41that are linked to this record:, therefore if a Basic Search is done, it will appear that this article has only been cited 7
42 on September 23, 2021 by guest. Protected copyright. 43times. The second to fifth rows are additional citations found using the Cited Reference Search, and show variants in 44how the monograph has been cited, with variations in abbreviations of the title (, with Health and Hlth, and Asses and 45 46Assess), and in page numbering. These rows show that half the citations are missed if only a Basic Search is used, 47because these citations are not linked to the record in the Core Collection. 48 49 50 51Overall, thirty monographs had citations to one unlinked cited reference variant, and the remaining 54 52 records had between 2 and 12 variants with unlinked citations. This resulted in a total of 25.5% (491) citations 53 54to the monographs being unlinked to their Full Record in the Web of Science Core Collection. 55 56 57As Scopus does not have an equivalent Cited Reference Search function it was not possible to determine the unlinked 58 59citations due to cited reference variants in this database. The remaining citation analyses were performed using Scopus, 60unless otherwise stated. For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 29 of 37 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from
1 Further results from citation analysis in Scopus. 2 3 An analysis using Scopus of citations to the HTA monographs showed the following: 4 i) The authors citing the monographs came from the UK (24.1%), then the USA (17.5%), Germany (5.7%), Canada 5 6 (4.8%), and Australia (4.4%). The remaining 43.5% were from 75 different countries. 7 ii) The journal that most commonly cited the HTA monographs was PLOS ONE (6.8%), followed by the citations for 8 9 the HTA journal (3.5%). The remaining citations came from 78 different journals. 10 11iii) The document types that cited the HTA monographs were: 63.4% from articles, 26.5% from reviews, 2.9% from 12 editorials, with 7.2% from other document types. 13 14 15 For peer review only 16 17 18 19 20 212. Bibliometric analysis of journal publications related to the HTA monographs 22 23 24Number of related publications. 25 26Overall 56% (62) of the monographs had published one or more related journal publication; of these monographs, half (31) 27had one journal publication, and half had between 2 and 8 related publications. Of the monographs that had at least one 28 29related journal article, their research types were: 76% primary research, 43% evidence synthesis, and 56% methodology. 30 There was no significant difference between citations to the monographs that had a related journal publication and those 31 32that did not (p=0.942). 33
34 http://bmjopen.bmj.com/ 35Cited reference variants to the journals in WoS. 36 37Fewer of the related journal articles had cited reference variants unlinked to the source items in WoS than was seen for 38the monographs (see above): i.e. 29% of the journal articles had at least one unlinked citation (only viewable in a Cited 39 40Reference Search) and the number of variants per article ranged from 1 to 5, also fewer than for the monographs. 41
42 on September 23, 2021 by guest. Protected copyright. 43Citations to the related journal articles by research type. 44The number of citations to the related journal articles, broken down by research type, are shown in Table 42. Of the 128 45 46journal publications in Scopus, 45 (35%) were from evidence synthesis monographs, 69 (54%) from primary research and 4714 (11%) from methodology. 48 49 50 51 52 53 54 55 56 57 7 58 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 59 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ OpenBMJ Open Page 8Page of 12 30 of 37 BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from
11 22 33 Table 42 Scopus citations to the related ournal publications 44 55 Total 66 citations Medians IQR Range 77 88 All Research Types 9 9 (n=128) 4850 11 4 to 25 0 to 707 1010 111 1 Evidence 1212 Synthesis 1313 1414 (n=45) 604 6 4 to 22 0 to 73 15 15 Primary For peer review only 1616 1717 (n=69) 3358 12 4 to 27 0 to 707 1818 Methodology 1919 220 0 (n=14) 888 12 4 to 82 1 to 308 221 1 222 2 223 3 224 4 Although primary research and methodology journal articles had a higher median number of citations (12 for each) 225 5 than evidence synthesis (6), there was no statistically significant difference (p=0.3516) in citations between the 2 6 26 research types. However, the ranges of citations were much wider for primary research and methodology (707 and 227 7 228 8 307 respectively) than evidence syntheses (73). 229 9 3 0 30 Citations and JIFs. 331 1 332 2 The related publications were most frequently published in the BMJ (16 articles) and the Lancet (6). The remaining 333 3 112 articles were published in 79 different journals. For the 127 articles that were in a journal with a JIF, the mean 334 4 http://bmjopen.bmj.com/ 335 5 JIF was 7.202, and the range was from 0.768 to 51.658. The mean JIFs for each research type wereas: evidence 3 6 36 synthesis = 4.698, primary research=9.957 and methodology = 5.040. There was a moderate correlation between the 337 7 338 8 citations to the related journal publications and their JIF; Spearman’s rho = 0.5561 (p<0.00001). 339 9 440 0 441 1 Countries of authors citing the journal publications.
4 2 on September 23, 2021 by guest. Protected copyright. 42 Of the citing authors, 21% came from the United Kingdom, 20% from the United States, then Germany (6%), 443 3 444 4 Canada (5%) and Australia (5%) The remaining citations were from 91 different countries. 445 5 446 6 447 7 448 8 Difference in the citations between the monograph and journal versions. 449 9 550 0 In most cases, monographs were added to PubMed at about the same time (a median of 0.03 months difference, 551 1 552 2 IQR -10.71 to + 0.33 months), but the range was wide i.e. with one journal article appearing 9 years before its 553 3 monograph, and one monograph appearing 4 years before the journal article. Such large differences in publication 554 4 555 5 dates mean that some monographs and articles will vary substantially in the time they have had to accrue citations. 556 6 Therefore, we compared numbers of citations to monographs with their 557 7 558 8 8 559 9 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 660 0 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Page 31 of 37 Page 9 of 12 BMJ Open BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from related journal articles in a subset, containing only those 80 pairs of monograph and journal article that were published 11 1 within 12 months of each other. 222 333 444 Overall, there were slightly fewer (3) citations to the related journal articles than to their monographs, but these differences in 555 666 citations varied significantly by research types (p=0.0001). For evidence syntheses and methodology the monograph was more 777 frequentlyhighly cited (by a median of 6 and 29 citations respectively) than its related journal article; this is in contrast to 888 999 primary research, where the journal articles were more frequentlyhighly cited than their monographs by a median of 4 citations. 101010 11111 1 The differences in citations between the monograph and its related journal articles wereas highly skewed, but mostly driven by 121212 some highly cited primary research articles published in general medical journals, with the citations to the journal article greatly 131313 141414 exceeding those to their related monographs. The largest difference was to the Lancet article by Peek et al[17]which received 151515 544 citations, comparedFor to its monograph[18]peer which review received 32 citations. only The top 10 percent (8 articles) of the differences 161616 171717 favouring citations to the journal articles all came from primary research published in general medical journals (BMJ =2, Lancet 181818 =4, New England Journal of Medicine=1, JAMA=1). 191919 22020 0 221 1 21 DISCUSSION 22222 2 22323 3 This analysis of 111 HTA monographs and comparison of three citation databases showed that overall GS gave 78% more 224 4 24 citations than WoS and 65% more than WoS, and Scopus gave 7% more than WoS; however the rankings of numbers of 22525 5 22626 6 citations amongstbetween the databases were highly correlated. When citations to the 111 HTA monographs were analysed 227 7 27 by research type, all databases showed the highest number of citations to evidence syntheses, then methodology, and fewest 22828 8 22929 9 for primary research. The lower proportion of citations for primary research monographs may be because there were more 330 0 30 journal articles published from the primary research projects, with at least one journal article from 76% of primary research, 33131 1 33232 2 compared to 56% from methodology and 43% from evidence syntheses. 333 3 33 33434 4 http://bmjopen.bmj.com/ When overall citations to the related journal articles were analysed by research type,with primary research received the 33535 5 33636 6 highest number of citations, and evidence syntheses the fewest. A comparison of differences in the number of citations in 33737 7 33838 8 Scopus between monograph publications paired with each of their related journal articles from the same project, found that 33939 9 the monographs received more citations than their related journal article(s) for evidence syntheses and methodology projects, 44040 0 44141 1 but for primary research the journal articles were more highly cited than their monograph version. The monographs and
442 2 on September 23, 2021 by guest. Protected copyright. 42 journals were both shown to have a large international readership, as 76% and 79% respectively of the citing authors came 44343 3 44444 4 from outside the UK, with the largest percentage of these coming from the USA. Also, the journal that most cited the 44545 5 monographs, PLOS ONE, is based in the USA. 44646 6 44747 7 44848 8 44949 9 The strengths of this study were that it was the first study to perform a detailed bibliometric analysis of a cohort of HTA 550 0 50 monographs and their related outputs published in external journals. We compared citations to the HTA monographs, using 55151 1 55252 2 three citation databases, and compared citations by type of evidence. A limitation is that we obtained details of the related 55353 3 55454 4 journal publications from the HTA website[9], where the number of publications may be underestimated by 15.8%.[19] 55555 5 However, this should not affect our results unless there is some systematic bias in reporting of external publications. 55656 6 55757 7 55858 8 55959 9 journal publications may be underestimated by 15.8%[19]. However, this should not affect our results unless 66060 0 there is some systematic bias in reporting of external publications. Also, we used the date that a publication For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml was first added to PubMed as a surrogate for the publication date; this was because the actual date of journal publications are difficult to determine, as some journalsBMJ Open are electronic only and many print journals now Page 32 of 37 BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from make their articles available online before appearing in print[20] (these epub ahead of print articles are 1 included in PubMed). However, there may be variation amongst journals with respect to the time taken to be 2 3 included in PubMed. 4 5 6 Although the median number of citations to the monographs was 69% higher for GS compared to Scopus and 7 and WoS, there was a high correlation between the three databases for the number of citations, indicating that 8 9 even though numbers of citations differ, the relative rankings are very similar. Although the absolute number 10 11 of citations to the individual monographs usually differed significantly between GS, Scopus and WoS (e.g. the 12 Waugh et al monograph[13] received 121 citations in GS, but 67 in Scopus, and 40 in WoS), the 13 14 rankings of the monographs based on the citations were highly correlated between the three databases. In the 15 recent responses toFor the call for peer evidence for the review review of the role of metricsonly in the assessment of 16 17 research[7] it was reported that 17 universities were using Scopus, 15 using Google Scholar, and 18 18 using WoS. Hence it is important that individuals quoting citation numbers for their papers (e.g. on 19 20 applications for jobs, promotion or grants) specify the database they used, to ensure that there is a fair 21 comparison between researchers from different institutions 22 23 24 Higher numbers of citations using GS compared to Scopus and WoS was also found by Kulkarni et al [21] 25 26 The higher citations using GS This is understandable, due to the greater range of types of literature it GS 27 covers, such as non-peer reviewed and grey literature. The high correlation between the ranking of citations 28 29 using GS and WoS, as observed in our study, has also been observed by Kousha et al.[21] Also, in a 30 comparison of the number of citations using WoS and GS for articles by 20 Nobel Prize Winners in four 31 32 disciplines, Harzing et al[8] found their ranking for Medicine to be nearly identical for GS and WoS data, but 33 for the other disciplines the rankings differed between the two databases. 34 http://bmjopen.bmj.com/ 35 36 37 [7]. Our data showed that 56% of the monographs had published at least one external journal article, but the 38 proportions by research type varied; i.e. 76% primary research, 56% methodology and 43% evidence 39 40 syntheses. The higher proportion of HTA primary research being published in an external journal was also 41 found by Chinnery et al[19] who reported that 62% of primary research and 44% of evidence syntheses
42 on September 23, 2021 by guest. Protected copyright. 43 monographs were also published in an external journal. This higher proportion of journal articles coming from 44 primary research may be because journals consider primary research more newsworthy than reviews of 45 46 already published evidence. In our study, the monographs of evidence syntheses and methodology were more 47 highly cited than their related journal articles, whereas journal articles of primary research (especially those in 48 49 general medical journals) were more highly cited than the monographs. Primary research also appeared in 50 journals with higher impact factors. 51 52 53 The monographs have the advantage over their related journal articles in that they allow for more complete 54 55 description of the study. However we found that monograph authors could be potentially disadvantaged in a 56 bibliometric analysis if those using the WoS Core Collection do not perform a Cited Reference Search to 57 58 capture all citations. This was because 76% of the monographs in the WoS had at least one cited reference 59 variant unlinked to their full record (this was 2.6 times more frequent than for the related journal articles), and 60 this resulted in 26% of monograph citations being missed unless the much more time consuming method of a For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml Cited Reference Search was used. The major source of unlinked citations to the monographs in WoS appeared Page 33 of 37 to bewere due to the variations in how the journal nameBMJ, Openissue andthe page numbers (the monographs are BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from paginated using both Roman and Arabic numerals) were cited. 1 2 3 We also noted inaccuracies in citations to monographs in Scopus. Our detailed analysis of one monograph for the reasons 4 for the missing citations (i.e. those that were in WoS but not Scopus) showed incomplete entries of the cited monographs 5 6 in Scopus and missing citations links to the monograph. Such errors were not noted for the other articles cited alongside 7 the monographs. We speculated that the number of variations in the form in which HTA journal is cited contributed to the 8 9 much higher rate of unlinked citations to the monographs in Scopus. 10 11 12 13 Conversely, p Publishing in an external journal can improve dissemination of findings. Clinicians are likely to 14 15 prefer a short journalFor article compared peer to a 170-page review monograph. However, only academics 16 17 18 1 0 19 20 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml 21 22 23 24 25 26 27 28 29 30 31 32 33
34 http://bmjopen.bmj.com/ 35 36 37 38 39 40 41
42 on September 23, 2021 by guest. Protected copyright. 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 34 of 37
Page 11 of 12 BMJ Open BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from
11 22 who publish journal papers from their monograph may spread their citations across more papers and hence 3 3 reduce their h-indexh-index; i.e. an academic with an h-indexh-index of 30 will increase that by publishing 44 55 one paper that receives 32 citations, but not by publishing two papers that are cited 16 times each. The h- 66 index is of course only one of a number of citation indicators, and has advantages and disadvantages (, 77 88 summarised in the RAND report [5]), but it the h-index is easily available, and we know it is being used in 99 1010 medical schools. 1111 1212 1313 1414 The decision on publication strategy will depend on several factors: the key audience, dissemination and 1515 citations. PublishingFor primary researchpeer in general review medical journals appears only to be a good strategy for 1616 1717 maximising citations. 1818 1919 2020 The HTA monograph series is entirely open access which might be thought to increase numbers of 2121 2222 citations. However Davis[22] reported that in a randomised trial, open access articles were cited no more 2323 oftenr than subscription-access ones. 24 24 2525 2626 Problems with indexing in WoS have been reported by a Canadian study exploring the use of bibliometric 27 27 indicators for rating researchers.[23] The Universities UK report also noted that not all citations are collated, 2828 2929 partly because authors use a variety of abbreviations for journal titles.[24] Meanwhile, people using 3030 3131 citation databases for bibliometric assessments should know about possible sources of error in citation 3232 databases, and how best to use them, or should seek advice from experts. Those citing numbers of 3333
3434 citations should always specify the database used since numbers vary considerably. http://bmjopen.bmj.com/ 3535
3636 3737 Future research could involve investigateing if the relative citations and rankings between the different 3838 3939 citations databases found in this study areis generalizable to other disciplines, and which citation database 4040 gives the most reliable and consistent results. Also, it has been reported that a higher proportion of 4141 statistically significant findings are reported in journal articles when compared with the outcomes reported 4242 on September 23, 2021 by guest. Protected copyright. 4343 in HTA monographs.[25] Therefore, it would be interesting to see whether monographs and journal 4444 4545 publications with significant findings are cited more often than those with non-significant outcomes. 4646 4747 4848 4949 5050 CONCLUSIONS 51 51 Conclusions: The numbers of citations to the HTA monographs differed considerably between the databases, 5252 5353 but rankings were highly correlated. When a HTA monograph had a journal article published from the 54 54 same study, there were more citations to the related journal article versions for primary research, but 5555 5656 more to the monographs than their related journal articles for evidence syntheses. Citations to related 5757 journal articles were more reliably recorded than citations to the HTA monograph series, which showed 5858 5959 many variant citation forms, leading to unlinked citations. 6060
For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml PageThose 35 usingof 37 bibliometric data for decisions regarding academicBMJ career Open progression and research funding need to be BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from aware of how differences amongst databases will affect their results. Publishing in an external journal as well as a 1 monograph can aid dissemination of the work and increase overall citations to the work; alternatively,, if the related 2 3 journal article is cited instead of the monograph, it could reduce the citations to the monograph but may have less impact 4 than a single publication, and could therefore adversely affect an author’s h-indexh-index. However there does seem to 5 6 be a clear citation advantage in publishing primary research articles in general medical journals, as well as via the 7 monograph series. 8 9 10 11 12 13 14Contribution of authors: PR and NW conceived the study; PR designed the study, extracted the data and carried out 15all analyses; PR and NW draftedFor the manuscript peer and approved review the final version. only 16 17Funding: Internal 18Competing interests: The authors publish reports in the Health Technology Assessment monograph series and sometimes 19 20as related journal articles. 21 22 23REFERENCES 24 25 26 27 28 29 30 31 32 33
34 http://bmjopen.bmj.com/ 35 36 37 38 39 40 41
42 on September 23, 2021 by guest. Protected copyright. 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml BMJ Open Page 1236 of of 12 37 BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from
11 1. Wright D, Young A, Iserman E, et al. The clinical relevance and newsworthiness of NIHR HTA-funded research: a 22 33 cohort study. BMJ open 2014;4(5). 44 2. Davies S. NIHR and the HTA Programme. Secondary NIHR and the HTA Programme 2013. 55 66 http://www.profbriefings.co.uk/nihrhta2013/presentations/Plenary%20Day%201%20Sally%20Davies.pdf 7 7 (accessed 21/12/2014). 88 99 3. Douet L, Milne R, Anstee S, Habens F, Young A, Wright D. The completeness of intervention descriptions in 1010 1111 published National Institute of Health Research HTA-funded trials: a cross-sectional study. BMJ open 12 12 2014;4(1):e003713. 1313 1414 4. Turner S, Wright D, Maeso R, Cook A, Milne R. Publication rate for funded studies from a major UK health 1515 For peer review only 1616 research funder: a cohort study. BMJ open 2013;3(5). 1717 5. Ismail S, Nason E, Marjanovic S, Grant J. Bibliometrics as a tool for supporting prospective R&D decision-making 1818 1919 in the health sciences: strengths, weaknesses and options for future development: RAND Technical Report 20 20 Secondary Bibliometrics as a tool for supporting prospective R&D decision-making in the health sciences: 2121 2222 strengths, weaknesses and options for future development: RAND Technical Report 2009. 2323 2424 http://www.rand.org/pubs/technical_reports/TR685.html (accessed 10/09/2014). 25 256. Higher Education Funding Council for England (HEFCE). Report on the pilot exercise to develop bibliometric 2626 2727 indicators for the Research Excellence Framework. Secondary Report on the pilot exercise to develop 2828 2929 bibliometric indicators for the Research Excellence Framework 2009. 3030 http://www.hefce.ac.uk/pubs/year/2009/200939/ (accessed 10/09/2014). 3131 3232 7. Higher Education Funding Council for England (HEFCE). Independent review of the role of metrics in research 33 33 assessment. Secondary Independent review of the role of metrics in research assessment 2014. 3434 http://bmjopen.bmj.com/ 3535 http://www.hefce.ac.uk/whatwedo/rsrch/howfundr/metrics/ (accessed 10/09/2014). 3636 3737 8. Harzing AW. A preliminary test of Google Scholar as a source for citation data: A longitudinal study of Nobel 38 38 prize winners. Scientometrics 2013;94(3):1057-75. 3939 4040 9. National Institute for Health Research. NIHR Journals Library: Health Technology Assessment. Secondary NIHR 4141 42 Journals Library: Health Technology Assessment 2014. http://www.journalslibrary.nihr.ac.uk/hta (accessed 42 on September 23, 2021 by guest. Protected copyright. 4343 10/09/2014). 4444 4545 10. Shaw L, Rodgers H, Price C, et al. BoTULS: a multicentre randomised controlled trial to evaluate the clinical 4646 effectiveness and cost-effectiveness of treating upper limb spasticity due to stroke with botulinum toxin type 4747 4848 A. Health Technol. Assess. 2010;14(26):1-+. 4949 5050 11. Ashfaq K, Yahaya I, Hyde C, et al. Clinical effectiveness and cost-effectiveness of stem cell transplantation in the 51 51 management of acute leukaemia: a systematic review. Health Technol. Assess. 2010;14(54):1-+. 5252 5353 12. Song F, Parekh S, Hooper L, et al. Dissemination and publication of research findings: an updated review of 5454 5555 related biases. Health Technol. Assess. 2010;14(8):1-+. 5656 13. Waugh N, Cummins E, Royle P, et al. Newer agents for blood glucose control in type 2 diabetes: systematic 5757 5858 review and economic evaluation. Health Technol. Assess. 2010;14(36):1-+. 5959 14. Measuring the impact of knowledge: A comparison of Web of Science and google scholar; 2009. 6060 15. Cooper K, Squires H,For Carroll peer C, review et al. Chemopreventiononly - http://bmjopen.bmj.com/site/about/guidelines.xhtml of colorectal cancer: systematic review and economic evaluation. Health Technol. Assess. 2010;14(32):1-+. Page16. 37Mowatt of 37 G, Zhu S, Kilonzo M, et al. Systematic review ofBMJ the Openclinical effectiveness and cost-effectiveness of BMJ Open: first published as 10.1136/bmjopen-2014-006595 on 18 February 2015. Downloaded from photodynamic diagnosis and urine biomarkers (FISH, ImmunoCyt, NMP22) and cytology for the detection 1 and follow-up of bladder cancer. Health Technol. Assess. 2010;14(4):1-+. 2 3 17. Peek GJ, Mugford M, Tiruvoipati R, et al. Efficacy and economic assessment of conventional ventilatory support 4 versus extracorporeal membrane oxygenation for severe adult respiratory failure (CESAR): a multicentre 5 6 randomised controlled trial. Lancet 2009;374(9698):1351-63. 7 8 18. Peek GJ, Elbourne D, Mugford M, et al. Randomised controlled trial and parallel economic evaluation of 9 conventional ventilatory support versus extracorporeal membrane oxygenation for severe adult respiratory 10 11 failure (CESAR). Health Technol. Assess. 2010;14(35):1-+. 12 1319. Chinnery F, Young A, Goodman J, Ashton-Key M, Milne R. Time to publication for NIHR HTA programme- 14 funded research: a cohort study. BMJ open 2013;3(11):e004121. 15 For peer review only 1620. Tort AB, Targino ZH, Amaral OB. Rising publication delays inflate journal impact factors. PLoS One 17 2012;7(12):e53374. 18 1921. Kulkarni AV, Aziz B, Shams I, Busse JW. Comparisons of citations in web of science, Scopus, and Google 20 21 Scholar for articles published in general medical journals. JAMA - Journal of the American Medical 22 Association 2009;302(10):1092-96. 23 2422. Davis PM. Open access, readership, citations: a randomized controlled trial of scientific journal publishing. 25 26 FASEB journal : official publication of the Federation of American Societies for Experimental Biology 27 2011;25(7):2129-34. 28 2923. Campbell D, Picard-Aitken M, Côté G, et al. Bibliometrics as a performance measurement tool for research 30 evaluation: The case of research funded by the national cancer institute of Canada. American Journal of 31 32 Evaluation 2010;31(1):66-83. 33
3424. Universities UK. The use of bibliometrics to measure research quality in UK higher education institutions. http://bmjopen.bmj.com/ 35 Secondary The use of bibliometrics to measure research quality in UK higher education institutions 36 37 15/12/2014 2007. http://www.universitiesuk.ac.uk/highereducation/Documents/2007/Bibliometrics.pdf 38 39 (accessed 29/08/2014). 4025. Matthews GA, Dumville JC, Hewitt CE, Torgerson DJ. Retrospective cohort study highlighted outcome reporting 41
42 bias in UK publicly funded trials. Journal of clinical epidemiology 2011;64(12):1317-24. on September 23, 2021 by guest. Protected copyright. 43
44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml