BMJ
Confidential: For Review Only Academic criteria for promotion and tenure in biomedical sciences faculties: a cross-sectional analysis of an international sample of universities
Journal: BMJ
Manuscript ID BMJ-2019-053055.R1
Article Type: Research
BMJ Journal: BMJ
Date Submitted by the 10-Feb-2020 Author:
Complete List of Authors: Rice, Danielle; McGill University; Ottawa Hospital Research Institute, Raffoul, Hana; University of Waterloo; Ottawa Hospital Research Institute Ioannidis, John; Stanford University, Stanford Prevention Research Center, Department of Medicine and Department of Health Research and Policy Moher, David; Ottawa Hospital Research Institute, Ottawa Methods Centre
Keywords: Promotion, Tenure, Faculty of Medicine, Incentives
https://mc.manuscriptcentral.com/bmj Page 1 of 58 BMJ
Faculty of Biomedical Sciences Promotion and Tenure Criteria 1 2 3 1 Academic criteria for promotion and tenure in biomedical sciences faculties: a cross-sectional 4 5 6 2 analysis of an international sample of universities 7 8 3 9 10 4 Danielle B Rice1,2, Hana Raffoul2,3, John PA Ioannidis4,5,6,7, David Moher8,9 11 Confidential: For Review Only 12 5 13 14 15 6 1Department of Psychology, McGill University, Montreal, Quebec, Canada; 2Ottawa Hospital 16 17 7 Research Institute, Ontario, Canada; 3Faculty of Engineering, University of Waterloo, Waterloo, 18 19 8 Ontario, Canada; 4Departments of Medicine, 5Health Research and Policy, 6Biomedical Data 20 21 7 22 9 Science, and Statistics, and Meta-Research Innovation Center at Stanford (METRICS), Stanford 23 24 10 University, Stanford, California, USA; 8Centre for Journalology, Clinical Epidemiology Program, 25 26 11 Ottawa Hospital Research Institute, Ontario, Canada; 9School of Epidemiology and Public Health, 27 28 29 12 University of Ottawa, Ottawa, Canada 30 31 13 32 33 14 Corresponding Author 34 35 15 David Moher 36 37 16 [email protected] 38 17 39 40 18 Danielle Rice: ORCID 0000-0001-5615-7005 41 42 19 David Moher: ORCID 0000-0003-2434-4206 43 44 20 John Ioannidis: ORCID 0000-0003-3118-6859 45 46 21 47 48 49 50 51 52 53 54 55 56 57 58 1 59 60 https://mc.manuscriptcentral.com/bmj BMJ Page 2 of 58
Faculty of Biomedical Sciences Promotion and Tenure Criteria 1 2 3 1 Affiliations: 4 5 6 2 Centre for Journalology, Clinical Epidemiology Program, Ottawa Hospital Research Institute, 7 8 3 Ontario, Canada, K1H 8L6 9 10 4 Danielle B Rice (Doctoral student) 11 Confidential: For Review Only 12 5 Hana Raffoul (Undergraduate student) 13 14 6 David Moher (Director) 15 16 7 Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, 17 18 8 California, USA, 94305 19 20 9 John PA Ioannidis (Co-Director) 21 22 10 23 24 25 11 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 2 59 60 https://mc.manuscriptcentral.com/bmj Page 3 of 58 BMJ
Faculty of Biomedical Sciences Promotion and Tenure Criteria 1 2 3 1 ABSTRACT 4 5 6 2 Objectives: To determine the presence of a set of pre-specified traditional and non-traditional 7 8 3 criteria used to assess scientists for promotion and tenure in faculties of biomedical sciences among 9 10 4 universities worldwide. 11 Confidential: For Review Only 12 5 Design: Cross-sectional study. 13 14 15 6 Setting: Not applicable. 16 17 7 Participants: 170 randomly selected universities from the Leiden Ranking of world universities 18 19 8 list were considered. 20 21 22 9 Main outcome measures: Two independent reviewers searched for all guidelines applied when 23 24 10 assessing scientists for promotion and tenure for institutions with biomedical faculties. Where 25 26 11 faculty-level guidelines were not available, institution-level guidelines were sought. Available 27 28 29 12 documents were reviewed and the presence of 5 traditional (e.g., number of publications) and 7 30 31 13 non-traditional (e.g., data sharing) criteria was noted in guidelines for assessing assistant 32 33 14 professors, associate professors, professors, and the granting of tenure. 34 35 15 Results: A total of 146 institutions had faculties of biomedical sciences with 92 having eligible 36 37 38 16 guidelines available to review. Traditional criteria mentioned peer-reviewed publications, 39 40 17 authorship order, journal impact, grant funding, and national or international reputation in 95%, 41 42 18 37%, 28%, 67%, and 48% of the guidelines, respectively. Conversely, among non-traditional 43 44 45 19 criteria only citations (any mention in 26%) and accommodations for employment leave (37%) 46 47 20 were relatively commonly mentioned; while there was rare mention of alternative metrics for 48 49 21 sharing research (2%) and data sharing (1%), and 3 criteria (publishing in open access mediums, 50 51 52 22 registering research, and adhering to reporting guidelines) were not found in any institution 53 54 23 reviewed. Traditional criteria were more commonly reported than non-traditional criteria (p= .001). 55 56 57 58 3 59 60 https://mc.manuscriptcentral.com/bmj BMJ Page 4 of 58
Faculty of Biomedical Sciences Promotion and Tenure Criteria 1 2 3 1 We observed notable differences across continents on whether guidelines are accessible or not 4 5 6 2 (Australia 100%, North America 97%, Europe 50%, Asia 58%, South America 17%), and more 7 8 3 subtle differences on the use of specific criteria. 9 10 4 Conclusions: This study demonstrates that the current evaluation of scientists emphasizes 11 Confidential: For Review Only 12 5 traditional criteria as opposed to non-traditional criteria. This may reinforce research practices that 13 14 15 6 are known to be problematic while insufficiently supporting the conduct of better-quality research 16 17 7 and open science. Institutions should consider incentivizing non-traditional criteria. 18 19 8 Registration: Open Science Framework (https://osf.io/26ucp/) 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 4 59 60 https://mc.manuscriptcentral.com/bmj Page 5 of 58 BMJ
Faculty of Biomedical Sciences Promotion and Tenure Criteria 1 2 3 1 What is already known on this topic: 4 5 6 2 • Academics tailor their research practices based on the evaluation criteria applied within 7 8 3 their academic institution. 9 10 4 • Ensuring that biomedical researchers are incentivized by adhering to best practice 11 Confidential: For Review Only 12 5 13 guidelines for research is essential given the clinical implications of this work. 14 15 6 • While changes to the criteria used to assess professors and confer tenure have been 16 17 7 recommended, a systematic assessment of promotion and tenure criteria being applied 18 19 8 worldwide has not been conducted. 20 21 22 23 9 What this study adds: 24 25 26 10 • Across countries, university guidelines focus on rewarding traditional research criteria 27 28 29 11 (peer-reviewed publications, authorship order, journal impact, grant funding, and national 30 31 12 or international reputation). 32 33 13 • The minimum written requirements for promotion and tenure criteria are predominantly 34 35 36 14 objective in nature, although several of them are inadequate measures to assess the 37 38 15 impact of researchers. 39 40 16 • Developing and evaluating more appropriate, non-traditional indicators of research may 41 42 17 facilitate changes in the evaluation practices for rewarding researchers. 43 44 45 46 18 47 48 19 49 50 51 52 53 54 55 56 57 58 5 59 60 https://mc.manuscriptcentral.com/bmj BMJ Page 6 of 58
Faculty of Biomedical Sciences Promotion and Tenure Criteria 1 2 3 1 INTRODUCTION 4 5 6 2 There are important deficiencies in the quality and transparency of research conducted across 7 8 3 disciplines. 1 2 Numerous efforts have been made to combat these inadequacies by developing, for 9 10 4 example, reporting guidelines (e.g., the CONSORT and PRISMA Statements), registration of 11 Confidential: For Review Only 12 5 studies prior to data collection (e.g., clinicaltrials.gov), and data sharing practices.3 4 Despite these 13 14 15 6 strategies, poorly conducted and inadequately reported research remains highly prevalent.5 This 16 17 7 has important consequences, especially in the field of medicine, as research is heavily relied upon 18 19 8 to inform clinical decision-making. 20 21 22 9 Institutions have the ability to influence large-scale improvements among researchers, as 23 24 10 universities hire new faculty, and promote and tenure existing faculty. Universities can provide 25 26 11 incentives and rewards (e.g., promotions) for scholarly work that is conducted appropriately, 27 28 29 12 reported transparently, and adheres to best publication practices. A recent survey conducted in the 30 31 13 UK found that academics tailor their publication practices to align with their institutional 32 33 14 evaluation criteria.6 These criteria, however, may include metrics that are known to be problematic 34 35 15 for assessing researchers.7 Current incentives and rewards may also be misaligned with the needs 36 37 38 16 of society. Reward systems within universities typically include criteria within promotion and 39 40 17 tenure documents such as the quantity of publications and novelty of findings rather than the 41 42 18 reliability, accuracy, reproducibility and transparent reporting of findings.8 Inappropriate criteria 43 44 9 45 19 being applied for career advancement can inadvertently contribute to research waste, with billions 46 47 20 of dollars invested in non-usable research.10 For example, universities that emphasize the quantity 48 49 21 of published papers, can increase undeserved authorship, salami slicing, and publication in very 50 51 52 22 low-quality journals (e.g., predatory journals) without peer-review, and contribute to the problems 53 54 23 of reproducibility. 55 56 57 58 6 59 60 https://mc.manuscriptcentral.com/bmj Page 7 of 58 BMJ
Faculty of Biomedical Sciences Promotion and Tenure Criteria 1 2 3 1 Institutional criteria for promotion and tenure decisions can vary and may not be evidence-based.11 4 5 6 2 Some institutions set minimum quantitative “thresholds” for promotion, while others provide 7 8 3 qualitative phrasing of criteria that scientists must meet. Recent articles identifying the limitations 9 10 4 of the current criteria used to assess scientists for promotion and tenure have been largely 11 Confidential: For Review Only 12 5 conceptual in nature and have limited empirical evidence.11-14 In a recent study, evaluation 13 14 15 6 documents were reviewed, and potential limitations were identified.14 A group of 22 individuals, 16 17 7 including, academic leaders, health policy organizations, funders, and scientists participated in a 18 19 8 panel workshop about the perceived strengths and weaknesses of approaches used for assessing 20 21 22 9 career advancement. Further, strategies to encourage implementation and uptake of more 23 24 10 responsible indicators for assessing scientists were discussed, including six general principles for 25 26 11 assessing scientists. These principles included aspects such as rewarding researchers for open 27 28 14 29 12 science practices and the transparent and complete reporting of research. Prior to implementing 30 31 13 changes to existing criteria, however, we must better understand the current standards being 32 33 14 applied. Understanding the variability of criteria and thresholds for promotion and tenure applied 34 35 15 across institutions requires a systematic empirical assessment. Therefore, we aimed to identify and 36 37 38 16 document a set of pre-specified traditional and non-traditional criteria used to assess scientists for 39 40 17 promotion and tenure within faculties of biomedical sciences among a large number of universities 41 42 18 around the world. 43 44 45 19 METHODS 46 47 20 The protocol for this study was registered within the Open Science Framework database 48 49 21 (https://osf.io/26ucp/) prior to the study’s data collection. The STROBE Statement for cross- 50 51 52 22 sectional studies checklist was used to ensure that methods and findings are clearly reported (see 53 54 23 Appendix 1).15 55 56 57 58 7 59 60 https://mc.manuscriptcentral.com/bmj BMJ Page 8 of 58
Faculty of Biomedical Sciences Promotion and Tenure Criteria 1 2 3 1 Approach to Selecting University Institutions 4 5 6 2 The Centre for Science and Technology Studies (CSTS) Leiden Ranking of world universities 7 8 3 from 2017 (http://www.leidenranking.com/ranking/2017/list) was used to select institutions for 9 10 4 inclusion in the study.16 A random sample of 20% of institutions from the Leiden ranking list 11 Confidential: For Review Only 12 5 (170/854) were selected using an online random sampling software.17 The CSTS ranking list for 13 14 15 6 the field of “Biomedical and Health Sciences” was selected, which represents the field that 16 17 7 publications from universities are assigned to. We planned to include all randomly selected 18 19 8 institutions on this list, regardless of the faculties present in each university. The default settings 20 21 22 9 on the CSTS website were used. The default indicator settings include: type of indicator 23 24 10 (‘impact’) and indicators [‘P, P(top 10%), PP(top 10%)’]. This indicator represents the number 25 26 11 and proportion of a university’s publications that when compared with other publications in the 27 28 29 12 same field and in the same year, are among the top 10% of most frequently cited publications. 30 31 13 The list was ordered by publications, and the calculation of impact indicators using fractional 32 33 14 counting option selected. A minimum publication output was set at the default value of 100. 34 35 15 Searching of Institution Criteria 36 37 38 16 Searching for institutional criteria involved an iterative process by two reviewers. Each institution 39 40 17 (i.e., university) website was searched for their criteria and policies used for evaluation, promotion 41 42 18 and tenure in the faculty of medicine or biomedical sciences faculty. In instances where medical 43 44 45 19 schools were their own entity, the medical school website was searched for evaluation, promotion, 46 47 20 and tenure documents. We first searched if each institution had a relevant biomedical sciences 48 49 21 department or faculty (e.g., faculty of medicine, department of science). If a faculty of biomedical 50 51 52 22 sciences was found, we searched for keywords on the faculty websites including “academic 53 54 23 performance”, “career mobility”, “criteria”, “evaluation”, “guidelines”, “policy”, and “tenure” and 55 56 57 58 8 59 60 https://mc.manuscriptcentral.com/bmj Page 9 of 58 BMJ
Faculty of Biomedical Sciences Promotion and Tenure Criteria 1 2 3 1 “promotion” to find documents related to promotion and tenure. If there was no faculty related to 4 5 6 2 biomedical sciences, or if promotion and tenure guidelines were not publicly available at the 7 8 3 faculty level, we referred to the available institution level guidelines. If publicly available criteria 9 10 4 could not be located after searching with these methods, human resource personnel, professors, 11 Confidential: For Review Only 12 5 and academic affairs administrators for the institution were contacted directly on up to two 13 14 15 6 occasions to request access to faculty or institution level criteria. Within some countries, 16 17 7 promotions first require meeting criteria set at a state or national level. After searching for faculty- 18 19 8 and institution- level guidelines if these were not available, we also searched for state or national 20 21 22 9 guidelines by applying the same search techniques used for universities. For institution websites 23 24 10 that were published in languages other than English or French, the promotion and tenure 25 26 11 information available on the institution website was searched by an individual who was fluent in 27 28 29 12 the relevant language to facilitate data collection and emails were sent in the language of the 30 31 13 institution website. Twelve translators searched university, regional, and national websites for 32 33 14 documents to facilitate data extraction. These individuals also translated an email to send to 34 35 15 institution representatives when guidelines could not be found online. 36 37 38 16 39 40 17 Approach to Selecting List of Criteria 41 42 18 Twelve criteria related to promotion and tenure were purposively selected to enable a comparison 43 44 45 19 between traditional (e.g., quantity of research) and non-traditional (e.g., reproducibility of 46 47 20 research) criteria used to assess scientists for promotion and tenure (see Box 1). Although the 48 49 21 characterization of traditional and non-traditional criteria was ultimately subjective, we based our 50 51 12-14 52 22 decisions on evidence and policy initiatives from multiple jurisdictions . We used an iterative 53 54 23 process to select the characteristics. An early version of the criteria included 10 items, however, 55 56 57 58 9 59 60 https://mc.manuscriptcentral.com/bmj BMJ Page 10 of 58
Faculty of Biomedical Sciences Promotion and Tenure Criteria 1 2 3 1 after a set of 5 institutions were pilot tested, two additional items were added, and revisions were 4 5 6 2 made to the wording of select items. The final set of criteria included 5 traditional criteria (i.e., 7 8 3 peer-reviewed publications, authorship order, journal impact, grant funding, national or 9 10 4 international reputation) and 7 non-traditional criteria (i.e. citations, data sharing, publishing in 11 Confidential: For Review Only 12 5 open access mediums, registration of research, adherence to reporting guidelines, alternative ways 13 14 15 6 for sharing research, accommodations for employment leave). 16 17 7 Data Collection 18 19 8 For each eligible institution, we extracted the following information: university name, faculty 20 21 16 22 9 name, country, and human development index rating of country. Faculty of biomedical sciences 23 24 10 or institution guidelines used for the evaluation of professors were reviewed, where available, to 25 26 11 determine if each of the 12 items from our list of criteria for faculty promotion and tenure were 27 28 29 12 present. We recorded the relevant mentions for each criterion, regardless of how exactly the 30 31 13 criterion was considered or operationalized. We did not intend to arbitrate if the proposed version 32 33 14 of the criterion was appropriate and technically sophisticated; however, we collected information 34 35 15 about if guidelines applied thresholds for each criterion. 36 37 38 16 When promotion and tenure guidelines were available, we first reviewed the table of contents and 39 40 17 located the section on promotion and tenure criteria and reviewed this section of the document, 41 42 18 including any sections that the document referred to for context. If a table of contents was not 43 44 45 19 provided, we reviewed the document in its entirety. We then reviewed and extracted the presence 46 47 20 of criteria and the relevant statement for each level of promotion criteria published for universities, 48 49 21 including promotions to assistant professor, associate professor, full professor, and the granting of 50 51 52 22 tenure, as well as whether a criterion was mentioned for at least one of these levels. These levels 53 54 23 of promotion were considered based on a North American framework of career advancement. 55 56 57 58 10 59 60 https://mc.manuscriptcentral.com/bmj Page 11 of 58 BMJ
Faculty of Biomedical Sciences Promotion and Tenure Criteria 1 2 3 1 Where institutions applied different labels to ranks/positions (e.g., researcher level A), we sought 4 5 6 2 documentation for the appropriate equivalent categorization of the promotion levels. If 7 8 3 documentation describing the position was not available, we consulted with professors from the 9 10 4 institution country in order to equate positions with those being applied in our study. If no 11 Confidential: For Review Only 12 5 equivalent position was available, we did not include the institution within our sample (n=3 13 14 15 6 institutions). This information was extracted for tenure-track positions rather than non-tenure 16 17 7 track. We did not extract promotion and tenure criteria for aspects of career advancement related 18 19 8 to teaching or clinical duties, or for positions that were comprised of more educational or clinical 20 21 22 9 activities as compared to research activities. The level of the promotion criteria available (i.e. 23 24 10 faculty level criteria, departmental level criteria), the year that the promotion guideline was 25 26 11 published, the associated URL of the criteria, and the date that the website was searched were also 27 28 29 12 extracted. Two reviewers (DBR, HR) independently extracted all data and results were compared 30 31 13 for consistency. Where consensus was not achieved between reviewers after discussion, a third 32 33 14 team member (DM) was consulted to address discrepancies in the interpretation of criteria. For 34 35 15 guidelines published in languages other than English or French, translation of the relevant 36 37 38 16 documents was conducted by one individual and verified by a second reviewer (DBR) using 39 40 17 Google Translate. A Hungarian translator was not available for one guideline. For this guideline, 41 42 18 one reviewer (DBR) used Google translate to conduct data extraction. Data collection was 43 44 45 19 performed using a standardized electronic data collection form in Distiller Systematic Reviewer 46 47 20 (Evidence Partners, Ottawa, Canada). 48 49 21 Approach to Synthesis 50 51 52 22 Institution characteristics and promotion and tenure criteria were summarized in table format to 53 54 23 facilitate inspection and discussion of findings. The percentage of criteria that were included in 55 56 57 58 11 59 60 https://mc.manuscriptcentral.com/bmj BMJ Page 12 of 58
Faculty of Biomedical Sciences Promotion and Tenure Criteria 1 2 3 1 promotion and tenure guidelines were compared through a paired sample t-test. Categorical 4 5 6 2 variables were presented as percentages and counts, and continuous variables were presented as 7 8 3 means and standard deviations (SD) or median and interquartile range (IQR). Institutions that had 9 10 4 criteria available were compared to institutions that did not have criteria available through 11 Confidential: For Review Only 12 5 independent samples t-test, chi-square tests, or non-parametric tests. 13 14 15 6 Exploratory analyses were conducted for the full professor position as this had the most data 16 17 7 available. Two multiple linear regressions were conducted to assess the associations between 18 19 8 institution characteristics (independent variables: level of criteria, continent, human development 20 21 22 9 index, and Leiden ranking) and the number of criteria present for traditional and non-traditional 23 24 10 items for guidelines for professors (dependent variable) as the majority of institutions had 25 26 11 guidelines present for this promotion level. Logistic regressions were conducted for each criterion 27 28 29 12 present among 10% to 90% of institutions at the rank of full professor to assess the associations 30 31 13 between institutional characteristics (independent variables) and the presence of each criterion 32 33 14 (dependent variable). Institution characteristics were selected as covariates given their availability 34 35 15 and the potential relevance to the type of criteria applied (e.g., regional differences in career 36 37 38 16 advancement procedures). If an independent variable did not have at least one institution both with 39 40 17 and without the item criteria, that variable was removed from the logistic regression. Prior to 41 42 18 conducting regression analyses, preliminary tests were performed to confirm that there were no 43 44 45 19 violations of multiple regression assumptions. Microsoft Excel was used for summing study 46 47 20 results, while SPSS Statistics version 21.0 (Chicago, IL) was used for statistical tests. All statistical 48 49 21 analyses were two-tailed with p<.005 significance level to adhere to recent recommendations for 50 51 14 18 52 22 a lowered threshold of statistical significance. 53 54 23 Patient and Public Involvement 55 56 57 58 12 59 60 https://mc.manuscriptcentral.com/bmj Page 13 of 58 BMJ
Faculty of Biomedical Sciences Promotion and Tenure Criteria 1 2 3 1 This research did not involve consultation with patients or the public. 4 5 6 2 RESULTS 7 8 3 Deviations from Protocol 9 10 4 We refined our inclusion criteria to exclude institutions that did not have a faculty of medicine, 11 Confidential: For Review Only 12 5 biomedical sciences, life sciences, health sciences, or medical sciences in order to focus on 13 14 15 6 institutions that have a department directly related to studying and subsequently disseminating 16 17 7 biomedical research. In the regression analyses, we excluded data from continents that had fewer 18 19 8 than 2 institutions with guidelines available. This resulted in excluding one institution from South 20 21 22 9 America and one institution from Africa from regression analyses. 23 24 10 Institution Characteristics 25 26 11 Of the 170 institutions reviewed, 146 had faculties of biomedical sciences. We were able to obtain 27 28 29 12 a total of 92 (63%) institutions’ guidelines for promotion and tenure (see Appendix 2 and Figure 30 31 13 1). For the other institutions we could not find or access such guidelines either online or after 32 33 14 communication (see Appendix 3). Of institutions with available guidelines, 39 (42%) were specific 34 35 15 to faculties of biomedical sciences. 36 37 38 16 The universities that we could evaluate had mostly a very high development index rating (n=68, 39 40 17 74%) and they were almost equally split between Europe (n=27, 29%), Asia (n=29, 32%), and 41 42 18 North America (n=28, 30%). Guidelines referred to were last updated between 1993 and 2018 43 44 45 19 (median = 2016, IQR 2011 to 2017). Based on Leiden ranking of world universities, institution 46 47 20 rankings ranged from 6 to 842 (median = 345, IQR 157 to 549) (see Table 1). Of the 92 guidelines 48 49 21 reviewed, the evaluation for promotion to positions equivalent to assistant professor (n=49, 53%), 50 51 52 22 associate professor (n=79, 86%), full professor (n=83, 90%), and tenure (n=26, 28%) were present, 53 54 23 with most (n=82, 89%) institutions having guidelines available for more than one level of 55 56 57 58 13 59 60 https://mc.manuscriptcentral.com/bmj BMJ Page 14 of 58
Faculty of Biomedical Sciences Promotion and Tenure Criteria 1 2 3 1 promotion. There were no statistically significant differences between institutions that did versus 4 5 6 2 did not have criteria available for institution rankings (p = .142) or human development index 7 8 3 (x2(2, n = 93, p = .918). We observed notable differences across continents on whether guidelines 9 10 4 were accessible or not (Australia 100%, North America 97%, Europe 50%, Asia 58%, South 11 Confidential: For Review Only 12 5 America 17%) (x2(5, n = 93, p = .001) (see Table 1). 13 14 15 6 The traditional criteria that were present most often were peer-reviewed publications (any mention: 16 17 7 n=87, 95%; assistant professor: n= 39, 80%; associate professor: n=76, 96%; professor: n=79, 18 19 8 95%; tenure: n=22, 85%) and grant funding (any mention: n=62, 67%; assistant professor: n=26, 20 21 22 9 53%; associate professor n=50, 63%; professor: n=56, 67%; tenure: n=15, 58%) with less frequent 23 24 10 use of authorship order (any mention: 34, 37%), journal impact factor (any mention: n=26, 28%), 25 26 11 and national or international reputation (any mention: n=44, 48%). The exact mentions and how 27 28 29 12 they were supposed to be operationalized varied across guidelines and only a minority were 30 31 13 quantitative. Thirty-two institutions (35%) had at least one mention to a specific number of peer- 32 33 14 reviewed publications. The requirements for publications was heterogeneous between institutions 34 35 15 with some, for example requiring a specific number per year, or over the past 10 years (e.g., as 36 37 38 16 few as 1 total publication required with as many as 53 publications required in the past 10 years). 39 40 17 Institutions that required fewer publications often reported that publishing in journals with higher 41 42 18 impact factors was necessary (i.e., one publication in a journal with an impact factor of at least 10, 43 44 45 19 or two publications in journals with impact factors of at least 5). Fourteen institutions (15%) had 46 47 20 at least one mention to a specific amount of money for funding [range 300,000 Rem Min Bi 48 49 21 (41,766 USD) to 3,000,000 Rem Min Bi (417,660 USD]. For authorship order, 24 (26%) 50 51 52 22 institutions encouraged first author publications, 20 (22%) encouraged last or corresponding 53 54 23 author publications, although many of these institutions also promoted first author publications, 55 56 57 58 14 59 60 https://mc.manuscriptcentral.com/bmj Page 15 of 58 BMJ
Faculty of Biomedical Sciences Promotion and Tenure Criteria 1 2 3 1 and 4 institutions (11%) encouraged sole authored publications. No institutions mentioned that 4 5 6 2 middle author publications or multi-authored papers were favourable. For journal impact, 11 7 8 3 institutions of 26 (42%) mentioned specific numbers for desirable impact factor metrics, but the 9 10 4 desirable impact factor thresholds varied enormously across institutions (≥ 3, 4, 5, 9, 10, 11, 20, 11 Confidential: For Review Only 12 5 30, see Appendix 4). No institutions had any numerical recommendations on assessment of 13 14 15 6 national or international reputation. 16 17 7 Non-traditional criteria that were present included, adjustments to expectations when professors 18 19 8 go on leave (any mention: n=34, 37%; assistant professor: n=22, 45%; associate professor: n=28, 20 21 22 9 35%; professor: n=29, 35%; tenure: n=13, 50%), citations of research (any mention: n=24, 26%; 23 24 10 assistant professor: n=12, 24%; associate professor: n=23, 29%; professor n=23, 28%; tenure: n=6, 25 26 11 23%), and rarely mention of alternative metrics for sharing research (any mention: n= 3, 3%; 27 28 29 12 assistant professor: n=3, 6%; associate professor: n=3, 4%; professor: n=2, 2%; tenure: n=1, 4%). 30 31 13 Data sharing was mentioned only in 1 institution (1%). Mention of publishing in open access 32 33 14 outlets, registering research, and adhering to reporting guidelines were absent from all institutions 34 35 15 (see Table 2 and Figure 2). Non-traditional criteria were mostly qualitative. For citations, however, 36 37 38 16 25% of institutions (6 of 24) that included this item proposed specific numbers (see Appendix 4). 39 40 17 Characteristics Associated with the Presence of Traditional and Non-traditional Items for 41 42 18 Professors 43 44 45 19 In tests of multicollinearity, independent variable tolerance values ranged from 0.4 to 0.9, and 46 47 20 the variance inflation factors ranged from 1.1 to 2.8 for both traditional and non-traditional 48 49 21 analyses, indicating that multicollinearity was not a major issue.19 There was no deviation in the 50 51 52 22 assumption of normality based on the inspection of the normal probability plots of the residuals 53 54 55 56 57 58 15 59 60 https://mc.manuscriptcentral.com/bmj BMJ Page 16 of 58
Faculty of Biomedical Sciences Promotion and Tenure Criteria 1 2 3 1 or evidence of violations of assumptions of outliers, linearity, homoscedasticity, and 4 5 6 2 independence of residuals based on the standardized residual and scatter plot inspections. 7 8 3 Regressions for Total Scores 9 10 4 Institutions from Australia (unstandardized regression coefficient [β] = 1.8, standard error [SE] = 11 Confidential: For Review Only 12 5 0.61, p = .004) and North America (β = 0.99, SE = 0.43, p = .025) tended to have a slightly larger 13 14 15 6 number of traditional criteria present in guidelines (Table 3). Australia had an average of 70% 16 17 7 (mean=3.5 of 5 items, SD = 1.0) of traditional criteria present while North America had an average 18 19 8 of 64% (mean=3.2 of 5 items, SD = 1.1). A significantly greater percentage of traditional items 20 21 22 9 (54.2%) compared to non-traditional items (9.5%) were reported among institutions (mean 23 24 10 difference= 44.8%, 95% confidence interval [CI]= 39.6, 50.0, p= .001). Institutions located in 25 26 11 Australia (β = 1.07, SE = 0.38, p = .006) had modestly more non-traditional criteria in their 27 28 29 12 guidelines (Table 3). Institutions from Australia had an average of 1.5 (SD=0.5) non-traditional 30 31 13 criteria present in guidelines. Mean percentage of traditional and progressive criteria present by 32 33 14 continent are presented in Table 4. 34 35 15 Regressions for Individual Items 36 37 38 16 Six of the 12 items were present in 10% to 90% of promotion and tenure guidelines for 39 40 17 professors, including grant funding, authorship order, impact factor, national or international 41 42 18 reputation, citations, and adjustments to expectations. Encouraging researchers to have a national 43 44 45 19 or international reputation was significantly more present among institutions from North 46 47 20 America (β = 27.44, 95% CI = 3.26, 231.16, p = .002) (Table 5). 48 49 21 50 51 52 22 DISCUSSION 53 54 55 56 57 58 16 59 60 https://mc.manuscriptcentral.com/bmj Page 17 of 58 BMJ
Faculty of Biomedical Sciences Promotion and Tenure Criteria 1 2 3 1 We found that traditional criteria are relied on in guidelines for assessing faculty members for 4 5 6 2 promotion and tenure among an international sample of 92 institutions with faculties of 7 8 3 biomedicine or health sciences. Almost all institutions promotion criteria included the presence of 9 10 4 peer-reviewed publications, many of which also required a minimum number of papers published 11 Confidential: For Review Only 12 5 per year. Conversely, only about a third of institutions discussed citations and none referenced 13 14 15 6 publishing in open access mediums, registering research, and adhering to reporting guidelines for 16 17 7 transparently presenting research was absent from all examined documents. 18 19 8 There was substantial variability across continents on whether any guidelines were available at all. 20 21 22 9 This may be especially important to consider given our work was based on a North American 23 24 10 framework of career advancement which impacts the terms applied when searching for documents 25 26 11 and the interpretation of criteria. Given the substantial rate of non-response from specific regions, 27 28 29 12 we cannot exclude the possibility that such documents exist but could not be retrieved. For some 30 31 13 universities, the criteria and related guidelines are not set at the level of the medical faculty or even 32 33 14 the whole university, but by a higher state authority (e.g. the ministry of education in Greece) for 34 35 15 all universities, or government regulated laws (e.g., employment leaves). While the process for 36 37 38 16 achieving promotion and tenure vary internationally, the concept of career advancement and need 39 40 17 for appropriate criteria is common among all regions. Availability of criteria is probably helpful 41 42 18 for transparency; however, the availability of guideline documents does not mean that these are 43 44 45 19 also faithfully adhered to. Criteria and rules may be bent in everyday academic practice. Assessing 46 47 20 the adherence to guideline documents for career advancement through surveys or interviews may 48 49 21 allow for an improved understanding of how to most meaningfully align the promotion and tenure 50 51 52 22 criteria to best practices in research. This approach to data collection could also shed light on 53 54 23 criteria that may be applied but are not stated in promotion and tenure documents. 55 56 57 58 17 59 60 https://mc.manuscriptcentral.com/bmj BMJ Page 18 of 58
Faculty of Biomedical Sciences Promotion and Tenure Criteria 1 2 3 1 An important barrier to the implementation of non-traditional criteria relates to the difficulty of 4 5 14 6 2 selecting and integrating more appropriate measures, which was described in several promotion 7 8 3 documents reviewed. Institutions have noted the imperfections of traditional criteria, such as the 9 10 4 impact factor, but reported that few alternatives exist.20 21 Integrating non-traditional criteria to 11 Confidential: For Review Only 12 5 incentivize scientists requires evidence on the accuracy and the validity of non-traditional 13 14 15 6 indicators 14; and such indicators are starting to emerge. 16 17 7 As non-traditional metrics are available, implementing their use more widely, for example through 18 19 8 Declaration on Research Assessment (DORA)’s advisory board, may be one avenue to aid in the 20 21 22 9 dissemination of more appropriate tools for assessing scientists. 23 24 10 Institutions relying on traditional metrics, such as number of publications and associated journal 25 26 11 impact factors may misinterpret what these metrics mean.21 Beyond evidence, there are other 27 28 29 12 reasons to consider alternative criteria. They may better align with a university’s mission. 30 31 13 Similarly, some criteria, such as data sharing, have a high research integrity value; patients support 32 33 14 sharing of their data 22 and it facilitates assessments of reproducibility. To facilitate data sharing it 34 35 15 is likely that the FAIR (Findability, Accessibility, Interoperability and Reusability) principles will 36 37 38 16 need to be in place.9 An additional difficulty to including non-traditional criteria in evaluations is 39 40 17 the need for resources to support this change. The institution reviewed in our sample that had the 41 42 18 greatest number of non-traditional criteria, Ghent University, described having invested in an 43 44 20 45 19 online system to help assess some non-traditional metrics of their researchers. Decreasing the 46 47 20 barriers to using non-traditional metrics in evaluations will be necessary for systematic changes to 48 49 21 occur. 50 51 52 22 There are limitations that should be considered when interpreting our study results. Direct 53 54 23 involvement from patients and the public was absent from this review. Incorporating the 55 56 57 58 18 59 60 https://mc.manuscriptcentral.com/bmj Page 19 of 58 BMJ
Faculty of Biomedical Sciences Promotion and Tenure Criteria 1 2 3 1 perspectives of patients and the public in future research of promotion and tenure criteria can 4 5 6 2 incentivize research practices that better align with the needs and expectations of society. This 7 8 3 could also allow for international differences to be highlighted by speaking with stakeholder 9 10 4 groups in various regions. Next, despite searching websites and contacting institutions, not all 11 Confidential: For Review Only 12 5 institutions use prespecified criteria for assessing promotion and in some instances, documents 13 14 15 6 were not found. This resulted in only a subset of the intended sample being available for review 16 17 7 and included in our analyses. South America and Africa being underrepresented in our sample and 18 19 8 few conclusions can be drawn about the criteria of institutions in these regions. An additional 20 21 22 9 limitation is that incentives for professors can occur through other pathways, e.g. financial 23 24 10 bonuses, which may not be publicly available or included in the documents reviewed. Obtaining a 25 26 11 more complete understanding of the criteria used for providing financial and reputational 27 28 29 12 incentives in medical faculties may include reviewing internal documentation regarding bonuses 30 31 13 and awards or recognitions in addition to formal promotions. Furthermore, medical faculties often 32 33 14 take into account clinical work and teaching which we did not include. 34 35 15 Finally, we should acknowledge that both for traditional and for non-traditional criteria, the exact 36 37 38 16 way they are proposed and operationalized can make a difference on whether they can have a 39 40 17 positive or negative impact on research quality. With a plethora of metrics being developed for 41 42 18 non-traditional criteria, some of them may be much better than others. For example, while citations 43 44 45 19 may be a more accurate representation of one’s research impact than journal impact factor, 46 47 20 considering the number of citations in isolation from the field of research may not motivate those 48 49 21 who work in otherwise important research fields which have low citation density (e.g., because 50 51 52 22 few other scientists work in them). Integrating appropriately framed criteria that encourage best 53 54 23 practice research high could result in improvements in medical research and evidence-based 55 56 57 58 19 59 60 https://mc.manuscriptcentral.com/bmj BMJ Page 20 of 58
Faculty of Biomedical Sciences Promotion and Tenure Criteria 1 2 3 1 medicine. Systematic changes require collaborative efforts and creativity in order to address 4 5 6 2 barriers to developing and adopting the best metrics. Considering the benefits of creating 7 8 3 sustainable changes to the criteria and associated that drive poor medical research internationally, 9 10 4 however, would be a turning point in facilitating the transparency, openness, and reproducibility 11 Confidential: For Review Only 12 5 in research practices. 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 20 59 60 https://mc.manuscriptcentral.com/bmj Page 21 of 58 BMJ
Faculty of Biomedical Sciences Promotion and Tenure Criteria 1 2 3 1 4 2 Box 1. Criteria of Interest for Promotion and Tenure. 5 6 3 7 8 Traditional Criteria 9 10 1. Is any quantitative or qualitative mention made about publications required? If quantitative, please 11 specifyConfidential: the requirement. For Review Only 12 13 2. Is any quantitative or qualitative mention made about the specific authorship order in publications? If 14 so, please specify order (e.g. first, senior, single) required. 15 16 3. Is any mention made of journal impact factors? If quantitative, what are the minimum thresholds? 17 18 19 4. Is any mention made of grant funding? If quantitative, what are the minimum thresholds (i.e., amount 20 of funding and/or number of grants as principal investigator)? 21 22 5. Is any mention made requiring that research is recognized at a national or international level? If so, 23 please specify the requirement. 24 25 Non-Traditional Criteria 26 27 6. Is any mention made of citations? If quantitative, what are the thresholds of minimum requirement? 28 29 Are specific citation databases mentioned? 30 31 7. Is any mention made of data sharing? If quantitative, what are the minimum thresholds (e.g., percentage 32 of data that is to be made available)? 33 34 8. Is any mention made of publishing in open access mediums? If quantitative, what are the minimum 35 thresholds (e.g., percentage of studies to be published in open access journals)? 36 37 38 9. Is any mention made of registration (including preregistration challenge) of studies? If yes, are there 39 thresholds of minimum requirement (e.g., percentage of studies that are to be registered). 40 41 10. Is any mention made of adherence to reporting guidelines for publications? If so, are specific guidelines 42 mentioned? 43 44 11. Is any mention made of alternative metrics for sharing research (e.g., social media and print media)? 45 If so, are specific metrics mentioned? 46 47 48 12. Is any mention made of accommodations or adjustments to expectations due to employment leave? If 49 so, please specify the description of accommodations (e.g., an extra year to defer tenure consideration) 50 and the type of eligible circumstances (e.g., parental leave, medical leave)? 51 52 53 54 55 56 57 58 21 59 60 https://mc.manuscriptcentral.com/bmj BMJ Page 22 of 58
Faculty of Biomedical Sciences Promotion and Tenure Criteria 1 2 3 ACKNOWLEDGEMENTS 4 We would like to thank Dr. Juan Pablo Alperin for providing us with access to the database 5 6 which included documents for review from North American Universities. We would like to 7 thank Becky Skidmore for her recommended search terms, Nikesh Acharya Chander for his help 8 with table formatting and Dr. Tim Ramsay for his guidance with analyses. We would also like to 9 thank each of the individuals that searched websites and translated documents including: Xiaoqin 10 Wang, AndreaConfidential: Carboni-Jiménez, Michal Dedys,For Fatemeh Review Yazdi, Philipp-Clemens Only Nowotny, 11 Song Xiaoyang, Yan Jin, Kednapa Thavorn, Chen He, Alexander Tsertsvadze, and Francesca 12 Ruggiero. Finally, we would like to acknowledge each of the institutions that responded to our 13 14 requests for promotion and tenure documents. 15 16 COPYRIGHT FOR AUTHORS 17 The Corresponding Author has the right to grant on behalf of all authors and does grant on 18 behalf of all authors, a worldwide licence to the Publishers and its licensees in perpetuity, in all 19 forms, formats and media (whether known now or created in the future), to i) publish, reproduce, 20 21 distribute, display and store the Contribution, ii) translate the Contribution into other languages 22 create adaptations, reprints, include within collections and create summaries, extracts and/or, 23 abstracts of the Contribution, iii) create any other derivative work(s) based on the Contribution, 24 iv) to exploit all subsidiary rights in the Contribution, v) the inclusion of electronic links from 25 the Contribution to third party material where-ever it may be located; and, vi) licence any third 26 party to do any or all of the above. 27 28 29 The Corresponding Author has the right to grant on behalf of all authors and does grant on behalf 30 of all authors, an exclusive licence (or non exclusive for government employees) on a worldwide 31 basis to the BMJ Publishing Group Ltd to permit this article (if accepted) to be published in BMJ 32 editions and any other BMJPGL products and sublicences such use and exploit all subsidiary 33 rights, as set out in our licence. 34 35 36 37 CONFLICTS OF INTEREST 38 All authors declare: no support from any organisation for the submitted work; no financial 39 relationships with any organisations that might have an interest in the submitted work in the 40 previous three years 41 42 43 ETHICAL APPROVAL 44 As this study did not involve any human data, ethics approval was not required. 45 46 TRANSPARENCY DECLARATION 47 The manuscript’s guarantor affirms that the manuscript is an honest, accurate, and transparent 48 account of the study being reported; that no important aspects of the study have been omitted; 49 and that any discrepancies from the study as planned (and, if relevant, registered) have been 50 51 explained. 52 53 DISSEMINATION DECLARATION 54 55 56 57 58 22 59 60 https://mc.manuscriptcentral.com/bmj Page 23 of 58 BMJ
Faculty of Biomedical Sciences Promotion and Tenure Criteria 1 2 3 This work did not involve study participants. This work will be shared through conference 4 presentations, and shared with relevant stakeholder groups and initiatives such as the Declaration 5 6 on Research Assessment (DORA). 7 8 9 FUNDING 10 There wasConfidential: no funding received for this work. For Ms. Rice Review is funded through Only a Canadian Institutes of 11 Health Research Vanier Graduate Scholarship. Dr. Moher is funded by a University Research 12 Chair. METRICS is funded by a grant from the Laura and John Arnold Foundation. 13 14 15 DATA SHARING 16 All data associated with this study is posted on the open science framework 17 (https://osf.io/26ucp/). The study protocol, data extraction forms, and data are also available at 18 this link. 19 20 21 AUTHOR CONTRIBUTIONS 22 DM and JPAI conceived of the study. DBR and DM wrote the study protocol and the initial draft 23 of the article. DM, DBR and HR agreed on the criteria applied for promotion and tenure after 24 pilot testing a set of university criteria. DBR and HR found institution guidelines and extracted 25 information regarding promotion criteria. All authors were involved in subsequent protocol 26 revisions. DBR conducted the statistical analyses. All authors provided critical feedback and 27 agreed to the final version of the paper. 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 23 59 60 https://mc.manuscriptcentral.com/bmj BMJ Page 24 of 58
Faculty of Biomedical Sciences Promotion and Tenure Criteria 1 2 3 REFERENCES 4 5 6 1. Moja LP, Telaro E, D'Amico R, Moschetti I, Coe L, Liberati A. Assessment of 7 8 methodological quality of primary studies by systematic reviews: results of the metaquality 9 10 cross sectional study. BMJ 2005;330:1053. 11 Confidential: For Review Only 12 2. Ioannidis JPA. Acknowledging and overcoming nonreproducibility in basic and preclinical 13 14 15 research. JAMA 2017;317:1019-20. 16 17 3. Moher D, Liberati A, Tetzlaff J, Altman DG, Prisma G. Preferred reporting items for 18 19 systematic reviews and meta-analyses: the PRISMA statement. PLoS MED 2009;6:e1000097. 20 21 22 4. Schulz KF, Altman DG, Moher D. CONSORT 2010 statement: updated guidelines for 23 24 reporting parallel group randomised trials. BMC Med 2010;8:18. 25 26 5. Moher D, Glasziou P, Chalmers I, et al. Increasing value and reducing waste in biomedical 27 28 29 research: who's listening? Lancet 2016;387:1573-86. 30 31 6. Wolff C, Rod AB, Schonfeld RC. UK Survey of Academics 2015: Ithaka S+ R| Jisc| RLUK”. 32 33 Last modified 15 June 2016. https://doi.org/10.18665/sr.282736 34 35 7. Hammarfelt B. Recognition and reward in the academy: Valuing publication oeuvres in 36 37 38 biomedicine, economics and history. Aslib J Inf Manag 2017;69:607-23. 39 40 8. Ioannidis JPA, Greenland S, Hlatky MA, et al. Increasing value and reducing waste in 41 42 research design, conduct, and analysis. Lancet 2014;383:166-75. 43 44 45 9. Wilkinson MD, Dumontier M, Aalbersberg IJ, et al. The FAIR Guiding Principles for 46 47 scientific data management and stewardship. Sci Data 2016;3. 48 49 10. Chan A-W, Song F, Vickers A, et al. Increasing value and reducing waste: addressing 50 51 52 inaccessible research. Lancet 2014;383:257-66. 53 54 55 56 57 58 24 59 60 https://mc.manuscriptcentral.com/bmj Page 25 of 58 BMJ
Faculty of Biomedical Sciences Promotion and Tenure Criteria 1 2 3 11. Moher D, Goodman SN, Ioannidis J. Academic criteria for appointment, promotion and 4 5 6 rewards in medical research: where's the evidence? Eur J Clin Invest 2016;46:383-85. 7 8 12. McKiernan EC. Imagining the “open” university: Sharing scholarship to improve research 9 10 and education. PLoS Biol 2017;15:e1002614. 11 Confidential: For Review Only 12 13. Flier J. Faculty promotion must assess reproducibility. Nature News 2017;549:133. 13 14 15 14. Moher D, Naudet F, Cristea IA, Miedema F, Ioannidis JPA, Goodman SN. Assessing 16 17 scientists for hiring, promotion, and tenure. PLoS Biol 2018;16:e2004089. 18 19 15. Von Elm E, Altman DG, Egger M, et al. The Strengthening the Reporting of Observational 20 21 22 Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. 23 24 PLoS Med 2007;4:e296. 25 26 16. Available from: http://www.leidenranking.com/ranking/2017/list last accessed October 4, 27 28 29 2019. 30 31 17. Available from: https://www.randomizer.org/ last accessed October 4, 2019. 32 33 18. Benjamin DJ, Berger JO, Johannesson M, et al. Redefine statistical significance. Nat Hum 34 35 Behav 2018;2:6. 36 37 38 19. Tabachnick BG, Fidell LS. Using multivariate statistics, 6th edn Boston. Ma: Pearson 2013 39 40 20. Ghent University. Principles for the evaluation of research 2017 [Available from: 41 42 https://www.ugent.be/en/research/research-evaluation.htm last accessed October 4, 2019. 43 44 45 21. McKiernan EC, Schimanski LA, Muñoz Nieves C, Matthias L, Niles MT, Alperin JP. Use of 46 47 the Journal Impact Factor in academic review, promotion, and tenure evaluations. eLife 48 49 2019;8:e47338. doi: 10.7554/eLife.47338 50 51 52 22. Mello MM, Lieou V, Goodman SN. Clinical Trial Participants’ Views of the Risks and 53 54 Benefits of Data Sharing. N Engl J Med 2018;378:2202-11. 55 56 57 58 25 59 60 https://mc.manuscriptcentral.com/bmj BMJ Page 26 of 58
1 2 3 Figure 1. Included Universities 4 5 6 7 Random sample of institutions (n= 170) 8 9 10 11 Confidential: For ReviewGuidelines Only not available (n= 60) 12 Institution did not respond after two 13 attempts to contact (n= 43) 14 Institution provided hiring 15 16 documents only (n= 8) 17 Institution reported not being 18 willing or able to share promotion 19 and tenure guidelines (n= 6) 20 21 Guidelines only available for 22 positions that were not reviewed in 23 the current study (n= 3) 24 25 Available 26 Documents 27 28 29 30 31 32 33 34 35 Guidelines available (n= 110) 36 37 38 39 40 41 No faculty of biomedical sciences (n= 18) 42 43 44 45 46 Guidelines available (n= 92) 47 48 Faculty guideline (n= 39) 49 Institution or national guideline (n= 53) 50 51 52 53 54 55 56 57 58 59 60 https://mc.manuscriptcentral.com/bmj Page 27 of 58 BMJ
1 2 3 Figure 2. Mean Number of Criteria Present by Level of Promotion 4 5 6 Traditional and Non-Traditional Criteria for Promotion and Tenure 7 8 5 9 10 11 Confidential: For Review Only 12 4 13 14 15 3 16 17 18 Non-Traditional Criteria 19 2 20 Traditional Criteria 21 22 of Criteria Median Number 1 23 24 25 26 0 27 Assistant Associate Professor Tenure 28 Professor Professor 29 Level of Evaluation 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 https://mc.manuscriptcentral.com/bmj BMJ Page 28 of 58
1 2 3 Table 1. Summary Characteristics of Included Institutions (n= 146). 4 5 Variables Institutions with Guidelines Available Institutions with Guidelines not p value 6 7 (n=92) Available (n=54) 8 Confidential: For Review Only 9 10 Criteria Level Available 11 12 Institution (%) 53 (58) N/A N/A 13 14 Faculty (%) 39 (42) 15 16 17 Median Leiden Ranking 346 (392) 462 (418) .230 18 (Interquartile Range) 19 20 Continent .001 21 22 South America (n=6), (%) 1 (17) 5 (83) 23 24 Australia (n=6) (%) 6 (100) 0 (0) 25 26 Europe (n=54), (%) 27 (50) 27 (50) 27 28 North America (n=29), (%) 28 (97) 1 (3) 29 30 Asia (n= 50), (%) 29 (58) 21 (42) 31 32 Africa (n=1) (%) 1 (100) 0 (0) 33 34 Human Development Index .918 35 Very High (n=107), (%) 68 (64) 39 (36) 36 37 High (n=37), (%) 23 (62) 14 (38) 38 39 Medium (n=2), (%) 1 (50) 1 (50) 40 41 42 43 44 45 https://mc.manuscriptcentral.com/bmj 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 Page 29 of 58 BMJ
1 2 3 4 Table 2. Criteria of Interest for Promotion and Tenure 5 6 Criteria Presence of Presence of Presence of Presence of Criteria Example of Example of Relevant 7 Criteria for Criteria for Criteria for Full for Tenure (n=26), Quantitative Quote From University 8 Confidential:Assistant Associate ForProfessor Reviewn(%) InformationOnly if Website 9 Professor Professor, (n=83), n(%) Present 10 (n=49), n(%) (n=79), n(%) 11 12 Traditional Criteria 13 14 1. Is any quantitative or “Minimum 2 "Publication in refereed 15 qualitative mention made research papers or journals or series, or by 16 about publications one paper and one publishers recognized 39 (80%) 76 (96%) 79 (95%) 22 (85%) book” as leaders in the field" 17 required? If quantitative, 18 please specify the 19 requirement. 20 21 2. Is any quantitative or “Three papers as “In case of multi- 22 qualitative mention made first or authored work, at least 23 about the specific corresponding one of the peer 24 author” reviewed publications authorship order in 25 11 (22%) 28 (35%) 28 (34%) 9 (35%) must be sole authored." 26 publications? If so, 27 please specify order (e.g. 28 first, senior, single) 29 required. 30 31 “At least one "At least 3 publications 32 impact factor>3.0; in international 33 3. Is any mention made of at least one journals with 34 journal impact factors? impact factor>5.0, reasonable impact 35 If quantitative, what are 12 (25%) 24 (30%) 23 (28%) 2 (8%) or accumulate factor are required." 36 the minimum impact 37 thresholds? factor>11.0” 38 39 40 41 42 43 44 45 https://mc.manuscriptcentral.com/bmj 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Page 30 of 58
1 2 3 “Principal "The candidate has 4 investigator of engaged in research 5 4. Is any mention made of one provincial grants/contracts as 6 grant funding? If project, principal Principal or Co- 7 quantitative, what are investigator or the Investigator at a 8 Confidential: For Review mainOnly member (top funding level the minimum thresholds 9 26 (53%) 50 (63%) 56 (67%) 15 (58%) three) of project appropriate to the (i.e., amount of funding 10 with more than discipline, possibly in and/or number of grants 11 500,000 RMB” collaboration with 12 as principal other Universities or 13 investigator)? organizations" 14 15 16 17 “Performance of 18 exceptional distinction 5. Is any mention made and achievements that 19 requiring that research is 20 are recognized as recognized at a national distinguished 21 11 (22%) 26 (33%) 39 (47%) 11 (42%) N/A or international level? If internationally or 22 nationally (meeting the 23 so, please specify the benchmarks)" 24 requirement. 25 26 27 Non-Traditional Criteria 28 29 6. Is any mention made of “One paper cited "Achieves a citation 30 more than 20 rate or proportion of 31 citations? If quantitative, what are the thresholds times” research outputs in 32 most prestigious outlets of minimum 33 12 (24%) 23 (29%) 23 (28%) 6 (23%) […] in line with 34 requirement? Are discipline and leading 35 specific citation universities" 36 databases mentioned? 37 38 39 7. Is any mention made of 1 (2%) 1 (1%) 1 (1%) 0 (0%) N/A "Sound data 40 data sharing? If management is a basic 41 requirement for this 42 43 44 45 https://mc.manuscriptcentral.com/bmj 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 Page 31 of 58 BMJ
1 2 3 quantitative, what are (academic analysis) 4 the minimum thresholds and provides additional 5 (e.g., percentage of data guarantees for a 6 flawless methodology, that is to be made 7 for sharing and reusing 8 available)?Confidential: For Review Only data by other 9 researchers in an Open 10 Science context and for 11 the accountability of a 12 researchers own 13 academic integrity" 14 15 16 17 8. Is any mention made of 18 publishing in open access 19 mediums? If 20 quantitative, what are 0 (0%) 0 (0%) 0 (0%) 0 (0%) N/A N/A 21 the minimum thresholds 22 (e.g., percentage of 23 studies to be published in 24 open access journals)? 25 26 27 9. Is any mention made of 28 registration (including 29 preregistration 30 challenge) of studies? If 31 yes, are there thresholds 0 (0%) 0 (0%) 0 (0%) 0 (0%) N/A N/A 32 of minimum requirement 33 (e.g., percentage of 34 studies that are to be 35 registered). 36 37 10. Is any mention made of 0 (0%) 0 (0%) 0 (0%) 0 (0%) N/A N/A 38 39 adherence to reporting 40 guidelines for 41 publications? If so, are 42 43 44 45 https://mc.manuscriptcentral.com/bmj 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Page 32 of 58
1 2 3 specific guidelines 4 mentioned? 5 6 "Ghent University has 7 invested in a proper 8 Confidential: For Review Only information system of 9 research output (biblio, 10 IWETO/FRIS) for 11 many years and is 12 currently extended this 13 11. Is any mention made of to other research- 14 alternative metrics for related areas (Gismo). As soon as publications 15 sharing research (e.g., 3 (6%) 3 (4%) 2 (2%) 1 (4%) N/A and activities are 16 social media and print 17 properly registered in media)? If so, are specific 18 the information system, 19 metrics mentioned? the administrative 20 burden on researchers 21 in the context of an 22 evaluation (promotion, project applications) 23 should be reduced to a 24 minimum." 25 26 27 12. Is any mention made of 22 (45%) 28 (35%) 29 (35%) 13 (50%) "A female "Where staff have had a 28 accommodations or faculty […] career break, long term 29 adjustments to may extend absence or other 30 her contract extenuating expectations due to 31 up to two circumstances which 32 employment leave? If so, years in impact on their 33 please specify the association output/performance, 34 description of with they are encouraged to 35 accommodations (e.g., an pregnancy provide this information 36 extra year to defer and delivery including what impact 37 tenure consideration) and up to such breaks/absences 38 one year in have had on them and the type of eligible 39 case of undertaking their role." circumstances (e.g., 40 adopting a 41 parental leave, medical child six 42 43 44 45 https://mc.manuscriptcentral.com/bmj 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 Page 33 of 58 BMJ
1 2 3 leave)? years old or 4 younger." 5 6 7 8 Confidential: For Review Only 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 https://mc.manuscriptcentral.com/bmj 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Page 34 of 58
1 2 Table 3. Multiple Linear Regression Analyses of Institution Characteristics and Presence of Traditional and Non-Traditional 3 Criteria Among Full Professors (N = 81). 4 5 6 Variables 7 Traditional Criteria Non-Traditional Criteria 8 Confidential: For Review Only 9 Unstandardized β p- Unstandardized β p-value 10 value 11 (95% CI) (95% CI) 12 13 Criteria Level (Reference group = -.60 (-1.21 to 3.63) .062 -.08 (-.48 to .31) .675 14 15 Faculty Level) 16 17 Leiden Ranking -.01 (-.06 to .032) .884 .01 (-.476 to .026) .656 18 19 Human Development Index (HDI) 20 21 22 (Reference group = High HDI) 23 24 Very High HDI (n= 61) -.63 (-1.45 to 1.9) .132 .29 (-.22 to .81) .260 25 26 Continent (Reference group = Asia) 27 28 29 Australia (n= 6) 1.78 (.58 to 2.99) .004 1.07 (.32 to 1.82) .006 30 31 Europe (n= 22) .48 (-.41 to 1.36) .286 .32 (-.23 to .86) .256 32 33 North America (n= 27) .99 (.13 to 1.84) .025 .43 (-.11 to .96) .115 34 35 36 Note: bolded confidence intervals = p < .005 37 38 39 40 41 42 43 44 45 https://mc.manuscriptcentral.com/bmj 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 Page 35 of 58 BMJ
1 2 3 Table 4. Percentage of Institution Characteristics and Presence of Traditional and Non-Traditional Criteria Among Full Professors by 4 Region (N = 83). 5 6 Variables Mean Percent of Traditional Criteria Mean Percent of Non-Traditional 7 8 Confidential: For ReviewCriteria Only 9 10 11 Continent 12 13 South America (n=1) 20 0 14 15 Australia (n=6) 70 21 16 17 Europe (n=22) 45 10 18 19 North America (n=27) 64 13 20 21 Asia (n= 26) 49 3 22 23 Africa (n=1) 80 0 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 https://mc.manuscriptcentral.com/bmj 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Page 36 of 58
1 2 Table 5. Logistic Regression Analyses of Institution Characteristics and Presence of Traditional and Non-Traditional Criteria 3 Among Full Professors (N = 81). 4 5 6 Variables Criteria Criteria Not 7 Present Present 8 Confidential: ForN (%) ReviewN (%) OnlyAdjusted 95% Confidence 9 Odds Ratio Intervals 10 Lower Upper 11 Mention of authorship order 28 (34.6) 53 (65.4) 12 13 14 Criteria level (Reference group = faculty level) 17 (50.0) 17 (50.0) .351 .10 1.22 15 16 Institution Level 11 (23.4) 36 (76.6) 17 18 19 Leiden Ranking [Median (IQR)] 303.0 (556.8) 354.0 (378.0) .99 .89 1.10 20 21 Human Development Index (HDI) (Reference group 18 (30.0) 42 (70.0) 22 23 = Very High HDI) 24 25 High HDI 10 (47.6) 11 (52.4) 1.53 .30 7.69 26 27 28 Continent (Reference group =Asia) 12 (46.2) 14 (53.8) 29 30 Australia 2 (33.3) 4 (66.7) 1.21 .12 12.61 31 32 Europe 3 (13.6) 19 (86.4) .33 .05 2.22 33 34 35 North America 11 (40.7) 16 (59.3) .73 .14 3.87 36 37 38 Mention of journal impact factor 22 (27.2) 59 (72.8) 39 40 41 Criteria level (Reference group = faculty level) 9 (26.5) 25 (73.5) .73 .18 2.86 42 43 44 45 https://mc.manuscriptcentral.com/bmj 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 Page 37 of 58 BMJ
1 2 Institution Level 13 (27.7) 34 (72.3) 3 4 Leiden Ranking [Median (IQR)] 385.0 (390.0) 309.1 (448.0) .96 .85 1.08 5 6 7 Human Development Index (HDI) (Reference group 11 (18.3) 49 (81.7) 8 Confidential: For Review Only 9 = Very High HDI) 10 11 12 High HDI 11 (52.4) 10 (47.6) 3.35 .65 17.41 13 14 Continent (Reference group = Asia) 12 (46.2) 14 (53.8) 15 16 17 Australia 2 (33.3) 4 (66.7) 1.50 .13 16.87 18 19 Europe 5 (22.7) 17 (77.3) .84 .13 5.27 20 21 North America 3 (11.1) 24 (88.9) .30 .65 17.41 22 23 24 Mention of grant funding 55 (67.9) 26 (32.1) 25 26 Criteria level (Reference group = faculty level) 27 (79.4) 7 (20.6) 27 28 29 Institution Level 28 (59.6) 19 (40.4) .49 .13 1.84 30 31 Leiden Ranking [Median (IQR)] 345.0 (493.0) 324.5 (317.8) 1.01 .90 1.14 32 33 Human Development Index (HDI) (Reference group 43 (71.7) 17 (28.3) 34 35 = Very High HDI) 36 37 38 High HDI 12 (57.1) 9 (42.9) 1.9 .38 9.5 39 40 Continent (Reference group = Asia) 13 (50.0) 13 (50.0) 41 42 43 44 45 https://mc.manuscriptcentral.com/bmj 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Page 38 of 58
1 2 Europe 12 (54.5) 10 (45.4) 2.40 .44 13.10 3 4 North America 24 (88.9) 3 (11.1) 9.96 1.52 65.31 5 6 7 Mention of national or international recognition 38 (46.9) 43 (53.1) 8 Confidential: For Review Only 9 Criteria level (Reference group = faculty level) 20 (58.8) 14 (41.2) .49 .11 2.26 10 11 12 Institution Level 18 (38.3) 29 (61.7) 13 14 Leiden Ranking [Median (IQR)] 365.0 (504.5) 321.0 (408.0) 1.01 .89 1.16 15 16 Human Development Index (HDI) (Reference group 36 (60.0) 24 (40.0) 17 18 = Very High HDI) 19 20 21 High HDI 2 (9.5) 19 (90.5) .47 .05 4.14 22 23 24 Continent (reference group =Asia) 2 (7.7) 24 (92.3) 25 26 Australia 5 (83.3) 1 (16.7) 52.36 2.30 1190.95 27 28 29 Europe 9 (40.9) 13 (59.1) 6.86 .71 66.85 30 31 North America 22 (81.5) 5 (18.5) 27.44 3.26 231.16 32 33 Mention of citations 23 (28.4) 58 (71.6) 34 35 36 Criteria level (Reference group = faculty level) 8 (23.5) 26 (76.5) .66 .16 2.75 37 38 Institution Level 15 (31.9) 32 (68.1) 39 40 Leiden Ranking [Median (IQR)] 252.0 (486.0) 351.5 (409.8) .99 .88 1.11 41 42 43 44 45 https://mc.manuscriptcentral.com/bmj 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 Page 39 of 58 BMJ
1 2 Human Development Index (HDI) (Reference group 19 (31.7) 41 (68.3) 3 4 = Very High HDI) 5 6 7 High HDI 4 (19.0) 17 (81.0) 1.64 .22 12.53 8 Confidential: For Review Only 9 Continent (reference group =Asia) 4 (15.4) 22 (84.6) 10 11 Australia 5 (83.3) 1 (16.7) 47.81 2.25 1016.33 12 13 14 Europe 8 (36.4) 14 (63.6) 5.21 .58 46.80 15 16 North America 6 (22.2) 21 (77.8) 1.97 .24 16.33 17 18 Mention of accommodations/adjustments to 29 (35.8) 52 (64.2) 19 20 21 circumstances 22 23 Criteria level (Reference group = faculty level) 15 (44.1) 19 (55.9) .93 .19 4.45 24 25 26 Institution Level 14 (29.8) 33 (70.2) 27 28 Leiden Ranking [Median (IQR)] 400.0 (490.0) 324.5 (357.5) 1.00 .89 1.12 29 30 Continent (reference group =Asia) 2 (7.7) 24 (92.3) 31 32 33 Australia 4 (66.7) 2 (33.3) 5.07 .45 57.79 34 35 Europe 6 (27.3) 16 (72.7) 1.01 .15 6.95 36 37 North America 17 (63.0) 10 (37.0) 4.54 .62 33.44 38 39 40 Note: bolded confidence intervals = p < .005, IQR = interquartile range. 41 42 43 44 45 https://mc.manuscriptcentral.com/bmj 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Page 40 of 58
1 2 3 Appendix 1. Adherence to STROBE Guideline. 4 5 Item Recommendation Page Number 6 No 7 8 Title and abstract 1 (a) Indicate the study’s design with a 1 9 commonly used term in the title or the 10 abstract 11 Confidential:(b) Provide For in the abstractReview an informative Only2-3 12 and balanced summary of what was done 13 and what was found 14 Introduction 15 Background/rationale 2 Explain the scientific background and 6-7 16 rationale for the investigation being 17 reported 18 Objectives 3 State specific objectives, including any 7 19 prespecified hypotheses 20 Methods 21 Study design 4 Present key elements of study design 7 22 23 early in the paper 24 Setting 5 Describe the setting, locations, and N/A 25 relevant dates, including periods of 26 recruitment, exposure, follow-up, and 27 data collection 28 Participants 6 Cross-sectional study—Give the 8-10 29 eligibility criteria, and the sources and 30 methods of selection of participants 31 Variables 7 Clearly define all outcomes, exposures, 10 32 predictors, potential confounders, and 33 effect modifiers. Give diagnostic 34 criteria, if applicable 35 Data sources/ 8* For each variable of interest, give 10-11, Appendix 2 36 measurement sources of data and details of methods of 37 assessment (measurement). Describe 38 comparability of assessment methods if 39 40 there is more than one group 41 Bias 9 Describe any efforts to address potential N/A 42 sources of bias 43 Study size 10 Explain how the study size was arrived 8 44 at 45 Quantitative variables 11 Explain how quantitative variables were 11-12 46 handled in the analyses. If applicable, 47 describe which groupings were chosen 48 and why 49 Statistical methods 12 (a) Describe all statistical methods, 11-12 50 including those used to control for 51 confounding 52 (b) Describe any methods used to N/A 53 examine subgroups and interactions 54 55 (c) Explain how missing data were N/A 56 addressed 57 58 59 60 https://mc.manuscriptcentral.com/bmj Page 41 of 58 BMJ
1 2 3 Cross-sectional study—If applicable, N/A 4 describe analytical methods taking 5 account of sampling strategy 6 (e) Describe any sensitivity analyses N/A 7 8 9 Results 10 Participants 13* (a) Report numbers of individuals at each 13, Figure 1 11 Confidential:stage ofFor study—eg Review numbers potentially Only 12 eligible, examined for eligibility, 13 confirmed eligible, included in the study, 14 completing follow-up, and analysed 15 (b) Give reasons for non-participation at Appendix 3 16 each stage 17 Descriptive data 14* (a) Give characteristics of study 13, Table 1 18 participants (eg demographic, clinical, 19 social) and information on exposures and 20 potential confounders 21 (b) Indicate number of participants with 13-15 22 23 missing data for each variable of interest 24 Outcome data 15* Cross-sectional study—Report numbers 13-15 25 of outcome events or summary measures 26 Main results 16 (a) Give unadjusted estimates and, if 12-16 27 applicable, confounder-adjusted 28 estimates and their precision (eg, 95% 29 confidence interval). Make clear which 30 confounders were adjusted for and why 31 they were included 32 (b) Report category boundaries when N/A 33 continuous variables were categorized 34 (c) If relevant, consider translating N/A 35 estimates of relative risk into absolute 36 risk for a meaningful time period 37 Other analyses 17 Report other analyses done—eg analyses 15-16 38 39 of subgroups and interactions, and 40 sensitivity analyses 41 Discussion 42 Key results 18 Summarise key results with reference to 17-18 43 study objectives 44 Limitations 19 Discuss limitations of the study, taking 18-19 45 into account sources of potential bias or 46 imprecision. Discuss both direction and 47 magnitude of any potential bias 48 Interpretation 20 Give a cautious overall interpretation of 18-20 49 results considering objectives, 50 limitations, multiplicity of analyses, 51 results from similar studies, and other 52 relevant evidence 53 Generalisability 21 Discuss the generalisability (external 17 54 55 validity) of the study results 56 Other information 57 58 59 60 https://mc.manuscriptcentral.com/bmj BMJ Page 42 of 58
1 2 3 Funding 22 Give the source of funding and the role 23 4 of the funders for the present study and, 5 if applicable, for the original study on 6 which the present article is based 7
8 9 10 11 Confidential: For Review Only 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 https://mc.manuscriptcentral.com/bmj Page 43 of 58 BMJ
1 2 3 4 Appendix 2. Included Institution Characteristics 5 6 Level of Leiden Ranking Most Relevant 7 University Criteria of University Biomedical Faculty 8 Assessed 9 10 Faculty Bar-Ilan University 495 Faculty of Medicine 11 Confidential:Criteria For Review Only 12 Institution Department of Health 13 Brigham Young University 565 14 Criteria Sciences 15 Institution Cardiff University 213 School of Medicine 16 Criteria 17 Case Western Reserve Faculty 18 62 School of Medicine 19 University Criteria 20 Central China Normal Institution 21 790 Life Sciences 22 University Criteria 23 Faculty 24 Central South University 84 School of Medicine 25 Criteria 26 Faculty 27 China Medical University 96 Medical University 28 Criteria 29 China Pharmaceutical Faculty Basic Medicine and 30 298 31 University Criteria Clinical Pharmacy 32 Faculty 33 Colorado State University 400 School of Medicine 34 Criteria 35 Institution 36 Dankook University 575 College of Medicine Criteria 37 38 Faculty 39 Duke University 15 School of Medicine 40 Criteria 41 Institution Faculty of Medicine and 42 Ghent University 94 43 Criteria Health Sciences 44 Faculty Guangxi Medical University 328 Medical University 45 Criteria 46 47 Guangzhou Medical Faculty 354 Medical University 48 University Criteria 49 50 Faculty Gunma University 440 Faculty of Medicine 51 Criteria 52 53 54 55 56 57 58 59 60 https://mc.manuscriptcentral.com/bmj BMJ Page 44 of 58
1 2 3 4 Institution 5 Harbin Medical University 152 Medical University 6 Criteria 7 8 Johannes Gutenberg Faculty 9 232 University Medicine 10 UniversityConfidential: Mainz Criteria For Review Only 11 Department of Faculty 12 Kansas State University 580 Diagnostic Criteria 13 Medicine/Pathobiology 14 Katholieke Universiteit Institution 15 56 Faculty of Medicine 16 Leuven Criteria 17 Graduate School of Korea Advanced Institute of Faculty 18 547 Medical Science and Science and Technology Criteria 19 Engineering 20 Institution 21 Kyung Hee University 183 College of Medicine 22 Criteria 23 Lomonosov Moscow State Institution Faculty of Fundamental 24 533 25 University Criteria Medicine 26 27 Institution School of Veterinary and Murdoch University 761 28 Criteria Life Sciences 29 30 Faculty Nagasaki University 349 School of Medicine 31 Criteria 32 33 Faculty Nanchang University 420 Medical College 34 Criteria 35 36 Institution 37 Nanjing Medical University 72 Medical University 38 Criteria 39 40 National Autonomous Institution 260 School of Medicine 41 University of Mexico - UNAM Criteria 42 43 National University of Faculty 44 639 School of Medicine 45 Cordoba Criteria 46 47 Faculty School of Medicine & 48 North Dakota State University 695 Criteria Health Sciences 49 50 51 Institution Northeast Normal University 767 School of Life Science 52 Criteria 53 54 55 56 57 58 59 60 https://mc.manuscriptcentral.com/bmj Page 45 of 58 BMJ
1 2 3 4 Norwegian University of Institution Faculty of Medicine and 5 303 6 Science and Technology Criteria Health Sciences 7 8 Faculty 9 Ohio University 694 Biomedical Sciences 10 Criteria 11 Confidential: For Review Only 12 Institution College of Health Old Dominion University 728 13 Criteria Sciences 14 15 Institution 16 Philipps-Universität Marburg 348 Department of Medicine 17 Criteria 18 19 Institution Prince Songkla University 624 Faculty of Medicine 20 Criteria 21 Barts and the London 22 Queen Mary University of Institution 23 240 School of Medicine and London Criteria 24 Dentistry 25 Institution Seoul National University 14 College of Medicine 26 Criteria 27 28 Shahid Beheshti University of Institution 317 School of Medicine 29 Medical Sciences Criteria 30 31 32 Shenyang Pharmaceutical Institution 479 Medicinal Chemistry 33 University Criteria 34 35 Shiraz University of Medical Institution 36 423 School of Medicine 37 Science Criteria 38 South China Agricultural Institution 39 629 College of Life Sciences 40 University Criteria 41 42 South China University of Institution 43 630 School of Medicine Technology Criteria 44 45 Faculty St Louis University 385 School of Medicine 46 Criteria 47 48 Faculty Sun Yat-sen University 26 Medical Sciences 49 Criteria 50 51 Faculty Tulane University 308 School of Medicine 52 Criteria 53 54 55 56 57 58 59 60 https://mc.manuscriptcentral.com/bmj BMJ Page 46 of 58
1 2 3 4 Uniformed Services University Institution 421 School of Medicine 5 of the Health Sciences Criteria 6 7 8 Università degli Studi di Institution School of Medical and 253 9 Genova Criteria Pharmaceutical Sciences 10 11 Confidential: For Review Only 12 Université Catholique de Institution Faculty of Medicine and 257 13 Louvain Criteria Dentistry 14 15 Institution 16 University College Dublin 252 School of Medicine 17 Criteria 18 19 Faculty 20 University of Akron 764 Medicine 21 Criteria 22 23 Institution University of Belgrade 214 School of Medicine 24 Criteria 25 26 27 Institution School of Medical 28 University of Campinas 186 29 Criteria Sciences 30 31 Institution 32 University of Chile 415 Faculty of Medicine 33 Criteria 34 35 University of Colorado, Faculty 36 51 School of Medicine Denver Criteria 37 38 39 Institution University of East Anglia 504 Medical School 40 Criteria 41 42 43 Institution Faculty of Medicine and University of Florence 177 44 Criteria Dentistry 45 46 47 Institution University of Freiburg 142 Faculty of Medicine 48 Criteria 49 50 Faculty 51 University of Kansas 181 School of Medicine 52 Criteria 53 54 55 56 57 58 59 60 https://mc.manuscriptcentral.com/bmj Page 47 of 58 BMJ
1 2 3 4 University of Maryland, Faculty 5 98 School of Medicine 6 Baltimore Criteria 7 8 Institution 9 University of Melbourne 21 Medicine School 10 Criteria 11 Confidential: For Review Only University of Minnesota - Faculty 12 32 Department of Medicine 13 Twin Cities Criteria 14 15 University of Modena and Institution Faculty of Medicine and 16 397 Reggio Emilia Criteria Surgery 17 18 19 Institution University of Montpellier 321 Faculty of Medicine 20 Criteria 21 22 23 Institution University of Navarra 433 School of Medicine 24 Criteria 25 26 27 University of Nevada, Las Faculty 686 School of Medicine 28 Vegas Criteria 29 30 31 Faculty University of Nevada, Reno 657 School of Medicine 32 Criteria 33 34 35 University of New South Institution 36 87 Medicine 37 Wales Criteria 38 39 Faculty College of Osteopathic 40 University of North Texas 567 41 Criteria Medicine 42 43 Faculty 44 University of Oklahoma 234 College of Medicine Criteria 45 46 47 Institution University of Oslo 73 Faculty of Medicine 48 Criteria 49 50 51 Faculty 52 University of Pittsburgh 6 School of Medicine Criteria 53 54 Faculty 55 University of Pretoria 549 School of Medicine Criteria 56 57 58 59 60 https://mc.manuscriptcentral.com/bmj BMJ Page 48 of 58
1 2 3 4 Institution University of Queensland 50 Faculty of Medicine 5 Criteria 6 7 8 Faculty 9 University of Saskatchewan 345 College of Medicine 10 Criteria 11 Confidential: For Review Only 12 Institution School of Pharmacy and 13 University of South Australia 437 14 Criteria Medical Sciences 15 16 Institution 17 University of St Andrews 689 School of Medicine 18 Criteria 19 20 21 Institution Institute of Pharmacy University of Strathclyde 619 22 Criteria and Biomedical Sciences 23 24 25 Institution 26 University of Sussex 555 Medical School 27 Criteria 28 29 Institution 30 University of Szeged 470 Faculty of Medicine 31 Criteria 32 33 34 Faculty Faculty of Medicine and University of Tampere 386 35 Criteria Life Sciences 36 37 38 University of Tennessee, Institution Graduate School of 187 39 Knoxville Criteria Medicine 40 41 42 Faculty College of Medicine and University of Toledo 487 43 Criteria Life Sciences 44 45 46 Faculty University of Utah 80 School of Medicine 47 Criteria 48 49 50 Faculty Faculty of Medicine and University of Valencia 278 51 Criteria Odontology 52 53 54 University of Western Institution 178 Medical School 55 Australia Criteria 56 57 58 59 60 https://mc.manuscriptcentral.com/bmj Page 49 of 58 BMJ
1 2 3 4 Institution Schulich School of 5 University of Windsor 784 6 Criteria Medicine and Dentistry 7 8 Institution 9 Utrecht University 37 Faculty of Medicine Criteria 10 11 Confidential: For Review Only 12 Faculty Faculty of Medical Wuhan University 138 13 Criteria Sciences 14 15 16 Faculty School of Basic Medical Xi'an Jiaotong University 157 17 Criteria Sciences 18 19 Institution School of Life Sciences 20 Xidian University 842 21 Criteria and Technology 22 23 24 Institution Yamagata University 634 Faculty of Medicine 25 Criteria 26 27 28 Institution Yeungnam University 476 College of Medicine 29 Criteria 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 https://mc.manuscriptcentral.com/bmj BMJ Page 50 of 58
1 2 3 Appendix 3. Categorization of Institutions Not Included in Analyses 4 5 6 Emailed Institution Twice with No Response 1. Akdeniz University* 7 2. Budapest University of Technology and Economics* 8 Confidential:3. ChungnamFor National Review University Only 9 4. COMSATS Institute of Information Technology 10 5. Ehime University 11 6. Ernst-Moritz-Arndt University of Greifswald 12 13 7. Federal University of São Carlos - UFSCar 14 8. Fourth Military Medical University 15 9. Graz University of Technology* 16 10. Jiangnan University 17 11. Jilin University 18 12. Kindai University 19 13. Kobe University 20 21 14. Lille 1 University of Science and Technology 22 15. Medical University of Graz 23 16. Metropolitan Autonomous University 24 17. National Technical University of Athens* 25 18. National Tsing Hua University 26 19. National Yang Ming University 27 20. Nicholas Copernicus University of Torun 28 29 21. Osaka University 30 22. Peking Union Medical College 31 23. Saarland University 32 24. Shanghai Jiao Tong University 33 25. Tokyo Medical and Dental University 34 26. Tokyo University of Agriculture and Technology 35 36 27. Universidad Nacional de La Plata* 37 28. Universidade Federal do Paraná 38 29. Universidade Federal do Rio de Janeiro 39 30. Université Paul Sabatier 40 31. Universiti Teknologi Malaysia 41 42 43 44 45 https://mc.manuscriptcentral.com/bmj 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 Page 51 of 58 BMJ
1 2 3 32. University of Hyderabad 4 33. University of La Laguna 5 6 34. University of Minho 7 35. University of Paris XII - Paris-Est Créteil Val de Marne 8 Confidential:36. UniversityFor of SilesiaReview Only 9 37. University of Southampton 10 38. University of Thessaly 11 39. University of Tokyo 12 13 40. University of Toyama 14 41. University of Valladolid 15 42. Waseda University 16 43. Wroclaw University of Technology 17 18 Institution Reported Not being Willing or 1. City University London 19 Able to Share Documents 2. Erasmus University Rotterdam* 20 3. Freie Universität Berlin 21 4. Heidelberg University 22 5. Karolinska Institute 23 24 6. University of Tartu 25 26 Institution Provided Documents for Hiring 1. University of São Paulo 27 Only 2. Tohoku University 28 3. University of Bonn 29 4. University of Paris VII - Paris Diderot 30 5. University of Santiago de Compostela 31 32 6. Aalborg University 33 7. Gifu University 34 8. Yamaguchi University 35 36 Guidelines Only Available for Positions Not 1. University of Münster 37 Included in Analyses 2. Julius Maximilian University of Würzburg 38 3. University of Girona 39 40 41 42 43 44 45 https://mc.manuscriptcentral.com/bmj 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Page 52 of 58
1 2 3 Documents Available but No Relevant 1. Anna University* 4 Faculty 2. Beihang University* 5 6 3. Carnegie Mellon University* 7 4. Delft University of Technology* 8 Confidential:5. EindhovenFor University Review of Technology* Only 9 6. George Mason University* 10 7. Georgia Institute of Technology* 11 8. Hefei University of Technology* 12 13 9. Iowa State University* 14 10. Nanjiang Agricultural University* 15 11. Roma Tre University* 16 12. Sichuan Agricultural University* 17 13. Technische Universität Darmstadt* 18 14. University of Bath* 19 15. University of Nebraska, Lincoln* 20 21 16. University of Salento* 22 17. University of Stuttgart* 23 18. Vienna University of Technology* 24 Note: * = institutions did not have a relevant biomedical faculty 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 https://mc.manuscriptcentral.com/bmj 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 Page 53 of 58 BMJ
1 2 3 Appendix 4. Thresholds for Criteria 4 5 6 Criteria Relevant Information from Institution Document (number of institutions) 7 Peer Reviewed Publications At least 3 publications. (n=1) 8 Confidential: For Review Only 9 Continued scholarly publication (40 publications typically). (n=1) 10 11 Three articles for a 4 year contract; five articles for a 6-year contract. (n=1) 12 13 14 Three first authored papers published on important journals. (n=1) 15 16 One to three publications are required. (n=1) 17 18 Two publications per year are expected. (n=2) 19 20 21 Three to two papers required (varies based on impact factor of journals published in). 22 (n=1) 23 24 Three to six papers required (varies based on impact factor of journals published in). 25 (n=1) 26 27 28 Five to ten papers required (varies based on impact factor of journals published in). 29 (n=2) 30 31 Two to four papers required (varies based on impact factor of journals published in, 32 awards won, and citations). (n=1) 33 34 Two published papers required. (n=3) 35 36 37 Four published papers required. (n=1) 38 39 Five published papers since the last promotion. (n=2) 40 41 42 43 44 45 https://mc.manuscriptcentral.com/bmj 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Page 54 of 58
1 2 3 Six published papers required. (n=1) 4 5 6 Seven published papers required. (n=3) 7 8 Confidential:Eight published papers For required. Review (n=3) Only 9 10 Ten published papers required. (n=1) 11 12 13 Twelve published papers required. (n=1) 14 15 Twenty-eight publications preferred. (n=1) 16 17 One to six papers required (varies based on impact factor of journals published in). (n=1) 18 19 Requirement varies based on field of medicine, ranging from 14 - 53 publications in the 20 21 last 10 years. (n=3) 22 23 Authorship Order First or senior author on 1/3 of manuscripts. (n=1) 24 25 Greater points earned for single author, first author, or corresponding author on papers. 26 (n=1) 27 28 29 Sole or lead authored publications are encouraged. (n=1) 30 31 Three first authored papers published in important journals. (n=1) 32 33 Expectation that candidates for promotion will have at least two publications per year 34 with a significant portion as first or last author. (n=1) 35 36 37 Substantial research activity [..] as evidence by a substantial list of first or senior 38 authored high quality peer-reviewed publications. (n=3) 39 40 Two papers as first or corresponding author. (n=1) 41 42 43 44 45 https://mc.manuscriptcentral.com/bmj 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 Page 55 of 58 BMJ
1 2 3 4 Six papers as first or corresponding author. (n=2) 5 6 7 Seven papers as first or corresponding author. (n=1) 8 Confidential: For Review Only 9 Eight papers as first or corresponding author. (n=1) 10 11 Publishes as primary, senior, or corresponding author at least twice per year. (n=1) 12 13 14 At least 1 paper as first author. (n=1) 15 16 At least 1 sole authored paper. (n=1) 17 18 Five papers as first or corresponding author. (n=1) 19 20 21 First author on one of five papers. (n=1) 22 23 The specific role, such as lead author or otherwise, should be noted. (n=1) 24 25 Candidate will be expected to be lead, senior or corresponding author on most recent 26 publications. (n=1) 27 28 29 Lead contributor or corresponding contributor of publications. (n=1) 30 31 Promotion requires sole, major, or senior authorship of high-impact publications. (n=1) 32 33 Corresponding author of at least five publications since previous promotion. (n=1) 34 35 36 Recent publications which the applicant is first or senior author. (n=1) 37 38 Applicant must be leading author (first or last) in the last three years. (n=1) 39 40 Independence as documented by senior authorship. (n=1) 41 42 43 44 45 https://mc.manuscriptcentral.com/bmj 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Page 56 of 58
1 2 3 4 At least 5 accredited publications with some as first author. (n=1) 5 6 7 Impact Factor At least one paper with impact factor >3.0; at least one paper with impact factor >5.0, or 8 Confidential:cumulate impact factor For rates totalling Review to >11.0. (n=1) Only 9 10 At least one paper with impact factor >9.0; or cumulate impact factor rates totalling to 11 >30.0. (n=1) 12 13 14 At least one paper with impact factor higher than the average level of the specific area (if 15 too few publications as first or senior author the impact factor must be higher). (n=1) 16 17 At least three paper with impact factors ranked as second level or higher on the journal 18 citation reports. (n=1) 19 20 21 At least three papers with impact factor >5.0; at least one paper with impact factor 22 >10.0, or cumulate impact factor rates totalling to >20.0. (n=1) 23 24 At least one paper with impact factor >10.0; or at least one paper with impact factor 25 >15.0. (n=1) 26 27 28 At least three papers after the last evaluation with a cumulative impact factor of >4.0. 29 (n=1) 30 31 At least 60% of papers in journals that have an impact factor. (n=1) 32 33 At least one paper in a Journal Citation Reports II area; or two papers with impact factor 34 >3.0. (n=1) 35 36 37 Table provided from guideline with varying requirements for impact factor dependent on 38 the journal. (n=1) 39 40 41 42 43 44 45 https://mc.manuscriptcentral.com/bmj 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 Page 57 of 58 BMJ
1 2 3 At least 7 papers with total impact factor >5.0; one paper with an impact factor >3.0; or 4 one paper in journal citation reports II area. A second option is that papers reach an 5 6 impact factor of at least 2 on the journal citation reports. A third option is that the 7 professor has at least 7 relevant papers as first or corresponding author with all of the 8 Confidential:impact factors >3.0. For (n=1) Review Only 9 10 Grant Funding Principal investigator of one provincial (or higher level) grant. (n=1) 11 12 13 Five to seven years of funding. (n=1) 14 15 Principal investigator of one provincial (or higher level) grant; principal investigator of 16 one national science foundation general project; principal investigator of vertical 17 projects with total amount reach 500,000 RMB or above; or principal investigator of one 18 national science foundation youth project; or principal investigator of vertical projects 19 with total amount reach 300,000 RMB or above. (n=1) 20 21 22 Acquiring a multi-year extramural grant award as a principal investigator at the funding 23 level of NIH R01, or higher; acquiring as a principal investigator a grant award 24 comparable to NIH R01 from other agencies or foundations. (n=1) 25 26 Two national grants or one with 1,800,000 RMB. (n=1) 27 28 29 Principal investigator of one national science foundation general project; or principal 30 investigator of sub-project of National Science and Technology Major Project; or 31 principal investigator of National Key Research and Development Program, with 32 research fund 1,000,000 RMB; or 500,000 RMB for non-provincial key disciplines. 33 (n=1) 34 35 36 Principal investigator of two national natural science foundation project; or Principal 37 investigator of project with more than 2,000,000 RMB. 38 39 Principal investigator of one national project and some publications. (n=1) 40 41 42 43 44 45 https://mc.manuscriptcentral.com/bmj 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 BMJ Page 58 of 58
1 2 3 At least 1 grant funded since the last promotion. 4 5 6 7 Awarded with national second prize or provincial first prize; or expenses for the 8 Confidential:technology transfers For reached 100,000 Review RMB, or scientific achievementOnly application 9 created more than 500,000 RMB, or gained 1000,000 RMB economic benefit. These 10 could replace the requirements for the publication. (n=1) 11 12 Principal investigator of category A project with disposable funding amounting 250,000 13 14 for level four professors, 400,000 for level three professors, and 600,000 for level two 15 professors. (n=1) 16 17 Principal investigator of one National Natural Science Foundation project. (n=1) 18 19 Must be principal investigator of two national grants. (n=1) 20 21 22 Citations Threshold required for number of citations in the last 10 years for promotion. H-index 23 threshold for the past 10 years also required, ranges from 5-12. (n=1) 24 25 Threshold required for number of citations in the last 10 years ranging from 16 - 406 26 based on medical specialty. H-index threshold for the past 10 years also required, ranges 27 from 5-12. (n=2) 28 29 30 One paper need to be cited 20 to 30 times. Or 100- 150 cumulative citations among less 31 than five papers. (n=1) 32 33 […] published one SCI paper with more than 30 times citation by others. (n=1) 34 35 36 "Citations from 10 hetero citations." (n=1) 37 Note: Data from non-English schools has been translated English. 38 39 40 41 42 43 44 45 https://mc.manuscriptcentral.com/bmj 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60