BMJ Confidential: For Review Only Academic criteria for promotion and tenure in biomedical sciences faculties: a cross-sectional analysis of an international sample of universities Journal: BMJ Manuscript ID BMJ-2019-053055.R1 Article Type: Research BMJ Journal: BMJ Date Submitted by the 10-Feb-2020 Author: Complete List of Authors: Rice, Danielle; McGill University; Ottawa Hospital Research Institute, Raffoul, Hana; University of Waterloo; Ottawa Hospital Research Institute Ioannidis, John; Stanford University, Stanford Prevention Research Center, Department of Medicine and Department of Health Research and Policy Moher, David; Ottawa Hospital Research Institute, Ottawa Methods Centre Keywords: Promotion, Tenure, Faculty of Medicine, Incentives https://mc.manuscriptcentral.com/bmj Page 1 of 58 BMJ Faculty of Biomedical Sciences Promotion and Tenure Criteria 1 2 3 1 Academic criteria for promotion and tenure in biomedical sciences faculties: a cross-sectional 4 5 6 2 analysis of an international sample of universities 7 8 3 9 10 4 Danielle B Rice1,2, Hana Raffoul2,3, John PA Ioannidis4,5,6,7, David Moher8,9 11 Confidential: For Review Only 12 5 13 14 15 6 1Department of Psychology, McGill University, Montreal, Quebec, Canada; 2Ottawa Hospital 16 17 7 Research Institute, Ontario, Canada; 3Faculty of Engineering, University of Waterloo, Waterloo, 18 19 8 Ontario, Canada; 4Departments of Medicine, 5Health Research and Policy, 6Biomedical Data 20 21 7 22 9 Science, and Statistics, and Meta-Research Innovation Center at Stanford (METRICS), Stanford 23 24 10 University, Stanford, California, USA; 8Centre for Journalology, Clinical Epidemiology Program, 25 26 11 Ottawa Hospital Research Institute, Ontario, Canada; 9School of Epidemiology and Public Health, 27 28 29 12 University of Ottawa, Ottawa, Canada 30 31 13 32 33 14 Corresponding Author 34 35 15 David Moher 36 37 16 [email protected] 38 17 39 40 18 Danielle Rice: ORCID 0000-0001-5615-7005 41 42 19 David Moher: ORCID 0000-0003-2434-4206 43 44 20 John Ioannidis: ORCID 0000-0003-3118-6859 45 46 21 47 48 49 50 51 52 53 54 55 56 57 58 1 59 60 https://mc.manuscriptcentral.com/bmj BMJ Page 2 of 58 Faculty of Biomedical Sciences Promotion and Tenure Criteria 1 2 3 1 Affiliations: 4 5 6 2 Centre for Journalology, Clinical Epidemiology Program, Ottawa Hospital Research Institute, 7 8 3 Ontario, Canada, K1H 8L6 9 10 4 Danielle B Rice (Doctoral student) 11 Confidential: For Review Only 12 5 Hana Raffoul (Undergraduate student) 13 14 6 David Moher (Director) 15 16 7 Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, 17 18 8 California, USA, 94305 19 20 9 John PA Ioannidis (Co-Director) 21 22 10 23 24 25 11 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 2 59 60 https://mc.manuscriptcentral.com/bmj Page 3 of 58 BMJ Faculty of Biomedical Sciences Promotion and Tenure Criteria 1 2 3 1 ABSTRACT 4 5 6 2 Objectives: To determine the presence of a set of pre-specified traditional and non-traditional 7 8 3 criteria used to assess scientists for promotion and tenure in faculties of biomedical sciences among 9 10 4 universities worldwide. 11 Confidential: For Review Only 12 5 Design: Cross-sectional study. 13 14 15 6 Setting: Not applicable. 16 17 7 Participants: 170 randomly selected universities from the Leiden Ranking of world universities 18 19 8 list were considered. 20 21 22 9 Main outcome measures: Two independent reviewers searched for all guidelines applied when 23 24 10 assessing scientists for promotion and tenure for institutions with biomedical faculties. Where 25 26 11 faculty-level guidelines were not available, institution-level guidelines were sought. Available 27 28 29 12 documents were reviewed and the presence of 5 traditional (e.g., number of publications) and 7 30 31 13 non-traditional (e.g., data sharing) criteria was noted in guidelines for assessing assistant 32 33 14 professors, associate professors, professors, and the granting of tenure. 34 35 15 Results: A total of 146 institutions had faculties of biomedical sciences with 92 having eligible 36 37 38 16 guidelines available to review. Traditional criteria mentioned peer-reviewed publications, 39 40 17 authorship order, journal impact, grant funding, and national or international reputation in 95%, 41 42 18 37%, 28%, 67%, and 48% of the guidelines, respectively. Conversely, among non-traditional 43 44 45 19 criteria only citations (any mention in 26%) and accommodations for employment leave (37%) 46 47 20 were relatively commonly mentioned; while there was rare mention of alternative metrics for 48 49 21 sharing research (2%) and data sharing (1%), and 3 criteria (publishing in open access mediums, 50 51 52 22 registering research, and adhering to reporting guidelines) were not found in any institution 53 54 23 reviewed. Traditional criteria were more commonly reported than non-traditional criteria (p= .001). 55 56 57 58 3 59 60 https://mc.manuscriptcentral.com/bmj BMJ Page 4 of 58 Faculty of Biomedical Sciences Promotion and Tenure Criteria 1 2 3 1 We observed notable differences across continents on whether guidelines are accessible or not 4 5 6 2 (Australia 100%, North America 97%, Europe 50%, Asia 58%, South America 17%), and more 7 8 3 subtle differences on the use of specific criteria. 9 10 4 Conclusions: This study demonstrates that the current evaluation of scientists emphasizes 11 Confidential: For Review Only 12 5 traditional criteria as opposed to non-traditional criteria. This may reinforce research practices that 13 14 15 6 are known to be problematic while insufficiently supporting the conduct of better-quality research 16 17 7 and open science. Institutions should consider incentivizing non-traditional criteria. 18 19 8 Registration: Open Science Framework (https://osf.io/26ucp/) 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 4 59 60 https://mc.manuscriptcentral.com/bmj Page 5 of 58 BMJ Faculty of Biomedical Sciences Promotion and Tenure Criteria 1 2 3 1 What is already known on this topic: 4 5 6 2 • Academics tailor their research practices based on the evaluation criteria applied within 7 8 3 their academic institution. 9 10 4 • Ensuring that biomedical researchers are incentivized by adhering to best practice 11 Confidential: For Review Only 12 5 13 guidelines for research is essential given the clinical implications of this work. 14 15 6 • While changes to the criteria used to assess professors and confer tenure have been 16 17 7 recommended, a systematic assessment of promotion and tenure criteria being applied 18 19 8 worldwide has not been conducted. 20 21 22 23 9 What this study adds: 24 25 26 10 • Across countries, university guidelines focus on rewarding traditional research criteria 27 28 29 11 (peer-reviewed publications, authorship order, journal impact, grant funding, and national 30 31 12 or international reputation). 32 33 13 • The minimum written requirements for promotion and tenure criteria are predominantly 34 35 36 14 objective in nature, although several of them are inadequate measures to assess the 37 38 15 impact of researchers. 39 40 16 • Developing and evaluating more appropriate, non-traditional indicators of research may 41 42 17 facilitate changes in the evaluation practices for rewarding researchers. 43 44 45 46 18 47 48 19 49 50 51 52 53 54 55 56 57 58 5 59 60 https://mc.manuscriptcentral.com/bmj BMJ Page 6 of 58 Faculty of Biomedical Sciences Promotion and Tenure Criteria 1 2 3 1 INTRODUCTION 4 5 6 2 There are important deficiencies in the quality and transparency of research conducted across 7 8 3 disciplines. 1 2 Numerous efforts have been made to combat these inadequacies by developing, for 9 10 4 example, reporting guidelines (e.g., the CONSORT and PRISMA Statements), registration of 11 Confidential: For Review Only 12 5 studies prior to data collection (e.g., clinicaltrials.gov), and data sharing practices.3 4 Despite these 13 14 15 6 strategies, poorly conducted and inadequately reported research remains highly prevalent.5 This 16 17 7 has important consequences, especially in the field of medicine, as research is heavily relied upon 18 19 8 to inform clinical decision-making. 20 21 22 9 Institutions have the ability to influence large-scale improvements among researchers, as 23 24 10 universities hire new faculty, and promote and tenure existing faculty. Universities can provide 25 26 11 incentives and rewards (e.g., promotions) for scholarly work that is conducted appropriately, 27 28 29 12 reported transparently, and adheres to best publication practices. A recent survey conducted in the 30 31 13 UK found that academics tailor their publication practices to align with their institutional 32 33 14 evaluation criteria.6 These criteria, however, may include metrics that are known to be problematic 34 35 15 for assessing researchers.7 Current incentives and rewards may also be misaligned with the needs 36 37 38 16 of society. Reward systems within universities typically include criteria within promotion and 39 40 17 tenure documents such as the quantity of publications and novelty of findings rather than the 41 42 18 reliability, accuracy, reproducibility and transparent reporting of findings.8 Inappropriate criteria 43 44 9 45 19 being applied for career advancement can inadvertently contribute to research waste, with billions 46 47 20 of dollars invested in non-usable research.10 For example, universities that emphasize the quantity 48 49 21 of published papers, can increase undeserved authorship, salami slicing, and publication in very 50 51 52 22 low-quality journals (e.g., predatory journals) without peer-review, and contribute to the problems 53 54 23 of reproducibility.