Competency-Based Assessment for the Training of Phd Students and Early-Career Scientists
Total Page:16
File Type:pdf, Size:1020Kb
FEATURE ARTICLE POINT OF VIEW Competency-based assessment for the training of PhD students and early-career scientists Abstract The training of PhD students and early-career scientists is largely an apprenticeship in which the trainee associates with an expert to become an independent scientist. But when is a PhD student ready to graduate, a postdoctoral scholar ready for an independent position, or an early-career scientist ready for advanced responsibilities? Research training by apprenticeship does not uniformly include a framework to assess if the trainee is equipped with the complex knowledge, skills and attitudes required to be a successful scientist in the 21st century. To address this problem, we propose competency-based assessment throughout the continuum of training to evaluate more objectively the development of PhD students and early-career scientists. DOI: https://doi.org/10.7554/eLife.34801.001 MICHAEL F VERDERAME†*, VICTORIA H FREEDMAN†, LISA M KOZLOWSKI† AND WAYNE T MCCORMACK† The quality of formal training assessment mechanisms. This variation in the quality and received by PhD students and early-career scien- substance of training assessment for PhD stu- tists (a label that covers recent PhD graduates in dents and early-career scientists (Maki and Bor- a variety of positions, including postdoctoral kowski, 2006) underscores the need for an trainees and research scientists in entry-level improved approach to such assessment. positions) is highly variable, and depends on a The value of bringing more definition and number of factors: the trainee’s supervisor or structure to the training environment has been research adviser; the institution and/or graduate recognized by professional organizations such as *For correspondence: mxv8@psu. program; and the organization or agency fund- the National Postdoctoral Association, the edu ing the trainee. The European approach, for American Physiological Society/Association of †These authors contributed example, relies more on one final summative Chairs of Departments of Physiology, and some equally to this work assessment (that is, a high stakes evaluation at educational institutions and individual training Competing interests: The the conclusion of training, e.g. the dissertation programs. In addition, a recent NIH Funding authors declare that no and defense), whereas US doctoral programs Opportunity Announcement places increased competing interests exist. rely more on multiple formative assessments emphasis on the development of both research Funding: See page 5 (regular formal and informal assessments to eval- and career skills, with a specific charge that uate and provide feedback about performance) “Funded programs are expected to provide evi- Reviewing editor: Emma before the final dissertation defense dence of accomplishing the training objectives”. Pewsey, eLife, United Kingdom (Barnett et al., 2017). Funding agencies in the Lists of competencies and skills provide guide- Copyright Verderame et al. US such as the National Science Foundation lines for training experiences but they are rarely This article is distributed under (NSF) and the National Institutes of Health (NIH) integrated into training assessment plans. the terms of the Creative have recently increased expectations for formal Based on our experience as graduate and Commons Attribution License, which permits unrestricted use training plans for individuals supported by indi- postdoctoral program leaders, we recognized and redistribution provided that vidual or institutional training grants the need both to identify core competencies the original author and source are (NIH, 2012); but these agencies support only a and to develop a process to assess these com- credited. small fraction of PhD trainees via these funding petencies. To minimize potential confirmation Verderame et al. eLife 2018;7:e34801. DOI: https://doi.org/10.7554/eLife.34801 1 of 5 Feature article Point of View Competency-based assessment for the training of PhD students and early-career scientists bias we deliberately chose not to begin this proj- and executing protocols, troubleshooting, lab ect with a detailed comparison of previously safety, and data management. described competencies. Each author indepen- Computational Skills encompasses relevant dently developed a list of competencies based statistical analysis methods and informatics on individual experiences. Initial lists were wide- literacy. ranging, and included traditional fundamental Collaboration and Team Science Skills research skills (e.g., critical thinking skills, includes openness to collaboration, self- and dis- computational and quantitative skills), skills ciplinary awareness, and the ability to integrate needed for different career pathways, (e.g., information across disciplines. teaching skills), and business and management Responsible Conduct of Research (RCR) and skills (e.g., entrepreneurial skills such as the abil- Ethics includes knowledge about and adherence ity to develop a business or marketing plan). to RCR principles, ethical decision making, moral Although we recognize that many of the compe- courage, and integrity. tencies we initially defined are important in spe- Communication Skills includes oral and writ- cific careers, from the combined list we defined ten communication skills as well as communica- 10 core competencies essential for every PhD tion with different stakeholders. scientist regardless of discipline or career path- Leadership and Management Skills includes way (Table 1). the ability to formulate a research vision, man- age group dynamics and communication, orga- nize and plan, make decisions, solve problems, Core competencies and and manage conflicts. subcompetencies Survival Skills includes a variety of personal Broad Conceptual Knowledge of a Scientific characteristics that sustain science careers, such Discipline refers to the ability to engage in pro- as motivation, perseverance, and adaptability, as ductive discussion and collaboration with col- well as participating in professional development leagues across a discipline (such as biology, activities and networking skills. chemistry, or physics). Because each core competency is multi-fac- Deep Knowledge of a Specific Field encom- eted, we defined subcompetencies. For exam- passes the historical context, current state of the ple, we identified four subcompetencies of art, and relevant experimental approaches for a Critical Thinking Skills: (A) Recognize important specific field, such as immunology or questions; (B) Design a single experiment nanotechnology. (answer questions, controls, etc.); (C) Interpret Critical Thinking Skills focuses on elements data; and (D) Design a research program. Each of the scientific method, such as designing core competency has between two to seven sub- experiments and interpreting data. competencies, resulting in a total of 44 subcom- Experimental Skills includes identifying petencies (Table 1—source data 1: Core appropriate experimental protocols, designing Competencies Assessment Rubric). Table 1. Ten Core Competencies for the PhD Scientist. 1. Broad Conceptual Knowledge of a Scientific Discipline 2. Deep Knowledge of a Specific Field 3. Critical Thinking Skills 4. Experimental Skills 5. Computational Skills 6. Collaboration and Team Science Skills 7. Responsible Conduct of Research and Ethics 8. Communication Skills 9. Leadership Skills 10. Survival Skills DOI: https://doi.org/10.7554/eLife.34801.002 The following source data available for Table 1: Source data 1. Core Competencies Assessment Rubric. DOI: https://doi.org/10.7554/eLife.34801.005 Verderame et al. eLife 2018;7:e34801. DOI: https://doi.org/10.7554/eLife.34801 2 of 5 Feature article Point of View Competency-based assessment for the training of PhD students and early-career scientists Assessment milestones research adviser or job supervisor, to determine Individual competencies could be assessed using if a PhD student or early-career scientist had a Likert-type scale (Likert, 1932), but such rat- reached the milestone for their stage of training ings can be very subjective (e.g., “poor” to (Table 1—source data 1: Core Competencies “excellent”, or “never” to “always”) if they lack Assessment Rubric). A sample for the Critical specific descriptive anchors. To maximize the Thinking Skills core competency is shown in usefulness of a competency-based assessment Table 3. rubric for PhD student and early-career scientist training in any discipline, we instead defined Recommendations for use observable behaviors corresponding to the core We suggest that such a competency-based competencies that reflect the development of assessment be used to guide periodic feedback knowledge, skills and attitudes throughout the between PhD students or early-career scientists timeline of training. and their mentors or supervisors. It is not meant We used the “Milestones” framework to be a checklist. Rather than assessing all 44 described by the Accreditation Council for Grad- subcompetencies at the same time, we recom- uate Medical Education: “Simply defined, a mile- mend that subsets of related competencies (e. stone is a significant point in development. For g., “Broad Conceptual Knowledge of a Scientific accreditation purposes, the Milestones are com- Discipline” and “Deep Knowledge of a Specific petency-based developmental outcomes (e.g., Field”) be considered during any given evalua- knowledge, skills, attitudes, and performance) tion period (e.g., month or quarter). Assessors that can be demonstrated progressively