THE SURGICAL PROCEDURE FEEDBACK RUBRIC FOR ASSESSING RESIDENT PERFORMANCE IN THE OPERATING ROOM: DOES IT WORK? by Ayca Melis Toprak A thesis submitted to the Department of Education In conformity with the requirements for the degree of Masters of Education Queen’s University Kingston, Ontario, Canada (July, 2015) Copyright © Ayca Toprak, 2015 Abstract Competency-based training models in surgical education challenge programs to adequately document and assess trainees’ clinical decision-making, problem solving, and procedural skills in the operating room. While a variety of tools for the assessment of procedural skills exist, validated intraoperative assessment tools are scarce. Those that have been validated often employ checklists or numerical rating scales, which are prone to bias and provide limited feedback to residents. The Surgical Procedure Feedback Rubric (SPR) was developed by the Department of Surgery at Queen’s University to document the quality of resident performance during a single, directly observed operative encounter and provide targeted feedback to support learning. It differs from other assessment tools because it defines performance criteria by increasing complexity through the use of behavioural anchors, thus embedding standards of performance in the tool. This study begins the process of building a validity argument for the SPR. Validity in educational assessment is an evaluation of the appropriateness of inferences made with the use of an assessment tool. Kane and Stobart’s validity frameworks guided this study. The purpose of this study was to examine the inference that the SPR is able to distinguish between the intraoperative performances of different levels of learners. A 14-month observational study was conducted in the General Surgery, Orthopaedic and Obstetrics and Gynecology training programs. Document analysis of the SPR provided evidence that the SPR was being used as intended. Exploratory factor analysis identified a 3-factor structure of the SPR consisting of Operating Room Preparation, Technical Skills and Intrinsic Competencies. Three-way (PGY x Program x Role) ANOVA confirmed the results of factor ii analysis, showed the utility of the SPR in discriminating between residents by postgraduate training year (PGY), and provided evidence for partial transferability of the SPR between programs and different roles in surgery. These results contributed to evidence to support scoring, generalizability and extrapolations inferences in the validity argument of the SPR. In addition to supporting the validity argument for the SPR itself, results of this study provide evidence for the use of rubric-based assessment tools to support competency-based assessment systems in medical education. iii Acknowledgements I would like to take this opportunity to thank all the people who have supported me during the pursuit of this Master’s degree. Without them none of this would have been possible. I would like to start by acknowledging the Clinical Investigator Program, Postgraduate Medical Education office and Division of General Surgery at Queen’s University for giving me the opportunity to take the time out of my residency training to pursue my research interests. Thank you to the Social Sciences and Humanities Research Council of Canada and the Postgraduate Medical Education office for funding this research. To my supervisor, Lyn Shulha, and committee members, Don Klinger and Laura McEwen, you have all provided great inspiration, guidance and encouragement throughout the last two years. Thank you for always being there with open doors to answer questions, eliminate fears, guide my learning, and expand my thinking. A special thank you to Laura McEwen. If you had not taken a chance on me four years ago and introduced me to the world of medical education I would not be here today. You have been my champion, teacher, friend and confidant from the beginning. I look forward to the next four years working together and what it will bring. To all the residents, staff, and faculty in the Departments of Surgery and Obstetrics and Gynecology, thank you for participating in this study and taking a chance on a new idea. In addition, I would like to recognize my mentors in the Department of Surgery. Dr. Jones, thank you for trusting me to continue the work on the Surgical Procedure Feedback Rubric. Even with the distance, I have felt your unwavering support and encouragement. Dr. Winthrop, your insight and support has been invaluable. Dr. Reznick, I have no words to express my gratitude for your iv encouragement in my academic pursuits. You are a pioneer in the field of surgical education. It has been an honor to learn from you. Finally, to my mother and husband, my accomplishments would never be possible without your unconditional love, patience, and understanding. Thank you for always encouraging me to pursue my dreams and for providing the emotional and practical supports to make it through the journey. I love you both. v Table of Contents Abstract ........................................................................................................................................... ii Acknowledgements ........................................................................................................................ iv List of Figures ................................................................................................................................. x List of Tables ................................................................................................................................. xi List of Abbreviations .................................................................................................................... xii Chapter 1 : Introduction .................................................................................................................. 1 Context ........................................................................................................................................ 5 Queen’s Approach to Assessment in Support of CBME: The Rubric ........................................ 5 Development of the Surgical Procedure Feedback Rubric (SPR)............................................... 7 Study Purpose ............................................................................................................................ 11 Setting: How is the SPR Incorporated into Surgical Residency?.............................................. 13 Chapter Summary ...................................................................................................................... 14 Chapter 2 : Literature Review ....................................................................................................... 16 Introduction ............................................................................................................................... 16 Workplace-based Assessments (WBA) .................................................................................... 16 Are WBA Assessment Tools Valid and Reliable? .................................................................... 19 Are WBA Being Used As Intended?......................................................................................... 23 Barriers to Implementation of WBA ......................................................................................... 24 Assessment of Procedural Skills ............................................................................................... 25 Challenges with Rating-scales and Checklists: Justifying a Need for Better Assessment Tools ................................................................................................................................................... 32 Rubrics as Assessment Tools .................................................................................................... 34 Rubrics in Health Professional Education ................................................................................. 34 Theoretical Framework ............................................................................................................. 36 Chapter Summary ...................................................................................................................... 42 Chapter 3 : Methods ...................................................................................................................... 45 Introduction ............................................................................................................................... 45 Study Design and Participants................................................................................................... 46 Ethical Clearance....................................................................................................................... 47 vi Description of the Surgical Procedure Feedback Rubric .......................................................... 48 Data Collection .......................................................................................................................... 49 Data Analyses ............................................................................................................................ 50 Enabling Question 1: Is the SPR Being Used As Intended? ..................................................... 52 Enabling Question 2: What is the Factor Structure of the SPR? ............................................... 54 Enabling Question 3: How is a Resident’s SPR Score Influenced by the Resident’s