Review Article The Use of Standardized Patient Assessments for Certification and Licensure Decisions John R. Boulet, PhD; Although standardized patients have been employed for formative assessment for over Sydney M. Smee, PhD; 40 years, their use in high-stakes medical licensure examinations has been a relatively Gerard F. Dillon, PhD; recent phenomenon. As part of the medical licensure process in the United States and Canada, the clinical skills of medical students, medical school graduates, and residents 07/05/2021 on BhDMf5ePHKav1zEoum1tQfN4a+kJLhEZgbsIHo4XMi0hCywCX1AWnYQp/IlQrHD3i3D0OdRyi7TvSFl4Cf3VC4/OAVpDDa8KKGKV0Ymy+78= by http://journals.lww.com/simulationinhealthcare from Downloaded John R. Gimpel, DO are evaluated in a simulated clinical environment. All of the evaluations attempt to provide the public with some assurance that the person who achieves a passing score Downloaded has the knowledge and/or requisite skills to provide safe and effective medical services. Although the various standardized patient-based licensure examinations from differ somewhat in terms of purpose, content, and scope, they share many common- http://journals.lww.com/simulationinhealthcare alities. More important, given the extensive research that was conducted to support these testing initiatives, combined with their success in promoting educational activities and in identifying individuals with clinical skills deficiencies, they provide a framework for validating new simulation modalities and extending simulation-based assessment into other areas. (Sim Healthcare 4:35–42, 2009) Key Words: Licensure, Certification, Simulation, Standardized patient, Simulated patient, by OSCE BhDMf5ePHKav1zEoum1tQfN4a+kJLhEZgbsIHo4XMi0hCywCX1AWnYQp/IlQrHD3i3D0OdRyi7TvSFl4Cf3VC4/OAVpDDa8KKGKV0Ymy+78= There are many types of simulations that are currently being tion activities.8–10 Individuals were trained to portray specific used to assess healthcare professionals.1–4 In both Canada patient conditions, allowing medical students to practice and the United States (US), many of these simulation modal- their clinical skills and receive immediate feedback concern- ities, including multiple choice questions, part-task trainers, ing strengths and weaknesses. In the 1980s, with an increased and computer-based case simulations, have been used as part emphasis on evaluating what medical trainees could do, as of the examination process used to certify and license physi- opposed to what they knew, various organizations started cians.1,5,6 These simulation-based examinations, which can research programs aimed at determining how assessments vary somewhat in terms of purpose and focus, all attempt to employing SPs could be structured to make valid skills-based provide the public with some assurance that the person who proficiency decisions. Over the next two decades, the end achieves a passing score has the knowledge and/or requisite result of these research activities was the implementation of a skills to provide safe and effective medical services, either number of high-stakes assessments all aimed at measuring independently or under supervision. Here, as with any sim- abilities in key clinical skills domains. Although these re- ulation-based assessment, the structure, content, fidelity, and search efforts required extensive resources, they were suc- difficulty of the modeled exercises, combined with the scores, cessful in identifying the specific conditions and structures will determine what inferences one can make about the indi- that are needed to produce defensible scores and decisions for vidual test taker. multistation, performance-based, simulation activities.11–17 From a simulation perspective, the use of standardized The introduction of SP-based certification and licensure patients (SPs) for certification and licensure decisions has examinations in medicine was a monumental achievement. on been a relatively recent phenomenon.7 Historically, SP-based 07/05/2021 Although other high-stakes simulation-based assessments assessments were implemented as part of formative evalua- have been developed and used in other professions, the logis- tical, economical, and psychometric challenges associated From the Foundation for Advancement of International Medical with national multistation clinical skills assessments were Education and Research (J.R.B.), Philadelphia, PA; Medical Council of staggering.18,19 Organizations that built these assessments all Canada (S.M.S.), Ottawa, ON, Canada; National Board of Medical Examiners (G.F.D.), Philadelphia, PA; and National Board of Osteopathic had to address concerns regarding test content (eg, types of Medical Examiners (J.R.G.), Conshohocken, PA. scenarios to model), test administration models (eg, fixed Reprints: John R. Boulet, PhD, Foundation for Advancement of versus temporary sites; number, timing and sequencing sta- International Medical Education and Research, 3624 Market Street, tions), measurement rubrics (eg, holistic or analytic), eligi- Philadelphia, PA 19104 (e-mail: [email protected]). The authors have indicated that they have no conflict of interest to bility requirements, scoring models (eg, compensatory or disclose. conjunctive), and the establishment of defensible standards, Copyright © 2009 Society for Simulation in Healthcare just to name a few. Nevertheless, even with these hurdles, and DOI: 10.1097/SIH.0b013e318182fc6c despite numerous objections concerning the need to measure Vol. 4, No. 1, Spring 2009 35 clinical skills as part of certification/licensure process,20 each who interview the same SP with the same presenting com- of these organizations was able to produce a high-quality plaint will receive, on questioning, the same patient history. simulation-based assessment that was appropriate for their The physical findings relevant to the case, either real or sim- particular needs. In doing so, many lessons were learned, the ulated, need to be stable and, for a given modeled scenario, most important being that simulation-based summative as- they must not vary from one SP to another. sessment of clinical skills was viable, even with large examinee populations, differing testing purposes, and varying exami- nation administration protocols. LARGE-SCALE SP EXAMINATIONS Medical Council of Canada Qualifying Examination Part II PURPOSE Since 1912, the MCC has been setting an examination that The purpose of this article was to describe and contrast the is a prerequisite for medical licensure in Canada; the Licenti- Clinical Skills Assessment (CSA) programs that are employed ate of the MCC is granted to those who successfully complete in Canada and the US as part of the certification and licensure it. In 1992, the MCC added the Qualifying Examination Part process for physicians. These assessments include the Medi- II (MCCQE Part II) to the assessment sequence. Initially the cal Council of Canada (MCC) Qualifying Examination Part MCCQE Part II was a 20-station Objective Structured Clini- 7,28 II (MCCQE Part II),21 the United States Medical Licensing cal Examination (OSCE). Although the use of OSCEs is Examination (USMLE) Step 2 Clinical Skills (USMLE Step now commonplace throughout the world, implementing a 2 CS),22 and the National Board of Osteopathic Medical national summative, performance-based, assessment based Examiners (NBOME) Comprehensive Osteopathic Medical on a series of SP encounters was, at the time, unprecedented. Licensing Examination Level 2-Performance Evaluation The impetus for implementing the MCCQE Part II came (COMLEX-USA Level 2-PE).23 To better understand the largely from the licensing authorities. In the late 1980s, be- USMLE Step 2 CS, a brief overview of the Educational Com- cause of the number and nature of related complaints that mission for Foreign Medical Graduates (ECFMG) CSA is also they received each year, members of these authorities began provided.24 The CSA was used to assess the clinical skills of calling for an assessment of clinical and communication international medical graduates (IMGs) before the introduc- skills. The existing paper-and-pencil test of medical knowl- tion of USMLE Step 2 CS. Following this overview, a brief edge and problem solving (MCC Qualifying Examination synthesis of the similarities and differences in the assessments Part I—MCCQE Part I) was not sufficient to address the and assessment programs is provided. With these distinc- emergent belief that candidates for medical licensure should be tions in mind, and knowing the success and scope of the assessed more broadly. individual testing programs, it is possible to envision where To qualify for the MCCQE Part II, candidates must have summative simulation-based assessment activities could be completed successfully 12 months of postgraduate clinical enhanced, applied in other areas, and used for the evaluation training and passed the MCCQE Part I, currently a computer- of nonphysician healthcare professionals. adaptive test of knowledge and clinical decision-making. The number of candidates who qualify for the MCCQE Part II ASSESSMENT OF CLINICAL SKILLS continues to grow. In 1992, 401 candidates took the exami- nation. In 2007, 3481 candidates completed this assessment, a In general terms, clinical skills refer to information gath- more than eightfold increase. ering and communication skills, applied during the patient As the measurement qualities of the MCCQE Part II be- encounter, that help to establish an accurate diagnosis and came better understood, the number of stations was reduced support high-quality