E O Educational Outcomes Service S Group G Evaluating Core Competencies

June 2003

FUTURE FOR GME ACCREDITATION The following section will identify some of the (Lynne Tomasa, PhD) various tools included in the ACGME toolbox.

The ACGME Outcomes Project places EVALUATING CORE COMPETENCIES emphasis on a program’s actual (Craig McClure, MD) accomplishments by asking the following set of questions: Suggestive list of methods: 1. Do the residents achieve the learning 1. Direct Observation or Shadowing objectives identified by the program? 2. Standardized Oral Examinations 2. What evidence or documentation can the 3. Objective Standardized Exams program provide that residents achieve 4. Simulations and Models these objectives? 5. Chart-Stimulated Recall 3. Using the data collected, how does the 6. Global Rating program demonstrate continuous 7. 360 Degree Evaluation improvement in its educational processes? 8. OSCE 9. Portfolios In order to answer these questions, programs 10. Record Review must first identify specific learning objectives related to each of the ACGME’s six general The ACGME has left responsibility for competencies. Programs must enlist faculty implementation and evaluation of core participation in the process. The learning competencies to individual programs to objectives must define in clear and concise accomplish. Once the characteristics that behavioral terms, what residents are expected define competency are known, the selection of to know and do. Sample objectives for Medical assessment methods can occur. An Knowledge include: introduction to some methods follows. 1. By the end of Year 1, residents will be able to independently do procedure X and Y. Direct Observation or Shadowing: Some 2. By the end of Year 2, residents will be able version of the faculty member directly to independently do procedure A and observing the behavior of the resident has long receive an attending evaluation of #5 been used in a variety of settings and with (always performs a specific varying formality and documentation. The Mini- behavior/competency) in 90 percent of their CEX of the American Board of Internal evaluations. Medicine reflects an effort to provide a systematic definition of competency Once the objectives are defined, programs implemented in a way that provides resident must use dependable and objective methods of feedback and documentation of that feedback. assessing residents’ attainment of the This method can be very expensive of faculty competencies identified in the learning members’ time and without the proper tool can objectives. The methods or tools selected must be very subjective, yet has the potential for a be valid and reliable. Once the appropriate tool rich immediate assessment and feedback of all is selected, faculty must understand what the of the six general competencies coupled with anchors mean and recognize the importance of the now mandated documentation of the effort. accurately filling out the forms. In addition, faculty must agree on how the data from the Standardized Oral Examinations (SOE): tool will be collected, analyzed and ultimately typically requires the resident to review a used for resident and program improvement. patient case vignette in the presence of a faculty member and orally present the thought surveys appropriate for the array of people in a process involved in collecting history and resident’s life: residents, faculty members, physical examination information, synthesizing nurses, patients, clerical staff members, etc. data into a differential diagnosis list and The 360 Degree is useful for measuring developing an appropriate diagnostic and communication/interpersonal and professional therapeutic care plan. This technique is more qualities. Challenges include developing the appropriate for measuring patient care, surveys, managing the mass of information, knowledge, interpersonal/ communication and balancing feedback to the resident with the skills, and systems based practice, than confidentiality needed for valid information. professionalism or practice-based learning and improvement. Use of faculty time runs high. OSCE: the Objective Structured Clinical Examination provides a direct measure of Objective Standardized Exam: such as the performance in the doctor-patient encounter. InTraining Examination, mini-quizzes in An OSCE is very useful to measure specific preparation for Board exams or computerized clinical skills and requires a significant modules combining both education and investment of time and money to develop and evaluation can be useful for assessing medical administer, making it most cost-effective when knowledge. a number of residents are examined in a single session. Simulations and Models: can be useful in evaluating procedural skills and medical Portfolios: is a set of materials collected by knowledge. While tending to be expensive and about a resident reflecting on clinical because of the materials used (cadavers, performance. One example is the resident file animal preparations, computer simulations, maintained by the Residency Office. The tool OSCE’s these are valuable when resources can be expanded with copies of videotapes, support their availability. reflective statements from the resident about progress in learning, an educational plan for Chart-Stimulated Recall (CSR): provides an what has been learned, is yet to be learned, opportunity for the resident to verbally review and how that will occur. Carefully established the processes involved in patient care, so may criteria are important if the portfolio is to be provide insight into patient care, medical used in comparisons between residents. Some knowledge, systems-based practice and programs find it a way to measure activities perhaps practice-based learning. M & M otherwise not evaluated; some find the time conferences and chart reviews are two settings committed excessive for the return. for this method. A trained examiner questions the resident about the reasoning behind the Record Review: involves trained staff using documented care in the progress note and standardized coding to abstract information chart. from the written record of patient care, useful for measuring clinical decision-making, Global Rating of Live or Recorded Action: documentation, follow-thru in management, includes the familiar standardized form appropriate use of resources, etc. Since the completed at the end of a block rotation. It note may involve the consensus of consultation requires the observer to judge a general with other physicians (e.g., inpatient resident competency as a category rather than team or attending physician), caution must be evaluating specific behaviors, skills, or tasks. used in attributing the diagnostic and An advantage is that the tool is readily therapeutic plans solely to the resident signing available and can address all six general the note. competencies. Problems include risk of great subjectivity, difficulty discriminating between References: levels of trainee and reliability even with the 1. http://www.acgme.org/outcome/project/ same faculty member. OutIntro_fnl1.htm 2. Swing, SR. Assessing the ACGME 360 Degree Evaluation: is a tool often seen in general competencies: general Human Resource and business settings. In considersations and assessment brief, the instrument involves developing methods. Acad Emerg Med. 2002; 9: 1278-1288