Appendix 1: VALIDATED EDUCATIONAL ADJUNCTS

Determination of a Remediation Plan In order to design an appropriate remediation plan to assess deficiencies noted within the ACGME competencies, the program director and resident must first meet to define a plan for improvement. They should establish a time-line with specific goals, determine a method of evaluation for meeting these goals, schedule follow-up meetings to assess progress, and define consequences if the objectives are not met.

a. Shapiro J, Prislin MD, Larsen KM, Lenahan PM. Working with the resident in difficulty. Family Medicine. 1987 Sep-Oct; 19(5): 368-375. b. Steinert Y and Levitt C. Working with the “problem” resident: Guidelines for definition and intervention. Fam Med. 1993 Nov-Dec; 25(10): 627-632. c. Yao DC and Wright SM. The challenge of problem residents. J Gen Intern Med. 2001 Jul; 16(7): 486-492. d. Yao DC and Wright SM. A national survey of internal medicine residency program directors regarding problem residents. JAMA. 2000 Sep; 284(9):1099–104.

Designation of a Mentor Webster’s dictionary defines a mentor as a “trusted counselor or guide”. A systemic review of articles related to mentoring in academic medicine by Sambunjak et al revealed that mentorship has a positive impact on a resident’s personal development, career guidance, career choice, and research productivity. Yao further advocates that the positive impact of mentors can also be utilized to positively influence the personal and clinical development of a problem resident.

a. Sambunjak D, Straus SE, Marusic A. Mentoring in academic medicine: A systematic review. JAMA. 2006 Sep; 296(9): 1103-1115. b. Yao DC and Wright SM. The challenge of problem residents. J Gen Intern Med. 2001 Jul; 16(7): 486-492. c. Yao DC and Wright SM. A national survey of internal medicine residency program directors regarding problem residents. JAMA. 2000 Sep; 284(9): 1099-1104.

Simulation Medical simulation is the use of a series of devices to emulate a real patient care situation for the purpose of education and training. While simulation has become a mainstay of emergency medicine resident education, the setting also provides an ideal forum for the remediation of problem resident behaviors as well as allows for the opportunity for improvement in procedural skills.

a. Barsuk JH, McGaghie WC, Cohen ER, O'Leary KJ, Wayne DB. Simulation-based mastery learning reduces complications during central venous catheter insertion in a medical intensive care unit. Crit Care Med. 2009 Oct; 37(10):2697-701. b. Issenberg SB, McGaghie WC, Hart IR, et al. Simulation technology for health care 1 professional skills training and assessment. JAMA. 1999 Sep; 282(9): 861-6. c. Okuda Y and Quinone J. The use of simulation in the education of emergency care providers for cardiac emergencies. Int J Emerg Med. 2008 Jun; 1(2): 73-77. d. Vozenilek J, Huff JS, Reznek M, Gordon JA. See one, do one, teach one: advanced technology in medical education. Acad Emerg Med. 2004 Nov; 11(11): 1149-54. Core Content Review Didactic curriculums are developed to convey information and the RRC for emergency medicine requires that residents participate in an average of at least five hours per week of educational time. The problem resident may benefit from additional core content review, such as additional reading assignments targeting specific areas of knowledge deficiencies that are then discussed with faculty members or a chief resident.

a. Jones RG. The lecture as a teaching method in modern nurse education. Nurse Education Today. 1990 Aug; 10(4): 290-293. b. Mallin M, Schlein S, Doctor S, Stroud S, Dawson M, Fix M. A Survey of the Current Utilization of Asynchronous Education Among Emergency Medicine Residents in the United States. Acad Med: 2014 Feb online. c. Menkes DL and Reed M. Structured didactic teaching sessions improve medical student neurology clerkship test scores: a pilot study. Open Neurol J. 2008; 2:8-11. d. Ogrinc G, Headrick L, Mutha S et al. A framework for teaching medical students and residents about practice-based learning and improvement, synthesized from a literature review. Academic Medicine. 2003 Jul; 78(7): 748-756. e. Torbeck L and Canal DF. Remediation practices for surgical residents. The American Journal of Surgery. 2009 Mar; 197(3): 397-402.

Web-based Education The web-based educational format is computer-assisted instruction that may encompass multimedia capabilities. The most notable advantage is ubiquitous access for asynchronous learning. Furthermore, material can more readily be revised and updated with current information. Reviews of the literature have shown that web-based learning is as effective as traditional teaching for knowledge acquisition and possibly even for some procedures. Of note, recent publications demonstrate that EM residents believe podcasts, such as those found on itunes may be the most beneficial use of their education time, thereby providing an important educational adjunct for resident remediation.

a. Chenkin J, Lee S, Huynh T, Bandiera G. Procedures can be learned on the Web: a randomized study of ultrasound-guided vascular access training. Acad Emerg Med 2008;15:949-54. b. Chumley-Jones HS, Dobbie A, Alford CL. Web-based learning: sound educational method or hype? A review of the evaluation literature. Academic medicine : journal of the Association of American Medical Colleges 2002;77:S86-93. c. Vozenilek J, Huff JS, Reznek M, Gordon JA. See one, do one, teach one: advanced technology in medical education. Acad Emerg Med 2004;11:1149-54. d. Burnette K, Ramundo M, Stevenson M, Beeson MS. Evaluation of a web-based asynchronous pediatric emergency medicine learning tool for residents and medical students. Acad Emerg Med 2009;16 Suppl 2:S46-50. e. Mallin M, Schlein S, Doctor S, Stroud S, Dawson M, Fix M. A Survey of the Current Utilization of Asynchronous Education Among Emergency Medicine Residents in the United States. Academic

2 medicine : journal of the Association of American Medical Colleges 2014.

Lecture Creation and Delivery A lecture is a method of communication that is predominantly information-oriented. The creation and delivery of lectures aids in resident remediation in several manners. First and foremost, creation of a didactic conference forces the lecturer to learn and review information in a novel fashion. In addition, the delivery of lectures allows for the resident to improve his or her presentation skills.

a. Domizio P. Giving a good lecture. Diagnostic Histopathology. 2008 Jun; 14(6): 284-288. b. Exley K and Dennick R. Giving a lecture: from presenting to teaching. Teaching Theology and Religion. 2011 Jan; 14(1): 85-86. c. Torbeck L and Canal DF. Remediation practices for surgical residents. The American Journal of Surgery. 2009 Mar; 197(3): 397-402.

Completion of Written Boards-style Test Questions Multiple choice questions are extensively used for knowledge assessment in medicine and other professions. It has been demonstrated that these tests of knowledge have significant relationships to later markers of quality clinical care. Furthermore, there is a correlation between the EM Inservice Training Examination and ABEM qualifying examination scores. A resident undergoing medical knowledge remediation may therefore greatly benefit from using written boards-style test questions as a measure to augment medical knowledge.

a. Frederick RC, Hafner JW, Schaefer TJ, Aldag JC. Outcome measures for emergency medicine residency graduates: do measures of academic and clinical performance during residency training correlate with American Board of Emergency Medicine test performance? Acad Emerg Med 2011;18 Suppl 2:S59-64. b. Goyal N, Aldeen A, Leone K, Ilgen JS, Branzetti J, Kessler C. Assessing medical knowledge of emergency medicine residents. Acad Emerg Med. 2012; 19:1360-1365. c. Lurie SJ, Mooney CJ, Lyness JM. Measurement of the general competencies of the accreditation council for graduate medical education: a systematic review. Acad Med. 2009; 84(3):301-9.

Assessment of Evidence-Based Literature The practice of evidence-based medicine is defined as “the conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients”. The ACGME evaluates evidence-based medicine principles through the practice-based learning core competency. Resources, such as UpToDate and online medical journals, are now readily accessible and stress the importance of examination of evidence from clinical research as the basis of medical decision making. The influence of evidence-based medicine on clinical practice and medical education continues to grow.

a. Carpenter CR, Kane BG, Carter M, Lucas R, Wilbur LG, Graffeo CS. Incorporating evidence -based medicine into resident education: a CORD survey of faculty and resident expectations. Acad Emerg Med 2010 Oct;17 Suppl 2:S54-61. 3 b. Friedman S, Sayers B, Lazio M, Friedman S, and Gisondi MA. Curriculum design of a case -based knowledge translation shift for emergency medicine residents. Acad Emerg Med. 2010 Oct; 17 Suppl 2: S42-8. c. Green ML and Ellis PJ. Impact of an evidence-based medicine curriculum based on adult learning theory. J Gen Intern Med. 1997 Dec; 12(12): 742-50. d. Guyatt G, Cairns J, Churchill D, et al. Evidence-based medicine: A new approach to teaching the practice of medicine. JAMA. 1992 Nov; 268(17): 2420-2425. e. Haynes RB, Devereaux PJ, Guyatt GH. Physicians' and patients' choices in evidence based practice. BMJ. 2002 Jun; 324(7350):1350. f. Kazdin AE. Evidence-based treatment and practice: New opportunities to bridge clinical research and practice, enhance the knowledge base, and improve patient care. American Psychologist. 2008 Apr; 63(3): 146-159. g. Rosenberg W and Donald A. Evidence based medicine: an approach to clinical problem -solving. BMJ. 1995 Apr; 310:1122-1126

Participation in Case-based Discussions Problem-based learning is an instructional technique that focuses on case discussions in a collaborative atmosphere. During these sessions, the chief resident or faculty facilitator can identify deficiencies in a resident’s knowledge base, as well as respond to the learner’s specific needs through discussion of pre-assigned readings, review of clinical problems, and analysis of clinical reasoning. Through the use of structured case scenarios, similar to the qualifying oral boards examination, problem-based learning allows for improved understanding and retention of information which may then be utilized in subsequent clinical situations.

a. Bounds R, Bush C, Aghera A, Rodriguez N, Stansfield RB, Santen SA. Emergency medicine residents' self-assessments play a critical role when receiving feedback. Acad Emerg Med. 2013 Oct; 20:1055-61. b. Frederick RC, Hafner JW, Schaefer TJ, Aldag JC. Outcome measures for emergency medicine residency graduates: do measures of academic and clinical performance during residency training correlate with American Board of Emergency Medicine test performance? Acad Emerg Med. 2011 Oct;18 Suppl 2:S59-64. c. Goodman CJ, Lindsey JI, Whigham CJ, and Robinson A. The problem resident: the perspective of chief residents. Academic Radiology. 2000 Jun; 7(6): 448-450. d. Negrini B, Zeitz H, Cohen W, and Bardwell S. Problem based learning: making it work for resident education. Workshop e. Steinert Y and Levitt C. Working with the “problem” resident: Guidelines for definition and intervention. Fam Med. 1993 Nov-Dec; 25(10): 627-632. f. Thistlethwaite JE, Davies D, Ekeocha S, et al. The effectiveness of case-based learning in health professional education. A BEME systematic review: BEME Guide No. 23. Medical teacher 2012; 34:e421-44.

Direct Observation of Clinical Encounters Kilminster et al define supervision as “the provision of guidance and feedback on matters of personal, professional, and educational development in the context of a trainee’s experience of providing safe and appropriate clinical care”. Emergency medicine educators recommend direct observation as the optimal evaluation tool for patient care, systems-based practice, interpersonal and communication skills, and professionalism (Chisolm 2004). Direct observation of a problem resident during components of an ED encounter, including history, physical exam, 4 interpersonal communications, discharge communication, and informed consent discussions positively affects resident development and allows for essential constructive feedback in areas of deficiency.

a. Chisolm CD, Whenmouth LF, Daly EA et al. An evaluation of emergency medicine resident interaction time with faculty in different teaching venues. Acad Emerg Med. 2004 Feb; 11(2):149-55. b. Kilminster S, Cotrell D, Grant J, and Jolly B. AMEE Guide No. 27: Effective educational and clinical supervision. Medical Teacher. 2007 Jan; 29(1): 2-19. c. Manthey D, Ander D, and Khandelwal S. Direct observation of competence. Presentation for 2010 SAEM meeting. June 6, 2010. d. Shapiro J, Prislin MD, Larsen KM, Lenahan PM. Working with the Resident in Difficulty. Family Medicine. 1987 Sep-Oct; 19(5): 368-375. e. Steinert Y and Levitt C. Working with the “problem” resident: Guidelines for definition and intervention. Fam Med. 1993 Nov-Dec; 25(10): 627-632. f. Yao DC and Wright SM. A national survey of internal medicine residency program directors regarding problem residents. JAMA. 2000 Sep; 284 (9):1099–104.

Bedside Teaching Bedside teaching is a type of medical education that occurs in real-time on a clinical shift, namely using a specific patient interaction to allow for a transfer of information. This tool may encompass a variety of medical skills including history taking, physical examination, and procedures. It may take on a demonstrative or observational form and allows for immediate feedback and reflection.

a. Aldeen AZ and Gisondi MA. Bedside teaching in the emergency department. Acad Emerg Med. 2006 Aug; 13(8): 860-66. b. Cykulka RK et al. Evaluation of resident performance and intensive bedside teaching during direct observation. Acad Emerg Med. 1996 Apr; 3(4): 345-51. c. Gisondi M. Improving ED bedside teaching and resident evaluation. Lecture May 21, 2003. d. Hedges JR. Pearls for teaching of procedural skills at the bedside. Acad Emerg Med. 1994 Jul-Aug; 1(4): 401-404. e. Janicik RW and Fletcher KE. Teaching at the bedside: a new model. Med Teach. 2003 Mar; 25(2): 127-130. f. Ramani S. Twelve tips to improve bedside teaching. Med Teach. 2003; 25(2): 112-115. g. Shayne P et al. Protected clinical teaching time and a bedside clinical evaluation instrument in an emergency medicine training program. Acad Emerg Med. 2002 Nov; 9(11): 1342-1349.

Creation of Study Materials Similar to the creation of didactic lectures, the creation of study materials also increases a resident’s knowledge by obligatory review of educational materials. Many residency programs utilize online sites for the asynchronous component of resident education. Development and posting of study materials not only aids in the remediation of the problem resident, but has a secondary benefit of improving the education of his or her residency colleagues.

a. Hauer KE, Ciccone A, Henzel TR et al. Remediation of the deficiencies of physicians across

5 the continuum from medical school to practice: a thematic review of the literature. Academic Medicine. 2009 Dec; 84(12): 1822-1832. b. Shin JH, Haynes RB, and Johnston ME. Effect of problem-based, self-directed undergraduate education on life-long learning. Academic Medicine. 2006 Mar; 81(3): 207-212. c. Torbeck L and Canal DF. Remediation practices for surgical residents. The American Journal of Surgery. 2009 Mar; 197(3): 397-402.

Role Modeling A role model is a person who serves as an example and whose behavior is emulated by others. Multiple studies have established that role modeling is an effective teaching method within the realm of medical education. Wright et al assert that the ideal role model possesses admirable personal characteristics, exceptional clinical skills, and superb teaching abilities and that these positive attributes represent modifiable behavior and acquirable skills. Therefore, a problem resident can learn strategies for improvement from direct observation of a physician modeling appropriate behavior.

a. Kenny NP, Mann KV, MacLeod H. Role modeling in physicians’ professional formation: Reconsidering an essential but untapped educational strategy. Acad Med. 2003 Dec; 78(12): 1203-1210. b. Wright SM and Carrese JA. Excellence in role modeling: insight and perspectives from the pros. CMAJ. 2002 Sep; 167(6): 638-43. c. Wright SM, Kern DE, Kolodnerk K et al. Attributes of excellent attending-physician role models. N Eng J Med. 1998 Dec; 339(27): 1986-93. d. Yao DC and Wright SM. The challenge of problem residents. J Gen Intern Med. 2001 Jul; 16(7): 486-492.

Receiving Clinical Teaching During Shift Effective clinical teaching during emergency department shifts may be challenging given significant patient volumes, high acuity, and the pressure to practice in an efficient manner. However, teaching during shifts is essential to clinical education and provides a forum for remediation of the problem resident. Clinical teaching models such as the five-step “microskills model” and “the one minute clinical preceptor” have all been developed to aid educators in improving of their teaching skills such that they can positively impact the development of resident education.

a. Houghland JE and Druck J. Effective clinical teaching by residents in emergency medicine. Annals of Emergency Medicine. 2010 May; 55(5): 434-439. b. Neher JO, Gordon KC, Meyere B, and Stevens N. A five-step “microskills” model of clinical teaching. J Am Board Fam Prac. 1992 Jul-Aug; 5(4): 419-424. c. Wald DA. Teaching techniques in the clinical setting: the emergency medicine perspective. Acad Emerg Med. 2004 Oct; 11(10): 1028.

Utilization of Pocket Reference Material Clinical information-retrieval technology, allowing healthcare professionals the ability to access up to date medical information, has expanded to previously incomprehensible levels over the

6 past decade. When used appropriately, these readily accessible references can help promote safe, efficient, and effective patient care.

a. Beasley BW. Utility of palmtop computers in a residency program: a pilot study. South Med J. 2002 Feb; 95(2):207-11. b. Pluye P, Grad RM, Dunikowski LG, Stephenson R. Impact of clinical information-retrieval technology on physicians: A literature review of quantitative, qualitative and mixed methods studies. International Journal of Medical Informatics. 2005 Sep; 74(9): 745-768. c. Torre DM and Wright SM. Clinical and educational uses of handheld computers. South Med J. 2003 Oct; 96(10):996-9.

Utilization of Checklists The practice of emergency medicine requires the delivery of efficient and coordinated care in a high-pressure clinical environment. Checklists, or a written series of tasks that need to be completed, have been used in high-risk industries, such as aviation and nuclear power, for many years and are now gaining popularity for use within the medical field to aid in performance.

a. Areskoug P and Hammarlund P. Evaluation of checklists as a diagnostic tool in the Emergency Department: a pilot study. Lunds University. Lecture 2012. b. Arriaga AF, Bader AM, Wong JM, et al. Simulation-based trial of surgical-crisis checklists. N Engl J Med. 2013 Jan; 368: 246-253. c. Gorter S, Rethans JJ, Scherpbier A, van der Heijde D, Houben H, van der Vleuten C et al. Developing case-specific checklists for standardized-patient-based assessments in internal medicine: a review of the literature. Acad Med. 2000 Nov 75(11):1130–7. d. Hales B, Terblanch M, Fowler R, Sibbald W. Development of medical checklists for improved quality of patient care. International Journal for Quality in Healthcare. 2008 Dec; 20(1): 20- 30.

e. Hales, Brigette M., and Peter J. Pronovost. "The checklist—a tool for error management and performance improvement." Journal of critical care 21.3 (2006): 231-235.

f. Wolff AM, Taylor SA, McCabe JF. Using checklists and reminders in clinical pathways to improve hospital inpatient care. Med J Aust 2004 Oct; 181(8):428–31.

Special Resources: educational and behavior specialists, psychiatric support, stress management Given the stressful nature of emergency medicine residencies, with constantly changing schedules combined with the physical and emotional demands of the work, certain residents who require remediation may benefit from the utilization of specific adjunctive resources, such as meetings with education and/or behavior specialists, psychiatric support, and stress management workshops in order to address a problematic behavior.

7 a. Katz ED, Goyal DG, Char D, Coopersmith CM, Fried ED. A novel concept in residency education: case-based remediation. J Emerg Med. 2013 Feb; 44(2): 493-8. b. McCue JD, Sachs CL. A stress management workshop improves residents' coping skills. Arch Intern Med. 1991 Nov; 151(11):2273–7. c. Quill TE, Williamson PR. Healthy approaches to physician stress. Arch Intern Med. 1990 Sep; 150(9):1857–61. d. Shapiro J, Prislin MD, Larsen KM, Lenahan PM. Working with the resident in difficulty. Family Medicine. 1987 Sep-Oct; 19(5): 368-375. e. Steinert Y and Levitt C. Working with the “problem” resident: Guidelines for definition and intervention. Fam Med. 1993 25(10): 627-632. e. Yao DC and Wright SM. The challenge of problem residents. J Gen Intern Med. 2001 Jul; 16(7): 486-492.

Impairment Screening Substance abuse and impairment are societal problems that have specific occupation hazards for the emergency medicine resident. Previous studies have demonstrated that substance abuse is likely underreported in this population and is an important consideration in the remediation of the problem resident.

a. Aach RD, Girard DE, Humphrey H, et al. Alcohol and other substance abuse and impairment among physicians in residency training. Annals of Internal Medicine. 1992 Feb; 116(3): 245 -254. b. McNamara R and Margulies JL. Chemical dependency in emergency medicine residency programs: Perspective of the program directors. Annals of Emergency Medicine. 1994 May; 23(5): 1072-1076. c. Steffen PD and Dalley RH. Appropriate management of chemical dependency in emergency medicine residents. Annals of Emergency Medicine. 1992 May; 21(5): 559-564. d. Steinert Y and Levitt C. Working with the “problem” resident: Guidelines for definition and intervention. Fam Med. 1993 Nov-Dec; 25(10): 627-632.