The Role for Virtual Patients in the Future of Medical Education

source: https://doi.org/10.7892/boris.80291 | downloaded: 27.9.2021 Copyright © by the Association of American Medical Colleges. Unauthorized reproduction ofthis article is prohibited. on student mental health, thecurrent educational model of impact Much shows evidence thenegative inmany instruction schools. medical of means remain learning and book primary knowledge management,of yet lectures teaching but methods alsotheteaching which demands notonly more efficient Medical knowledge isexpanding rapidly, question not only the quality of the question of notonly thequality concerns error aboutmedical one’s limits. own Further, legitimate andtheacknowledgment of expertise diagnostic thedevelopmentboth of knowledge indiverse contexts to foster foundational more on theapplication of education shouldfocus that medical on learning. havemotivation will apositive impact intrinsic mental healthandcapturing indicatesevidence thatimproving problem of diagnostic error diagnostic problem of practice, medical thecontinuedof clinical reasoning isacornerstone education system intheUnited States. challenges thecurrent facing medical Academic Medicine,Vol. XX,No. X/XXXXXX doi: 10.1097/ACM.0000000000001146 First publishedonline Acad Med. 2016;XX:00–00. [email protected]. 03756; telephone:(603)653-9888;e-mail: norman. Berman, OneMedicalCenterDrive,Lebanon, NH Correspondence shouldbeaddressed toNormanB. about theauthors. Please seetheendofthisarticleforinformation M as multimedia,screen-based interactive virtual patients(VPs),whichtheydefine possible. Theauthorspropose arole for challenges inwaysnotpreviously promise ofaddressing theseimportant Educational technologyoffers the to improve educationaloutcomes. system andtoimplementstrategies challenges facingthemedicaleducation the continuum—toaddress thecurrent working—across disciplinesandacross The medicaleducationcommunityis Abstract Soren Huwendiek,MD,MME,andMarc M.Triola, MD MD,PhD,MartinR.Fischer,Norman B.Berman,MD,StevenJ.Durning, MD,MME, Medical Education The RoleforVirtual Patients intheFuture of uch has been written aboutthe uch written hasbeen 3 Additionally, although 2 andindirect 4 suggests 5 call into call 1

challenges innew ways. addressingthe promise educational of into technology education offers of educational practice. incorporation The sciences, into andeducational technology psychology, education, thelearning fields such as cognitiveeducational and system must advances translate in practice, education medical amodern basic andclinical sciences into medical compelled to advances translate inthe Just healthcare asamodern system is these new educational strategies. theeffectivenessunderstanding of of remain difficult to solvewithoutbetter a however, thecurrent challenges will to improve educational outcomes identify andimplement strategies disciplines andacross thecontinuum—to education community isworking—across competence itsgraduates. medical The of toeducation needs better assess the U.S.contemporary medical system of health care system but the alsowhether application offoundationalknowledge VPscanalsoenhancetrainees’ learning. motivation andpromote mastery fromlearning VPscancapture intrinsic medical knowledge.Clinicallyoriented needed tohandletherapidgrowth in can promotethatis thedeeplearning activities interactive VP-basedlearning facing medicaleducation.Well-designed, well suitedtoaddressing thechallenges capabilities andbenefitsparticularly patient scenarios.TheybelieveVPsoffer and, then, proposingeducational specific their current roles education inmedical this by, first,what describing VPs are and education. medical facing We do will (VPs)—in addressing specific challenges patients enhanced education—virtual technology- roles forof form aspecific thisPerspectiveaim with isto suggest education. inmedical technology Our educational this to occur intheuseof solutions, andthere isthepotential for technologies offer more hopethanactual 7,8 Often, modern 6 ; colleagues incorporated into a VP. Huwendiek and technologies or features be thatmight are the insufficient in characterizing believe, however, thatthesedefinitions Talbot al et VP classificationdeveloped initially by other teaching methods that might other teaching thatmight methods patient scenarios; thisdefinition excludes multimedia,be screen-based interactive Perspective, we are considering VPs to addressed. this For of thepurposes andthecompetencytechnology being Kononowicz andcolleagues andfeedback. interactivity the useof the scenario hasbranch pointsand include, for example, or not whether factors for classifying VPs; factors including 19different derived typology and therapeutic decisions.”and therapeutic a physical exam, andmake diagnostic providers to ahistory, obtain conduct emulatelearners healthcare theroles of life clinical scenarios [through which] computer thatsimulates program real- improved systemofmedical education. and tomeasure progress toward an effectiveness ofeducationalinterventions both tobetterunderstandthe enable themedicaleducationcommunity for multi-institutionalresearch thatwill the useofVPscanserveasbasis based education.Thedatacreated by solution, VPscansupportcompetency- practice. Althoughnottheentire reasoning, thefoundationofmedical to promote thedevelopmentofclinical assessment” training,and medical education, or healthcare of scenarios for thepurpose computer real-life clinical simulation of defined patient” “virtual as “an interactive Medical educators andothers have What Are VPs? facilitate. educational outcomes we believe VPs can for theuseof VPsstrategies andthe 11 13 presented anempirically 9 to include theunderlying or of “aﬁc type speci 10 12 We We adapted a Perspective 1 Perspective

be considered VPs in Kononowicz example of the use of VPs in medical the need for better assessment). and colleagues’ classification such as education; specifically, it demonstrates Fortunately, advances in the science of simple case presentations, VP games, that a consortium project can lead cognition and learning give educators high-fidelity software simulation, to multi-institutional, national, and a better foundation for designing mannequin-based simulators, and virtual international use of a shared collection educational strategies to address some standardized patients. Our definition of VPs.23 Currently, VPs are in use in of these challenges. Here we discuss five also excludes other forms of computer- more than 130 medical schools in the educational strategies, each of which based education such as digital slide United States and Canada24 and in many is intended to address an important presentations and educational videos. European countries.25 challenge in medical education. We propose ways in which VPs can be For VPs to be effective, they must be used, incorporated into these strategies, and Current Role of VPs and there are strategies for integrating we suggest the educational outcomes that Medical educators use VPs to achieve VPs, which will promote their use. can be improved with these strategies. 26 widely varied instructional goals Berman and colleagues showed that The challenges, VP-based educational including not only teaching core VPs can be effectively integrated into strategies, examples of VP educational 14 15 knowledge, clinical reasoning, and clinical education by coordinating their activities, and expected educational 16 communication skills but also assessing use with other learning activities (e.g., outcomes are outlined in Table 1. learners’ progress.17 Lehmann and didactics, clinical experiences) and colleagues18 demonstrated the value of assessments and by making room in the Leverage interactive learning activities VPs, when blended with simulation, in course through the elimination of some to promote deep learning supporting the teaching of clinical skills, lectures and textbook assignments. Hege Chi,29 an education researcher, has and Fall and colleagues19 reported on VP and colleagues27 investigated a wide range proposed a conceptual framework and development based on comprehensive of scenarios for integrating VPs into provided empirical evidence supporting coverage of nationally accepted curricula. the medical curriculum and suggested Finally, Berman and colleagues20 reported a voluntary rather than an obligatory a hierarchy of learning activities. This on collaborative development of VPs approach. Huwendiek and colleagues28 framework suggests that instructional across multiple institutions; their work identified learner preferences, suggesting interventions that incorporate overtly shows that collaboration makes the task the importance of sequencing and aligning active, constructive, and/or interactive of covering broad curricular objectives VPs with other activities and assessments. activities will promote deeper learning, more manageable while also taking which emphasizes understanding and the application of knowledge over advantage of the ability to deliver VPs at Future Role of VPs scale, as suggested by Ellaway et al.8 memorization and recall. Having learners In our introduction above, we have track key findings presented in a VP is VP use in medical education is outlined a series of challenges facing an example of an active learning activity. substantial. In the United States, VP use medical education (e.g., rapidly Creating a summary statement from is most common in clinical clerkship expanding medical knowledge, the the history and physical exam findings education. By 2007 the Computer- ongoing occurrence of diagnostic and of a VP is an example of a constructive assisted Learning In Pediatrics Program other cognitive errors, the evolving activity. VPs offer the advantage of a (CLIPP), a VP program for pediatrics, understanding of learning preferences, standardized case presentation, making was used in more than 70 medical schools.19 A mixed-methods study of VP adoption based on the CLIPP program21 Table 1 demonstrated that the program’s ability Current Challenges of Medical Education and Virtual Patient (VP)-Based Strategiesa to fill gaps in students’ exposure to core clinical problems, the use of a Potential national curriculum, and the program’s VP educational VP educational educational development by clerkship directors Challenge strategy activity outcome were important factors leading to broad Expansion of medical Interactive learning VP assigned prior to seminar Deep learning adoption of the program. VP use is not knowledge activities (i.e., the flipped classroom) limited to pediatrics. Surveys of internal Negative impact of Capture student’s VPs recommended Mastery medical education on intrinsic motivation to by system, based on and lifelong medicine (IM) clerkship directors in student mental health learn assessment of performance learning 2009 and 2011 included questions on the Diagnostic error Focus on application VPs incorporating learner- Clinical uses and purposes of VPs in IM training of foundational constructed summary reasoning 22 programs in the United States. On the knowledge statements and prioritization expertise basis of these data, Lang and colleagues22 of differential diagnosis reported that meeting regulatory High prevalence of Competency-based VP-based assessment Reduced requirements was an important initial medical error education and aligned with VP cases for medical error motivator and that improving the quality assessment learning of learning became more important Difficulty identifying Analyze educational VPs incorporating learning Improved over time. The NetWoRM case-based improved outcomes from data analytics to earlier identify learning educational strategies and support learners at risk outcomes e-learning project in occupational medicine serves as another successful aThe authors define VP as a multimedia, screen-based interactive patient scenarios.

reliable and valid assessment of such be designed to capitalize on intrinsic Cook and Triola38 have proposed—on summary statements more feasible. motivation, mastery goal orientation, the basis of learning theory and a review of Further, Smith and colleagues30 recently and achievement emotions to improve the literature—VPs as an ideal instructional described a framework for evaluating a learning.35 method to prepare learners for clinical summary statement which can be applied reasoning in real patients. In a review of to real patients, as well as, importantly, To illustrate, novice learners using educational strategies to promote clinical VPs. Finally, responding to multiple- VPs can engage with content that reasoning, Bowen39 emphasized the choice questions, long menu questions, is authentic but designed to avoid following techniques: asking open-ended or other novel question types supported cognitive overload. Medical educators questions; providing single-sentence by VP technology, and receiving rich can design VP courses to match learning summaries of patient problems in individual feedback on those responses, or cognitive demands with student abstract terms; asking for discriminating is an example of an interactive learning capabilities—an ideal that is difficult features of a set of diagnostic hypotheses; activity. Advances in VP software could to achieve with real patients. Further, probing early for differential diagnoses; provide structured feedback on student educators can develop VPs at different prioritizing diagnoses; comparing and answers to free-text questions, an even levels of difficulty to complement contrasting diagnostic hypotheses based higher level of interactivity. Kopp and several levels of learner performance. An on real clinical data; demonstrating colleagues31 have shown that a learner- adaptive VP system could recommend typical presentations of different centered and VP-driven environment additional activities for learners based diagnostic hypotheses; and presenting incorporating active, constructive, and on their prior performance, resulting in the relative probabilities of different interactive learning activities, like the better matching of learner ability and diagnoses. Educators can design VPs ones mentioned above, can foster gains in demands. Learners can receive highly to incorporate these techniques. In a diagnostic knowledge. individualized and timely feedback via focus group study of medical students, 40 self-assessment dashboards that show Huwendiek and colleagues found Ellaway32 proposes that medical educators strengths and where improvements are that students perceive many of these can develop VP-based activities to achieve needed. Learners can also repeat VPs techniques as helpful for fostering clinical various specific objectives. In this context, or complete additional VPs to improve reasoning when learning with VPs. VPs are well suited to support emerging their performance. This approach can instructional interventions such as Assess learner competence to reduce give learners a choice of instructional the “flipped classroom.” In the flipped medical errors activities, allowing them to determine classroom model, learners might practice their own pace as they progress through Educators in both undergraduate and a particular concept on their own with the activity. graduate medical education are rapidly the VP and, then, attend a seminar or a adopting competency frameworks in problem-based learning or team-based Apply knowledge to support the an effort to move away from a purely learning session. VPs with embedded development of clinical reasoning time-based progression through training learning analytics can measure student expertise to, instead, a progression that is also engagement in these activities or predict informed by milestones of achieved Extensive research shows clearly that learning outcomes. mastery of a skill. Initiatives such as the clinical reasoning expertise cannot exist Physician Competency Reference Set41 Capture intrinsic motivation to foster without content knowledge,36 yet students and the Next Accreditation System42 have mastery and lifelong learning can have difficulty applying knowledge defined sets of common learning goals of foundational concepts when solving or Several educational theories suggest that graduating medical students and 33 explaining clinical problems.37 Norman’s the importance of motivation and the residents must meet at varying levels affective nature of learning.34 Intrinsic review of the educational psychology of training. VPs can play a key role as motivation, which arises from a desire literature presents a number of strategies medical education transitions to these to learn a topic because it is enjoyable to facilitate transfer of conceptual competency-based assessment systems.43 and/or interesting, is closely correlated knowledge to the clinical setting; with a mastery goal orientation. these strategies include embedding Although the medical education Research from a variety of disciplines the concept in a problem context and community knows that medical error is has shown that mastery goal orientation incorporating active problem solving at often the result of problems in the health (e.g., focusing on mastery of the the time of the initial learning.37 Further, care delivery system,5 the community subject) improves learning more than both mixed practice (through which also knows that it is better for learners a performance goal orientation (e.g., problems illustrating different concepts to make mistakes on virtual rather focusing on getting an “A” in a course are presented together) and distributed than real patients. Given their nature or clerkship).3 Further, achievement practice (in which experiences are as a screen-based intervention, VPs emotions (i.e., the feelings learners have dispersed over time) can result in large can be readily and flexibly integrated toward an academic activity) relate to and significant learning gains.37 VPs can into assessment activities in almost any goal orientation, affect motivation, and support each of these approaches. Basic setting, can be delivered at any time, and impact learning outcomes. Learning science and other foundational concepts have been used with learners at every activities that foster positive emotions, such as statistical analysis and population level and in multiple disciplines in health such as enjoyment and pride, rather than health can be incorporated into VPs, and professions education and practice.44–47 negative emotions, such as boredom conversely VPs can be integrated into This asynchronous capability is a natural or frustration, are preferable. VPs can basic science education. fit for assessing learners who may

vary in the timing of their progression a VP. Applied analytics could reflect the In Sum through training. The key feature exam student’s understanding of key clinical Case-based and patient-based learning format, which focuses specifically on findings and show how the student’s will always be at the heart of medical testing for decision making, has been actions affect the differential diagnosis. education, and technology will play an successfully incorporated into VP-based Another domain that educators could increasingly important role in education 48,49 assessment. Two advantages of VP- potentially assess using VP-generated data in the future. VPs are fundamentally a based competency assessment strategies is learning strategy. Analytic techniques patient-based means of learning enabled are that medical educators can align them may be able to identify a mastery learning by technology. VPs, if incorporated into with VP learning cases and that they can orientation, a marker of a deeper learning medical education more broadly, can be truly standardized from one learner to strategy. be an efficient and effective method to 50 the next. achieve the goal of creating a medical An area of potential future development education system that better educates the Conversely, VPs could dynamically is the integration of VP datasets with 53 next generation of providers to serve in a adapt to the performance level of an large educational and clinical databases. transformed health care system that better individual learner, highlight particular Combining such large amounts of suits the needs of patients and society. patient factors, and teach the effects data from two different arenas could of the determinants of health. Medical facilitate much more powerful multi- Well-designed and interactive VP-based students, for example, could learn institutional research. Such research and learning activities can be used to promote through a progressive pattern of other applications of analytics will be the deep learning necessary in an era increasingly challenging VPs, the timing more feasible when standardized VPs of rapid growth in medical knowledge. of which is generated from their actual are common or shared across multiple Clinically oriented learning from VPs patient encounters as recorded in clinical institutions and when resources can be can capture intrinsic motivation and experience logs and the electronic pooled to co-create high-quality teaching promote mastery learning. VPs can medical record (EMR). These linkages content and assessment instruments. help enhance the integration of the between VPs and experiences, when foundational sciences and clinical coupled with established standards for Challenges With VPs education to promote the development of authoring and exchanging VPs,51 could clinical reasoning skills. We believe that Despite all the promise of VPs, their enable a learning health care system in VPs have the potential to be an important impact on medical education to date which learners are taught and assessed component of medical education reform remains limited. In 2008, Berman and by VP cases—perhaps even cases that that incorporates critical interdisciplinary colleagues54 pointed out several barriers are machine generated directly from topics, interprofessional education, and to broad implementation of computer- deidentified comprehensive patient competency-based learning. assisted instruction programs in medical records in the EMR. This progressive education, and the same challenges exist model also suggests natural integrations We believe that VPs have the potential for VPs today. There is often a disconnect with other patient simulation modalities to make a significant impact on medical between available VP programs and such as mannequin-based simulators and education and that their use will likely the needs of the educators who might standardized patients. grow. The data generated by the use of VPs incorporate them into their teaching can facilitate multi-institutional research or courses. A lack of clarity among Analyze educational data to develop a that will enable the medical education educators and learners regarding the better understanding of educational community both to better understand the educational role of VPs leads to difficulties outcomes effectiveness of educational interventions in effectively integrating VPs in clinical and to measure progress. There is strong VPs, if implemented broadly, can create education. A widely accepted system for conceptual support for the role that VPs large amounts of educational data. These ongoing financial and technical support can play in the transformation of medical types of data, unique to the use of VPs, of VPs does not yet exist, and significant education, but much more can and should are relatively new to medical educators, efforts to support dissemination and be done to take advantage of the benefits and their use, though not yet well adoption of VPs8,9 have not yet resulted they offer. established, has great potential. Learning in widespread sharing or repurposing. analytics refers to the use of educational Finally, a lack of sufficient evidence for Acknowledgments: The authors wish to thank Elyse data to assess current performance and the features of VPs that create effective Payson for her support in organizing the authoring predict future performance. Medical learning remains a significant barrier for of this Perspective and to Sarah Berman for her educators can apply learning analytic those skeptical educators who may have grammatical editing and many helpful suggestions and educational-data-mining techniques seen educational fads come and go in toward improving the Perspective. across a large number of students and the past. Schifferdecker and colleagues21 Funding/Support: None reported. institutions to assess educational gains, identified factors leading to broad VP and they can use VP-generated learning use (e.g., the ability to fill gaps in clinical Other disclosures: None reported. data to predict success or failure in exposure, the use of a national curriculum Ethical approval: Reported as not applicable. specific domains.52 To illustrate, clinical and development by educators) which instructors could assess the development are consistent with existing models of Disclaimer: The ideas and views in this Perspective are those of the authors alone of a learner’s clinical reasoning and adoption of innovation, so we believe and do not necessarily represent those of the clinical decision making by applying that with attention to these challenges the U.S. Department of Defense or the federal analytics to actions the student makes in impact of VPs will grow. government of the United States.

N.B. Berman is professor, Department of of virtual patients. Med Teach. 2009;31: other activities in the undergraduate medical Pediatrics, Geisel School of Medicine at Dartmouth, 743–748. curriculum: A focus group study. Med Teach. Hanover, New Hampshire. 12 Kononowicz AA, Zary N, Edelbring S, 2013;35:920–929. Corral J, Hege I. Virtual patients—what are 29 Chi MT. Active–constructive–interactive: S.J. Durning is professor of medicine and we talking about? A framework to classify A conceptual framework for differentiating pathology, Uniformed Services University, Bethesda, the meanings of the term in healthcare learning activities. Top Cogn Sci. 2009;1:73– Maryland. education. BMC Med Educ. 2015;15:11. 105. M.R. Fischer is professor and chair for 13 Talbot TB, Sagae K, Bruce J, Rizzo AA. 30 Smith S, Kogan JR, Berman NB, Dell MS, medical education, Institut für Didaktik und Sorting out the virtual patient: How to Brock DM, Robins LS. The development Ausbildungsforschung in der Medizin, University exploit artifical intelligence, game technology and preliminary validation of a rubric to Hospital, LMU Munich, Germany. and sound education practices to create assess medical students’ written summary engaging role-playing simulations. Int J statements in virtual patient cases. Acad Med. S. Huwendiek is senior lecturer and head of Gaming Comput Mediat Simul. 2012;4:1–19. 2016;91:94–100. the department, Department of Assessment and 14 Sanders CL, Kleinert HL, Free T, et al. 31 Kopp V, Stark R, Fischer MR. Fostering Evaluation, Institute of Medical Education, University Caring for children with intellectual diagnostic knowledge through computer- of Bern, Bern, Switzerland. and developmental disabilities: Virtual supported, case-based worked examples: M.M. Triola is associate professor and associate patient instruction improves students’ Effects of erroneous examples and feedback. dean for educational informatics, New York knowledge and comfort level. J Pediatr Nurs. Med Educ. 2008;42:823–829. University School of Medicine, New York, New York. 2007;22:457–466. 32 Ellaway RH. Virtual patients as activities: 15 Garrett BM, Callear D. The value of Exploring the research implications of an intelligent multimedia simulation for activity theoretical stance. Perspect Med References teaching clinical decision-making skills. Educ. 2014;3:266–277. 1 Irby DM, Cooke M, O’Brien BC. Calls Nurse Educ Today. 2001;21:382–390. 33 Kusurkar RA, Ten Cate TJ, van Asperen M, for reform of medical education by the 16 Sijstermans R, Jaspers MW, Bloemendaal Croiset G. Motivation as an independent and Carnegie Foundation for the Advancement PM, Schoonderwaldt EM. Training a dependent variable in medical education: of Teaching: 1910 and 2010. Acad Med. inter-physician communication using the A review of the literature. Med Teach. 2010;85:220–227. dynamic patient simulator. Int J Med Inform. 2011;33:e242–e262. 2 Dyrbye LN, Thomas MR, Shanafelt TD. 2007;76:336–343. 34 Artino AR Jr, Holmboe ES, Durning SJ. Systematic review of depression, anxiety, and 17 Round J, Conradi E, Poulton T. Improving Can achievement emotions be used to other indicators of psychological distress assessment with virtual patients. Med Teach. better understand motivation, learning, and among U.S. and Canadian medical students. 2009;31:759–763. performance in medical education? Med Acad Med. 2006;81:354–373. 18 Lehmann R, Bosse HM, Simon A, Nikendei Teach. 2012;34:240–244. 3 Artino AR Jr, Holmboe ES, Durning SJ. C, Huwendiek S. An innovative blended 35 Yavner SD, Pusic MV, Kalet AL, et al. Twelve Control-value theory: Using achievement learning approach using virtual patients as tips for improving the effectiveness of Web- emotions to improve understanding of preparation for skills laboratory training: based multimedia instruction for clinical motivation, learning, and performance in Perceptions of students and tutors. BMC learners. Med Teach. 2015;37:239–244. medical education: AMEE guide no. 64. Med Med Educ. 2013;13:23. 36 Eva KW. What every teacher needs to Teach. 2012;34:e148–e160. 19 Fall LH, Berman NB, Smith S, White CB, know about clinical reasoning. Med Educ. 4 Graber ML, Franklin N, Gordon R. Woodhead JC, Olson AL. Multi-institutional 2005;39:98–106. Diagnostic error in internal medicine. Arch development and utilization of a computer- 37 Norman G. Teaching basic science to optimize Intern Med. 2005;165:1493–1499. assisted learning program for the pediatrics transfer. Med Teach. 2009;31:807–811. 5 Kohn LT, Corrigan J, Donaldson MS. To Err clerkship: The CLIPP project. Acad Med. 38 Cook DA, Triola MM. Virtual patients: A Is Human: Building a Safer Health System. 2005;80:847–855. critical literature review and proposed next Washington, DC: National Academy Press; 20 Berman NB, Fall LH, Chessman AW, et al. steps. Med Educ. 2009;43:303–311. 2000. A collaborative model for developing and 39 Bowen JL. Educational strategies to promote 6 Cooke M, Irby DM, Sullivan W, Ludmerer maintaining virtual patients for medical clinical diagnostic reasoning. N Engl J Med. KM. American medical education 100 years education. Med Teach. 2011;33:319–324. 2006;355:2217–2225. after the Flexner report. N Engl J Med. 21 Schifferdecker KE, Berman NB, Fall LH, 40 Huwendiek S, Reichert F, Bosse HM, et al. 2006;355:1339–1344. Fischer MR. Adoption of computer-assisted Design principles for virtual patients: A focus 7 Sandars J, Haythornthwaite C. New horizons learning in medical education: The educators’ group study among students. Med Educ. for e-learning in medical education: perspective. Med Educ. 2012;46:1063–1073. 2009;43:580–588. Ecological and Web 2.0 perspectives. Med 22 Lang VJ, Kogan J, Berman N, Torre D. The 41 Association of American Medical Colleges. Teach. 2007;29:307–310. evolving role of online virtual patients Physician competency reference set. August 8 Ellaway R, Poulton T, Fors U, McGee in internal medicine clerkship education 2013. https://www.aamc.org/initiatives/cir/ JB, Albright S. Building a virtual patient nationally. Acad Med. 2013;88:1713–1718. about/348808/aboutpcrs.html. Accessed commons. Med Teach. 2008;30:170–174. 23 Kolb S, Wengenroth L, Hege I, et al; NetWoRM January 27, 2016. 9 Ellaway R, Candler C, Greene P, Smothers V. Group. Case based e-learning in occupational 42 Accreditation Council on Graduate Medical An Architectural Model for MedBiquitous medicine—A European approach. J Occup Education. Next accreditation system. https:// Virtual Patients. Baltimore, MD: Environ Med. 2009;51:647–653. www.acgme.org/acgmeweb/tabid/435/ MedBiquitous; 2006. http://groups.medbiq. 24 MedU. Homepage. 2013. www.med-u.org. ProgramandInstitutionalAccreditation/ org/medbiq/display/VPWG/MedBiquitous Accessed January 27, 2016. NextAccreditationSystem.aspx. Accessed +Virtual+Patient+Architecture. Accessed 25 Smothers V, Ellaway R, Balasubramaniam C. January 27, 2016. February 23, 2016. eViP: Sharing virtual patients across Europe. 43 Frank JR, Snell LS, Cate OT, et al. Competency- 10 AAMC Institute for Improving Medical AMIA Annu Symp Proc. November 6, based medical education: Theory to practice. Education. Effective use of educational 2008:1140. Med Teach. 2010;32:638–645. technology in medical education: 26 Berman N, Fall LH, Smith S, et al. Integration 44 Lowdermilk DL, Fishel AH. Computer Colloquium on educational technology: strategies for using virtual patients in clinical simulations as a measure of nursing students’ Recommendations and guidelines for clerkships. Acad Med. 2009;84:942–949. decision-making skills. J Nurs Educ. medical educators. Washington, DC: 27 Hege I, Ropp V, Adler M, et al. Experiences 1991;30:34–39. Association of American Medical Colleges; with different integration strategies of case- 45 Schittek Janda M, Mattheos N, Nattestad 2007. https://members.aamc.org/eweb/ based e-learning. Med Teach. 2007;29: A, et al. Simulation of patient encounters upload/Effective%20Use%20of%20 791–797. using a virtual patient in periodontology Educational.pdf. Accessed February 1, 2016. 28 Huwendiek S, Duncker C, Reichert F, et al. instruction of dental students: Design, 11 Huwendiek S, De leng BA, Zary N, Fischer Learner preferences regarding integrating, usability, and learning effect in history-taking MR, Ruiz JG, Ellaway R. Towards a typology sequencing and aligning virtual patients with skills. Eur J Dent Educ. 2004;8:111–119.

46 Gesundheit N, Brutlag P, Youngblood P, opportunities. Med Teach. 2005;27:450– 52 Ellaway RH, Pusic MV, Galbraith RM, Gunning WT, Zary N, Fors U. The use of 455. Cameron T. Developing the role of big virtual patients to assess the clinical skills 49 Bronander KA, Lang VJ, Nixon LJ, et al. data and analytics in health professional and reasoning of medical students: Initial How we developed and piloted an electronic education. Med Teach. 2014;36:216–222. insights on student acceptance. Med Teach. key features examination for the internal 53 Cook DA, Andriole DA, Durning SJ, Roberts 2009;31:739–742. medicine clerkship based on a US national NK, Triola MM. Longitudinal research 47 Waldmann UM, Gulich MS, Zeitler curriculum. Med Teach. 2015;37:807–812. databases in medical education: Facilitating HP. Virtual patients for assessing 50 Stevens A, Hernandez J, Johnsen K, et al. the study of educational outcomes over medical students—Important aspects The use of virtual patients to teach medical when considering the introduction of students history taking and communication time and across institutions. Acad Med. a new assessment format. Med Teach. skills. Am J Surg. 2006;191:806–811. 2010;85:1340–1346. 2008;30:17–24. 51 Triola MM, Campion N, Mcgee JB, et al. 54 Berman NB, Fall LH, Maloney CG, Levine 48 Fischer MR, Kopp V, Holzer M, Ruderich An XML standard for virtual patients: DA. Computer-assisted instruction in clinical F, Jünger J. A modified electronic key Exchanging case-based simulations in education: A roadmap to increasing CAI feature examination for undergraduate medical education. AMIA Ann Symp Proc. implementation. Adv Health Sci Educ Theory medical students: Validation threats and October 11, 2007:741–745. Pract. 2008;13:373–383.