Sequencing Discovery Learning and Direct Instruction for Simulation-Based Technical Skills Training

S37

7–9 Impact of Assessment of Impact Advocates of direct Advocates = 0.27). 2 11,12 In these models, these models, In 10 which are valuable when when valuable are which 6 vast amount of literature attesting to vast amount of to attesting literature ofthe benefits instruction, direct particularly for faster and more accurate accurate particularly and more faster for optimizing However, performance. not always does performance immediate transfer ofguarantee future learning retention, or instruction counter that discovery that discovery instruction counter unnecessarylearning induces cognitive mental weak or in inaccurate results load, Conclusions Sequencing discovery learning before to improve appeared instruction direct transfer performance in simulation- based skills training. Implications for design are and curricular research future discussed. Results significantly did not differ The groups or retention on immediate posttest P < 0.33). The Do test (F[1,30] = 0.96, (N = 16) outperformed then See group on the (N = 16) the See then Do group transfer test; 2.99 versus 2.52 (F[1,28] = 10.14, P < 0.004, η considering the range ofconsidering similar but tasks that traineesnonidentical will in the workplace. experience eventually learning involves discovery contrast, By minimal guidance, trainees receiving the learning explore allowing to them experiment withtask and to different variations include a problem; to solutions inquiry-based learning and problem- based learning. students autonomously address the address autonomously students learning task and construct own their them having to as opposed meanings, Proponents an instructor. imposed by of learning argue that it discovery transfer ofenhances learning and learner learner- and is a more self-regulation centric approach. yet medical yet 5,6 educators have little evidence to draw on on draw little evidence to have educators the combine to how when determining has Optimizing combination their two. learning. optimize to the potential involves usually teaching Traditional or A teacher instruction:direct deconstructsinstructor the first the explains skill or oftopic interest, and demonstrates essential concepts, instruction Direct the application. time when students precedes usually with problems solve to attempt later and feedback. guidance the teacher’s of find this sequence teaching People pause to they that hardly so intuitive a is There its effectiveness. question combine direct instruction direct and learner- combine in an idiosyncratic likely SRL, driven at the curricular level, Further, fashion. include now most medical schools periods of without clear assigned SRL, integrate those to how for guidelines direct periods with involving sessions in educational Researchers instruction. that the best learningpsychology suggest arise when educators may outcomes thoughtfully and meaningfully integrate instruction,SRL and direct Method medical students In 2015, first-year two randomized into (N = 36) were to learn mattress horizontal groups had Do group suturing. The See then independent access to instructors before the Do then See group practice, whereas the task independently before explored accessing instructors. Participants were assessed at the transition between interventions, and as training ended. and transfer to a novel Skill retention, variation of the suturing task, were assessed after one week. Performance on a five-point global rating was scored scale by a blinded rater.

1 —but the —but 3,4 During a typical in session 3 and effective 2

ncouraging learners to direct their their direct ncouraging learners to

Abstract extent of learner involvement and ofextent learner involvement as whether as well and how control, integrate SRL withto instructor-led Indeed, understudied. remains sessions, review and meta- systematic a recent ofanalysis training simulation-based SRL compare that researchers showed often, interventions and instructor-led the two combine to how study few yet approaches. Academic Medicine, Vol. 93, No. 11 / November 2018 Supplement 93, No. 11 / November Academic Medicine, Vol. Acad Med. 2018;93:S37–S44. Acad Med. doi: 10.1097/ACM.0000000000002378 Copyright © 2018 by the Association of American Medical Colleges

Please see the end of this article for information about the authors. to Kulamakan should be addressed Correspondence General Kulasegaram, Wilson Centre–Toronto Hospital, University Health Network, 200 Elizabeth Ontario M5G 2C4; e-mail: St., 1ES-603, Toronto, Twitter: [email protected]; @mahanmeded. E Kulamakan Kulasegaram, PhD, Daniel Axelrod, MD, Charlotte Ringsted, PhD, MD, Charlotte Ringsted, Daniel Axelrod, Kulasegaram, PhD, Kulamakan and Ryan Brydges, PhD See One: Sequencing Discovery Sequencing Then See One: Do One for Instruction Learning and Direct Skills Training Technical Simulation-Based Discussant: Rachel Yudkowsky, MD, MHPE MD, Yudkowsky, Rachel Discussant: Be Determined To Facilitator: own learning has been an active area area own learning has been an active medical educators. for of investigation learners can “activating” agree that Most traditional Self- enhance education: learning (SRL) can be efficient, regulated any medical curriculum, educators may may educators medical curriculum, any diagnostic, Purpose skills, educators When teaching technical learneroften include a mix of self- instruction. and direct regulation sequencing of these Appropriate activities—such as allowing learners a period of discovery learning prior to been shown instruction—has direct transfer in other domains to improve the of learning. This study compared efficacy of learners trying a novel formal simulated suturing task before instruction (Do then See) versus the more typical sequence of formal instruction, followed by practice (See then Do) on and transfer. skill acquisition, retention, Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited. American Medical Association of Copyright © by the

Downloaded from https://journals.lww.com/academicmedicine by BhDMf5ePHKav1zEoum1tQfN4a+kJLhEZgbsIHo4XMi0hCywCX1AWnYQp/IlQrHD3IOi9dz/r5OE4P0ATcru8NtY2rObcGw3vezXYu/yBmdbaI0iiOhhr4g== on 12/04/2018 Downloaded from https://journals.lww.com/academicmedicine by BhDMf5ePHKav1zEoum1tQfN4a+kJLhEZgbsIHo4XMi0hCywCX1AWnYQp/IlQrHD3IOi9dz/r5OE4P0ATcru8NtY2rObcGw3vezXYu/yBmdbaI0iiOhhr4g== on 12/04/2018 Impact of Assessment

representations of content, and poses suturing skills in undergraduate students. Procedures 13 undue stress on trainees. Indeed, head- We hypothesized that the sequence of After consent and randomization, to-head comparisons of direct instruction discovery, followed by direct instruction, participants completed a short survey versus pure discovery learning have would benefit students’ transfer of skills, estimating previous time spent on (a) typically found that pure discovery is a but not their retention or immediate 13 any suturing, (b) horizontal mattress weaker form of instruction. posttest performance. We also examined suturing, and (c) knot tying. After how the two sequences impacted this, the research assistant introduced Contemporary research in education has students’ perceived self-efficacy and them to the relevant procedures for moved beyond an “either–or” approach, competence. their respective groups. All participants focusing instead on the optimal completed an initial session in their sequencing of direct instruction and assigned group, working in small groups discovery learning.6 The specific sequence Method of six to eight. That session included of discovery learning prior to direct Our experimental study compared two two performance assessments. After a instruction, known as guided discovery, forms of training for suturing skills at one-week delay, participants returned to has been studied, for example, in statistics the University of Toronto Surgical Skills complete a retention test and a transfer instruction, where students solved Centre at Mt. Sinai Hospital. Participants test. Performances on all tests were scored problems on the concept of standard were randomized to learn suturing skills by a single rater blinded to participants’ deviation.14 Crucially, the problems on a skin-pad simulator through (i) group assignment. required students to examine some data direct instruction followed by discovery and develop their own approaches to learning (See then Do), or (ii) discovery Interventions and design measuring consistency. After this initial learning followed by direct instruction In the See then Do group, participants discovery phase, instructors then formally (Do then See). explained standard deviation and its first learned with instructors, who applications. Alternatively, a comparison Population demonstrated and explained the steps in group attended a traditional lecture completing the suturing task twice. The We recruited first- and second-year instructors also allowed participants to followed by SRL of practice problems. undergraduate MD students (N = 32) The primary outcome included students’ practice independently, and provided at the University of Toronto, who were individualized feedback as requested in a knowledge of standard deviation, naïve to the suturing tasks and thus, we and also how well they applied their semistructured environment. Participants expected, could be influenced by both were encouraged to ask questions knowledge to novel, related transfer forms of instruction. We based our 15,16 and solicit feedback. We aimed for a tasks. On transfer, the guided sample size on an expected moderate discovery group far outperformed the 1:3 instructor ratio during this phase. effect size, and in keeping with previous Instructors were junior- and senior- traditional training group. Similar studies 1 research using our primary outcome. level residents in the orthopedic surgery have been conducted in concept learning, Participants completed two study with consistent findings for transfer of postgraduate residency program, all with sessions one week apart in which session learning17–19; notably, these studies often basic technical skill teaching experience. 1 involved delivery of the interventions show little benefit of guided discovery on In an effort to standardize instruction, and immediate posttest and session 2 retention. Focusing on assessing retention we gave instructors a teaching script (see involving retention and transfer testing. performance may obscure the benefits of Appendix 1). Although this limited the Two cycles of the study took place, one in guided discovery, which tests of transfer ability of instructors to initially adapt April and one in November 2015. may make visible.20 Even some dedicated their instruction to their participants, it provided a measure of control for proponents of direct instruction Ethics acknowledge the utility of guided our comparison with the Do then See discovery8 when training for transfer. We received ethics approval for group. After 30 minutes of instruction, recruitment and data collection from the participants completed a self-efficacy Most studies of guided discovery14–19 University of Toronto Health Sciences rating and performed a horizontal have focused on concept learning, Research Ethics Board. mattress suture (midpoint test), without with research only now emerging in any instructor feedback. Participants Materials and tasks skills training in medical education.20 then had 30 minutes to practice on their Thoughtful incorporation of guided For both practice and all testing, own in small groups, without instructors. discovery into training modalities such as participants used custom-made The full session ended with participants simulation-based training may optimize silicone simulated skin pads and completing the self-efficacy rating and transfer to the clinical setting. Guided a standardized suturing kit (Faux performing a horizontal mattress suture discovery may have other benefits, such Medical Corporation). We assigned the (immediate posttest), without any as using limited instructor time with horizontal mattress suturing task, using instructor feedback. learners more efficiently. It may also help Ethilon nylon monofilament suture, learners calibrate their self-efficacy and because we expected that it would be The Do then See group had the sequence actual skill level.1 In the present study, challenging for participants but could be of training inverted, with a discovery we experimentally tested the efficacy of taught and assessed relatively efficiently. phase prior to contact with instructors. two sequences of discovery learning and For the transfer test, we placed the For the first 30 minutes, participants direct instruction on the acquisition, same skin pads in an abdominal cavity received the skin pads, suturing material retention, and transfer of simulated simulator. and equipment, and an example diagram

S38 Academic Medicine, Vol. 93, No. 11 / November 2018 Supplement Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited. Impact of Assessment

of a completed horizontal mattress suture. The RA encouraged participants to use the Table 1 diagram as a guide as they attempted the Experience of Participants With Suturing and Knot Tying, Shown in Hours horizontal mattress suture. Participants did not receive any guidance, feedback, or Do then See, See then Do, further instructions. We asked participants median (SD) median (SD) to complete the task alone, with minimal Experience with any suture 2 (1.01) 2 (0.99) interactions with each other. After 30 Experience with horizontal sutures 1 (0.78) 1 (0.39) minutes, participants completed the Experience with knot tying 1 (1.18) 1 (0.79) midpoint test of suturing performance and a rating of their self-efficacy. Next, the participants worked directly with (GRS) from the Objective Structured during the delay period; thus, we analyzed instructors, who demonstrated, and Assessment of Technical Skills system, for 16 participants per group. The majority provided feedback in a 1:3 instructor- which there is considerable and strong of participants had less than one hour of to-participant ratio for 30 minutes. validity evidence for assessing performance experience with each task (see Table 1). The session ended when participants of novice learners in a research setting.22 Participants’ self-reported number of completed the immediate posttest of their The GRS consists of six items evaluating hours suturing correlated with their self-efficacy and suturing performance. elements of the procedural skill, and one transfer performance at r = 0.49 (P < .005) overall process item; all items are on a five- and, thus, was included as a covariate in After a one-week delay, participants point Likert scale, with verbal anchors at the analysis of transfer test performance. completed a retention test, consisting 1, 3, and 5 (see Appendix 2), and a higher of two horizontal mattress sutures score indicates superior performance. Performance on the transfer task is on the same skin pad on a table top. The GRS score used in our analyses is an presented in Figure 1. The ANCOVA Immediately following that test, they also average across all items. showed that the Do then See group completed a transfer test consisting of outperformed the See then Do group two horizontal mattress sutures on the Analysis on the transfer task at 2.99 (SE = 0.10; skin pad placed in an abdominal cavity Analysis of transfer performance was 95% CI: 2.78–3.20) compared with 2.52 simulator, using elongated instruments through analysis of covariance (ANCOVA) (SE = 0.10; 95% CI: 2.31–2.73) (F[1,28] and the same suturing materials. All with intervention as the between-subjects = 10.14, P < .004, η2 = 0.27). There participants completed self-efficacy factor and retention performance as was no significant interaction between questionnaires prior to both retention the covariate. We included participants’ participants’ self-reported number and transfer tests. retention score as a covariate to gauge the of hours suturing and their group effect of individual variation in suturing assignment (F[1,28] = 1.12, P < .9). Outcomes skill, and because other studies using Our primary outcome was performance transfer as the primary outcome often do The repeated-measures ANOVA on the delayed “near-transfer” test. We not include a test prior to their transfer of suturing performance showed a categorized the test as near-transfer test; hence, we included the retention test significant effect of test (F[2,60] = 18.01, because the fundamentals of horizontal score as a covariate to account for likely P < .001, η2 = 0.38) and significant mattress suturing had not changed, though interdependence between performances interaction between group and time important contextual task elements were on the two tests. We included other (F[2,60] = 5.23, P < .008, η2=0.15) altered, requiring participants to perform covariates, such as previous hours of (see Figure 2). The interaction was the same motor skills with moderate experience, and previous suturing driven by the Do then See group’s poor variations in instrument handling and attempts, only if they were shown to performance on the midpoint test, and motor skills requirements. predict transfer performance. We analyzed the absence of a statistically significant participants’ scores on the other tests group effect on immediate post and Our secondary outcomes were (midpoint, immediate post, and retention) retention tests. Both groups improved performance on the midpoint, immediate using a repeated-measures ANOVA (group from midpoint to immediate posttest post, and retention tests. We also collected × test). For the self-efficacy scores, we (P < .001) and experienced a significant data on participants’ self-efficacy, using analyzed using another repeated-measures drop in performance on retention test a visual analogue scale of 0 to 100 which ANOVA, across all four tests (midpoint, (P = .003). We found no significant asked participants to judge themselves immediate post, retention, and transfer). differences between groups overall 21 according to a previously studied scale : We set significance at an alpha of 5% (F[1,30] = 0.96, P < .33). “On a scale from 0 to 100 with 10 being two sided, and applied the Bonferroni not sure, 40 being somewhat sure, 70 correction when conducting multiple The ANOVA of self-efficacy across the being pretty sure, and 100 being very comparisons. Analysis software was done four performance tests is reported in sure, how sure are you that you will with SPSS 22 (IBM Corporation, Armonk, Figure 3. Data were missing for two perform your next suturing and knot- New York). additional participants in the Do then tying attempt effectively and safely?” See group (N = 14), and one participant in the See then Do group (N = 15). We Scoring Results detected a significant main effect of test A single blinded rater scored participants’ We recruited 18 participants per study (F[3,81] = 9.52, P < .0001, η2=0.26), performance using the global rating scale arm, and 2 from each group dropped out where participants’ self-efficacy was

Academic Medicine, Vol. 93, No. 11 / November 2018 Supplement S39 Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited. Impact of Assessment

technical skills23 provide an ideal modality and curricular structure for using guided discovery approaches. Yet, the dominant training paradigm in simulation, mastery learning,24 includes little time for discovery learning in any formal guidelines25 and, instead, emphasizes direct instructor feedback and attainment of performance standards. Our results suggest that allotting some time to discovery learning may enhance transfer learning outcomes while also using less instructor time. Indeed, guided discovery curricula would use half the time required of faculty in a mastery-based course. Future studies might compare the Do then See Figure 1 Performance on transfer suturing task by group; a significant effect for the Do then See sequence with a mastery learning model group (F[1,28] = 10.14, P < .004, η2 = 0.27). by evaluating learning, self-efficacy, and training efficiency (i.e., faculty time, low after the midpoint test, improved performance. Our findings confirmed use of consumable resources) as key significantly at immediate posttest those hypotheses. Crucially, learners outcomes. (P < .001), and declined significantly in the Do then See group had lower at retention (P = .01) and transfer tests self-efficacy than the See then Do group, The See then Do group had higher (P = .003) for both groups. Overall, the despite having similar or statistically overall self-efficacy across all tests, See then Do group had significantly superior performance outcomes. Indeed, including the transfer test, despite having higher self-efficacy than the Do then See the group mean self-efficacy for the Do lower performance. Ample evidence group for all tests (F[1,27] = 8.5, P = .007, then See group never rose above the demonstrates the perils of relying 2 η = 0.24). lowest group mean for the See then Do on learners’ self-assessments26,27 and group. Our results align with evidence self-efficacy to gauge competence. Our from concept and classroom learning, Discussion results further add to this phenomenon showing that the Do then See sequence by showing that increasing learners’ We tested the impact of the usual enhances learning of simulated suturing perceived fluency for a task might apprenticeship model (See then Do) skills and seemingly limits self-efficacy. not align with actual learning.28 This against guided discovery: a sequence of occurred with the See then Do group, discovery followed by direct instruction Discovery learning is not a novel concept which likely used their higher immediate (Do then See). We hypothesized that for medical education, especially in performance (on the midpoint test) allowing learners to experiment with a preclinical training of fundamental to judge the extent of their learning,29 task before interacting with an instructor concepts (e.g., problem-based learning, despite the retention and transfer test would benefit their near-transfer test case-based learning). The ubiquity data showing that their higher self- performance and would have no benefit of simulation centers and the use of efficacy was not warranted. This pattern for immediate posttest or retention test intensive “boot camps” for training is consistent with research showing that learning conditions that lead to increased immediate performance (i.e., our midpoint and posttests) often yield increased judgments of learning and, paradoxically, decreased future performance.30 Conversely, the Do then See group may have had low self-efficacy because they experienced the “desirable difficulty” effect of struggling prior to being shown “how to do” the task. While inappropriate as a stand-alone measure, we suggest that self-efficacy is an imperfect metric for evaluating learning interventions, in addition to measuring learners’ actual performance.

Previous studies provide many potential Figure 2 Performance on midpoint, immediate post, and retention tests. A significant effect of reasons why the Do then See sequence test (F[2,60] = 18.01, P < .001, η2 = 0.38) and significant interaction between group and time appears to improve transfer outcomes. (F[2,60] = 5.23, P < .008, η2 = 0.15) but not of group (F[1,30] = 0.96, P < .33). First, a discovery period essentially forces

S40 Academic Medicine, Vol. 93, No. 11 / November 2018 Supplement Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited. Impact of Assessment

between discovery learning and direct instruction, and time delay are surely worth exploring. Moreover, although our study had a randomized design, a relatively small sample size is still an issue because randomization may not balance the unknown confounders that may influence the effect. Future randomized studies should aim to replicate our effect with larger samples.37 Lastly, we studied effects on learning outcomes alone, and encourage further study of the proposed mechanisms of these two learning sequences using behavioral and process measures. Figure 3 Self-efficacy across the four performance tests. We found a main effect of test (F[3,81] = 9.52, P < .0001, η2 = 0.26), and the See then Do group had significantly higher self- Conclusions efficacy than the Do then See group for all tests (F[1,27] = 8.5, P = .007, η2 = 0.24). Our study presents experimental evidence for the benefits of a Do then learners to experiment with, and thus with the task; such activation prior to See sequence for transfer of learning in produce, a number of possible variations direct instruction may be an affordance simulation-based surgical skills training. 34,35 in how to perform the assigned task of the discovery phase. Clarifying Allowing learners to experiment before 17,18 successfully. Alternatively, direct which of these potential mechanisms they interact with an instructor may help instruction often exposes trainees to is at play requires studies focusing on balance learner autonomy and instructor “the way to do it,” and they may then both behavioral measures and learning time pressures. Moreover, our study work on replicating the routine variation outcomes. shows that educators and curriculum they have just observed, rather than developers will likely better serve learners experimenting with different variations. We must note some study limitations. by pausing to consider how to combine There is evidence that variation during Firstly, given logistical and funding learner SRL and direct instruction within learning yields improved acquisition and challenges, we used a single blinded training sessions and across broader subsequent transfer of performance.31 rater, which prevents us from producing curriculum plans. This form of task variation is often a measure of interrater reliability. characterized as “contextual interference” Fortunately, previous studies have Acknowledgments: The authors wish to thank and may be afforded by the discovery shown that the intraclass correlation Polina Mirinova for her assistance with data 32 collection, the Surgical Skills Centre at Mount phase. In combination with direct coefficient is typically acceptable when Sinai Hospital, and the MD Program at the instruction, learners may experience assessing suturing skills using the University of Toronto. the necessary variation for successful same GRS.36 Secondly, our transfer transfer. Second, a degree of struggle task involved minimal change in the Funding/Support: Funding for this work when first learning a task may help contextual elements from the original was provided by the BMO Chair in Health Professions Education at the Wilson Centre in the learners take better advantage of task, which we chose to illustrate the University Health Network. subsequent time with the instructor, effect of our training sequences on including asking more relevant or higher-order learning outcomes, beyond Other disclosures: None reported. targeted questions19 as well as attending strict replication and retention. We Ethical approval: Provided by the University of to critical features of the movement. recommend, however, that future studies Toronto Health Sciences Research Ethics Board. Our research assistants, participants, and isolate retention and transfer outcomes instructors noted differences in their separately, rather than including both K. Kulasegaram is assistant professor, Department interactions, depending on the sequence tests in a single design. Alternative of Family and Community Medicine, and education of learning, though these anecdotes transfer outcomes studied in the concept scientist, MD Program and Wilson Centre, Faculty of Medicine, University of Toronto, Toronto, Ontario, must be confirmed through rigorous, learning literature include far transfer Canada. prospective data collection of behavioral tasks where learners must innovate D. Axelrod is a resident in orthopaedic surgery, measures. Third, a broader mechanism solutions or learn new ideas by using the McMaster University, Hamilton, Ontario, Canada. may also be found in the underlying concepts taught in the original session.17 C. Ringsted is vice dean and professor of rationale for constructivist education Thirdly, our study represents only one education and director, Centre for Health Science approaches including discovery learning. approach to integrating discovery and Education, Faculty of Health, Aarhus University, Constructivists stress the importance of direct instruction. For feasibility reasons, Aarhus, Denmark. allowing learners to construct their own we constrained the amount of time R. Brydges is assistant professor, Department of “meaning” of the task by scaffolding it in discovery, the ratio of discovery to Medicine, holds the Professorship in Technology- on their previous knowledge.33 To do direct instruction, and the delay between Enabled Education at St. Michael’s Hospital, and is education scientist, Wilson Centre, Faculty of so successfully, learners must activate discovery and direct instruction. Other Medicine, University of Toronto, Toronto, Ontario, previous knowledge while engaging forms of discovery learning, ratios Canada.

Academic Medicine, Vol. 93, No. 11 / November 2018 Supplement S41 Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited. Impact of Assessment

References 12 Kuhn D, Black J, Keselman A, Kaplan D. The 26 Eva KW, Regehr G. Knowing when to look development of cognitive skills to support it up: A new conception of self-assessment 1 Brydges R, Nair P, Ma I, Shanks D, Hatala inquiry learning. Cogn Instr. 2000;18:495. ability. Acad Med. 2007;82(10 suppl):S81–S84. R. Directed self-regulated learning versus 13 Mayer RE. Should there be a three-strikes 27 Eva KW, Regehr G. Exploring the divergence instructor-regulated learning in simulation rule against pure discovery learning? The between self-assessment and self-monitoring. training. Med Educ. 2012;46:648–656. case for guided methods of instruction. Am Adv Health Sci Educ Theory Pract. 2 Artino AR Jr, Cleary TJ, Dong T, Hemmer Psychol. 2004;59(1):14–19. 2011;16:311–329. PA, Durning SJ. Exploring clinical reasoning 14 Kapur M. Productive failure in learning 28 Kornell N, Bjork RA. Learning concepts in novices: A self-regulated learning the concept of variance. Instr Sci. and categories: Is spacing the “enemy of microanalytic assessment approach. Med 2012;40(4):651–672. induction”? Psychol Sci. 2008;19:585–592. Educ. 2014;48:280–291. 15 Schwartz DL, Martin T. Inventing to prepare 29 Kornell N, Hausman H. Performance bias: 3 Brydges R, Manzone J, Shanks D, et al. for future learning: The hidden efficiency Why judgments of learning are not affected by Self-regulated learning in simulation-based of encouraging original student production learning. Mem Cognit. 2017;45:1270–1280. training: A systematic review and meta- in statistics instruction. Cogn Instr. 30 Bjork EL, Bjork RA. Making things hard analysis. Med Educ. 2015;49:368–378. 2004;22:129–184. on yourself, but in a good way: Creating 4 Murad MH, Coto-Yglesias F, Varkey P, 16 Mylopoulos M, Brydges R, Woods NN, desirable difficulties to enhance learning. Prokop LJ, Murad AL. The effectiveness of Manzone J, Schwartz DL. Preparation for In: Gernsbacher MA, Pew RW, Hough LM, self-directed learning in health professions future learning: A missing competency in Pomerantz JR, eds. Psychology and the Real education: A systematic review. Med Educ. health professions education? Med Educ. World: Essays Illustrating Fundamental 2010;44:1057–1068. 2016;50:115–123. Contributions to Society. New York, NY: 5 Lee HS, Anderson JR. Student learning: What 17 Kapur M. Productive failure in learning Worth Publishers; 2011:56–64. has instruction got to do with it? Annu Rev math. Cogn Sci. 2014;38:1008–1022. 31 Kulasegaram K, Min C, Howey E, et al. Psychol. 2013;64:445–469. 18 Klahr D, Nigam M. The equivalence of The mediating effect of context variation 6 Schwartz DL, Bransford JD. A time for learning paths in early science instruction: in mixed practice for transfer of basic telling. Cogn Instr. 1998;16(4):475–522. Effect of direct instruction and discovery science. Adv Health Sci Educ Theory Pract. 7 Lee HS, Anderson A, Betts S, Anderson JR. learning. Psychol Sci. 2004;15:661–667. 2015;20:953–968. When does provision of instruction promote 19 DeCaro MS, Rittle-Johnson B. Exploring 32 Willis RE, Curry E, Gomez PP. Practice learning? In: Carlson L, Hoelscher C, Shipley mathematics problems prepares children to schedules for surgical skills: The role of task T, eds. Proceedings of the 33rd Annual learn from instruction. J Exp Child Psychol. characteristics and proactive interference on Conference of the Cognitive Science Society. 2012;113:552–568. psychomotor skills acquisition. J Surg Educ. Austin, TX: Cognitive Science Society; 20 Mylopoulos M, Woods N. Preparing medical 2013;70:789–795. 2011:3518–3523. students for future learning using basic 33 Pressley M, Wood E, Woloshyn VE, Martin V, 8 Kirschner PA, Sweller J, Clark RE. Why science instruction. Med Educ. 2014;48:667– King A, Menke D. Encouraging mindful use minimal guidance during instruction does 673. of prior knowledge: Attempting to construct not work: An analysis of the failure of 21 Cleary TJ, Zimmerman BJ. Self-regulation explanatory answers facilitates learning. Educ constructivist, discovery, problem-based, differences during athletic practice by Psychol. 1992;27:12–18. experiential, and inquiry-based teaching. experts, non-experts, and novices. J Appl 34 Schmidt HK, Loyens SMM, van Gog T, Paas Educ Psychol. 2006;41(2):75–86. Sport Psychol. 2001;13(2):185–206. F. Problem-based learning is compatible with 9 Clark RE, Kirschner PA, Sweller J. Putting 22 Hatala R, Cook DA, Brydges R, Hawkins R. human cognitive architecture: Commentary students on the path to learning: The case Constructing a validity argument for the on Kirschner, Sweller. Educ Psychol. for fully guided instruction. Am Educ. Objective Structured Assessment of Technical 2007;42:10–14. 2012;36(1):6–11. Skills (OSATS): A systematic review of 35 Metcalfe J, Kornell N. Principles of cognitive 10 Barrows HS. Problem-Based Learning validity evidence. Adv Health Sci Educ science in education: The effects of Applied to Medical Education. Springfield, Theory Pract. 2015;20:1149–1175. generation, errors, and feedback. Psychon IL: Southern Illinois University Press; 23 Wayne DB, Cohen ER, Singer BD, et Bull Rev. 2007;14:225–229. 2000. al. Progress toward improving medical 36 Brydges R, Carnahan H, Safir O, Dubrowski 11 Hmelo CE, Lin X. Becoming self-directed school graduates’ skills via a “boot camp” A. How effective is self-guided learning of learners: Strategy development in problem- curriculum. Simul Healthc. 2014;9:33–39. clinical technical skills? It’s all about process. based learning. In: Evensen D, Hmelo CE, 24 Inui TS. The charismatic journey of mastery Med Educ. 2009;43:507–515. eds. Problem-Based Learning: A Research learning. Acad Med. 2015;90:1442–1444. 37 Altman DG. Statistics and ethics in medical Perspective on Learning Interactions. 25 McGaghie WC. When I say … mastery research: III How large a sample? Br Med J. Mahwah, NJ: Erlbaum; 2000:227–250. learning. Med Educ. 2015;49:558–559. 1980;281:1336–1338.

S42 Academic Medicine, Vol. 93, No. 11 / November 2018 Supplement Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited. Impact of Assessment

Appendix 1 Instructor Template and Script Suturing Instructor Script:

Today we will be learning the horizontal mattress suture.

Begin by selecting the 3-0 nylon suture. Open the package and grasp the needle of the suture with the suture driver. Position the needle so that it sits right at the tip of the needle driver and such that the needle driver grasps the needle 1/2 to 2/3 of the way from the sharp end to the blunt end of the needle. Sutures have “memory,” meaning they will stay curled up and difficult to work with after removing them from the package. Pull gently on the suture material in a constant manner for a few seconds to reduce the coiling.

With the suture driver in your dominant hand, and the toothed Adson forceps in your nondominant hand, begin the procedure for the horizontal mattress suture. Start by elevating the skin gently on one side of the wound. Pronate your hand to position the needle 5 to 10 mm from the wound edge, and 90 degrees to the plane of the skin. Supinate your hand as you pierce the epidermal and dermal layers to curve the needle through and into the wound. Release your needle driver and regrasp the needle in the wound. Continue to supinate to bring the full needle through the skin. Next elevate the other side of the wound. Enter at the same depth (under the dermal layer) as you exited on the opposite side. Supinate aiming to have the needle exit the skin the same distance from the wound edge you entered on the opposite side. Release the needle driver and grasp the needle outside the skin. Continue to supinate to curve the needle completely out of the skin. Pull the suture through leaving a short tail.

At this point the suture needle must be loaded in the opposite direction, aiming back towards the original wound edge. You will enter the skin on the same side you just exited approximately 1 cm further down the wound edge. In a backhand fashion, pronate as you enter the skin at 90 degrees. Repeat the process of grasping the needle with wound and then after repositioning, complete the second bite on the original wound edge. Pull the suture through and prepare for an instrumented tie.

At this point the long tail of the suture should be farthest from you and the short end closer to you, both on the same side of the wound. Place your needle driver between the two tails. Loop the long tail around the needle driver end towards the short tail twice; this produces a surgeon’s knot. Grasp the short tail with the needle driver and pull it through the loop, cross your hands as you pull the knot tight to lay it down flat. Place the needle driver back between the two tails. Again loop the long tail towards the short tail, which should now be on opposite sides than previous. Continue this process until at least four throws are completed.

Grasp the short tail in the suture driver when you have completed tying the knot. Pass the needle driver and long end of the suture into your nondominant hand. Grab the straight Mayo scissors in your dominant hand and cut both suture tails at least 1 cm in length.

Academic Medicine, Vol. 93, No. 11 / November 2018 Supplement S43 Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited. Impact of Assessment

Appendix 2 Wound Closure Global Rating Scale