Consensus-Based Training and Assessment Model for General Surgery

Original article

Consensus-based training and assessment model for general surgery

P. Szasz1, M. Louridas1, S. de Montbrun1, K. A. Harris2 and T. P. Grantcharov1

1Department of Surgery, University of Toronto, Toronto, and 2Royal College of Physicians and Surgeons of Canada, Ottawa, Ontario, Canada Correspondence to: Dr P. Szasz, Department of Surgery, St Michael’s Hospital, 30 Bond Street 16CC-056, Toronto, Ontario, M5B 1 W8, Canada (e-mail: [email protected])

Background: Surgical education is becoming competency-based with the implementation of in-training milestones. Training guidelines should reflect these changes and determine the specific procedures for such milestone assessments. This study aimed to develop a consensus view regarding operative procedures and tasks considered appropriate for junior and senior trainees, and the procedures that can be used as technical milestone assessments for trainee progression in general surgery. Methods: A Delphi process was followed where questionnaires were distributed to all 17 Canadian general surgery programme directors. Items were ranked on a 5-point Likert scale, with consensus defined as Cronbach’s 𝛂 of at least 0⋅70. Items rated 4 or above on the 5-point Likert scale by 80 per cent of the programme directors were included in the models. Results: Two Delphi rounds were completed, with 14 programme directors taking part in round one and 11 in round two. The overall consensus was high (Cronbach’s 𝛂=0⋅98). The training model included 101 unique procedures and tasks, 24 specific to junior trainees, 68 specific to senior trainees, andnine appropriate to all. The assessment model included four procedures. Conclusion: A system of operative procedures and tasks for junior- and senior-level trainees has been developed along with an assessment model for trainee progression. These can be used as milestones in competency-based assessments.

Presented to the Annual Clinical Congress of the American College of Surgeons, Chicago, Illinois, USA, October 2015; published in abstract form as JAmCollSurg2015; 221(Suppl 1): S45

Paper accepted 10 December 2015 Published online 22 March 2016 in Wiley Online Library (www.bjs.co.uk). DOI: 10.1002/bjs.10103

Introduction account11–16. It is also important to identify procedures that can be used for these technical milestone assessments Surgical education is changing as a result of a number within a competency-based framework. As surgical train- of complex and interdependent factors, including trainee ing evolves, the goal of training programmes should be to working-hour restrictions, decreased operating room time, optimize the available operating time with procedures and increased patient complexity, the growing use of technol- ogy and the recent focus on surgeon volume–outcome tasks that trainees will perform on a regular basis, in order 16–18 data1–7. to increase their competence and confidence .Train- As a result, educational bodies, including the Royal Col- ing should be directed away from focusing on procedures lege of Physicians and Surgeons of Canada (RCPSC), the and tasks that trainees may not perform after the comple- Intercollegiate Surgical Curriculum Programme (ISCP) in tion of training, particularly complex surgical procedures the British Isles, and the Accreditation Council on Gradu- that are nowadays directed to high-volume centres in order 19,20 ate Medical Education (ACGME) based in the USA, are to achieve improved patient outcomes . implementing competency-based education frameworks The first objective of this study was to outline a consensus with a focus on milestones8–10. At this critical juncture, on the procedures and tasks that are appropriate for junior- it seems an appropriate time to re-evaluate current gen- and senior-level trainees in general surgery. The second eral surgery training guidelines and see how to implement objective was to outline a consensus on the procedures that more progressive guidelines that take recent changes into can be used for technical milestone assessments, in order

to determine whether junior-level trainees can progress to Table 1 Participating programmes and programme director’s senior-level trainees. specialization

The Delphi technique was used to achieve consensus Institution Programme director’s specialization among a group of content experts while minimizing inter- University of British Columbia General surgery, MIS, surgical oncology personal interactions, often used in areas where there is University of Alberta General surgery, trauma/acute care 21 26 a lack of evidence – . The major cornerstones of the surgery Delphi technique include: the use of experts, participant University of Calgary General surgery, surgical oncology, anonymity, iterations with interim feedback and statistical trauma/acute care surgery University of Saskatchewan General surgery, CCM 21,24,27 aggregation of results . Although the classical Delphi University of Manitoba* General surgery, trauma/acute care technique has been used for over 50 years, multiple varia- surgery, CCM tions have been developed, each of which has modified one Western University General surgery, colon and rectal 27,28 surgery, trauma/acute care surgery or more of the cornerstones described above .There University of Toronto General surgery, trauma/acute care 29 is agreement, however, as described by Cuhls , that the surgery, CCM Delphi technique is a survey administered to experts in McMaster University* General surgery, two or more iterations, in which after the first round the hepatopancreaticobiliary surgery Queen’s University General surgery, answers (or results) are returned to the experts as a method hepatopancreaticobiliary surgery, MIS of interim feedback. Experts in the context of a Delphi University of Ottawa General surgery, colon and rectal technique are individuals who understand a topic of study surgery 28 University of Montreal* General surgery, surgical oncology more than others in the same field . Programme direc- Sherbrooke University General surgery, MIS tors were chosen as the panellists in this Delphi technique Dalhousie University General surgery, as they are experts in training general surgery residents, hepatopancreaticobiliary surgery, they are up-to-date on changes within their specialty, and transplant, trauma/acute care surgery Memorial University General surgery, MIS are aware of the evolving educational initiatives that have placed an emphasis on achieving specific competencies and *These programmes completed part one of the Delphi technique only. milestones8. Anonymity refers to the participants being MIS, minimally invasive surgery; CCM, critical care medicine. unaware of which experts provided which answers28.The benefits of anonymity include a lack of an overly influential participant and the ability of participants to change their described as the median (i.q.r.) value, for each response , , opinions without the other participants’ knowledge24 27 28. item24,27,28. The survey was administered using the online platform SurveyMonkey™ (Palo Alto, California, USA). All pro- Methods gramme directors were sent an e-mail to their secure The research ethics board at the University of Toronto university-affiliated addresses with instructions and a link approved the study. to participate. Participation was voluntary. The first round closed 6 weeks after the initial contact, and after three e-mail reminders had been distributed. The second round Delphi methodology was returned only to those programme directors who The experts employed in this study were the general responded in the first round. Procedures and tasks sug- surgery programme directors across Canada within an gested to be out of the scope of practice of general surgery accredited RCPSC postgraduate training programme. trainees, by both a lack of first round responses and current Anonymity was maintained as all interactions between guidelines11–13 (such as vascular, thoracic and transplanta- participants took place via an online platform. tion surgery), were not included in the second round. The Two rounds of the Delphi technique were completed second round closed 6 weeks after it was distributed, and to achieve consensus. The content of the first round was after a subsequent three e-mail reminders. generated by the programme directors, then supplemented The first objective of the study was accomplished byask- with a review of the literature and current guidelines11–13. ing the programme directors which procedures and tasks, All responses and associated comments were redistributed characterized by anatomical region categories (upper gas- to the programme directors for ranking in the second trointestinal tract, lower gastrointestinal tract, breast, etc.), round. they considered a junior trainee (end of postgraduate year Responses of all participating experts were combined (PGY) 2) and a senior trainee (end of PGY-5) should be to give an overall opinion of the entire group28.Thiswas competent to perform. Competence in the context of this

study was defined for the programme directors as ‘suf- Table 2 Junior-level procedures and tasks included in the final ficiency of qualification’ by the Oxford English Dictionary consensus training model, arranged by anatomical category second edition30: the ability to perform the task or proce- 31 Anatomical categorization of tasks Cronbach’s Likert score dure safely and efficiently without guidance . and procedures α per question per item* The second objective of the study was accomplished by Open and minimally invasive upper 0⋅75 asking the programme directors to identify the procedures gastrointestinal tract (performed by all levels of trainees) for which variations in Insertion of laparoscopic trocars 5⋅0(4⋅0–5⋅0) Use of surgical staplers 5⋅0(4⋅0–5⋅0) performance were most often exhibited. Open and minimally invasive lower 0⋅93 gastrointestinal tract ⋅ ⋅ ⋅ Closure of stoma† 4 0(375–4 0) Ranking and consensus determination Laparoscopic 5⋅0(4⋅75–5⋅0) appendicectomy – simple† All question items were ranked on a 5-point Likert scale. Open appendicectomy – simple† 5⋅0(4⋅0-5⋅0) Open and minimally invasive 0⋅85 Consensus was reached when Cronbach’s α was at least hepatopancreaticobiliary 0⋅70 per question. Accepted guidelines suggest an inter- ⋅ ⋅ ⋅ Opening/closing chevron incision 5 0(40–5 0) nal consistency using Cronbach’s α of 0⋅70 or above Clip application to cystic artery/duct 4⋅0(4⋅0–5⋅0) 32–34 Laparoscopic mobilization of 4⋅0(4⋅0–5⋅0) for research/education recommendations . The closer gallbladder off liver Cronbach’s α is to 1⋅0, the greater the consistency in ⋅ ⋅ ⋅ Laparoscopic 4 0(40-5 0) responses by the expert participants34. cholecystectomy – simple† Open and minimally invasive hernias 0⋅95 Inguinal hernia incision 5⋅0(4⋅0–5⋅0) Inguinal hernia mobilization of 5⋅0(4⋅0–5⋅0) Inclusion in final models spermatic cord Inguinal hernia sac identification⋅ 4 5(4⋅0–5⋅0) Once consensus was achieved for each question, the next Primary repair of inguinal 4⋅0(3⋅75–5⋅0) hernia – simple† step was to identify positive, negative or neutral consen- Mesh repair of inguinal hernia – simple† 4⋅0(4⋅0–5⋅0) sus. Items ranked on the 5-point Likert scale as 5 (strongly Umbilical hernia incision 5⋅0(4⋅0–5⋅0) agree) or 4 (agree) by at least 80 per cent of the pro- Umbilical hernia sac identification⋅ 5 0(4⋅0–5⋅0) Umbilical hernia repair 4⋅5(4⋅0–5⋅0) gramme directors reached positive consensus and were , Ventral hernia incision 4⋅0(3⋅75–5⋅0) included in the final models35 36. Procedures and tasks Perianal 0⋅87 that ranked 1 (unimportant) or 2 (less important) by 80 Incision and drainage of anorectal 5⋅0(4⋅0–5⋅0) abscess† per cent or more of the programme directors reached EUA† 5⋅0(4⋅0–5⋅0) negative consensus and were not included in the final Fistula identification 4⋅0(4⋅0–5⋅0) models. All other combinations of rankings received neu- Haemorrhoid banding† 4⋅0(4⋅0–4⋅25) Excision pilonidal sinus 4⋅0(4⋅0–4⋅0) tral consensus and, again, were not included in the final Flexible sigmoidoscopy 4⋅0(4⋅0–5⋅0) models. Rigid sigmoidoscopy 4⋅5(4⋅0–5⋅0) Breast 0⋅90 Incision and drainage of breast abscess 4⋅5(4⋅0–5⋅0) Closure of mastectomy incision 5⋅0(4⋅0–5⋅0) Statistical analysis Breast biopsy (FNA) 4⋅0(4⋅0–5⋅0) Breast biopsy (surgical) 4⋅0(4⋅0–5⋅0) Internal consistency for all data collected was deter- ⋅ Emergency and trauma 0 83 mined using Cronbach’s α33. For each Delphi survey Opening/closing of laparotomy 5⋅0(3⋅75–5⋅0) FAST 4⋅0(3⋅5–4⋅0) item, the median (i.q.r.) of the programme direc- Chest tube placement 5⋅0(4⋅0–5⋅0) tors’ responses was determined. SPSS® version 22⋅0 ⋅ ⋅ ⋅ Central venous catheter placement 5 0(40–5 0) (IBM, Armonk, New York, USA) was used for statistical Soft tissue n.a.‡ Incision and drainage of subcutaneous 5⋅0(4⋅75–5⋅0) analysis. abscess Excision of benign skin and soft tissue 5⋅0(4⋅0–5⋅0) lesion Results *Values are median (i.q.r.). †Tasks and procedures appropriate for both junior and senior trainees. ‡Level of consensus not reached; not included Of 17 programme directors approached, 14 took part in in final model (see text for full explanation). EUA, evaluation under round one and 11 in round two. The participating pro- anaesthesia; FNA, fine-needle aspiration; FAST, focused assessment with sonography for trauma; n.a., not applicable. grammes and directors’ areas of specialization are shown in Ta b l e 1 .

Table 3 Senior-level procedures and tasks included in the final Table 3 Continued. consensus training model, arranged by anatomical category Anatomical categorization of tasks Cronbach’s Likert score Anatomical categorization of tasks Cronbach’s Likert score and procedures α per question per item* and procedures α per question per item* Open intra-abdominal hernia repair 4⋅0(3⋅75–4⋅25) Open and minimally invasive upper 0⋅92 (Petersen’s space) gastrointestinal tract Perianal 0⋅89 OGD 5⋅0(4⋅0–5⋅0) EUA† 5⋅0(4⋅0–5⋅0) Open and minimally invasive lower 0⋅93 Incision and drainage of anorectal 5⋅0(4⋅75–5⋅0) gastrointestinal tract abscess† Laparoscopic mobilization of right colon 4⋅5(4⋅0–5⋅0) Fistulotomy 5⋅0(4⋅0–5⋅0) Open mobilization of right colon 5⋅0(4⋅75–5⋅0) Fistulectomy 5⋅0(3⋅5–5⋅0) Laparoscopic mobilization of 4⋅5(4⋅0–5⋅0) Seton placement 5⋅0(4⋅75–5⋅0) sigmoid/left colon Haemorrhoidectomy 5⋅0(4⋅0–5⋅0) Open mobilization of sigmoid/left colon 5⋅0(4⋅75–5⋅0) Haemorrhoid banding† 5⋅0(4⋅75–5⋅0) Laparoscopic mobilization of rectum 4⋅0(3⋅75–5⋅0) Lateral internal sphincterotomy 4⋅5(4⋅0–5⋅0) Open mobilization of rectum 5⋅0(4⋅75–5⋅0) Colonoscopy 5⋅0(4⋅75–5⋅0) Open splenectomy 4⋅5(4⋅0–5⋅0) Breast 0⋅78 Laparoscopic right hemicolectomy 4⋅5(4⋅0–5⋅0) Lumpectomy 5⋅0(4⋅75–5⋅0) Open right hemicolectomy 5⋅0(4⋅75–5⋅0) Excision of ducts 5⋅0(4⋅75–5⋅0) Laparoscopic left hemicolectomy 4⋅5(4⋅0–5⋅0) Mastectomy – simple 5⋅0(4⋅75–5⋅0) Open left hemicolectomy 5⋅0(4⋅75–5⋅0) Mastectomy – skin sparing 5⋅0(4⋅0–5⋅0) Laparoscopic sigmoid resection 4⋅5(4⋅0–5⋅0) Mastectomy – modified radical⋅ 5 0(4⋅0–5⋅0) Open sigmoid resection 5⋅0(4⋅75–5⋅0) SLNB 5⋅0(4⋅0–5⋅0) Open LAR 4⋅0(4⋅0–5⋅0) ALND 5⋅0(4⋅75–5⋅0) Laparoscopic small bowel resection 4⋅5(4⋅0–5⋅0) Emergency and trauma 0⋅91 Open small bowel resection 5⋅0(4⋅0–5⋅0) Damage control laparotomy 5⋅0(4⋅75–5⋅0) Intracorporeal anastomosis 4⋅0(4⋅0–5⋅0) Abdominal packing 5⋅0(4⋅75–5⋅0) Intracorporeal vessel ligation 5⋅0(4⋅0–5⋅0) Trauma splenectomy 5⋅0(4⋅0–5⋅0) (ileocolic/IMA) Surgical management of SBO 5⋅0(4⋅75–5⋅0) Creation of loop/end stoma 5⋅0(4⋅75–5⋅0) Surgical management of LBO 5⋅0(4⋅75–5⋅0) Closure of loop stoma† 5⋅0(4⋅75–5⋅0) Oversewing of bleeding duodenal ulcer 5⋅0(4⋅0–5⋅0) Laparoscopic 5⋅0(4⋅75–5⋅0) Laparoscopic repair of duodenal ulcer 4⋅0(4⋅0–5⋅0) appendicectomy – simple† Open repair of duodenal ulcer 5⋅0(4⋅0–5⋅0) Laparoscopic 5⋅0(4⋅75–5⋅0) Laparoscopic repair of enterotomy 4⋅0(4⋅0–4⋅25) appendicectomy – complex Open repair of enterotomy 5⋅0(4⋅75–5⋅0) Open appendicectomy – simple† 5⋅0(4⋅75–5⋅0) Emergency department thoracotomy 4⋅0(3⋅75–5⋅0) Open appendicectomy – complex 5⋅0(4⋅75–5⋅0) Surgical airway 5⋅0(4⋅0–5⋅0) Open and minimally invasive 0⋅82 Soft tissue 0⋅84 hepatopancreaticobiliary Debridement for necrotizing fasciitis 5⋅0(5⋅0–5⋅0) ⋅ ⋅ ⋅ Control of portal structures for bleeding 4 0(40–5 0) Debridement of wounds 5⋅0(4⋅0–5⋅0) ⋅ ⋅ ⋅ Kocher manoeuvre 5 0(475–5 0) Lymph node biopsy 5⋅0(4⋅75–5⋅0) ⋅ ⋅ ⋅ Mobilization of liver 4 5(40–5 0) Excision of benign skin and soft tissue 5⋅0(4⋅0–5⋅0) Exposure of pancreas 5⋅0(4⋅0–5⋅0) lesion Laparoscopic 5⋅0(4⋅75–5⋅0) Excision of malignant skin and soft 5⋅0(4⋅0–5⋅0) cholecystectomy – simple† tissue lesion Laparoscopic 5⋅0(4⋅75–5⋅0) cholecystectomy – complex *Values are median (i.q.r.). †Tasks and procedures appropriate for both Open cholecystectomy – simple 5⋅0(4⋅0–5⋅0) junior and senior trainees. OGD, oesophagogastroduodenoscopy; LAR, Open cholecystectomy – complex 5⋅0(4⋅0–5⋅0) low anterior resection; IMA, inferior mesenteric artery; CBD, common Open CBD exploration 4⋅0(3⋅5–5⋅0) bile duct; EUA, evaluation under anaesthesia; SLNB, sentinel lymph Open repair of liver injuries – simple 4⋅0(3⋅75–5⋅0) node biopsy; ALND, axillary lymph node dissection; SBO, small bowel Open and minimally invasive hernias 0⋅78 obstruction; LBO, large bowel obstruction. Primary repair of inguinal hernia† 5⋅0(4⋅75–5⋅0) Primary repair of inguinal 5⋅0(4⋅75–5⋅0) hernia – recurrent Mesh repair of inguinal hernia – simple† 5⋅0(4⋅75–5⋅0) Consensus training model for operative procedures Mesh repair of inguinal 5⋅0(4⋅75–5⋅0) and tasks hernia – recurrent Femoral hernia repair 5⋅0(4⋅0–5⋅0) The final consensus training model included 101 unique Laparoscopic ventral hernia repair 4⋅0(3⋅75–4⋅25) procedures and tasks, both open and minimally inva- ⋅ ⋅ ⋅ Open ventral hernia repair 5 0(40–5 0) sive. Twenty-four procedures and tasks were specific to Parastomal hernia repair 5⋅0(4⋅0–5⋅0) junior-level trainees (Ta b l e 2 ), 68 to senior-level trainees

Table 4 Procedures and tasks excluded from the final consensus Table 5 Procedures for technical milestone assessments training model, arranged by anatomical category Likert score per item Anatomical categorization of tasks No. of Likert score Laparoscopic cholecystectomy 4⋅0(4⋅0–5⋅0) and procedures procedures distribution* Laparoscopic appendicectomy 4⋅0(4⋅0–4⋅0) Open and minimally invasive Inguinal hernia repair 4⋅0(4⋅0–4⋅0) upper gastrointestinal tract Small bowel resection 4⋅0(4⋅0–4⋅0) Senior trainees 7 2⋅5–4⋅0(1⋅0–1⋅75) ⋅ Junior trainees 3 3⋅0–4⋅0(0–0⋅75) Values are mean (i.q.r.). Overall Cronbach’s α=0 71. Open and minimally invasive lower gastrointestinal tract or negatively) that these should be included in, or excluded Senior trainees 4 3⋅5–4⋅0(1⋅0–1⋅5) Junior trainees 9 2⋅0–4⋅0(0⋅75–2⋅0) from, the final training modelTa ( b l e 4 ; Appendix S1; sup- Open and minimally invasive porting information). Of these, 30 were procedures and hepatopancreaticobiliary tasks considered specific to junior trainees and 38 spe- ⋅ ⋅ ⋅ ⋅ Senior trainees 11 2 0–4 0(075–1 75) cific to senior trainees. These procedures and tasks were Junior trainees 4 2⋅0–4⋅0(1⋅0–1⋅75) Open and minimally invasive embedded within each anatomical region category. The endocrine ‘Open and minimally invasive endocrine procedures and Senior trainees 6 3⋅0–4⋅0(1⋅0–2⋅5) tasks’ for both junior and senior trainees had all of its ⋅ ⋅ ⋅ Junior trainees 2 4 0(175–2 0) procedures and tasks fall within the neutral consensus Open and minimally invasive hernias group and included: opening and closing thyroid incision, Senior trainees 3 3⋅0(0–1⋅75) thyroid fine-needle aspiration, total thyroidectomy, subto- Junior trainees 1 4⋅0(1⋅0) tal thyroidectomy, hemithyroidectomy, parathyroidecomy Perianal and adrenalectomy (open and laparoscopic). Senior trainees 3 2⋅5–3⋅0(1⋅0–1⋅75) Junior trainees 3 3⋅0–4⋅0(0⋅75–1⋅75) Breast Consensus model of procedures for technical Senior trainees 1 4⋅0(1⋅75) Junior trainees 4 3⋅0–4⋅0(0⋅75–1⋅75) milestone assessments Emergency and trauma The procedures in the final consensus assessment model Senior trainees 0 − Junior trainees 3 4⋅0(1⋅5–2⋅0) for the progression of a junior-level trainee to senior-level Soft tissue trainee are shown in Ta b l e 5 . Senior trainees 3 3⋅0–4⋅0(1⋅0–2⋅75) Junior trainees 1 3⋅0(1⋅0) Discussion *Distribution of median (i.q.r.) ranges for all procedures in that category per trainee group. Using a consensus-based methodology, a training and assessment model has been developed for general surgery (Ta b l e 3 ), and nine were common across all trainees (identi- trainees. fied by an asterisk in Tables 2 and 3). When categorized by A previous study, based on the opinions of US pro- anatomical region, there were three upper gastrointestinal, gramme directors on procedures that general surgery 24 lower gastrointestinal, 13 hepatopancreaticobiliary, trainees should be competent to undertake upon the 16 hernia, 13 perianal, 11 breast, five soft tissue, and 16 completion of residency, demonstrated wide disparity trauma and emergency representative procedures. Overall compared with their actual log books37,38. Although an consensus was high, with Cronbach’s α of 0⋅98. Consensus excellent reference for end-of-training objectives, it did on individual questions ranged from 0⋅69 to 0⋅95. All not address the issue of creating trainee level-specific questions, except junior trainee ‘Soft tissues procedures guidelines, or milestone assessments. The present model and tasks’, reached consensus. describes operative procedures and tasks appropriate Only a single procedure, laparoscopic mobilization of the to junior and senior trainees in general surgery, along rectum, for junior trainees in the ‘Open and minimally with a consensus-based assessment model on operative invasive lower gastrointestinal tract procedures and tasks’ procedures that can be used as milestone assessments, category demonstrated negative consensus; this was not to determine whether trainees are progressing appropri- included in the final model. ately through their surgical training and ultimately for There were 68 procedures and tasks, both open and min- in-training promotion. imally invasive, that received neutral consensus, signifying Regarding the first objective of the present study, proce- that programme directors did not feel strongly (positively dures and tasks where positive consensus was achieved were

included in the final training model. This included all of trainees, all procedures and tasks achieved neutral con- the major general surgical categories, divided by anatom- sensus. The procedures included in this category were ical region. These procedures and tasks are in keeping all related to the thyroid or parathyroid, except for the with previous literature published regarding the most com- adrenalectomy. Programme directors did not feel that mon surgical exposures for general surgery residents39 and, junior or senior trainees should be competent to perform with regard to senior trainees, the list compiled by Bell an open or laparoscopic adrenalectomy, likely reflecting and colleagues37. The procedures and tasks included in the evolution of this operation37,45. Programme directors final training model also represented the more common also did not feel strongly that general surgery trainees procedures performed by practising general surgeons16. should be competent in performing thyroid or parathyroid Their inclusion signified the programme directors’ agree- procedures or tasks, in contrast to previous literature ment of the skill set required by trainees to enter indepen- supporting both an expectation of and experience with dent practice without fellowship training. This was further thyroid procedures37,39. Again this may reflect recent shifts demonstrated by the common procedures and tasks per- in volume–outcome research in many general surgical formed by general surgeons16 having the highest median subspecialties, and the increasing numbers of trainees and smallest interquartile ranges within this study. Further- undertaking fellowship training before completing these more, the procedures with the highest median and small- procedures independently17,18,46. Most procedures and est interquartile ranges were also in keeping with the list tasks in the ‘Open and minimally invasive upper gastro- compiled by Brennan and Debas14, depicting how previ- intestinal tract procedures and tasks’ category and the ously trained general surgeons (often in the absence of a ‘Open and minimally invasive hepatopancreaticobiliary fellowship) at the time of their 10-year recertification had procedures and tasks’ category for senior trainees also self-differentiated their own practice into only a small sub- achieved neutral consensus. Programme directors did not set of their original training. feel that a senior trainee should be competent to perform In the present study, procedures in each anatomical cate- gastrectomy (total or partial), antireflux surgery, pyloromy- gory reached positive consensus, except junior trainee ‘Soft otomy, pancreatic resections, pancreatic anastomoses or tissue procedures and tasks’. This was possibly a result of a biliary anastomoses. Although logical that trainees without true lack of consensus, but more likely because there were subspecialized training in advanced minimally invasive only a few item choices (3) for that question. The lack and hepatopancreaticobiliary surgery should not be per- of item choices has been documented in the literature to forming such procedures, this has not yet been reflected deflate the true value of Cronbach’s α40. Given a lack of in current training curricula5,6,11–13. What seems most consensus for ‘Soft tissue procedures and tasks’, they were useful about this training model is not necessarily the not included in the final model. procedures and tasks present in the final model, but rather The only procedure achieving negative consensus was the procedures and tasks that are absent. Keeping these laparoscopic mobilization of the rectum for junior trainees tasks and procedures in mind for the future of curricu- in the ‘Open and minimally invasive lower gastrointesti- lum design to the RCPSC, ISCP, ACGME and the US nal tract procedures and tasks’ category. The fact that only Surgical Council On Resident Education (SCORE) may a single procedure achieved negative consensus probably prove useful in streamlining the process8–11. reflects programme directors’ reluctance to remove proce- The development of milestones9,47 –49 can serve as a dures completely from this new training model. The rea- link between competency-based education paradigms and sons for this reluctance are likely to be complex, but may competency-based assessments. By definition, milestones involve a perception that a rarely performed procedure may are markers of expected trainee performance at different have elements of transferability to other more commonly stages of training50. Provided they are defined in a clear performed procedures or tasks. Views on skill transfer are, manner, milestones allow programme directors to identify however, mixed19,41–44. The lack of excluded procedures specific trainees who are ready to move on to the next and in the present study is in keeping with the results from more complex phase of training48. Bell and colleagues37, who identified only eight procedures Regarding the second objective of the present study, from a list of 300 that US programme directors did not feel four procedures met consensus as suitable milestone were appropriate for general surgery resident training. assessments including: laparoscopic cholecystectomy, Although all of the major general surgery categories laparoscopic appendicectomy, inguinal hernia repair and had procedures and tasks that achieved neutral consen- small bowel resection. To ensure that these performance sus, within the ‘Open and minimally invasive endocrine assessments are reliable representations of performance, procedures and tasks’ category, for both junior and senior procedures must be assessed on multiple occasions, by

multiple assessors51. When technical performance deficits References are identified, they can then be addressed and remedied 1 Barden CB, Specht MC, McCarter MD, Daly JM, Fahey 52 early, rather than near the completion of training . TJ III. Effects of limited work hours on surgical training. The present study has some limitations. There were J Am Coll Surg 2002; 195: 531–538. three non-responders (14 of 17 responders) in the first 2 Drolet BC, Sangisetty S, Tracy TF, Cioffi WG. Surgical round, with a subsequent loss of a further three pro- residents’ perceptions of 2011 Accreditation Council for gramme directors in the second round. Although these Graduate Medical Education duty hour regulations. JAMA are acceptable non-responder and attrition rates, the reli- Surg 2013; 148: 427–433. 3 Rigberg D, Cole M, Hiyama D, McFadden D. Surgery in ability of Delphi techniques increases with higher par- the nineties. Am Surg 2000; 66: 813–816. ticipant numbers, although no clear minimum number 4 Pofahl WE, Pories WJ. Current status and future directions 21,23,36,53,54 has been documented . It has been shown that of geriatric general surgery. J Am Geriatr Soc 2003; both responders and non-responders have similar under- 51(Suppl): S351–S354. lying characteristics and that the opinions of responders 5 Ricci C, Casadei R, Buscemi S, Taffurelli G, D’Ambra M, are representative of colleagues who did not respond55. Pacilio CA et al. Laparoscopic distal pancreatectomy: what Although programme directors seemed the most appro- factors are related to the learning curve? Surg Today 2015; priate individuals to make decisions regarding training 45: 50–56. 6 Jeong O, Ryu SY, Choi WY, Piao Z, Park YK. Risk factors guidelines, the fact that they worked predominantly in aca- and learning curve associated with postoperative morbidity demic centres may have generated a list biased towards of laparoscopic total gastrectomy for gastric carcinoma. Ann surgery performed in large specialist centres rather than Surg Oncol 2014; 21: 2994–3001. a list representing the needs of the wider surgical popu- 7 Traynor O. Surgical training in an era of reduced working lation. The study was conducted exclusively with Cana- hours. Surgeon 2011; 9(Suppl 1): S1–S2. dian programme directors, limiting transferability to other 8 Frank JR, Jabbour M, Fréchette D, Marks M, Valk N, healthcare systems. With a worldwide transition towards Bourgeois G et al.(eds)Report of the CanMEDS Phase IV competency-based education models, the likelihood is that Working Groups. Royal College of Physicians and Surgeons programmes will have increasing similarities as best prac- of Canada: Ottawa, 2005. 9 Accreditation Council for Graduate Medical Education. tice protocols are incorporated. The limitation imposed Milestones. http://www.acgme.org/acgmeweb/tabid/430/ by variations in surgical training programmes interna- ProgramandInstitutionalAccreditation/NextAccreditation tionally must also be acknowledged. There are wide dif- System/Milestones.aspx [accessed 8 August 2014]. ferences in the structure and duration of basic training, 10 Intercollegiate Surgical Curriculum Programme. The with varying entry and exit points into more subspecial- Intercollegiate Surgical Curriculum: Educating the Surgeons of ized training. Therefore, the models presented here may the Future; 2013. http://www.surreyandsussex.nhs.uk/ be more applicable to programmes similar to those in wp-content/uploads/2014/09/syllabus_CORE_2013.pdf Canada. [accessed 1 June 2015]. The present procedure and task list can be used by those 11 Surgical Council on Resident Education. Curriculum Outline for General Surgery Residency 2013–2014. http://www. involved in surgical education to inform and develop new absurgery.org/xfer/curriculumoutline2013-14.pdf [accessed training guidelines, while the development of in-training 1 June 2015]. milestones to assess technical competence and progression 12 The American Board of Surgery. Specialty of General Surgery can be adjusted to suit specific healthcare systems. Defined; 2015. http://www.absurgery.org/default.jsp?about surgerydefined [accessed 18 August 2014]. 13 Royal College of Physicians and Surgeons of Canada Acknowledgements (RCPSC). Objectives of Training in the Specialty of General Surgery. RCPSC: Ottawa, 2010. The authors thank the following Canadian general surgery 14 Brennan MF, Debas HT. Surgical education in the programme directors who kindly took part in this study: United States: portents for change. Ann Surg 2004; 240: A. Meneghetti, K. Fathimani, L. A. Mack, P. R. Hayes, 565–572. B. McCarthy, M. Ott, N. A. Ahmed, M. J. Marcaccio, 15 Pellegrini CA. Surgical education in the United States: navigating the white waters. Ann Surg 2006; 244: 335–342. D. Jalink, S. Tadros, P. Dubé, É. Comeau, M. Walsh and 16 Royal College of Physicians and Surgeons of Canada. The M. Wells. This work was funded through the RCPSC, H. Future of General Surgery: Evolving to Meet a Changing S. Morton Exchange Fellowship Fund. The authors had Practice. Final Report of the Task Force on the Future of General complete access to the study data. Surgery; 2014. http://www.royalcollege.ca/portal/ Disclosure: The authors declare no conflict of interest. page/portal/rc/common/documents/educational_initiatives/

future_of_general_surgery_report_e.pdf [accessed 1 January laparoscopic colorectal surgery: a Delphi methodology. Am 2016]. JSurg2011; 201: 251–259. 17 Coleman JJ, Esposito TJ, Rozycki GS, Feliciano DV. Early 36 Zevin B, Levy JS, Satava RM, Grantcharov TP. A subspecialization and perceived competence in surgical consensus-based framework for design, validation, and training: are residents ready? J Am Coll Surg 2013; 216: implementation of simulation-based training curricula in 764–771. surgery. J Am Coll Surg 2012; 215: 580–586.e3. 18 Fonseca AL, Reddy V, Longo WE, Gusberg RJ. Graduating 37 Bell RH Jr, Biester TW, Tabuenca A, Rhodes RS, Cofer JB, general surgery resident operative confidence: perspective Britt LD et al. Operative experience of residents in US from a national survey. JSurgRes2014; 190: 419–428. general surgery programs: a gap between expectation and 19 Mattar SG, Alseidi AA, Jones DB, Jeyarajah DR, Swanstrom experience. Ann Surg 2009; 249: 719–724. LL, Aye RW et al. General surgery residency inadequately 38 Bell RH Jr. Why Johnny cannot operate. Surgery 2009; 146: prepares trainees for fellowship: results of a survey of 533–542. fellowship program directors. Ann Surg 2013; 258: 39 Chung RS. How much time do surgical residents need to 440–449. learn operative surgery? Am J Surg 2005; 190: 351–353. 20 Birkmeyer JD, Finks JF, O’Reilly A, Oerline M, Carlin AM, 40 Cortina JM. What is coefficient alpha? An examination of Nunn AR et al. Surgical skill and complication rates after theory and applications. J Appl Psychol 1993; 78: 98–104. bariatric surgery. N Engl J Med 2013; 369: 1434–1442. 41 Ahlering TE, Skarecky D, Lee D, Clayman RV. Successful 21 Hsu CC, Sandford BA. The Delphi technique: making sense transfer of open surgical skills to a laparoscopic environment of consensus. Practical Assessment, Research & Evaluation using a robotic interface: initial experience with laparoscopic 2007; 12:1–8. radical prostatectomy. J Urol 2003; 170: 1738–1741. 22 Jones J, Hunter D. Consensus methods for medical and 42 Seymour NE, Gallagher AG, Roman SA, O’Brien MK, health services research. BMJ 1995; 311: 376–380. Bansal VK, Andersen DK et al. Virtual reality training 23 Fink A, Kosecoff J, Chassin M, Brook RH. Consensus improves operating room performance: results of a methods: characteristics and guidelines for use. Am J Public randomized, double-blinded study. Ann Surg 2002; 236: Health 1984; 74: 979–983. 458–463. 24 Goodman CM. The Delphi technique: a critique. JAdv 43 Figert PL, Park AE, Witzke DB, Schwartz RW. Transfer of Nurs 1987; 12: 729–734. training in acquiring laparoscopic skills. J Am Coll Surg 25 Nathens AB, Rivara FP, Jurkovich GJ, Maier RV, Johansen 2001; 193: 533–537. JM, Thompson DC. Management of the injured patient: 44 Park AE, Lee TH. Evolution of minimally invasive surgery identification of research topics for systematic review using and its impact on surgical residency training. In Minimally the delphi technique. JTrauma2003; 54: 595–601. Invasive Surgical Oncology, Matteoti R, Ashley SW (eds). 26 Marchant EW. Methodological problems associated with Springer: Berlin, 2011; 11–22. the use of the Delphi technique – some comments. Fire 45 Walz MK, Alesina PF, Wenger FA, Deligiannis A, Szuczik Te ch n o l o g y 1988; 24: 59–62. E, Petersenn S et al. Posterior retroperitoneoscopic 27 Rowe G, Wright G. The Delphi technique as a forecasting adrenalectomy – results of 560 procedures in 520 patients. tool: issues and analysis. International Journal of Forecasting Surgery 2006; 140: 943–948. 1999; 15: 353–375. 46 Sosa JA, Bowman HM, Tielsch JM, Powe NR, Gordon TA, 28 Martino JP. Technological Forecasting for Decision Making (3rd Udelsman R. The importance of surgeon experience for edn). McGraw-Hill: New York. 1993. clinical and economic outcomes from thyroidectomy. Ann 29 Cuhls K. Delphi Method: UNIDO Foresight Seminars 2005. Surg 1998; 228: 320–330. http://www.unido.org/fileadmin/import/16959_Delphi 47 Nasca TJ, Philibert I, Brigham T, Flynn TC. The next Method.pdf [accessed 5 August 2014]. GME accreditation system – rationale and benefits. NEngl 30 The Oxford English Dictionary (2nd edn). Oxford University JMed2012; 366: 1051–1056. Press: Oxford, 1989. 48 Frank JR, Snell LS, Sherbino J (eds). Draft CanMEDS 2015 31 Szasz P, Louridas M, Harris KA, Aggarwal R, Grantcharov Milestones Guide. Royal College of Physicians and Surgeons TP. Assessing technical competence in surgical trainees: a of Canada: Ottawa, 2015. systematic review. Ann Surg 2015; 261: 1046–1055. 49 Paddick JS, Le Rolland P. The Annual Review of Competence 32 Shrout PE, Fleiss JL. Intraclass correlations: uses in Progression (ARCP) Process – A Short Review by the assessing rater reliability. Psychol Bull 1979; 86: 420–428. Postgraduate Medical Education and Training Board. General 33 Bland JM, Altman DG. Cronbach’s alpha. BMJ 1997; 314: Medical Council: London, 2010. 572. 50 Royal College of Physicians and Surgeons of Canada. 34 Graham B, Regehr G, Wright JG. Delphi as a method to Competence by Design (CBD): Moving Towards establish consensus for diagnostic criteria. J Clin Epidemiol Competency-Based Medical Education – Benefits of CBD for 2003; 56: 1150–1156. Medical Educators in Each Specialty. http://www.royal 35 Palter VN, MacRae HM, Grantcharov TP. Development of college.ca/portal/page/portal/rc/resources/cbme [accessed 2 an objective evaluation tool to assess technical skill in September 2014].

51 Marriott J, Purdie H, Crossley J, Beard JD. Evaluation of 54 de Villiers MR, de Villiers PJ, Kent AP. The Delphi procedure-based assessment for assessing trainees’ skills in technique in health sciences education research. Med Teach the operating theatre. Br J Surg 2011; 98: 450–457. 2005; 27: 639–643. 52 Wu JS, Siewert B, Boiselle PM. Resident evaluation and 55 McKee M, Priest P, Ginzler M, Black N. How remediation: a comprehensive approach. J Grad Med Educ representative are members of expert panels? Qual Assur 2010; 2: 242–245. Health Care 1991; 3: 89–94. 53 Sierles FS. How to do research with self-administered surveys. Acad Psychiatry 2003; 27: 104–113.

Supporting information

Additional supporting information may be found in the online version of this article: Appendix S1 Procedures and tasks excluded in the final consensus training model, arranged by anatomical category (Word document)

If you wish to comment on this, or any other article published in the BJS, please visit the online Your Views section of the website (www.bjs.co.uk).