DYNAMIC PROCEDURE AIDS SUPPORT CRISIS ATTENTION ADISSERTATION SUBMITTED TO THE DEPARTMENT OF COMPUTER SCIENCE AND THE COMMITTEE ON GRADUATE STUDIES OF STANFORD UNIVERSITY IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY Leslie Wu December 2014 © 2014 by Leslie Wu. All Rights Reserved. Re-distributed by Stanford University under license with the author. This work is licensed under a Creative Commons Attribution- Noncommercial 3.0 United States License. http://creativecommons.org/licenses/by-nc/3.0/us/ This dissertation is online at: http://purl.stanford.edu/xm083hr3885 ii I certify that I have read this dissertation and that, in my opinion, it is fully adequate in scope and quality as a dissertation for the degree of Doctor of Philosophy. Pat Hanrahan, Co-Adviser I certify that I have read this dissertation and that, in my opinion, it is fully adequate in scope and quality as a dissertation for the degree of Doctor of Philosophy. Stuart Card, Co-Adviser I certify that I have read this dissertation and that, in my opinion, it is fully adequate in scope and quality as a dissertation for the degree of Doctor of Philosophy. Scott Klemmer Approved for the Stanford University Committee on Graduate Studies. Patricia J. Gumport, Vice Provost for Graduate Education This signature page was generated electronically upon submission of this dissertation in electronic format. An original signed hard copy of the signature page is on file in University Archives. iii Acknowledgments Much appreciation for my colleague, Jesse Cirimele, who jointly carried out much of the work described in this thesis. Thanks to my Ph.D. thesis committee, Stuart K. Card, Pat Hanrahan, Terry Winograd, Scott Klemmer, and Larry Chu. They all provided invaluable feedback and guidance over the years. In fact, many of the good ideas and approaches in this thesis were, no doubt, bubbled up in the brain of Stuart K. Card. The Stanford Anesthesia Informatics Media (AIM) Lab, including Drs. Larry Chu, Kyle Harrison and Sara Goldhaber-Fiebert, Anna Clemenson, graciously o↵ered us laboratory space and equipment. Dr. David Gaba and the Li-Ka Shing Center for Immersive and Learning-Based Simulation let us observe medical simulations over the course of many months. Both Drs. Larry Chu and Kyle Harrison provided the initial motivation for the work, and contributed greatly at all stages of the design process. CURIS students Kristen Leach, Justin Lee, Tanya Yu, Kyle M. Barrett, and Katherine Chen spent hours prototyping visual and interaction designs, interaction designs, and helped to run several of the studies described. Jon Bassen, Ti↵any Dharma and Nicholas Stevens helped run additional pilot studies. Lahiru Jayatilaka helped build aid design tool prototypes. Wendy Mackay provided insight into video prototyping and theories of participatory design. iv Contents Acknowledgments iv 1 Introduction and Background: Checklist Aids 2 1.1 Abstract.................................. 2 1.2 Thesis contributions . 3 1.3 Dissertation roadmap . 4 1.4 ChecklistsinMedicineandAviation. 5 1.5 Medicine&Aviation: Complex,High-Risk . 5 1.5.1 Medical Errors and Preventable Adverse Events . 6 1.5.2 Volatile, Uncertain, Complex, Ambiguous (VUCA) . 7 1.5.3 Aviationvs.Medicine ...................... 7 1.6 Checklists: Previous Work . 9 1.6.1 Aviation.............................. 9 1.6.2 SpaceTravelandNuclearPower. 9 1.6.3 ChecklistsinAviationvs. Medicine . 9 1.6.4 Medical Checklists . 10 1.7 MedicalAids:Theory&Benefits . 11 1.8 CognitiveAidsinMedicine: InPractice. 13 1.9 DigitalAidsinMedicine ......................... 15 1.10 ChecklistsTell“What’sImportantNow” . 15 1.11 Software Aids Could Emphasize What’s Important . 16 1.12 DynamicProcedureAids. 17 v 2 Designing for Complex High-Risk Procedures 22 2.1 Chapter Overview . 22 2.2 Checklist Challenges . 22 2.2.1 Checklists: Promise vs. Performance . 22 2.2.2 Learning from, and Adapting from Aircraft Checklists . 23 2.2.3 Checklist Development and Deployment . 25 2.2.4 ImplementationandAdoption . 25 2.2.5 Evaluating Checklist E↵ectiveness . 25 2.3 UserObservation ............................. 26 2.3.1 AnesthesiaCrisisResourceManagement . 26 2.3.2 Teaching Methodology . 26 2.3.3 OperatingRoom(OR)anesthesiology. 27 2.3.4 Example ACRM Scenarios . 28 2.3.5 Crisis Resource Management . 28 2.4 Participatory Design: Process and Key Concepts . 31 2.4.1 Design Process . 31 2.4.2 Design Experience: Four Key Problems . 32 2.4.3 ReadyAccess(Problem1) . 32 2.4.4 RapidAssimilation(Problem2) . 35 2.4.5 ProfessionalAcceptance(Problem3) . 38 2.4.6 LimitedAttention(Problem4) . 41 2.4.7 iCogAid .............................. 43 2.4.8 DesignVignette.......................... 44 2.5 dpAid:Systemdesign .......................... 45 2.5.1 Humanfactors .......................... 45 2.5.2 Interaction design . 45 2.5.3 Informationdesign ........................ 45 2.5.4 Visualdesign ........................... 45 2.5.5 Technical Implementation . 45 vi 3 Crisis Attention: Analysis and Design 47 3.1 Chapter Overview . 47 3.2 Checklists: Time and Attention . 48 3.2.1 Checklist Settings A↵ect Usage Criteria and Patterns . 49 3.2.2 Checklist Attention: a formative gaze coding and pattern analysis 52 3.3 RapidRead: Step-at-a-Glance Crisis Checklists . 55 3.3.1 Introduction: Designing for Discretionary Use . 55 3.3.2 Chapter Contributions . 57 3.3.3 Design Patterns for RapidRead Checklists . 59 3.3.4 Dynamic Focus Balances Simplicity and Complexity . 59 3.3.5 Object-Action Language Provides Brevity & Structure . 61 3.3.6 Information Patches Aggregate Related Content . 63 3.4 Experiment1: Answer-timeMeasurement . 64 3.4.1 Method . 65 3.4.2 Procedure . 68 3.4.3 Results . 71 3.4.4 Discussion . 73 3.4.5 Heat-maps:Background . 74 3.4.6 TroubleshootingCognitiveAids . 75 3.4.7 Experiment 3: Structure Reduces Variance . 79 3.4.8 FurtherAnalysis ......................... 80 4 Crisis Attention: Evaluation 85 4.1 Chapter Overview . 85 4.2 Medical Simulation . 86 4.2.1 History............................... 86 4.2.2 LearningfromAviation. 87 4.2.3 Medical Education and Training . 87 4.2.4 Paradigm . 87 4.2.5 ResearchunderSimulation. 88 4.2.6 Costs and Benefits . 88 vii 4.3 SimulationParadigms .......................... 89 4.3.1 High-vs.Low-Fidelity. 89 4.3.2 Tradeo↵sandTechniques..................... 89 4.3.3 Medium-FidelitySimulations . 90 4.4 NarrativeSimulation ........................... 90 4.4.1 Approach ............................. 91 4.4.2 Benefits . 91 4.4.3 Drawbacks . 91 4.4.4 Future Work . 92 4.5 DynamicProcedureAids: AnEvaluation . 92 4.5.1 Method . 92 4.5.2 Procedure . 95 4.5.3 StatisticalAnalysisandDataCleaning . 96 4.5.4 Results . 97 4.5.5 Discussion: BenefitsofDynamicAids . 99 4.5.6 DynamicProcedureAids. 101 5 Discussion and Future Work 104 5.1 Chapter Overview . 104 5.2 TheFutureRoleofProcedureAids . 105 5.2.1 ChecklistUsebyExpertandNoviceUsers . 105 5.2.2 Checklist Errors . 105 5.2.3 Social E↵ectsofAidUse..................... 106 5.2.4 Checklist Compliance and Big Data . 106 5.2.5 Whatcanwelearnfromdrivingaids?. 107 5.3 GeneralizingDynamicAids . 107 5.3.1 DynamicProcedureAids: Abstractions . 107 5.4 Conclusion . 108 5.4.1 Looking Forward . 109 5.5 Impact . 111 5.6 Thanks . 111 viii .1 Experiment 1 Questions . 116 ix List of Tables 1.1 Contrasting properties of routine cognitive skill (like much office work) and complex, high-risk procedures (like surgery).. 6 3.1 Task and Checklist Setting . 51 3.2 UsagePatternsandWorkPractices . 51 5.1 GeneralizingProcedureAids . 108 x List of Figures 1.1 Medical team in simulation center: an example of distributed action and attention (photo cc-by Stanford EdTech) . 8 1.2 WHOsurgicalsafetychecklistforroutineuse. 10 1.3 Bradycardia aid in hospital setting, photographed by Dr. Larry Chu . 13 1.4 ACLS Pulseless Electrical Activity cognitive aid (Stanford 2011) . 14 1.5 ACLS Pulseless Electrical Activity aid (Stanford AIM lab, 2011) . 19 1.6 A gallery of five aid styles. Note di↵erences in visual and structural design . 20 1.7 ACLS Pulseless Electrical Activity aid (Stanford HCI software aid 2012) 20 1.8 Medical Doctor uses dpAid system in medical simulation center . 21 2.1 Laerdal SimMan 3G (left) at the Immersion Learning Center: Stanford Li-Ka Shing . 27 2.2 Crisis team responds in a simulated medical emergency at Stanford LKSC ................................... 28 2.3 In a control room, simulation pioneer Dr. David Gaba observes and directs. (photo cc-by Stanford EdTech) . 29 2.4 Doctor references aid in simulation (photo cc-by Stanford EdTech) . 30 2.5 Dynamic Procedure Aid for VT & VFib . 32 2.6 The four key issues; their induced design shifts, and proposed solution components . 33 2.7 Example OR layout . 34 2.8 Doctor refers to digital aid (iCogAid prototype) on large-screen display 35 2.9 DynamicProcedureAiddisplayoncrashcart . 36 xi 2.10 This checklist [Ziewacz 2011] exemplifies how static information pre- sentation can be hard to skim during crisis response . 37 2.11 Early (iCogAid) design, note timeline, dock, vitals display, stock (blood) 41 2.12 ProcedureAidarchitectureandmirroring. 46 3.1 Stanford Emergency Manual binder in use (photo courtesy of Stanford Simulation Group) . 50 3.2 Gaze times by location in simulated crisis . 53 3.3 Ranked list of gaze times . 54 3.4 [DynamicFocus] Treatment aid progresses from dosing atropine to con- sider: transcutaneous pacing or infusions . 60 3.5 Semiformal instructions . 62 3.6 Information patches highlighted: procedures, drugs, and objects (left) 64 3.7 Asystole/Pulseless Electrical Activity aid, style comparison: Standard Text, Structured Text, Color Block, Pictographic, Dynamic Focus. 65 3.8 Standard Text, PEA . 66 3.9 Structured Text, PEA . 67 3.10 ColorBlock, PEA . 68 3.11 PictographicAid,PEA. 69 3.12DynamicAid,PEA............................ 70 3.13 Answer times: means (seconds) by style. The symbol indicates ± coefficient of variation, defined as the standard deviation divided by the mean . 71 3.14 Fittedlog-normaldistributionofanswertimes . 72 3.15 Answertimemeans(sec)bystyle+1stdevbars . ..