Dynamic Procedure Aids Support Crisis Attention

DYNAMIC PROCEDURE AIDS SUPPORT CRISIS ATTENTION

ADISSERTATION SUBMITTED TO THE DEPARTMENT OF COMPUTER SCIENCE AND THE COMMITTEE ON GRADUATE STUDIES OF STANFORD UNIVERSITY IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY

Leslie Wu December 2014

This work is licensed under a Creative Commons Attribution- Noncommercial 3.0 United States License. http://creativecommons.org/licenses/by-nc/3.0/us/

This dissertation is online at: http://purl.stanford.edu/xm083hr3885

ii I certify that I have read this dissertation and that, in my opinion, it is fully adequate in scope and quality as a dissertation for the degree of Doctor of Philosophy.

Pat Hanrahan, Co-Adviser

I certify that I have read this dissertation and that, in my opinion, it is fully adequate in scope and quality as a dissertation for the degree of Doctor of Philosophy.

Stuart Card, Co-Adviser

I certify that I have read this dissertation and that, in my opinion, it is fully adequate in scope and quality as a dissertation for the degree of Doctor of Philosophy.

Scott Klemmer

Approved for the Stanford University Committee on Graduate Studies. Patricia J. Gumport, Vice Provost for Graduate Education

This signature page was generated electronically upon submission of this dissertation in electronic format. An original signed hard copy of the signature page is on file in University Archives.

iii Acknowledgments

Much appreciation for my colleague, Jesse Cirimele, who jointly carried out much of the work described in this thesis. Thanks to my Ph.D. thesis committee, Stuart K. Card, Pat Hanrahan, Terry Winograd, Scott Klemmer, and Larry Chu. They all provided invaluable feedback and guidance over the years. In fact, many of the good ideas and approaches in this thesis were, no doubt, bubbled up in the brain of Stuart K. Card. The Stanford Anesthesia Informatics Media (AIM) Lab, including Drs. Larry Chu, Kyle Harrison and Sara Goldhaber-Fiebert, Anna Clemenson, graciously o↵ered us laboratory space and equipment. Dr. David Gaba and the Li-Ka Shing Center for Immersive and Learning-Based Simulation let us observe medical simulations over the course of many months. Both Drs. Larry Chu and Kyle Harrison provided the initial motivation for the work, and contributed greatly at all stages of the design process. CURIS students Kristen Leach, Justin Lee, Tanya Yu, Kyle M. Barrett, and Katherine Chen spent hours prototyping visual and interaction designs, interaction designs, and helped to run several of the studies described. Jon Bassen, Ti↵any Dharma and Nicholas Stevens helped run additional pilot studies. Lahiru Jayatilaka helped build aid design tool prototypes. Wendy Mackay provided insight into video prototyping and theories of participatory design.

iv Contents

Acknowledgments iv

1 Introduction and Background: Checklist Aids 2 1.1 Abstract...... 2 1.2 Thesis contributions ...... 3 1.3 Dissertation roadmap ...... 4 1.4 ChecklistsinMedicineandAviation...... 5 1.5 Medicine&Aviation: Complex,High-Risk ...... 5 1.5.1 Medical Errors and Preventable Adverse Events ...... 6 1.5.2 Volatile, Uncertain, Complex, Ambiguous (VUCA) ...... 7 1.5.3 Aviationvs.Medicine ...... 7 1.6 Checklists: Previous Work ...... 9 1.6.1 Aviation...... 9 1.6.2 SpaceTravelandNuclearPower...... 9 1.6.3 ChecklistsinAviationvs. Medicine ...... 9 1.6.4 Medical Checklists ...... 10 1.7 MedicalAids:Theory&Beneﬁts ...... 11 1.8 CognitiveAidsinMedicine: InPractice...... 13 1.9 DigitalAidsinMedicine ...... 15 1.10 ChecklistsTell“What’sImportantNow” ...... 15 1.11 Software Aids Could Emphasize What’s Important ...... 16 1.12 DynamicProcedureAids...... 17

v 2 Designing for Complex High-Risk Procedures 22 2.1 Chapter Overview ...... 22 2.2 Checklist Challenges ...... 22 2.2.1 Checklists: Promise vs. Performance ...... 22 2.2.2 Learning from, and Adapting from Aircraft Checklists . . . . . 23 2.2.3 Checklist Development and Deployment ...... 25 2.2.4 ImplementationandAdoption ...... 25 2.2.5 Evaluating Checklist E↵ectiveness ...... 25 2.3 UserObservation ...... 26 2.3.1 AnesthesiaCrisisResourceManagement ...... 26 2.3.2 Teaching Methodology ...... 26 2.3.3 OperatingRoom(OR)anesthesiology...... 27 2.3.4 Example ACRM Scenarios ...... 28 2.3.5 Crisis Resource Management ...... 28 2.4 Participatory Design: Process and Key Concepts ...... 31 2.4.1 Design Process ...... 31 2.4.2 Design Experience: Four Key Problems ...... 32 2.4.3 ReadyAccess(Problem1) ...... 32 2.4.4 RapidAssimilation(Problem2) ...... 35 2.4.5 ProfessionalAcceptance(Problem3) ...... 38 2.4.6 LimitedAttention(Problem4) ...... 41 2.4.7 iCogAid ...... 43 2.4.8 DesignVignette...... 44 2.5 dpAid:Systemdesign ...... 45 2.5.1 Humanfactors ...... 45 2.5.2 Interaction design ...... 45 2.5.3 Informationdesign ...... 45 2.5.4 Visualdesign ...... 45 2.5.5 Technical Implementation ...... 45

vi 3 Crisis Attention: Analysis and Design 47 3.1 Chapter Overview ...... 47 3.2 Checklists: Time and Attention ...... 48 3.2.1 Checklist Settings A↵ect Usage Criteria and Patterns . . . . . 49 3.2.2 Checklist Attention: a formative gaze coding and pattern analysis 52 3.3 RapidRead: Step-at-a-Glance Crisis Checklists ...... 55 3.3.1 Introduction: Designing for Discretionary Use ...... 55 3.3.2 Chapter Contributions ...... 57 3.3.3 Design Patterns for RapidRead Checklists ...... 59 3.3.4 Dynamic Focus Balances Simplicity and Complexity ...... 59 3.3.5 Object-Action Language Provides Brevity & Structure . . . . 61 3.3.6 Information Patches Aggregate Related Content ...... 63 3.4 Experiment1: Answer-timeMeasurement ...... 64 3.4.1 Method ...... 65 3.4.2 Procedure ...... 68 3.4.3 Results ...... 71 3.4.4 Discussion ...... 73 3.4.5 Heat-maps:Background ...... 74 3.4.6 TroubleshootingCognitiveAids ...... 75 3.4.7 Experiment 3: Structure Reduces Variance ...... 79 3.4.8 FurtherAnalysis ...... 80

4 Crisis Attention: Evaluation 85 4.1 Chapter Overview ...... 85 4.2 Medical Simulation ...... 86 4.2.1 History...... 86 4.2.2 LearningfromAviation...... 87 4.2.3 Medical Education and Training ...... 87 4.2.4 Paradigm ...... 87 4.2.5 ResearchunderSimulation...... 88 4.2.6 Costs and Beneﬁts ...... 88

vii 4.3 SimulationParadigms ...... 89 4.3.1 High-vs.Low-Fidelity...... 89 4.3.2 Tradeo↵sandTechniques...... 89 4.3.3 Medium-FidelitySimulations ...... 90 4.4 NarrativeSimulation ...... 90 4.4.1 Approach ...... 91 4.4.2 Beneﬁts ...... 91 4.4.3 Drawbacks ...... 91 4.4.4 Future Work ...... 92 4.5 DynamicProcedureAids: AnEvaluation ...... 92 4.5.1 Method ...... 92 4.5.2 Procedure ...... 95 4.5.3 StatisticalAnalysisandDataCleaning ...... 96 4.5.4 Results ...... 97 4.5.5 Discussion: BeneﬁtsofDynamicAids ...... 99 4.5.6 DynamicProcedureAids...... 101

5 Discussion and Future Work 104 5.1 Chapter Overview ...... 104 5.2 TheFutureRoleofProcedureAids ...... 105 5.2.1 ChecklistUsebyExpertandNoviceUsers ...... 105 5.2.2 Checklist Errors ...... 105 5.2.3 Social E↵ectsofAidUse...... 106 5.2.4 Checklist Compliance and Big Data ...... 106 5.2.5 Whatcanwelearnfromdrivingaids?...... 107 5.3 GeneralizingDynamicAids ...... 107 5.3.1 DynamicProcedureAids: Abstractions ...... 107 5.4 Conclusion ...... 108 5.4.1 Looking Forward ...... 109 5.5 Impact ...... 111 5.6 Thanks ...... 111

viii .1 Experiment 1 Questions ...... 116

ix List of Tables

1.1 Contrasting properties of routine cognitive skill (like much oce work) and complex, high-risk procedures (like surgery)...... 6

3.1 Task and Checklist Setting ...... 51 3.2 UsagePatternsandWorkPractices ...... 51

5.1 GeneralizingProcedureAids ...... 108

x List of Figures

1.1 Medical team in simulation center: an example of distributed action and attention (photo cc-by Stanford EdTech) ...... 8 1.2 WHOsurgicalsafetychecklistforroutineuse...... 10 1.3 Bradycardia aid in hospital setting, photographed by Dr. Larry Chu . 13 1.4 ACLS Pulseless Electrical Activity cognitive aid (Stanford 2011) . . . 14 1.5 ACLS Pulseless Electrical Activity aid (Stanford AIM lab, 2011) . . . 19 1.6 A gallery of ﬁve aid styles. Note di↵erences in visual and structural design ...... 20 1.7 ACLS Pulseless Electrical Activity aid (Stanford HCI software aid 2012) 20 1.8 Medical Doctor uses dpAid system in medical simulation center . . . 21

2.1 Laerdal SimMan 3G (left) at the Immersion Learning Center: Stanford Li-Ka Shing ...... 27 2.2 Crisis team responds in a simulated medical emergency at Stanford LKSC ...... 28 2.3 In a control room, simulation pioneer Dr. David Gaba observes and directs. (photo cc-by Stanford EdTech) ...... 29 2.4 Doctor references aid in simulation (photo cc-by Stanford EdTech) . . 30 2.5 Dynamic Procedure Aid for VT & VFib ...... 32 2.6 The four key issues; their induced design shifts, and proposed solution components ...... 33 2.7 Example OR layout ...... 34 2.8 Doctor refers to digital aid (iCogAid prototype) on large-screen display 35 2.9 DynamicProcedureAiddisplayoncrashcart ...... 36

xi 2.10 This checklist [Ziewacz 2011] exempliﬁes how static information presentation can be hard to skim during crisis response ...... 37 2.11 Early (iCogAid) design, note timeline, dock, vitals display, stock (blood) 41 2.12 ProcedureAidarchitectureandmirroring...... 46

3.1 Stanford Emergency Manual binder in use (photo courtesy of Stanford Simulation Group) ...... 50 3.2 Gaze times by location in simulated crisis ...... 53 3.3 Ranked list of gaze times ...... 54 3.4 [DynamicFocus] Treatment aid progresses from dosing atropine to consider: transcutaneous pacing or infusions ...... 60 3.5 Semiformal instructions ...... 62 3.6 Information patches highlighted: procedures, drugs, and objects (left) 64 3.7 Asystole/Pulseless Electrical Activity aid, style comparison: Standard Text, Structured Text, Color Block, Pictographic, Dynamic Focus. . . 65 3.8 Standard Text, PEA ...... 66 3.9 Structured Text, PEA ...... 67 3.10 ColorBlock, PEA ...... 68 3.11 PictographicAid,PEA...... 69 3.12DynamicAid,PEA...... 70 3.13 Answer times: means (seconds) by style. The symbol indicates ± coecient of variation, defined as the standard deviation divided by the mean ...... 71 3.14 Fittedlog-normaldistributionofanswertimes ...... 72 3.15 Answertimemeans(sec)bystyle+1stdevbars ...... 73 3.16 Heatmap of participants’ gaze: modified pictographic Bradycardia aid 75 3.17 Heatmap of modified Bradycardia aid, Dynamic style ...... 76 3.18 HeatmapofmodifiedBradycardiaaid,Textstyle ...... 77 3.19 A comparison of mean answer times (seconds) of questions from each aid style to Standard (plotted along y=x)...... 78 3.20 Heatmap of modified Bradycardia aid, Structured style ...... 79

xii 3.21 Answer times (seconds) by trial (phase I data). Each circle corresponds to a single, timed answer response...... 81 3.22 Average answer times (y-axis) decrease as trial numbers increase (x- axis). Non-dotted lines correspond to each style, dotted lines denote trend lines (phase I data only) ...... 82 3.23 Learning curve/e↵ect, averaged over all styles ...... 83

4.1 Medical students attend lecture at the Li-Ka Shing Center for Knowl- edge (photo cc-by Stanford EdTech) ...... 88 4.2 Participants using Dynamic Procedure Aids responded correctly sig- niﬁcantly more often than those using paper aids or no aid ...... 98

5.1 Base image: simulated crisis with dpAid superimposed ...... 110 5.2 Proposed heads-up display for crisis response: iPad integration . . . . 111 5.3 Proposed heads-up display for crisis response: people in room . . . . 112 5.4 Proposed heads-up display for crisis response: compression aid . . . . 113 5.5 Patient history on HUD: iPad plus large-screen display ...... 113 5.6 Proposed heads-up display for crisis response: procedure aid . . . . . 114

xiii Contents

1 Chapter 1

Introduction and Background: Checklist Aids

1.1 Abstract

Medical checklists can improve performance in the volatile, uncertain, complex, and ambiguous domain of emergency medicine [Wu et al. 2014]. Such procedure aids are only now being adopted, in paper form, in hospitals and clinics across the world [Gawande 2009]. Software-based aids o↵er an opportunity to improve upon the ef- fectiveness of static, paper aids, since digital aids a↵ord interactivity, high-speed distribution, and dynamic content. However, software-based medical checklists must work seamlessly in uncertain, time-pressured scenarios. In these team-based, multitasking environments, a users attention is limited. These attentional aspects of crisis computing—supporting highly trained teams as they respond to real-life emergencies—have been underexplored in the user interface community. This dissertation describes research that begins to address these challenges with design prototyping, constraint analysis, engineering, simulation, and evaluation. This dissertation describes an interactive software system, dpAid, that supports medical teams in action by displaying relevant checklist aids. dpAid helps doctors

2 CHAPTER 1. INTRODUCTION AND BACKGROUND: CHECKLIST AIDS 3

follow Advanced Cardiac Life Support (ACLS) protocol, as they respond to emergency codes such as heart attacks in operating rooms or hospital wings. The design of this system was based on 18 months spent observing Stanford medical residents responding to simulated crises in high-ﬁdelity medical simulators with realistic patient mannequins. Idevelopandanalyzeusers’time-andattention-constraints,leadingtofour design concepts: shared displays for ready access, step-at-a-glance for rapid assimilation, resources-at-a-glance for professional acceptance, and attention aids for limited attention. I present a simulation/evaluation approach, narrative simulation that comparatively assesses expert performance through a score-and-correct approach. I present data on the e↵ectiveness of dynamic and interactive aids for supporting doctors as they respond to simulated ACLS scenarios, suggesting that the software- based dpAid system e↵ectively supports crisis attention and performance. More broadly these experimental results suggest that the Dynamic Procedure Aid (dpAid) approach can improve user performance in complex, high-risk domains.

1.2 Thesis contributions

This thesis contributes to the understanding of doctors’ attention during time-limited, stressful situations, and of how to design dynamic procedure aids for these scenarios. Speciﬁcally, it unpacks the four key design problems of ready access, professional acceptance, and limited attention. I introduce the Dynamic Procedure Aid (dpAid) approach, involving shared displays for access, step- and resource-at-a-glance, and attention aids. To evaluate and understand the e↵ectiveness of this approach, I introduce the narrative simulation method and apply it simulated Advanced Cardiac Life Support (ACLS) scenarios. I presents data gathered from over 150 medical simulations and laboratory experiments involving close to ﬁfty medical doctors, students, residents, fellows and professionals. This process revealed the importance of time constants and limited attention, leading to experimental work involving eye trackers and gaze analysis. CHAPTER 1. INTRODUCTION AND BACKGROUND: CHECKLIST AIDS 4

Broadly speaking, I attempt to answer four research questions. Q1: Analyze and model the role of attention in crisis response: Can we bracket upper- and lower-bounds for gaze periods? Q2: What are e↵ective design principles for crisis software? Q3: How do we evaluate interfaces for limited-time & attention domains? Q4: How e↵ective is the dpAid system at supporting crisis attention and improving performance in simulated scenarios?

1.3 Dissertation roadmap

Chapter 1: Checklists in Medicine and Aviation

This chapter introduces the background material relevant to the use of checklists in medicine and aviation. Checklist beneﬁts and barriers are enumerated. The chapter ends by describing the opportunities available for software-based aids for medicine.

Chapter 2: Designing for Complex High-Risk Procedures

Medicine often involves Complex High-Risk Procedures such as surgery and code blue calls, where responders treat in-hospital cardiac arrests. This chapter describes the design process of dpAid, a system for Dynamic Procedure Aids, involving co-design with medical doctors and iterative prototyping and medical simulation. I present four key design concepts distilled from this work: Ready Access, Rapid Assimilation, Professional Acceptance, and Limited Attention.

Chapter 3: Crisis Attention: Analysis and Design

In such complex, high-risk scenarios, users’ time and attention are severely limited. This chapter provides a gaze and attentional analysis as part of a large understanding of the problem domain of crisis attention in medicine. Given this analysis, it provides the Dynamic Procedure Aid (dpAid) design approach involving at-a-glance displays and dynamic focus+context techniques. It concludes with a medical answer time CHAPTER 1. INTRODUCTION AND BACKGROUND: CHECKLIST AIDS 5

study, where medical professionals are asked questions and are timed as they use aids to answer these questions.

Chapter 4: Crisis Attention: Simulation and Evaluation

Prototyping and evaluating software on live patients in real hospitals can be costly in many ways. Medical simulation has been used to train new doctors and recertify existing ones. We propose the use of simulators for evaluating novel software paradigms and describe the beneﬁts and drawbacks of such an approach. This chapter introduces the narrative simulation approach and applies it to the evaluation of Dynamic Procedure Aids.

Chapter 5: Discussion and Future Work

Many factors are involved in the deployment and adoption of checklists and aids beyond their ecacy in simulation. This chapter describes barriers for professional acceptance. It discusses broader issues of team attention and shared displays, leading to implications for the design of general software for attention-limited domains. I describe a vision for the future of attention in ubiquitous computing and how this thesis serves as a stepping stone towards that goal.

1.4 Checklists in Medicine and Aviation

This chapter introduces medicine as an example of a complex high-risk domain, and describes previous work in managing such complexity with checklists.

1.5 Medicine & Aviation: Complex, High-Risk

Medicine and aviation are two examples of complex and high-risk domains with highly trained users. In both ﬁelds, doctors and pilots undergo years of schooling followed by decades of experience before they are entrusted with large numbers of lives, whether through patients or passengers. CHAPTER 1. INTRODUCTION AND BACKGROUND: CHECKLIST AIDS 6

In medicine, these complex, high-risk procedures may take the form of drawn-out surgeries or rapid response code blue calls, where responders treat in-hospital cardiac arrests. These procedures are at the edge of tractable complexity [Patterson 2007; Rochlin et al. 2005], and errors are easy to make. Acute / crisis care & emergency medicine is arguably a new frontier for HCI. Un- like single-user oce work, crisis medicine is paced, operated under risky emergency conditions, involve multi-tasking with people and equipment, and is generally team- based. Table 1.1 contrasts the properties of routine cognitive skill work and complex, high-risk procedures.

OFFICE WORK SURGERY Routine cognitive skill Complex high-risk procedures Learning: Few hours of task training required 10,000+ hours of training Errors: Errorsmostlytrivial Errorprone/Errorintolerant Time: Onetaskatatime;Unpaced Multitasked;Pacedcrisis Actor: Individual Team

Table 1.1: Contrasting properties of routine cognitive skill (like much oce work) and complex, high-risk procedures (like surgery).

Complex procedures require intricately coordinated multi-tasking, are team-based, and strongly time-paced. Errors are easy to make, yet high-risk environments are severely unforgiving of even small errors.

1.5.1 Medical Errors and Preventable Adverse Events

Medical errors and adverse events are challenging to avoid given systematic complexity. An estimated 400,000 deaths per year in U.S. hospitals are associated with preventable harm, and serious complications may be ten to twenty times more common [James 2013]. Managing the risk of medical error and increasing adherence to best practices can result in safer health systems and improved patient safety [Dekker 2011, Kohn, Corrigan, and Donaldson 2000]. CHAPTER 1. INTRODUCTION AND BACKGROUND: CHECKLIST AIDS 7

1.5.2 Volatile, Uncertain, Complex, Ambiguous (VUCA)

Previous work on high-reliability organizations has investigated the way that such organizations deal with volatile, uncertain, complex, ambiguous (VUCA) tasks, whether in medicine [Patterson 2007], aviation [Rochlin et al. 2005], or the military [Paparone, Anderson, and McDaniel 2008]). These organizations have developed methods of training, practice, and reﬂection that have helped their practitioners operate at high levels of safety and e↵ectiveness. One risk-management technique for VUCA tasks is the use of simulation. As part of a training curriculum, simulation has been commonplace in aviation education for decades, and in other ﬁelds for even longer. Pilots must log thousands of simulator hours before they graduate to real planes. A longer discussion of the history of simulation follows in Chapter 4. Two other important techniques include the formal instruction of crew resource management (CRM) through classes [Gaba et al. 2001], and mandatory use of checklists in both routine and emergent situations.

1.5.3 Aviation vs. Medicine

Aviation and medicine are examples of high-risk, high-tempo domains [Degani and Wiener 1990] which can place great demands on users’ cognitive facilities [Gaba, Fish, and Howard 1994]. However, when compared to aviation, medical crisis care is more extreme in its co-located team size [Sarcevic, Marsic, and Burd 2012], which can range from a single doctor to a dozen or more team members responding to a cardiac arrest “code blue”. Medical crisis care is also characterized by a multiplicity of information sources and controls, distributed widely throughout the environment [Sarcevic, Marsic, and Burd 2010]. This interaction complexity extends from the physical to the social—while responding to a code, doctors and their team members may be spread out through a room (Figure 1.1), and arrive at very di↵erent times. In contrast, aviation pilots are seated next to each other with similar controls. Whereas in aviation, pilots and co-pilots train and are educated similarly, medical team members may come from very di↵erent work-practice cultures. For example, CHAPTER 1. INTRODUCTION AND BACKGROUND: CHECKLIST AIDS 8

Figure 1.1: Medical team in simulation center: an example of distributed action and attention (photo cc-by Stanford EdTech)

medical team members may use di↵erent language and terms to describe the same patient condition. Since medical professionals play di↵erent roles in a crisis setting, they may have di↵ering views on teamwork and safety culture [Mills, Neily, and Dunn 2008]. Thus, interface design for medical crisis care represents an extreme point in the space of interaction contexts, even when compared to aviation. It o↵ers a number of challenging applications, such as collaborative work and decision support under stress. Crisis care demands multi-user, multi-role interactions, and requires usable human-computer interaction techniques that span devices distributed throughout a ﬂuid, information-rich environment. CHAPTER 1. INTRODUCTION AND BACKGROUND: CHECKLIST AIDS 9

1.6 Checklists: Previous Work

Checklists have the opportunity for tremendous impact in many domains [Gawande 2009, Gaba, Fish, and Howard 1994]. We describe a few important domains below, and summarize previous ﬁndings.

1.6.1 Aviation

Whereas maverick pilots initially embraced danger and complexity, organizations in aviation eventually adopted and mandated checklist use by all commercial pilots [Gawande 2009]. Since then, checklist use has been shown to improve performance in aviation [Boorman 2001; Burian, Barshi, and Dismukes 2005; Degani and Wiener 1990]. Thus pilots are trained with checklists in simulation and use them regularly. Aircraft pilots use checklists for both routine operation (taking-o↵and landing) [Degani and Wiener 1990] and abnormal events (crashing in the ocean) [Burian, Barshi, and Dismukes 2005]. These checklists may be paper-based, or digital and interactive. Researchers have developed guidelines on the importance of human factors and performance in the design of aircraft checklists [Federal Aviation Administration 1995]. One important design factor (see Chapter 3.3.2) is the e↵ective use of typography in checklists and ﬂight-deck documentation [Degani 1992].

1.6.2 Space Travel and Nuclear Power

Checklists are also used in space travel and nuclear power operation. NASA continues to investigate the role of design in checklists [Degani and Wiener 1990], and more general operating documents.

1.6.3 Checklists in Aviation vs. Medicine

In contrast to aviation and other domains, where checklists have been commonplace for decades, their adoption has been slower in medicine due to cultural and complexity reasons [Gawande 2009; Winters et al. 2009]. Checklist usage can a↵ect the social CHAPTER 1. INTRODUCTION AND BACKGROUND: CHECKLIST AIDS 10

Figure 1.2: WHO surgical safety checklist for routine use hierarchy (Chapter 5) and doctors may stigmatize the use of aids as incompetence, or as having to rely a handicap (Chapter 5). There may also be a gap between perceived value and perceived cost (Chapter 4).

1.6.4 Medical Checklists

Studies have demonstrated the e↵ectiveness of checklists in surgical care [Haynes et al. 2009; Makary et al. 2006], operating room crises [Arriaga et al. 2013; Ziewacz et al. 2011], and rare medical events [Harrison et al. 2006]. These checklists have been studied for both routine (expected) tasks and emergent events which may not happen every time. Routine tasks may include putting in central lines [Pronovost et al. 2006] or stopping for pre-surgery time-outs (Figure 1.2) in operating rooms (ORs) [Makary et al. 2006]. CHAPTER 1. INTRODUCTION AND BACKGROUND: CHECKLIST AIDS 11

Introducing a routine checklist into Michigan hospitals decreased infection rates by 66%. This simple change saved more than 1500 lives, and about $175 million, in the ﬁrst 18 months of deployment [Pronovost et al. 2006]. Across a diverse set of multiple hospital sites, checklists led major complications from surgery to drop 36, and deaths to fall by 47 percent [Gawande 2009; Haynes et al. 2009]. Research into how static checklists may be made dynamic and interactive is limited. This thesis begins to address how dynamic and interactive aids could improve their e↵ectiveness.

1.7 Medical Aids: Theory & Beneﬁts

Preventable adverse events can be categorized as

Errors of commission, • Errors of omission, • Errors of communication, • Errors of context, and • Diagnostic errors • [James 2013]. In general these errors are not often caused by lack of skill, but, on the individual level, by cognitive overload [Coiera, Tombs, and Clutton-Brock 1996; Gaba, Fish, and Howard 1994]. Checklists are one strategy to manage this overload. Checklists are an example of a cognitive aid in medicine or other ﬁelds [Chu and Fuller 2011a]. This thesis instead uses the term procedure aid rather than cognitive aid or checklist. The term procedure aid focuses on how external aids can assist users while they perform tasks, whether to increase speed or reduce the risk of error. As procedure aids, checklist augment individual and team cognition. Medical aids can help direct attention [Burden et al. 2012], o↵-load demands on short term CHAPTER 1. INTRODUCTION AND BACKGROUND: CHECKLIST AIDS 12

memory [Federal Aviation Administration 1995] and support long term memory when diagnoses are rare [Harrison et al. 2006]. Used in teams, procedure aids such as checklists may help ground communication [Clark and Brennan 1991] by providing a shared referent. They can improve team coordination, by supporting distributed cognition [Kirsh 2005]. When visible to more than one person, they may encourage a shared mental model, which has been shown to be associated with improved team performance [Mathieu et al. 2000]. Reading aids aloud makes explicit what is underway, encouraging situational awareness [Harrison et al. 2006]. Institutionally, checklists can standardize procedures and terminology, improving protocol adherence even in crisis [Ziewacz et al. 2011; Arriaga et al. 2013]. As a physical or digital artifact, they are an important “vessel of safety culture” [McConnell, Fargen, and Mocco 2012]. That is, simply introducing a checklist may not result in improved outcomes or performance. Instead they must be deployed as part of a larger whole: whether through systems of quality control and process improvement, or via host organizations and cultures that champion and operationalize their use. Summarizing, checklists can serve as

adefensestrategytopreventhumanerrors • a memory and attention aid to enhance task and team performance • standardization of tasks to facilitate coordination and hand-o↵s • ameanstocreateandmaintainasafetyculture • amechanismforprocessimprovementandmeasurement • support for quality control by management, government and inspectors • [Degani and Wiener 1990; Degani 1992; Verdaasdonk et al. 2009]. CHAPTER 1. INTRODUCTION AND BACKGROUND: CHECKLIST AIDS 13

Figure 1.3: Bradycardia aid in hospital setting, photographed by Dr. Larry Chu

1.8 Cognitive Aids in Medicine: In Practice

What do these aids look like in practice? Cognitive aids may be written by hand (Figure 1.3), or printed (Figure 1.4). Note that the handwritten aid (Figure 1.3) is placed less accessibly on a wall, with simple reminders for important diagnostic approaches. In contrast, the cognitive aid pictured (Figure 1.4) is part of a ring-bound set of aids developed by and used at the Stanford VA hospital. This card-sized set of aids is closer in size to reference guides commonly carried by medical professionals for Advanced Cardiac Life Support. A newer iteration (Figure 1.5), developed by the Stanford AIM lab is available as CHAPTER 1. INTRODUCTION AND BACKGROUND: CHECKLIST AIDS 14

Figure 1.4: ACLS Pulseless Electrical Activity cognitive aid (Stanford 2011)

aspiralboundpaperedition,withaidsprintedonbothsides[ChuandFuller2011a]. The cover page shows a relevant index, and each cognitive aid spans a single page or consecutive pages. Later in this thesis, I describe several aid design variations (Figure 1.6). CHAPTER 1. INTRODUCTION AND BACKGROUND: CHECKLIST AIDS 15

1.9 Digital Aids in Medicine

The Stanford AIM lab set of aids is now available in a digital form, through the Apple app store and iPad tablets. A binder is replaced by a button with drop-down menu, and scrolling replaces page turning. The digital version seems to o↵er myriad possible beneﬁts. Updates to aids can be published and distributed quickly and easily. Additional information could be displayed on demand. Figure 1.7 displays a mock-up such a possible interface, displaying medical protocol information as well as information about relevant services and electronic medical records. One might dream that such software would positively impact medical team e↵ectiveness and patient outcomes. However, the history of information technology in medicine has shown that designing, developing, and deploying software for the medical ﬁeld is a challenging endeavor. While software is hard, generally, medical software faces a number of addied barriers compared to other domains that involve less user risk. This includes legal hurdles, technology and evaluation cycles on the order of decades, and large chasms between theoretical advances and practical adoption. Given the added degree of user risk, information displays may be misleading or easy to misinterpret [Thimbleby 2013], leading to potentially dangerous results. In the case of the Therac-25 radiation therapy device [Leveson 1993], interface design choices and programming errors combined unexpectedly, resulting in several patient deaths.

1.10 Checklists Tell “What’s Important Now”

Checklists o↵er a way to help manage attention in time-limited domains, encouraging users to attend to matters they would otherwise forget about. They do so by exter- nalizing knowledge in a way that can be acted upon quickly, in the form of a simple list or aid. Checklists triage attention—letting readers quickly prioritize. They do so since checklists designers must decide what steps to include and what to omit. CHAPTER 1. INTRODUCTION AND BACKGROUND: CHECKLIST AIDS 16

In Intervention, Dan John describes three important tools for every situation and setting: checklists, rituals, and deliberate practice [DanJohn2012]. In the context of medicine, checklists take the form of procedure and cognitive aids. Rituals— “checklists come alive” in John’s world—are embodied in patterns such as the pre- surgery time-out. Deliberate practice comes in the form of simulation exercises focused narrowly and speciﬁcally, paired with expert, rapid feedback. John explains that the more we do the same thing, the more we will slip on the basics. Checklists, then, shift focus from “anything” to “what’s important now.” John writes: “The very act of making a checklist demands that you—

Tell me everything you need • Tell me what’s really important.” •

1.11 Software Aids Could Emphasize What’s Im- portant

While paper checklists are valuable, they are static, slow to access, and show both too much and too little information. Thus they poorly manage their own information overload—some paper aids are poster sized and measure by the yard—and their own information scarcity. A poster may contain a large amount of useful information, but in a time-critical setting, there may not be time to ﬁnd the important sections. A binder of aids may contain four (the AIM lab’s ACLS emergency aids) or a dozen, as in the digital StanMed iPad “Crisis Aids” or the OR Critical Events Check- lists set [Ziewacz et al. 2011]. However, the WHO international disease classiﬁcation system lists 13,600 diagnoses, 6000 drugs, and 4000 medical and surgical procedures [World Health Organization 2005]. Clearly, paper by itself may not easily scale, at least if access and attention time is a concern. Software can scale in many ways, but perhaps with checklists it can best help in narrowing attention and focusing action. If making checklists means deciding both “everything you need” and yet chosing only “what’s really important” then software CHAPTER 1. INTRODUCTION AND BACKGROUND: CHECKLIST AIDS 17

could help individuals and teams have access to the former (everything you need) while focusing on the latter (what’s important now).

1.12 Dynamic Procedure Aids

In the following chapters, I analyze the problem domain—how might we design aids for complex high-risk procedures? Based on observation and co-design, I introduce technology prototypes and describe potential beneﬁts of digital, software-based aids in medicine, to reduce errors, manage risk, and improve patient safety and outcomes. There is notable evidence that, with organizational support, well-designed procedure aids can help manage the complexity and increase the safety of medical procedures [Haynes et al. 2009; De Vries et al. 2011; Haynes et al. 2011]. How can we reliably harvest and amplify this potential? Toward this end, this thesis presents the following contributions:

A16-monthparticipatorydesigninvestigationwithanesthesiologyteams,high- • lighting intervention opportunities for di↵erent roles, and the cost-beneﬁt appeal of integrating checklists with other information resources.

Dynamic Procedure Aids that generalize checklists to expand their beneﬁts and • lower their costs. They comprise four key design concepts:

– shared displays for ready access, – step-at-a-glance for rapid assimilation, – resources-at-a-glance for professional acceptance, and – attention aids for limited attention.

dpAid, a software system for crisis medicine that manifests the key concepts of • Dynamic Procedure Aids. The dpAid architecture provides dynamic, software- based cognitive aids across multiple, mirrored displays (Figure 1.8). CHAPTER 1. INTRODUCTION AND BACKGROUND: CHECKLIST AIDS 18

The dpAid system employs shared displays, introducing the techniques of step-at- a-glance and resources-at-a-glance, and emphasizes importance of showing only what matters in the present moment. I close with a discussion of how these ﬁndings and techniques may generalize to other domains beyond the important case of medicine, foreshadowing future work in supporting such domains with wearable computing. CHAPTER 1. INTRODUCTION AND BACKGROUND: CHECKLIST AIDS 19

Figure 1.5: ACLS Pulseless Electrical Activity aid (Stanford AIM lab, 2011) CHAPTER 1. INTRODUCTION AND BACKGROUND: CHECKLIST AIDS 20

Figure 1.6: A gallery of ﬁve aid styles. Note di↵erences in visual and structural design

Figure 1.7: ACLS Pulseless Electrical Activity aid (Stanford HCI software aid 2012) CHAPTER 1. INTRODUCTION AND BACKGROUND: CHECKLIST AIDS 21

Figure 1.8: Medical Doctor uses dpAid system in medical simulation center Chapter 2

Designing for Complex High-Risk Procedures

2.1 Chapter Overview

This chapter frames the following problem: how might we design usable and e↵ective aids for complex and dangerous procedures in medicine? It describes an analysis of the problem and unpacks various design dimensions, from time to resources, visual to information design.

2.2 Checklist Challenges

2.2.1 Checklists: Promise vs. Performance

Chapter 1 described how checklist aids can improve performance in high-risk domains such as medicine. The performance of checklists does not, however, always live up to these promises. Checklist use can interrupt workﬂow. Finding and searching through checklists induces additional time, attentional demand, and complexity [McConnell, Fargen, and Mocco 2012]. Ironically, given that checklists are designed to help, some medical professionals perceive them as an externally imposed barrier or disruption.

22 CHAPTER 2. DESIGNING FOR COMPLEX HIGH-RISK PROCEDURES 23

Thomassen et al. note that “Despite the increasing use of checklists in healthcare worldwide, few studies have explored personnel experiences in using this new tool.” [Thomassen et al. 2010] Those that have ﬁnd barriers such as the checklist slowing down or otherwise disrupting the procedure [Fourcade et al. 2012]. This perceived interference—and cultural skepticism—has slowed checklist adoption by medical teams [Gawande 2009; Winters et al. 2009]. As Verdaasdonk et al. put it, “Time governs willingness and compliance in the use of checklists.”

2.2.2 Learning from, and Adapting from Aircraft Checklists

Medical Interaction Contexts

Though medical checklists draw inspiration from aviation, there are important di↵erences between the domains, especially in team composition and work. In aviation, the physical ergonomics are static and highly regulated. Aircrews sit in cockpits where controls and displays are co-designed and co-located [Hutchins 1995]. By contrast, in operating rooms, the sensors, information displays, and interaction points are spread throughout the environment [Mentis et al. 2012; Sarcevic, Marsic, and Burd 2010]. In routine care, medical doctors are beginning to adopt electronic health records but during a crisis, information technology use is typically limited to interacting with patient monitors, such as vitals displays and surgery-speciﬁc views [Gaba, Fish, and Howard 1994].

Complexity

Medical checklists have additional challenges. Human bodies are complex and treatment methods may not be as linear as checklists for engineered processes [Gaba, Fish, and Howard 1994]. As we have already noted in Chapter 1, the vast number of medical and disease conditions also increases complexity. These di↵erences in complexity, and work practice imply that new protocols, norms, and best practices must be developed for medical checklists. CHAPTER 2. DESIGNING FOR COMPLEX HIGH-RISK PROCEDURES 24

Social & Team Concerns

Cockpit crews work in small teams of two or three, and typically have similar back- grounds. Hospital crisis care teams may comprise surgeons, anesthesiologists, phar- macists, nurses, technicians, and other specialists, arranged around the patient, each with their own cultures, roles, and equipment. These di↵erences in education and work-practice translate into challenges for the adoption of technology such as paper checklists [Winters et al. 2009]. They may respond in small teams (2) or large, in the case of trauma teams (15+). Not only must sta↵work under time pressure, risk, and uncertainty, but they must cope with the coordination and communication complexity inherent in team-based crisis care [Hunziker et al. 2011]. These complexities lead to breakdowns in e↵ective crisis care: missed steps, timing errors, lack of a shared mental model, and poor resource management [Gaba, Fish, and Howard 1994]. Checklists are potentially awaytomitigateandrecoverfromthesebreakdowns,buttheymustbecarefully implemented or they could make things worse.

Who Checks the Checklist?

Within a medical team, checklist use and roles are still evolving. Who should ask for, hold, read, or otherwise use checklists in these settings? Are they simply a pedagogical tool for medical students and residents, or should fellows and all doctors feel obligated to pull out aids? Some groups have experimented with a reader role for checklists, where a team member is explicitly tasked with bringing out and reading a checklist aloud [Burden et al. 2012]. However, there is not yet a consensus about best practices.

Crutch or clutch?

Arelatedconcernrelatestode-skillingof(hopefully)expertperformance.Doaids serve as cognitive crutches that impair learning and memory, or “clutch” instruments that save lives and improve quality of care? We discuss these issues further in Chapter 4. CHAPTER 2. DESIGNING FOR COMPLEX HIGH-RISK PROCEDURES 25

2.2.3 Checklist Development and Deployment

Checklists are valuable, but they can add their own costs. Designing them them to be both e↵ective and ecient is a task with some subtleties. Even Gawande, one of checklists’ foremost promoters, noted the usability failure of his ﬁrst attempt to make a viable checklist [Gawande 2009]. Little is known empirically about how to e↵ectively design medical checklists, or how to build systems and tools that enable organizations to source, tailor, and deploy checklists systematically.

2.2.4 Implementation and Adoption

One ﬁnal challenge is that evaluating checklist interventions in situ requires substan- tial time, money, and political capital. Furthermore, interventional researchers may have conﬂicting agendas with other hospital departments, and reporting of successful interventions must be carefully scrutinized before being widely adopted. If implemented, will checklists simply sit unused? Doctors have consistently rejected decision support tools, from simple decision trees to more complex computer- based systems.

2.2.5 Evaluating Checklist E↵ectiveness

Finally, how might checklist e↵ectiveness be evaluated? In the United States, HIPAA and other regulation make recording private health information (PHI) tricky to im- plement. Many health organizations do not collect systematic performance metrics, and designing multi-site randomized controlled trials with technological interventions is extremely challenging. Furthermore, little tool support and research has gone into developing paradigms for evaluating checklist e↵ectiveness in simulation and clinical care. CHAPTER 2. DESIGNING FOR COMPLEX HIGH-RISK PROCEDURES 26

2.3 User Observation

Doctors spend an extreme number of hours spent in training, to develop expert performance [Ericsson and Lehmann 1996]. Given the depth of knowledge users have in this CHiRP domain, we begin with user observation and co-design with medical doctors, developing insight into the problem space.

2.3.1 Anesthesia Crisis Resource Management

As part of the design process, I engaged in participatory design work with doctors from the Stanford School of Medicine. We held regular meetings, and over 18 months, Ihadachancetoobserve50hoursofmockcrisisscenariosinhigh-ﬁdelitymedical simulation facilities (Figure 2.1) at the Li-Ka Shing Center and the VA Palo Alto. These were organized as part of an on-going Anesthesia Crisis Resource Management (ACRM) class for ﬁrst, second, and third year Stanford anesthesia residents [Gaba et al. 2001]. A typical ACRM simulation day may have involved 4-5 simulations in a row, with debriefs following each simulation. In each ACRM scenario, 1-2 anesthesia residents were in the “hot seat”, with one resident starting in the room and, typically, another resident waiting in the wings. The second resident would serve as backup when called. During each simulation, residents treat mannequin-based patients and act as team leaders, coordinating tasks and managing teams of medical professionals. These professionals were typically 3-5 confederate actors: doctors, nurses, technicians and patients played by doctors, nurses, technicians and patients (Figure 2.2).

2.3.2 Teaching Methodology

After each simulation session, participants debrief with clinical instructors and fellow students who observe remotely. Some instructors and confederates watch from behind a one-way mirror (Figure 2.3). Others watch a live video feed. Recordings are kept and stored securely, and may be used to bring up talking points during a debrief. CHAPTER 2. DESIGNING FOR COMPLEX HIGH-RISK PROCEDURES 27

Figure 2.1: Laerdal SimMan 3G (left) at the Immersion Learning Center: Stanford Li-Ka Shing

2.3.3 Operating Room (OR) anesthesiology

Our observation focused on the practice of Operating Room anesthesiology. OR anesthesiologists are responsible for managing emergent events during peri-operative patient care, which includes care pre- and post-surgery. Anesthesiologists are trained to recognize and respond to emergencies, and assume the role of crisis team leader when appropriate. Like pilots, anesthesiologists prepare for the beginning of surgery (takeo↵), keep an eye on the controls during the procedure (ﬂight), and monitor its completion and initial recovery (landing). Their job has been characterized as having hours of boredom punctuated by moments of terror [Gaba 2007]. CHAPTER 2. DESIGNING FOR COMPLEX HIGH-RISK PROCEDURES 28

Figure 2.2: Crisis team responds in a simulated medical emergency at Stanford LKSC

2.3.4 Example ACRM Scenarios

Anesthesia residents must deal with expected and unexpected events in medical simulation. These may include misconﬁgured equipment, common [Ziewacz et al. 2011] and rare medical emergencies [Harrison et al. 2006], with patients that are sometimes conscious, sometimes unconscious. In one scenario, a Kobayashi Maru1-style simulation, the patient dies no matter what. The real test comes when the patient’s wife comes and the residents must tell her, the actor, that her husband has died.

2.3.5 Crisis Resource Management

As part of these classes, the key points taught [Gaba et al. 2001] are:

1In Gene Roddenberry’s Star Trek, the Kobayashi Maru test was an unwinnable training exercise set before cadets in Starfleet academy. In a battlefield bridge simulation, a starship crew receives a distress call from a civilian ship in the Klingon Neutral Zone. Cadets must either chose to enter the Neutral Zone and risk provoking interstellar war, or abandon the ship leaving its crew to certain death. The test then is not to fight valiantly but to see how crews and would-be captains grapple with a no-win scenario. CHAPTER 2. DESIGNING FOR COMPLEX HIGH-RISK PROCEDURES 29

Figure 2.3: In a control room, simulation pioneer Dr. David Gaba observes and directs. (photo cc-by Stanford EdTech)

Call for Help Early • Anticipate & Plan • Distribute Workload • Allocate Attention Wisely • Communicate E↵ectively • Mobilize all Available Resources • Designate Leadership • These resources include both people in the room as well as cognitive aids. CHAPTER 2. DESIGNING FOR COMPLEX HIGH-RISK PROCEDURES 30

Figure 2.4: Doctor references aid in simulation (photo cc-by Stanford EdTech)

In this chapter we describe how the dpAid system can help promote these ACRM principles, with an emphasis on the use of procedure aids.

Errors in Simulation

In these classes, we observed a number of errors, both individual and team. For example, some individual errors observed included

change-blindness / inattentional blindness • incorrect diagnosis • get treatment algorithms wrong • lose time (late) • CHAPTER 2. DESIGNING FOR COMPLEX HIGH-RISK PROCEDURES 31

2.4 Participatory Design: Process and Key Con- cepts

2.4.1 Design Process

The participatory design work spanned 16 months and generated more than 60 dif- ferent prototypes at various ﬁdelities. A large fraction of these prototypes were non- interactive but narrative prototypes developed in Apple Keynote by myself, Jesse Cirimele, and Stanford CURIS undergraduates. In general, we held weekly design reviews with 3-4 computer scientists and 2-3 doctors or medical professionals. These took place in meeting rooms as well as in situ in medical simulation facilities..

Design Prototyping

Iinitiallyexploredtablet-based,general-purposeORdesigns.Toevaluatedesigns in a concrete domain, we focused on Advanced Cardiac Life Support (ACLS) [Chu and Harrison 2012], because it is important, widespread, and required for professional certification. The prototypes expressed four stages in our thinking. Initial prototypes were tablet-based. However, reading required walking to the tablet, and only one person could see it. A second set of prototypes added large, mirrored displays. Mindful of acceptance issues, these prototypes integrated doctor-requested resources, such as blood availability, test results, and vitals with history. Doctors reacted negatively to the resource-rich displays, which showed too much. Instead, doctors wanted fast-to-assimilate information. The third series of prototypes, therefore, simplified the display. While successful, this lead to fragmented information, obscuring the larger picture. Therefore the final prototype series used attention-reactive, focus+context techniques to dynamically shift the detail on the display (Figure 2.5). CHAPTER 2. DESIGNING FOR COMPLEX HIGH-RISK PROCEDURES 32

Figure 2.5: Dynamic Procedure Aid for VT & VFib

2.4.2 Design Experience: Four Key Problems

2.4.3 Ready Access (Problem 1)

Observed Needs/Problem: Invisible Resources Go Unused

Open-loop communication, misaligned mental models, and “invisible” work cause many medical crisis errors [Parush et al. 2011]. Common examples include requests without a speciﬁed recipient, lack of acknowledgement, and/or lack of follow-up. For example, “we need to get the crash cart” rather than “Jon can you call for the crash cart”, Jon—“yes I will call for the crash cart”. We observed one doctor inform another of an important change in patient vitals. The other doctor failed to hear, but the ﬁrst did not notice. As a result, neither realized they held di↵erent mental models of the situation. This is exacerbated at large hospitals, where team members commonly do not know all their colleagues’ names. Shared artifacts like paper aids [Harrison et al. 2006] and whiteboards [Xiao et al. CHAPTER 2. DESIGNING FOR COMPLEX HIGH-RISK PROCEDURES 33

Figure 2.6: The four key issues; their induced design shifts, and proposed solution components

2001] facilitate medical coordination. However, poor ergonomics and static content discourage use. We observed that doctors responding to crises would start using paper aids until another task required attention. Then, they would put the aid on a ﬂat surface, where it would invariably get covered and never picked up again. Other times, doctors would hold a binder of aids in one hand, without a convenient way to make it visible and accessible to others. Furthermore, doctors’ use varied. Given identical scenarios, some doctors never picked up the aid, others looked at it once, and others made personal and/or public use of its information. Consequently, the aids were often invisible, hidden physically, or held by only one team member.

Prototype Experience

The patient’s body provides an important coordination focus [Kyng, Nielsen, and Kristensen 2006], and dpAid’s design evolved in turn. Early prototypes featured a single wall-mounted display. However, because surgical teams often circle around CHAPTER 2. DESIGNING FOR COMPLEX HIGH-RISK PROCEDURES 34

Figure 2.7: Example OR layout the patient, any single location had blind spots. During the training exercise, the participants did not use the display as much as initially expected. In debriefs and walking through the operating room, the doctors reported that the screen was not always easy to see depending on where they were standing or where they were looking. To address this, we added a second, mirrored display (Figure 2.7). These displays can be permanently mounted, and/or wheeled in on emergency “crash carts”.

Emergent Concept: Shared Displays

We hypothesize that large, shared displays (Figure 2.8) can improve awareness and visibility. They provide a consistent physical location, legible from most locations, CHAPTER 2. DESIGNING FOR COMPLEX HIGH-RISK PROCEDURES 35

Figure 2.8: Doctor refers to digital aid (iCogAid prototype) on large-screen display

supporting common ground [Clark and Brennan 1991]. Research in aviation [Mathieu et al. 2000] and medicine [Manser et al. 2009] has connected team performance and asharedmentalmodel.Sharedvisualreferentstotheprocedure,itsstate,andthe resources involved may increase the shared understanding.

2.4.4 Rapid Assimilation (Problem 2)

Static aids are too slow to rapidly absorb. We simplify text and focus the interface more concisely.

Problem: Too Much Information, Much Too Slow

Checklists must be fast to use by someone who is attending to something else. Check- lists are rejected when they are slow to use and compete with time and attention CHAPTER 2. DESIGNING FOR COMPLEX HIGH-RISK PROCEDURES 36

Figure 2.9: Dynamic Procedure Aid display on crash cart needed for the patient [Verdaasdonk et al. 2009; Winters et al. 2009]. Conceptually, it is useful to distinguish rare procedures from common ones. For rare events, checklists provide new or poorly recalled information. Here, checklists must be easy to read. By contrast, for common events, checklists cover routine and familiar material and serve as a reminder to not skip steps or make assumptions too quickly. Here, checklists should be easy to skim, and remind e↵ectively. In between, checklists are used to look-up or confirm a fact, such as a drug dosage. In all cases, checklist aids must work well as part of a multi-tasking workflow. To support rapidly shifting visual attention, steps must be fast to find and re-find if the doctor looks away to attend to their main tasks.

Emergent Concept: Step-at-a-Glance

Ausefulwayofdesigningformulti-taskingistoestimateatypicaltimeintervalduring which the dominant task can be neglected and to design steps of the secondary task so that they can be completed in this turn length [Green 1999]. I introduce the CHAPTER 2. DESIGNING FOR COMPLEX HIGH-RISK PROCEDURES 37

Figure 2.10: This checklist [Ziewacz 2011] exempliﬁes how static information presentation can be hard to skim during crisis response step-at-a-glance concept that information artifacts should be designed so steps can be assimilated in one glance. This chunking speeds use and facilitates attentional shifts when needed. Our participatory design led to three techniques that reduce the time of assimilating a step.

Prototype Experience: Focus on current context

In reviewing prototypes, doctors preferred clear, simple presentation of the current step, even when that sacriﬁced peripheral information. Like turn-by-turn map directions, the whole screen can be focused on the current protocol step, simultaneously increasing relevant information and reducing cognitive load. While paper is restricted to a static display, software can emphasize currently needed information, such as a CHAPTER 2. DESIGNING FOR COMPLEX HIGH-RISK PROCEDURES 38

speciﬁc treatment. Information that has already been used, is not yet needed, or provides additional explanation can be minimized by default and expanded if necessary. This approach expands the focus+context layout strategy to procedural documents.

Object/Action checklist language

Early checklists were presented as full sentences with little visual structure. These were slow to read and scan. Because checklists have a highly-constrained structure, visual design can carry more of the information load and improve usability [Chu and Fuller 2011a]. To continue in this vein, we extracted the basic procedural structure from written descriptions and represented it graphically as appropriate. Increas- ing visual structure and shortening text speeds reading and improves scanning. We designed a stylized language for re-expressing medical procedures in a compressed object/action format. This language, loosely inspired by aircraft conﬁguration checklists [Degani 1992], reduces the number of words in a check-list, sometimes by as-much-as half. Whenever possible, each step begins with an object followed by an action or state setting to be achieved for the object. For example, the steps

Increase FiO2 to 100% Verify ischemia with 12 lead EKG if possible can be re-expressed as FiO : 100% 2 " Ischemia: Verify (Use 12-lead EKG) We further exploit structure by listing the object to the left, in larger, bold type. This leads to a consistent information mapping between content and form. dpAid expands the focus steps to reveal additional details. Collectively, these treatments seek to increase speed for the several types of procedure reading: direct reading, skimming, and searching.

2.4.5 Professional Acceptance (Problem 3)

We incentivize use through a shift towards resource management. CHAPTER 2. DESIGNING FOR COMPLEX HIGH-RISK PROCEDURES 39

Problem: Bridging the Gap between Promise and Adoption

In addition to increasing speed and reducing error, checklists should foster a safety culture, supporting quality-control and coordination through standardization [De- gani and Wiener 1993; Verdaasdonk et al. 2009]. How-ever, highly-skilled professionals rarely welcome the over-sight implied by standardization, despite improved outcomes. Consequently, checklists are underused because some perceive an unfavor- able cost:beneﬁt ratio or an unwelcome restriction on professional autonomy. Even in aviation, where checklists are standard, excess checklists reduce compliance [Hales and Pronovost 2006]. Medical professionals seek better, timely, resource and personnel information [Bardram, Hansen, and Soegaard 2006]. About 39% of surveyed anesthesiologists admitted to having made errors due to lack of medical information found in handbooks, and 74% reported a need for real-time medical knowledge at least monthly [Perel et al. 2004]. We believe that improving adoption requires tack- ling these issues head on: reduce the usage costs, expand and emphasize beneﬁts to practitioners.

Emergent Concept: Resource-at-a-Glance

Often, current information resources impede tight collaboration rather than encourage it. In one simulation we observed, a resident pulled out a smartphone to search for information about a competing diagnosis: malignant hyperthermia vs. thyroid storm. Because the form factor of the information was ill-suited for the device and task, he spent about 5 minutes out of a 20-25 minute crisis reading his device. This illustrates both the importance and the diculty of considering multiple options with current OR resources. A varying set of information needs arise mid-crisis, and require rapid access. Di↵erential diagnoses are one resource that should be at teams’ ﬁngertips; physical and personnel resources are another. To address these perceived and actual cost:beneﬁt problems, dynamic aids reframe checklists as a centerpiece of an integrated resource view. Doctors trained in crisis resource management are taught to use resources such as cognitive aids and people in the room. To support di↵erential diagnoses, aids prominently feature a Signs section CHAPTER 2. DESIGNING FOR COMPLEX HIGH-RISK PROCEDURES 40

at the top of each checklist that includes Consider suggestions for common competing diagnoses. To foreground available resources, dpAid shows pictures, names and roles of the current team and those on their way, and physical supplies like blood that can be requested. Integrated resource visibility may improve decision-making and team communication, because as participants gather information, they look to the same screen. This glanceable view of common resources seeks to lower the activation energy for acquiring information, facilitate serendipitous reminding, and create the habit of more frequently consulting these resources.

Prototype Experience

Balance simplicity and amount of information. Early designs included nearly every piece of information that participants suggested, and consequently su↵ered from clutter. This led to a display where, in principle, everything was available but in practice little was ﬁndable. Technical, information rich domains face this tension. The challenge was exacerbated by the wall-scale form factor, which requires legibility at a distance. A lesson learned repeatedly was that clear presentation of less information was much more important than displaying all potentially useful information. Of course, availability must be balanced against overload. Some of our early prototypes presented a laundry list of information resources desired by the doctors (Figure 2.11), including: inventories of blood, medicine, and other supplies available; the expected time to availability of laboratory tests; patient identiﬁcation, medical record highlights and images, procedure site, and plan; and names and roles of the operating team. When the medical team members saw all of this information together during design reviews, they very reasonably found it to be overwhelming. To balance the access/overload tension, the revised dpAid design shows some of this information only when relevant or on required. CHAPTER 2. DESIGNING FOR COMPLEX HIGH-RISK PROCEDURES 41

Figure 2.11: Early (iCogAid) design, note timeline, dock, vitals display, stock (blood)

2.4.6 Limited Attention (Problem 4)

Attention is narrow and limited under stress. We design the aid as an attention regulator, employing dynamic focus+context. A deeper analysis of attention, time- pressure, and usage constraints can be found in chapter 3.

Problem: Complex Setting Fragments Team Attention

Crisis response is attention-limited [Takahashi, Kojima, and Okada 2011]. Co-located teams work across multiple surfaces, on interdependent tasks. For example, anesthesiologists may split visual attention between the patient, vitals and a drug vial they are preparing, while simultaneously ensuring that others continue high-quality CPR. Medical personnel must re-orient physically to attend cognitively and socially. This physically-distributed attention [Srinivasan et al. 2009] di↵ers from desktop and mobile work, complicating software design. CHAPTER 2. DESIGNING FOR COMPLEX HIGH-RISK PROCEDURES 42

Prototype Experience: Alerts as a Hook

Administering recurring drugs provides a frequent and important example. Frenetic pacing and multiple responsibilities cause teams to miss doses, or forget prior doses and redose too often. Some ORs rely completely on memory, other ORs use clip- boards or whiteboards. Precisely timed attention to multiple activities is dicult for people, but easy for software. We saw timers as potentially a clear, high-value draw that would engender broader use. In participatory design sessions, the doctors were extremely enthusiastic about integrating timers and other reminders. How might dpAid e↵ectively present these alerts in the chaotic context of the operating theater? We initially explored audio alarms, as they are agnostic to physical orientation. However, ORs are extremely noisy: during routine operation, music combines with device alerts, chatter, and work-related discussion. Anesthesiologists may be listening to a surgeon while requesting an arterial blood gas, peripherally listening to O2 saturation, but ignoring a false alarm from a di↵erent machine. Medical alarms are unregulated, so tones, volume, and frequency are as varied as the device manufactur- ers. Crises make matters worse: though chatter dissipates and music is turned o↵, the number and frequency of genuine and false alarms increases dramatically, as does the speed and volume of communication. Consequently, “demanding” attention through audio is often fruitless and possibly detrimental. However, medical professionals (like pilots) are trained to cycle rapidly through relevant displays they are monitoring, and avisualalertcanbereadyforthemwhentheydo. Checklist users can regulate progress more or less tightly, depending on the situation. In aviation, electronic checklists for routine operation (pre-ﬂight) sometimes mandate step-by-step armation, called the READ-DO method [Gawande 2009]. However, the required speed of crisis response makes this unworkable—marking items siphons time away from handling crises. Medical doctors are not alone in resisting lockstep adherence. Pilots mostly use the READ-CONFIRM method of performing several items from memory, then consulting the checklist to see if they missed anything. Professionals modulate their care in response to challenge and risk. To mitigate the hazards of transoceanic ﬂight and engender greater diligence, pilots often employ the CHAPTER 2. DESIGNING FOR COMPLEX HIGH-RISK PROCEDURES 43

more cautious READ-DO. We hypothesize that enabling ﬂexibility increases adoption. However, because people underestimate risk, safe checklists should make adherence easy and fast.

Key Concept: Attention Aid

Given these complexities, the design shifted from checklists as attention regulators to checklists as attention aids. To foreground current state, speed the path to action, and re-duce errors, dpAid provides context-specific drug timers and alternate diagnoses to consider. The timers embed a dose and countdown at the relevant aid step, concentrating relevant information where it is needed (see Figure 1). Sugges- tions such as “consider...” flag similar diagnoses and diagnoses the current condition may evolve into. These suggestions lower the cost of switching to another aid and discourage fixation on initial diagnosis, a common issue under duress [Burian 2006; Gaba et al. 2001]. Like the timers, dpAid places these suggestions within the aid at the relevant action step.

2.4.7 iCogAid

IpresentiCogAid,whichembodiesthedesignconceptsofaccessandacceptance. iCogAid is an interactive software system which o↵ers interactive cognitive aids and dynamic checklists, whose displays are mirrored across tablets and large-screen displays. iCogAid was implemented as a HTML5 web application which synchronizes state via WebSockets, to ensure low latency communication overhead. The web architecture allows portable rendering across di↵erent form factors, such as large screens mounted on the wall and wheeled in on crash carts. It facilitates the authoring of checklists and cognitive aids using checkML, a markup language designed for the task domain. I tested iCogAid in several high-ﬁdelity medical simulation scenarios, operating the interactive system as a confederate participant in simulations with doctors and nurses. Based on feedback from these sessions, we further identiﬁed the importance of fast assimilation, and developed new prototypes which more drastically reduced information CHAPTER 2. DESIGNING FOR COMPLEX HIGH-RISK PROCEDURES 44

displayed and introduced the “step-at-a-glance” approach previously described. This resulted in the next-generation system called dpAid for Dynamic Procedure Aids.

2.4.8 Design Vignette

The dpAid system embodies these design shifts to proactively aid attention and support a rich, shared mental model across a medical team. It facilitates adoption by serving as a resource management system and reduces load through selective emphasis and rapid-read checklists. Here is an example of how dpAid might be used in practice. Katherine is a resident anesthesiologist at a large hospital. She is paged to help an emergent event during a routine surgery. Entering the OR, she sees the crash cart next to the patient, with a deﬁbrillator and mounted large-screen display. As she approaches her colleague Justin, he reports that they have a 65-year-old patient who came in for laparoscopic knee surgery. They both look at dpAid, which displays patient information and a personnel roster. As they review the patient’s vitals and history, the patient’s pulse becomes erratic and blood pressure drops. Eventually, the patient is pulseless, resulting in a state of pulseless electrical activity (PEA). Katherine asks a nurse to bring up the PEA aid. dpAid reminds her to switch to 100% oxygen and ventilate at 10 breaths/minute. Katherine moves away and gives epinephrine, triggering an on-screen timer to ensure redosing every 3-5 minutes. Meanwhile CPR begins while Justin monitors compression quality and depth. After these immediate actions, Justin and Katherine begin re-viewing possible causes, such as anaphylaxis or hypovolemia. They rule out several diagnoses quickly and review several other options to consider. Katherine calls for an arterial blood gas, and later notices an important electrolyte abnormality. She uses dpAid to verify these numbers and see what additional resources she can call upon. CHAPTER 2. DESIGNING FOR COMPLEX HIGH-RISK PROCEDURES 45

2.5 dpAid: System design

2.5.1 Human factors

In a code blue situation, it is not uncommon for a dozen sta↵members to enter a hospital room at di↵erent times, and for crash carts and other equipment to be wheeled in and out at will. This creates a complexity of interaction points and attentional sinks.

2.5.2 Interaction design

We gave tablet input to nurses, allowing doctors to give them verbal commands. Noticing the cost of information access, we built an always on, always visible system useful in routine care, transitioning seamlessly when a crisis occurs. Nurses are a good candidate for controlling and updating the display because of a professional inclination to process adherence and functional role in organization and support.

2.5.3 Information design

As described previously, the dpAid system uses dynamic focus+context to limit the amount of information displayed at any one time.

2.5.4 Visual design

To make aids visible across the room, dpAid uses white text on a black background. It intentionally makes minimal use of colors, using color mainly for emphasis. This is a controversial choice, as some guidelines encourage only black (or dark) text on a white (light) background [Gawande 2013].

2.5.5 Technical Implementation

The iCogAid prototype was implemented as a web-based single-page application. The front-end was written in JavaScript with the Backbone.js framework. Display CHAPTER 2. DESIGNING FOR COMPLEX HIGH-RISK PROCEDURES 46

Figure 2.12: Procedure Aid architecture and mirroring was rendered in HTML5 with the use of CSS and the

Future Work

I developed a simple HTML-based markup language for expressing checklists pro- grammatically. However, more work needs to be done to understand how to develop information systems that support these adaptive level-of-detail displays. Also, the dpAid prototype only has one view, regardless of the display device. It may be useful instead to customize displays to the size of the screen, and the user’s specialty. When is it appropriate to mirror screens and interactions, and when do individuals require di↵erent information displays? Chapter 3

Crisis Attention: Analysis and Design

3.1 Chapter Overview

Complex, high-risk domains such as medical crisis response require accurate responses under extreme time constraints. Checklists improve important outcomes in these domains. However, current designs are based largely on intuition; there is little theory or empirical work about designing e↵ective procedure aids. Furthermore, discretionary checklist use is fragmented and bursty rather than predictable and continuous. Working with doctors and studying successful aids, we developed the RapidRead design approach [Cirimele et al. 2014]. It distills three patterns for designing rapidly readable aids: Dynamic Focus, Object-Action, and Information Patches. Two experiments compared medical professionals’ search time, eye-gaze, and retention with alternative checklist designs. Applying RapidRead patterns resulted in signiﬁcantly faster aid usage, reducing answer time and importantly minimizing the frequency of slow responses to medical queries. Contents 3.1 Chapter Overview

3.2 Checklists: Time and Attention

47 CHAPTER 3. CRISIS ATTENTION: ANALYSIS AND DESIGN 48

3.2.1 Checklist Settings A↵ect Usage Criteria and Patterns

3.2.2 Checklist Attention: a formative gaze coding and pattern analysis

3.3 RapidRead: Step-at-a-Glance Crisis Checklists

3.3.1 Introduction: Designing for Discretionary Use

3.3.2 Chapter Contributions

3.3.3 Design Patterns for RapidRead Checklists

3.3.4 Dynamic Focus Balances Simplicity and Complexity

3.3.5 Object-Action Language Provides Brevity & Structure

3.3.6 Information Patches Aggregate Related Content

3.4 Experiment 1: answer-time measurement

3.4.1 Method

3.4.2 Procedure

3.4.3 Results

3.4.4 Discussion

3.4.5 Heat-maps: Background

3.4.6 Troubleshooting Cognitive Aids

3.4.7 Experiment 3: Structure Reduces Variance

3.4.8 Further Analysis

3.2 Checklists: Time and Attention

The successes of procedure aids such as checklists are tempered by the fact that they can slow procedures, increasing complexity and demand on attention [Gawande 2009; McConnell, Fargen, and Mocco 2012; Winters et al. 2009]. As Verdasdonk et al. puts it, “Time governs willingness and compliance in the use of checklists.” [Verdaasdonk et al. 2009] How, then, should these variables of time and attention inform the design of new checklists? CHAPTER 3. CRISIS ATTENTION: ANALYSIS AND DESIGN 49

This chapter presents a formative gaze analysis of how medical doctors use checklists under time pressure. These preliminary results, in concert with previous work, suggest that aids must support fast and non-linear use. I describe a basic design space of checklist type and use. how checklists are used in practice, contrasting everyday and routine checklist use with crisis care and treatment of emergent events. The former is typically predictable, consistent, and linear. The latter, instead, is chaotic, non-linear, and places a higher demand on ease and speed-of-use. To address these concerns of speed, I present and evaluate a design approach for fast aids called RapidRead. Based on a study of existing medical aids, I distill three patterns: (1) Dynamic Focus, (2) Information Patches, and (3) Object-Action Language. Idescribehowtheyhavebeendevelopedincontextandexplainaprincipled approach for each pattern: 1) Dynamic Focus: add detail around current subtask. Minimize or omit other information 2) Information Patches: map knowledge into graphically-deﬁned blocks 3) Object-Action language: codify language structure for brevity and consistency. Current checklists are largely based on expert-driven intuition and iterative testing. There is little theory or empirical work about how to e↵ectively design procedure aids. To address this, I evaluate ﬁve checklist designs, analyzing how they manifest (or not) these three principles. Applying RapidRead patterns results in faster aids, as measured by answer times in a laboratory experiment with medical professionals.

3.2.1 Checklist Settings A↵ect Usage Criteria and Patterns

To better understand how doctors attended to checklists, I observed high-ﬁdelity medical simulations and studied ﬁrst-person training videos to understand patterns of use. As part of these ACRM simulation exercises (described previously in Chapter 2), Stanford University medical anesthesia residents had voluntary access to paper-based checklists in binder form (Figure 3.1). These aids were designed by the Stanford CHAPTER 3. CRISIS ATTENTION: ANALYSIS AND DESIGN 50

Figure 3.1: Stanford Emergency Manual binder in use (photo courtesy of Stanford Simulation Group)

Anesthesia and Informatics Media Lab [Chu and Fuller 2011a]. Initially, I had expected that checklist use by doctors would match my mental model of how airline pilots use checklists. That is, medical teams would stop and iterate through a list of items verbally. A single doctor might either physically check o↵items, or a pair of doctors may use protocols such as challenge-response [Degani and Wiener 1990] or cross-check. In these protocols, one doctor might say a line item out loud, and another doctor conﬁrms that the item, if relevant, was completed. There are cases when checklist use is linear and predictable. For example, when starting a procedure such as an operating room surgery, it has now become common to employ pre-surgery time-outs [Makary et al. 2006]. In a time-out, the whole surgery team stops and verbally walks through a known number of steps. Social protocol is scripted, and focus is “single-threaded,” that is, the team does only one thing at a time. Another example is the checklist for putting in central lines [Pronovost et al. 2006], where an individual linearly checks through a consistent set of items, with no other concurrent tasks. For both these two routine, linear tasks, early research in medical checklists CHAPTER 3. CRISIS ATTENTION: ANALYSIS AND DESIGN 51

demonstrated the e↵ectiveness of simple paper checklists. Table 3.1 and 3.2 describe the characteristics of the settings and usage of these routine, mandatory checklists.

Setting: Routine Crisis Care Everyday (pre- surgery timeout) Emergent (MH or Code Blue) Orderly,Predictable Chaotic,Uncertain “Single- threaded” Focus / Operation Multiple/Concurrent Focus, Operation Less pressure on time and attention More pressure Table 3.1: Task and Checklist Setting

Aid Usage: Mandatory Discretionary Standardized and Scripted Voluntary, Ad hoc Social norms, roles On-the-ﬂy team use, conﬁguration Linear Use (Contiguous) Non-linear Use (Skip & Skim) Long (Minutes), Consistent Short (Seconds), Variable Gaze Patterns Table 3.2: Usage Patterns and Work Practices

In contrast to this linear and often mandatory use, I observed di↵erent patterns in medical simulation exercises. Doctors used aids inconsistently, and sometimes not at all. Although residents were taught to use cognitive aids as a key behavior principle of ACRM, not all did. Design sessions and debriefs held after simulation exercises revealed a gap between checklist use in routine, everyday care and emergent, chaotic, crisis care. In contrast, crisis or trauma care [Sarcevic, Marsic, and Burd 2012] is non-linear, and highly concurrent. Thus, these settings have a correspondingly di↵erent work practice of checklist use (Table 2). Instead of a linear ﬂow, doctors mix usage styles: skimming, skipping sections, looking up information for rare procedures, or answering speciﬁc questions, such as how much Dantrolene to administer [Harrison et al. 2006]. As mentioned in Chapter 2, I investigated the use of digital-based checklists deployed on tablets and large-screen displays. Their usage patterns mirrored that of paper-based aids, that is, usage was discretionary, fragmented, and sometimes minimal. In this case, doctors may opt not to use any aids at all, due to social concerns, work style, or time pressure. CHAPTER 3. CRISIS ATTENTION: ANALYSIS AND DESIGN 52

3.2.2 Checklist Attention: a formative gaze coding and pattern analysis

The attentional aspects of crisis computing—supporting highly trained teams as they respond to real-life emergencies—have been underexplored in the user interface community. Systems that aim to support crisis teams must then intelligently manage attention. To make these claims slightly more concrete, I present a gaze analysis based on the START2011 anesthesia crisis response training video. In this educational video, aclinicalinstructorparticipatesinasimulatedmedicalemergencyinanoperating room. They play the role of an anesthesiologist monitoring a patient during surgery, as a head-mounted camera captures their ﬁrst-person perspective.

Methodology

Approximately 8 minutes of the video was manually coded. Using visibility as a proxy for eye gaze, I identiﬁed six important features in the scene: sta↵, patient, vitals, intravenous lines (IV), crash cart, cognitive aid. Speech actions were transcribed and the relevant actors identiﬁed, along with the time of action. For each time block, I estimated gaze length and target.

Results

The following graph describes the results (Figure 3.2. Each gaze target is assigned adistinctcolor,andtimeprogressesasyougodowntherows.Verticallycontiguous colored blocks represent a roughly consistent gaze focus over that period of time. Again, the point of view is from the anesthesiologist’s.

Discussion

First, note that attention is split, in time-paced environment. In this chapter I focus on this distribution of attention, temporally and spatially. Sorting the gaze-times in decreasing order results in the following graph (Figure 3.3): CHAPTER 3. CRISIS ATTENTION: ANALYSIS AND DESIGN 53

Figure 3.2: Gaze times by location in simulated crisis

Note the decay of gaze times—most gaze times are short, less than 10 seconds. That is, there is a notable tail—almost all looks are less than 10-15 seconds, with a few exceptions at the head of the distribution. CHAPTER 3. CRISIS ATTENTION: ANALYSIS AND DESIGN 54

Figure 3.3: Ranked list of gaze times

Related Work

Attention in Medical Operating Rooms: Monitoring and Readability

In medicine things can quickly go wrong if left unattended. Liu et al. used head- mounted displays in clinical anesthesia settings and found that anesthesiologists “looked at the anesthesia workstation for 3.7 seconds per head turn on average and 7.2 seconds per head turn toward the patient/surgical ﬁeld.” [Liu et al. 2010]. Similar numbers were found later for head turns toward vital sign displays [Kusunoki et al. 2013]. Our user observation of medical doctors in simulation mirrors what other researchers have found [Kusunoki et al. 2013], namely that attention is a limited resource, and often most gaze times are short (less than 10 seconds). This in turn drives the design of interactive displays for medical trauma and crisis response, where researchers have reported the importance of avoiding a glut of information [Kusunoki et al. 2014]. CHAPTER 3. CRISIS ATTENTION: ANALYSIS AND DESIGN 55

Limited Attention

Much work in psychology has investigated the e↵ect of stress and pressure on performance, working memory, and attention. As in aviation, where pilots are taught speciﬁc gaze scan patterns and timings, medical doctors are taught in their specialties to attend to certain elements in a clinical setting, depending on the context. In the medical domain, interruptions are common [Chisholm et al. 2000; Healey, Sevdalis, and Vincent 2006], and doctors must adapt to this style of work.

Summary and Implications for Design

This formative gaze coding suggests the importance of understanding the nature of how doctors attention is distributed. E↵ective interfaces for continuous use may not work when use is bursty and non-linear. A temporal distribution which has a longer tail also implies that one must consider the case when most looks are short, which has implications for the design of interfaces in this domain. More work should be done, with more participants and in di↵ering simulations, to better understand how attention patterns vary based on expertise, team size, and time pressure.

3.3 RapidRead: Step-at-a-Glance Crisis Checklists

3.3.1 Introduction: Designing for Discretionary Use

Earlier, we argued that discretionary checklist use is fragmented and bursty rather than predictable and continuous. Medical professionals must rapidly absorb and process information as they address emergent events. As medical informatics increas- ingly supports doctors in real-time, it becomes more important to e↵ectively design and evaluate medical interfaces for such time-compressed domains, when attention is scarce. To inform design requirements for readable aids, we summarize ﬁndings from three relevant areas of study: information search and the use of vital signs and displays CHAPTER 3. CRISIS ATTENTION: ANALYSIS AND DESIGN 56

in medical settings, attentional patterns of in-car navigation systems, and work on mobile attention.

Previous Work Relates Aid Performance, Usage, and Attention: Medicine and Search

One important design requirement is that interfaces should support rapid, chunkable reading patterns. In externally-paced tasks like driving and surgery, diversion from the primary task impairs performance. Routine monitoring performance degrades and reaction time slows [Monk, Trafton, and Boehm-Davis 2008; Wickens and McCarley 2007]. Long secondary task times also increase the chance of prospective memory errors. [Wickens and McCarley 2007]. Interface speed also matters when use is discretionary. Faster information acquisition speeds increase usage [Kalnikait´eand Whittaker 2007; Verdaasdonk et al. 2009]. Slower Web search times decrease usage over time [Brutlag2009b]. This implies that users may implicitly invoke a cost model for discretionary interface use, informed by perceived knowledge beneﬁt versus perceived acquisition cost [Pirolli 2007].

Navigation Aids: Cars and Aviation

Aviation and car-based navigation are two task domains where users must split time and attention between technological aids and a complex set of other tasks. The concept of divided attention [Iqbal and Bailey 2010] describes this split nature of attention. In aviation, slow and dicult-to-read checklists contribute to accidents [Degani 1992]. When people drive, distraction and o↵-road gaze time correlates with more accidents. Speciﬁcally, the distribution of glance durations has a long tail, and longer glances correlate with longer response times to hazardous events [Horrey and Wickens 2007]. There is also a correlation between higher average glance time and worse driving performance. Researchers have thus suggested a 10-15 second limit for the use of in-car navigation systems [Green 1999]. CHAPTER 3. CRISIS ATTENTION: ANALYSIS AND DESIGN 57

Mobile Attention

More broadly, researchers have investigated attention in HCI since the development of multi-tasking terminals. Recent work has also focused on understanding interruption in computing systems [Iqbal and Bailey 2010]. An interesting comparison can be made with general mobile interaction. Oulasvirta et al. looked at mobile interaction across a range of contexts including walking on busy streets, riding an escalator, and sitting on a bus. They found that average continuous spans of attention ranged from 4 to 14 seconds [Oulasvirta et al. 2005]. For complex navigation situations, such as walking on a busy street, the interaction patterns are quite similar to those found for driving.

Design requirements

Summarizing, I propose the following design requirements for supporting non-linear use of crisis aids: 1) Interfaces must support rapid, chunkable reading 2) When use is discretionary, information acquisition speed matters 3) Readability and speed must ﬁt into time & attentional parameters of the domain and task Interfaces that support these requirements also may be used for routine and linear care. In fact, faster interfaces in those domains may also lead to increased usage.

3.3.2 Chapter Contributions

Previous work in aviation has suggested the importance of typography and visual design in authoring checklists that are easy and fast to read [Degani 1992]. A checklist for (medical) checklists [Gawande 2013] posits guidelines such as: -sansserif,darkonalightbackground -“fewerthan10itemsperpausepoint” -ﬁtononepage -“simple,uncluttered,andlogicalformat” CHAPTER 3. CRISIS ATTENTION: ANALYSIS AND DESIGN 58

However, these guidelines have not been empirically validated, and not all successful checklists adhere to these heuristics. For example, many aids used at the Stanford hospital span two pages, sometimes even three. When used with a binder, two pages may be a manageable way to show more information, if they are both visible on adjacent pages. Almost no prior work has compared alternative layout styles or form factors, and no existing guidelines have been empirically tested. This chapter focuses on the impact of design, and contributes: (1) a formative set of comparative measurements of information ﬁnding tasks, and (2) a set of design patterns found in these checklists that empirically improve performance on these tasks.

RapidRead

This chapter introduces RapidRead design principles. Based on these principles I develop Structured Text aids, static aids based on the content from the Standard Text aids [Ziewacz et al. 2011]. Then, the first experiment compares five alternative checklist presentation styles from the literature: Standard Text, Structured Text, Color Block [Chu and Fuller 2011a], Pictographic [Chu and Harrison 2012], and Dynamic Focus. These five aids support Advanced Cardiac Life Support (ACLS) crisis response. In awithin-subjectsstudy,medicalparticipants(n=13)respondedfastestwithDynamic Focus aids. Eye-tracking analysis showed the importance of clear visual navigation paths, anchors, and rapid scanning. A second experiment compared Dynamic Focus aids to a new design that applied the RapidRead principles to the Dynamic aids. This revision further reduced performance variation. We discuss reasons for these benefits, reflect on performance and eye-tracking data, and suggest future work. CHAPTER 3. CRISIS ATTENTION: ANALYSIS AND DESIGN 59

3.3.3 Design Patterns for RapidRead Checklists

Given the goal of creating crisis checklist aids that are fast to use, I present design principles and patterns. These principles distill and extend strategies used in existing aids, principles derived from human perception and multitasking research, and insights from participatory design. Some principles were briefly sketched in the prior chapter. These were refined and improved after experimentally analyzing their usage and e↵ectiveness. These techniques are designed to increase the speed of information search in procedure aids. The design concept of fitting a checklist or cognitive aid step into a multi-tasking cycle we call a step-at-a-glance user interface.

RapidRead Design Concept: Approaches

The RapidRead design concept combines three approaches:

Dynamically add detail around the current step while reducing it elsewhere with • dynamic focus;

Express information concisely in a stereotyped format called object-action lan- • guage; and

Map knowledge into graphically-deﬁned information patches to increase speed • of search.

We discuss how these patterns and principles were uncovered and when they may be fruitfully applied.

3.3.4 Dynamic Focus Balances Simplicity and Complexity

We found, as prior work did, that in crises, doctors’ attention is a limited resource, and most gaze times are short (< 10 s) [Chen]. To address the limited attention in medical trauma and crisis response, prior work has winnowed information on interactive displays to just the most important elements [Kusunoki et al. 2014]. Ideally, CHAPTER 3. CRISIS ATTENTION: ANALYSIS AND DESIGN 60

Figure 3.4: [DynamicFocus] Treatment aid progresses from dosing atropine to consider: transcutaneous pacing or infusions crisis checklists should show only information that is relevant to the current operating context. Interfaces should minimize or omit less-related content. To achieve this, we introduce Dynamic Focus, an extension of the focus+context visualization [Card, Mackinlay, and Shneiderman 1999] that combines overview (context) and detail information (focus) with no occlusion. Dynamic Focus displays extend this approach with a situation-speciﬁc focus. For example, in car navigation, turn-by-turn directions often auto-update to show only the next turn, rather than the entire route. Drivers make fewer errors and lane deviations with auto-updating turn-by-turn directions than route overviews. Consequently, guidelines suggest that CHAPTER 3. CRISIS ATTENTION: ANALYSIS AND DESIGN 61

“drivers should not be expected to process complex information to obtain the desired route, i.e., the systems should not display a map with a highlighted route” and a limit of 15 seconds for task interactions [Green 1999]. Many cognitive aids partition information into sections, for example Signs, Treat- ment Protocol, and Di↵erential Diagnosis. In a Dynamic Focus aid, typically one section is expanded; the others are collapsed. Within a block, there is a selective focus on the current step; future steps are de-emphasized and executed steps crossed out (see Figure 3.4). The dynamic focus pattern works well if there is a reasonable amount but not massive amount of text to distill.

3.3.5 Object-Action Language Provides Brevity & Structure

This pattern codifies a strategy developed in aviation . For example, the checklist for an MD-80 airliner emphasizes the airplane configuration for takeo↵, landing, etc. [Degani 1992]. The left lists the object; the right lists the action to be taken on it, usually the configuration state to be set. For example: BRAKES...... SET WINDSHIELD HEAT...... ON This checklist language is compact, even terse. This compactness has at least four benefits: 1) more steps fit in a small space; 2) the checklist can be searched quickly because objects (left) and actions (right) are aligned; 3) the steps are quick to read because they have been reduced to a canonical form; and 4) people can verbally refer to elements using spatial language. The following example shows the application of RapidRead to medical checklists. The more usual checklist language:

Increase FiO2 to 100% CHAPTER 3. CRISIS ATTENTION: ANALYSIS AND DESIGN 62

Figure 3.5: Semiformal instructions

Verify ischemia with 12 lead EKG if possible is re-expressed as: FiO 100% 2 " Ischemia Verify Use 12-lead EKG. We call this object-action notation. Figure 3.5 has an example of applying this structure to an existing aid (left), resulting in semiformal instructions (right). The Twitter @cookbook is an example of extreme shortening–recipes in 140 char- acters or less. While brevity can help, in medicine one must take care when abbrevi- ating.

Drug parameter sub-language

Drug dosages appear frequently in checklist statements. Misreading these statements is so consequential that it is necessary to have a canonical dose presentation. In this CHAPTER 3. CRISIS ATTENTION: ANALYSIS AND DESIGN 63

sublanguage, the drug name is given (even if it repeats object), followed by the dose and units (in square brackets if an interval), then additional instructions like ‘IV’ or ‘max dosage’. For example: Calcium chloride 1g IV epinephrine [2˜10g/min]

Machine Parameter sub-language

Generally, the machine name is the object, and the action relates to the parameter of the machine, either as Parameter = Value or Parameter: Action. For example: Pacer Electrodes: Place on chest Mode= Pacer Current: Increase mA until capture

3.3.6 Information Patches Aggregate Related Content

To support rapid, random access, RapidRead uses visual patches to focus information search to a small, quickly-recognizable region. RapidRead separates steps spatially, and information types typographically (see Figure 3.6).

Procedure blocks

Procedure blocks group a small number of steps (up to five). Blocks can be of several types including signs, do immediately, treatment, or (di↵erential) diagnosis. Proce- dure blocks have a subtly colored background. This color cue both identifies the block type and defines the patch perceptually with a low spatial frequency region [Tovée 2008].

Drug patches

A gray background under the drug parameter speciﬁcation creates another low spatial frequency region. CHAPTER 3. CRISIS ATTENTION: ANALYSIS AND DESIGN 64

Figure 3.6: Information patches highlighted: procedures, drugs, and objects (left)

Object patches

With complex information, there are often multiple relevant groupings. To present objects as a group, we employ Tufte’s concept of layering and separation [Tufte 1990]. For example, in Figure 3.6, “Objects” form a vertically aligned cluster. This helps people quickly locate objects by consistently placing them on the left and rendering them in bold. Other elements are rendered in lighter type.

3.4 Experiment 1: Answer-time Measurement

We chose to study aid design in a controlled laboratory setting gathering fine-grained data from many participant trials. The first experiment compared five di↵erent sets of checklists (Figure 3.7) in a within-subjects experiment on medical professionals. Four sets were drawn from the literature. We created the fifth set (Structured Text) CHAPTER 3. CRISIS ATTENTION: ANALYSIS AND DESIGN 65

Figure 3.7: Asystole/Pulseless Electrical Activity aid, style comparison: Standard Text, Structured Text, Color Block, Pictographic, Dynamic Focus. by modifying the Standard aids to employ the object-action, information patch, drug patch, and object patch patterns. This experiment asked participants to ﬁnd information embedded in aids for Ad- vanced Cardiac Life Support (ACLS) [Neumar et al. 2010]. ACLS was chosen for its ubiquity and importance: U.S. medical school and advanced emergency medical technician (EMT) programs require ACLS coursework. We hypothesized that the Dynamic Focus aid would be faster than other styles because it reduces the amount of information displayed at one time. The other aid styles could not incorporate dynamic focus because they are static. We also hypothesized that Structured Text and Color Block would be faster than Standard Text due to increased structure, and that Pictographic would outperform the Standard Text on questions where the images were easily interpretable, but slower when the images were not easy to interpret.

3.4.1 Method

To ensure sucient understanding of aid terms and usage, the 13 participants comprised 2 ACLS trained EMTs and 11 medical doctors. Participants were compensated CHAPTER 3. CRISIS ATTENTION: ANALYSIS AND DESIGN 66

Figure 3.8: Standard Text, PEA

$40. The experiment compared ﬁve presentation styles.

Design Variations

Standard Text

This set of aids has shown to be e↵ective in high-ﬁdelity medical simulation [Ziewacz et al. 2011] (Figure 3.8).

Structured Text

These aids build on Standard Text, but distill their presentation into an abridged format (Figure 3.9). CHAPTER 3. CRISIS ATTENTION: ANALYSIS AND DESIGN 67

Figure 3.9: Structured Text, PEA

ColorBlock

These aids use color and visual design to delineate di↵erent conceptual chunks [Chu and Fuller 2011b] (Figure 3.10).

Pictographic

These aids have similar content and wording to Color Block, but have drastically di↵ering visual presentation. They use graphical images for each step of the checklist in addition to textual information as a way to provide visual landmarks [Chu and Harrison 2012] (Figure 3.11). CHAPTER 3. CRISIS ATTENTION: ANALYSIS AND DESIGN 68

Figure 3.10: ColorBlock, PEA

Dynamic Focus

These aids also draw their content from Color Block. They change display, showing more detail for the current step than other steps (Figure 3.12). All styles were presented on the same display at the same resolution density on screen. For example, one page of the Standard checklist aid used the same number of pixels as one page of the Structured Text aid, and half the pixels of a two-page Pictographic aid.

3.4.2 Procedure

In a within-subjects Latin square design, participants were timed on answering 15 information look up questions for each of ﬁve distinct styles of medical aids, totaling 75 questions. Some questions were simple lookup; others required some inference. CHAPTER 3. CRISIS ATTENTION: ANALYSIS AND DESIGN 69

Figure 3.11: Pictographic Aid, PEA

Examples: Drug Parameter: What is the correct dose for atropine? Procedure Parameter: What is the appropriate ventilation rate during CPR? Drug Selection: What drug and dose would you use to treat a calcium channel blocker overdose? To ensure that responses were not memorized, question answers were altered. For example, instead of putting down the correct Epinephrine drug dosage of 1mg, we put down similar numbers like 2mg or 3mg. Questions spanned 4 ACLS top- ics: Pulseless Electrical Activity (4), Supra-ventricular Tachycardia (3), VT/VF (4), and Bradycardia (4). A full list of questions is available online (“List of Questions: https://gist.github.com/icogaid/6604919”), as well as in the appendix. CHAPTER 3. CRISIS ATTENTION: ANALYSIS AND DESIGN 70

Figure 3.12: Dynamic Aid, PEA

Sequence

After a short pre-study questionnaire to record demographic information (occupation and years of experience) participants were given two example questions as a brief training. Participants were seated in a chair at a ﬁxed distance of approximately 24” from a 22” monitor with a 1680x1050 pixel resolution. Participants paced themselves using a keyboard. After reading a question, they pressed the spacebar to show the aid. Once they found the answer, they said the answer aloud, and pressed the spacebar again to advance to the next question. The experiment measured response time for answers as the interval between spacebar presses. Each session was videotaped, and a SMI RED eye-tracker captured participants’ eye movements. This eye-tracker requires no restraint or equipment to be worn, and is accurate to approximately .5 1 degree of arc. CHAPTER 3. CRISIS ATTENTION: ANALYSIS AND DESIGN 71

Measures

The primary measure was the time participants took to locate the requested piece of information within the aid. Response times are compared using a ﬁxed-e↵ects linear model that uses participant, question, and condition. Second, we analyzed the data by comparing the fraction of responses that exceed task-relevant thresholds—10 and 20 seconds. Third, we compare variation in response times using the coecient of variation. This metric is useful as it scales the standard deviation by the mean, allowing easy comparison between conditions. Threshold and variation analyses are important for paced tasks like crisis response and driving to measure the likelihood that an information task ﬁts into a safe cycle time for diverting attention from the primary task [Salvucci2010d].

3.4.3 Results

Figure 3.13: Answer times: means (seconds) by style. The symbol indicates coef- ﬁcient of variation, deﬁned as the standard deviation divided by the± mean

Dynamic Focus response times were fastest: 41% faster (avg. 5.7s) than Standard Text (avg. 9.6s) (Figure 3.13). This di↵erence was statistically signiﬁcant ( = 4.3, t(796) = -6.8, p<.001). Color Block was 16% faster ( = 1.5, t(796) =-2.4,p<.05) than Standard Text. Average response times for the other aids were statistically indistinguishable from Standard Text. Since long answer times are particularly dangerous, Tables 3 also reports the percent of trials exceeding 10 or 20 CHAPTER 3. CRISIS ATTENTION: ANALYSIS AND DESIGN 72

Figure 3.14: Fitted log-normal distribution of answer times seconds; the percent beyond 20 s ranged from 0% for the Dynamic Aid to 34% for the Standard Text aid.

Answer Times

Answer times followed an asymmetric log-normal distribution: for the log-transformed distribution, skewness was 0.5 and the excess kurtosis was 0.1, both close to the expected value of 0 for a normal distribution. Consequently, all statistical analyses that depend on data normality use log-transformed data. While most answer times were short, some were long, and these answer times in the tail of the distribution had a large e↵ect on the means of the answer times. Answer times for test taking have been previously characterized using log-normal or related (e.g. Gamma) distributions. I ﬁtted a log-normal plot to gaze times in phase I (Figure 3.14). Note the skewed distribution with a long tail. CHAPTER 3. CRISIS ATTENTION: ANALYSIS AND DESIGN 73

Figure 3.15: Answer time means (sec) by style + 1 stdev bars

3.4.4 Discussion

Two hypotheses were conﬁrmed: Dynamic Focus aids were fastest, and Color Block aids outperformed Standard aids (Figure 3.15). However, the hypothesis that Object Action aids would be faster than Standard Text aids was not substantiated. The Pictographic aid had mixed results in comparison to the Standard Text aids.

Why were the Dynamic aids so much faster?

Which attributes correlated with faster search? Eye traces highlight three e↵ective strategies. Successful designs reduced searchers’ eye movements by laying out a search path, quickly guiding them to a salient patch, or reducing the e↵ort of digesting information once found.

Only the necessary information

Reducing the amount of information makes choices easier. In static layouts, there is a tradeo↵between the amount of information and search complexity. Dynamically CHAPTER 3. CRISIS ATTENTION: ANALYSIS AND DESIGN 74

expanding step-relevant information and minimizing irrelevant information sped participants’ search. Since less information was shown, more pixels could be dedicated to the relevant information, compared to other aids.

Consistent Structure

Information blocks helped participants ﬁnd information faster. By grouping related information, blocks allowed participants to quickly dismiss or focus on a patch. Consistent with information foraging theory [Pirolli 2007], most eye traces began with a broad scanning phase to locate the right patch, followed by focused consump- tion of that patch’s information. Participants’ eyes followed the object column until they found the drug name, then moved to the action column to read the dosage information. By contrast, the standard text aids have less visual structure, requiring participants to scan all of the text.

3.4.5 Heat-maps: Background

Eye-tracking analysis supported the use of visual hierarchy. Heat map images demonstrated consistent patterns as participants parsed various cognitive aids. Output from eye-tracking was used to characterize di↵erent information scan and search patterns for various information seeking tasks. Eye-tracking has previously been used for usability studies of websites and consumer shopping behavior, as well as for investigating various psychological concepts. It relies on a calibration step and for the participant to remain largely still and seated. Eye trackers use reﬂection to track pupil movements, including saccades and dwell times. Plotting saccades and gaze vectors on a screen results in a dwell map which characterizes where a user looks. This can be turned into a “heat map” by recording where one or more participants look over the course of a period of time. In the heat-maps that follow, red corresponds to more activity, blue less. CHAPTER 3. CRISIS ATTENTION: ANALYSIS AND DESIGN 75

Heat-maps: Figures and Results

Here are several examples from eye-tracking heat maps which show prototypically fast and prototypically slow information search patterns. They provide insight into how di↵erent designs support or hinder rapid information search. For example, pictographic aids can often provide a distraction in some cases (Fig- ure 3.16) and a useful guide in another.

Figure 3.16: Heatmap of participants’ gaze: modiﬁed pictographic Bradycardia aid

3.4.6 Troubleshooting Cognitive Aids

This study also illuminated design ﬂaws and opportunities for improvement in all of the aids styles. By analyzing questions with highly di↵erential response times across the designs, we could focus on places where information design had a signiﬁcant impact. In Figure 3.19, points along the y=x line indicate questions where response times for an aid were equivalent to the Standard Aid. The top-left or bottom-right quadrants indicate questions where a design is performs much better or much worse CHAPTER 3. CRISIS ATTENTION: ANALYSIS AND DESIGN 76

Figure 3.17: Heatmap of modiﬁed Bradycardia aid, Dynamic style than the Standard aids. Here are three especially salient design issues identiﬁed with response time data and understood using the eye-tracking data. These issues highlight useful design patterns, or anti-patterns, that can be used to improve aid design.

Grouping Ecacy Depends on Use

In one case, participants spent long amounts of time in the consistently wrong part of an aid. For the question, “What is the appropriate ventilation rate during CPR for a patient in PEA?”, Structured Text had average response time of 6.1s, with 3.0s sd. Color Block had many more slow responses, with an average of 15.9s and sd 13.2s. What led to this large di↵erence? It turns out that there are multiple ways to group procedure aid content. Struc- tured Text aids (Figure 3.20) had a single procedure block related to CPR, and thus fast answer times. Color Block organized procedures more temporally. That is, a top block would contain Diagnose steps, and then next steps that were to be done CHAPTER 3. CRISIS ATTENTION: ANALYSIS AND DESIGN 77

Figure 3.18: Heatmap of modiﬁed Bradycardia aid, Text style immediately. While useful in certain situations, this led to CPR information being split across multiple blocks.

Lost without an anchor

Asecondissuewasthatkeyinformationinblocktitleswerevisuallyde-accentuated, which resulted in participants repeatedly missing the information (see Figure 8c). For the question, “Patient is in unstable SVT. Should shock be synchronized or unsynchronized for a narrow complex regular rhythm?”, the Dynamic Aid had average response time of 8.9s and sd of 3.2s. The Structured Text aid had average 18.7s and sd 6.5s. In the Structured Text aid, key information that the shock should be ‘unsychro- nized’ (a modiﬁer introduced speciﬁcally for this experiment) was displayed in a small font and all caps, even though it was the title for a block of information. Many participants missed this when scanning larger, bold items below. CHAPTER 3. CRISIS ATTENTION: ANALYSIS AND DESIGN 78

Figure 3.19: A comparison of mean answer times (seconds) of questions from each aid style to Standard (plotted along y=x).

Support rapid scanning

Athirdissuewaswhenmachineparametersettingswererepeatedorsplitacross blocks. They created visual distractors and participants often wasted time making sure answers were consistent before reporting them. For the question, “How many Joules should you shock at?”, the Dynamic Focus had average response time of 4.8s and sd of 1.6s. The Standard Text aid had average 18.7s and sd 13.7s. Participants were ecient when using the Dynamic aid. They first hit the title ‘Defibrillate” and looked to the right to see the Joules. In contrast, with the Stan- dard aid, participants looked in four separate areas because machine parameters were spread over three di↵erent sections. The largest distractor was the middle right, where ablocktitledDefibrillatordidnotcontaintheshocksetting.Theseconddistractor was the top-right box titled During CPR. Actual content was located on the left side. Repeated information seemed to hurt rather than help, as participants sometimes cross-checked to verify their answer was consistent. This analysis drove a new design pattern for the RapidRead principles, the machine parameter sub-language. CHAPTER 3. CRISIS ATTENTION: ANALYSIS AND DESIGN 79

Figure 3.20: Heatmap of modiﬁed Bradycardia aid, Structured style

3.4.7 Experiment 3: Structure Reduces Variance

Based on the results of Experiment 1, we updated the RapidRead principles. We added the machine parameter sub-language, and more systematically consolidated information patches. Would applying the revised guidelines to one of the aid styles improve performance? Our base was the Dynamic Focus aid because it best instanti- ated the RapidRead principles. We created an updated version called Rapid Dynamic that incorporated object and drug patches, as well as the machine and drug parameter sub-language.

Method

Eleven of the thirteen participants from the ﬁrst experiment returned for the follow- up study. For taking part in the second experiment, participants were compensated $40. The new RapidDynamic design was created to compare to the Dynamic. Pre- sentation format stayed the same. Participants ﬁrst repeated seven of the experiment CHAPTER 3. CRISIS ATTENTION: ANALYSIS AND DESIGN 80

1 questions chosen for high variance and similar overall average to the full set. They then answered these questions again for both the original Dynamic design and the new RapidDynamic redesign.

Results

Dynamic averaged 3.7s with a sd of 4.2 and a coecient of variance of 0.57. Rapid- Dynamic averaged 3.1s with a sd of 0.95 and a coecient of variance of 0.29. The di↵erence between means was not signiﬁcant, but RapidDynamic had signiﬁcantly less variance (F(48,48)=3.4, p<0.001).

Why Variance Matters

For paced tasks such as emergency medicine, reducing variance may be even more important than increasing average speed. As discussed earlier in this chapter, while the di↵erence between 8 and 10 seconds may not have a big impact, an information lookup that takes 30 or 60 seconds could be very distracting. By increasing design consistency, these techniques can reduce answer time outliers and make aids more dependable.

3.4.8 Further Analysis

Learning

Looking at the distribution of answer times, we observe a notable learning e↵ect. Participants learned over time—they responded more quickly as the trials contin- ued (Figure 3.21). However, it was not clear that any aid was necessarily easier to learn than any other (Figure 3.23). One important ﬁnding was that a participant’s performance on an aid they were familiar with (after multiple new exposures) was comparable on a “slower” aid when compared to a faster aid they they were not familiar with. For example, looking at Figure 3.23, we see that the answer times of 8-10 seconds on the right hand side CHAPTER 3. CRISIS ATTENTION: ANALYSIS AND DESIGN 81

Figure 3.21: Answer times (seconds) by trial (phase I data). Each circle corresponds to a single, timed answer response. is comparable to the fast answer times for the Dynamic Aid in orange. More concisely, Figure 3.23 aggregates data over all styles, noting that participants on average performed 34% faster at the end of Phase I than on ﬁrst use. These data perhaps explains why doctors are hesitant to switch aids–experience using and thinking through an aid is perhaps at least as important as the design of the aid itself. CHAPTER 3. CRISIS ATTENTION: ANALYSIS AND DESIGN 82

Figure 3.22: Average answer times (y-axis) decrease as trial numbers increase (x-axis). Non-dotted lines correspond to each style, dotted lines denote trend lines (phase I data only)

Discussion

Previous checklist design guidance has suggested that such aids be graphically minimal, ﬁt on a single page, with fewer listed items [Gawande 2013]. The experimental data presented support guidelines which encourage careful paring of information, but do not suggest that aids must be limited to a single page. Selective use of graphical content and visual layering may be minimally useful. An analysis of checklist design failures revealed several pitfalls for aid design. Based on these data, we suggest the following guidelines for checklist design: -ifpossible,usesoftwaretoprovidecontext-speciﬁc,dynamicallywinnowedfocus. CHAPTER 3. CRISIS ATTENTION: ANALYSIS AND DESIGN 83

Figure 3.23: Learning curve/e↵ect, averaged over all styles

-twopagesmaynotbeworsethanone -usegraphicssparinglytodrawattentionandprovideinformationscent -usevisualhierarchyandlayers -avoidinformationvomit Can interfaces be too fast, or checklists too minimal or over-designed?

Conclusion

This chapter introduced the RapidRead approach for designing procedure aids. Two studies compared search times and eye traces for six presentation styles, ﬁnding that Dynamic Focus aids sped information search. Previous checklist design guidance has suggested that such aids be graphically minimal, and ﬁt on a single page, with fewer listed items [Gawande 2013]. The experimental data presented support guidelines which encourage careful paring of information, but do not suggest that aids must be limited to a single page. Selective use of graphical content and visual layering may be minimally useful. An analysis of checklist design failures revealed several pitfalls for aid design. Eye-tracking analysis supported the use of visual hierarchy. CHAPTER 3. CRISIS ATTENTION: ANALYSIS AND DESIGN 84

Minimizing presented information by using context proved the most e↵ect technique for supporting rapid assimilation. In addition, visual and information design were found to impact speed. Eye-trace analysis suggested the beneﬁt of object-action language and information patches. Analysis uncovered several pitfalls of checklist design and new guidelines were presented for the design of future checklists and medical aids.

Future Work

There are various open questions regarding the use of pictographic designs and interaction methods. Preliminary work done by our medical collaborator shows that pictographic aids can be more e↵ective in certain situations. Future work could develop a predictive theory for estimating how visual design a↵ects performance. Besides visuals, the form factor also seems to matter—binders act di↵erent than laminated aids, which work di↵erently than smartphone, tablets, heads-up and large- screen displays. Visual design and form factors aside, how should practitioners make use of these aids in teams? Dynamic Focus aids require the user (pilot), or a 3rd party (navigator) to drive the system forward. Is this navigator the same person? Another person? Remote or local? What happens when AI-based systems attempt to assist in Dynamic Focus? Also, more work needs to be done in understanding how interactive, multi-surface displays can support dynamic aids in team settings. There are also many paced domains outside of medicine where these principles could be more broadly applied. Chapter 4

Crisis Attention: Evaluation

4.1 Chapter Overview

This chapter reviews the use of medical simulation in training, education, and evaluation. It introduces a design space of computer-based simulation paradigms, and explores various trade-o↵s in this space. The chapter introduces the narrative simulation paradigm for comparatively assessing expert procedural performance through a score-and-correct approach. I evaluate dynamic procedure aids with this approach and show their ecacy in simulated crisis situations. A study compared Dynamic Pro- cedure Aids, paper, and no aid conditions, ﬁnding that participants with Dynamic Procedure Aids performed signiﬁcantly better than with paper or no aid. Contents 4.1 Chapter Overview

4.2 Medical Simulation

4.2.1 History

4.2.2 Learning from Aviation

4.2.3 Medical Education and Training

4.2.4 Paradigm

4.2.5 Research under Simulation

85 CHAPTER 4. CRISIS ATTENTION: EVALUATION 86

4.2.6 Costs and Benefits 4.3 Simulation Paradigms 4.3.1 High- vs. Low- Fidelity 4.3.2 Tradeo↵s and Techniques 4.3.3 Medium-Fidelity Simulations 4.4 Narrative Simulation 4.4.1 Approach 4.4.2 Benefits 4.4.3 Drawbacks 4.4.4 Future Work 4.5 Dynamic Procedure Aids: An Evaluation 4.5.1 Method 4.5.2 Procedure 4.5.3 Statistical Analysis and Data Cleaning 4.5.4 Results 4.5.5 Discussion: Benefits of Dynamic Aids 4.5.6 Dynamic Procedure Aids

4.2 Medical Simulation

4.2.1 History

David Gaba was an early pioneer in mannequin-based simulation at Stanford [Gaba et al. 2001]. An experienced airplane pilot, Gaba adapted ideas from aviation into medicine, and began building simulation hardware, including patient mannequins. Stanford pioneers developed courses in Anesthesia Crisis Resource Management [Gaba, Fish, and Howard 1994], as a parallel to cockpit/crew resource management in aviation. These teach non-technical skills such as e↵ective communication and coordination [Gaba et al. 2001]. CHAPTER 4. CRISIS ATTENTION: EVALUATION 87

4.2.2 Learning from Aviation

In the early 20th century, pilot skills were passed from pilot to student ﬂying in actual airplanes. This was costly and dangerous. After a number of highly publicized accidents, the American army became interested in the use of simulation for training, resulting in the use of the Link Trainer, a physical ﬂight simulator. These simulators became widely adopted during World War II, when over 10,000 “blue box” simulators were used to train over 500,000 pilots [History and Heritage Committee 2010].

4.2.3 Medical Education and Training

Nowadays, pilots routinely logs thousands of hours in simulation. However, unlike aviation, simulation is not yet a standard part of medical school training worldwide. Just as the aviation Link Trainer was mainly purchased by carnivals in its early days [History and Heritage Committee 2010], the medical ﬁeld viewed the use of patient simulators with skepticism, considering them akin to video games rather than a serious training method or technique [Gaba et al. 2001]. In modern day medical education, classroom (and later clinical) work is emphasized, with some medical schools are providing iPads to incoming medical students (Figure 4.1). iPad and other tablets have now become commonplace amongst medical professionals.

4.2.4 Paradigm

High-ﬁdelity medical simulation was created to provide a safe, realistic setting for medical education and training. It places one or more practitioners in a hospital or operating room with a confederate crew of nurses and doctors, who, in turn are supported by a team of simulationists. Behind the scenes, teaching sta↵remotely control patient mannequin responses [Gaba et al. 2001] including heart rate, breathing, eye movement, and transmitted speech. CHAPTER 4. CRISIS ATTENTION: EVALUATION 88

Figure 4.1: Medical students attend lecture at the Li-Ka Shing Center for Knowledge (photo cc-by Stanford EdTech)

4.2.5 Research under Simulation

In human-computer interaction, the technique of Wizard of Oz refers to the use of abehind-the-scenesoperator(“wizard”)thatsimulatesthebehaviorofanintelligent or otherwise fully functional computer program [Kelley 1984]. As such, the theater of high-ﬁdelity medical simulation, with one-way mirrors and remotely controlled robotic mannequins, o↵ers an amazing opportunity to develop test scenarios and Wizard of Oz both patient responses as well as team interactions and novel software designs.

4.2.6 Costs and Beneﬁts

While high-ﬁdelity medical simulation o↵ers a great deal of verisimilitude, it can be costly in time and money. Even if some members of the support sta↵volunteer, CHAPTER 4. CRISIS ATTENTION: EVALUATION 89

paying multiple doctors, nurses, and simulation sta↵adds up, as well as the cost of renting and/or maintaining the building facilities. In response, technologists have developed approaches such as tabletop simulation [Zadow et al. 2013], resulting in high levels of immersion and engagement. Others have developed simulators that use virtual avatars rendered on large-screens tilted sideways.

4.3 Simulation Paradigms

4.3.1 High- vs. Low- Fidelity

Whereas high-fidelity simulation typically involves multiple participants, robotic mannequins and a large amount of physical equipment (monitors, tables, instruments, and so on), low-fidelity simulation is relatively inexpensive, and may refer to the use of just one communication channel (audio playback). High-fidelity simulation may not always be superior to low-fidelity simulation, cost aside. For example, students trained in high-fidelity simulation may not necessarily be better at identifying heart sounds than those trained using audio recordings only [Giovanni, Roberts, and Norman 2009]. Screen-based simulation is another low-fidelity simulation technique. Participants use a single screen, working on one task at a time (no or minimal multi-tasking), typically without other participants. Such screen-based simulators can also be e↵ective in training [Schwid et al. 2001]. Researchers have also investigated the use of so-called zero-fidelity simulation in fire emergency response [Toups et al. 2011], finding that zero-fidelity simulation can be a fruitful technique for practicing team communication.

4.3.2 Tradeo↵s and Techniques

There are clearly reasons to use higher-ﬁdelity simulations in teams. From a research perspective, the study of interventions and new technologies may require realistic environments to show validity. However, work in zero-ﬁdelity simulation shows that team communication can still be trained without expensive simulation equipment. CHAPTER 4. CRISIS ATTENTION: EVALUATION 90

On the other hand, low-ﬁdelity simulation methods such as the use of screen-based simulators are cheap and can be e↵ective teaching tools. However, they are typically not designed to evaluate teamwork, verbal communication patterns, attention management, and physical behavior.

4.3.3 Medium-Fidelity Simulations

This thesis introduces a medium-fidelity simulation technique called narrative simulation that attempts to bridge the gap between low-fidelity and high-fidelity simulation methods.

4.4 Narrative Simulation

The narrative simulation paradigm supports the comparative assessment of expert procedural performance through a score-and-correct approach. Inspired by video training [ACLS-Algorithms 2012], narrative simulation presents a consistently unfolding scenario to all participants. The scenario asks participants questions, records their response, and then reveals recommended best practice. The scenario continues from that action. This cross-participant consistency enables rapid, controlled experiments of how presentation a↵ects medical performance. Participants are asked to verbalize proper procedure under attentional stress and time limits. These scenarios placed single participants in the role of team-leader for cardiac arrest crisis, the role typically responsible for using the aids to support decision-making. This allowed us to test aids and verify their merit before requiring investment into larger teams and expensive simulation. Though not as realistic as a high-ﬁdelity team simulation, narrative simulation incorporates narrative elements that emulate teamwork, for example, involving virtual team members that report vitals and ask for next steps. CHAPTER 4. CRISIS ATTENTION: EVALUATION 91

4.4.1 Approach

When compared to single screen-simulation, narrative simulation often uses multiple screens and multiple, interleaved tasks. Since the narrative is linear and predictable, that is, non-branching, screens can be easily synchronized in advance. Distractor, or secondary tasks, can be easily added to induce load on a participant’s attention or scheduling.

4.4.2 Beneﬁts

There are notable beneﬁts in terms of cost. Without needing additional participants to play the part of secondary roles, the simulation method is cheaper than high- ﬁdelity, mannequin-based simulation. From a research perspective, since the narrative is linear, participant’s responses can be compared across conditions. Unlike screen-based simulation, additional, synchronized screens can be added to study the impact of attentional demands via secondary tasks. Finally, when studying novel interaction techniques or software prototypes, a non- branching scenario greatly reduces the cost of developing software, as interfaces can be more quickly mocked up and puppeted if the scenario is known in advance.

4.4.3 Drawbacks

However, there is still a role for other forms of simulation. It is not clear how gen- eralizable the results are of narrative simulation, especially when it comes to split attention, teamwork, and physical skills. It is also important to di↵erentiate between showing ecacy and e↵ectiveness. Ecacy refers to performance under ideal situations and controlled circumstances, whereas e↵ectiveness refers to real-world results [Singal, Higgins, and Waljee 2014]. CHAPTER 4. CRISIS ATTENTION: EVALUATION 92

4.4.4 Future Work

Narrative simulation as presented in this chapter uses screenshots and slide-based pre- sentations that were manually synchronized. However, future systems could support the easier creation of synchronized scenarios across multiple devices. For example, researchers could deﬁne scenarios via a spreadsheet, and then automatically deploy them across multiple devices. Such systems would support evaluating software and interaction techniques, especially in domains such as ubiquitous and wearable computing. For example, rather than testing how a user might navigate a city through various public transportation systems, a narrative simulation could virtually lead them through a scenario with pre-planned interaction points. While anecdotally, it is much cheaper to develop interfaces for non-branching scenarios, I have not done a cost analysis that shows how much cheaper it is. A comparative cost analysis and further cost/beneﬁt studies could support researchers and educators as they choose appropriate simulation methods for study and training.

4.5 Dynamic Procedure Aids: An Evaluation

There are many ways to present medical information aids. To understand the impact of interface presentation style on medical decision-making, we need both better theory and better empirical tools. The rest of this chapter applies the technique of narrative simulation towards this end.

4.5.1 Method

37 people (28 MDs, 9 medical students) were recruited from our university to par- ticipate in a one-hour study: 20 female and 17 male. Common specialties included: Internal Medicine (8), Anesthesia (7), and Emergency Medicine (7). All were trained in ACLS, which requires re-certification every 2 years. The distribution of recertified participants was: two years ago (4), one year ago (13), in the current year (16), and not yet certified (4). In this hospital, residents run cardiac arrest response teams. CHAPTER 4. CRISIS ATTENTION: EVALUATION 93

There are 2-4 “codes” per month (a “code blue” is used in hospitals to alert sta↵ that a patient requires resuscitation or other immediate attention). On average, each resident participates every few months.

Materials

Apre-studysurveyaskedparticipantsfor(expected)graduationyearfrommedical school, specialty, date of first (and most recent) ACLS certification. Participants were counterbalanced based on number of certifications (0, 1, 2 or more). This within-subjects experiment compared speed and quality of medical responses in three conditions: with paper aids, with dpAid, and with no aids. I hypothesized that narrative simulation would reflect the attention and time-limited nature of crises, and that dynamic aids would improve participant response quality relative to other conditions.

Paper Cognitive Aids

This condition provided participants with paper ACLS aids. I chose widely-used aids that have been shown to support crisis teams in high-ﬁdelity simulations [Ziewacz et al. 2011]. These aids were not standard in our hospital, so none of the participants used these aids in their regular work. We printed the paper aids on 8.5” 11” paper and laminated them so they would be sturdy and easy to handle. They were placed on a table nearby, a common practice.

Dynamic Procedure Aids

The dynamic aid, shown on a screen adjacent the scenario display, appeared to respond to scenario events as they happened. These pre-timed interfaces slides were synchronized with the scenario slides, advancing automatically as if a nurse or reader were controlling the interface via a mirrored tablet. The medical content in this condition was substantively equivalent to the paper condition, but presented using Step-at-a-Glance. Content from existing paper aids was divided into 2-4 steps, and the dynamic aid changed the focus step to match the scenario. CHAPTER 4. CRISIS ATTENTION: EVALUATION 94

Scenario Design and Slide Simulators. This study used narrative encapsulations of authentic medical scenarios, enabling fast and inexpensive medical challenges. Sce- narios were designed to test participants’ medical knowledge and crisis management decisions under time pressure. These medical scenarios were adapted from online training videos [ACLS-Algorithms 2012] and updated by our medical collaborators. This simulation approach focuses on psychological fidelity over physical fidelity, and is used widely in training [Beaubien and Baker 2004]. The scenario advanced slides every 5 seconds, revealing information about the patient and unfolding crisis. Each scenario contained 20 to 30 questions like “What is the next important step?” or “What is this [EKG] rhythm?” Participants had 10 seconds to verbally answer each question. Responses after 10 seconds were not counted; speed had no other impact on score. Regardless of response, scenarios revealed a fixed narrative. Scoring comprised three steps. First, we defined a rubric with the help of a doctor collaborator who teaches medical crisis response. Second, two authors jointly graded 1/3 of participants to align expectations, and split the other 2/3 equally. Partial credit was given as appropriate (e.g., for incorrect dosage but appropriate drug or defibrillation). Finally, answers with non-obvious grades were re-evaluated with the doctor who helped create the rubric.

Experimental Setting & Apparatus

The experimental room was conﬁgured with an empty patient bed, a secondary task display, and a scenario screen showing the simulation narrative and questions. In the dynamic condition, an external display showed the dpAid. In the paper condition, participants received laminated paper aids on a table.

Secondary Task

To simulate the additional cognitive load and multi-tasking required in crises, participants had to attend to a secondary task. On a separate screen, a ﬁlled circle randomly changed colors from gray to red, yellow, or blue approximately 50 times each scenario. CHAPTER 4. CRISIS ATTENTION: EVALUATION 95

Participants had 10 seconds to press a matching color-labeled key, reverting the color to gray. This induced an additional load on the participant’s attention, since they had to turn physically to see the secondary task display. The diculty of this task was chosen such that participants would uniformly do well.

4.5.2 Procedure

Experimental Sequence

The experiment comprised the following steps: consent form, pre-study survey, training, 3 scenarios, post-scenario surveys, post-study survey, and debrieﬁng. Total study time was 1 hour. Simulation runs were video recorded. Participants were alone— nurses and other doctors were implicitly present in the scenario design.

Training (10 mins)

Participants were guided through a 10-minute training period to familiarize them with simulation slides, secondary task, paper cognitive aids, and the dynamic checklists. Participants ran through two abbreviated versions of ACLS slide simulations, ﬁrst with paper cognitive aids and next with a synchronized dynamic checklist.

Scenarios (3x 8 mins)

All participants responded to three simulations, always in the same order. These were the progression of medical conditions for each: Male, 65, Pneumonia: Bradycardia, Asystole, Ventricular Fibrillation (25 questions) Male, 65, Syncope: Unstable Supraventricular Tachycardia, Ventricular Fibrilla- tion (25 questions) Female, 78, Unresponsive: Ventricular Fibrillation, Asystole, Ventricular Tachy- cardia (24 questions) CHAPTER 4. CRISIS ATTENTION: EVALUATION 96

Conditions

Each participant saw three conditions: dpAid, paper aids, and no aid. Participants saw each condition once; order was counterbalanced using a Latin square design. In the aid conditions, participants were told, “In this condition you will be given access to an aid. It will be located here.” They were told aid use was discretionary.

Post-Scenario Self-Assessment (3 1 min)

After each scenario, participants ﬁlled out a survey on their perceived performance for the scenario and secondary task:

How many times do you feel like you selected the incorrect color or missed one • entirely?

How many questions do you feel like you missed? • If you used a cognitive aid/checklist, how much do you feel it changed your • score on the questions?

Post-Study Survey & Debrief (10 mins)

Participants ﬁlled out a survey including demographic information and open response questions about ACLS and checklist experience. All materials used in the experiment, including the secondary task, surveys, scenarios, aids, and experimental protocols are available at https://github.com/icogaid/study- 2013.

4.5.3 Statistical Analysis and Data Cleaning

Scores are reported as the percentage of correct trials. Results were compared in R using the lm fixed e↵ects model, a type of linear regression. Unlike the t-test and similar to the ANOVA, linear regression accounts for the probability of multiple pair-wise tests being simultaneously true. Regression models have two benefits over repeated- measures ANOVA. First, fixed-e↵ects linear models are strictly more powerful than CHAPTER 4. CRISIS ATTENTION: EVALUATION 97

an ANOVA because they can handle unbalanced or missing data, but are otherwise equivalent to a multivariate ANOVA. Second, random e↵ects can be added to account for factors such as participant and scenario di↵erences that in practice cannot be ex- haustively sampled [Baayen, Davidson, and Bates 2008]. This paper primarily uses fixed-e↵ects regression models. Each result report comprises three pieces: first, per- condition averages; second, the e↵ect-size , indicating the slope di↵erence reported by the mixed e↵ects model; third, the key statistic and p-value. Note that is slightly di↵erent than simply subtracting the condition averages because incorporates the model’s estimate of underlying variation in random and fixed e↵ects.

Data Cleaning

29 of 37 starting participants had usable data for all scenarios: 6 had at least one scenario removed due to synchronization issues; 2 saw incorrect conditions. In the Pneumonia scenario, we removed questions 16 to 24 from the analysis after discovering that for many participants, a software bug caused Dynamic Aids not to advance with the scenario. We report results after this data cleaning.

4.5.4 Results

Aid type

Dynamic Aids reduced medical procedure errors. Participants responded correctly significantly more often in the Dynamic condition than in the unaided condition (79.6% vs. 69.1% correct; =9.46,t(82)=3.3,p<.01); the paper condition was not statistically better than unaided (70.0% vs. 69.1%; =.30,t(82)=.104, p = .92). Moreover, more use of Dynamic Aids correlated with fewer errors (Adj R2 = 0.28, F(4,82) = 8.01, p<.001) (Figure 4.2). Analyzing only the first scenario creates a between-subjects comparison that avoids the risk of priming or fatigue e↵ects. With this first-scenario analysis, the e↵ect of Dynamic Aids was even stronger: those using Dynamic Aids responded correctly significantly more often than unaided participants (80.0% vs. 63.6%; =16.4, CHAPTER 4. CRISIS ATTENTION: EVALUATION 98

Figure 4.2: Participants using Dynamic Procedure Aids responded correctly signiﬁ- cantly more often than those using paper aids or no aid t(26) = 4.3, p<0.01). Again, there was no signiﬁcant di↵erence between paper and no aids (67.6% vs. 63.6%; =3.95,t(26)=.974,p = .34).

Signiﬁcant factors & interaction e↵ects

To determine what factors were important in predicting scores, we compared several di↵erent models. To compare two lm models, we used R’s ANOVA function on pairs of model outputs. A significant ANOVA indicates the two models di↵er. Incrementally adding and testing factors and interaction e↵ects revealed that scenario, experience level, and experimental condition were all important. There were no significant interaction e↵ects between scenario and experience level, between experience level and experimental condition, and between experimental condition and scenario. Scenarios varied in diculty, as measured by error rate. The Pneumonia and Syncope scenarios did not di↵er significantly ( =-1.2,t(82)=-.042,p = .67), but Unresponsive was easier than Pneumonia ( =9.1,t(82)=3.17,p<.01). CHAPTER 4. CRISIS ATTENTION: EVALUATION 99

Experience

As might be expected, advanced medical personnel (residents and fellows) had more correct trials than medical students when controlling for condition and scenario (74% vs. 67%) ( =8.3,t(80)=2.81,p<.01).

Secondary task

Across all scenarios, participants successfully responded to 92% of colors. There was alearninge↵ect:responseratesimprovedasscenariosprogressed(88%,93%,97%). There was a marginally signiﬁcant e↵ect of condition on total missed responses on the color task (85 dynamic, 88 none, 115 paper, 2(2, n=30)=5.7, p =0.06).

Perceived Utility

In a post-test survey, participants reported both paper and dynamic aids as beneﬁcial. However, participants perceived a larger score increase with Dynamic Aids (15.3%) than paper (4.4%) (t = -4.52, df = 56.0, p<.001).

4.5.5 Discussion: Beneﬁts of Dynamic Aids

Dynamic Procedure Aids focus on four key problems: ready access, rapid assimilation, professional acceptance, and limited attention. I discuss observations for each in turn.

Ready Access

Paper aids can be tough to ﬁnd, easy to lose, and inconvenient to hold. Dynamic aids address this through a shared display with context-relevant information and resources. The study found that indeed participants used dynamic aids more than paper ones (mean 22.9 vs. 18.1 times per participant. t=-2.2, df=54, p<.05). In a post-survey interview, one participant (P21) discussed the possible beneﬁt of shared display. “The dosages thing is always scary to residents. . . having it sit there... I know it’s right, everyone can see it’s right... Everyone in the room can see where we are in the sequence...” CHAPTER 4. CRISIS ATTENTION: EVALUATION 100

Rapid Assimilation

Current aids are slow to read and search, diverting attention away from the patient. Dynamic aids address this through “step-at-a-glance”: cuing attention to the current step, and displaying relevant information. To make steps glanceable, aid content was expressed in a consistent object/action language and layout. Peripheral steps were summarized. Selecting a step as the focus dynamically expands it to present additional details. The secondary task simulated doctors’ multiple attentional demands. This dual-task methodology converts attentional load into errors. Consequently, dynamic aids’ lower error rate suggests that step-at-a-glance reduced attentional load.

Professional Acceptance dpAid integrates multiple resources. This presentation appears to have succeeded: participants estimated that Dynamic Aids improved their score by 15.3%; paper aids by 4.4%. This di↵erence is signiﬁcant (t=-4.52, df=56.0, p<.001). It is important that procedure aids both improve performance and are perceived to do so. These actual and perceived beneﬁts suggest that dynamic aids can facilitate aid acceptance. Because this study relied on volunteers, future work should assess perceived ecacy and directly measure acceptance in the broader community.

Limited Attention

Crises have multifarious activities competing for scarce resources. Describing the challenge of limited attention in medical response, one study participant remarked that “90% of your time is people trying to distract you.” Attentional overload acutely a↵ects people with less experience, because tasks requires more conscious e↵ort [Eric- sson and Lehmann 1996]. Consequently, a change in the novice/expert performance spread may indicate a change in the attentional bandwidth required. Improving newcomers’ performance is especially important because they commit more errors [Phillips and Barker 2010]. In this study, unsurprisingly, doctors had a higher accuracy rate than students CHAPTER 4. CRISIS ATTENTION: EVALUATION 101

(74.5% vs. 67.0%). However, students’ performance increased far more in the dynamic condition (21% for students, 7.5% for doctors). This suggests that dynamic aids are more attentionally ecient, providing more headroom for intrinsic task demands. Note that students seemed to outperform residents in the dynamic condition. Ihypothesizethatstudentsreliedmoreonaids,whileresidentsreliedmoreonexpe- rience. When designed well, external representations can be faster and more reliable.

(When) do paper aids help?

Notably, the study found no significant advantage of paper compared to no aid. In contrast, prior studies have found increases in team performance and adherence [Arriaga et al. 2013; Harrison et al. 2006; Ziewacz et al. 2011]. We posit three factors for this di↵erence: usage, teams, and training. First, the study assessed discretionary use; aid use was not required. Everyone referred to the dynamic aid at least ten times. By contrast, five participants used paper fewer than ten times—essentially placing themselves in a no-aid scenario. Sec- ond, prior work measured teams. This experiment studied individuals, even though they operated as if they were part of a virtual response crew. (To supplement this individually-focused experiment, I have run high-fidelity team simulations, finding that teams made successful crisis response decisions with dpAid.) Third, prior work may have provided more training on aids used. This study provided two minutes of training for each aid style. In the debrief, participants reported lack of familiarity as a major impediment to using paper aids. Many had experience with other aids. Given this, it is striking that participants used the digital aid well with minimal training. Aggregating these results with prior work suggests that paper aids are valuable when used, underuse minimizes impact, and that dynamic aids can encourage adoption.

4.5.6 Dynamic Procedure Aids

The study ﬁnds that a checklists’ design inﬂuences e↵ectiveness. I note examples of how digital aids helped. CHAPTER 4. CRISIS ATTENTION: EVALUATION 102

Dynamic aids track changes in best practices

Medical best practices change frequently, so even a doctor who perfectly remembers medical school may not be up-to-date. Prior to 2010, best practice was to check for pulse and rhythm changes immediately after shock. In newer versions, responders immediately perform post-shock CPR for all patients in cardiac arrest, even if they have a pulse [Link et al. 2010]. Performing CPR before checking for a pulse (the hoped-for outcome of the shock) was counter-intuitive and contrary to prior training for many participants. 24 participants studied ACLS before 2010, learning a dated protocol. The results reflect this: 9ofthe11participantswhosawthisinthedynamicconditionrespondedcorrectly; only 3 of 10 in the paper condition and 2 of 8 in the no aid condition responded correctly. One benefit of digital aids is that revisions can instantly propagate globally as knowledge evolves. Digital aids provide access to more information. Participants often forgot protocol specifics such as dosing, timing, joules, and appropriate ordering. A dynamic aid provides appropriate detail when needed, with less clutter.

Digital aids can reduce costs & variability of access

Paper aids can be tough to find, easy to lose, and inconvenient to hold. Two di↵erent participants dropped paper aids on the floor while trying to use them. Multiple participants missed questions while reading paper aids. Some became so frustrated after first use that they put them down permanently.

Digital aids (and simulation) help the low performers more

An important goal of medical crisis response—and many technology sca↵olds—“is to raise up the lowest performers to the level of the average performers” [Harrison 2012]. As we saw, medical students without aids performed the worst, and aids helped their performance dramatically. CHAPTER 4. CRISIS ATTENTION: EVALUATION 103

Digital aids combine with simulation for e↵ective training

This chapter introduced narrative simulation to evaluate time-constrained behavior. Three attributes suggest this approach. First, consistent scenario structure enables comparison across participants. Second, enforced pacing provides an element of real- ism, and assesses performance under tight time demands. Third, narrative simulation is relatively fast and cost-e↵ective. My experience is that simulation provides an ex- cellent venue for introducing and evaluating aids. This builds on decades of research in simulation [Degani and Wiener 1993; Gaba et al. 2001]. Chapter 5

Discussion and Future Work

5.1 Chapter Overview

This chapter discusses the role of procedure aids in medicine and other complex, high- risk domains–if and when adopted, how will they shape use, training, and education? For example, do aids de-skill experts or impair learning? Beyond medicine, how do insights from the design of Dynamic Procedure Aids generalize? The chapter looks forward to the role of wearable computing for procedure aids and, more broadly, its role in healthcare. Contents 5.1 Chapter Overview

5.2 The Future Role of Procedure Aids

5.2.1 Checklist Use by Expert and Novice Users

5.2.2 Checklist Errors

5.2.3 Social E↵ects of Aid Use

5.2.4 Checklist Compliance and Big Data

5.2.5 What can we learn from driving aids?

5.3 Generalizing Dynamic Aids

5.3.1 Dynamic Procedure Aids: Abstractions

104 CHAPTER 5. DISCUSSION AND FUTURE WORK 105

5.4 Conclusion

5.4.1 Looking Forward

5.5 Impact

5.6 Thanks

5.2 The Future Role of Procedure Aids

5.2.1 Checklist Use by Expert and Novice Users

Some may worry: do checklists and aids de-skill experts? People as far back as Socrates have worried that knowledge recorded on paper and elsewhere will become a crutch that de-skills memory [Plato, Hamilton, and Cairns 1961]. However, with checklists as with books, this is not a zero-sum game. People delegate the memory of knowledge to recorded media (when they believe they can access it later) [Sparrow, Liu, and Wegner 2011]. Given the fragile nature of memory, this is often wise. Concur- rently, people strengthen their search, assessment, and integration skills—improving quality of diagnosis and treatment. On the ﬂipside, my interviews with medical doctors and practitioners suggest that professionals ﬁnd it straightforward to imagine that checklists and related aids can and should be used by students and novices. Aids sca↵old learning and can provide step-by-step guidance. However, opinion di↵ers as to whether or not experts can or should continue to rely on aids as they develop knowledge, skills, and “muscle memory.”

5.2.2 Checklist Errors

Arelatedworry:mightchecklists,whetherpaperorsoftware,increaseerrors,or change the kinds of errors made? One could overfocus on an aid and respond slowly to unexpected events. A low-ranking sta↵member charged with reading checklists aloud [Burden et al. 2012] may feel uneasy speaking up, leading to missed steps or diagnoses. Social challenges aside, checklists have shown to be broadly useful, even CHAPTER 5. DISCUSSION AND FUTURE WORK 106

though best practices have yet to be formalized. In crises, both paper and software aids have the beneﬁt of being non-blocking, that is, practitioners can chose to attend to other matters if usage is too slow or otherwise non-functional.

5.2.3 Social E↵ects of Aid Use

Clinical instructors emphasized the importance of using cognitive aids and calling for help. However physicians vary in their approach. In the study presented in chapter 4, one participant (P01), when handed the paper aids said, “Why do you give me the answer? Don’t give me the answer.” In a post-study interview, one participant (P25) mentioned how paper aid use may a↵ect how physicians are perceived. “To be honest—physicians have an ego. [You] don’t want to seem unprepared by ru✏ing through papers. . . ” Contrastly, would physicians feel more empowered using tablets rather than rummaging through binders full of aids? Or would both be considered crutches? Further work needs to be done here.

5.2.4 Checklist Compliance and Big Data

In previous chapters, I discussed how dynamic aids might help individuals perform more e↵ectively, by serving as an aid to cognition, attention, and memory. In teams, aid use can promote a shared mental model and o✏oad other burdens on shared cognitive tasks. Zooming out, I argue that software-based checklists can also positively impact healthcare systems. Besides encouraging compliance and providing a distribution mechanism for up- to-date knowledge, checklists can also serve as a point for data collection. In the United States, healthcare records and databases are siloed, with each organization or institution making use of di↵erent systems and schemas. In comparison, the World Health Organization has successfully published checklists such as the pre-surgery timeout, which has been widely studied [Gawande 2009]. Although sites often customize checklists based on procedure and institutional di↵erences, the core checklist CHAPTER 5. DISCUSSION AND FUTURE WORK 107

may be somewhat standard. Thus, software-based checklists could o↵er an opportunity to collect standardized data points on a regular basis. Mandatory use of aids on a national scale and of data sharing would also beneﬁt medical research and consumer understanding of how healthcare systems are performing.

5.2.5 What can we learn from driving aids?

Driving is another complex, high-risk domain that can help reveal the potential impact of software-based procedure aids. Just as physicians vary, drivers di↵er as to how much they ask for help, both socially (in car or on the street) and practically (using GPS devices). Practically, interactive mapping technology has largely supplanted paper maps for navigation, in the form of web-based applications such as Google Maps as well as in- car navigation systems. Although legal and attention concerns come to the forefront once users start relying on such aids, they reduce the e↵ort required to navigate via unknown routes as well as provide the ability to re-route if driving conditions change or errors are made (turns are missed etc.). Will improved interaction design and AI similarly a↵ect how medicine is practiced?

5.3 Generalizing Dynamic Aids

5.3.1 Dynamic Procedure Aids: Abstractions

The approaches presented in this thesis address complex, high-risk procedures, but the principles can be used more broadly. While the focus has been on medicine, the Dynamic Aid interface paradigm is broadly useful for real-time assistive interfaces. For example, driving is a di↵erent sort of paced, perilous task. Using Dynamic Aids to analyze a GPS display shows how the same components combine to reduce drivers’ attentional burden. GPS navigation, unlike paper maps, provides a quickly ﬁnd-able display visible to drivers and passengers. Input is best delegated to those in a support role (passenger). CHAPTER 5. DISCUSSION AND FUTURE WORK 108

Abstraction Surgery Driving Shared Display Mirrored displays & cart Car GPS display Steps-at-a-Glance Simplify, focus on current step Turn-by-turn instructions Resource-at-a-Glance Names, supplies, lab results Roads, ETA, shops AttentionAids Drugtimers Location-drivendisplay

Table 5.1: Generalizing Procedure Aids

Turn-by-turn reveals directions with step-at-a-glance. Displays provide resources-at- a-glance: estimated arrival time, distance, nearby shops (for gas, cash, or ca↵eine) (Figure 5.1). This lens can also be applied to the domain of wearable computing. In the Glass Design Principles, Google emphasized the importance of relevance, telling designers to build apps that “don’t get in the way”. In other words, glass is designed, also, as an attention aid rather than an attention regulator. The Glass timeline, with its emphasize on the now (glass timeline here), mirrors the procedure aid approach of step-at-a-glance and resource-at-a-glance. Similarly, the Glass design team initially reported cramming too much information on the screen at first, finding eventually that they had to balance information needs with assimilation speed and complexity. It is not surprising that the Glass design implements the abstractions suggested by the Dynamic Procedure Aid approach. Wearable computing may work best when it aids rather than regulates attention, and emphasizes assisting the user in real-life, embodied tasks. Thus, the insights of the dpAid approach could broadly benefit designers and developers of future wearable computing platforms and applications.

5.4 Conclusion

Deploying aids through software has broad beneﬁts for authoring and distributing best practices. Creating e↵ective checklists requires both medical and design expertise. Encoding best layout practices in software would enable more experts to create and revise checklists. Digital aids also provide a mechanism for automatic logging and recording. Designing tools to support crisis response can be a challenge given the pace, risk, CHAPTER 5. DISCUSSION AND FUTURE WORK 109

multi-tasking and team nature of medicine. Dynamic aids o↵er the ability to reduce the impedance between a doctor’s needs and the information shown, improving adoption and adherence to best practice.

5.4.1 Looking Forward

Following on these promising results, further work should be done to look at the e↵ects of dynamic aids on teamwork in high-ﬁdelity simulations including the social impact on team communication and the possibility of distraction. Additional work should look at interaction issues, and in-situ professional acceptance. Finally, practical issues of technology availability and security are worth exploring.

Ecacy vs. E↵ectiveness

One must importantly distinguish between medical ecacy and e↵ectiveness [Singal, Higgins, and Waljee 2014]. What works in a simulation or clinical trial (ecacy) may not translate to real-world e↵ectiveness. This thesis presents preliminary evidence that software-based checklists manifesting the Dynamic Procedure Aid approach improves performance in controlled settings. However it does not show that these techniques are necessary or sucient to result in improved patient outcomes.

Team-based Displays

Little work has been done in developing multi-surface displays which see regular use in important tasks. However, in medicine and other domains, tablets, large-screens, and wearable computing platforms are beginning to be broadly adopted. How will interaction or design happen across these devices?

Designing Software-based Medical Aids

More work needs to be done to make it easier to design, share, and modify aids. These aids will need to work in di↵erent situations, for di↵erent users on di↵erent platforms. Every hospital may have di↵erent guidelines, and so customization becomes an important need for users. CHAPTER 5. DISCUSSION AND FUTURE WORK 110

Figure 5.1: Base image: simulated crisis with dpAid superimposed

Wearable Computing

Google Glass developers have been enthusiastic about the role of wearable computing in medicine. These sketches suggest how checklists and dynamic procedure aids may support physicians as they respond to crises.

The future of attention

Given the expansion of display technologies, more screens compete for users’ attention. How might we mitigate this possible negative impacts of information complexity? For example, how might we reduce the number of visual shifts required in an operating room, if this is a metric that matters. CHAPTER 5. DISCUSSION AND FUTURE WORK 111

Figure 5.2: Proposed heads-up display for crisis response: iPad integration

5.5 Impact

Finally, have we succeeded in our original intent? First to do no harm, then to improve healthcare systems and medical performance? More work needs to be done in understanding how to evaluate the impact of design research in medicine, as well as to build systems that get used, eventually, on live patients. Little is known about the role of software systems in healthcare outcomes and system eciency, but research must consider all these bottom lines to evaluate, advise, and encourage the work that follows this.

5.6 Thanks

Thanks to colleagues and collaborators, including members of the Stanford HCI group and the Stanford AIM lab. Friends and family. Thanks to Tableau software and VMware for providing software used in the analysis presented. Over the years, I have been supported by the National Science Foundation, the Stanford DARE graduate fellowship, the Stanford Computer Science department, and CHAPTER 5. DISCUSSION AND FUTURE WORK 112

Figure 5.3: Proposed heads-up display for crisis response: people in room a Google Scholarship. CHAPTER 5. DISCUSSION AND FUTURE WORK 113

Figure 5.4: Proposed heads-up display for crisis response: compression aid

Figure 5.5: Patient history on HUD: iPad plus large-screen display CHAPTER 5. DISCUSSION AND FUTURE WORK 114

Figure 5.6: Proposed heads-up display for crisis response: procedure aid Appendices

115 116

.1 Experiment 1 Questions

Fire (Training aid) q1 What is the code red number q2 if NON-AIRWAY fire, what do you Extinguish burning materials with?

Bradycardia q1 What is the correct dose for atropine? q2 What rate should the pacer be set to? q3 What infusions and doses would you select to treat bradycardia q4 What drug and dose would you use to treat a calcium channel blocker overdose.

PEA (Pulseless Electrical Activity) q1 What is the dose for Vasopressin q2 How often do you dose Epinephrine? q3 What is the appropriate ventilation rate during CPR? q4 In Pulmonary Thrombosis how do you rule out right ventricle failure?

VT/VF q1 what is the CPR compression rate? q2 How many Joules should you shock at? q3 how long is a CPR cycle? q4 What is the correct dosing for amiodarone?

SVT (Supra-ventricular tachycardia) q1 What are the minimum Joules for shocking for a narrow complex and regular rhythm? q2 Patient is in unstable SVT. Should shock be synchronized or unsynchronized for a narrow complex regular rhythm? 117

q3 What are the maximum Joules for shocking for a narrow complex and irregular rhythm? q4 What are the minimum Joules for shocking for a wide complex and regular rhythm? Bibliography

ACLS-Algorithms (2012). MegaCode. url: http://acls-algorithms.com/acls- megacode-simulator. Arriaga, Alexander F et al. (2013). “Simulation-based trial of surgical-crisis checklists”. In: N Engl J Med. 368.3, pp. 246–53. issn:1533-4406.doi: 10 . 1056 / NEJMsa1204720. url: http://www.ncbi.nlm.nih.gov/pubmed/23323901. Baayen, R.H., D.J. Davidson, and D.M. Bates (2008). “Mixed-e↵ects modeling with crossed random e↵ects for subjects and items”. In: Journal of Memory and Lan- guage,pp.390–412.issn: 0749596X. doi: 10.1016/j.jml.2007.12.005. url: http://linkinghub.elsevier.com/retrieve/pii/S0749596X07001398. Bardram, Jakob E, Thomas Riisgaard Hansen, and M Soegaard (2006). “AwareMedia: Asharedinteractivedisplaysupportingsocial,temporalandspatialawarenessin surgery”. In: Proc. CSCW,pp.109–118. Beaubien, J M and D P Baker (2004). “The use of simulation for training teamwork skills in health care: how low can you go?” In: Qual Saf Health Care,pp.i51–i56. Boorman, D. (2001). “Safety beneﬁts of electronic checklists: An analysis of commercial transport accidents”. In: Proceedings of the 11th International Symposium on Aviation Psychology.Columbus,pp.5–8.url: http://www.faa.gov/library/ online\_libraries/aerospace\_medicine/sd/media/Boorman.pdf. Burden, Amanda R et al. (2012). “Does every code need a ”reader?” improvement of rare event management with a cognitive aid ”reader” during a simulated emergency: a pilot study.” In: Simulation in Healthcare 7.1, pp. 1–9. issn: 1559-713X. url: http://www.ncbi.nlm.nih.gov/pubmed/22113440.

118 BIBLIOGRAPHY 119

Burian, Barbara K, Immanuel Barshi, and Key Dismukes (2005). The Challenge of Aviation Emergency and Abnormal Situations. Tech. rep. June. Mo↵ett Field, California: NASA. Burian, B.K. (2006). “Aeronautical emergency and abnormal checklists: expectations and realities”. In: Human Factors. Vol. 50. October. San Francisco: Human Fac- tors and Ergonomics Society, pp. 101–105. url: http://www.ingentaconnect. com / content / hfes / hfproc / 2006 / 00000050 / 00000001 / art00022http : / / www.ingentaconnect.com/content/hfes/hfproc/2006/00000050/00000001/ art00023. Card, Stuart K., Jock D. Mackinlay, and Ben Shneiderman (1999). Readings in information visualization: using vision to think.SanFrancisco:MorganKauf- mann Publishers Inc., pp. 306–309. isbn:1-55860-533-9.url: http://dl.acm. org / citation . cfm ? id = 300679http : / / dl . acm . org / citation . cfm ? id = 300679.300766papers3://publication/uuid/54F88F29- C991- 42A0- B6C5- E6EB538EF0DF. Chisholm, Carey D et al. (2000). “Emergency Department Workplace Interruptions: Are Emergency Physicians ”Interrupt-driven” and ”Multitasking”?” In: Academic Emergency Medicine,pp.1239–1243.doi: 10.1111/j.1553-2712.2000.tb00469. x. url: http://onlinelibrary.wiley.com/doi/10.1111/j.1553-2712.2000. tb00469.x/full. Chu, Larry and A Fuller (2011a). Appendix C: Crisis Management Cognitive Aids in Manual of Clinical Anesthesia.Lippincott.url: papers3://publication/uuid/ 247778C0-0951-4690-BED0-5CE2B031B486. —(2011b).Appendix C: Crisis Management Cognitive Aids in Manual of Clinical Anesthesia. Lippincott. url: papers3://publication/uuid/247778C0-0951- 4690-BED0-5CE2B031B486. Chu, Larry and T Kyle Harrison (2012). ACLS Emergency Aids. url: http://www. anesthesiaillustrated.org/cogaids/acls-emergency-aids/. Cirimele, Jesse et al. (2014). “RapidRead: Step-At-A-Glance Crisis Checklists”. In: Pervasive Health. url: http://bit.ly/rapidRead. BIBLIOGRAPHY 120

Clark, Herbert H. and Susan E. Brennan (1991). “Grounding in Communication”. In: Perspectives on Socially Shared Cognition. Ed. by Lauren B. Resnick, John M. Levine, and Stephanie D. Teasley. Vol. 135. 10. Washington, D.C.: American Psychologcal Association. Chap. 7. Coiera, E W, V J Tombs, and T H Clutton-Brock (1996). “Attentional overload as a fundamental cause of human error in monitoring”. In: pp. 1–31. De Vries, E N et al. (2011). “Prevention of surgical malpractice claims by use of a surgical safety checklist”. In: Ann Surg,pp.624–628. Degani, A (1992). On the typography of ﬂight-deck documentation.Tech.rep.Mo↵ett Field, CA. url: http://citeseerx.ist.psu.edu/viewdoc/download?doi= 10.1.1.184.8985\&rep=rep1\&type=pdfhttp://www.biblio.com/degani- asaf/on-the-typography-of-flight~67356943~title. Degani, Asaf and Earl L Wiener (1990). Human Factors of Flight-Deck Checklists : The Normal Checklist.Tech.rep.May.url: http://www.csa.com/partners/ viewrecord.php?requester=gs\&collection=TRD\&recid=N9127144AHhttp: //www.csa.com/partners/viewrecord.php?requester=gs\&collection= TRD\&recid=N9127144AH. Degani, Asaf and Earl L. Wiener (1993). “Cockpit Checklists: Concepts, Design, and Use”. In: Human Factors: The Journal of the Human Factors and Ergonomics Society,pp.345–359.url: http://hfs.sagepub.com/content/35/2/345. Dekker, Sidney (2011). Patient Safety: A Human Factors Approach. Boca Raton, FL: CRC Press, p. 247. isbn:978-1-4398-5225-5. Ericsson, K.A. and A.C. Lehmann (1996). “Expert and exceptional performance: evidence of maximal adaptation to task constraints.” In: Annual Review of Psy- chology,pp.273–305.issn:0066-4308.doi: 10.1146/annurev.psych.47.1.273. url: http://www.ncbi.nlm.nih.gov/pubmed/15012483. Federal Aviation Administration (1995). Human Performance Considerations in the Use and Design of Aircraft Checklists. U.S. Department of Transportation, Federal Aviation Administration, p. 63. url: http://www.faa.gov/about/office\ _org/headquarters\_offices/avs/offices/afs/afs200/branches/afs210/ training\_aids/media/checklist.doc. BIBLIOGRAPHY 121

Fourcade, Aude et al. (2012). “Barriers to sta↵adoption of a surgical safety checklist.” In: BMJ Quality & Safety,pp.191–7.issn:2044-5423.doi: 10.1136/bmjqs- 2011- 000094. url: http://www.pubmedcentral.nih.gov/articlerender. fcgi?artid=3285141\&tool=pmcentrez\&rendertype=abstract. Gaba, David M (2007). “Out of this nettle, danger, we pluck this ﬂower, safety: healthcare vs. aviation and other high-hazard industries.” In: Simulation in Healthcare, pp. 213–7. issn:1559-2332.doi: 10.1097/SIH.0b013e31815ca23e. url: http: //www.ncbi.nlm.nih.gov/pubmed/19088624http://content.wkhealth. com/linkback/openurl?sid=WKPTLP:landingpage\&an=01266021-200700240- 00001papers3://publication/doi/10.1097/SIH.0b013e31815ca23e. Gaba, David M, Kevin J. Fish, and Steven K. Howard (1994). Crisis Management in Anesthesiology. New York: Churchill Livingstone. isbn:0443089108.url: http: //www.getcited.org/pub/103104567. Gaba, David M et al. (2001). “Simulation-Based Training in Anesthesia Crisis Re- source Management (ACRM): A Decade of Experience”. In: Simulation & Gam- ing,pp.175–193.issn:1046-8781.doi: 10 . 1177 / 104687810103200206. url: http://sag.sagepub.com/cgi/doi/10.1177/104687810103200206. Gawande, Atul (2009). The Checklist Manifesto: How to Get Things Right. NY: Metropolitan Books. isbn:0805091742. —(2013).Project Check. url: http://www.projectcheck.org/checklist-for- checklists.html. Giovanni, Daniela de, Trudie Roberts, and Geo↵Norman (2009). “Relative e↵ectiveness of high- versus low-ﬁdelity simulation in learning heart sounds.” In: Medical education 43.7, pp. 661–8. issn:1365-2923.url: http://www.ncbi.nlm.nih. gov/pubmed/19573189. Green, Paul (1999). “The 15-second rule for driver information systems”. In: Proc of the ITS America. url: http://www.umich.edu/~driving/publications/ITSA- Green1999.pdf. Hales, Brigette M and Peter J Pronovost (2006). “The checklist–a tool for error management and performance improvement.” In: Journal of Critical Care,pp.231–5. issn:0883-9441. BIBLIOGRAPHY 122

Harrison, T Kyle (2012). personal communication. Harrison, T Kyle et al. (2006). “Use of Cognitive Aids in a Simulated Anesthetic Crisis”. In: Anesthesia & Analgesia 103.3, pp. 551–556. issn:0003-2999.doi: 10 . 1213 / 01 . ane . 0000229718 . 02478 . c4. url: http : / / www . anesthesia - analgesia.org/content/103/3/551.short. Haynes, A. B. et al. (2011). “Changes in safety attitude and relationship to decreased postoperative morbidity and mortality following implementation of a checklist- based surgical safety intervention”. In: Qual Saf Health Care,pp.102–107. Haynes, Alex B et al. (2009). “A surgical safety checklist to reduce morbidity and mortality in a global population.” In: N Engl J Med. 360.5, pp. 491–9. issn:1533- 4406. doi: 10.1056/NEJMsa0810119. url: http://www.ncbi.nlm.nih.gov/ pubmed/19144931. Healey, A N, N Sevdalis, and C A Vincent (2006). “Measuring intra-operative interference from distraction and interruption observed in the operating theatre.” In: Ergonomics,pp.589–604.issn:0014-0139.doi: 10.1080/00140130600568899. url: http://www.ncbi.nlm.nih.gov/pubmed/16717011. History and Heritage Committee (2010). The Link Flight Trainer. url: http:// files.asme.org/asmeorg/Communities/History/Landmarks/5585.pdf. Horrey, WJ and CD Wickens (2007). “In-vehicle glance duration: Distributions, tails and a model of crash risk”. In: Journal of the Transportation Research Board, pp. 1–16. url: http://trb.metapress.com/index/W870353NUP5664V6.pdf. Hunziker, Sabina et al. (2011). “Teamwork and leadership in cardiopulmonary resuscitation.” In: Journal of the American College of Cardiology,pp.2381–8.issn: 1558-3597. doi: 10.1016/j.jacc.2011.03.017. url: http://www.ncbi.nlm. nih.gov/pubmed/21658557. Hutchins, Edwin (1995). “How a cockpit remembers its speeds”. In: Cognitive Science, pp. 265–288. url: http://www.sciencedirect.com/science/article/pii/ 0364021395900209. Iqbal, Shamsi T. and Brian P. Bailey (2010). “Oasis”. In: ACM Transactions on Computer-Human Interaction 17.4, pp. 1–28. issn:10730516.doi: 10 . 1145 / BIBLIOGRAPHY 123

1879831 . 1879833. url: http : / / portal . acm . org / citation . cfm ? doid = 1879831.1879833. James, John T (2013). “A new, evidence-based estimate of patient harms associated with hospital care.” In: Journal of Patient Safety,pp.122–8.issn:1549-8425. doi: 10 . 1097 / PTS . 0b013e3182948a69. url: http : / / www . ncbi . nlm . nih . gov/pubmed/23860193http://journals.lww.com/journalpatientsafety/ Fulltext/2013/09000/A\_New,\_Evidence\_based\_Estimate\_of\_Patient\ _Harms.2.aspx. Kalnikait´e, V and S Whittaker (2007). “Software or wetware?: discovering when and why people use digital prosthetic memory”. In: Proc. of the SIGCHI conference on Human Factors in Computing Systems. isbn:9781595935939.url: http:// dl.acm.org/citation.cfm?id=1240635. Kelley, J. F. (1984). “An iterative design methodology for user-friendly natural language oce information applications”. In: ACM Transactions on Information Sys- tems 2.1, pp. 26–41. issn:10468188.doi: 10.1145/357417.357420. url: http: //dl.acm.org/citation.cfm?id=357417.357420. Kirsh, David (2005). “Metacognition, distributed cognition and visual design”. In: Cognition, education, and communication technology,pp.1–22.url: http : / / books.google.com/books?hl=en\&lr=\&id=eSOgC81ChcgC\&oi=fnd\&pg= PA147\&dq=Metacognition+,+Distributed+Cognition+and+Visual+Design\ &ots=LeKIhxGu7s\&sig=mPBtB2zOjg2kKkNdgPZVNKpLgL8. Kohn, L. T., J. Corrigan, and M. S. Donaldson (2000). To Err is Human: Building a Safer Health System. Tech. rep. Washington, D.C.: National Academy of Sciences. Kusunoki, Diana S. et al. (2013). “Understanding visual attention of teams in dynamic medical settings through vital signs monitor use”. In: Proceedings of the 2013 conference on Computer supported cooperative work - CSCW ’13. New York, New York, USA: ACM Press, p. 527. url: http://dl.acm.org/citation.cfm?id= 2441776.2441836. Kusunoki, D.S. et al. (2014). “Balancing design tensions: Iterative display design to support ad hoc and interdisciplinary medical teamwork”. In: Proc. of the ACM SIGCHI Conference on Human Factors in Computing Systems. BIBLIOGRAPHY 124

Kyng, M., E.T. Nielsen, and M. Kristensen (2006). “Challenges in designing interactive systems for emergency response”. In: Proc. DIS,pp.301–310. Leveson, Nancy G. (1993). “An Investigation of the Therac-25 Accidents”. In: IEEE Computer 26, pp. 18–41. url: http : / / citeseerx . ist . psu . edu / viewdoc / summary?doi=10.1.1.130.1077. Link, M. S. et al. (2010). “Part 6: Electrical Therapies Automated External Defib- rillators, Defibrillation, Cardioversion, and Pacing”. In: 2010 AHA Guidelines for Cardio-pulmonary Resuscitation and Emergency Cardiovascular Care,S706–S719. Liu, David et al. (2010). “Monitoring with head-mounted displays in general anesthesia: a clinical evaluation in the operating room.” In: Anesthesia and analgesia 110.4, pp. 1032–8. issn:1526-7598.doi: 10.1213/ANE.0b013e3181d3e647. url: http://www.ncbi.nlm.nih.gov/pubmed/20357147. Makary, Martin A et al. (2006). “Operating room briefings: working on the same page.” In: Joint Commission Journal on Quality and Patient Safety,pp.351–5. issn:1553-7250.url: http://www.ncbi.nlm.nih.gov/pubmed/16776390. Manser, Tanja et al. (2009). “Coordination patterns related to high clinical performance in a simulated anesthetic crisis.” In: Anesthesia & Analgesia,pp.1606–15. issn:1526-7598.doi: 10.1213/ane.0b013e3181981d36. url: http://www.ncbi. nlm.nih.gov/pubmed/19372344. Mathieu, J E et al. (2000). “The influence of shared mental models on team process and performance.” In: The Journal of applied psychology 85.2, pp. 273–83. issn: 0021-9010. url: http://www.ncbi.nlm.nih.gov/pubmed/10783543. McConnell, Douglas J, Kyle M Fargen, and J Mocco (2012). “Surgical checklists: A detailed review of their emergence, development, and relevance to neurosurgical practice.” en. In: Surgical Neurology International 3.1, p. 2. issn:2152-7806.url: http://www.surgicalneurologyint.com/article.asp?issn=2152- 7806; year=2012;volume=3;issue=1;spage=2;epage=2;aulast=McConnell. Mentis, Helena M. et al. (2012). “Interaction proxemics and image use in neuro- surgery”. In: Proceedings of the SIGCHI Conference on Human Factors in Com- puting Systems. New York, New York, USA: ACM Press, p. 927. url: http : //dl.acm.org/citation.cfm?id=2207676.2208536. BIBLIOGRAPHY 125

Mills, Peter, Julia Neily, and Ed Dunn (2008). “Teamwork and communication in surgical teams: implications for patient safety.” In: Journal of the American College of Surgeons 206.1, pp. 107–12. issn:1879-1190.doi: 10.1016/j.jamcollsurg. 2007.06.281. url: http://www.ncbi.nlm.nih.gov/pubmed/18155575. Monk, Christopher a, J Gregory Trafton, and Deborah a Boehm-Davis (2008). “The e↵ect of interruption duration and demand on resuming suspended goals.” In: Journal of Experimental Psychology,pp.299–313.issn: 1076-898X. doi: 10.1037/ a0014402. url: http://www.ncbi.nlm.nih.gov/pubmed/19102614. Neumar, Robert W et al. (2010). “Part 8: adult advanced cardiovascular life support: 2010 AHA Guidelines for Cardiopulmonary Resuscitation and Emergency Car- diovascular Care.” In: Circulation 122.18 suppl 3, S729–S767. issn:1524-4539. url: http://circ.ahajournals.org/content/122/18\_suppl\_3/S729. fullpapers3://publication/doi/10.1161/CIRCULATIONAHA.110.970988. Oulasvirta, Antti et al. (2005). “Interaction in 4-second bursts”. In: Proceedings of the SIGCHI conference on Human factors in computing systems - CHI ’05. New York, New York, USA: ACM Press, p. 919. isbn:1581139985.doi: 10.1145/1054972. 1055101. url: http://dl.acm.org/citation.cfm?id=1054972.1055101. Paparone, C. R., R. A. Anderson, and R. R. McDaniel (2008). “Where Military Professionalism Meets Complexity Science”. In: Armed Forces & Society 34.3, pp. 433–449. issn: 0095-327X. doi: 10.1177/0095327X07310337. url: http: //afs.sagepub.com/content/34/3/433.short. Parush, A. et al. (2011). “Communication and team situation awareness in the OR: Implications for augmentative information display”. In: Journal of Biomedical Informatics,pp.477–485. Patterson, Emily S (2007). “Communication Strategies From High-reliability: Trans- lation is Hard Work”. In: Annals of Surgery,pp.170–172.doi: 10.1097/01.sla. 0000253331.27897.fe. url: http://www.ncbi.nlm.nih.gov/pmc/articles/ PMC1876978/papers3://publication/doi/10.1097/01.sla.0000253331. 27897.fe. BIBLIOGRAPHY 126

Perel, A. et al. (2004). “Anaesthesiologists’ views on the need for point-of-care information system in the operating room: a survey of the European Society of Anaesthesiologists”. In: Eur. J. Anaesthesiol. Pp. 898–901. Phillips, David P and Gwendolyn E C Barker (2010). “A July spike in fatal medication errors: a possible e↵ect of new medical residents.” In: Journal of general internal medicine, pp. 774–9. issn:1525-1497.url: http://www.pubmedcentral.nih. gov/articlerender.fcgi?artid=2896592\&tool=pmcentrez\&rendertype= abstract. Pirolli, Peter (2007). Information Foraging Theory: Adaptive Interaction with Infor- mation. Oxford University Press. url: http://www.ncbi.nlm.nih.gov/entrez/ query.fcgi?db=pubmed\&cmd=Retrieve\&dopt=AbstractPlus\&list\_uids= 106114098413092665related:Ob-S1TX-eAEJpapers3://publication/uuid/ 5C3CC0C0-5900-472F-9515-8CF41BBA08FE. Plato, Edith Hamilton, and Huntington Cairns (1961). The collected dialogues of Plato, including the letters. Princeton, NJ, USA: Princeton University Press., xxv, 1743 p. isbn:0691097186. Pronovost, Peter et al. (2006). “An intervention to decrease catheter-related blood- stream infections in the ICU.” In: N Engl J Med. 355.26, pp. 2725–32. issn:1533- 4406. url: http://www.nejm.org/doi/full/10.1056/NEJMoa061115http: //www.ncbi.nlm.nih.gov/pubmed/17192537. Rochlin, Gene I et al. (2005). “The Self-Designing High-Reliability Organization : Aircraft Carrier Flight Operations at Sea”. In: Naval War College Review,pp.75– 90. url: http://www.caso- db.uvek.admin.ch/Documents/MTh\_SM\_22. pdfpapers3://publication/uuid/62B6CC6F-7E71-4540-94CB-32E1575D24CA. Sarcevic, Aleksandra, Ivan Marsic, and Randal S. Burd (2012). “Teamwork Errors in Trauma Resuscitation”. In: ACM Transactions on Computer-Human Interaction, pp. 1–30. issn:10730516.url: http://dl.acm.org/citation.cfm?id=2240156. 2240161. Sarcevic, Aleksandra, Ivan Marsic, and Randall S Burd (2010). “Does size and location of the vital signs monitor matter? A study of two trauma centers.” In: Proceed- ings of the American Medical Informatics Association 2010 Annual Symposium BIBLIOGRAPHY 127

(AMIA 2010).Vol.2010,pp.707–11.url: http://www.pubmedcentral.nih. gov/articlerender.fcgi?artid=3041385\&tool=pmcentrez\&rendertype= abstract. Schwid, H A et al. (2001). “Screen-based anesthesia simulation with debriefing improves performance in a mannequin-based anesthesia simulator.” In: Teaching and learning in medicine 13.2, pp. 92–6. issn:1040-1334.doi: 10.1207/S15328015TLM1302\ _4. url: http://www.ncbi.nlm.nih.gov/pubmed/11302037. Singal, Amit G, Peter D R Higgins, and Akbar K Waljee (2014). “A primer on e↵ectiveness and ecacy trials.” In: Clinical and translational gastroenterology 5, e45. issn: 2155-384X. url: http://dx.doi.org/10.1038/ctg.2013.13. Sparrow, Betsy, Jenny Liu, and Daniel M Wegner (2011). “Google e↵ects on memory: cognitive consequences of having information at our fingertips.” In: Science 333, pp. 776–778. issn:0036-8075.doi: 10.1126/science.1207745. Srinivasan, Narayanan et al. (2009). “Focused and distributed attention”. In: Progress in Brain Research,pp.87–100.url: http://www.sciencedirect.com/science/ article/pii/S0079612309176069. Takahashi, Yuki, Hiroaki Kojima, and Ken-ichi Okada (2011). “Injured person information management during second triage”. In: Proc. UIST. New York, New York, USA: ACM Press, p. 97. url: http://dl.acm.org/citation.cfm?id=2047196. 2047208. Thimbleby, Harold (2013). “Reasons to question seven segment displays”. In: Pro- ceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI ’13. New York, New York, USA: ACM Press, p. 1431. isbn:9781450318990. doi: 10.1145/2470654.2466190. url: http://dl.acm.org/citation.cfm?id= 2470654.2466190. Thomassen, Oyvind et al. (2010). “Checklists in the operating room: Help or hurdle? A qualitative study on health workers’ experiences.” In: BMC Health Services Research,p.342.issn:1472-6963.doi: 10 . 1186 / 1472 - 6963 - 10 - 342. url: http://www.biomedcentral.com/1472-6963/10/342. Toups, Zachary O. et al. (2011). “Zero-fidelity simulation of fire emergency response”. In: Proceedings of the 2011 annual conference on Human factors in computing BIBLIOGRAPHY 128

systems - CHI ’11. New York, New York, USA: ACM Press, p. 1959. url: http: //dl.acm.org/citation.cfm?id=1978942.1979226. Tov´ee, Martin J. (2008). An Introduction to the Visual System. Cambridge University Press, p. 222. isbn:0521709644.url: http://www.amazon.com/Introduction- Visual-System-Martin-Tov/dp/0521709644. Tufte, Edward R. (1990). Envisioning Information. Cheshire, CT, USA: Graphics Press. url: http://www.amazon.com/Envisioning-Information-Edward-R- Tufte/dp/0961392118. Verdaasdonk, E G G et al. (2009). “Requirements for the design and implementation of checklists for surgical processes.” In: Surgical Endoscopy 23.4, pp. 715– 726. issn:0930-2794.doi: 10 . 1007 / s00464 - 008 - 0044 - 4. url: http : / / www . springerlink . com / index / 10 . 1007 / s00464 - 008 - 0044 - 4papers3 : //publication/doi/10.1007/s00464-008-0044-4http://www.ncbi.nlm. nih.gov/pubmed/18636292. Wickens, Christopher D and Jason S McCarley (2007). Applied Attention Theory. CRC Press. isbn:978-0-8058-5983-6.doi: http : / / dx . doi . org / 10 . 1201 / 9781420063363. Winters, Bradford D et al. (2009). “Clinical review: checklists - translating evidence into practice.” In: Critical Care,p.210.issn: 1466-609X. doi: 10.1186/cc7792. url: http : / / www . pubmedcentral . nih . gov / articlerender . fcgi ? artid = 2811937\&tool=pmcentrez\&rendertype=abstract. World Health Organization (2005). ICD-9-CM International Classiﬁcation of Dis- eases.9th. Wu, Leslie et al. (2014). “Supporting crisis response with dynamic procedure aids”. In: Proceedings of the 2014 conference on Designing interactive systems - DIS ’14. New York, New York, USA: ACM Press, pp. 315–324. isbn:9781450329026.doi: 10.1145/2598510.2598565. url: http://dl.acm.org/citation.cfm?id= 2598510.2598565. Xiao, Y. et al. (2001). “Cognitive properties of a whiteboard: A case study in a trauma centre”. In: Proc. ECSCW,pp.259–278. BIBLIOGRAPHY 129

Zadow, Ulrich von et al. (2013). “SimMed”. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI ’13. New York, New York, USA: ACM Press, p. 1469. url: http://dl.acm.org/citation.cfm?id=2470654. 2466196. Ziewacz, John E et al. (2011). “Crisis checklists for the operating room: development and pilot testing.” In: J. of the American College of Surgeons,212–217.e10.issn: 1879-1190. url: http://www.ncbi.nlm.nih.gov/pubmed/21658974.