PHYSICAL-VIRTUAL PATIENT SIMULATORS: BRINGING TANGIBLE HUMANITY TO SIMULATED PATIENTS by SALAM DAHER M.S. University of Central Florida, 2015 M.S. University of Florida, 2006 B.S. Lebanese American University, 2004 A dissertation submitted in partial fulfilment of the requirements for the degree of Doctor of Philosophy in the School of Modeling, Simulation, and Training in the College of Engineering and Computer Science at the University of Central Florida Orlando, Florida Fall Term 2018 Major Professor: Gregory F. Welch c 2018 Salam Daher ii ABSTRACT In lieu of real patients, healthcare educators frequently use simulated patients. Simulated patients can be realized in physical form, such as mannequins and trained human actors, or virtual form, such as via computer graphics presented on two-dimensional screens or head-mounted displays. Each of these alone has its strengths and weaknesses. I introduce a new class of physical-virtual patient (PVP) simulators that combine strengths of both forms by combining the flexibility and richness of virtual patients with tangible characteristics of a human-shaped physical form that can also exhibit a range of multi-sensory cues, including visual cues (e.g., capillary refill and facial expressions), auditory cues (e.g., verbal responses and heart sounds), and tactile cues (e.g., localized temperature and pulse). This novel combination of integrated capabilities can improve patient simulation outcomes. In my Ph.D. work I focus on three primary areas of related research. First, I describe the realization of the technology for PVPs and results from two user-studies to evaluate the importance of dynamic visuals and human-shaped physical form in terms of perception, behavior, cognition, emotions, and learning. Second, I present a general method to numerically evaluate the compatibility of any simulator-scenario pair in terms of importance and fidelity of cues. This method has the potential to make logistical, economic, and educational impacts on the choices of utilizing existing simulators. Finally, I describe a method for increasing human perception of simulated humans by exposing participants to the simulated human taking part in a short, engaging conversation prior to the simulation. iii To my parents Amal and William for all their continuous sacrifices, care, and support that shaped the person I am today and helped me go higher and further during every step of the way. To my brother Bassam for always being there for me, especially when it matters the most. To my grandmother Teta Nabiha for your wisdom, love, and prayers. I am proud of being your first grand daughter to graduate with a PhD. To my second family away-from-home, my Orlando friends for sharing my special moments and offering their physical and psychological support. iv ACKNOWLEDGMENTS This work would not have been possible without the financial support primarily by National Sci- ence Foundation (NSF) Award #1564065, CHS:¨ Medium: Physical-Virtual Patient Bed for Health- care Training and Assessment,” Program Director Dr. Ephraim P. Glinert, and in part by the Office of Naval Research (ONR) Code 30 under Dr. Peter Squire, Program Officer (ONR awards N00014- 14-1-0248 and N00014-12-1-1003). I deeply appreciate the scholarships and fellowships from the National Center for Women and Technology (NCWIT), the I/ITSEC committee for the RADM Fred Lewis scholarship, the Link Foundation, and the UCF Modeling and Simulation graduate program. I also appreciate Florida Hospital for their support of my advisor Prof. Welch via their Endowed Chair in Healthcare Sim- ulation. I am especially indebted to my advisor and chair of my committee Prof. Gregory Welch for caring, advising, and supporting me and the team in every aspect of the research. I am inspired by your creativity, innovation, energy, collaboration, and strong work ethics. Thank you for taking the time to and having no reservations in directly or indirectly passing all of these positive qualities to people around you. As the proverb says: "He gives twice who gives quickly", and I double my thanks to you for always giving quickly! I am grateful to my committee Prof. Laura Gonzalez, Prof. Juan Cendan, and Prof. Michael Proctor for your time, expertise, and advice. Similarly, I am also grateful for the feedback from Prof. Gerd Bruder, Prof. Andrew Raij, Prof. Arjun Nagendran, and Prof. Charlie Hughes from the SREAL lab, and for Prof. Mindi Andreson, and Prof. Desiree Diaz from College of Nursing at UCF, and for Prof. Jeremy Bailenson from Stanford University. v It takes a village to do the work described in this dissertation. First, I acknowledge my team-mates and co-authors on many papers (not in any specific order): Jason Hochreiter, Ryan Schubert, Nahal Norouzi, Kangsoo Kim, and Myungho Lee. I acknowledge behind the scenes work of Barbara Lee, Katie Ingraham, and Eric Imperiale at the SREAL lab, and the overall support from the Institute of Simulation and Training and the school of Modeling and Simulation faculty and staff at University of Central Florida. I would like to mention by name Prof. Randall Shumaker, Prof. Paul Wiegand, Dr. Sabrina Gordon, Kirsten Seitz, and Naya Ramirez. I also acknowledge the behind the scenes work of people from College of Nursing, I would like to mention by name: Syretta Spears, Jorge Nieves, and Chris Upchurch. Furthermore, I recognize the efforts of my teachers, instructors, professors, and anyone who con- tributed in planting the seeds of knowledge throughout my life, this includes a very long list of people from College des Soeurs des Saints Coeurs, the Lebanese American University, University of Florida, and University of Central Florida. A special thank you to a unique friend, family member, and researcher, Dr. Joseph Najem for the insightful research discussions, perspective, and motivation. A huge thank you for my friends who have been there for me (in no specific order): Dr. Shainna Ali Borenstein, Dr. Rachel Eyma, Jo Bohn, Colleen O’Sullivan, Cindy Vincent, Rita Carnero, Veronica Lavenworth, Belda Stack, Dr. Linda Rosa-Lugo, Suspira Tiouat, Yana Maxwell, Kim Moore, Marie Hewitt, Shehan Sirigampola, and Ravi Melaram. Last but not least, nobody has been more important to me than my family Amal, William, Bassam, and Teta Nabiha. You are my rock, my strength, and my inspiration. I love you. vi TABLE OF CONTENTS LIST OF FIGURES . xvi LIST OF TABLES . .xxiii LIST OF ACRONYMS AND ABBREVIATIONS . .xxiv CHAPTER 1: INTRODUCTION . 1 1.1 Simulation Fidelity . 1 1.1.1 Importance of Simulation Fidelity . 2 1.1.2 Patient Simulation Fidelity Space . 2 1.1.3 Patient Simulation Fidelity Challenges . 3 1.2 Current Healthcare Education . 4 1.2.1 Didactic vs. Experiential Learning . 4 1.2.2 Current Types of Patient Simulators . 5 1.2.2.1 Standardized Patients (real humans) . 5 1.2.2.2 Mannequins (physical simulators) . 6 1.2.2.3 Computer-Based (Virtual Simulators) . 7 1.2.2.4 Augmented Reality (Physical-Virtual Simulators) . 8 1.2.3 Challenge . 9 1.3 Research Overview . 9 1.3.1 Thesis Statements . 11 1.3.2 Support for Thesis Statements . 11 1.3.3 Notable Findings . 12 1.4 Terminology . 13 1.5 Dissertation Overview . 15 vii CHAPTER 2: RELATED WORK . 18 2.1 Healthcare Setting . 18 2.1.1 Real World Clinical Setting . 18 2.1.2 Healthcare Simulation Setting . 19 2.2 Patient Simulators . 20 2.2.1 Standardized Patients . 21 2.2.1.1 Description and History . 21 2.2.1.2 Pros and Cons . 22 2.2.2 Mannequins . 23 2.2.2.1 Description . 24 2.2.2.2 Examples and History . 25 2.2.2.3 Pros and Cons . 25 2.2.3 Computer Based Patients . 26 2.2.3.1 Description . 27 2.2.3.2 Examples . 28 2.2.3.3 Pros and Cons . 28 2.2.4 Augmented and Mixed Reality Simulators . 29 2.2.4.1 Description . 29 2.2.4.2 Examples . 30 2.2.4.3 Pros and Cons . 30 2.3 Fidelity and Realism for Healthcare Simulation . 31 2.3.1 Usage of the Term . 31 2.3.2 Importance of Fidelity and Realism . 33 2.3.2.1 Multi-sensory Realism . 34 2.3.2.2 Separation of Sensory Cues . 34 2.3.2.3 Co-location of Sensory Cues . 35 viii 2.4 Human Senses and Technology . 35 2.4.1 Sight: Visual Displays . 35 2.4.2 Touch: Haptic Displays . 36 2.4.3 Hearing: Auditory Displays . 37 2.5 Human Information Processing . 37 2.5.1 Perceptual Stage . 39 2.5.1.1 Visual Sense and Perception . 40 2.5.1.2 Haptic Sense and Perception . 40 2.5.1.3 Auditory Sense and Perception . 41 2.5.1.4 Priming . 41 2.5.2 Cognitive Stage . 42 2.5.2.1 Working Memory . 43 2.5.2.2 Cognitive Load Theory . 44 2.5.3 Action Stage . 45 2.6 Select Metrics of Interest for Simulated Patients . 45 2.6.1 Perceptual . 45 2.6.2 Cognitive . 47 2.6.3 Emotions and Feelings . 48 2.6.4 Behavioral . 49 CHAPTER 3: PHYSICAL-VIRTUAL PATIENT DEVELOPMENT AND TESTING . 50 3.1 Physical-Virtual Patient Head . 51 3.1.1 Development . 51 3.1.1.1 Hardware . 51 3.1.1.2 Software . 51 3.1.2 Evaluation of the PVHead . 53 ix 3.2 Physical-Virtual Patient Bed . 58 3.2.1 Development . 58 3.2.1.1 Hardware . 58 3.2.1.2 Software . 61 3.2.1.3 Scenario-Driven Content . 66 3.2.2 Evaluation . 70 3.2.2.1 Technical Evaluation . 70 3.2.2.2 Human-Subject Experiment . 73 3.2.2.2.1 Protocol . 75 3.2.2.2.2 Instrument .