University of Massachusetts Amherst ScholarWorks@UMass Amherst Masters Theses Dissertations and Theses July 2018 The Application of Usability Engineering Methods to Evaluate and Improve a Clinical Decision Support System Kristine DeSotto University of Massachusetts Amherst Follow this and additional works at: https://scholarworks.umass.edu/masters_theses_2 Part of the Industrial Engineering Commons Recommended Citation DeSotto, Kristine, "The Application of Usability Engineering Methods to Evaluate and Improve a Clinical Decision Support System" (2018). Masters Theses. 635. https://doi.org/10.7275/11941974 https://scholarworks.umass.edu/masters_theses_2/635 This Open Access Thesis is brought to you for free and open access by the Dissertations and Theses at ScholarWorks@UMass Amherst. It has been accepted for inclusion in Masters Theses by an authorized administrator of ScholarWorks@UMass Amherst. For more information, please contact [email protected]. THE APPLICATION OF USABILITY ENGINEERING METHODS TO EVALUATE AND IMPROVE A CLINICAL DECISION SUPPORT SYSTEM A Thesis Presented by KRISTINE M. DESOTTO Submitted to the Graduate School of the University of Massachusetts Amherst in partial fulfillment of the requirements for the degree of MASTER OF SCIENCE IN INDUSTRIAL ENGINEERING AND OPERATIONS RESEARCH May 2018 Mechanical and Industrial Engineering © Copyright by Kristine M. DeSotto 2018 All Rights Reserved THE APPLICATION OF USABILITY ENGINEERING METHODS TO EVALUATE AND IMPROVE A CLINICAL DECISION SUPPORT SYSTEM A Thesis Presented by KRISTINE M. DESOTTO Approved as to style and content by: ____________________________________ Jenna Marquard, Chair ____________________________________ Hari Balasubramanian, Member ____________________________________ Elizabeth A. Henneman, Member ____________________________________ Tamar Taddei, Member ____________________________________ Sundar Krishnamurty, Department Head Mechanical and Industrial Engineering ACKNOWLEDGMENTS I would like to sincerely thank my advisor, Professor Jenna Marquard, for her guidance, encouragement, and support in helping me to complete this study and further my knowledge in the field of human factors. The different opportunities I was able to experience while working with her were the most rewarding part of my time at the University of Massachusetts. I also wish to express my gratitude to the other members of my committee: Professor Elizabeth Henneman, Professor Hari Balasubramanian, and Dr. Tamar Taddei. I continue to be inspired by their meaningful work and their passion for mentoring and I am thankful that they were willing to be part of my journey. Throughout the design and execution of this study I have received generous support from Dr. Tamar Taddei, Rajni Mehta, and the rest of the Cancer Coordination and Tracking System (CCTS) team. I am so thankful for the time and support from all of these individuals and the opportunity to collaborate on such an important topic. The Department of Veterans Affairs is filled with passionate, kind, and intelligent individuals and I am proud to work alongside them. I would also like to thank my teammates in Professor Marquard’s lab, Vanessa Martinez and Swaminathan Kandaswamy, and the other students at UMass who supported me along the way. Finally, I am unbelievably grateful to my family and boyfriend for their endless encouragement and belief in me. iv ABSTRACT THE APPLICATION OF USABILITY ENGINEERING METHODS TO EVALUATE AND IMPROVE A CLINICAL DECISION SUPPORT SYSTEM May 2018 KRISTINE M. DESOTTO, B.S., WORCESTER POLYTECHNIC INSTITUTE M.S., UNIVERSITY OF MASSACHUSETTS AMHERST Directed by: Professor Jenna Marquard Delays in the process of diagnosing and treating cancer are common and lead to confusion and undesirable outcomes. Care coordinators are often embedded within the system of care to manage follow-up care. Electronic and real-time reminder systems can be used to support the care coordinator’s work, but electronic health record (EHR) usability is known to be poor. This study, completed in collaboration with the Department of Veterans Affairs (VA) Connecticut Healthcare System, evaluated the Cancer Coordination and Tracking System (CCTS), an EHR-linked, web-based tool for cancer care management. A set of expert-driven and user-driven usability engineering methods was applied to comprehensively identify and analyze usability problems within the system. Ten current CCTS users were engaged in the study to help identify problem. 101 (62.3%) problems were identified through expert-driven methods, 56 (34.6%) were identified by user-driven methods, and 5 (3.1%) were identified through both types of methods. The list of 162 unique problems were prioritized and twelve high priority problems were highlighted. Design recommendations were developed to address each of these high priority problems. v TABLE OF CONTENTS Page ACKNOWLEDGMENTS ................................................................................................. iv ABSTRACT .........................................................................................................................v LIST OF TABLES ........................................................................................................... viii LIST OF FIGURES ........................................................................................................... ix CHAPTER 1. INTRODUCTION ...........................................................................................................1 2. LITERATURE REVIEW ................................................................................................3 2.1 Cancer Care and Coordination ...........................................................................3 2.2 Electronic Health Record Usability Problems ...................................................4 2.3 Usability Engineering Methods .........................................................................5 2.3.1 Heuristic Evaluation............................................................................6 2.3.2 Cognitive Walkthrough .......................................................................6 2.3.3 Observations using screen capture and think aloud ............................8 2.3.4 Debriefing Interviews .........................................................................9 2.3.5 Usability Questionnaires .....................................................................9 3. METHODS ....................................................................................................................10 3.1 Overview ..........................................................................................................10 3.2 Cancer Coordination and Tracking System .....................................................12 3.3 Expert-Driven Problem Identification .............................................................13 3.3.1 Heuristic Evaluation..........................................................................13 3.3.1.1 Approach ............................................................................13 3.3.1.2 Data and Analysis ..............................................................14 3.3.2 Cognitive Walkthrough .....................................................................15 3.3.2.1 Approach ............................................................................15 3.3.2.2 Data and Analysis ..............................................................15 3.4 User-Driven Problem Identification ................................................................16 3.4.1 Observations using screen capture and think aloud ..........................17 3.4.1.1 Approach ............................................................................17 3.4.1.2 Data and Analysis ..............................................................17 3.4.2 Debriefing Interview .........................................................................18 3.4.2.1 Approach ............................................................................18 3.4.2.2 Data Analysis .....................................................................19 3.4.3 User Perception of Cognitive Walkthrough Problems......................19 vi 3.4.3.1 Approach ............................................................................19 3.4.3.2 Data Analysis .....................................................................20 3.4.4 Usability Questionnaire ....................................................................20 3.4.4.1 Approach ............................................................................20 3.4.4.2 Data and Analysis ..............................................................20 3.5 Design Recommendations ...............................................................................21 3.5.1 Development of Design Recommendations......................................21 3.5.1.1 Approach ............................................................................21 3.5.1.2 Data and Analysis ..............................................................22 4. RESULTS ......................................................................................................................23 4.1 Overview ..........................................................................................................23 4.2 Results from Expert-Driven Methods ..............................................................26 4.2.1 Heuristic Evaluation..........................................................................26 4.2.2