A Knowledge-Based Assessment of Compliance to the Longitudinal Application of Clinical Guidelines Thesis submitted in partial fulfillment of the requirements for the degree of “DOCTOR OF PHILOSOPHY” by Avner Hatsek Submitted to the Senate of Ben-Gurion University of the Negev November 2014 Beer-Sheva A Knowledge-Based Assessment of Compliance to the Longitudinal Application of Clinical Guidelines Thesis submitted in partial fulfillment of the requirements for the degree of “DOCTOR OF PHILOSOPHY” by Avner Hatsek Submitted to the Senate of Ben-Gurion University of the Negev Approved by the advisor__________ Approved by the Dean of the Kreitman School of Advanced Graduate Studies__________ November 2014 Beer-Sheva This work was carried out under the supervision of Prof. Yuval Shahar In the Department: Information System Engineering Faculty: Engineering Acknowledgments I would like to thank to those who supported and encouraged me in completing this research. My academic supervisor, Professor Yuval Shahar, who guided me for several years, in which he shared much of his infinite knowledge with me, and granted me with an opportunity to learn and to develop my skills in many new areas. I would like to thank Dr. Irit Hochberg, Dr. Deeb Daoud and Prof. Aya Biderman who participated in this research and donated from their time and knowledge to assist me in evaluating the system I developed in this research, and to better understand the implications of such systems to real clinical settings. My colleges at the Ben Gurion University Medical Informatics Research Center, Erez Shalom, Denis Klimov, Ayelet Goldstein, Elior Ariel, Mata Lion and all the others, whom I was lucky to share many experiences and develop new ideas, and who never hesitated to assist when any help was needed. I would like to thank the Department of Information System Engineering and the Faculty of Engineering at Ben Gurion University, where I studied, taught and researched for many years, for assisting me financially, and mainly, for providing me the environment to acquire so much knowledge and so many skills. I would also like to thank the Israeli Ministry of Science Office and the "GERTNER" National Institute for research in health-care policy, who supported in part of the funding for my research. This work is dedicated to my wife Merav, who I really love, and together expecting a first son to arrive very soon… I Abstract Clinical guidelines are developed by professional medical associations as a tool to standardize medical care. These guidelines are published by the associations in order to assist clinicians in basing their medical decisions on state of the art, research-based evidence. Although these guidelines are in general, accessible to clinicians, it is almost impossible for busy clinicians to constantly follow every new guideline that is published, and to comply with all of the current recommendations. Several methods were presented in the past for the development of automated systems for clinical-guideline-based plan recognition, critiquing, and quality assessment. However, additional work is still needed in order for such systems to be widely accepted in healthcare. In this study, I designed, implemented, and evaluated the DiscovErr system, which is a comprehensive system for guideline-based critiquing and quality assessment, that includes an integrated set of modules with a graphical knowledge specification interface; a clinical guideline library; a full-fledged guideline-based quality-assessment engine that assesses care over long time periods according to a well-defined, formal representation of the guideline; and an interface for the visualization of the compliance analysis results. The DiscovErr system uses a formal representation of the procedural workflow knowledge inherent in clinical guidelines and of the declarative domain- specific data-interpretation knowledge they explicitly or implicitly use, to perform quality assessment of the medical care by analyzing the longitudinal clinical data. Furthermore, by using a flexible reasoning mechanism based on fuzzy temporal logic, the compliance analysis algorithm addresses the inherent ambiguity of clinical guidelines, the uncertainty of clinical data, and the fact that care providers typically do not strictly follow clinical guidelines but rather adhere to these guidelines in a partial manner that reflects the essence of the guidelines' recommendations. To evaluate the DiscovErr system, I formally represented a complex, state of the art guideline for management of type II diabetes patients. I then performed several experiments by applying the system to significant numbers of real patient data, comparing the critique generated by the system to the comments made by three expert internists (two of whom are diabetes-therapy experts, and the third a family medicine expert) who reviewed the original clinical records as well as the system’s comments. II The completeness of the DiscovErr system's comments was assessed by comparing the system's comments to those of the three expert clinicians, defined as the portion of the experts' comments that were reproduced by the system. The completeness of the DiscovErr system's comments increased from 66% for comments made by only one expert, to 83% for comments that were mentioned by exactly two of the experts, and reached 98% for comments made by all three experts. The completeness of the system was 91% for comments made by at least a majority of the three experts, which I considered as the system's completeness level. The correctness of the comments made by the system was assessed by the two diabetes experts, who assessed each comment as correct, partially correct, or incorrect. The level of agreement on the correctness of the system's comments, among the two diabetes experts, was measured using Cohen's Kappa, and was found to be significantly high. A comment was considered as correct if it was assessed as correct by one expert and as at least partially correct by the other expert. The correctness of the systems' comments, as assessed by the two diabetes experts, was also 91%. Correctness was higher for comments concerning issues that were judged as being of greater importance, as opposed to comments concerning issues judged as being of lesser importance. I have also assessed the experts' correctness and completeness in an indirect fashion, through the comments they made and through their assessments of the system's comments. The correctness of the experts' comments was defined as having one's comment agreed to by at least one additional expert. I defined an additional measure to assess the global "indirect correctness" score, in which the DiscovErr system was essentially considered as a fourth expert, and which was used for the overall assessment of the system versus the three experts. Completeness of the experts' comments was defined relative to the overall set of comments made by the DiscovErr system that were assessed as correct by both of the diabetes experts. Overall, when compared to the three human experts, using these measures to assess system and human correctness, the DiscovErr system could be placed between the family medicine expert and the two diabetes experts; with respect to the system and human measures of completeness, it displayed a higher level of completeness than any of the three human experts. I conclude that systems such as DiscovErr can be effectively used to assess the quality of longitudinal guideline-based care of large numbers of patients. Table of Contents 1 INTRODUCTION ............................................................................................................... 5 2 BACKGROUND ................................................................................................................ 13 2.1 Planning, Plan Understanding, and Plan Recognition .................................................... 15 2.2 Formal Models for Guideline Representation ................................................................ 16 2.3 Methods for Clinical Guidelines Recognition and Critiquing ........................................ 20 2.4 Fuzzy Sets and Fuzzy Logic .......................................................................................... 37 3 THE DISCOVERR SYSTEM .......................................................................................... 41 3.1 Desiderata for the System’s Design ............................................................................... 42 3.2 Overall Architecture ....................................................................................................... 44 3.3 The Knowledge Framework ........................................................................................... 46 3.3.1 Clinical Guidelines Representation ....................................................................... 47 3.3.2 Clinical Steps Representation ................................................................................ 53 3.3.3 Domain Knowledge Representation ...................................................................... 55 3.3.4 The Knowledge Specification Tool ....................................................................... 59 3.4 The Patient Data Access Module ................................................................................... 62 3.5 The Analysis Framework ............................................................................................... 64 3.5.1 The Fuzzy Temporal Reasoner .............................................................................