A PRACTICE-BASED EVIDENCE APPROACH FOR CLINICAL DECISION SUPPORT

Hamzah Bin Osop BComp. (National University of Singapore) MSc. in Information Systems (Nanyang Technological University)

Submitted in fulfilment of the requirements for the degree of Doctor of Philosophy

Electrical Engineering and Computer Science School Science and Engineering Faculty Queensland University of Technology November 2018

To my wife and parents

Keywords

Clinical data warehouse, clinical decision support system, decision-making, eHealth, electronic health records, electronic medical records, evidence-based practice, patient health records, practice-based evidence

A Practice-Based Evidence Approach for Clinical Decision Support i

Abstract

Dynamic and complex patient care needs are creating an increasing demand for useful tools that support well-informed clinical decision making by healthcare professionals. Here, we introduce a new Practice-Based Evidence (PBE) approach to clinical decision making that builds on existing healthcare ICT infrastructure.

In general, Practice-Based Evidence is defined as the use of current best evidence that is drawn from actual clinical practice settings to make specific decisions about individual patient care needs. An example is healthcare professionals collaborating to form research networks where valuable clinical data gathered during routine clinical practices are accumulated and shared. Such PBE approach has been proposed as a complement to Evidence-Based Practice (EBP) due to limitations of outcomes from randomised controlled trials. Instead of accumulating clinical data via research networks, our newly proposed PBE approach strategises an ICT architecture that exploits Electronic Health Records (EHRs) and a Clinical Decision Support System (CDSS) to provide automated support for decision making, customising it to individual patients.

Currently, huge volumes of digital data are being generated by healthcare information systems. These data have the potential to be highly useful for decision making support. However, the healthcare information systems usually run on proprietary software and databases, resulting in a ‘diverse and disparate’ systems environment. This inhibits sharing of information across different information systems and ultimately to healthcare professionals. As such, we contend that an effective and efficient healthcare delivery requires an integrated, data-driven, information sharing approach. Our PBE approach to decision making support has three key components, (1) a data integration architecture, (2) Electronic Health Records as a key source of evidence, and (3) a Clinical Decision Support System as a tool to assist in decision making.

To evaluate the effectiveness of our PBE approach, we assessed its potential to integrate multiple sources of data and provide comprehensive information that could be used in a decision support tool to assist healthcare professionals in making well-

ii Feasibility of Practice-Based Evidence for Clinical Decision Support

informed decisions. To do this, a study in three stages was conducted with IT professionals and doctors in a public hospital.

The first stage of the study used interviews to elicit the hospital’s current Information and Communication Technology (ICT) architecture. Content analysis of the interview data revealed the clinical information systems in use and informed the design of a suitable data integration architecture, effectively a clinical data warehouse.

The second stage of the study used a survey to identify doctors’ perceptions regarding the benefits of using EHR systems and the usefulness of EHR data for clinical decision making. The outcome from this stage of the study helped to identify the practical value and suitability of electronic health records as a source of evidence to inform decisions.

The third and final stage consisted of a field test and a focus group discussion to evaluate the feasibility of our Practice-Based Evidence approach to clinical decision making, implemented through a prototype Clinical Decision Support System developed specifically for this research. Anonymised inpatient data was used as the source of evidence for our prototype CDSS. The system was able to provide several significant patient-centric statistics, one being the prediction of probable length of stay which illustrates how our PBE approach provides healthcare professionals with useful data that supports decision making. Outcomes from a qualitative thematic analysis of the focus group discussion revealed an overall positive perception regarding the feasibility and potential of our PBE approach to assist healthcare professionals with decision making.

This thesis contributes to the body of knowledge within the healthcare ICT domain in several aspects. First, the conceptualisation of our PBE approach to decision making identifies key components essential for its implementation. Second, it identifies the use of EHR as alternative evidence for PBE approach, based on the general perception among doctors in Singapore regarding the potential of data in EHRs to assist with decision making. Third, it presents a data warehouse architecture that supports PBE adoption, which is based on an existing IT infrastructure design of a public hospital in Singapore. Fourth, it evaluates the PBE approach to decision making through the use of a prototype CDSS and anonymised patient health records, representing a case study of a potential real world application.

A Practice-Based Evidence Approach for Clinical Decision Support iii

Table of Contents

Keywords ...... i Abstract ...... ii Table of Contents ...... iv List of Figures ...... viii List of Tables ...... x List of Abbreviations ...... xi Statement of Original Authorship ...... xiii Acknowledgements ...... xiv Publications Arising from this Thesis ...... xv INTRODUCTION ...... 1 1.1 Background ...... 1 1.2 Research Context ...... 5 1.3 Purposes ...... 6 1.3.1 Aims & Objectives ...... 6 1.3.2 Research Questions ...... 8 1.4 Significance, Scope and Definitions ...... 9 1.4.1 Research Significance, Scope and Limitation ...... 9 1.4.2 Methodology...... 11 1.4.3 Research Plan ...... 13 1.5 Research Contribution ...... 13 1.6 Thesis Outline ...... 17 LITERATURE REVIEW ...... 19 2.1 Background ...... 19 2.2 Chronic Disease Prevalence in General Cases ...... 21 2.3 ICT Adoption in Healthcare...... 26 2.4 Evidence-based Practice ...... 29 2.4.1 Evolution of Evidence-based Practice ...... 30 2.4.2 Benefits of Evidence-based Practice...... 33 2.4.3 Effectiveness of Evidence-based Practice Limited by Evidence ...... 34 2.5 Practice-based Evidence as a Complementary Paradigm To Evidence-Based Practice ...... 38 2.5.1 Practice-based Evidence in the eyes of Barkham and Mellor-Clark ...... 41 2.5.2 Practice-based Evidence in the eyes of Horn & Gassaway ...... 42 2.6 Electronic Health Records ...... 42 2.6.1 Understanding Electronic Health Records ...... 43 2.7 Clinical Data Warehouse ...... 49 2.7.1 Data Warehousing for Healthcare ...... 50 2.8 Clinical Decision Support System in Healthcare ...... 52 2.8.1 Adoption of CDSSs ...... 53 2.8.2 Limitations of current CDSSs ...... 53 2.9 Summary and Implications ...... 55 RESEARCH DESIGN AND METHODOLOGY ...... 59

iv Feasibility of Practice-Based Evidence for Clinical Decision Support

3.1 Research Collaboration with National University Hospital, Singapore ...... 59 3.2 Methodology and Research Design ...... 60 3.2.1 Methodology ...... 60 3.2.2 Research Design Study (1): Conceptualising Our Practice-Based Evidence Approach to Decision Making ...... 61 3.2.3 Research Design Study (2): Architecting for Practice-based Evidence Approach ...... 61 3.2.4 Research Design Study (3): Perceived Clinical Benefits of EHR Systems and Perceived Usefulness of EHR Data for Decision Making ...... 63 3.2.5 Research Design Study (4): Evaluating our Practice-based Evidence Approach ...... 65 3.3 Ethics and Limitations ...... 66 CONCEPTUALISING OUR NEW PBE APPROACH ...... 69 4.1 Overview ...... 69 4.2 Current Implementation of Practice-Based Evidence ...... 69 4.3 Defining Our Approach of Practice-based Evidence To Decision Making ...... 71 4.4 EHRs as Suitable Evidence for PBE ...... 73 4.5 Fundamental Architecture for PBE – Data Warehousing ...... 80 4.5.1 Warehouse Architecture and Model ...... 80 4.6 Clinical Decision Support System for PBE Approach ...... 82 4.7 Summary ...... 82 ARCHITECTING OUR NEW PRACTICE-BASED EVIDENCE APPROACH 85 5.1 Overview ...... 85 5.2 Research Design...... 86 5.2.1 Research Design ...... 86 5.3 Participants ...... 87 5.4 Instruments...... 89 5.5 Procedure and Timeline ...... 90 5.5.1 Pre-Interview ...... 90 5.5.2 Main Interview ...... 92 5.6 Analysis ...... 93 5.6.1 Organising and preparing data ...... 94 5.6.2 Defining unit of analysis ...... 96 5.6.3 Developing categories and a coding scheme ...... 96 5.6.4 Code Text ...... 97 5.6.5 Analysing Findings ...... 99 5.7 Designing a Data Warehouse Architecture for Our PBE approach ...... 106 5.8 Limitations ...... 110 5.9 Conclusion ...... 112 ELICITING DOCTORS’ PERCEPTIONS REGARDING BENEFITS OF EHR SYSTEMS AND THE USEFULNESS OF EHR DATA FOR DECISION MAKING ...... 115 6.1 Overview ...... 115 6.2 Research Model ...... 116 6.2.1 Research Design ...... 116 6.2.2 Pre-survey content validation ...... 116 6.2.3 Pilot Survey Testing ...... 117 6.3 Research Constructs ...... 118 6.3.1 Research questions...... 118 6.3.2 Survey items and constructs ...... 118

A Practice-Based Evidence Approach for Clinical Decision Support v

6.4 The Survey ...... 125 6.4.1 Instruments ...... 125 6.4.2 Survey participants selection criteria ...... 127 6.4.3 Survey Administration ...... 127 6.4.4 Ethics ...... 128 6.5 Analysis of the Survey Results ...... 128 6.5.1 Response ...... 128 6.5.2 Survey response reliability test ...... 129 6.5.3 Participants ...... 129 6.5.4 Descriptive Analysis ...... 133 6.5.5 Findings and Discussions ...... 145 6.5.6 Limitations ...... 155 6.6 Conclusions...... 157 EVALUATING OUR NEW PRACTICE-BASED EVIDENCE APPROACH .. 159 7.1 Overview ...... 159 7.2 Research Study Components ...... 160 7.3 Data Preparation Phase ...... 161 7.3.1 Data source ...... 161 7.3.2 Reviewing data ...... 162 7.3.3 Importing data into MySQL ...... 164 7.3.4 Data Preparation ...... 165 7.3.5 Summary of finalised data ...... 176 7.3.6 Verifying PBE data warehouse architecture with the finalised data set ...... 177 7.4 Prototype CDSS Design & Development ...... 182 7.4.1 Requirements Analysis (Planning phase) ...... 183 7.4.2 Wireframes as mock-up design (User interface) ...... 187 7.4.3 Prototype Development ...... 194 7.5 PBE Approach Evaluation ...... 202 7.5.1 Participants ...... 203 7.5.2 Instruments ...... 204 7.5.3 Conducting the evaluation process ...... 204 7.6 Analysis ...... 206 7.6.1 Organising and preparing data ...... 206 7.6.2 Coding Transcripts ...... 207 7.6.3 Drawing Findings ...... 212 7.7 Limitations ...... 221 7.8 Conclusion ...... 222 DISCUSSION & CONCLUSION ...... 225 8.1 Discussions Of Research Findings ...... 225 8.2 Contributions ...... 230 8.3 Limitations ...... 231 8.4 Future Work ...... 233 8.5 Conclusion ...... 235 REFERENCES ...... 237 APPENDICES ...... 254 Appendix A. Table of direct medical costs of diabetes mellitus paid by the hospital (Rieman et al., 1995) 254 Appendix B. Table of prevalence of Diabetes-related comorbidities and complications in Singapore from 2005 to 2008 (ISO, 1998) ...... 255

vi Feasibility of Practice-Based Evidence for Clinical Decision Support

Appendix C. Paradigm of EBP ...... 256 Appendix D. Evidence Pyramid ...... 257 Appendix E. Interview questions for IT Professionals ...... 258 Appendix F. Data access and use agreement...... 259 Appendix G. Finalised Survey Question ...... 261 Appendix H. Survey Email Invitation Template ...... 268 Appendix I. Ethics Application ...... 269 Appendix J. Ethics Clearance Certification...... 277 Appendix K. Frequency and Distribution ...... 279 K1. Frequency and percentage distribution of EHR systems use and perceived clinical benefits...... 279 K2. Frequency and percentage distribution of EHR systems use and perceived clinical benefits...... 281 K3. Frequency and distribution of information believed to be contained in EHR data ...... 282 K4. Frequency and percentage distribution of perceived EHR data quality ...... 282 K5. EHR data quality associated with clinical benefits ...... 283 K6. Perceived Clinical Benefits and Decision Making ...... 284 K7. Frequency and percentage distribution of Perceived Usefulness of EHR data with Decision Making ...... 284 K8. Frequency and percentage distribution of availability of information and decision making ...... 285 Appendix L. Cronbach’s alpha results ...... 286 Appendix M. Data fields of data source provided ...... 295 Appendix N. Structure of data set (presented in R) ...... 296 Appendix O. Summary of data sets (presented in R) ...... 297 Appendix P. Deyo Quan list of 17 comorbidities for CCI values ...... 298 Appendix Q. Nielsen’s (1994) Heuristic Evaluation Checklist ...... 299 Appendix R. Prototype design checklist based on Nielsen’s list of heuristics ...... 300 Appendix S. Usability Design Gap Solutions ...... 301 Appendix T. Interview Transcript CIO ...... 303 Appendix U. Interview Transcript Principal Systems Specialist ...... 306 Appendix V. Focus Group Transcript on the Evaluation of PBE Approach ...... 308 Appendix W. PBE Evaluation Participant Sign Off ...... 324 Appendix X. List of Healthcare Institutions under the new cluster...... 327

A Practice-Based Evidence Approach for Clinical Decision Support vii

List of Figures

Figure 1. Research Stages ...... 15 Figure 2, Research plan identifying how research questions are answered through the objectives identified and the corresponding methods to achieve them ...... 16 Figure 3. Percentage of all deaths by NCDs or chronic diseases in Australia and Singapore adapted from World Health Organization (2015) ...... 24 Figure 4. Factors influencing the quality of decision making through a causal-loop diagram (Osop & Sahama, 2016a) ...... 30 Figure 5. Graphical view of Electronic Health Record adapted from Sahama et al. (2013) ...... 45 Figure 6. Resource-based view model adopted ...... 64 Figure 7. Logical model of the key components in our Practice-based evidence approach ...... 71 Figure 8. Adoption of Evidence-based practice in a clinical practice setting (Osop & Sahama, 2015) ...... 74 Figure 9. Illustrating the Practice-based evidence approach to decision making (Osop & Sahama, 2015) ...... 75 Figure 10. Core competencies of a CIO based on the 9 identified capabilities required for a CIO (Chun & Mooney, 2009) ...... 87 Figure 11. 2x2 matrix representing 4 types of CIO roles (Chun & Mooney, 2009)...... 88 Figure 12. Sample coding is done to interview corpus ...... 97 Figure 13. Identifying concepts based on coding schemes using thematic networks ...... 98 Figure 14. Finalised concepts after iterations which included the addition of sub-themes or parent-themes ...... 99 Figure 15. NUH’s Clinical Application Map represented by NUHS (Tan & Ong, 2009) ...... 101 Figure 16. ICT and data warehouse architecture [Top left:(Hamoud & Obaid, 2014), top-right: (Choi et al., 2013), bottom-left: (Lu & Keech, 2015), bottom-right: (Chen et al., 2014)] ...... 104 Figure 17. Envisioned NUH enterprise ICT infrastructure (Osop & Sahama, 2016a) ...... 107 Figure 18. Flow of data enabling a PBE approach through a data warehouse architecture ...... 109 Figure 19. Data warehouse architecture for our proposed PBE approach ...... 111 Figure 20. Factors affecting the quality of decision making reflected in a causal-loop diagram (Osop & Sahama, 2016a) ...... 123 Figure 21. Distribution of participants based on healthcare organisation type ...... 129 Figure 22. Distribution of participants based on medical specialty ...... 130 Figure 23. Distribution of participants based on designation ...... 130 Figure 24. Number of participants based on designation ...... 131 Figure 25. Distribution of participants based on age group ...... 131 Figure 26. Distribution of respondents based on years of service ...... 131 Figure 27. Respondent distribution by years of EHR system usage ...... 133 Figure 28. Distribution of EHR systems use leading to clinical benefits ...... 136 Figure 29. Distribution of perception of how useful information captured in EHR is...... 137 Figure 30. Perceived Data Quality of EHR ...... 138

viii Feasibility of Practice-Based Evidence for Clinical Decision Support

Figure 31. Distribution of perceived EHR data quality ...... 139 Figure 32. Distribution of the quality of data by doctors using EHR systems in association with perceived clinical benefits gained ...... 142 Figure 33. Distribution of perceived clinical benefits ...... 143 Figure 34. Distribution of perceived EHR data usefulness ...... 144 Figure 35. Distribution of availability of information and decision making ...... 145 Figure 36. Distribution of the usage trends with regards to EHR systems in Singapore ...... 148 Figure 37. Comparing the survey results from EHR functionalities and the corroboration with decision making ...... 150 Figure 38. Items with less than 50% of the doctors in strong agreement ...... 151 Figure 39. Items with more than 50% of doctors in strong agreement ...... 151 Figure 40. Research study phase ...... 161 Figure 41. The phpMyAdmin tool used to store, retrieve and update data ...... 164 Figure 42. Star-schema data mart ...... 179 Figure 43. Extraction process based on data mart design using Pentaho CE Data Integration ...... 181 Figure 44. Mock-up layout ...... 188 Figure 45. Design with content in mind ...... 189 Figure 46. Functional process flow ...... 189 Figure 47. Logical flow of system with design layout ...... 190 Figure 48. The need for a user-friendly and easy-to-use prototype (Usability design gap) inspired by Davis (1993) Technology Acceptance Model (TAM) ...... 191 Figure 49. Finalised mock-up design ...... 193 Figure 50. Prototype clinical decision support system adopting the PBE approach to decision making ...... 196 Figure 51. Prototype CDSS with patient-centric statistics and prediction of length of stay ...... 199 Figure 52. Differing patient-centric statistics and prediction of probable length of stay based on patient profile and collection of health records ...... 205 Figure 53. Themes emerging through thematic networks from evaluation of PBE approach ...... 211

A Practice-Based Evidence Approach for Clinical Decision Support ix

List of Tables

Table 1. Dimensions of Data Quality by different authors ...... 48 Table 2. Dimensions of data quality for our PBE approach to decision making ...... 79 Table 3. Table of participants for the interview ...... 89 Table 4. Qualitative analysis process (Zhang & Wildemuth, 2016) ...... 94 Table 5. Coding manual ...... 97 Table 6. Identified survey questionnaire items from the three studies...... 120 Table 7. EHR Usage Trends list of questionnaire items ...... 121 Table 8. System functionalities with clinical benefits questionnaire items ...... 121 Table 9. Useful Information questionnaire items ...... 122 Table 10. Data Quality and clinical benefits questionnaire items ...... 122 Table 11. Data quality and usefulness questionnaire items ...... 123 Table 12. Data availability questionnaire items ...... 124 Table 13. Perceived clinical benefits and decision making questionnaire items ...... 124 Table 14. Table of constructs and corresponding questionnaire items ...... 125 Table 15. Likert 6-point scale of usage trends ...... 126 Table 16. Likert 5-point rating scale of agreement ...... 127 Table 17. Participant’s attributes and demographics...... 132 Table 18. Mean rank and standard deviation ...... 154 Table 19. Table of the proportion of high LOS outliers and non-outliers ...... 174 Table 20. First 10 rows of result from the application of function on the anonymised data set ...... 175 Table 21. An example of the first 10 rows of ICD_CCI values ...... 176 Table 22. Root concept of the proposed clinical decision support system ...... 183 Table 23. Stakeholder profiles ...... 183 Table 24. Use case ...... 184 Table 25. Development of a user scenario ...... 186 Table 26. List of functional requirements ...... 187 Table 27. Recommendations to usability design gaps based on Nielsen (1994) heuristics ...... 194 Table 28. Table of participants in focus group ...... 204 Table 29. Concepts matching themes developed deductively based on questions asked ...... 208 Table 30. Coding identified concepts and themes done inductively...... 210

x Feasibility of Practice-Based Evidence for Clinical Decision Support

List of Abbreviations

AARNET - Australia’s Academic Research Network ADHA - Australian Digital Health Agency AUD - Australian Dollar CAQDAS - Computer Aided Qualitative Data Analysis Software CCDR - Centralised Clinical Data Repository CCI - Charlson Comorbidity Index CCOE - Computerised Clinician Order Entry CDM - Chronic Disease Management CDoc - Clinical Documentation CDSS - Clinical Decision Support System CDW - Clinical Data Warehouse CIO - Chief Information Officer CPSS2 - Computerised Physician Support System 2 DRG - Diagnosis Related Group EBM - Evidence-based Medicine EBP - Evidence-based Practice EDW - Enterprise Data Warehouse eHIDS - Electronic Hospital Inpatient Discharge EHR - Electronic Health Record eIMR - Electronic Inpatient Medication Records eMARS - Electronic Medication Administration Recording System EMR - Electronic Medical Record ETL - Extraction-Transformation-Load HCP - Healthcare Professionals HDFS - Hadoop Distributed File System HE - Heuristics Evaluation HIE - Health Information Exchange HIMSS - Health and Information System Society HIS - Healthcare Information System HIT - Health Information Technology ICD - International Classification of Diseases ICT - Information and Communication Technology IHiS - Integrated Health Information Systems IPAS - Inpatient Pharmacy Automation System KBMA - Knowledge Based Medication Administration KTPH - Khoo Teck Phuat Hospital LIS - Laboratory Information System LOS - Length of stay MOHH - Ministry of Health Holdings NEHR - National Exchange Health Record NUH - National University Hospital NUHS - National University Health System ODS - Operational Data Store OLAP - Online Analytical Processing OLTP - Online Transaction Processing OPAS - Outpatient Pharmacy System

A Practice-Based Evidence Approach for Clinical Decision Support xi

PBE - Practice-based Evidence PBE-CPI - Practice-based Evidence for Clinical Practice Improvement PHR - Patient Health Record PMR - Patient Medical Record PRN - Practice Research Networks QUT - Queensland University of Technology RBV - Resource-based View RCT - Randomised Controlled Trials SBD - Scenario-Based Design SDLC - Software Development Life Cycle SGD - Singaporean Dollar SGH - Singapore General Hospital TAM - Technology Acceptance Model USD - United States Dollar

xii Feasibility of Practice-Based Evidence for Clinical Decision Support

Statement of Original Authorship

The work contained in this thesis has not been previously submitted to meet requirements for an award at this or any other higher education institution. To the best of my knowledge and belief, the thesis contains no material previously published or written by another person except where due reference is made.

Signature: ______

Date: ___23rd November 2018______

Hamzah Bin Osop, B. Computing, MSc (Info Systems)

A Practice-Based Evidence Approach for Clinical Decision Support xiii

Acknowledgements

First and foremost, I would like to extend my gratitude to my principal supervisor Dr Tony Sahama, for your constant belief, expertise, knowledge, patience and persistence given throughout my PhD journey. Your relentless guidance, support and motivation were invaluable, and for that I sincerely thank you.

To my associate supervisors, Dr Wayne Kelly and Dr Colin Fidge, thank you for assisting and supporting me in my most trying times and ensuring that the journey was ever smooth and painless.

For those who had assisted with my data collection, I am genuinely grateful. Notably, special thanks go to Dr Sue-Anne Toh and Dr Tan Xin Quan from National University Hospital for your endless support and assistance.

Most importantly, to my dearest wife, Norhayati Bte Amat, for your unwavering and unconditional love, belief and support which had guided and motivated me throughout the tough and challenging of times. Not forgetting, my loving parents, Mak (Hawa Bte Hassan) and Bak (Osop Bin Samad), for continuously spurring and inspiring me to always strive for the best. Also, to Mak Yah (Rokiah Bte Tahir) and Abah (Amat Bin Salleh) for supporting in what I do. To my brothers, sisters-in-law and niece, thank you for your constant support, cheers and concern.

To my dear friends in QUT, your unending support and encouragement were invaluable and had made this whole journey all the more worthwhile, so thank you for that.

Finally, for those who were not mentioned, my sincerest thanks and appreciation for all your kind and generous support.

xiv Feasibility of Practice-Based Evidence for Clinical Decision Support

Publications Arising from this Thesis

The outcomes of this thesis have been published or are under review in peer-reviewed journals and conferences as follows.

PUBLISHED PAPERS

Refereed Book Chapters 1. Osop, Hamzah & Sahama, Tony (2016). Data driven and practice-based evidence: Design and development of efficient and effective clinical decision support system. In Improving Health Management through Clinical Decision Support Systems. IGI Global Publishing, Hershey, PA.

Refereed Conference Papers 1. Osop, Hamzah & Sahama, Tony (2016). Electronic Health Records: Improvement to Healthcare Decision-making. Paper presented at IEEE Healthcom 2016 Conference (Healthcom’16), Munich, Germany.

2. Osop, Hamzah & Sahama, Tony (2016). Quality Evidence, Quality Decisions: Ways to Improve Security and Privacy of EHR Systems. Paper presented at IEEE Healthcom 2016 Conference (Healthcom’16) 3rd International Workshop Reliability of eHealth Information Systems, Munich, Germany

3. Osop, Hamzah & Sahama, Tony (2017). Doctors’ perception of the potential of EHR: A Singapore insight. Paper presented at the 11th Australasian Conference on Health Informatics and Knowledge Management (HIKM 2018), Brisbane, Australia.

Conference Presentations (posters)

1. Osop, Hamzah & Sahama, Tony (2015). Effective clinical decision- making from practice-based evidence. Paper presented at 15th International Conference on Advances in ICT for Emerging Regions (ICTer2015), Colombo, Sri Lanka.

A Practice-Based Evidence Approach for Clinical Decision Support xv

“Twenty years from now you will be more disappointed by the things that you didn’t do than by the ones you did. So throw off the bowlines. Sail away from the safe harbor. Catch the trade winds in your sails. Explore. Dream. Discover.” - MARK TWAIN

xvi Feasibility of Practice-Based Evidence for Clinical Decision Support

Introduction

“It is health that is real wealth and not pieces of gold and silver.” - MAHATMA GANDHI

This chapter begins with the conceptualisation of this research project. Section 1.1 presents the background of the research and Section 1.2 provides its context. Research purposes, aims and objectives, questions and hypotheses are illustrated in Section 1.3, while the significance and scope of the research are discussed in Section 1.4. This chapter concludes with the details of the remaining chapters in Section 1.6, outlining the structure of this thesis.

1.1 BACKGROUND

The growing complexity and dynamism of the healthcare industry creates a critical need to make effective and well-informed clinical decisions. The ability to make quality decisions helps to ensure that care provided to patients is both safe and appropriate. It is therefore not a surprise to witness the widespread adoption of Information and Communication Technology (ICT) within healthcare industries, as it is the conduit towards the enablement of effective delivery of care. ICT implementation can assist healthcare professionals (HCPs) with healthcare decision making as well as improving the overall quality of care provided to patients (Buntin et al., 2011; Chaudhry et al., 2006; Cresswell & Sheikh, 2015).

Through ICT, the utilisation of various hardware devices and software applications is bringing massive potential to the delivery and management of patient care. ICT systems facilitate capturing and storing important healthcare related data, integrating disparate sources of information and analysing integrated data to assist with decision making. Such is the potential of ICT that it has the possibility of transforming healthcare delivery and management into something much more effective and efficient. To assure that current and future healthcare ICT implementations continue to provide better care and outcomes, global standards and certification bodies for healthcare ICT implementation such as the international Health and Information Management System Society (HIMSS) have been set up. This society helps ensure that healthcare organisations adhere to well-developed guidelines by having their ICT

Chapter 1: Introduction 1

implementations and competencies validated through certifications. Similarly, national agencies such as the Australian Digital Health Agency (ADHA), formerly known as Australian National E-Health Transition Authority (NEHTA), and Singapore’s Ministry of Health (MOH), or national policies like the United States’ HITECH Act, govern each country’s proper development, implementation and use of electronic health records and systems to ensure better healthcare management throughout. Internationally recognised standards such as ISO TC 215 regulate healthcare device communications, terminologies used in clinical systems, and the list of functions to be present in EHR systems (ISO, 2018). Nationally, guidelines such as Australia’s Digital Hospital Handbook (SA HB 163:2017) provide guides to the implementation of “good governance, engagement and process” (The Australian Hospital Healthcare Bulletin, 2018).

While ICT adoption is seen as key to delivering effective care to patients (Cresswell & Sheikh, 2015), the approach to the practice of clinical medicine is equally, if not, even more important. The current practice of clinical medicine is directed by Evidence-based practice (EBP) or Evidence-based medicine (EBM). EBP utilises evidence from systematic research to assist in clinical decision making. The evidence is generally elicited through systematic studies, such as the outcomes from randomised controlled trials (RCTs). Through such studies, EBP results in care that is both empirically grounded and safe (Greenhalgh et al., 2014, p. 1). An example of EBP is the establishment of well-developed clinical guidelines that standardise care, which healthcare professionals can adhere to. By following these guidelines, healthcare professionals have well supported decision making, for example, being able to provide correct treatments or prescribe appropriate medications. However, healthcare is a dynamic industry. There is a global expectation that there will be an increasing prevalence of patients with chronic conditions and comorbidities. This prevalence has been attributed to the general population growth, in particular, the growing ageing population, and overall poor personal health management. While EBP is an approach appropriate to manage such issues, more can be done to handle this growing phenomenon. This is especially so when there are critics who question the supposed effectiveness of EBP. This ineffectiveness has primarily been attributed to the limitations brought about by the outcomes in RCT studies. RCT studies tend to produce small data sets that are not useful for analysis or the studies conducted hamper

2 Chapter 1: Introduction

useful findings by excluding specific groups of patients (such as those with comorbidities) from participating. Due to such limitations, research has begun into offering new or alternative approaches that can be easily adopted by healthcare professionals. Researchers have also begun to investigate improvements to the quality of evidence used for decision making.

ICT implementation has provided more and more healthcare professionals with the ability to collect and store patient-related healthcare information digitally, to manage and assist their patients with care delivery. The continued growth in the implementation and use of various clinical information systems suggests an area where useful data is already in abundance. Patients, more often than not, are treated by more than one doctor or specialist. Data is constantly entered into clinical information systems by such healthcare professionals. Therefore, when healthcare professionals are able to gain access to clinical information systems and relevant information documented by other care providers, this allows them to conveniently analyse the data available and subsequently make well-informed decisions. Unfortunately, integrated and comprehensive data may not be readily available and easily accessible in healthcare organisations. Most clinical information systems manage their own data sources, resulting in large isolated collections of digital health data or electronic health records across healthcare organisations (Osop & Sahama, 2016b). This has created an environment where the nature of ICT within healthcare organisations becomes “diverse and disparate”, with collections of data being frequently uncomprehensive. With clinical information systems primarily designed to cater to the needs of individual departments, they are usually proprietary-based, scattered all over the organisations and operating in “silos”. Therefore, they are sparsely integrated and do not share data with other information systems (Sahama & Croll, 2007). However, data from these systems regularly contain rich sources of administrative, clinical and medical information that could potentially be used as evidence of clinical practices to assist in decision making. Therefore, integrating these data can potentially improve the quality of evidence used in directing the practice of clinical medicine, which could lead to improved decision making and better delivery of care. Such an approach could complement some of the limitations highlighted about EBP.

Achieving data integration is necessary to share information and gain knowledge for improved decision making and healthcare delivery. Clinical Data Warehouses

Chapter 1: Introduction 3

(CDWs) are gaining wider use in the healthcare industry. CDWs overcome the obstacle of scattered and unintegrated healthcare data, thus meeting the demands of harmonising them. They become a place not only for healthcare professionals to access all clinical data gathered in the patient care process but to do so with vastly improved quality for real-time decision-making processes (Sahama & Croll, 2007).

With integrated data, a clinical decision support system (CDSS) can assist healthcare professionals in their ability to make well-informed clinical decisions. As a type of computer program, a decision support system is designed to deal with clinical data or knowledge that is intended to provide decision support to its users (Musen et al., 2014). Through the use of a CDSS, healthcare professionals are not only able to greatly improve care delivery and safety of care provision but also reduce the gap between the knowledge available and what can then be practised (Bates et al., 2003). For example, one of the three focuses to having an efficient CDSS is to provide patient- specific recommendations, supporting healthcare professionals with care provision by giving preventive reminders and correct drug dosages for medications. The use of decision support systems has been effective in improving the management of patients with chronic conditions such as the discovery of disease-disease relations (Onitilo et al., 2014) or disease-drug relations (Chen et al., 2008). Besides decision support systems, many recent studies have focused on the secondary uses of electronic health record (EHR) data. These studies include utilising information captured in EHRs to detect and screen patients with chronic conditions (Anderson et al., 2016). In other similar studies, information from EHR data can also be used to investigate the relationships between hospital lengths of stay in relation to costs and quality of care (Bowers & Cheyne, 2016). Therefore, the potential of utilising EHR data together with a clinical decision support system looks promising. Healthcare professionals are not only assisted with the ability to make well-informed decisions but are able to do so in an actual clinical environment, at the point-of-care.

As the limitations of outcomes from RCT studies continue to challenge the effectiveness of EBP, several researchers and academics have proposed an approach called Practice-based evidence (PBE). This approach has the potential of improving the applicability of evidence generated and, in turn, uses it for decision making. PBE focuses on the use of evidence drawn from routine practice settings rather than those generated from efficacy studies (Evans et al., 2003) such as RCTs. However, such PBE

4 Chapter 1: Introduction

approaches require setting up research networks of clinicians or conducting practical clinical trials in order to elicit the much-needed practical evidence, which can be financially costly. Therefore, with the possibility of integrating the vast collection of patients’ electronic health records, which are largely available in most healthcare organisations, and its potential in supporting evidence of actual clinical practices, a new Practice-based evidence approach to decision making, using EHRs as practical clinical evidence can instead be adopted.

Hence, in this study, we introduce a new Practice-based evidence approach to decision making. With the inefficiencies of EBP previously highlighted by the issues and limitations of evidence elicited from outcomes of systematic research, PBE instead aims at utilising existing clinical evidence captured from practices performed by healthcare professionals in actual clinical settings, such as EHRs, to assist and support in clinical decision making.

1.2 RESEARCH CONTEXT

The challenges in the healthcare domain relate to how healthcare professionals can be provided with meaningful information to make well-informed decisions. Numerous research studies conducted have highlighted growing concerns regarding the effectiveness of clinical evidence used to direct the practice of medicine, especially when applying these evidences to individual patients or patients with comorbidities. However, when data which has the potential to be used as an alternative source of clinical evidence is discovered, these data, unfortunately, tend to be isolated and are not shared with other information systems. It seems like a wasted resource as the data can be useful to healthcare professionals. The “disparate and diverse” information systems that can be found in most healthcare organisations mean that data about patients’ health tends to be incomplete, unintegrated and difficult to retrieve for analysis. This severely impacts healthcare professionals’ ability to make informed decisions, as they are unable to rely on conceivably high quality and comprehensive data or information to do so.

This research study therefore emanates from the need to support healthcare professionals with the capability of making well-informed decisions. For this reason, we identify a novel Practice-based evidence approach to decision making as having the potential of achieving that. In describing how such an approach can be

Chapter 1: Introduction 5

implemented and adopted, IT professionals and doctors from public hospitals in Singapore were identified as participants and collaborators in this study. With the rising prevalence of patients with chronic conditions, the study also concentrated on diabetes and diabetic patients.

This study begins with the conceptualisation of our new Practice-based evidence approach, its design, and evaluation of this new approach by healthcare professionals. Included in the design of the approach is the design and implementation of a clinical data warehouse and the development of a clinical decision support system. In the evaluation process of our Practice-based evidence approach to decision making for healthcare professionals, anonymised patient data from a public hospital in Singapore was provided for use with our clinical decision support system. The anonymised data was used to generate patient-related information that can be utilised to make decisions about patients with diabetic conditions.

1.3 PURPOSES

1.3.1 Aims & Objectives In Section 1.2, we identified the need for this research study. It established the need to improve clinical care practices so that it becomes relevant and applicable to current patient demographics population, where the majority are likely to be patients with comorbidities (Heng et al., 2010; Holzinger, 2005). Our Practice-based evidence approach therefore attempts to achieve precisely that. Hence for such an approach to be adopted in a complex domain such as healthcare, it is also essential that we review comprehensively the related issues pertaining to technologies and stakeholders involved in healthcare organisations. This forms the objectives of our research study.

This research study aims to evaluate, based on the opinions and viewpoints of doctors, our Practice-based evidence approach, implemented in a clinical setting, as a tool that assists healthcare professionals with clinical decision making. In meeting this aim, we identified six broad objectives that guided our research study as follows:

I. Define the components of our new Practice-based evidence approach to decision making (Stage 1 in Section 1.4.2 below).

II. Investigate and understand the current healthcare ICT infrastructure of one of Singapore’s public hospitals, to identify the feasibility of implementing a clinical data warehouse (Stages 1 & 2).

6 Chapter 1: Introduction

III. Implement a clinical data warehouse, if it does not currently exist, to integrate different data sources from clinical information systems presently used as identified in the ICT infrastructure of one of Singapore’s public hospitals (Stages 2 & 4). If one exists, develop a “data mart” design and perform a simulation to verify the feasibility of the existing data warehouse architecture.

IV. Evaluate the perceptions of healthcare professionals regarding the benefits of using EHR systems and the usefulness of EHR data as an alternative source of evidence for use in decision making (Stage 3).

V. Develop a prototype clinical decision support system that adopts our new Practice-based evidence approach to decision making. As a case study to demonstrate the capabilities of our PBE approach to support decision making, the prototype clinical decision support system assists by providing (1) patient-centric statistical analysis and (2) prediction modelling of the probable hospital length of stay (Stage 5).

VI. Evaluate qualitatively the effectiveness of our new Practice-based evidence approach to assist in clinical decision making (Stage 6).

The objectives above support the overall aim of our research study. These are integrating extensive data from different clinical systems that are scattered throughout the healthcare organisation using a data warehouse architecture, identifying EHRs as a suitable alternative source of evidence so that healthcare professionals can be supported with making well-informed decisions, and qualitative measurement of healthcare professionals’ capability to make well-informed decisions through the use of a clinical decision support system. Meeting these objectives contributes to the implementation and adoption of our conceptualised PBE approach. This approach has the potential to be effective and cost-efficient, as it utilises existing evidence of clinical practices captured in the form of electronic health records.

To achieve the objectives above, a research plan and research questions were developed. The different stages of the research plan are further illustrated in Figure 1, while the research questions are discussed in the next section.

Chapter 1: Introduction 7

1.3.2 Research Questions The fundamental or overarching question that needs to be answered in this research is:

“What are the ICT architecture and processes required for a new Practice- based evidence approach to assist healthcare professionals make well-informed decisions?”

To answer the fundamental research question above fully, we have broken it down into three sub-questions. Each sub-question has a set of corresponding objectives that are met throughout the thesis. The consolidation of the findings from all these questions forms the overall answer to the fundamental research question above. The research sub-questions (RQ) are as follows:

RQ1: What will be an effective Practice-based evidence approach to decision making in a clinical setting?

• Objective 1.1: Conceptualising and defining the components essential for a Practice-based evidence approach to decision making.

RQ2: What changes to the current state of a healthcare organisation’s ICT architecture are required to adopt a Practice-based evidence approach in assisting healthcare professionals with clinical decision making?

• Objective 2.1: Identify the current information systems used in healthcare organisations. • Objective 2.2: Identify the types of data stored in these systems.

• Objective 2.3: Identify how current organisation of information is implemented.

• Objective 2.4: Identify how a data warehouse can improve data integration for reliable use.

RQ3: What are the processes required that ensure a Practice-based evidence approach can assist healthcare professionals in clinical decision making?

• Objective 3.1: Identify the limits of our ability to make well-informed decisions.

8 Chapter 1: Introduction

• Objective 3.2: Identify how clinical evidence from electronic health records can be used to assist in decision making.

• Objective 3.3: Identify how a clinical decision support system can assist with making well-informed decisions. (Healthcare providers need assistance in managing the increased number of patients with chronic diseases. Such a requirement is overwhelming due to the extensive amount of data in the information systems.)

• Objective 3.4: Identify how a Practice-based evidence approach assists with decision making. (PBE promotes the use of clinically relevant evidence from practice settings).

By achieving these research objectives, we will have made a valuable contribution towards identifying an effective approach that assists healthcare professionals with useful data and information to make well-informed decisions. The research questions are therefore addressed accordingly throughout this thesis.

1.4 SIGNIFICANCE, SCOPE AND DEFINITIONS

This research study aims to provide both theoretical and practical contributions to the ICT healthcare domain. In theory, this study conceptualises the components of a new Practice-based evidence approach to decision making and envisions the ICT infrastructure suited to the new approach for Singapore’s public healthcare organisations such as hospitals and polyclinics. As a case study, we identified one out of the eight public hospitals in Singapore. In practice, a clinical decision support system implementing the new PBE approach was developed and evaluation of its ability to assist healthcare professionals in making well-informed decisions was performed.

1.4.1 Research Significance, Scope and Limitation The implementation of our Practice-based evidence approach to decision making is a significant move towards assisting healthcare professionals in making well- informed decisions. With healthcare professionals constantly needing to make important and appropriate decisions, they could do with sufficient data and information. Our PBE approach begins by presenting an ICT architecture that can be emulated by most healthcare organisations. This architecture enables the integration

Chapter 1: Introduction 9

of valuable digital health data which is normally dispersed throughout healthcare organisations. This integration approach creates a resource where comprehensive patient health data can be made easily accessible to healthcare professionals. This enables valuable findings to be uncovered from the health data and lets them be presented to healthcare professionals using a clinical decision support system. The CDSS thus becomes useful in assisting and supporting clinical decision making. Our PBE approach also has the potential to address the limitations of evidence-based guidelines and outcomes from RCT studies because the evidence used to direct decision making in PBE is based on actual health data that captures information pertaining to care provided to actual patient types. At the same time, our PBE approach addresses some of the grand challenges of developing an effective decision support system. This potentially provides significant contributions to the practice of clinical medicine.

However, the use of ICT to manage patients with chronic conditions is a very broad subject that requires reasonable effort in scoping it for this research study. Thus, our focus narrowed towards the effective implementation of our PBE approach through the use of electronic health records and a clinical decision support system. Essentially, this research study was conducted with the collaboration of two diabetic specialists from Singapore. This further helped to scope our research study, focusing on chronic diseases, in particular, diabetes management. While the umbrella of chronic diseases covers more than just diabetes, the findings from this research are still applicable to all other forms of disease-related conditions. We identified the utilisation of electronic health records as evidence because of its potential to support decision making. Findings such as being able to detect populations with chronic conditions, predict clinical outcomes, determine disease incidences or analyse drug reactions and interactions (Anderson et al., 2016; Bagley & Altman, 2016; Cox et al., 2016; Ibrahim et al., 2016; Jang et al., 2016) highlight the benefits of secondary uses of electronic health records. Therefore, we also investigated how findings from secondary uses of EHRs can be used to assist decision making. However, issues of data privacy and security were beyond the scope of this research as they entail both an elaborate data security architecture as well as development of policies for data privacy management, which are outside our expertise.

10 Chapter 1: Introduction

It is also pertinent that we highlight the perceived limitations in an exploratory study such as this. Given the availability of resources and domain experts, this research only aimed at implementing a working clinical decision support system prototype that implements our PBE approach through the utilisation of anonymised patient health data, provided by the collaborating doctors from one of the public hospitals in Singapore. Nonetheless, we are confident that the findings generalise readily to other similar situations.

1.4.2 Methodology This is a qualitative research study evaluating our Practice-based evidence approach to decision making, implemented through a clinical decision support system, to assist healthcare professionals in making well-informed clinical decisions.

In particular, we followed a qualitative case study research approach that enabled us to understand the current limitations of information systems, propose a new approach, and evaluate the potential of this new approach to improve decision making support. It allowed us to study and explore doctors’ knowledge and perceptions regarding our new approach, as well as understanding the ICT architecture of healthcare organisations (Baxter & Jack, 2008). These findings in turn helped to identify the relationships required for our PBE approach to decision making. To do so, three cases, identified in Section 3.2.1, were conducted sequentially and the outcomes of these studies helped to shape the findings of this research. The cases are also mentioned in the research stages described below.

Stage 1 – Contextual Literature Review to define our new Practice-based evidence approach to decision making. Stage 1 of this research study focused on the contextual preliminary review to define the research problem. Common limitations and issues with the use of ICT pertaining to making clinical decisions were identified at this stage. Academic research papers, scientific journals, conference papers, book chapters and industry white papers were employed during this review. Research and knowledge gaps were identified and used to formulate the research scope, objectives and questions. Findings from the literature review identified limitations and gaps that exist in current approaches to clinical decision making. Most importantly, the outcomes of this stage established the components of our new Practice-based evidence approach to decision making.

Chapter 1: Introduction 11

Stage 2 – Interviews to establish requirements for a data warehouse architecture supporting clinical decision making. In this stage, IT professionals were interviewed for information regarding the current ICT infrastructure in place at a hospital. Findings from the interview identified the range of clinical information systems currently used in the hospital, the types of data captured by these systems, how information is presented to system users and the existence of a data warehouse or data integration architecture. The outcome of this stage helped to establish the design of a data warehouse architecture for our new Practice-based evidence approach to decision making.

Stage 3 – A survey on doctors’ perceptions regarding electronic health records and EHR systems. In Stage 3, an online survey study was conducted to evaluate doctors’ perceptions regarding the benefits of using EHR systems and the usefulness of EHR data. Findings from the study explained whether EHR systems help in the management of care and if EHR data can be used as evidence of clinical practices captured for decision making. Therefore, the outcomes of this stage of the study helped establish the potential use of EHRs as a suitable source of evidence to support healthcare professionals’ decision making.

Stage 4 – Design a Clinical Data Warehouse (CDW). The outcome at this stage was the design of a clinical data warehouse for our PBE approach, based on findings regarding the ICT infrastructure uncovered in Stage 2. The designed CDW integrates data from multiple systems currently used in the case-study hospital and was an important aspect of the implementation of our PBE approach to decision making. To verify the feasibility of the CDW architecture design, a simulation was conducted.

Stage 5 – Implement a Decision Support System (DSS) Prototype. A prototype decision support system was designed, developed and implemented during this stage of the study. Integrated data from the data warehouse designed in Stage 4 was extracted and fed to the prototype CDSS as the source of evidence of clinical practices. The prototype decision support system assists healthcare professionals with decision making through patient-centric statistical analysis and prediction modelling. The outcome of this stage was the development of a clinical decision support system that illustrates our Practice-based evidence approach to support clinical decision making.

Stage 6 – Final Evaluation & Review of our PBE approach to decision making. In this stage, the decision support system implementing our PBE approach to

12 Chapter 1: Introduction

decision making was demonstrated to collaborating doctors. As a case study to illustrate our PBE approach, patient-centric information to assist with decision making was presented visually in the prototype CDSS in the form of detailed statistical analyses and a model that predicts the probable length of hospital stays. Following the demonstration, a focus group discussion was conducted. Findings from the focus group discussion disclosed the perceptions of doctors regarding the feasibility, usability, limitations and potential of adopting our PBE approach in assisting and supporting decision making during actual clinical practices. The outcome of this stage of the study was a confirmation that our Practice-based evidence approach to decision making, using a prototype CDSS and EHRs as alternative sources of evidence that capture clinical practices performed by healthcare professionals, can assist them in making well-informed clinical decisions that could improve the delivery of care to their patients.

1.4.3 Research Plan We also drew up a research plan that identified the objectives and corresponding research questions that needed to be answered. This was to ensure that our research questions were answered thoroughly by the objectives that we identified. Furthermore, each objective was also linked to the different stages of our research methods to ensure that appropriate research methodology was conducted, to gain the correct outcomes.

A diagrammatic version of the research plan is illustrated in Figure 2.

1.5 RESEARCH CONTRIBUTION

Following the execution of the research stages and plan mentioned in the previous sections, findings from our research study conducted provided a number of theoretical and practical contributions within the domain of healthcare ICT.

First, our research study has contributed to the body of knowledge by introducing a new Practice-based evidence approach that assists healthcare professionals with decision making. Secondly, in implementing our PBE approach to decision making, our study has made a practical contribution to designing an ICT architecture that is based on existing and current healthcare organisations’ ICT infrastructure. Thirdly, by employing the use of a clinical decision support system and actual patient health data as tools in our PBE approach to assist with decision making, our research study has provided a case study for a possible real-world application.

Chapter 1: Introduction 13

Detailed descriptions of the research study conducted, data collected, and findings uncovered are presented in the coming chapters as highlighted in the thesis outline.

14 Chapter 1: Introduction

Stage 1: Contextual Literature to Inform our New Practice-Based Evidence Approach to Decision Making

Limitations & issues of Practice-Based Evidence Data integration current clinical Clinical decision-making Approach architecture information systems

Stage 2: Interview to Establish Requirements for a Data Warehouse Architecture Supporting Clinical Decision Making

Interview with healthcare organisation's IT experts to determine the current ICT infrastructure and feasibility of introducing a data warehouse architecture to integrate data

Stage 3: Survey on Doctors' Perceptions Regarding EHR and EHR systems

Survey on doctors' perceptions regarding the perceived clinical benefits of using EHR systems and usefulness of data captured in EHR to be used as evidence for decision making support. And identifying the suitability of EHR to be used in our PBE approach.

Stage 4: Design of a Clinical Data Warehouse (CDW)

Design CDW architecture for our PBE approach Simulate data integration based on the designed based on existing ICT infrastructure CDW architecture for use in CDSS

Stage 5 : Implement a Decision Support System (DSS) Prototype

Design DSS prototype Implement DSS together with CDW to Develop DSS prototype development assist with decision making

Stage 6: Final Evaluation & Review of our PBE Approach to Decision Making

Support decision-making Support decision making Demonstrate our PBE Evaluate our PBE through statistical through prediction of approach through DSS approach analysis LOS

Figure 1. Research Stages

Chapter 1: Introduction 15

Figure 2, Research plan identifying how research questions are answered through the objectives identified and the corresponding methods to achieve them

16 Chapter 1: Introduction

1.6 THESIS OUTLINE

The remainder of the thesis is outlined as follows.

Chapter 2 – Literature Review

This chapter details the problems and issues faced by healthcare professionals and their organisations, especially in the realm of effective decision making. This chapter builds the background surrounding the conceptualisation of our Practice-based Evidence approach to effective clinical decision making.

Chapter 3 – Research Design and Methodology

This chapter discusses the research collaboration, design and methodology for the four related research studies conducted in this thesis. The chapter ends with the discussion on the ethics and limitations of this research.

Chapter 4 – Conceptualising Our New Practice-Based Evidence Approach

This chapter discusses the first of four studies conducted. The chapter details the conceptualisation of our Practice-based evidence approach to decision making, which aims at assisting healthcare professionals in their ability to make well-informed decisions, utilising electronic health records as best evidence of clinical practices gathered during actual clinical practice. This chapter also discusses the model for implementing our PBE approach.

Chapter 5 – Architecting Our New Practice-Based Evidence Approach

This chapter focuses on the interview procedures and participants of the interview. In this chapter, we uncover the ICT infrastructure of the collaborating hospital, National University Hospital (NUH) in Singapore, and design the corresponding data warehouse architecture supporting our PBE approach.

Chapter 6 – Perceived Clinical Benefits of EHR Systems and Perceived Usefulness of EHR Data for Decision Making

This chapter discusses the survey study conducted with doctors, regarding their perception of the benefits of using EHR systems and usefulness of EHR data for decision making. It presents the development and deployment of survey procedures, detailed findings from the survey responses and ends with the suitability of EHRs to be used as evidence in our PBE approach.

Chapter 1: Introduction 17

Chapter 7 – Evaluating Our New Practice-Based Evidence Approach

This chapter evaluates the potential of our approach in practice and begins with an overview of the research study components. This is followed by the data preparation phase, prototype CDSS design and development phase, and our PBE approach evaluation. A case study is conducted with a clinical system demonstration to illustrate the adoption of our PBE approach and how it can assist healthcare professionals in making well-informed decisions. This facilitates the evaluation of our PBE approach to support decision making in a clinical setting. This chapter ends with a detailed summary regarding the analysis and findings from the evaluation conducted in a focus group discussion and interview.

Chapter 8 – Discussion & Conclusion

This chapter reflects on the research findings, contributions, research limitations and plans for future work.

18 Chapter 1: Introduction

Literature Review

“A good decision is based on knowledge and not on numbers”. - PLATO

This literature review chapter discusses the preliminary investigation conducted in the implementation of our new Practice-based evidence approach. The chapter begins with the background to the research in Section 2.1. This is followed by a review of the chronic disease prevalence (Section 2.2) and the current ICT adoption in the healthcare industry in Section 2.3. Next is the introduction of the current paradigm of clinical care, Evidence-based practice in Section 2.4. The chapter continues with the introduction and background of a complementary approach to clinical care, Practice- based evidence in Section 2.5. This is followed by a review of Electronic Health Records (Section 2.6), Clinical Data Warehouses (Section 2.7) and Clinical Decision Support Systems (Section 2.8). The chapter ends with a summary and implications.

2.1 BACKGROUND

The increasing prevalence of chronic diseases among patients is quickly becoming a global healthcare issue. Such prevalence has been attributed to issues such as increased healthcare costs and poor quality of life, and thus, considerable action needs to be taken to improve the delivery of care.

Effective clinical care can positively improve patients’ health outcomes, be cost- effective and provide care in a much safer manner (Greenhalgh et al., 2014). One feature of effective clinical care delivery is the ability to make well-informed decisions. Healthcare professionals, such as doctors and clinicians, routinely encounter episodes of clinical care, where the right decisions made can mean the difference between improved or deteriorating health conditions. However, the ability to make sound clinical decisions is a challenge. These challenges are brought about as treatments and care for patients become more complicated and complex (Akyürek et al., 2015; Hunink et al., 2014).

Hence, the use of healthcare related Information and Communication Technologies (ICT) such as Electronic Health Record (EHR) systems, Computerised

Chapter 2: Literature Review 19

Physician Order Entry (CPOE) and Laboratory Information Systems (LIS), has become an essential mechanism in the effort to improve the delivery of healthcare services to patients (Buntin et al., 2011). The potential benefits of implementing ICT are numerous, such as the ability to improve the health of patients (Buntin et al., 2011), reduce medical errors (Cresswell & Sheikh, 2015; Goldzweig et al., 2009) and reduce healthcare costs (Buntin et al., 2011; Cresswell & Sheikh, 2015; Goldzweig et al., 2009). Despite these apparent benefits, there are studies that argue the total clinical benefits arising from ICT adoption are still minimal. As a result, the focus has now shifted towards the huge collection of digital health data generated from various information systems implemented across healthcare organisations and how healthcare organisations can make full use of them to improve patient care and delivery.

These huge collections of digital health data represent an enormous opportunity for healthcare organisations to discover new medical and clinical findings. Nonetheless, it also presents a considerable obstacle. Without the help of information technologies, it will be impossible for decision-makers to analyse and make decisions just by looking at raw data (Akyürek et al., 2015). So, with big data analytics becoming a trend in discovering new findings, healthcare analytics takes on the same approach. Healthcare analytics can be a solution that is potentially more supportive of clinical care delivery. Since health data contains rich information about patients’ health and medical conditions, it can be used to shape and inform future clinical practices. Furthermore, by having access to all this information and relevant data in a timely manner, it can also assist healthcare professionals with improved decision making (Akyürek et al., 2015).Therefore, in taking advantage of the availability of digital health data, the benefits of using healthcare analytics and ultimately improve decision making, a Practice-based evidence approach is introduced. PBE aims at improving decision making through the use of meaningful practical evidence (Barkham & Margison, 2007; Evans et al., 2003).

Our narrative literature review focuses on a Practice-based Evidence approach to clinical decision making implementation involving electronic health records, data warehouse architectures and decision support systems. The literature reviewed is from 1992 to 2017. The publications were sourced from many library databases including Science Direct, ACM Digital and IEEE. The literature review provides the background on current issues associated with the prevalence of chronic diseases, the approach

20 Chapter 2: Literature Review

adopted by healthcare professionals for decision making and treatments, understanding the role clinical information systems play alongside healthcare professionals and tools that improve decisions.

The literature review is analysed according to five themes of Chronic Disease Prevalence in General Cases (Section 2.2), ICT Adoption in Healthcare (Section 2.3), Evidence-based Practice (Section 2.4), Practice-based Evidence as a Complementary Paradigm to EBP (Section 2.5) and Electronic Health Records (Section 2.6) and two methodologies on Clinical Data Warehouse (Section 2.7) and Clinical Decision Support System in Healthcare (Section 2.8) which highlight the implications from the literature and develop this thesis study.

2.2 CHRONIC DISEASE PREVALENCE IN GENERAL CASES

Chronic disease prevalence is a growing healthcare concern that is impacting both developing and developed countries. According to the data on Global Health Observatory, deaths caused by chronic diseases worldwide have totalled 39.5 million in 2015 (World Health Organization, 2018). A total of 81% of these deaths have been attributed to four major groups of chronic diseases; cardiovascular disease, cancer, chronic respiratory diseases and diabetes (World Health Organization, 2018). By the year 2030, it is predicted that the (annual) total death count will have risen to 52 million. In Australia alone, chronic diseases have been responsible for 91% of all deaths while in Singapore, the figure stands at 76% (World Health Organization, 2015). The burden of chronic diseases spans beyond human and social consequences. Individuals, organisations, as well as society at large, are equally affected by the financial strains brought about by this prevalence (Ng et al., 2015).

Chronic diseases, in particular, the global diabetes mellitus burden, has been described by Hu et al. (2015) as “pandemic” and has affected over 380 million people worldwide. This number is expected to grow to 592 million people by the year 2035. However, the growth in the number of people diagnosed with diabetes will come as no surprise, given the increase in the general population growth, aging population, poor personal health management such as a lack of physical activity, smoking and alcohol consumption, and urbanisation (Chen et al., 2012; Su et al., 2016; Yeo et al., 2012). As of 2010, 11.3% or about 426,200 Singaporean adults aged 18 to 69 years, have been diagnosed with diabetes (Ministry of Health Singapore, 2010). In fact, recent

Chapter 2: Literature Review 21

estimates revealed that the prevalence of diabetes in Singapore rose to 12.3% in 2013 (Ng et al., 2015) and 13.7% in 2017, according to the International Diabetes Federation (2017). In Australia, the diabetic population is approximated to be around one million Australians (Australian Institute of Health & Welfare, 2014) which is about 4% of its total population (Australian Institute of Health & Welfare, 2015). However, the Australian diabetic population may well be an underestimate, as data from International Diabetes Federation suggests that the percentage of adults with diabetes undiagnosed stands at 45.8%. The undiagnosed rate of people with diabetes in Singapore, on the other hand, is about 46.9% (International Diabetes Federation, 2015), higher when compared to Australia. Worryingly, with the anticipated number of people with diabetes set to increase, likewise, the increasing number of patients with associated diabetic health risks such as obesity, sleeping disorder and depression (Chen et al., 2012). Furthermore, if left undiagnosed or untreated, patients with diabetes tend to suffer from other health problems such as heart disease, kidney failure or stroke. Therefore, it will persist to be a worrying trend if patients continue to not receive proper treatments.

Diabetes and diabetic-related complications, such as kidney or eye diseases, and health risks, such as comorbidities or deaths, have been studied and are found to be responsible for the considerable need for healthcare expenses and resources (Su et al., 2016). This expenses and resources range from patients having to go through routine check-ups, health reviews, hospitalisation episodes, to treatments as well as medication prescriptions. The predicted increase in the diabetic population is instigating global social and financial strain, therefore managing patients with diabetes becomes a prime concern. In a study on the direct costs of Type 2 diabetes conducted by Ng et al. (2015), which was based on a final sample size of 500 diabetic patients, the calculated mean total costs for diabetic patients with at least one inpatient admission was SGD 8,787.80, with about 84.8% (SGD 7,453.30) of that cost relating only to inpatient services (refer to Appendix A) (Ng et al., 2015). In 2008-09, the diabetes expenditure in Australia amounted to AUD 1.5 billion, with 42% of the expenditure spent on hospital admissions alone. Globally, the estimated annual global health expenditure for diabetes is USD 612 billion, an increase of 12% from the previous year. This increased in expenditure was attributed to a rise in the total number of people with diabetes, from 382 million in 2013 to 387 million in 2014 (da Rocha

22 Chapter 2: Literature Review

Fernandes et al., 2016). Therefore, expenditure is expected to rise even further in the future unless improvements in the delivery of care is achieved.

Besides being responsible for the compelling mortality and associated health risks mentioned above, patients with diabetes are also likely to develop comorbidities (Su et al., 2016), which is defined as the “coexistence of two or more chronic conditions” (Salive, 2013, p. 1). For example, diabetic patients may have comorbidity or develop conditions with multiple diseases such as hypertension, chronic kidney disease or dyslipidaemia. Following a study conducted on one of Singapore’s healthcare cluster diabetes registry by Heng et al. (2010), more than half (50.2%) of all the Type 2 diabetic patients from the registry were those aged 45 to 64 years. A staggering 98.5% of these Type 2 diabetic patients had at least one diabetes-related comorbidity. In the same year, the prevalence rate of comorbidity for patients with diabetes, hypertension and dyslipidaemia in the primary care clinics was 77.7% and in the specialist outpatient clinic at 69.2% (reference in Appendix B) (Heng et al., 2010). In a 2011-12 Australian Health Survey study, statistics revealed that, for every 10 Australians, there were two Australians that had two or more chronic diseases. Furthermore, comorbidity cases are becoming more common as patients age. About 39% of Australians aged 45 and above were identified as having at least two cases of chronic disease (Holzinger, 2005). Moreover, patients with comorbidities, are at a higher risk of deaths, disability, hospitalisation, and adverse drug reactions and would likely have a poorer quality of life (Falsetti et al., 2016; Salive, 2013). The statistics above show compelling diabetes-related issues faced by patients, and the impacts it can have on healthcare systems.

With the possibility of longer-term mortality and an increasingly aging population, comorbidity is also considered as a reflection of the quality of health one possess (Falsetti et al., 2016). Patients with comorbidities are usually associated with the increased rate of adverse events such as mortality. Some of them may be subjected to complex treatments, increasing the risk of other adverse effects such as drug-drug interaction (Falsetti et al., 2016). Additionally, comorbidity is also related to increased frailty, disability, the risk of hospitalisation and death, which are further signs of poor health quality correlated with long-term mortality (Falsetti et al., 2016). Therefore, while patients get to live longer, they may instead develop multiple chronic conditions that show the poor state of health they are actually in. Therefore, it is not a surprise

Chapter 2: Literature Review 23

that patients with comorbidities tend to associate with a greater need for healthcare resources, which inevitably contributes to the overall healthcare expenditure. As illustrated by earlier statistics, the increase in the aging population may also be a sign indicating the increase in number of patients with comorbidities. Therefore, it is imperative that healthcare decision making considers patients with comorbidities so that decisions made becomes applicable and thus more effective.

Figure 3. Percentage of all deaths by NCDs or chronic diseases in Australia and Singapore adapted from World Health Organization (2015)

Comorbidity, as an indicator to the presence of multiple chronic conditions, is also considered to be one of the predictors for sustained high healthcare costs (Charlson et al., 2014). This can be determined using the Charlson Comorbidity Index or CCI. CCI was initially developed as a weighted index to predict the mortality of patients (Charlson et al., 1987). This weighted index references the number of comorbidities and the level of seriousness of the comorbid diseases of a patient. The greater the comorbidity level, the higher is the probability of mortality for the patient. However currently, CCI has a much wider area of application and is extensively studied as a predictive factor for high hospital charges (Charlson et al., 2014). Therefore, using this predictive index, efforts in reducing resource utilisation and even improvement to care outcomes can be focused towards patients with multiple chronic conditions.

Healthcare policy makers are finding ways to reduce the healthcare expenditure, especially when the budget is scarce and limited. Schnipper et al. (2012, p. 1715) reveal that the reason behind the rise in the cost of healthcare in the United States is

24 Chapter 2: Literature Review

partly attributed to the “unnecessary use of healthcare resources”. This refers to the “unnecessary tests, procedures, physician visits, hospital stays and other services that do not improve a patient’s health” (Schnipper et al., 2012, p. 1715). Furthermore, Hillestad et al. (2005) iterate that due to the lack of use of information systems, with most records still in paper form, has resulted in information being inaccessible and a challenge when used in coordinated care for patients with chronic conditions. Consequently, decision making becomes less effective and thus may be a contributing factor to increased healthcare cost.

While both sets of authors highlighted different examples as to how expenditure has risen, both have also underlined the lack of information being made available and accessible to healthcare decision makers as a contributing factor. The lack of availability and accessibility of information have in a way, limited the ability for healthcare professionals to make well-informed decisions. In addition, when such healthcare professionals are unable to provide patients with the necessary information, the patients in turn, are unable to make informed decisions about their care and thus unwittingly become “participants” of the unnecessary use of healthcare resources as highlighted by Schnipper et al. (2012). Therefore, if information can be made available and accessible to all healthcare professionals seamlessly, the issue of unnecessary utilisation of healthcare resources may be easily overcome. For that reason, information systems have been designed to provide information to healthcare professionals more efficiently as compared to the paper-based records. Therefore, maybe the availability of not just information, but comprehensive and valuable information, can make a difference to how healthcare resources can be efficiently utilised or how patients can receive proper and appropriate treatments.

Fortunately, with the increasing and widespread adoption of ICT across healthcare organisations, efforts are on the way to address issues highlighted above. This is extensively reviewed in Section 2.3. Clinical information systems implemented in healthcare organisations aim at supporting healthcare professionals in providing the best treatment possible to individual patients, given the unique circumstances surrounding them. However, comorbidities and disease-related complications like those experienced by diabetic patients are making treatment rather complex and require more than just clinical evidence and, doctors’ expertise and experiences to be able to make effective decisions. While outcomes from systematic studies supposedly

Chapter 2: Literature Review 25

generate quality clinical evidence, in most cases, they may be limited or inapplicable of supporting decision making especially for those with comorbidities. Electronic health records, on the other hand, contain valuable clinical information about numerous patients with diverse conditions, including those with comorbidities. Essentially, EHRs may also contain evidence of clinical practices regarding similar past patients that can be used as evidence to inform decision making. However, querying from large collections of EHRs is not entirely convenient, plus issues of data quality and comprehensiveness may limit the usefulness of them as evidence. Thus, the relevance of our new Practice-based evidence approach, incorporating the ICT architecture and processes required to assist decision making efficiently. This will be comprehensively covered in the sections that follow.

2.3 ICT ADOPTION IN HEALTHCARE

The adoption of Information and Communication Technologies (ICT) in healthcare has the potential to improve the quality of healthcare services and reduce burgeoning healthcare expenditure, amid the increasing prevalence of patients with chronic conditions.

With this increasing prevalence, the care for patients has become both complex and expensive. As highlighted in Section 2.2, patients with chronic conditions such as cardio-respiratory diseases or diabetes have been partly responsible for the increase in annual healthcare service expenses (Haluza & Jungwirth, 2015). Therefore, the immense growth in the adoption of ICT within the healthcare industry comes as no surprise.

As healthcare organisations and policymakers seek to find ways to reduce increasing healthcare costs and reduce healthcare resource utilisations, the majority of them have turned to ICT to improve the quality and efficiency of care (Adler-Milstein et al., 2014; Agha, 2014). The healthcare industry has also been relatively late in leveraging the potential of ICT adoptions. Therefore the adoption of ICT has been described as a strategy that has helped to modernise and improve the delivery of healthcare. Healthcare ICT adoption has been responsible for improving the quality of care, on top of managing healthcare costs and expenditure. It is also driven by the need to improve patient safety which arises from the increased diversity in both patient types and patient care (Bélanger et al., 2012). The improvements in care quality and patient

26 Chapter 2: Literature Review

safety have been observed through health information system implementations that support medical error prevention, redundant test reductions and overall health outcomes improvement (Agha, 2014; Cresswell & Sheikh, 2015). The adoption of ICT is achievable in various ways. Most commonly, healthcare organisations adopt ICT through the use of a myriad of clinical information systems such as electronic health record (EHR) system and clinical decision support systems (CDSS), employing information exchange protocol such as health information exchange (HIE) and management and engagement tools like telehealth and patient self-management tools (Adler-Milstein et al., 2014).

In the use of clinical information systems such as EHR systems, patients’ medical and personal information, as well as doctors’ notes, are electronically maintained in secured databases. These databases allow for efficient storage, retrieval and access by multiple authorised users, as compared to the use of traditional paper- based records (Agha, 2014; Bélanger et al., 2012; Häyrinen et al., 2008). In fact, the use of a paper-based record system makes information retrieval cumbersome and complex, as it is physical in nature, and takes up storage spaces across healthcare organisations (Sahama et al., 2013).

Besides being able to track patients’ health over a period of time through an EHR system, doctors are also able to access inputs from other consulting doctors. This allows the doctors to gain an accurate understanding of a patient’s actual medical condition. The accessibility and availability of such comprehensive information enable doctors to make clinical decisions which will be more informed (Agha, 2014). Computerised Physician Order Entry (CPOE), on the other hand, is a system that has been designed to help doctors manage medication prescriptions, and laboratory and radiology tests (Bélanger et al., 2012). When EHR and CPOE systems are used together with a clinical decision support system, the CDSS is able to provide advice and prompts to doctors according to clinical guidelines. For example, the use of CDSS offers timely reminders and information to doctors by recommending screening tests, and identifying drug-drug interaction as well as drug allergy information, all with the purpose of improving the quality of care through well-informed decisions (Agha, 2014; Bélanger et al., 2012). In fact, there have also been studies that show, EHR systems, especially those equipped with effective CDSS, have been associated with improved quality of care (Poon et al., 2010).

Chapter 2: Literature Review 27

Over the years, the advances in ICT have also enabled more computing capabilities and power to be harnessed. Predictive analytics, machine learning and artificial intelligence (Chen et al., 2017; Chennamsetty et al., 2015; Qureshi & Gupta, 2014; Raghupathi & Raghupathi, 2013; Wills, 2014) represent some of the latest manner in which, ICT enables the drawing of usable inferences from raw healthcare data without the need for input from healthcare professional themselves. Inferences drawn through these computational approaches can assist healthcare professionals in various ways, such as diagnosis, treatment, health management and preventive care (Raghupathi & Raghupathi, 2013). Some of the examples of inferences that can be drawn include disease detection, predicting the onset of diseases or medication recommendations. Especially with the increasing prevalence of patients with comorbidities, the need for effective personalised medicine or personalised care (Coulter et al., 2013) can greatly benefit from such advances in ICT capabilities. This helps to further enhance the care of individual and unique patients. Valuable statistical analytics and information regarding individual patients can also be generated effectively from the vast collection of raw data available. Therefore, the approaches above are capable of assisting healthcare professionals’ in decision making. For example, in personalised cohort studies, where “data regarding past similar patients are queried to inform decision making” (Gallego et al., 2015), inferences drawn from such data can help identify relevant and applicable interventions that have been successfully performed on similar patients. Subsequently, the same interventions can therefore be performed or applied safely to other similar patients.

Overall, better utilisation of ICT has the potential to improve the quality of care (Buntin et al., 2011; Chao et al., 2013; Cresswell & Sheikh, 2015), the efficiency of healthcare providers in providing care (Buntin et al., 2011), promote patient safety (Cresswell & Sheikh, 2015), improve patient health outcomes (Buntin et al., 2011; Cresswell & Sheikh, 2015), allow for cost saving measures (Buntin et al., 2011; Chao et al., 2013; Haluza & Jungwirth, 2015) and empower patients with their own health engagement (Buntin et al., 2011; Haluza & Jungwirth, 2015). While initial implementation of ICT can be considered to be expensive, the benefits and improvements that it brings later to patient care delivery show that ICT has the potential for a high cost-saving in the future (Haluza & Jungwirth, 2015).

28 Chapter 2: Literature Review

Naturally, the increased adoption and reliance on ICT has resulted in the implementation of varying types of clinical information systems throughout healthcare organisations. However, many of the stand-alone information systems implemented, poorly communicate with other systems (De Mul et al., 2012). It may be due to information systems being developed in different programming languages and environment, that lacked in application and programming interfaces to allow for data to be shared or transferred, or simply because the systems make use of proprietary databases. This, unfortunately, creates an environment of disparate and diverse systems (Sahama & Croll, 2007) which inhibits systems communication and sharing of information. For example, according to Chen et al. (2014), online self-management information systems, which are integrated with biometric sensors, are often isolated and seldom connected, as the information systems and databases are usually scattered. While the benefits of implementing and utilising ICT systems are aplenty, they have undoubtedly created some challenges which impede the direct gain to improve patient care further.

2.4 EVIDENCE-BASED PRACTICE

Effective clinical decision making is the essence of efficient delivery of care, making it one of the fundamental requirements of healthcare professionals. However the clinical decision-making process is not one that is simple to make, in fact, it has been described as stressful by Akyürek et al. (2015) because it is a combination of processes that are both complicated and complex (Akyürek et al., 2015; Hunink et al., 2014). The decision-making process involves the need to consider multiple complex interrelated factors that can be both clinical and non-clinical in nature (Akyürek et al., 2015). An example of such interrelated factor is, having to consider the “therapeutic and diagnosis uncertainties” together with “patients’ needs and values as well as costs” (Hunink et al., 2014, p. 1).

However, decision making can be improved when factors of availability and accessibility of information are addressed (Akyürek et al., 2015; Tunis et al., 2003). The availability of data results in the availability of evidence and knowledge, which improves the quality of decision making. Equally important is having timely access to information and information that is organised. The depiction in Figure 4 shows how these factors have a positive reinforcing effect towards improving the quality of decision making (Osop & Sahama, 2016a).

Chapter 2: Literature Review 29

Figure 4. Factors influencing the quality of decision making through a causal-loop diagram (Osop & Sahama, 2016a)

Also, for healthcare professionals to truly make effective clinical decisions, they have to be equipped with the necessary skills, values and knowledge. Therefore, the aim of introducing the paradigm of Evidence-based practice is also to encourage healthcare professionals to incorporate the use of clinical evidence together with their individual expertise when making decisions about providing care to respective patients.

2.4.1 Evolution of Evidence-based Practice Prior to the formal conceptualisation of Evidence-based practice (EBP), the requirements needed to guide the practice of medicine and care by clinicians were based on four simple assumptions. According to Guyatt et al. (1992), these four assumptions that guided the practices for clinicians, could be achieved simply by (1) being able to build and maintain knowledge regarding patient prognosis, value of diagnostic tests and the effectiveness of treatment through unsystematic observations of clinical experiences, (2) having had sufficient study and knowledge of basic mechanisms of disease and pathophysiologic principles, (3) having a combination of traditional medical training and basic common sense to effectively evaluate new tests and treatments, and (4) being a content expert and having clinical experience, as satisfactory requirements to develop credible clinical guidelines. In essence, when faced with clinical problems, clinicians looked for answers either by relying on their

30 Chapter 2: Literature Review

prior clinical experiences, the knowledge and the understanding they had acquired regarding diseases, through medical literature or by eliciting the views of fellow clinical experts. While this practice might have seemed reliable at that point of time, some level of “flaws of biases” would have been present, questioning its integrity as an effective approach to management of care (Hoffmann et al., 2009). Consequently, in the early 1990s, a conscientious shift started towards a new scientific paradigm, which incorporated the use of systematic research evidence, clinical expertise and the preferences and needs of patients.

A scientific paradigm, according to Thomas Kuhn as cited by Guyatt et al. (1992, p. 2420), was described as “ways of looking at the world that define both the problems that can legitimately be addressed and the range of admissible evidence that may bear on their solution”. In short, Evidence-based practice was considered as a paradigm shift, because the developments in clinical research were generating new undeniable evidence that was helping to improve clinical practices, such that the old approach to guiding clinical practice was no longer viable and justifiable.

However, before the practice was formally termed as Evidence-based practice, it actually started out as Evidence-based medicine (EBM) in 1992 when Guyatt et al. (1992, p. 2420) strongly emphasised the need to have reference to evidence from clinical research about the use and reliance of doctors’ “intuition, unsystematic clinical experience and pathophysiologic rationale”, to guide clinical decision making. It was described as a process where healthcare professionals needed to acquire new skills for efficient literature searching and for applying the evidence to guide clinical practice.

In fact, even as early as the year 1061, the practice of medicine had already shown signs of relying on some level of evidence, to support decision making and direct clinical practices. According to Claridge and Fabian (2005), during the period of the Song Dynasty, Ben Cao Tu Jing intended to evaluate the effectiveness of ginseng. He ordered the finding of two people and had each of them run; one after consuming ginseng, while the other without. According to him, the person who was likely to develop shortness of breath would be the one who had not taken any ginseng (Claridge & Fabian, 2005). The use of evidence to support any decisions, however, did not end there. Instead, it continued well into the seventeenth century.

Chapter 2: Literature Review 31

Nonetheless, it was not until 1996 that Sackett et al. (1996) formally introduced the approach as Evidence-based practice (EBP). In it, Sackett et al. (1996, p. 71) defined Evidence-based practice as

“the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients. The practice of evidence-based medicine means integrating individual clinical expertise with best available external clinical evidence from systematic research”.

Sackett et al. (1996) explained that with increased clinical expertise, it would have reflected a more effective and efficient diagnosis. Not only that, but it would have also meant that clinicians had thoughtfully included their patients’ dilemmas, rights and choices as part of the clinical decision-making process. The availability of best external clinical evidence was referred to as acquiring evidence from clinically relevant research, especially those that were patient-centered. Sackett et al. (1996) added that clinicians would need both their clinical expertise and the help from the best clinical evidence available to make decisions, as, without evidence, the practices risked being out-dated and causing harm to patients instead. On the other hand, without clinical expertise, even the best evidence available might not be useful and applicable to the patients. Thus, the ‘components’ to Evidence-based practice are best illustrated by Hoffmann et al. (2009) in Appendix C.

Over the years, the definition of Evidence-based practice may have broadened and been further refined, but the key to the paradigm of EBP still remains, and that is about making the right decisions in the process of delivering care to every patient. Thus, the most frequently used and widely known definition of EBP is aptly described by Hoffmann et al. (2009, p. 3), which characterises EBP as the “integration of the best research evidence with clinical expertise and client’s values and circumstances” together with the “characteristics of the practice context where the health professionals work”. Therefore, the underlying philosophy behind the paradigm of Evidence-based practice is to assist clinicians in making effective decisions (Friesen-Storms et al., 2015; Hoffmann et al., 2009; Hoyt & Hersh, 2014), and to do it based on available clinical evidence, the knowledge and experience of attending doctor and the consideration of the needs of patient’s current medical and non-medical conditions.

32 Chapter 2: Literature Review

2.4.2 Benefits of Evidence-based Practice With the increasing prevalence of patients with chronic diseases and those with comorbidities, compelling healthcare issues such as rising healthcare costs, patient safety concerns and quality of care have never been more critical.

With the introduction of Evidence-based practice, the nature of clinical practice has evolved into one that is “more scientific and empirically grounded” (Greenhalgh et al., 2014, p. 1). This has resulted in patient care that is safer, consistent and cost- effective. EBP warrants that, besides clinical expertise and experience, clinical evidence be considered when decisions are being made, thus ensuring that the reason behind any clinical decisions made, becomes clear and transparent (Hoffmann et al., 2009). Clinical evidence is therefore a key attribute of EBP. According to Bower and Gilbody (2010), “evidence” is considered as a basis of belief, where the belief is that a particular treatment will work for a patient. Doctors often use such beliefs to guide them when attending to patients. While it can be argued that beliefs may be perceived as something intangible, in EBP, clinical evidence is derived from systematic reviews, meta-analytic reviews, observational studies and randomised controlled trials (Evans et al., 2003; Greenhalgh et al., 2014). These studies thus offer evidence that is highly empirical and credible. One of the most preferred tools in generating evidence for EBP is randomised controlled trials. It is considered as the gold standard design for establishing causality on scientific research and testing therapeutic interventions (Horn & Gassaway, 2007; Hoyt & Hersh, 2014). The evidence pyramid diagram in Appendix D best illustrates the relative ranking, in terms of effectiveness, of the different types of studies conducted to elicit evidence. Although systematic reviews and meta-analysis are placed at the top of the pyramid, Hoyt and Hersh (2014) argue that such studies are impractical, as they are expensive and labour intensive. Case reports and case series, on the other hand, provide less scientific significance, as they are merely collections of treatment reports of individual patients without control groups. The introduction of EBP in healthcare has also benefitted in many ways, for instance, in the development of clinical guidelines. The practice of EBP has propelled towards the outcome of well- developed evidence-based guidelines over consensus-based guidelines. An excellent clinical guideline thus offers concise instructions on many aspects of healthcare including screening, diagnosis, management and prevention (Bower & Gilbody, 2010). According to Grimshaw et al. (2006) as cited in Bower and Gilbody (2010),

Chapter 2: Literature Review 33

EBP enables the ability to evaluate the effectiveness of such guidelines, thus influencing clinical practice and improving patient outcomes.

The reality is, clinical decision making is becoming a more complex and complicated process. It needs the collaboration of multiple healthcare professionals and specialists to make it far more effective (Hunink et al., 2014; Smith et al., 2008). With clinical decisions also under constant scrutiny from fellow professionals and the medical audit board, EBP facilitates professional accountability. Furthermore, the clinical decisions made in EBP are based on evidence and adherence to well-designed clinical guidelines, making accountability transparent. It also has the potential to help reduce healthcare costs and expenditure, as healthcare resources are being used wisely when the decisions made consider relevant evidence.

In 1998, McKibbon (1998) recognised that healthcare needed to be personalised and was continually evolving due to many uncertainties and probabilities. McKibbon also realised that individualised care was where care should be heading towards. Thus, McKibbon’s definition of Evidence-based practice was an “approach to healthcare where healthcare professionals use the best evidence possible, that is, the most appropriate information available to make clinical decisions for individual patients”. McKibbon also noted that clinical expertise played an important role, as decision making was getting complex and needed to be conscientious. These decisions needed to take into consideration the patient’s characteristics, situations and preferences as well as the appropriate application of evidence, to be effective as care solutions.

In short, Evidence-based practice has strategised the approach to direct clinical practice using evidence to support decisions, providing to both the caregivers and patients, valuable information to make well-informed decisions both from and for the overall benefit of giving and receiving quality care.

2.4.3 Effectiveness of Evidence-based Practice Limited by Evidence The evolution of Evidence-based practice as a paradigm to direct clinical medicine highlights the importance of evidence for decision making. EBP has undoubtedly improved the strategy towards providing effective clinical care. However, it is without its drawbacks and limitations. While EBP is still committed to making clinical care systematic and empirically grounded, those who may have supported EBP now argue that the approach is in fact, facing a serious crisis (Greenhalgh et al., 2014).

34 Chapter 2: Literature Review

Many academics and researchers alike have debated the effectiveness of adopting EBP, while others questioned the validity of the evidence in use. Many critics have also doubted the applicability of EBP in current clinical practice settings. With evidence from studies said to bear little or no relevance to an actual real-world backdrop, how effective and appropriate has the care to patients been? Therefore, when posed with such a question, the most repeated reason provided points towards the considerable limitations of the evidence in use, primarily, outcomes from RCT studies conducted (Fava et al., 2015; Horn & Gassaway, 2010).

Randomised controlled trial is one of the most common research methods used to elicit evidence for EBP. It is considered the “gold standard” for generating evidence and is excellent at providing guidance towards effective clinical practice (Barkham et al., 2010; Houser & Oman, 2010). Only systematic reviews and meta-analysis are considered to be better than RCT, as illustrated in Appendix D. However, systematic reviews and meta-analysis are impractical as it is both expensive to conduct and labour-intensive, thus the preference for RCT (Hoyt & Hersh, 2014). Randomised controlled trials, as defined in summary by Hoyt and Hersh (2014, p. 327), is a study where

“subjects are randomly assigned to a treatment or a control group that received placebo or no treatment. The randomisation assures to a great extent that patients in the two groups are balanced in both known and unknown prognostic factors, and that the only difference between the two groups is the intervention being studied. RCTs are often ‘double blinded’ meaning that both the investigator and the subjects do not know whether they received an active medication or a placebo. This assures that patients and clinicians are less likely to become biased during the conduct of the trial, and the randomisation effect remains protected throughout the trial.”

To understand how evidence is elicited from RCT studies for use in EBP, it is essential to understand the mechanism of conducting such studies. The mechanisms towards conducting RCT are based on three important aspects of control, comparison and randomisation. These aspects of RCT ensure the protection of threats against internal validity or threats towards researchers’ confidence in concluding that an experiment conducted has demonstrated a causal relationship (Bower & Gilbody, 2010).

Chapter 2: Literature Review 35

Control. In control, variables are kept constant or varied systematically to investigate their effect. For example, patients with comorbidities are usually excluded from RCT studies. This is to ensure that any differences in outcomes observed are not due to their existing medical conditions.

Comparison. In comparison, patients are given different degrees of treatment, and the differences in the outcome of the results are examined. For example, two groups of patients will be used for a study. The ‘control group’ represent a group of patients who will not be subjected to any treatment, while the ‘comparison group’ are those being offered alternative treatments. ‘Control’ and ‘comparison’ groups are used to protect against threats such as time and elements outside of the research context, like being attended to by a professional. Such threats may potentially alter the outcome of the results. Therefore, the use of ‘control’ and ‘comparison’ groups ensures that such threats affect both groups and cancel out any benefits or disadvantages encountered. The differences in outcome are down to the difference in the level of treatments provided during the study.

Randomisation. Randomisation is a process that ensures and assigns both ‘control’ and ‘comparison’ groups equal levels of each type of patient variable. For example, when a particular group only consists of patients with vast differences in characteristics, such as the severity of their disorders, then this may affect the result of the outcome rather than the treatments provided. Therefore, when assigning groups, any patient types or any distinctive value assigned to a group, the same should be assigned to other groups, to balance them out, ensuring that the differences in outcome are not attributed to any existing differences between groups.

The three important aspects of RCT ensure that it continues to provide the evidence needed for the paradigm of Evidence-based practice. Through RCT, clinicians are able to determine when a particular treatment is harmful rather than good (Bower & Gilbody, 2010). However, Horn and Gassaway (2007) reveal that RCT may not actually be suitable for everyday patient care. While it is agreed that RCT is important in determining if a new treatment does cause an effect, RCT is unable to reveal combinations of interventions or practices that can be useful during everyday patient care. Plus, the very nature of RCT, which makes it excellent at finding causal evidence for treatments, is also its drawback. Given the nature of how RCT studies are

36 Chapter 2: Literature Review

being conducted, it has resulted in its limitation as a source of evidence, useful for use in the support for the clinical practice of medicine.

Randomised controlled trial is an expensive study to conduct. The elaborate process in screening the right patients for a study, the effort in coordinating and monitoring care to the study groups as well as the collection of data makes RCT an expensive investment (Horn & Gassaway, 2007). As a result, not all treatment studies can be conducted. As a result, there may be many more treatments that have yet to be studied and evidence to be found. This may require other studies to be conducted to complement where outcomes from RCT studies are lacking.

RCT produces small data sets that are not useful. The strict and restrictive selection criterion of patients for the RCT study has adequately enabled the reduction in patient variation. This has unfortunately also restricted the amount of data available for analysis. Therefore, when RCT studies do take off, they produce such small data sets that the data becomes worthless. As Evans et al. (2003) aptly pointed out, a small data set is inadequate in making precise evaluations of the effects of the treatments. A small data set also limit the conclusions that can be obtained from it, thus inhibiting the general use of such findings to be applied in patient care (Evans et al., 2003; Horn & Gassaway, 2007).

Selection criteria hamper useful findings. RCTs’ lack of applicability and relevance to real-world clinical settings is another of its limitations. In real-world settings, patients vary in conditions. The selection criteria tend to exclude certain groups of patients from randomised controlled trials, such as those with severe disorders or comorbidities. Patients with comorbidities are, unfortunately, becoming more and more common (Salive, 2013). Horn and Gassaway (2007) thus, believe that, if clinical trials included patients of with severe disorders or comorbidities, it may have actually been more useful to analyse the outcome of these results. This can provide further analysis on the outcome of the results and support healthcare decisions specifically pertaining to patients with such conditions. RCT’s strict selection criteria inhibit the acceptance of such patients into the study, inevitably limiting the evidence to support them. This has placed some doubts in the minds of clinicians about the findings, while some have even dismissed using such RCT findings totally.

The above highlights the limitations of the outcomes from RCT studies, which in turn, impede the effectiveness or availability of evidence that can be utilised. This,

Chapter 2: Literature Review 37

in a way, contributes inevitably to the limitations of EBP due to its frequent use of evidence generated from RCT studies. As a consequence of the limitations of the evidence or outcomes from systematic studies, some researchers and academics have also pointed out a critical limitation in the applicability of EBP to effectively guide clinical practice. Bakken (2001) highlights the difficulty faced by clinicians in interpreting the evidence available to effectively apply it to certain patients. Bennett et al. (2012) surface doubts by clinicians on whether treatments are easily adaptable to actual clinical practice settings, even if such treatments have been proven effective and are supported by systematic evidence. Greenhalgh et al. (2014) question if EBP guidelines are indeed relevant to real-world patients, as the guidelines developed are often inadequately applicable to patients with comorbidities. All three authors raised issues regarding how evidence can be effectively applied to direct clinical medicine to patients due to the disparity between patient types tested during studies, as compared to the varied kinds of patients that are present in actual clinical settings.

Besides eliciting evidence from RCT studies, the rapid growth in medical knowledge development, such as the Cochrane reviews, has added to the growth of medical literature databases where clinicians are searching to elicit evidence. However, the rate of growth of these medical literature databases is way higher than the frequency where the reviews are being summarised and evidence extracted to make it easier for healthcare professionals to search for and apply them to patients. It has been predicted that the process will take as long as 10 years before all the reviews can be summarised and be applicable (Guyatt et al., 2004). Hence, clinicians still face the issue of not having sufficient access to important and relevant clinical information when needed (Bakken, 2001), which appears to be a recurring hindrance contributing to the limitations faced by Evidence-based practice. This has further added to the limitations of the effectiveness of the EBP approach to direct decision making.

2.5 PRACTICE-BASED EVIDENCE AS A COMPLEMENTARY PARADIGM TO EVIDENCE-BASED PRACTICE

As highlighted, the care for patients has currently become more complex and complicated, especially with the increasing prevalence of patients with comorbidities. It is therefore vital that healthcare professionals are able to make clinical decisions based on the unique medical conditions faced by individual patients so that treatments provided, and interventions made are appropriate and necessary. However, limited by

38 Chapter 2: Literature Review

the quality of evidence from the outcomes of systematic studies in particular RCT, this has resulted in the questionable applicability of EBP for effective decision making, as pointed out by Straus and McAlister (2000), Bower and Gilbody (2010) and Greenhalgh et al. (2014). For this reason, a Practice-based evidence (PBE) approach has been proposed by the likes of Barkham and Mellor‐Clark (2000), and Horn and Gassaway (2007). Together with what is possibly lacking from EBP, this PBE approach aims to improve and enhance decision-making capabilities of healthcare professionals by using evidence that is relevant and relatable to what really happens in an actual clinical practice setting. PBE therefore represents a complementary approach to EBP where evidence used to assist in clinical decision making is practical.

Evans et al. (2003) propose that Practice-based evidence uses evidence derived from routine settings rather than from efficacy studies, such as in RCT. It therefore engages practitioners in the collection and the ownership of data as well as in analyses of the data that will shape their practice. In coming up with an accurate definition for PBE, Barkham and Margison (2007, p. 446) amended the definition of Evidence-based practice by Sackett et al. (1996) to the definition of Practice-based evidence as

“the conscientious, explicit and judicious use of current evidence drawn from practice-settings in making decision about the care of individual patients. Practice-based evidence means integrating both individual clinical expertise and service level parameters with the best available evidence drawn from rigorous research activity carried out in routine clinical settings.”

In 2012, Green and Latchford (2012, p. 88) described Practice-based evidence as to “encompass any activity in which clinicians gather scientific evidence themselves as part of their routine practice”. Such activities included exploring the changes in a single case using qualitative or quantitative data, to contributing to a large database recording.

Based on the definition and description provided by the authors above of what Practice-based evidence should entail, the difference between Evidence-based practice and Practice-based evidence seems to point to the use of clinical evidence from routine clinical settings. Therefore, the context of Practice-based evidence is based on two critical aspects of (1) clinical expertise and (2) best available evidence drawn from routine clinical settings.

Chapter 2: Literature Review 39

The clinical expertise in PBE is still being referred to as clinicians’ wisdom, experience, necessary skills, knowledge and values when called upon to make decisions. It is still vital in the decision-making process and is developed through clinical practices. Having underlined the concerns regarding the limitations of outcomes from RCT studies as evidence used for EBP, continuing its use to generate evidence just makes no decision-making sense. Additional or alternative sources of evidence must instead be elicited, that is capable of improving the quality of evidence gained so that this provides healthcare professionals with the ability to make effective decisions. Therefore, as proposed by PBE approach to decision making, a new source of evidence that reflects “evidence drawn from practice-settings” or “activity carried out in routine clinical settings” has to be identified. With ICT adoption widespread across healthcare organisations, naturally, the source of evidence to look at should be there. Coincidentally, the adoption of ICT has resulted in the generation of extensive collections of digital health data such as electronic health records (EHR) (Osop & Sahama, 2016b). Such data generated is the result of information regarding patients being captured throughout their patient journey. EHRs represent a repository of patient data, which contain temporal records of a patient’s health and healthcare information. They contain past medical history, life style, physical examination notes, diagnoses, laboratory and radiology tests results, procedures information, treatments, medication and discharge summaries (Häyrinen et al., 2008).

EHRs fit well into the model of “evidence drawn from practice-settings” and “activity carried out in routine clinical settings”. Information systems in hospitals are in place to assist clinicians with documentation, process workflow and decision making. They are being used to record patient’s health conditions during visits to the healthcare organisations (i.e. patient journey). Records ranging from medications to treatments and laboratory tests are captured and stored in large databases. Such records are examples of EHRs. EHRs thus reflect the needs of PBE, to draw and use evidence captured from routine clinical practice settings. Furthermore, EHRs are inexpensive as data is readily available in hospital from the respective clinical information systems. EHRs also represent huge data sets that span an extended period of time, effectively containing meaningful information that can be analysed and used to assist decision making. Additionally, records in EHRs include those from a myriad source of patient types, making the use of EHRs as evidence, applicable to individual patients. EHRs

40 Chapter 2: Literature Review

have the potential to overcome the limitations that exist within RCTs, which have made EBP unpractical and inapplicable. Moreover, PBE can be current, real-time and relevant, as EHRs are stored within healthcare organisations, regularly updated and are accessible to clinicians for decision making at any time.

The two leading proponents of Practice-based evidence in the past two decades have been Michael Barkham and John Mellor-Clark, and Susan Horn and Julie Gassaway. Both groups of authors highlighted that the motivation in adopting the Practice-based evidence approach started from the need to improve the effectiveness of Evidence-based practice and to close the widening gap between what is available from research and what has been and can be adopted in practice.

2.5.1 Practice-based Evidence in the eyes of Barkham and Mellor-Clark Barkham and Mellor-Clark’s motivation to introduce a Practice-based evidence approach was based on the need to close the gap between research and practice. According to Barkham and Mellor‐Clark (2000), the direction of care adopted through Evidence-based practice has gone through several improvements, from efficacy, through quality improvement and towards increased effectiveness. The authors have suggested that collaboration between researchers and clinicians can realise the Evidence-based practice limitation through a complementary paradigm of Practice- based evidence. The authors view the concept of evidence as a continuum, emphasising the role of Practice-based evidence as the natural complement to Evidence-based practice. Barkham and Margison (2007) defined Practice-based evidence as the process of integrating clinical expertise and best evidence drawn from research activities that were carried out in actual clinical practice settings to decide on the care of individual patients.

The key towards effective care, underlined by the paradigm of Evidence-based practice, has been the use of outcomes from randomised controlled trials or efficacy studies. However, as discussed, the limitations of outcomes from these studies have undoubtedly had a limiting effect on the effectiveness of Evidence-based practice. Therefore, in the suggested approach of Practice-based evidence, Barkham and Mellor‐Clark (2000) have identified the use of practice research networks (PRN) as structures required in the delivery of Practice-based evidence. A PRN typically consists of a large group of clinicians that work together to accumulate, store and document clinical data. A PRN entails collaborating clinicians to collect, share and use

Chapter 2: Literature Review 41

data gathered during their routine clinical practice as standards and benchmarks to improve the delivery and development of care services (Barkham & Mellor‐Clark, 2000).

2.5.2 Practice-based Evidence in the eyes of Horn & Gassaway Horn & Gassaway’s motivation to introduce the approach of Practice-based evidence stems from the problems of evidence from Evidence-based practice. The main tools for generating evidence in EBP are randomised controlled trials and meta- analysis. The evidence generated in randomised controlled trials, while it is able to show its effectiveness, may turn out to be actually ineffective in the real world. A Practice-based evidence approach therefore aims at bridging the gap between what recommended care is and what patients receive as improved care, by generating new knowledge.

Because of the knowledge factor, the introduction of a Practice-based evidence for clinical practice improvement (PBE-CPI) is advocated (Horn et al., 2010). PBE- CPI is a clinical research methodology similar to how randomised controlled trial studies are being conducted. However, PBE-CPI is a methodology where the conduct of practical clinical trials can be executed effectively.

When compared to the conduct of RCT, PBE-CPI takes into account patient complexity and treatment variations in an actual clinical care practice. This means that PBE-CPI does not have to change treatments in order to evaluate if a particular intervention is effective, unlike in RCT. Therefore a PBE-CPI study is able to capture an extensive range of information such as “patient heterogeneity, treatment heterogeneity and outcome heterogeneity” (Horn & Gassaway, 2010). PBE-CPI is also able to verify the kind of care process that is responsible for the patient outcomes of different patient types (Horn & Gassaway, 2007).

This makes the outcomes from PBE-CPI research applicable to individual patients, unlike RCT, which has difficulties in applying evidence to certain patient types.

2.6 ELECTRONIC HEALTH RECORDS

The widespread use of ICT, as highlighted previously, has seen the increase in the generation of large volumes of patient data, which can be categorically referred to

42 Chapter 2: Literature Review

as Electronic Health Records (EHR). It is used to support, maintain and improve healthcare quality. Patient health records can be easily shared among participating clinicians, allowing for better understanding of a patient’s situation, making well- informed decisions and ultimately improving care.

EHRs contain patients’ health-related information that has been captured over a protracted period of time, possibly from multiple healthcare institutions, and are thus very extensive. Examples of such data include past medical history, treatment procedures and medication. By itself, an individual patient record can only do so much to inform a clinician about their health condition. However, collectively, EHRs contain valuable information which can be analysed, studied and applied to improve healthcare delivery.

With Practice-based evidence advocating the use of evidence from actual clinical practice settings, EHRs seem to be the right fit and therefore should be considered as a source of such evidence.

EHRs contain clinical, social as well as administrative data that illustrates the patient’s journey throughout their medical care. Records such as past medical history, treatment procedures, medication and other medically related data provide information that can effectively be used as reference and evidence for decision making. At the same time, EHRs record information routinely faced by healthcare professionals in a real clinical setting. Thus, using EHRs as an alternative source of evidence for Practice- based evidence approach to decision making is possible.

2.6.1 Understanding Electronic Health Records The Electronic Health Record is defined as a repository of patient data in digital form, stored and exchanged securely, and accessible by multiple users (Häyrinen et al., 2008). It contains comprehensive, cross-institutional and longitudinal collections of patient’s health and healthcare data (Hoerbst & Ammenwerth, 2010). EHRs contain information that can be categorised into referral, present complaint, past medical history, life style, physical examination, diagnoses, laboratory and radiology tests, procedures, treatment, medication and discharge summaries (Häyrinen et al., 2008). They also hold valuable patient information such as patient needs during episodes of care provided by different healthcare professionals. They record detailed information about outcomes of treatment, medication and tests conducted. EHRs are considered as

Chapter 2: Literature Review 43

the building blocks of an eHealth system (Sahama et al., 2013), where information and communication technologies are applied in the delivery of different types of healthcare services.

What is an EHR? According to Sahama et al. (2013), an EHR is a combination of Personal Health Records (PHR) and Electronic Medical Records (EMR). PHRs consist of information about individuals that is stored, collected, shared and controlled by the individuals themselves, while EMRs are information that authorised clinicians and healthcare organisations manage, amend and update. The relationship between PHRs, EMRs and EHRs is a complex data structure, as shown in Figure 5. EHRs contain both structured and unstructured data as well as narrative text. Structured data is those that are standardised and pre-defined, in a format that can be easily read and accessed by computers. Examples would be patient names, age and diagnosis codes (Datamark Incorporated, 2013). Unstructured data is data comprising of those in the form of diagnosis images and scanned medical records, among many others (Datamark Incorporated, 2013).

Goal of EHRs As mentioned, the main purpose of EHRs is to support, maintain and improve health care quality (Hoerbst & Ammenwerth, 2010). Hersh et al. (2013) believe that the usage of EHRs is able to offer great promise when it comes to improving the quality, safety and cost of healthcare. This is possible since in an EHR, care delivery can be easily recorded and documented, allowing for the outcome of care to be assessed (Häyrinen et al., 2008) and evaluated for future review. Through the use of EHRs, patient care can now be planned for, with each and every patient’s specific and unique objectives being set (Häyrinen et al., 2008).

EHRs enable information sharing For patient care to be successful, sharing and proper use of information is an important aspect. More often than not, patients are under the care of more than one healthcare professional and specialist, pushing for the growing need for information to be shareable. Healthcare professionals thus require tools that enable them to connect and share information with other specialists to help make better decisions towards the improvement of the quality of care. Patients, on the other hand, want to be more involved in their own healthcare processes and want control over their health records. Information sharing becomes a part of healthcare, which can alter the way care is being

44 Chapter 2: Literature Review

delivered. EHRs offer high accessibility, allowing health records to be viewed at multiple locations at any time. By sharing the records, co-treating clinicians can view and refer to each other’s patient records to have a better understanding of patient’s conditions (van Ginneken, 2002). For example, the development of a National Electronic Health Record (NEHR) in Singapore, aims at allowing the sharing of medical information by all public healthcare institutions and a selected few private clinics (Sinha et al., 2012; Woon, 2014). This allows attending doctors to gain access to records and information inputted by other caregivers from different healthcare institutions.

Figure 5. Graphical view of Electronic Health Record adapted from Sahama et al. (2013)

EHRs improve information quality A study evaluating the impact of the use of EHRs quoted that information quality is highlighted as an important factor. The improved information quality is perceived as a result of EHR implementation, according to doctors and nurses (Nguyen et al., 2014). The information quality or data quality is concerned with the correctness, timeliness, accuracy and completeness that make data appropriate for use (Orfanidis et al., 2004). Completeness can be referred to as the measure of the prevalence of missing information. According to Häyrinen et al. (2008), the use of information systems is conducive to creating complete documentation among healthcare professionals. This, as explained by van Ginneken (2002), is because information systems actively prompt for data to be entered. As a result, important data and

Chapter 2: Literature Review 45

information that is required are not missed, thus EHRs facilitate the collection of detailed data (Barkham & Mellor‐Clark, 2003).

EHRs have the potential to offer further benefits. The application of data mining of context-sensitive information and retrieval techniques on EHR systems may offer new development of care delivery models, performance reporting and public health surveillance. With the use of EHRs not confined to just tertiary care, their widespread use in both primary and secondary care as well, demonstrates the importance and dependence on EHRs for improvements to healthcare management.

Importance of integrating EHRs to gain comprehensive data While EHRs have been defined to be a repository of digital information regarding patients’ healthcare information, many of these digital data tend to be stored in proprietary databases (Wisniewski et al., 2003). In most cases, these collections of EHRs are not integrated, resulting in situations where data accessible by healthcare professionals are not comprehensive enough. Even when organisations have huge healthcare databases, the information available may not be big enough to provide solutions needed by healthcare professionals. There is a need to integrate with other internal data sources such as claims or administrative data and even to external data sources such as the like of Singapore’s NEHR (Brown et al., 2010). By integrating valuable data sources and increasing integration capabilities, there is potential for the setting off of new findings and probabilities (Henriques et al., 2013). Through this findings and probabilities, it is possible that healthcare professionals can be supported with effective decision making.

Secondary uses of EHRs The fact that EHRs contain patients’ healthcare-related information, makes them a valuable source of information that many researchers are currently capitalising for clinical research. In fact, secondary uses of EHR data is said to have the potential of increasing the conductivity of patient-centered research and the rate of new medical discoveries (Weiskopf & Weng, 2013). Together with the added pervasive use of ICT, more digital health data is expected to be generated, allowing healthcare organisations to gain access to this rich clinical data which has great promise for secondary uses. Besides clinical research, the practice-based data of EHRs has also been identified as being suitable for use in patient care purposes, for example, in identifying, undiagnosed post-stroke spasticity patients (Cox et al., 2016), drug interactions and

46 Chapter 2: Literature Review

effects (Henriksson et al., 2015; Ibrahim et al., 2016; Jang et al., 2016; White et al., 2016) and diseases risk factors among patients (Jonnagaddala et al., 2015; Karystianis et al., 2015). As a matter of fact, Martin-Sanchez et al. (2017) have identified that the use of EHR data is one of the “pillars that is enabling the practice of precision medicine”. This allows healthcare professionals to provide treatments and preventive methods based on the individual patients’ genes, environment and lifestyle.

Due to the nature of EHR data that is collected from diverse patient types, information captured regarding their medical statuses, care and outcomes is a close representation of the actual patient population. This data can range from patients with a simple or minor medical condition to those with multiple chronic conditions altogether. As such, secondary uses of EHR data can also be helpful in informing decision making that is patient centered. Information regarding similarly typed patients can be queried from these massive collections of EHR data which can be reused to direct decision making regarding other similarly typed patients at the point of care. In fact, studies by Gallego et al. (2015) on “personalised cohort” and “green button” by Longhurst et al. (2014) suggest the possibility of capitalising on EHR data as an alternative source of evidence for such purposes. Likewise, the use of predictive analysis, machine learning and data mining of EHR data, has the potential to further improve the management and care of varied patients (Chen et al., 2017).

EHRs not only function as collections of patient-specific clinical and medical information but information that captures individual patient’s treatments, interventions, medications and so forth, documented by the attending healthcare professionals. In essence, this resembles evidence of actual clinical practices performed by the healthcare professionals that record efficient treatments or effective medications which have resulted in better care outcomes for their patients. While such guarantees, regarding outcomes, are not explicitly captured in EHRs, what can be assured is that evidence of interventions can be reference upon and be used to help in making well-informed decisions.

EHR data quality The secondary uses of EHRs have shown benefits in patient-centered research like those mentioned above. However, these benefits rely on EHR data to be of quality (Weiskopf & Weng, 2013). One of the concerns with data quality is the fact that EHR systems which record it, have not been developed to ensure that data recorded can be

Chapter 2: Literature Review 47

directly reused for research. The use of EHR systems, according to Weiskopf & Weng (2013), has also led to the “recording of a greater quantity of bad data”. As highlighted by Burnum (1989), bad data arises from recording errors caused by doctors, patients and medical equipments in use as well as the recording of misinformation provided by patients. For this reasons, concerns with data quality suggests that EHRs may not be as useful as we may have initially thought.

Dimensions of Data Quality Author Dimension Synonyms/Definitions (Weiskopf & Completeness Accessibility, Accuracy, Availability, Missingness, Omission, Weng, 2013) Presence, Quality, Rate of recording, Sensitivity, Validity Correctness Accuracy, Corrections made, Errors, Misleading, Positive predictive value, Quality, Validity Concordance Agreement, Consistency, Reliability, Variation Plausibility Accuracy, Believability, Trustworthiness, Validity Currency Recency, Timeliness (Almutiry et Accuracy The extent to which registered data conforms to its actual al., 2013) value Completeness The state in which information is not missing and is sufficient for the task. Consistency Representation of data values remains the same in multiple data items in multiple locations. Relevance The extent to which information is appropriate and useful for the intended task Timeliness The state in which data is up to date and its availability is on time Usability The ease with which data can be accessed, used, updated, understood, maintained and managed Provenance The source of data, shown and linked to metadata about data Interpretability The degree to which data can be understood Security Prevents personal data from being corrupted and controls access to ensure privacy and confidentiality (Kahn et al., Conformance Compliance of data against internal or external formatting, 2016) relational or computational definitions Completeness Features that describe the data attributes in a data set, e.g. absence of data Plausibility Features that describe the believability or truthfulness of data values

Table 1. Dimensions of Data Quality by different authors

The study of data quality has begun ever since the beginning of EHR systems use. There have been many definitions of data quality and the best and most cited is

48 Chapter 2: Literature Review

“fitness for use”. Weiskopf and Weng (2013) describe “fitness for use” as when there is “sufficient quality in data that it serves the needs of a given user pursuing specific goals”. Leitheiser (2001) and Johnson et al. (2015) on the other hand describe data quality as data that is “fit for use”. “Fit for use” for data consumers and for purposes such as decision-making or planning. However, no matter which definition is adopted, data quality is frequently evaluated based on its quality dimensions (Almutiry et al., 2013; Batini et al., 2009; Weiskopf & Weng, 2013). Therefore, when a data quality problem occurs, it usually happens when there are issues with one or more of these dimensions that influences its fitness for use (Leitheiser, 2001). In fact, some authors have also used these quality dimensions to define the meaning of data quality. For example, Rousidis et al. (2014) define data quality as the “state of completeness, validity, consistency, timeliness and accuracy that makes data suitable for specific use”. Hence where there is potential in the reuse of data and with the need to integrate multiple sources of data to achieve comprehensiveness, data quality concerns have never been more important.

There have been many studies that investigate these different dimensions of data quality. However, there has been no generally accepted quantitative measure of data quality according to Johnson et al. (2015). Weiskopf and Weng (2013) describe five common dimensions of quality as Completeness, Correctness, Concordance, Plausibility and Currency. Almutiry et al. (2013), on the other hand address information-related dimensions in data quality as Accuracy, Completeness, Consistency, Relevance, Timeliness, Usability and the communication-related dimensions in data quality as Provenance, Interpretability and Security. Kahn et al. (2016), in assessing data quality for secondary uses of EHRs, identify the dimensions of data quality as Conformance, Completeness and Plausibility. These data quality dimensions thus help to evaluate the effectiveness of EHRs to be used as evidence in decision making.

2.7 CLINICAL DATA WAREHOUSE

The benefits of using EHR systems provide healthcare organisations with the ability to improve the quality and safety of patient care, and at the same time reduce costs (Hersh et al., 2013). Despite the apparent benefits mentioned, the nature of information systems being scattered within healthcare organisations raises an unwanted issue where many of these stand-alone systems lack communication

Chapter 2: Literature Review 49

capabilities with each other. Therefore, a data integration solution is required where data from multiple sources can be aggregated and used to assist in decision making.

A data warehouse is a subject oriented, integrated, time-variant and non-volatile collection of data that supports managerial decision making (Sen & Sinha, 2005). A data warehouse is a central repository where massive amounts of data from multiple sources are brought together, integrated, managed and made accessible to various users. It does so by extracting, cleaning, conforming and delivering the source data into this central information repository. It makes many disparate systems that hold unintegrated data becomes integrated (Ado et al., 2014). A data warehouse saves time for end users as it increases their ability to access and analyse the data (Scheese, 1998). It improves the quantity and quality of information made available through efficient storage (Ado et al., 2014). A data warehouse supports querying and implements analysis, making it suitable for decision making, research and management.

Data warehouses have been extensively deployed in banking and finance, consumer goods and demand-based manufacturing (Krishnan, 2013). They are also gaining popularity in use for non-commercial sectors such as medical fields, government, military services, education and research communities (Ado et al., 2014).

2.7.1 Data Warehousing for Healthcare It is inevitable that data warehousing methodologies will find their use in the healthcare and medical industry. Medical data is scattered and unstructured throughout healthcare organisations (Roelofs et al., 2013) with many stand-alone information systems that do not communicate with each other (De Mul et al., 2012) and often, the integration between these systems is poor (Pedersen & Jensen, 1998). For example, clinical departments like pharmacy, laboratory and medical records, each store data in a separate database server with each server interfaced to a single proprietary database (Wisniewski et al., 2003). The result is disparate systems with “islands” of information (Sahama & Croll, 2007), which in itself is extensive (Ado et al., 2014). This untapped clinical data continues to increase as healthcare organisations adopt electronic health records, which are a rich source of data for analysis. However, access to this extensive and scattered data is a challenge for end users (Sahama & Croll, 2007; Scheese, 1998) and requires the data to be integrated for effective use.

50 Chapter 2: Literature Review

Data warehouse assists decision making. There are several examples of the use of data warehousing in healthcare settings. Roelofs et al. (2013) describe the benefits of implementing data warehouse architecture together with data mining tools to assist in collecting data for radiotherapy trials. Ado et al. (2014), on the other hand, share how a diabetes data warehouse can be used to support decision makings, such as recommending food consumption and physical activity. In the two research papers mentioned above, Roelofs et al. (2013) and Ado et al. (2014) noted that the introduction of clinical data warehouse architecture presents healthcare professionals with tools for decision making. Ado et al. (2014) go further to add that it does not only provide but improves the quality of real-time decision making. For healthcare professionals, this is crucial, as decision making is an integral part of their profession. They are regularly faced with situations where the right decisions must be made.

Data warehouse promotes data integration and sharing. A clinical data warehouse primarily seeks to integrate data from multiple clinical systems into one. The integration of data allows a data warehouse to become “a place where healthcare providers gain access to all clinical data gathered in the patient care process” (Ado et al., 2014). While this is a key aspect of data warehouse architecture, De Mul et al. (2012) believe that it is a crucial feature, especially for the unique requirement of providing care in intensive care units (ICU). Treatment in ICUs requires constant monitoring from various high dependent devices to provide an up- to-date status of patients. However, it may be too demanding for clinical information systems to provide this decisive, consolidated information regularly.

Data warehouse facilitates research. The implementation of data warehouse architecture allows clinicians to capitalise on the benefits that integrated data provides. Besides improving the quantity and quality of information available for clinicians to refer to (Ado et al., 2014), it also aids in research. New treatment programs can be unfolded or discovered from the huge volume of clinical information, so too, the design of disease management plans for individual patients (Roelofs et al., 2013). According to Lyman et al. (2008), researchers have also made use of clinical data warehouses beyond just providing treatments. They have been utilised to find relationships between diseases.

Chapter 2: Literature Review 51

The main idea for the use of clinical information systems is to support, maintain and improve healthcare delivery. While it has been excellent in its use to support and maintain the healthcare of patients, the lack of communication and sharing between the systems has revealed a substantial limitation. Data warehouse architecture has therefore presented itself as a solution that not only solves this problem but opens up other avenues where data can be better utilised.

2.8 CLINICAL DECISION SUPPORT SYSTEM IN HEALTHCARE

Evaluating clinical data can lead to the discovery of trends and patterns that can enhance the understanding of disease progression and management (Prather et al., 1997). Such trends and patterns can be the detection of diseases, screening of patients for diagnosis and prediction of medical complications. Musen et al. (2014) define a clinical decision support system (CDSS) as any computer program designed to help healthcare professionals make clinical decisions. The use of CDSSs is widespread, from maintaining medication to improving disease management.

The use of CDSSs is very much varied. With the availability of digital health data such as electronic health records, a CDSS becomes a decision makers’ tool that further improves their patient care capabilities by bridging the gap between what they know and what can be adopted in practice (Bates et al., 2003). CDSSs have been used to support the safe practice of patient care. A CDSS that is embedded within patient care workflow supports decision making throughout the patient journey and helps to reduce errors. CDSSs can lessen the probability of healthcare professionals making hasty decisions, which may lead to inaccurate or misdiagnosis (Castaneda et al., 2015). The application of CDSSs in medication prescription, on the other hand, has allowed doctors prevent errors in prescribing medication (Jao et al., 2008). Another example of the use of a clinical decision support system is providing additional expert information to support accurate diagnosis (Parshutin & Kirshners, 2013).

The use of CDSSs has shown to be effective in helping healthcare professionals with assistance in making well-informed decisions. Similarly, a clinical decision support system can be useful in helping healthcare professionals make clinical decisions that can help improve healthcare expenditure, while at the same time improving the quality of care to patients.

52 Chapter 2: Literature Review

2.8.1 Adoption of CDSSs The adoption of CDSSs by healthcare professionals in healthcare organisations, however, has been met with several barriers. Rousseau et al. (2003) identify key issues, like the ease of use, relevance and accuracy of messages and the flexibility to respond to other factors when making clinical decisions as limitations in the practical adoption of CDSSs. Furthermore, the process of implementing a clinical decision support system can be both tedious and expensive. Therefore, it is vital that every support system that is implemented is used effectively. Unfortunately, there are factors that can be the difference between adoption and rejection.

According to Rousseau et al. (2003), system familiarity is suggested to be a factor in the adoption of a decision support system. Clinicians’ inability to navigate through a new system, while having limited time for training, does not help to improve their usage of the support system.

Another factor being considered is the relevancy and accuracy of messages provided in the support system. One of the factors that Rousseau et al. (2003) highlighted is, the decision support guideline is not able to provide sufficient support for individual patients. This is interesting to note, as EBP has a similar limitation, which restricts its ability to be effectively applied to individual patients. Further literature into CDSSs revealed that Evidence-based medicine and knowledge translation had been the approaches adopted for most CDSSs (Chignell et al., 2010). While those applications have adopted the Evidence-based practice approach, the implementation of a CDSS alongside a Practice-based evidence approach to decision making seems worth exploring.

2.8.2 Limitations of current CDSSs Clinical decision support systems, as we know, are tools developed to assist healthcare professionals from the point of consultation to diagnosis (Castaneda et al., 2015). It provides support to healthcare professionals in enhancing patient care through intelligently filtering computer-generated knowledge and patient-related information (Sittig et al., 2008).The potential of CDSSs is highly regarded and its use is widespread across healthcare organisations. While its implementation has managed to improve care delivery, there have been some cases where CDSSs have failed to meet their potential and promise.

Chapter 2: Literature Review 53

Sittig et al. (2008) manage to classify the limitations or challenges of current CDSSs into three broad categories of (1) improving the effectiveness of CDSS interventions, (2) creating new CDSS interventions and, (3) disseminating existing CDSS knowledge and interventions.

Improving the effectiveness of CDSS interventions In improving the effectiveness of CDSS interventions, Sittig et al. (2008) identify five related limitations which are presented as challenges in CDSS implementations. The five challenges are to (1) improve human-computer interface, (2) summarise patient-level information, (3) prioritise and filter recommendations to users, (4) combine recommendations for patients with comorbidities and, (5) use freetext information to drive decision support. Improving the human-computer interface ensures that a CDSS works effectively and does not obstruct the clinical workflow unnecessarily. This will aid the process where healthcare professionals are prevented from making omission and commission errors. Summarising patient-level information, prioritising and filtering recommendations to users, and combining recommendations for patients with comorbidities focus on delivering care that is catered to individual patient’s needs and requirements. Summary of patient-level information helps healthcare professionals have access to full patient data content, rather than recalling them. This enables healthcare professionals understand patient’s current condition and make appropriate decisions peculiar to each patient optimally. Similarly, a CDSS which recommends interventions that are prioritised and filtered according to patient attributes as well as those with comorbidities, becomes much more robust and reliable for healthcare professionals’ decision making. Finally, the ability of a CDSS to extract freetext information in EHR systems has the potential of further improving the capabilities of that CDSS because there is valuable information that can be useful to healthcare professionals which are currently documented in freetext.

Creating new CDSS interventions In creating new CDSS interventions, two crucial challenges are highlighted as (1) prioritising CDSS content development and implementation and, (2) mining large clinical databases to create new CDSSs. The challenge in implementing a CDSS that can deliver quality help may be costly and takes a long period of time to develop. To that end, Sittig et al. (2008) have recommended prioritising CDSS content development and implementation so that in the long run, the implementation of CDSSs

54 Chapter 2: Literature Review

can be widespread. In complementing the content development, the availability of large clinical databases presents an opportunity where clinical knowledge can be increased by implementing new algorithms and techniques to mine them. These challenges, when overcome, can create a robust CDSS of quality.

Disseminate existing CDSS knowledge and interventions For disseminating existing CDSS knowledge and interventions, three challenges highlighted are (1) disseminating best practices in CDSS design, development and implementation, (2) creating an architecture for sharing executable CDSS modules and services and, (3) creating internet-accessible clinical decision support repositories. These challenges are concerned with performance, capability and accessibility of a CDSS for effective care delivery. By being able to share any CDSS design, development and implementation best practices, other healthcare organisations that seek to develop their own CDSS is able to leverage upon this knowledge and ensure that a usable, economical and robust CDSS can be built. Similarly, to create an environment where quality CDSS services can be easily ‘subscribed’, an architecture that allows sharing of these services is capable of equipping healthcare professionals with robust CDSS tools on the fly. Together with internet accessibility, CDSS capabilities can be shared easily and in real-time, further illustrating the enormous potential of future CDSSs.

The three grand challenges highlight the limitations of current CDSSs and illustrate what can be done to enhance the capabilities of future CDSSs to ensure that they meet the potential of supporting effective patient care delivery.

2.9 SUMMARY AND IMPLICATIONS

This literature review has illustrated the need to improve delivery and management of patient care, especially amidst the increasing prevalence of patients with comorbidities. The challenges posed by these circumstances require healthcare professionals to make important and challenging decisions regularly. With the widespread implementation of ICT, certain aspects of care delivery have been effectively supported, such as the improvement in health outcomes or the promotion of patient safety. The consequence to the use of ICT in healthcare industries, on the other hand, has resulted in the large generation of digital health data, such as EHRs, which contain valuable information that can potentially be utilised as evidence to assist

Chapter 2: Literature Review 55

in clinical decision making. With limitations from outcomes of RCT studies having a direct impact towards the effectiveness of Evidence-based practice, several research studies have recommended the use of alternative sources of evidence. Evidence that is drawn from actual clinical practice settings to support clinical decision making. As a result, a complementary approach to decision making called Practice-based evidence has been proposed. However, the leading proponents of the PBE approach have identified different ways of implementing it.

Due to the widespread adoption of ICT, studies have identified the potential of utilising large collections of digital health data or electronic health records to support care delivery. The evident of information relating to patients’ healthcare information suggests that such data can be utilised and adapted for patients with similar health attributes. This is because data captured in EHRs echoes the clinical practices or decisions made by healthcare professionals when providing care to their patients.

Based on this literature review and findings, we are introducing our new Practice-based evidence approach to decision making. This approach aims at adopting the use of EHRs as a source of evidence in informing decision-making, through an architecture that can integrate the disparate data sources which are apparent in most healthcare organisations. Our approach thus identifies the advantage of using the abundant, available and accessible EHRs as practical clinical evidence in place of the outcomes from RCT studies. However, integrating the disparate sources of digital health data into one is integral to having a comprehensive source of information that is reliable. Therefore, data warehousing technique is introduced where integration of data is made possible. Finally, a clinical decision support system will be the tool that actualises the PBE approach, utilising EHRs as the evidence that potentially assists healthcare professionals in making well-informed decisions.

The literature review therefore has provided the conceptual view that shapes this research study into three themes:

(1) The engagement of Evidence-based practice to direct practice of clinical medicine has highlighted several key limitations that are impacting on the effectiveness of care delivery to patients. Thus, our new Practice-based evidence approach recommends the use of evidence from actual clinical practice settings, i.e. data captured in EHRs, to inform clinical decision

56 Chapter 2: Literature Review

making, potentially making it practical and applicable to different patient types present.

(2) The limitations of outcomes from RCT studies reveals a gap where decision making can be further improved. Therefore, designing a CDSS with our PBE approach, using data captured from actual clinical practice as evidence may potentially result in providing healthcare professionals with greater assistance in making well-informed decisions.

(3) Information and data are crucial in any aspect of decision making. While the architecture of a data warehouse is not a new phenomenon, the application of it in healthcare fits like a glove. Since overhauling legacy systems to new integrated systems is close to impossible, data warehouse architecture ensures that information can be integrated, shared and retrieved efficiently. Thus, it will form a foundation where other emerging technologies like a CDSS can take advantage of it.

The literature review concludes by presenting the research possibilities that have thus far impeded the decision-making capabilities among healthcare professionals. It examines the potential that our PBE approach can bring to improving decision making, as well as the benefits of utilising EHRs as providing practical evidence in the process of decision making.

Chapter 2: Literature Review 57

[This page is left intentionally blank.]

58 Chapter 2: Literature Review

Research Design and Methodology

“In much of society, research means to investigate something you do not know or understand.” - NEIL ARMSTRONG

This chapter begins with an introduction to our research collaboration with a public hospital in Singapore. This is followed by a detailed description of the research methodology adopted and the corresponding research study design for the four phases of the study conducted. The chapter ends with a discussion of the research ethics and limitations of the study.

3.1 RESEARCH COLLABORATION WITH NATIONAL UNIVERSITY HOSPITAL, SINGAPORE

As part of this research study, a collaboration with doctors from National University Hospital’s (NUH) Regional Health Strategic (RHS) Planning Office of National University Health System (NUHS) was formed. The NUHS is an integrated academic health system with regional and national health responsibilities. The NUHS is made up of one acute hospital which is National University Hospital (NUH), three specialty centres and three Health Science University facilities. Their responsibilities include collaborating and working with private, public and people sectors to provide integrated care (NUHS, 2016).

This collaborative effort involved two doctors, Dr Sue-Anne Toh, Clinical Director of RHS Planning Office and Senior Consultant at Division of Endocrinology, and Dr Tan Xin Quan, Deputy Head and Associate Consultant at the RHS Planning Office. Dr Toh also heads a Metabolic Diseases and Diabetes research program with a particular interest in translational physiology, population health and diabetes. Dr Tan is responsible for NUHS’ population health strategy and development of new care models, with research interest in the areas of infectious diseases and chronic diseases.

This collaboration lasted for a period of one month although discussions had begun in early 2016. During this collaboration period, we were provided with a sample data set of anonymised inpatient cases to be used during the evaluation of our Practice-

Chapter 3: Research Design and Methodology 59

based evidence approach phase. However, access to the data set was only available within the hospital’s premises and through a dedicated computer provided.

3.2 METHODOLOGY AND RESEARCH DESIGN

3.2.1 Methodology Qualitative research studies are growing in popularity in Computer Science (CS), Information Systems (IS) and Information Technology (IT) disciplines, evidenced by the numerous studies adopting them. However, approaching a qualitative study may be complex. Creswell (2007) identifies five different approaches that can be adopted when conducting a qualitative study, i.e. narrative research, phenomenology, grounded theory, ethnography and case study, each with its own philosophical assumptions. However, in this study, we have decided on adopting a qualitative case study approach, as it enables the creation of an in-depth understanding (Crowe et al., 2011) of the most practical way of implementing our PBE approach to decision making. Moreover, through the approach of a collective case study, we are able to study multiple cases in which we can gain a more comprehensive understanding of the issues (Crowe et al., 2011) that may arise. Notably, qualitative case studies have been adapted in evaluation studies of computer systems and information technology (Kaplan & Maxwell, 2005), as seen by the works of Yang et al. (2015), and Feldman et al. (2015).

To reiterate, the overall aim of this qualitative research study is to evaluate the adoption of our Practice-based evidence approach as a solution to assists healthcare professionals in making well-informed decisions. This is achieved through the use of a clinical decision support system and electronic health records. Based on the opinions and viewpoints of healthcare professionals obtained during an evaluation process, we will be able to assess and determine if our PBE approach is capable of assisting healthcare professionals with clinical decision making. In essence, this study is focused on discovering the perceptions of healthcare professionals towards verifying the feasibility of our PBE approach. This research study however does not evaluate the comparative effectiveness of our PBE approach over other approaches, even EBP, for example. Therefore, it makes this study suited for qualitative research. Furthermore, qualitative studies are designed to gain an understanding of people’s issues and specific situations investigated through their behaviours and perspectives within the

60 Chapter 3: Research Design and Methodology

social and cultural situation they are in (Kaplan & Maxwell, 2005; Myers & Avison, 1997).

In this qualitative case study, we begin with the conceptualisation of our Practice-based evidence approach to decision making followed by three separate but related cases to be conducted sequentially. This benefit of conducting collective case study as highlighted by Crowe et al. (2011), not only enables us to finally evaluate the effectiveness of our PBE approach but do so by first highlighting the significance and impact provided by each case. This type of case study help to generate a broader understanding of certain issues as mentioned by Crowe et al. (2011). In our study, this helps us to understand the approach that leads to improved clinical decision making. The three cases are (1) qualitative study on the ICT architecture development for our PBE approach, (2) quantitative survey on the perceived clinical benefits of EHR systems and perceived usefulness of EHR data for decision making, and (3) qualitative evaluation of our PBE approach to decision making as a solution to assist healthcare professionals make well-informed decisions.

In the following sections, the conceptualisation of the new PBE approach and the three studies are discussed in summary.

3.2.2 Research Design Study (1): Conceptualising Our Practice-Based Evidence Approach to Decision Making The goal of this first study is to define and conceptualise our Practice-based evidence approach to decision making. The purpose of this phase of the study is to answer the following research sub-question

RQ1: What will be an effective Practice-based evidence approach to decision making in a clinical setting? In order to answer the above question, the corresponding research objective has to be achieved. Based on the review of relevant literature, three components to the conceptualisation of our Practice-based evidence approach to decision making have been uncovered. The components are (1) electronic health records, (2) data warehouse and, (3) clinical decision support system.

3.2.3 Research Design Study (2): Architecting for Practice-based Evidence Approach The goal of this phase of the research is to define the ICT architecture required for our new Practice-based evidence approach. This qualitative study is achieved by

Chapter 3: Research Design and Methodology 61

interviewing the IT professionals from Integrated Health Information Systems (IHiS) Pte Ltd regarding the ICT infrastructure of National University Hospital (NUH). The aims of the interviews are (1) to identify the various clinical information systems implemented in the hospital, (2) to understand how the EHR systems support doctors with their care delivery through the flow of data and information, and (3) to find out if there exists any mechanism that integrates multi-sourced data. Therefore, the goal of this phase of the study is to answer the second research sub-question identified from the overarching research question. In doing so, the question that is directly answered is

RQ2: What changes to the current state of a healthcare organisation’s ICT architecture are required to adopt a Practice-based evidence approach in assisting healthcare professionals with clinical decision making? In order to answer the research question above, the corresponding research objectives have to be achieved. To do so, an appropriate research instrument and data collection methods are required. Semi-structured interviews and supporting document analysis are the research method employed for data collection. The outcome of this study is the envisioned ICT architecture of NUH. Through this architecture, the sources of clinical information systems, corresponding electronic health records for the clinical decision support system and the data integration method for our proposed Practice-based evidence approach can be identified.

Interviewing is one of the most common methods of data collection in a research study (Jamshed, 2014). Interviews enable the researcher to “contribute to the body of knowledge that is conceptual and theoretical and is based on the meanings that life experiences hold for the interviewees” (DiCicco-Bloom & Crabtree, 2006, p. 314). Therefore, through interviews, this research is able to elicit valuable information from the interviewees based on their knowledge and experience and bring to light the extent of the ICT architecture of NUH. Moreover, the use of interview sessions as a manner of collecting data from participants helps to understand the architecture or systems implementation in organisations, which have been conducted before in similar case studies, such as the one by Zakaria et al. (2010). However, the interview questions have to be specifically developed for this study, as there are no prior or related studies that have categorically conduct investigations to gain information regarding the ICT architecture of hospitals in Singapore.

62 Chapter 3: Research Design and Methodology

The decision to employ a semi-structured interview is also influenced by several factors. Firstly, it is essential to identify the extent of information systems’ use and implementation in the hospital in order to understand the distribution of unintegrated digital health data. Secondly, an extensive literature search regarding the ICT architecture of Singapore’s public healthcare organisations results in few or no publications. In fact, what little information that is available is in the form of presentation slides by Singapore’s Ministry of Health Holdings (MOHH) and IHiS, regarding Singapore’s National Electronic Health Record (NEHR) architecture (Tan & Ong, 2009). Additional documents include three yearbooks by IHiS. One that reports on the latest IT systems implemented in public hospitals in Singapore (IHiS, 2016) and two that detail the ICT systems implemented throughout the public hospitals and polyclinics in Singapore (IHiS, 2011, 2015). These additional documents are used to supplement and support findings from the interviews conducted.

The analysis is carried out using direct content analysis and thematic analysis through iterations on the primary interview data and additional supporting documents, in order to gain meaningful insights.

Based on the findings from the analysis, the outcome of this study is the envisioned ICT architecture of NUH. This architecture helps to determine the data sources for our PBE approach, the data integration method and the final data for the clinical decision support system.

3.2.4 Research Design Study (3): Perceived Clinical Benefits of EHR Systems and Perceived Usefulness of EHR Data for Decision Making The goal of the next phase of the research study is to investigate the suitability of EHR data to be used as evidence in our Practice-based evidence approach to decision making. This mixed qualitative and quantitative study is achieved through the use of an online survey questionnaire conducted with healthcare professionals from Singapore’s public hospitals and polyclinics. The aims of the survey are to understand doctors’ perception of the clinical benefits of EHR system use and the perceived usefulness of EHR data in assisting with clinical decision making. From these findings, we can therefore determine if EHRs are indeed a suitable source of evidence that captures actual clinical practices in digital form.

This phase of the study targets at accomplishing Stage 3 of the planned research stages. In doing so, the following research sub-question is asked:

Chapter 3: Research Design and Methodology 63

RQ3: What are the processes required that ensure a Practice-based evidence approach can assist healthcare professionals in clinical decision making?

Figure 6. Resource-based view model adopted

Following the works of Kwon et al. (2014), Resource-based view (RBV) theory is adopted to empirically identify how a firm’s attributes affect the adoption of big data analytics. Similarly, RBV theory can be utilised in this research study to understand how the perceived benefits of using EHR systems and the perceived usefulness of EHR data affect decision making. While the Technology Acceptance Model (TAM) can also be used in this Research Study 3, the study is not focused on the adoption of technology, but studying the effects of EHRs as tangible resources of a firm in gaining a competitive edge over others. In this case, the study concentrates on how EHR data that is captured within the healthcare organisation can be used as evidence of actual clinical practices to assist with clinical decision making.

In the RBV model above, the degree of EHR system functionalities represents how EHR systems are utilised by healthcare professionals in managing patient care. From this list of functionalities, healthcare professionals are surveyed on whether such functionalities provide any clinical benefits that help them with decision making, and how the corresponding data that is captured, relates to decision making. The degree of EHR data quality represents the types of quality of data captured in an EHR. In turn, these qualities are surveyed regarding their relationship with the clinical benefits provided by the EHR systems and the usefulness of the data for decision making.

64 Chapter 3: Research Design and Methodology

Finally in the model, how data is made available is surveyed to understand its relationship with decision making.

3.2.5 Research Design Study (4): Evaluating our Practice-based Evidence Approach The goal of the final phase of the research is to evaluate our Practice-based evidence approach, implemented in a clinical care setting at NUH, as a viable approach to assist with decision making. As part of this study, a prototype CDSS implementing our PBE approach is developed, utilising the sample anonymised data set of inpatient cases. The use of this data set has enabled the generation of patient-centric statistics and a prediction model. In this qualitative study, it consists of a field testing of the prototype CDSS and is followed up with a focus group discussion.

The research question that is asked during this phase of the study is similar to the one asked in the previous study. The answers to this question are obtained from research stages 4, 5 & 6.

RQ3. What are the processes required that ensures a PBE approach can assist healthcare professionals in clinical decision making? The motivation to conduct a field testing and focus group is influenced by the research aim, which is to evaluate the possibility of implementing our PBE approach to assist with decision making in an actual clinical care setting at NUH. A field test allows participants to understand how our Practice-based evidence approach to decision making can be actualised in a decision support system. A focus group, as evidenced by Kitzinger (1995, p. 299), is effective in discovering participants’ thought processes and why they think the way they do. It is also an efficient way for participants to be engaged in a group discussion, ask questions, “explore and clarify their views” and generally share their thoughts and beliefs. Hence, employing the focus group discussion is important, as it allows us to gain an understanding of participants’ observations and views regarding the perceived effectiveness of our PBE approach to assist with decision making, based on participants’ existing knowledge and experiences of using EHR systems.

In order to elicit their perceptions regarding our PBE approach to decision making, thematic analysis is performed on the data collected during the focus group session. Iterations are performed until clear and consistent themes are reached.

Chapter 3: Research Design and Methodology 65

Finalised themes arising from the analysis therefore describe the practicality of adopting or implementing our PBE approach.

Based on the findings from the analysis, the outcome of this study also represents the doctors’ views on the benefits of our PBE approach in its ability to assist healthcare professionals with decision making.

3.3 ETHICS AND LIMITATIONS

We have obtained the necessary ethical clearance needed for all phases of the research study before its commencement, from our educational institution, Queensland University of Technology (QUT).

There are no ethical issues or problems arising from any of the studies conducted. The ethical clearance certificate is also available as an appendix (Appendix J.)

Research Study 2 is limited to the IT professionals from IHiS Pte Ltd. The study is also limited to the understanding of the ICT architecture of NUH, although the actual layout, map or blueprint is not provided as a secondary source of data during this phase of the study. Instead, the layout of the ICT architecture envisioned is incorporated together with the data warehouse architecture as illustrated by supporting literature.

Research Study 3 is limited to doctors from at least three public hospitals and one polyclinic in Singapore. However, we did not involve other doctors from the private hospitals and private clinics in this study. The study aims at eliciting the perceptions of healthcare professionals towards the benefits of using EHR systems and the corresponding usefulness of the data captured in EHR to assist in decision making. The findings from the above perceptions determine the suitability of EHR data for our PBE approach. However, the study is not aimed at defining the determinants for each perception.

Research Study 4 aims at evaluating our Practice-based evidence approach that utilises electronic health records and clinical decision support system to assist in clinical decision making. The study is limited to healthcare professionals and IT professionals working in National University Hospital. The data provided by NUH in the study is limited to hospitalisation episodes of patients with diabetes. As part of the study, the use of a predictive model is limited to Poisson regression and not decision

66 Chapter 3: Research Design and Methodology

trees or other regression models. The predictive model is also limited to predicting the probable length of hospital stay.

Chapter 3: Research Design and Methodology 67

[This page is left intentionally blank.]

68 Chapter 3: Research Design and Methodology

Conceptualising Our New PBE Approach

“The value of an idea lies in the using of it”. - THOMAS A. EDISON

The research design and methods chapters comprise four major studies. In this chapter, these studies begin with the conceptualisation of our Practice-based evidence approach to decision making. Next in this chapter is the introduction of EHRs as a key component of the approach, with the fundamental data warehouse architecture and clinical decision support system as the essential tools to implement our PBE approach. The chapter ends with a summary and implications.

4.1 OVERVIEW

Our literature review on the increasing prevalence of patients with chronic conditions, current limitations of clinical practice approaches for decision making and the nature of healthcare ICT have provided a useful background for the conceptualisation of our new Practice-based evidence approach to decision making in this research. Barkham and Margison (2007, p. 446), one of the leading proponents of Practice-based evidence, have defined PBE as the “conscientious, explicit and judicious use of current evidence drawn from practice settings in making decisions about the care of individual patients”. Following this definition of Practice-based evidence, this chapter describes the conceptualisation of our Practice-based evidence approach to clinical decision making. The key components of our PBE approach are (1) employing a data warehouse architecture to integrate multiple data sources, (2) utilisation of electronic health records as alternative source of evidence and (3) the use of a clinical decision support system to aid healthcare professionals with assistance in decision making.

4.2 CURRENT IMPLEMENTATION OF PRACTICE-BASED EVIDENCE

The conceptualisation of our Practice-based evidence approach to decision making has been very much influenced by the works of Barkham and Mellor‐Clark (2003) and, Horn and Gassaway (2007). Their recommendation for Practice-based

Chapter 4: Conceptualising Our New PBE Approach 69

evidence has been primarily motivated by issues with the limitations of evidence used in Evidence-based practice, where the leading tool used to generate them is randomised controlled trials. This exposes a gap where the use of such evidence lacks applicability, especially when used to generalised care for all types of patients. Therefore, Practice-based evidence, which entails the gathering and using of evidence drawn from practice or clinical settings, aims to bridge this gap and improve the applicability of evidence used (Barkham & Margison, 2007; Green & Latchford, 2012).

To recap, in Barkham and Mellor-Clark’s approach to Practice-based evidence, the authors recommend the use of Practice research networks (PRN) to collect, share and use data gathered during actual practice to direct care (Barkham & Mellor‐Clark, 2000). A PRN enables a large group of clinicians to work together in accumulating, storing and documenting clinical data that can be used to improve the delivery and development of care services. Horn and Gassaway’s work however, introduce Practice-based evidence as a clinical research methodology approach designed for clinical practice improvement called PBE-CPI (Horn & Gassaway, 2007). Similar to how randomised controlled trials are conducted, PBE-CPI differs only in how patient complexities and treatment variations in an actual clinical care practice are taken into account. This allows patients with comorbidities to take part in such studies which is the opposite for RCTs. As a result, evidence drawn from PBE-CPI becomes more applicable to a greater population of varied patient types (Horn & Gassaway, 2007). Therefore, both the works of Barkham and Mellor-Clark, and Horn and Gassaway, focus on bridging the gap created with Practice-based evidence. In both implementations of PBE approaches above, the authors emphasise the importance of using data gathered or collected during actual practices in clinical settings. It is believed that by improving on the evidence generated, it can be used to make effective decisions about individual patients.

Our research is also greatly concerned with this ineffectiveness and poor applicability of evidence used to inform decisions regarding the care of actual patients. With that, one of the key components of our Practice-based evidence approach focuses towards identifying a suitable source of alternative evidence, specifically data that captures healthcare professionals’ clinical practices taking place in actual clinical settings, i.e. “practice-based” data, like electronic health records. Thus, an EHR

70 Chapter 4: Conceptualising Our New PBE Approach

represents precisely the type of practical evidence required. EHRs contain valuable information regarding clinical practices performed in actual clinical settings and can therefore be used in our approach to direct clinical care. It is anticipated that the benefits of leveraging on such evidence will be notable in the ability to make well- informed decisions regarding actual patients.

4.3 DEFINING OUR APPROACH OF PRACTICE-BASED EVIDENCE TO DECISION MAKING

Our research study together with the works of Barkham and Mellor-Clark, and Horn and Gassaway, look to focus on bridging the gap created, as mentioned above, with Practice-based evidence. However, this is where the similarity ends. Barkham and Mellor-Clark and, Horn and Gassaway focus on the outcome of evidence to be applicable and relevant to differing patients through the Practice-based evidence approach. Our focus is instead on designing a practical ICT approach where the use of existing evidence of actual clinical practice (electronic health records) can assist healthcare professionals in making well-informed decisions. In doing so, we have also drawn inspirations from the works of other authors on the use of a clinical decision support system as the mechanism to drive our Practise-based evidence approach to decision making and clinical data warehousing on integrating mostly disparate sources of data within the healthcare organisation.

Figure 7. Logical model of the key components in our Practice-based evidence approach

Chapter 4: Conceptualising Our New PBE Approach 71

Therefore, the practical conceptualisation of our approach in this thesis begins with defining our Practice-based evidence approach to decision making. From the ensuing definition, our approach identifies three key components: (1) a data warehouse architecture, (2) the use of electronic health records as an alternate source of evidence, and (3) a clinical decision support system. This is represented in a logical model representing the key components as illustrated in Figure 7.

The introduction of our Practice-based evidence aims at assisting the clinical decision-making process for healthcare professionals through the use of practical evidence, i.e. electronic health records, captured during actual clinical practice. The extensive adoption of healthcare ICT has helped in the generation of large collections of such evidence which capture many levels of patient health information digitally. Therefore, we define our Practice-based evidence as the approach where

“sources of meaningful evidence of clinical practices performed by healthcare professionals as part of their routine practice, such as electronic health records and electronic medical records, are integrated, and this practical and comprehensive evidence used in a decision support system to support and inform clinical decision making towards the care of individual patients”

However, with that being said, there have also been other research studies which appear to be similar to our new PBE approach, and therefore, such bodies of work and their contributions deserve to be mentioned and acknowledged. Research on “personalised cohort studies” and EHR system “green buttons” have similarly suggested the use of large clinical data repositories, such as EHRs, as an alternate source of evidence to direct or inform clinical decision making (Gallego et al., 2015; Longhurst et al., 2014) In “personalised cohort study”, Gallego et al. (2015) propose a framework where “EHRs of past similar patients can be queried as an alternate source of evidence to inform decision making”. This framework builds upon the EHR system “green button” approach, conceptualised by Longhurst et al. (2014). The “green button” approach is a function that can be built within EHR systems that helps “clinicians to leverage on aggregated patient data for decision making at the point-of- care” (Longhurst et al., 2014). Both bodies of work identify the value of EHRs and how they can be capitalised to aid healthcare professionals in their daily care routine, as do our PBE approach. While acknowledging the existence and potential of such similar works, we may have also inadvertently questioned the novelty of our PBE

72 Chapter 4: Conceptualising Our New PBE Approach

approach. But, at the same time, it also signals the significance and relevance of our PBE approach because we have identified the necessary ICT architecture and processes required to implement an approach which potentially contributes to a better practice of clinical medicine. In fact, at the point where this thesis research study is being conducted, both similar body of works appear to still be in its theoretical phase. The framework identified by Gallego et al. (2015) is said to be a “proposed process for generating real-time cohort studies at point-of-care”, thus it may not yet be actualised into existing EHR systems or CDSSs. Similarly, Longhurst et al. (2014) noted that while the technologies needed to implement the “green button” approach may have already existed, it has yet to be made into reality because the support systems, such as investments and policy frameworks required to deploy the necessary technology is still lacking. On that account, this may mean that a CDSS implementation based on our PBE approach to decision making would be among the first to be implemented. In addition, demonstrating our prototype in a clinical practice environment, with diabetes as the disease condition of research, further shows its significant contribution.

4.4 EHRS AS SUITABLE EVIDENCE FOR PBE

The adoption of EHR systems across healthcare organisation means that information related to patient care in clinical settings is captured meaningfully, stored and can be retrieved easily on a regular basis. EHRs essentially contain documentary “evidence” of the “practice” conducted by clinicians during care delivery. That makes EHRs data-rich and a suitable candidate for our PBE approach, as they collate evidence from the perspective of clinicians in practice in an actual clinical environment.

EHRs have primarily been used to document patient care, and the assessment of outcomes of care delivered. The amount and quality of information recorded in EHRs has an impact on the outcomes of patient care and continuity of care (Häyrinen et al., 2008). As highlighted in our literature review, EHRs contain vast collections of information such as present medical complaint, past medical history, diagnoses, laboratory and radiology tests, procedures, treatment and medication, to name a few (Häyrinen et al., 2008). With patients with multiple conditions, EHRs contain care information provided by different healthcare professionals. Hence, EHRs are able to record detailed information about the outcomes of treatment, medication and tests conducted as decided upon by the attending healthcare professionals. Häyrinen et al.

Chapter 4: Conceptualising Our New PBE Approach 73

(2008) even reported that recent studies have explored how EHRs can be a source for Evidence-based medicine because of the vast amount of information they capture.

To present the application of our Practice-based evidence approach to decision making through the utilisation of electronic health records as evidence drawn from actual clinical practice settings, we illustrate this approach through an e-Health scenario. In this process, the approach is detailed according to the processes that take place in an actual clinical practice setting as well as highlighting the importance of EHRs. We first published the conceptualisation of our Practice-based evidence in a book chapter entitled Data-driven and Practice-based evidence: Design and Development of Efficient and Effective Clinical Decision Support System (Osop & Sahama, 2015). In it, the concept of Practice-based evidence was described through a patient journey as illustrated in Figure 8 and Figure 9.

Figure 8. Adoption of Evidence-based practice in a clinical practice setting (Osop & Sahama, 2015)

Patient journey Patient A visits Doctor B at a local hospital. Information regarding Patient A’s health and medical background and current complaint is exchanged during the consultation. This enables Doctor B to understand Patient A’s ongoing health problems and past medical history. Once that understanding has been established, Doctor B

74 Chapter 4: Conceptualising Our New PBE Approach

makes a judgement, based on clinical expertise, on the next appropriate course of intervention. This decision-making process relies on the experience, knowledge, skills and values gained over the years of practice. On top of that, Doctor B refers to clinical guidelines and clinical evidence gathered from systematic research. Together with clinical expertise and the evidence found, Doctor B applies this to decision making and plans the intervention. This process illustrates the paradigm of Evidence-based practice as shown in Figure 8.

Now, through the adoption of ICT such as the use of EHR systems, extensive clinical data is captured regularly into electronic health records. This also includes the actions performed by Doctor B, based on the decisions made, which relied on the clinical expertise and clinical evidence from systematic research that determined the intervention provided. However, in the traditional approach of EBP, the health records are referred back again only when the need arises to assess the outcome of care (Häyrinen et al., 2008). Therefore, when Patient A returns for a follow-up visit, Doctor B uses the EHR system to view the patient’s historical records and reiterate the process of understanding the patient’s health condition.

Figure 9. Illustrating the Practice-based evidence approach to decision making (Osop & Sahama, 2015)

However, in our Practice-based evidence approach to decision making as illustrated in Figure 9, Doctor B begins with integrating the use of clinical expertise

Chapter 4: Conceptualising Our New PBE Approach 75

and the evidence from systematic research in deciding on the appropriate intervention for Patient A. This process is then meaningfully captured in electronic health records. However, instead of merely assessing the health records to retrieve Patient A’s medical history, data from other clinical information systems within the healthcare organisation can be integrated to provide detailed and extensive information about Patient A. Additionally, data of other patients within the same healthcare organisations are integrated and analysed to provide a comparative base of information where health and healthcare-related information of other patients similar to Patient A can be forwarded as evidence and knowledge, to improve the decision making for Doctor B.

In the following paper published for IEEE Healthcom’16 (18th International Conference on E-Health Networking, Application & Services), we illustrated a practical example of the adoption of Practice-based evidence to decision making through an e-Health scenario (Osop & Sahama, 2016a). Excerpts from the illustration are provided below.

“Adam suffers from diabetes and hypertension, and has adverse drug allergies to metformin and insulin. In order to treat his diabetic condition, Dr Henry prescribes a new medication, ‘X’, as described on a new clinical guideline he comes across. Dr Henry documents Adam’s drug allergies and orders medication ‘X’ for him. A few days later, Adam returns to complain that he has developed a rash after taking the new medication. Dr Henry decides to stop the medication and orders an allergy test for Adam. As it turns out, because Adam has been taking Atenolol as well for his hypertension, it has reacted with the new medication ‘X’ resulting in him getting the rash. Dr Henry captures his findings in the clinical notes and after researching other clinical guidelines, prescribes Adam with a new medication ‘Y’. Medication ‘Y’ is ordered and captured in Adam’s EHR. Adam has had no further allergic reactions to the new medication and both his chronic conditions are being managed effectively.

Meanwhile, Dr James, from the same healthcare organisation, is visited by a patient, John, who happens to suffer from diabetes and hypertension as well as an allergy to insulin. In the Practice-based evidence approach to decision-making scenario illustrated in Figure 9, the information and data from all the patient’s health records are integrated and fed into a decision support system where it is analysed and adapted to provide assisted decision making based on John’s medical condition. Through this PBE approach, Dr James is alerted to the possibility of prescribing John

76 Chapter 4: Conceptualising Our New PBE Approach

with two new medications to treat his diabetic condition, medication ‘X’ and ‘Y’. Dr James is also alerted that since John is on Atenolol, it is advisable to prescribe medication ‘Y’ since there is evidence that medication ‘X’ has a drug-drug interaction with Atenolol. That evidence is presented based on the existence of information captured in Adam’s health records. Without the Practice-based evidence approach, Dr James would have to spend more time finding suitable clinical guidelines that are applicable to John and may not have found out the drug-drug interaction medication ‘X’ has with Atenolol”.

Both the patient journey and e-Health scenario illustrated above, provided examples of how EHRs can potentially be used as evidence in our Practice-based evidence approach to assist healthcare professionals with clinical decision making. However, with such data scattered all over healthcare organisations, an architecture for integrating them is necessary, such as data warehousing.

However, there are potentially similar studies or bodies of work to the patient journey and e-Health scenario illustrated in our Practice-based evidence approach to decision making. It is therefore vital that we understand the similarities and differences between these bodies of work and that of ours. The similar bodies of work that we are referring to are PatientsLikeMe and “personalised cohort studies”.

According to Wicks et al. (2010), “PatientsLikeMe is an online quantitative personal research platform for patients with life-changing illness to share their experience using patient-reported outcomes, find other patients like them matched on demographic and clinical characteristics, and learn from the aggregated data reports of others to improve their outcomes”. Frost and Massagli (2009) describe PatientsLikeMe as a “web tool that merges the functionality of an online peer support community with patient-oriented treatment, symptom, and health outcome data”. PatientsLikeMe seeks out to enable the users of the tool to share health-related information with similar typed patients. Brubaker et al. (2010) describe PatientsLikeMe as a patient networking site that “facilitates information sharing between patients within disease-specific communities” where such patients are able to “track and share relevant information such as symptoms, treatments, and medical data”, allowing them to “compare their experiences to other patients and empower them to take a more active role in determining treatment options with their care providers”.

Chapter 4: Conceptualising Our New PBE Approach 77

Research on “personalised cohort studies” have similarly suggested the use of large clinical data repositories, i.e. EHRs, as an alternate source of evidence to direct or inform clinical decision making (Gallego et al., 2015). In “personalised cohort study”, Gallego et al. (2015) propose a framework where “EHRs of past similar patients can be queried as an alternate source of evidence to inform decision making”. This framework builds upon the “green button” approach, conceptualised by Longhurst et al. (2014). The “green button” approach is a function that can be built within EHR systems that helps “clinicians to leverage on aggregate patient data for decision making at the point-of-care”. Both bodies of work identify the value of EHRs and how they can be capitalised to aid healthcare professionals in their daily care routine, as does our PBE approach.

While PatientsLikeMe and the “green button” approach seem to be similar to our PBE approach, it does not however actively involve healthcare professionals in the decision making process. In our PBE approach to decision-making, the presence of healthcare professionals is vital in order for any decision making to take place. Plus, the decision making it is supported by the knowledge and expertise of the healthcare professionals. In fact, PatientsLikeMe has been described as being a research platform and not used in an actual clinical setting. Additionally, PatientsLikeMe is said to support members more psychologically and increases the levels of emotional well- being rather than treatment or outcomes wise (Frost & Massagli, 2009). On the other hand, there have been no studies to confirm that the use of PatientsLikeMe is damaging. While the concept of using information and data from similar patients appear to be the same, the application of PBE in actual clinical practice settings in assisting with making clinical decisions differs greatly to PatientsLikeMe.

EHR Data Quality Dimensions With great emphasis on the advantages of using EHRs, it is also vital that the EHR data to be used as evidence in our PBE approach has the quality dimensions that allow relevant, appropriate and correct inferences to be made from. This is because EHR data, which potentially contains evidence capturing actual clinical practices as indicated in the Patient Journey, has the potential to be used to support and assists other healthcare professionals in making well-informed decisions. Accordingly, we have considered several of the data quality dimensions recommended by Weiskopf and Weng (2013), Almutiry et al. (2013) and Kahn et al. (2016) highlighted in Table 1 and

78 Chapter 4: Conceptualising Our New PBE Approach

identified the following aspects of data quality that will be appropriate for our PBE approach to decision making.

Proposed Data Quality for PBE approach Dimension Definition Accuracy Data being free from error or defect Practical Data which consists of information resulting from clinical practices Relevant Data relating directly and significantly to clinical practices Reliable Data being dependable in accuracy, being able to put trust in Secure Data which is safe and not wrongfully tempered with Valid Data being sound and well-founded.

Table 2. Dimensions of data quality for our PBE approach to decision making

The six aspects of data quality which we have identified as quality suitable for our PBE approach are Accuracy, Practical, Relevant, Reliable, Secure and Valid. Accuracy, as identified by Almutiry et al. (2013) is the “extent to which registered data conforms to its actual value”. Accuracy is also identified as synonyms to three data qualities by Weiskopf and Weng (2013), Completeness, Correctness and Plausibility. Accuracy thus represent a critical quality dimension of data that ensures the data reflects its actual value where it is complete, correct and trustworthy. In fact, we have defined Accuracy as data that is free from error or defect. A key component of our PBE approach is the use of evidence from activities carried out in routine practice settings. The next data quality dimension we are concerned with is the Practical aspect of data. While this dimension is not identified by any of the authors above, it is in fact a significant dimension especially in our PBE approach because it identifies data that is recorded based on the actions or activities performed by healthcare professionals. Thus, it represents the evidence of clinical practices which is vital in its potential to guide and assists healthcare professionals with decision making. We have defined Practical as data that consists of information resulting from clinical practices. Similarly, Relevant is concerned with dimension of data quality where data relates directly and significantly with clinical practices. With our PBE approach focusing on providing capabilities of making well-informed decisions, it is therefore important that EHR data records information which is Relevant. In fact, we have included it as one of the six dimensions because based on the study conducted by Almutiry et al. (2013), they have identified it as an information-related dimension to indicate the appropriateness and usefulness of data. Reliable has been identified as a synonym for

Chapter 4: Conceptualising Our New PBE Approach 79

Concordance by Weiskopf and Weng (2013), which is also a synonym for Consistency. And Consistency has been defined by Almutiry et al. (2013) as values that “remains the same in multiple data items in multiple locations”. Thus, this is important especially when considering that data sources in most healthcare organisations are disparate, with the need to be integrated to make it comprehensive and valuable for analysis. Similarly, with our PBE approach identifying a data warehouse methodology to integrate multiple sources of data together, Reliable becomes an important dimension in our data quality definition. Similarly, Valid has the same standing as Reliable with the addition of Usability and Plausibility. Valid thus ensures that data is dependable and trusted to be used for with ease and that there is truthfulness in the values recorded in the data. Finally, Secure represents the dimension of data quality such that, the data is safe and not wrongfully tempered with. This is important because as defined by Almutiry et al. (2013), it prevents data from being corrupted and its validity, reliability and relevance questioned.

With concerns that data captured in EHR systems are not of the quality to be appropriately used in research or for clinical findings, many researchers have identified several dimensions of data quality as a way to measure its value. The incorporation of our six aspects of data quality dimensions which are influenced by the findings by Almutiry et al. (2013), Kahn et al. (2016) and Weiskopf and Weng (2013) aims at ensuring the data or evidence used in our PBE approach is suitable in generating usable inferences that can help in decision making. These data quality dimensions will be further analysed in Chapter 6.

4.5 FUNDAMENTAL ARCHITECTURE FOR PBE – DATA WAREHOUSING

4.5.1 Warehouse Architecture and Model With most sources of data from information systems across healthcare organisations being generally scattered and unintegrated, a clinical data warehouse represents the best solution to consolidate all the evidence into a single repository. According to Sahama and Croll (2007), a clinical data warehouse is one of the most efficient data repositories available that is capable of delivering quality patient care, however, CDWs are clearly very complex and time-consuming to use especially when used to review series of patient records. Therefore, these are some of the challenges that need overcoming when deciding on implementing a CDW architecture.

80 Chapter 4: Conceptualising Our New PBE Approach

Lyman et al. (2008) describe the development of a data warehouse as involving three tasks. First, is the design and implementation of the data warehouse architecture. There are several data warehousing architectures available with each being suitable for different situations. Sen and Sinha (2005) share five such warehousing architectures such as the enterprise data warehouse architecture, the data mart architecture, the hub- and-spoke data mart architecture, enterprise warehouse with the operational data store and the distributed data warehouse. Next, data marts can be designed, which represent a subset of the data in the data warehouse itself and are optimised for faster access. However, there is no “one size fits all” architecture that can satisfy the requirements of all institutions.

Second is populating the data warehouse through the process of data acquisition, extraction, transformation and loading, commonly known as Extraction- Transformation-Loading (ETL). This ensures that all information that is acquired from source systems are transported and loaded into the data warehouse whilst maintaining its integrity. Data is cleaned, duplicated and merged into the data warehouse. This ensures that the data is free from errors and irregularities. It is imperative that data quality is upheld since it will be used for decision making.

Third are the proper data access and query management, and security and user services. An OnLine Analytical Processing (OLAP) tool or reporting tools are used during this process, to generate the information required by the users. The use of data marts allows different users of the data warehouse faster access to information relevant to their requirements. For example, doctors can access information on clinical activity from a specified period of time or identify patients that were treated with a particular medication. Additionally, the use of decision support systems with OLAP tools can inform doctors of potential errors and assist in effective decision making for improvements in care delivery.

A data warehouse is therefore essential for our PBE approach because it facilitates the integration of many sources of data. With the key component of our PBE approach being the use of EHRs as the alternative source of evidence suitable in assisting with decision making, a data warehouse architecture allows the creation of a single repository of data that is a complete and comprehensive collection of all patients’ health information. This is especially true when there exist various standalone clinical information systems that do not share information and serves individual

Chapter 4: Conceptualising Our New PBE Approach 81

clinical department needs, across the healthcare organisations. Furthermore, this integrated data also improves the quantity and quality of information for healthcare professionals to consult on (Ado et al., 2014). This becomes beneficial for healthcare professionals as they can gain access to quality evidence that can greatly assist them in making well-informed decisions.

4.6 CLINICAL DECISION SUPPORT SYSTEM FOR PBE APPROACH

The use of a clinical decision support system (CDSS) has primarily been based on Evidence-based practice or Evidence-based medicine. Therefore, implementing the PBE approach to a clinical decision support system will address some of the limitations and challenges of CDSS implementation highlighted in the literature review.

CDSSs have often been used to assist healthcare professionals with a guided and directed approach to decision making. This has allowed decision making to be less tedious and complicated. It is therefore essential that CDSSs be a part of the solution in implementing our Practice-based evidence approach to decision making. With an integrated and comprehensive collection of patients’ health information, a decision support tool that is able to draw relevant inferences and recommendations is hugely required. It is both appropriate and fitting, since CDSSs will be able to guide the process of decision making effectively through the use of practical clinical evidence, such as electronic health records.

4.7 SUMMARY

The main aim of our Practice-based evidence approach is to assist healthcare professionals in making well-informed decisions and potentially improve their decision-making capabilities. It is an approach that complements issues with Evidence-based practice due to the limitations of evidence used. In the conceptualisation of the approach, the key components we have identified are the use of electronic health records as an alternative source of evidence, a data warehouse architecture to integrate sources of data and a clinical decision support system as a tool that assist healthcare professionals with decision making. This has been illustrated logically in the model depicting our Practice-based evidence approach in Figure 7.

Electronic health record is the alternative source of evidence identified in place of outcomes from systematic studies as it digitally captures information about actual

82 Chapter 4: Conceptualising Our New PBE Approach

clinical practices performed by healthcare professionals during care activities. Therefore, it can be used as practical clinical evidence or evidence of “practice-based” data, which has the potential to assist in informing healthcare professionals’ decision making. In our PBE approach, EHRs begin as records that are stores of patient health and healthcare-related information. When brought into practice through the use of a clinical decision support system, these records become evidence of actual clinical practices performed and inferred from when directing decision making. Together with data warehousing, these valuable data is integrated to provide a complete and comprehensive collection of information that is more informative and becomes applicable to differing patient types. The CDSS completes the implementation of our PBE approach by guiding and assisting healthcare professionals in making effective decisions through using inferences drawn from the integrated collection of electronic health records.

Chapter 4: Conceptualising Our New PBE Approach 83

[This page is left intentionally blank.]

84 Chapter 4: Conceptualising Our New PBE Approach

Architecting Our New Practice- Based Evidence Approach

“Research is creating new knowledge.” - NEIL ARMSTRONG

The chapter is the second out of four separate studies describing the three differing but integrated objectives of this overall research. This chapter discusses the study conducted for the architecture required for our Practice-based evidence approach. It begins with an overview of the research study, a recap of the research design. This is followed by detailed description of the research participants, instruments, procedures and timeline. The chapter ends with a thorough analysis of the research study findings.

5.1 OVERVIEW

Singapore’s healthcare system consists of eight public hospitals, six of which are acute general hospitals, one women’s and children’s hospital and one psychiatric hospital (Ministry of Health Singapore, 2016b). In addition, there are 20 polyclinics which are described as ‘one-stop’ healthcare centres. These polyclinics cater to about 20% of Singapore’s primary care needs (Khoo et al., 2014; Ministry of Health Singapore, 2018). The hospitals and polyclinics are managed by six public healthcare clusters, namely Alexandra Health System (AHS), Eastern Health Alliance (EHA), Jurong Health Services (JHS), National Healthcare Group (NHG), National University Health Systems (NUHS) and Singapore Health Services (SingHealth) (Yeo et al., 2012). The management of ICT infrastructure, support and services is however provided by an external entity, Integrated Health Information Systems (IHiS) Pte. Ltd., which is the healthcare IT leader in Singapore. IHiS partners with the clusters by managing the health IT systems and providing them with professional IT support. Within each cluster, IHiS has a dedicated team of IT professionals headed by a Chief Information Officer (CIO). The ICT infrastructure, support and services are extended to the polyclinics by the healthcare clusters they belong to.

Chapter 5: Architecting Our New Practice-Based Evidence Approach 85

However, the clusters have now been regrouped into three integrated clusters, West, Central and East clusters to meet Singaporean’s future healthcare needs (Ministry of Health Singapore, 2017; Poon Chian, 2018). The list of institutions and new clusters allocations are shown in detail in Appendix X.

Therefore, having defined the conceptualisation of our Practice-based evidence approach to decision making for healthcare professionals in the previous chapter, this following chapter is the second of four studies that discusses the overall development, implementation and evaluation of our PBE approach. In this chapter, the proposed ICT architecture for the adoption of our PBE approach is studied and discussed in further detail.

The aim in this phase of the research study was to uncover the existing enterprise ICT architecture of National University Hospital (NUH). The study began by discovering the various types of clinical information systems currently used by healthcare professionals. This included understanding the infrastructure of the clinical systems and if data sources from all the different clinical systems in use were integrated. This helped to unravel any existence of a clinical data warehouse. If one did not exist, then a design implementation of the data warehouse architecture would be required. Four objectives were identified for this phase of the study. The outcomes of meeting them would help to uncover the enterprise ICT architecture of NUH. Through this architecture of the hospital’s ICT system, the task of identifying relevant data sources and integrating them into a single data source would then be achieved. From there, the PBE data warehouse architecture design could be developed, and integrating the identified data sources to be used in a clinical decision support system as the source of evidence would be performed.

5.2 RESEARCH DESIGN

5.2.1 Research Design In this qualitative study to define the enterprise ICT architecture, IT professionals managing the ICT systems in NUH were interviewed. The aims of this interview were (1) to identify the various clinical systems implemented in the hospital, (2) to understand how the EHR system supports doctors with their care delivery and (3) to find out if there exists any mechanism that integrates multi-sourced data.

86 Chapter 5: Architecting Our New Practice-Based Evidence Approach

5.3 PARTICIPANTS

The participants to be interviewed in this study were identified as ICT professionals with the experience and knowledge of NUH’s network infrastructure and the types of clinical information systems that were being used. Since management of ICT support and services are provided by Integrated Health Information Systems (IHiS) Pte Ltd., the personnel of IHiS therefore became the intended participants.

In order to establish the working enterprise ICT architecture of NUH, information to be gathered was conducted through interviews with system engineers, software engineers and system analysts from IHiS. System engineers are IT professionals that are responsible for the network infrastructure as well as the database administration. Software engineers have the information systems knowledge at the development level, which includes having the understanding of the database structure. The system analysts manage the information systems within the hospital and have an understanding of how the clinical systems function and the kind of health information being stored.

Figure 10. Core competencies of a CIO based on the 9 identified capabilities required for a CIO (Chun & Mooney, 2009)

However, due to resource limitations and time constraints, none of the identified participants was available for interviews. Instead, the interview was only able to be conducted with the Chief Information Officer (CIO) assigned to NUH. This was the first hurdle we encountered and could have been a major stumbling block in our effort to elicit the information required. While the job responsibilities of a CIO differed from

Chapter 5: Architecting Our New Practice-Based Evidence Approach 87

that of the three identified participants, the many roles played by a CIO, together with the influences and knowledge the CIO brings to the healthcare organisation, was considered very important and relevant, and therefore still provided as significant contributions in this study.

In justifying the suitability of the CIO to provide us with the much-needed information, we referred to a study conducted by Feeny and Willcocks (1998) which identified the nine core IS capabilities required of a CIO to steer a company’s success through the implementation of IT and business strategies. This study is critical because it helped us to determine if indeed the CIO is the right personnel to provide us with the correct information. The nine capabilities identified by Feeny and Willcocks (1998) addressed three main challenges faced regularly by companies, and they are the challenge of (1) business and IT vision, (2) design of IT architecture and (3) delivery of IS services. Within the challenges of the design of IT architecture as illustrated in Figure 10, the CIO has to be a leader or has leadership qualities, needed to be responsible in ensuring the proper ICT architecture planning and to enable the implemented technologies to work and meet the organisation’s business strategies.

Figure 11. 2x2 matrix representing 4 types of CIO roles (Chun & Mooney, 2009)

88 Chapter 5: Architecting Our New Practice-Based Evidence Approach

However, ever-changing business needs and rapid advances in technology, made the role, capabilities and competencies of a CIO change constantly. Therefore, following the above study by Feeny and Willcocks (1998), Chun and Mooney (2009) developed a 2x2 matrix that depicted the current four roles played by CIOs, as illustrated in Figure 11. With close reference to this study, the role of a “Landscape Cultivator” typed CIO, among many, is responsible for the management of systems as well as being an IT architecture manager. This represented the attributes of participants who we identified for the interview. Thus, we believed that the CIO was indeed a suitable candidate with the relevant knowledge and experience to provide the necessary information required for this study. Additionally, to also support the information provided by the CIO, a Principal Systems Specialist who was involved with the development of NUH’s EMR system, CPSS2, was also recruited for the interview.

To ensure that the interview questions were purposefully developed, a pilot interview was conducted prior to the main interview. In this pilot phase, we interviewed IHiS’ Chief Architect. In total, one pilot interview and two main interviews were conducted with the personnel listed in Table 3.

Table of participants Designation Interview Type No. of participants Chief Architect, Director for Architecture & Innovation Pilot 1 Department Chief Information Officer (CIO) Main 1 Principal Systems Specialist Main 1

Table 3. Table of participants for the interview

5.4 INSTRUMENTS

The primary source of data collected during this phase of the research came from the interviews conducted. However, additional supporting documents, which included IHiS yearbooks and corporate presentation slides by the Ministry of Health Holdings (MOHH) and IHiS, were also analysed as secondary data to support and verify the findings made from the interviews.

Chapter 5: Architecting Our New Practice-Based Evidence Approach 89

A qualitative, semi-structured interview was developed, based on the themes derived from the research questions created, to collect data regarding the enterprise ICT architecture of NUH.

The method of analysis adopted was qualitative content analysis. The qualitative content analysis is defined as “a research method for the subjective interpretation of the content of text data through the systematic classification process of coding and identifying themes or patterns” (Hsieh & Shannon, 2005, p. 1278). Moreover, with qualitative content analysis, hidden or implicit meanings can be discovered, themes generated and conclusions are drawn out from the interview data (Zhang & Wildemuth, 2016). Therefore, this allowed us to identify the types of information systems used, the type of data stored and whether any integration techniques had been implemented.

To implement the qualitative content analysis, the process involved summarising the raw interview data into themes or categories based on a researcher’s interpretation and deduction. Deductive reasoning, which is a process by which themes and categories are directed by existing concepts through the research questions and prior literature, was applied in this research to develop the themes of the content analysis. Three different approaches to qualitative content analysis, conventional, directed and summative, can be employed. In this case, our research adopted the directed content analysis approach. Initial coding was created based on the literature reviews and the conceptualisation of our PBE approach. This represented the most common approach performed to validate or extend a conceptual framework or theory (Zhang & Wildemuth, 2016). During the data analysis phase, themes that emerged were carefully identified through a series of processes which was done iteratively.

5.5 PROCEDURE AND TIMELINE

5.5.1 Pre-Interview Planning The pre-interview planning included identifying suitable participants for the main interview session. Based on the objective of this study, which was to investigate (1) the existing enterprise ICT architecture of NUH, (2) the clinical information systems used and (3) the existence of a data warehouse, three types of IT professionals were identified, i.e. system engineers, software engineers and system analysts.

90 Chapter 5: Architecting Our New Practice-Based Evidence Approach

However, as highlighted in the previous section, this IT personnel were not able to participate during this phase of our study.

Developing the interview The process of developing the interview began by identifying the research question that needed to be answered and objectives to be met. Based on the research question, appropriate interview questions were then drafted. We managed to test the questions with a domain expert in a pilot interview test. This was done to validate the suitability and correctness of the interview questions in its ability to elicit information required to answer the research questions. Pilot testing represents an important aspect of the preparation of interview, because it can be used to discover interview design “flaws, limitations or other weakness” (Turner, 2010, p. 757).

For a pilot testing to be effective, it had to be conducted with similar typed participants (Turner, 2010). For that reason, the Chief Architect from IHiS was invited for a pilot interview session as part of the process of developing the interview. The Chief Architect represented the overall designer of the ICT architectures of all healthcare organisations in Singapore. As this was only a pilot interview to test the validity and applicability of the interview questions, no interview data was collected. The interview questions that would be asked of the interviewees, were evaluated based on their structure and validity. The structure of the interview questions was tested to ensure that the questions were clear and easily understood. Questions were also tested to ensure that they were not biased, confusing or complicated. Questions were assured to not be too broad, yet open-ended to allow interviewees to further elaborate and explain. The validity of the interview questions on the other hand, was tested to ensure that the questions were appropriate to be asked from IT professionals without infringing on any intellectual property of the interviewee’s direct employer.

The outcome of the pilot interview allowed the structuring of the interview process to be proper. An email invitation template was created to formally invite participants for an interview. It detailed the objectives of the interview, a brief background of the research and the expected duration of the interview. As the interview was conducted in Singapore, the email invitation also contained the possible interview dates available. Audio recording devices were also tested to ascertain that they were in working condition and that the recordings would be audible.

Chapter 5: Architecting Our New Practice-Based Evidence Approach 91

At the end of the pilot interview with the Chief Architect, the interview questions were given a final review and edited to improve them. The final interview questions were then categorised accordingly to three IT aspects of (1) system administration, (2) database administration and (3) networks. These three key IT aspects allowed for a more definite distinction of the topic to be interviewed. It also provided a structured approach to eliciting information, ensuring that enough information would be gathered during the interview process. The final semi-structured interview questions are illustrated in Appendix E.

5.5.2 Main Interview After being successfully granted QUT ethical approval, a field study to Singapore was initiated. Through the help of IHiS’ Chief Architect, an initial introductory email was sent to the CIOs of National University Hospital (NUH), Singapore General Hospital (SGH) and Khoo Teck Phuat Hospital (KTPH). The email provided a brief background regarding the research study and the objectives of the interview. This was followed by a formal email invitation to the three CIOs, requesting their help in identifying suitable interview participants, one from the following category of IT professionals, system engineers, software engineers and systems analyst. However, due to time and resource constraints, no suitable participants were available or able to take part. Only the CIO from NUH was available and volunteered to be interviewed. Following that, an email invitation was then sent out to the Principal Systems Specialist from IHiS, who was involved in the development of the Electronic Medical Record (EMR) system in NUH, requesting to also be a participant for an interview.

Next, the interested participants were once again contacted to confirm the agreed location and session for the interview. The interview with the CIO was conducted at NUH itself, while the other with the Principal Systems Specialist was held outside the work location due to time constraints which did not permit the participant to return to their work location for the interview.

Before the start of the interview session, the participants were briefed again regarding the background of the research and the range of questions that would be asked. The participants were informed that their participation in the interview session was entirely voluntary, and, as an indication that they have understood and agreed to be interviewed, they were required to sign off on the consent form. They were also

92 Chapter 5: Architecting Our New Practice-Based Evidence Approach

reminded that the interview would be audio recorded and the recorder placed closest to them to ensure that the recording would be most audible. Since the interview questions were not provided to the participants beforehand, ample time was afforded to the participants to further ponder over the questions and offer their respective responses. The format of the semi-structured interview also allowed us to proceed to other questions according to the flow of the interview, so as to keep the momentum of discussion moving. It had also enabled us to come up with new questions along the way, in order to gather richer information. All the interviews were conducted in English, which is the official business language in Singapore.

However, one of our main concerns with the interview as a form of data collection was the fact that we only had a total of two interviews conducted, one with the CIO and one with the Principal Systems Specialist. We were worried that the interviews would be biased and that the data that we analysed be flawed. However, there were only a total of seven CIOs altogether, six managing the different healthcare clusters each and one for the Agency for Aged Care in Singapore. Therefore, it was rather difficult to engage a substantial number of them in an interview session. Furthermore, we believed that the single CIO interview was sufficient as the information elicited was not about capturing the differing views or opinions of CIOs. Neither was the research about comparing answers from different individuals or groups that needed highlighting. Instead, this particular phase of the research study was solely focused on a single expertise, which was about eliciting information regarding the current enterprise ICT architecture. Moreover, the interview was substantiated with document analysis through the secondary sources of documents originated from Ministry of Health Holdings (MOHH) and IHiS.

5.6 ANALYSIS

There are many qualitative analysis methods that can be employed, such as thematic analysis, document analysis and content analysis. In fact, there are even cases where the use of thematic analysis and content analysis can be done interchangeably. Following a similar study conducted by Zakaria et al. (2010), the method of analysis that we engaged was a mixture of content and thematic analysis.

Chapter 5: Architecting Our New Practice-Based Evidence Approach 93

The analysis plan followed the eight-step process suggested by Zhang and Wildemuth (2016). The steps are summarised and illustrated in Table 4.

Step Description 1. Prepare the data Data has to generally be transformed into written text before analysis can be performed. The following questions arise when transcribing interviews: • Should all the questions of the interviewer or only the main questions from the interview guide be transcribed? • Should all the verbalisations be transcribed literally or only in a summary? • Should observations in an interview be transcribed or not? 2. Define the unit of The unit of analysis refers to the basic unit of text to be classified during analysis content analysis. Messages have to be unitised before they can be coded and differences in the unit definition can affect decisions as well as the comparability of outcomes with other studies. 3. Develop categories Categories and a coding scheme can be derived from three sources: the and a coding scheme data, previous related studies and theories. Coding schemes can be developed inductively and deductively. 4. Test coding scheme Validate the coding scheme early in the process to test the clarity and on a sample text consistency of the category definitions with a sample data. 5. Code all text Apply the coding rules to the entire corpus of text. Check the coding repeatedly during the coding process to prevent “drifting into an idiosyncratic sense of what the codes mean” (Schilling, 2006 as cited by Zhang and Wildemuth (2016)) 6. Assess coding Recheck coding consistency after coding the entire set to ensure that consistency coding of the entire corpus of text is consistent and in a reliable manner. 7. Draw conclusions Make inferences and present the reconstructions of meanings derived from from coded data the data by exploring the properties and dimensions of categories, identifying relationships between categories, uncovering patterns and testing categories with data (Bradley 1993 as cited by Zhang and Wildemuth (2016)). 8. Report methods and Monitor and report the analytical procedures and processes in order to findings make the study replicable.

Table 4. Qualitative analysis process (Zhang & Wildemuth, 2016)

5.6.1 Organising and preparing data The primary data was collected from two interview sessions, one conducted with the CIO and the other with the Principal System Specialist. The audio content of the interview sessions was recorded using a portable digital audio recorder. A copy of the audio was extracted from the recorder and saved on our working laptop. Another copy was kept on an external hard drive as well as online, in Australia’s Academic and Research Network (AARNet) CloudStor, provided through QUT. The additional or secondary data sources were in the form of three IHiS yearbooks and a presentation

94 Chapter 5: Architecting Our New Practice-Based Evidence Approach

slide regarding Singapore’s NEHR, jointly presented by MOHH and IHiS. As the content of the additional data sources was also in English, no translation work was needed.

The importance of the qualitative data analysis process is in the preparation and organisation of data collected for analysis: condensing it to themes through coding and finally presenting the findings in discussion (Creswell, 2007). Furthermore, with rapid developments in ICT, the use of computer aided qualitative data analysis software or CAQDAS in short, is swiftly improving the analysis process (Welsh, 2002). CAQDAS such as NVivo is frequently utilised in qualitative data analysis as it is easy to use, able to assist in the storage, search and retrieval of qualitative data, and supports an effective and efficient coding process thus allowing for a more thorough analysis of the qualitative data itself (Welsh, 2002; Wickham & Woods, 2005). Therefore, in this interview data that we had collected, the audio recordings were stored and transcribed using the NVivo 11 software.

The interview transcripts were the primary sources of data for the content analysis. Therefore, transcribing of audio interview recordings was required before any analysis could be performed. Since the interview was conducted in English, no prior translation work was needed. We performed the transcription ourselves with the help of the NVivo 11 software. All the audio recordings were imported into NVivo and saved in a folder labelled ‘Interviews’. NVivo has an inbuilt feature, which allowed us to easily play the audio recording and seamlessly typed the text transcription. It therefore helped in syncing the transcribed text with the audio timeline.

The transcription method adopted was intelligent verbatim as we conducted the interview as well as the transcribing ourselves in this study. Hence, having had firsthand experience with the content of the interview, words or sentences such as “ahh, hmmm, mmmm, ummm, I know...” and those that did not add value to or changed the context of the content of the interview were removed from the transcription. However, this process was done with due care, so as not to alter the overall meaning and sense of the content. Furthermore, as the research objective was qualitative, quantitative research elements such as calculating the repeated occurrence of words was however not critical in this study. Thus, the end-product of the transcription was one of an accurate and concise account of the interview, while keeping the content true to the perceptions of the interviewees.

Chapter 5: Architecting Our New Practice-Based Evidence Approach 95

5.6.2 Defining unit of analysis According to Zhang and Wildemuth (2016), in qualitative content analysis, individual themes are usually used as the units of analysis. Such a theme can be conveyed in a single word, phrase or an entire document. However, in directed content analysis, themes, codes and categories can be developed initially from a theory or relevant research findings, done deductively (Zhang & Wildemuth, 2016). In that sense, we established the initial coding for this particular study from the conceptualisation of our Practice-based evidence approach and relevant literature reviews. In particular, the coding was in relation to answering the objectives of the research question that was asked.

5.6.3 Developing categories and a coding scheme Based on our Practice-based evidence approach to decision making concept, the coding schemes and categories were developed deductively instead of inductively. This allowed the uncovering of the enterprise ICT architecture of NUH through the process of qualitative content analysis.

Therefore, the list of coding categories that was developed and adapted to the data collected was as follows:

• Information Systems • Data Type • Data Integration • Process To ensure that coding was consistent throughout, we created a coding manual, illustrated in Table 5, as suggested by Zhang and Wildemuth (2016). However, we were unable to test the coding scheme and categories on any sample data as the data source was limited to begin with. In total, there were only two interviews that were conducted and four related documents available for analysis. Thus, to ensure that the coding schemes and categories reached a level of consistency and clarity, we performed the coding process iteratively.

Based on the list of coding categories, four different colour codes were used: orange, green, yellow and red. Each of the colours used was for identification purposes only, as illustrated accordingly to the category listed above. Further information was

96 Chapter 5: Architecting Our New Practice-Based Evidence Approach

furnished to explain what each category meant, with suitable examples provided, as illustrated in Table 5.

Category Definition / Rule Example Information Systems Referring to the various types of clinical Medication system, information systems being used to Radiology system, manage patient care in the hospital Laboratory system Data Types The type of information / data captured, height, blood pressure, i.e. clinical, financial, demographics, by lab results, medication the information systems Data Integration Indicating any form of data integration Data warehouse, data interoperability Processes Clinical or system process Messaging, update

Table 5. Coding manual

5.6.4 Code Text Upon completion of the transcription, the coding rule was applied to the corpus of the document. The content was thoroughly read and analysed. Concepts that were related to the coding schemes and categories, were highlighted according to the colour code developed. Additionally, the concepts were also allowed to overlap in multiple coding schemes during the initial iteration, before consistency and clarity were reached through further iterations. The iterative processes incorporated both elements of content analysis and thematic analysis in order to draw out the meanings, gain understanding and develop the knowledge (Bowen, 2009). The concepts derived from the coding schemes and categories thus represented the answers to the research question asked.

Figure 12. Sample coding is done to interview corpus

Chapter 5: Architecting Our New Practice-Based Evidence Approach 97

After the first iteration, a thematic network, as per the works of Attride-Stirling (2001), was used to visualise the result of the qualitative content analysis. The first iteration revealed an extensive collection of items within each code or theme. For example, under the theme of Information Systems, we uncovered systems such as EMR, billing, patient registration and laboratory systems. Under Data Types, information or data that was captured or stored by the information systems used in the healthcare organisation included information such as laboratory results, prescription, and diagnosis, to name a few. This was visually presented using the thematic networks, as pictured in Figure 13.

Figure 13. Identifying concepts based on coding schemes using thematic networks

Subsequent iteration was performed to ensure that items were not missed or were incorrectly coded. The iteration process also allowed us to improve the themes further such as by creating sub-themes that enabled the reduction in the thematic networks, as

98 Chapter 5: Architecting Our New Practice-Based Evidence Approach

illustrated in Figure 14. For example, Data Types were further categorised as financial, clinical and patient demographics.

The finalised concept thus represented a list of major clinical information systems available in NUH, the type of data captured or stored by these information systems, the existence of clinical data warehouses and important processes pertaining to information sharing and integration. Finally, the outcome from the derivation of the themes provided us with an overview of the enterprise ICT architecture in NUH. This enabled us to eventually propose and develop the architecture which will be suitable for the implementation of our Practice-based evidence approach to decision making.

Figure 14. Finalised concepts after iterations which included the addition of sub-themes or parent- themes

5.6.5 Analysing Findings Next, we continued with the content analysis of the primary interview data and the additional data sources, which comprised of one presentation slide and three IHiS yearbooks. In this content analysis, we aimed at answering the following research question.

“What changes to the current state of a healthcare organisation’s ICT architecture are required to adopt a PBE approach in assisting healthcare professionals with decision making?”

To answer the above question, we identified five objectives that could collectively provide the required answers. The findings from the analysis were as follows:

Chapter 5: Architecting Our New Practice-Based Evidence Approach 99

Objective 2.1: Identify the current information systems used in healthcare organisations. Findings from the interview session managed to uncover the list of clinical information systems currently in use at NUH. According to the CIO, NUH had developed their own in-house, bespoke, integrated electronic medical records (EMR) system called CPSS2 (Computerised Physician Support System 2). CPSS2 is a single computer system interface that extracts relevant clinical information from multiple system sources across NUH. It allows doctors to view patient medical information and act upon this information as it is displayed. It provides various modules that link to other systems that provide clinical documentation, problems lists, CCOE (Computerised Clinician Order Entry) which provides order management, for example to laboratory and radiology imaging systems, Patient Registration and discharge summaries (eHIDS). We also performed content analysis on the additional data sources to supplement and complement the information provided by the CIO. For example, in the presentation by Tan and Ong (2009), the authors shared the clinical application map, which highlighted the different systems used by healthcare organisations across Singapore. In particular, we focused on the information systems implemented in NUH, which is represented as NUHS. In the clinical application map, it listed systems such as CPSS, CCOE, eIMR (electronic Inpatient Medication Records) and eHIDS (electronic Hospital Inpatient Discharge). Although the slides were presented almost nine years ago, the information provided evidence of similar systems currently being used in NUH as per the findings from the interview data. In addition, according to the 2011 IHiS yearbook (IHiS, 2011), public hospitals in Singapore, including NUH, have been equipped with Electronic Health Record (EHR) systems, medication management systems, laboratory and radiology imaging systems and clinical decision support systems (IHiS, 2011, 2016). Besides having a medication management system for both inpatients and outpatients, public hospitals have other intelligent systems such as Inpatient Pharmacy Automation System (IPAS), Outpatient Pharmacy Systems (OPAS), electronic Medication Administration Recording Systems (eMARS) and Knowledge Based Medication Administration (KBMA), that aim at improving medication administration and reducing preventable medical errors. Patient management details are electronically documented through systems like Electronic Clinical Documentation (CDoc), Health Diary, electronic nurse charting and Sunrise Clinical Manager.

100 Chapter 5: Architecting Our New Practice-Based Evidence Approach

Objective 2.2: Identify the types of data stored in these systems. Based on the interview data analysed, in general, the information that was displayed in CPSS2 included data regarding patient medication, laboratory and radiology tests results, medical history, patient information and more. As CPSS2 was the interface that displayed information from multiple information systems, the following list of clinical information systems that were used in NUH helped to reveal the types of data or information stored in them. CCOE allowed healthcare professionals to order laboratory and radiology test, therefore would naturally contain patients’ laboratory and radiology results. It also allowed the medication to be ordered and prescribed to patients. Thus, it contained the past and present medication prescribed information. Patient Registration systems stored patients’ demographics information like their contact information and other non-clinically related information. eHIDS generated discharge summaries which contained patients’ inpatient information including their past and present medical conditions, medication information, treatment information, investigation results, diagnosis, follow-up care arrangements and physical and cognitive examinations (Robelia et al., 2017; Shastri et al., 2014) to name a few. As indicated in IHiS’ three yearbooks, IPAS, OPAS, KBMA and eMARS also contained patients’ prescribed medication information.

Figure 15. NUH’s Clinical Application Map represented by NUHS (Tan & Ong, 2009)

Chapter 5: Architecting Our New Practice-Based Evidence Approach 101

Objective 2.3: Identify how current organisation of information is implemented. For CPSS2 to be able to provide an integrated view of patients’ clinical information from a myriad of information systems used, the data sources from the various systems were linked up through a system called CLOVERLEAF. CLOVERLEAF is an integration system, which “aggregates disparate system and data regardless of the data source, message format and transmission protocol” (Infor, 2016), providing interoperability capabilities to CPSS2. CLOVERLEAF was the medium between CPSS2 and the various clinical information systems used (modules). In turn, CPSS2 became the interface that linked the healthcare professionals with these diverse information systems. Through the different modules in CPSS2, CPSS2 communicated with CLOVERLEAF by sending and receiving HL7 messages. Health Level 7 or HL7 is a standard-setting organisation that develops communication protocol. An HL7 message is a standard used to represent clinical documents, such as discharge summaries. With HL7 messages, systems communication will be based on standard rules to exchange, manage, communicate and integrate data (Dolin et al., 2001).

Objective 2.4: Identify how a data warehouse can improve data integration for reliable use. The interview session revealed the availability of two enterprise data warehouses; a Centralised Clinical Data Repository (CCDR) and an Operational Data Store (ODS). This thus represented the existence and the capability of NUH’s current enterprise ICT infrastructure to integrate and merge data sources from the multiple information systems available. The ODS mainly contained operational data such as financial, patient registration and some clinical data. The CCDR, on the other hand, held clinical data such as laboratory and radiology test results, medication information, operating theatre reports as well as information regarding patient demographics. The content analysis of the secondary data sources also highlighted the existence of the Central Clinical Data Repository. The CCDR was said to be capable of holding patient medical records. It was also integrated with clinical decision support systems and care management tools. However, no information was provided with regard to the context in which the clinical decision support systems were used. The CCDR was described as the single source of information for patient medical records and linked to other health clusters through the National Exchange Health Record (NEHR) platform. The

102 Chapter 5: Architecting Our New Practice-Based Evidence Approach

enterprise data warehouse on the other hand, was utilised by NUH for the purpose of financial analytics such as the tracking of medication costs and, payment profiles and affordability of patients. The presence of the two data warehouses indicated to us that NUH had developed the proper architecture to integrate their many data sources. They even managed to segregate the data warehouses based on its functionalities. This allowed the healthcare institution to perform specific research and investigative purposes based on each data warehouse. This was evident in how the enterprise data warehouse was developed explicitly for financial analytics and research purposes.

Changes required to adopt PBE approach The findings above helped us to uncover and understand the current ICT architecture of NUH. This was useful in helping us to identify the changes that were required to adopt our PBE approach effectively. However, both the interview data and supporting documents did not explain or describe the current blueprint or extent of NUH’s enterprise ICT architecture. Having this information would have enabled us to easily design a suitable data warehouse architecture to implement our PBE approach. Instead, we had to envision the ICT architecture. Therefore, based only on information we have gathered thus far, our next task was to determine the ICT architecture.

Envisioning the NUH ICT architecture

Drawing inspiration and examples from the works of Chen et al. (2014), Choi et al. (2013), Hamoud and Obaid (2014) and Lu and Keech (2015), we started work on defining how NUH’s enterprise ICT architecture would look like.

In the case of the data warehouse architecture designed by Hamoud and Obaid (2014), the data warehouse was developed to support the need to perform online analytical processing or OLAP. OLAP requires an entirely different database structure as compared to most common relational databases. A relational database is a database structure used often by information systems such as the Electronic Health Record Systems to perform online transaction processing (OLTP). OLTP operations include inserting, updating and deleting rows of data in a database. In this particular data warehouse design, the multiple sources of data are then integrated through the ETL processes where the data is extracted, transformed, cleansed before inserted into the data warehouse. A data mart is then used to retrieve the required data, ready for OLAP operations.

Chapter 5: Architecting Our New Practice-Based Evidence Approach 103

Figure 16. ICT and data warehouse architecture [Top left:(Hamoud & Obaid, 2014), top-right: (Choi et al., 2013), bottom-left: (Lu & Keech, 2015), bottom-right: (Chen et al., 2014)]

104 Chapter 5: Architecting Our New Practice-Based Evidence Approach

Choi et al. (2013)’s design for a data warehouse was for the development a research database for prostate cancer. The authors found difficulty in incorporating direct extraction program on existing hospital’s information systems and therefore suggested the use of a clinical data warehouse instead. In fact, this approach required a much lesser effort yet provided the access and extraction of research information. In this data warehouse architecture design, patient data from EMR systems was fed into a central data warehouse housing the prostate cancer registry system. Since the data warehouse only contained relevant data of patients with prostate cancer, it did not require any data mart to be designed to extract specific information.

Lu and Keech (2015) introduced a Big Data Analytics conceptual architecture that integrated data mining, warehousing and big data technologies together to take advantage of the vast amount of digital health data generated in the UK. In a study conducted by Chen et al. (2014), a data warehouse methodology is adapted to increase and explore the knowledge of diabetes by integrating other data sources from their telehealth program as well as doctor regular visits. In the architecture by Lu and Keech (2015), the data integration process was performed through an ETL process and loaded into a Hadoop Distributed File Systems (HDFS). Only when data was required, a data schema was built to transform the data needed from HDFS into a data warehouse. In the architecture by Chen et al. (2014), the data warehouse was used to centralise the data from their online operational systems. It consisted of three stages of data transcription, data manipulation and data visualisation. Data transcription was performed in the similar fashion as the ETL process where data from the online systems was duplicated into a staging database. Data manipulation stage was where data was integrated and shaped according to the analysis requirements. A data mart was used in this process where only the required information was extracted and used for analysis in the data visualisation stage. Similar to the works of (Hamoud & Obaid, 2014), a data mart design or a multi-dimensional model, as indicated in the study conducted by Lu and Keech (2015), was created to extract relevant data from the data warehouse for the data mining or data analytics activities.

The examples analysed above showed how similar models of a data warehouse architecture could be developed. In the data warehouse design by Hamoud and Obaid (2014), Lu and Keech (2015) and Chen et al. (2014), data marts were used to extract the necessary data required for OLAP, visualisation or intelligence tools. Although the

Chapter 5: Architecting Our New Practice-Based Evidence Approach 105

examples still did not precisely illustrate the exact ICT architecture of individual healthcare organisations, they accurately demonstrated the general model of a data warehouse architecture. Taking that into consideration and based on the findings from the interview and supporting documents, we envisioned the ICT architecture of NUH incorporating a data warehouse as illustrated in Figure 17.

5.7 DESIGNING A DATA WAREHOUSE ARCHITECTURE FOR OUR PBE APPROACH

As we recap the logical model of the key components in our Practice-based evidence approach to decision making (Figure 7), we identified the need for a data warehouse architecture. Data warehousing is not only an essential methodology where data from multiple sources can be integrated together. It is also the pillar for our conceptualised Practice-based evidence approach adoption, where evidence of clinical practices, which are captured electronically from various information systems, are consolidated into a single comprehensive repository.

With the growth in biomedical knowledge, advancements in ICT and decreasing computing costs, the healthcare industry is thriving with abundant sources of database (Sujansky, 2001). Such databases include those that contain patients’ clinical information, general medical and biomedical information as well as valuable administrative data. In the case of NUH, examples of such databases are in the form of laboratory systems, patient registration systems, EMR systems and CPSS2, as well as other ancillary systems.

A data flow diagram, such as the one illustrated in Figure 18, describes how data generated from various information systems are captured and stored in corresponding database sources or a datastore. In enabling our Practice-based evidence approach, the data warehouse architecture is relevant in extracting this valuable information to be integrated, so as to result in a rich and comprehensive collection of data that is both meaningful and purposeful to assist in clinical decision making.

Therefore, from the envisioned NUH’s enterprise ICT architecture above, the data warehouse architecture for our PBE approach architecture can be designed accordingly. However, in this stage of the study, we have uncovered the existence of an enterprise data warehouse consisting of two data warehouses, the ODS and CCDR,

106 Chapter 5: Architecting Our New Practice-Based Evidence Approach

Figure 17. Envisioned NUH enterprise ICT infrastructure (Osop & Sahama, 2016a)

Chapter 5: Architecting Our New Practice-Based Evidence Approach 107

within the NUH ICT architecture. This means that the planned development of a data warehouse to implement our PBE approach is no longer required. We will be able to capitalise on the existing data warehouse architecture to gain access to an integrated source of healthcare data. Nonetheless, if one was not available, the process of building a data warehouse consists of three components. These three major components, while they may be termed differently by different authors, consist of identifying relevant data sources to be integrated, the integration process and data utilisation process. Ado et al. (2014) term the three components as processes: the data acquisition process, storage process and presentation process. Chen et al. (2014) on the other hand, refer to them as stages: transcription, manipulation and visualisation. Whatever they are called, these processes and stages are similar in nature.

The initial step in building a data warehouse is to identify the relevant data sources that will be integrated into the data warehouse. Following this process, data from the identified datastores is extracted and duplicated into a staging database, where erroneous information that may exist can be cleaned before being loaded into a data warehouse. This process can be assisted with the help of medical and clinical experts with the necessary domain knowledge, as they suggest the correctness of the extracted clinical information.

Following the duplication and cleansing of data sources, the next process requires the Extraction, Transformation and Load (ETL). The data sources are then extracted from the staging database, merged and transformed to match the data schema of the target data warehouse. Once matching is completed, the transformed data is loaded into the data warehouse.

The final step in the building of the data warehouse requires the design of data marts to extract the necessary information. The specific design of the data marts will be able to answer the relevant analysis questions asked. These questions may be financial, business or administrative in nature. In our PBE approach, such questions can therefore be clinical in nature. Depending on the analysis requirements, different data mart designs can be implemented, such as the Star schema or Snowflake schema design (Mishra et al., 2008).

Therefore, based on the current ICT architecture of NUH, the development of a new data warehouse is no longer required since an enterprise data warehouse already

108 Chapter 5: Architecting Our New Practice-Based Evidence Approach

Figure 18. Flow of data enabling a PBE approach through a data warehouse architecture

Chapter 5: Architecting Our New Practice-Based Evidence Approach 109

exists. Instead, the additions required to effectively implement our PBE approach are a data mart and a clinical decision support system. The data mart design will be created to extract the necessary data for use in our prototype clinical decision support system.

The data mart represents a subset of a larger data warehouse (Chhabra & Pahwa, 2014). From the design of the data mart, relevant information can be extracted to become data sources for the prototype CDSS. Different data marts are designed to serve different purposes and the kinds of analysis needed. In this research, we require data that will be used to generate the patient-centric statistics and prediction modelling. With the data warehouse containing various types of information such as administrative and clinical data, a specific data mart design is required to extract only the necessary information (to generate the patient-centric statistics and prediction modelling) for the prototype CDSS. As the data schema of the data warehouse is not available, both new additions (data mart design and prototype CDSS) will be discussed in detail in the upcoming research study chapters.

Therefore, the architecture that is ideal for our PBE approach will consist of an existing EDW that contains the integrated data sources, a data mart to extract the necessary data to draw inferences from and a prototype CDSS to assist with decision making. This architecture is aptly illustrated in Figure 19.

5.8 LIMITATIONS

We acknowledge that a limitation in this particular phase of the study stems from having only two participants available to be interviewed. With the unavailability of IT personnel that we have identified as participants, we rely on the contributions of the CIO and Principal System Specialist. The risk is that interviewing a small number of participants may result in generating limited data to analyse effectively. Therefore, the lack of content to analyse from the two participants may influence the accuracy of the findings, in this aspect, the enterprise ICT architecture of NUH. To counter such risks, we have substantiated the interviews with secondary data sources. As a consequence, the findings in this particular study may only hold true to NUH and may not be assumed for other healthcare organisations in Singapore.

110 Chapter 5: Architecting Our New Practice-Based Evidence Approach

Figure 19. Data warehouse architecture for our proposed PBE approach

Chapter 5:Architecting Our New Practice-Based Evidence Approach 111

5.9 CONCLUSION

In this phase of the study, we managed to answer the research question set out. In addition, it allowed us to uncover the overall ICT architecture of NUH which was crucial in discovering the different kinds of clinical information systems used, the need for a data integration solution and the development of the architecture for the adoption of our PBE approach.

Although the interviews and the content analysis of supporting documents did not reveal the actual layout of the overall ICT architecture, there was sufficient information available to allow us to envision the eventual ICT architecture of the hospital. Through the interviews, we discovered the existence of an enterprise data warehouse within the ICT architecture which significantly helped with the research study. A new data warehouse architecture was no longer needed to be developed. This was the first step towards the implementation of our Practice-based evidence approach. With an existing data warehouse, this allowed the focus to be more on the sources of information for the clinical decision support system. Therefore, the new additions necessary for our PBE approach were identified as the data mart design and the clinical decision support system. With the design of the PBE architecture in Figure 19, the use of the data mart would allow for the retrieval of the necessary and relevant data required from the EDW. However, since the data schema design of the data warehouse or the data schema of the various clinical information systems’ relational databases was not available during the interviews, the new data mart design will be developed during the prototype CDSS development phase.

As the study was limited to NUH, it might be difficult to broadly generalise the same ICT architecture exists in other hospital thus enabling the applicability of our PBE approach to decision making to other healthcare organisations across Singapore. However, we found out that the hospitals and polyclinics in Singapore are grouped into six different healthcare clusters, where each cluster adopts the use of the same clinical systems. Hence, this architecture may be easily replicated and be applicable to the healthcare organisations within the same cluster. Ideally, further studies can be conducted to understand and identify the ICT architecture of healthcare organisations under other clusters, so that the PBE approach can include these other public hospitals in Singapore.

112 Chapter 5: Architecting Our New Practice-Based Evidence Approach

Following the discovery of an existing data warehouse and the design of the PBE architecture, our next phase of the study focuses on evaluating the perception of doctors in Singapore with regard to the suitability of utilising electronic health records as evidence for our PBE approach.

Chapter 5: Architecting Our New Practice-Based Evidence Approach 113

[This page is left intentionally blank.]

114 Chapter 5: Architecting Our New Practice-Based Evidence Approach

Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making

“If we knew what it was we were doing, it would not be called research, would it?” - ALBERT EINSTEIN

In this third of four studies, the chapter begins with an overview of the research. This is followed by detailed description of the research model, participants and research instrument. Next, is the analysis of the findings of the research study.

6.1 OVERVIEW

In the previous chapter, our research study uncovered the ICT architecture of National University Hospital and this should enable us to decide on the design of the data warehouse architecture for the adoption of our PBE approach. Instead, findings from the interviews with IT professionals detailed the existence of an existing enterprise data warehouse. Through the ETL (Extraction, Transformation and Load) process, data from various information systems such as the EMR system (CPSS2), laboratory systems, patient registration systems as well as other ancillary systems is integrated into the enterprise data warehouse which consists of the Operational Data Store (ODS) and Centralised Clinical Data Repository (CCDR). Thus, the additions required for the adoption of our Practice-based evidence are a data mart design and a clinical decision support system.

Following the logical model of the key components of our PBE approach to decision making (Figure 7), we have identified electronic health records as the suitable source of evidence. Numerous studies have provided examples of the potential of secondary uses of EHRs for decision making. However, such studies have not been conducted on healthcare organisations in Singapore. Therefore, in this particular research study, we intend to gain the perspective of doctors from Singapore regarding the perceived benefits of utilising EHRs. This does not only allow us to understand the extent of EHR system use across healthcare organisations in Singapore but also enable

Chapter 6: Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making 115

us to evaluate the value of EHRs as potential evidence. Hence, this second study aims to examine doctors’ perception regarding how EHR systems are being used, the benefits of using EHR systems, the usefulness of EHR data, the quality of EHR data, the nature of EHR data availability and the usefulness of EHR data for decision making. Through this study, we establish the applicability and suitability of using EHRs as evidence for our prototype CDSS in our PBE approach for decision making.

The survey had the participation of doctors in Singapore’s public hospitals and polyclinics. This survey study focuses on eliciting data regarding doctors’ EHR system usage trends, the perceived clinical benefits of EHR system use, the perceived quality of EHR data, the perceived clinical benefits associated with quality of EHR data, the perceived usefulness of EHR data for decision making and the nature of data availability for decision making. It is important that we understand these concepts in order to identify the potential of EHRs for use as evidence in our PBE approach to decision making. The survey outcome will therefore provide us information about whether doctors in Singapore value the use of EHR systems in managing their patient care and how much they value EHR data as being useful in helping to direct decision making. This reflects the potential of using EHRs as evidence in our PBE approach to decision making.

6.2 RESEARCH MODEL

6.2.1 Research Design In this phase of the collective case study, a mixed quantitative and qualitative evaluation of the perceived clinical benefits of using EHR systems and the perceived usefulness of EHR data were performed after the survey data collection was conducted.

6.2.2 Pre-survey content validation Content validation represents a critical aspect in the development of a data collection instrument such as a survey questionnaire (Grant & Davis, 1997). Content validation helps to verify questionnaire items as adequately measuring the assessing content and the sampling of content items throughout the questionnaire (Rubio et al., 2003). Content validation ensures the correctness, appropriateness, sensibility and relevance (Holden, 2010) of the survey questionnaire structure and items before the survey can be presented to participants. It includes the process of evaluating and

116 Chapter 6: Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making

examining the survey questions so that participants consistently understand the questions as the researcher has intended them to be understood (Burns et al., 2008). In this study, the content validation method we adopted was face validity.

In performing this validation process, three content experts, who are doctors from Singapore’s public hospitals and private clinics, were engaged. Following the works of Grant and Davis (1997), it has been suggested that the minimum number of content experts can range from two to three, although, the eventual number is dependent on the expertise and the variance in knowledge (Grant & Davis, 1997; Rubio et al., 2003) of the content experts. In this study, the three doctors involved are experienced and active EHR system users, and are also representative of the survey participant population. Therefore, they represent the required expertise needed to be content expert validators (Rubio et al., 2003). Furthermore, this validity process allows the content experts to provide constructive feedback that can help us to improve the design of the survey in terms of the quality of questionnaire items and the way items are measured (Rubio et al., 2003).

Overall, the three doctors that have been engaged, validated the content of the questionnaire by reviewing the structure of the questions, examining individual questionnaire items and interpreting and verifying the items as intended by the research team.

During this validation process, the doctors highlighted the importance of defining the terms used in the questionnaire so as not to mislead the participants. Therefore, changes made were to include a short explanation at every new section of the survey questionnaire where new terms, ideas or processes were introduced.

The three doctors also reviewed the flow, salience, acceptability and administrative ease, identifying unusual, redundant, irrelevant or poorly worded questions as per the works of Burns et al. (2008). This exercise also allowed us to record the time taken to complete the survey questionnaire and ensured that it would not take too long for future participants.

6.2.3 Pilot Survey Testing A pilot study is conducted as a way to assess the feasibility and design of an eventually much larger main study (Arain et al., 2010; Thabane et al., 2010). In the pilot study that we conducted, the survey questionnaire items used during this session

Chapter 6: Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making 117

represented the final copy that was used during the main survey. The intention of the pilot survey was to generate a sample survey data which could be analysed and provide an overview of what could be expected during the main survey.

Four respondents participated in the pilot study to get a response based on the newly developed questionnaire items. From the response data collected, the possibilities of participants misinterpreting the questions, as well as verifying that items were reliable in eliciting the intended responses were examined. This process allowed the removal of questionnaire items that were redundant and retention of questionnaire items that were optimal. Based on the response, we realised that the questionnaires pertaining to the adoption of a Practice-based evidence approach were not applicable to be asked from the respondents as they had no prior access to the prototype clinical decision support system. These questions were therefore ultimately removed from the questionnaire.

6.3 RESEARCH CONSTRUCTS

6.3.1 Research questions This phase of the study aims at answering the following research question listed below.

“What are the processes required that ensure a PBE approach can assist healthcare professionals in clinical decision making?”

By answering the above research question, we will be able to identify the factors that may limit the ability to make well-informed decisions and how this gap can be bridged by using EHRs. In addition, we will also be able to evaluate how EHR data can be used to assist in decision making.

6.3.2 Survey items and constructs The design of the survey items and constructs for this study is adopted from the RBV model illustrated in Figure 8 that studies the effects of EHRs as an organisation’s tangible resources. This assisted in setting the objectives of the research study, to elicit the views of doctors from Singapore’s public healthcare organisations regarding the benefits gained from adopting the use of EHR systems and the usefulness of EHR data for decision making. The survey identifies the need to understand the perception of doctors regarding the benefits of using EHR systems to manage the care of their

118 Chapter 6: Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making

patients and the perception of how useful EHR data is in assisting with decision making. It is therefore essential to understand these perceptions, as EHRs have been identified as the source of evidence in the conceptualisation of our Practice-based evidence approach to decision making. Besides understanding the usage trends of EHR systems among doctors, the survey can also elicit data regarding the range of information captured in EHRs. Therefore, based on the RBV model, the following survey constructs have been identified as, (a) EHR usage trends (UT), (b) EHR system functionalities (SF) leading to clinical benefits, (c) Information in EHR that is useful, (d) Quality of EHR data associated with clinical benefits, (e) Quality of EHR which makes it useful, (f) Nature of data availability, (g) Perceived clinical benefits of EHR system leading to improved decision making, and (h) Perceived usefulness of EHR data for quality decision making.

In addition, the development of the survey items and constructs also adopted items or measures from existing studies conducted by King et al. (2014), Secginli et al. (2014) and Simon et al. (2009). Questionnaire items from the above studies were analysed, reviewed and adapted accordingly to the survey constructs explicitly developed to measure the perceived clinical benefits of using EHR systems and the perceived usefulness of EHR data for decision making in this study. Each of the constructs developed aimed at measuring different aspects of the survey objective and is discussed further in the next section.

Survey questionnaire items from the studies Author 1. Helped you access a patient’s chart remotely King, et al. (2014) 2. Alerted you to a potential medication error 3. Alerted you to critical lab values 4. Helped you order more on-formulary drugs 5. Reminded you to provide preventive care 6. Reminded you to provide care that meets clinical guidelines for patients with chronic conditions 7. Helped you order fewer tests due to better availability of lab results 8. Helped you identify needed lab tests 9. Facilitated direct communication with a patient Electronic Health Record System: Secginli et al. 1. Provides quick and reliable access to scientific research (2014) 2. Enables easy access to information from past medical records 3. Provides access to patient data and analysis 4. Provides access to practice standards 5. Enables following test results 6. Contributes to health professional’s ability to make patient care decisions

Chapter 6: Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making 119

Clinical Functions within EHR systems Simon et al. (2009) 1. Laboratory test results 2. Laboratory order entry 3. Radiology test results 4. Radiology order entry 5. Electronic visit notes 6. Reminders for care activities 7. Electronic medication lists 8. Electronic problem lists

Table 6. Identified survey questionnaire items from the three studies.

The survey items for each construct were explicitly designed to measure the factors that affect decision making by healthcare professionals. The questionnaire items were then verified by the research team and a healthcare professional, who is the Chief Medical Information Officer at one of the public hospitals in Singapore.

Constructs and questionnaire items. A total of six constructs were developed to investigate the factors that affect decision making and how EHR data could be useful in assisting with decision making. These constructs would lead in evaluating the feasibility and suitability of EHR as data sources or evidence for our Practice-based evidence approach. Each of the constructs’ measurements is explained in detail below. The survey questions are constructed to be personalised (using I, me and my) while others have been impersonalised. Personalised questions aim at understanding doctors’ personal perception as an EHR system user rather than what the systems can be used for. Impersonalised questions aim to gather doctors’ objective views regarding the useful of data captured in EHRs.

EHR Usage Trends (UT) EHR usage trends describe how EHR systems are used by doctors in the process of managing patient care. This provided us with the information regarding the range of functionalities available in the EHR system used by doctors, as well as the types of data captured in EHR systems.

Construct Questionnaire item Usage Trends (UT) I use an EHR system to access to _____. UT1: clinical guidelines UT2: patient chart remotely

I use an EHR to _____. UT3: communicate with my patients

I use an EHR system to document and/or view patient’s _____. UT4: clinical notes UT5: demographic information

120 Chapter 6: Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making

UT6: family medical history UT7: health and healthcare-related information UT8: medical diagnosis UT9: medical history UT10: smoking status UT11: vital signs

I use an EHR system to order _____. UT12: laboratory tests UT13: medication prescription UT14: radiology tests UT15: treatments

I use an EHR system to provide patients with _____. UT16: clinical summaries per visit UT17: copies of their health information

I use an EHR system to review _____. UT18: laboratory tests results UT19: medical history UT20: medication history UT21: problem lists UT22: radiology tests results

Table 7. EHR Usage Trends list of questionnaire items

EHR system functionalities (SF) leading to clinical benefits The use of EHR systems facilitates the electronic documentation of patient health and healthcare-related information, such as doctor’s notes, and enables the viewing of laboratory and radiology results and the electronic prescription of medication (Jha et al., 2008). This construct provided an understanding of what doctors perceive as clinical benefits. This is later used to investigate the association of perceived clinical benefits of using EHR systems with clinical decision making.

Construct Questionnaire item System Functionality The EHR system provides me with the clinical benefit of (SF) SF1: alerting me to critical laboratory test results and values SF2: assisting in creating treatment plans SF3: assisting in diagnosing medical condition(s) SF4: assisting in ordering more formulary drugs SF5: facilitating access to suitable clinical guidelines to use SF6: facilitating access to patient health information such as diagnosis, medication, treatment, laboratory test result, etc SF7: identifying patients who require laboratory and/or radiology tests to be performed SF8: identifying patients who need preventive care SF9: reducing redundant laboratory and/or radiology testing SF10: reducing similar laboratory and/or radiology testing SF11: preventing potential medical errors

Table 8. System functionalities with clinical benefits questionnaire items

Chapter 6: Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making 121

Information in EHR data that is useful EHRs contains patient health and healthcare-related data, which represents the collection of data meaningfully captured during clinical practices. This construct examines the range of information captured in EHRs that is perceived by doctors to be useful for decision making.

Construct Questionnaire item Useful Information EHR contains information regarding patient’s __ that is useful. (UI) UI1: chart UI2: clinical guidelines used UI3: clinical notes UI4: demographic information UI5: family medical history UI6: health and healthcare-related information UI7: laboratory and radiology test results UI8: medical diagnosis UI9: medical history UI10: medication prescription UI11: problem list UI12: smoking status UI13: treatment UI14: vital signs

Table 9. Useful Information questionnaire items

Quality of EHR data associated with clinical benefits Besides the primary use of the EHR data as capturing patient’s health and healthcare-related information for the EHR systems, secondary uses of EHR data have the potential to improve care delivery and even provide knowledge regarding diseases and suitable treatments (Botsis et al., 2010).

Construct Questionnaire item Data Quality Benefits DQB1: Alerting to critical laboratory test result values (DQB) DQB2: Assisting in creating treatment plans DQB3: Assisting in diagnosing medical condition(s) DQB4: Assisting in ordering more formulary drugs DQB5: Facilitate access to patient health information, such as diagnosis, medication, treatment, laboratory test results DQB6: Facilitate access to suitable clinical guidelines DQB7: Identifying patient who needs preventive care DQB8: Identifying patient who requires laboratory and/or radiology test DQB9: Preventing potential medication errors DQB11: Reducing redundant laboratory and/or radiology testing DQB11: Reducing similar laboratory and/or radiology testing

Table 10. Data Quality and clinical benefits questionnaire items

122 Chapter 6: Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making

This section of the survey evaluates doctors’ perception regarding the quality of EHR data that is associated with the clinical benefits experienced through the use of EHR systems. This will further validate the suitability of using EHRs as practical clinical evidence in our proposed PBE approach.

Quality of EHR data which makes it useful The previous section of the survey establishes the quality of the EHR data most associated with the benefits of using EHR systems. This section however, identifies the quality measure of EHR data that makes it useful.

Construct Questionnaire item Data Quality Since EHR contain information that is _____, it is useful. Usefulness (DQU) DQU1: accurate DQU2: practical DQU3: relevant DQU4: reliable DQU5: secure DQU6: valid

Table 11. Data quality and usefulness questionnaire items

Nature of data availability The nature of data availability has been identified as factors that affect the quality of decision making. Availability, accessibility, timely access and organised data influence how decision-makers consume information, from which to make decisions (Akyürek et al., 2015; Tunis et al., 2003). In a causal loop diagram illustrated by Osop and Sahama (2016a), the interaction between the factors mentioned above has a reinforcing effect on improving the ability to make quality decisions.

Figure 20. Factors affecting the quality of decision making reflected in a causal-loop diagram (Osop & Sahama, 2016a)

Chapter 6: Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making 123

The availability, accessibility, timely access and organised data, besides representing the characteristics of data impacting on the quality of decision making, also describes the architecture of our PBE approach, which is to integrate the disparate sources of clinical data together. Therefore, understanding doctors’ perceptions regarding data availability and how it affects their decision making, reaffirms the need for a data warehouse architecture.

Construct Questionnaire item Data Availability I am able to improve my decision-making when ___ (DA) DA1: information is available DA2: information available is organised DA3: I have timely access to information DA4: I have access to relevant information

Table 12. Data availability questionnaire items

Perceived clinical benefits of EHR system leading to improved decision making The perception of the clinical benefits of EHR systems suggests the belief that using an EHR system is improving the care provided to patients. By studying this construct, we will get an actual reflection of what doctors believe is beneficial in the EHR systems and how this leads to improved decision making.

Construct Questionnaire item Perceived Clinical PCB1: Being alerted to critical laboratory test result and values improves Benefits (PCB) my decision-making PCB2: Being assisted in creating treatment plans improves my decision- making PCB3: Being assisted in diagnosing medical condition(s) improves my decision-making PCB4: Being assisted in ordering more formulary drugs improves my decision-making PCB5: Being facilitated access to patient health information improves my decision-making PCB6: Being facilitated to suitable clinical guidelines improves my decision-making PCB7: Being able to identify patients who need preventive care improves my decision-making PCB8: Being able to identify patients who require laboratory tests to be performed improves my decision-making PCB9: Being able to prevent potential medication errors improves my decision-making PCB10: Being able to reduce redundant laboratory and/or radiology testing improves my decision-making PCB11: Being able to reduce similar laboratory and/or radiology testing improves my decision-making

Table 13. Perceived clinical benefits and decision making questionnaire items

124 Chapter 6: Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making

Perceived usefulness of EHR data for quality decision making Evaluating doctors’ perception regarding the usefulness of EHR data to assist with their clinical decision making, epitomises the value of EHRs. This perception serves to reaffirm the suitability of selecting EHRs as evidence for our PBE approach.

Construct Questionnaire item Perceived Data PDU1: Using EHR data to guide the alert to critical laboratory test result Usefulness (PDU) values can improve my decision-making PDU2: Using EHR data to guide the diagnosis of patient medical condition(s) can improve my decision-making PDU3: Using EHR data to guide the ordering of appropriate laboratory and/or radiology testing can improve my decision-making PDU4: Using EHR data to guide the prescription of appropriate medications can improve my decision-making PDU5: Using EHR data to guide the provision of preventive care to patients can improve my decision-making PDU6: Using EHR data to guide the provision of appropriate treatment plans can improve my decision-making PDU7: Using EHR data to guide the usage of appropriate clinical guidelines can improve my decision-making

Table 14. Table of constructs and corresponding questionnaire items

6.4 THE SURVEY

6.4.1 Instruments The aim of this survey study is to answer the research question that has been asked in this phase of the research. As part of the process of ensuring our PBE approach is able to assist healthcare professionals in decision making, we investigated the factors that influence decision making, the quality of EHRs that can help in decision making and the suitability of EHRs as evidence for our PBE approach. To achieve this aim, we investigated the perceived clinical benefits from the use of EHR systems and the perceived usefulness of EHR data in assisting with clinical decision making among healthcare professionals in Singapore’s public hospital and polyclinics.

An online survey tool, KeySurvey, provided by Queensland University of Technology (QUT), was used to develop the 27-item questionnaire survey. A ready template was used to ensure that the format, structure, readability and presentation of the survey was appropriate. The choice to use an online survey tool did not present an issue, since the doctors selected for the study were very familiar with the use of ICT. The finalised survey question is illustrated in Appendix G.

Chapter 6: Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making 125

The invitation to participate in the survey was done via email to prospective participants from Singapore’s public hospitals and polyclinics. An example of the email invitation template is illustrated in Appendix H.

The survey items were designed according to four major sections. The flow of the survey began with the first section, which aimed at the collection of doctor’s demographic information. The second section focused on doctors’ perceived clinical benefits from the use of the EHR systems, the third section focused on the perceived usefulness of EHR data and the final section focused on how the perceived clinical benefits and perceived usefulness of EHR data would affect quality decision making. The design of the survey flow was critical in ensuring that the questions were not misleading and potentially misinterpreted. Furthermore, each major section consisted of smaller sub-sections with clear objectives and instructions provided to ensure survey items were easily understood.

The first section consisted of selecting the relevant options to accurately depict participants’ demographics, while the rest made use of the Likert scale to indicate the participants’ level of agreement to the items being asked. The questionnaire items were adapted from studies by King et al. (2014), Secginli et al. (2014) and Simon et al. (2009), however all three studies used different measurement scales. Therefore, we developed our scales accordingly and based it on the response that we intend to investigate. Two types of Likert scales were used. First is a Likert 6-point scale of ‘Agree, used routinely’, ‘Agree, used occasionally’, ‘Agree, but used rarely’, ‘Agree, but never used before’, ‘Disagree’ and ‘Unsure’ were used to indicate their EHR system usage trends. Next, a Likert 5-point scale of ‘Strongly agree’, ‘Agree’, ‘Neutral’, ‘Disagree’ and ‘Strongly disagree’ was used to indicate what the participants believed to be their perceived opinions, attitudes and behaviours for the rest of the survey items. An example of the two scales used is provided below.

Agree, used Agree, used Agree, but Agree, but Disagree Unsure routinely occasionally used rarely never used before 1 2 3 4 5 6

Table 15. Likert 6-point scale of usage trends

126 Chapter 6: Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making

Strongly Agree Agree Neutral Disagree Strongly Disagree 1 2 3 4 5

Table 16. Likert 5-point rating scale of agreement

6.4.2 Survey participants selection criteria The participants identified for this survey study represented doctors from Singapore’s public hospitals and polyclinics and those who were active users of clinical information systems such as an Electronic Health Record system at their respective healthcare organisations. The wide range of doctors from different healthcare organisations involved aimed at ensuring that the results and response we obtained, would be more generalisable. Their participation however, was entirely voluntary and only depended upon their willingness to partake.

6.4.3 Survey Administration The survey was administered using a two-prong approach to achieve an ideal number of survey participants.

First, an initial email invitation was sent to the Chief Medical Information Officer (CMIO) and Chief Medical Board (CMB) from three public hospitals in Singapore to request permission and help in disseminating the survey invitations to doctors from their respective hospitals. Out of the three, one hospital agreed to assist and the CMIO aided in the dissemination of email invitation to the doctors from that particular hospital. The second approach involved collating the list of doctors attached to the other public hospitals in Singapore. The list of doctors was readily available from the respective hospital websites and some were even furnished with their email addresses. Those lists of doctors with the email addresses were then invited by email to participate in the survey study. However, as most email addresses of doctors were not publicly available, subsequent participants were recruited through snowball sampling. The collaborative doctors from NUH assisted in disseminating the email invitations to other doctors within NUH as well as those from other public hospitals and polyclinics.

A detailed description was provided in the survey to potential participants regarding the background of the study and the purpose of the survey. With the survey

Chapter 6: Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making 127

being divided into individual sections, each section had a short description to briefly explain the concepts or terms that were used. It was designed in such an order to outline the different perceptions that were intended to be captured throughout the survey. The survey questions were directed at understanding the doctors’ opinions and viewpoints towards the benefits and usefulness of EHR systems and information captured in EHRs to help with decision making.

The data collection period ran for three months before it was concluded.

6.4.4 Ethics We had obtained ethical clearance from our educational institution, QUT, prior to conducting the survey study. However, we were informed by collaborating doctors from NUH that it was not necessary to obtain additional ethics clearance for the doctors from the hospitals and polyclinics. Furthermore, no patient related data was gathered during the study which was actually a main concern of conducting the survey. We did not face any ethical issues or problems that arise from this study. The ethical clearance certificate is available as Appendix J.

6.5 ANALYSIS OF THE SURVEY RESULTS

To present the findings with regards to the perceived clinical benefits and perceived usefulness of EHR data in assisting with clinical decision making, a descriptive analysis of the data collected was performed. The analysis software used for the analysis was IBM SPSS Version 23.

6.5.1 Response We were unable to determine the number of participants who received the survey email invitation, as the dissemination was made with the help of the CMIO and collaborating doctors. As of 2017, the number of doctors in public healthcare institutions were 8,573 (Department of Statistics Singapore, 2018). However, through the KeySurvey system, the survey received a click-through rate of 63 clicks. Click- through represents the number of unique respondents that clicked on the survey link provided and had reached the first page of the survey but did not continue on submitting any result in the survey (Porter & Whitcomb, 2003). From this population, the total number of survey responses received was 38, representing a response rate of about 60%. Out of these 38, five were disregarded outright as they were considered as

128 Chapter 6: Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making

total non-response. Three were partially incomplete and were regarded as partial non- response with less than 50% of the survey answered. Therefore, final valid 30 responses were considered for the data analysis.

6.5.2 Survey response reliability test The Cronbach’s alpha method of measuring the internal consistency or the coefficient reliability was applied, to ensure that the survey items were measuring the same construct. The result from this measurement revealed a high internal consistency which recorded value above the accepted range (0.7). The detailed Cronbach alpha measurement results are available in Appendix J.

6.5.3 Participants Quantitative data was collected from doctors in Singapore’s public hospitals and polyclinics through the use of an online survey. The participants of the online survey consisted of one doctor from a polyclinic while the rest, from four public hospitals. The distribution of doctors based on the type of healthcare organisation is illustrated in Figure 21.

Distribution of participants based on healthcare organisations

Polyclinic

3.33% Public hospital

96.67%

Healthcare Organisation Type Organisation Healthcare 0.00% 20.00% 40.00% 60.00% 80.00% 100.00% Percentage Distribution

Figure 21. Distribution of participants based on healthcare organisation type

The participants comprised a wide range of specialists. Majority of the participants were from the Accident and Emergency Medicine department followed by those from Endocrinology. The distribution of participants based on their medical specialities is illustrated in Figure 22.

Chapter 6: Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making 129

Distribution of participants based on their medical specialty

Accident and Emergency Medicine Endocrinology General Medicine Gastroenterology Family / General Practice Others: Paediatrics Cardiology Radiology General Surgery Others: Preventive Medicine Others: Public Health Medical Specialities Medical Others: Renal Medicine Others: Nephrology Others: Infectious Disease Others: Haematology Others: Anaesthesiology 0.00 5.00 10.00 15.00 20.00 25.00 Percentage Distribution

Figure 22. Distribution of participants based on medical specialty

The participants also varied in terms of their job title and thus corresponding experience. A total of 50% of the doctors were consultants followed by senior consultants. The distribution is illustrated in Figure 23 and the breakdown of participants based on their designation is detailed in Figure 24.

Distribution of participants based on designation

Consultant 50.00%

Senior Consultant 23.33%

Senior Staff Physician 13.33%

Resident Physician 6.67% Designation

Others: Senior Resident 3.33%

Others: Associate Percentage Distribution 3.33% Consultant

0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% Percentage Distribution

Figure 23. Distribution of participants based on designation

130 Chapter 6: Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making

No. of participants by designation

Consultant 15 Senior Consultant 7 Senior Staff Physician 4 Resident Physician 2

Others: Senior Resident 1 No. of doctors Others: Associate Consultant 1

Figure 24. Number of participants based on designation

More than half of the participants were aged between 36 and 45 years. However, there was an equal distribution of those 35 years and below, and with those 46 years and older. The full distribution of participants by age is illustrated in Figure 25.

Distribution of participants based on age group

<= 35 years 20.00%

36 - 45 years 60.00%

46 - 55 years 13.33% Age Group

56 - 65 years 6.67%

0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% 70.00% Percentage Distribution

Figure 25. Distribution of participants based on age group

At least half of the participants had been practising medicine for more than 15 years, with 23% of the participants being in the service for at least five years. The distribution is illustrated in Figure 26.

Distribution of respondents based on years in service

5 - 10 years 23.33%

11 - 15 years 26.67%

> 15 years 50.00% Years in in service Years

0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% Percentage Distribution

Figure 26. Distribution of respondents based on years of service

Chapter 6: Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making 131

The detailed participant attributes are listed in Table 17. It provides a detailed background of the participants that were involved in the study.

AGE (years) Years in Service SPECIALTY DESIGNATION <= 35 36 - 45 46 - 55 56 - 65 5 - 10 11 - 15 > 15 Total Consultant 0 3 0 0 0 2 1 Accident and Senior 0 2 0 0 0 0 2 Emergency Consultant 7 Medicine Senior Staff 1 1 0 0 0 1 1 Physician Cardiology Consultant 0 1 0 0 0 1 0 1 Consultant 0 2 0 0 0 1 1 Endocrinology Senior 5 0 1 2 0 0 0 3 Consultant Consultant 0 0 0 1 0 0 1 Family / General Resident 2 Practice 1 0 0 0 1 0 0 Physician Consultant 0 1 0 0 0 1 0 Gastroenterology Resident 2 1 0 0 0 1 0 0 Physician Senior 0 0 1 0 0 0 1 General Consultant 2 Medicine Senior Staff 0 1 0 0 0 1 0 Physician General Surgery Consultant 0 1 0 0 0 0 1 1 Radiology Consultant 0 1 0 0 1 0 0 1 Others: Consultant 0 1 0 0 0 0 1 1 Anaesthesiology Others: Consultant 0 1 0 0 0 1 0 1 Haematology Others: Infectious Consultant 0 1 0 0 0 0 1 1 Disease Others: Consultant 0 0 0 1 0 0 1 1 Nephrology Others: Renal Senior Staff 0 0 1 0 0 0 1 1 Medicine Physician Others: Public Senior Resident 1 0 0 0 1 0 0 1 Health Associate 1 0 0 0 1 0 0 Others: Consultant 2 Paediatrics Senior 0 1 0 0 0 0 1 Consultant Others: Preventive Consultant 0 0 0 0 1 0 0 1 Medicine Total 30 Table 17. Participant’s attributes and demographics.

The distribution in terms of the number of years the participants had been using an EHR system was almost equally distributed according to the assigned age groups. The minimum number of years the participants used an EHR system was 2 years to a maximum of 20 years. The calculated mean number of years therefore was 9 years and a standard deviation of 4.65 years. The distribution of participants’ years of EHR system usage is illustrated in Figure 27.

132 Chapter 6: Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making

Distribution of respondents based on length of EHR system use

1 - 5 years 30.00%

6 - 10 years 40.00%

>= 11 years 30.00% Length of EHR usage 0.00% 5.00% 10.00% 15.00% 20.00% 25.00% 30.00% 35.00% 40.00% 45.00% Percentage Distribution

Figure 27. Respondent distribution by years of EHR system usage

6.5.4 Descriptive Analysis Based on the survey data collected, we performed descriptive analysis that allowed us to describe the survey data in a more meaningful way so that it could be easily understood.

Demographics Out of the 30 doctors who took part in the survey, 96.7% were from public hospitals and 3.3% from polyclinics. Majority of the doctors surveyed were from the Accident and Emergency Medicine department, at 23.3%, followed by Endocrinology, at 16.7%. General Medicine, Gastroenterology, Family and General Practice, and Paediatrics each representing 6.7% of doctors surveyed. Cardiology, Radiology, General Surgery, Preventive Medicine, Public Health, Nephrology, Infectious Disease, Haematology and Anaesthesiology departments all registered 3.3% of doctors surveyed.

Half (50%) of the doctors surveyed were consultants, 23.3% were senior consultants, 13.3% were senior staff consultants, 6.7% were resident physicians and 3.3% were senior resident and associate consultants. When asked about their age, most were between the ages of 36 and 45 years (60%). While, 20% were below the age of 36 and 13.3% were between the ages of 46 and 55 years. Only 6.67% of the participants were older than 55 years. This was reflective of the number of years the doctors had been in service. A total of 50% of the doctors had more than 15 years’ experience being a doctor. However, those who had been in service between 11 and 15 years (26.7%) were just slightly more than those with five to ten years’ experience (23.3%). Most importantly, was the number of years the doctors had been using EHR systems as part of their daily patient care routine. For this, more than half (66.7%) of the

Chapter 6: Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making 133

participants had more than six years of experience in using EHR systems. Only 10% used EHR systems for the past three years.

EHR system usage trends The participants were also asked regarding their EHR system usage trends in terms of how it helped manage their patients. More than 60% (62.1%) of the doctors disagreed that the EHR systems could be used as a platform to communicate with their patients. Most agreed that they could use the EHR systems to access clinical guidelines and patient charts remotely. However, only 76.7% had actually used them to access clinical guidelines while 86.7% had accessed patient charts remotely.

The majority of the uses of EHR systems identified were for documenting and viewing patients’ clinical and medical information. In fact, all the doctors surveyed indicated that they had used the EHR system to document and view their patients’ medical diagnoses and medical history. Almost all (96.7%) used it to document and view patient clinical notes. Further, 83.3% used it for patient health-related information such as height, weight, BMI, etc., 75.9% used it to capture vital signs such as blood pressure, temperature, heart rate, etc., 73.3% for demographic information, 63.3% for indicating a family’s medical history and 53.3% had made use of it to document patient smoking status.

EHR systems were also used by all the doctors to order clinical and medical tests as indicated by the survey results. More than 90% of the doctors surveyed stated that they had used the EHR systems to routinely order laboratory tests (93.3%), medication prescription (96.7%), radiology tests (93.3%) and treatments (93.1%). All of them also agreed that EHR systems allowed them to review all the laboratory and radiology tests conducted. All doctors surveyed also indicated that EHR systems were used to review patient medical and medication history, while only 93.3% used it routinely to review patient problem lists.

However, barely half of the doctors surveyed provided patients with their copies of health information, with only 46.4% of the doctors routinely doing it. Only 53.3% of the doctors provided patients with their clinical summaries per visit routinely. The detailed responses to the trend of EHR system used are illustrated in Appendix K1.

134 Chapter 6: Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making

EHR system use leading to clinical benefits Based on the trend of EHR system usage concerning the management of patient care, doctors were surveyed on how EHR system use led to perceived clinical benefits gained. About 73.3% of doctors surveyed strongly agreed that EHR systems facilitated access to patient’s health information such as their diagnosis, medication, treatment and laboratory test results among others; 66.7% also agreed strongly that EHR systems were able to alert them to critical laboratory test results and values. More than 60% (63.4%) agreed that EHR systems were assisting doctors in creating treatment plans, but only 16.7% strongly agreed to it with 13.3% disagreeing entirely and 23.3% being neutral to the statement. However, only half the number of doctors (50%) surveyed indicated that the EHR systems facilitated the access to suitable clinical guidelines to use, which could potentially help in improving the delivery of care. Slightly more than 50% (53.4%) indicated that EHR systems were able to identify patients who needed preventive care, but only 16.7% strongly agreed to that, and 10% disagreed.

Interestingly, less than 50% (44.8%) actually agreed that EHR systems were assisting with the diagnosis of medical conditions and 24.1% disagreed that such systems were helping them with the diagnosis. A high percentage (31%) of the participants was also neutral to the statement, an indication that they were probably unsure of the EHR systems’ capabilities.

A total of 83.4% of the doctors surveyed agreed that they were assisted in ordering more formulary drugs, and 86.6% collectively agreed that they could be prevented from making potential medication errors.

Regarding ordering laboratory and radiology testing, a total of 76.7% indicated that EHR systems were able to identify patients who required laboratory and/or radiology testing. A total of 76.7% also stated that such systems were helping to reduce redundant laboratory or radiology testing and 76.6% agreed that it also helped to reduce similar ordering of laboratory or radiology testing.

Overall, the majority of the doctors surveyed (> 70%) agreed that EHR systems were used to access patient health information, as well as being helpful in assisting with care management such as ordering medication, being alerted to critical information, reducing medication errors and preventing unnecessary testing. A detailed table, indicating the survey responses to the association of EHR system use and perceived clinical benefits, is shown in Appendix K2.

Chapter 6: Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making 135

Distribution of EHR functionalities leading to Clinical Benefits Strongly Agree Agree Neutral Disagree Strongly Disagree 80.0%

70.0%

60.0%

50.0%

40.0%

% Distribution% 30.0%

20.0%

10.0%

0.0% SF1 SF2 SF3 SF4 SF5 SF6 SF7 SF8 SF9 SF10 SF11 EHR System Functionalities

Figure 28. Distribution of EHR systems use leading to clinical benefits

Information captured in EHR data that is useful The perceived usefulness of EHR data depends on the information captured in EHRs through the use of EHR systems and how EHR data can be used to improve decision making. When asked about what doctors believed EHR data held, all the doctors (100%) surveyed unanimously agreed that EHR data contained information regarding patient’s laboratory and radiology test results, medical diagnosis, medical history, medication prescribed and treatment provided. However, 96.6% implied that EHR data contained health and healthcare-related information such as height, weight and body mass index (BMI). It was also unexpected to notice that not all doctors surveyed (93.3%) agreed that EHR data contained patient demographics and vital signs’ information, with 3.3% (patient demographics) and 6.7% (vital signs) disagreeing respectively.

A significant number of doctors (20%) indicated that EHR data did not contain information regarding the clinical guidelines used, although 40% agreed it did. This finding was reflective of what doctors had earlier perceived regarding the clinical benefits of EHR systems that EHR systems were not being able to facilitate access to

136 Chapter 6: Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making

suitable clinical guidelines to use. Almost 90% agreed that a patient’s chart (86.7%), clinical notes (90%) and problems list (93%) were available in EHR data. On the other hand, only 73.3% suggested that information regarding family medical history was found in EHR data. Similarly, 72.4% admitted that patient smoking status was captured.

The responses indicated the extent of information captured in EHR data. However, none of these responses noted any substantial disagreement with the kinds of information being captured. This may suggest that EHR data indeed contain information that can be used to assist with decision making. A detailed list of the responses is tabulated in Appendix K3.

Distribution of information captured in EHR as useful Strongly Agree Agree Neutral Disagree Strongly Disagree 100.0%

90.0%

80.0%

70.0%

60.0%

50.0%

40.0% % Distribution% 30.0%

20.0%

10.0%

0.0% UI1 UI2 UI3 UI4 UI5 UI6 UI7 UI8 UI9 UI10 UI11 UI12 UI13 UI14 Useful Information Type

Figure 29. Distribution of perception of how useful information captured in EHR is.

Quality of EHR data Having quality data demonstrates how useful it can be when utilising such meaningful evidence to assist with decision making, as per our approach of Practice- based evidence. Accordingly, the survey had also asked doctors regarding the quality of data captured in EHRs in the following categories of being ‘Accurate’, ‘Practical’, ‘Relevant’, ‘Reliable’, ‘Secure’ and ‘Valid’. ‘Accurate’ refers to data being free from error or defect. ‘Practical’ refers to data that consists of information resulting from actual clinical practices. ‘Relevant’ refers to data relating directly and significantly to

Chapter 6: Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making 137

clinical practices. ‘Reliable’ refers to data being dependable in accuracy, being able to be trusted. ‘Secure’ refers to data that is safe and has not been wrongfully tampered with. ‘Valid’ refers to data being sound and well-founded.

Accurate Practical Total Total Total Total Disagreement Agreement Disagreement Agreement 10% 77% 0% 83%

Neutral Neutral 17% 13%

Relevant Reliable Total Total Total Total Disagreement Disagreement Agreement Agreement 10% 3% 90% 70%

Neutral Neutral 20% 7%

Secure Valid Total Total Total Total Agreement Disagreement Agreement Disagreement 80% 77% 7% 3%

Neutral Neutral 20% 13%

Figure 30. Perceived Data Quality of EHR

138 Chapter 6: Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making

In general, at least 70% of the doctors surveyed agreed that data captured in the EHR systems used, could be rated as being accurate, practical, relevant, reliable, secure and valid. Accordingly, 90% indicated that the EHR data was relevant. Close to 80% recognised that the data was accurate (83%) and secure (80%). For being practical and valid, 77% of the doctors surveyed were in agreement. However, between 4% and 10% of the doctors disagreed that the EHR data was actually useful for decision making. Most of those who opposed, cited practical (10%) and reliable (10%) as the quality of data. This could potentially explain why some doctors believed that EHR systems failed to prevent potential medical errors. Detail of the frequency and distribution of survey responses on EHR data quality is illustrated in Appendix K3 and Figure 30.

Distribution of Perceived EHR data quality Strongly Agree Agree Neutral Disagree Strongly Disagree 60.0%

50.0%

40.0%

30.0%

20.0% % Distribution%

10.0%

0.0% DQU1 DQU2 DQU3 DQU4 DQU5 DQU6 Data Quality Usefulness

Figure 31. Distribution of perceived EHR data quality

Association between EHR data quality and perceived clinical benefits Subsequent survey questions asked doctors about the quality of EHR data that could be associated with the perceived clinical benefits of utilising EHR systems. The doctors were asked based on the categories of EHR data quality mentioned above and 11 aspects of clinical benefits identified from the use of such systems. While the survey was unable to discern that these different qualities of EHR data were directly responsible for the various clinical benefits, it was, however able to highlight, based

Chapter 6: Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making 139

on the doctors’ experiences and knowledge, their beliefs of which quality of EHR data might be responsible for its usefulness.

On the aspect of being alerted to critical laboratory test result values, ‘Accurate’ was scored highest with 70% of those surveyed, selecting it. This was followed by ‘Practical’ at 46.7%, ‘Reliable’ at 36.7% and ‘Relevant’ at 33.3%. ‘Secure’ and ‘Valid’ were lowest at 26.7% and 23.3% respectively.

For assisting in creating treatment plans, close to half (46.7%) indicated that it was due to EHR data containing information captured as a result of actual clinical practices, i.e. ‘Practical’. However, other aspects of data quality fared low amongst the doctors with only a third (33.3%) selecting ‘Relevant’ and 23.3% indicating ‘Accurate’.

‘Relevant’ was highly quoted (40%) as the quality of data that allowed EHR systems to be able to assist in diagnosing medical conditions. ‘Accurate’ and ‘Practical’ were not far behind with 33.3% of doctors selecting them. Next, ‘Reliable’ (20%), ‘Secure’ (16.7%) and ‘Valid’ (16.7%) were spread evenly out amongst doctors surveyed.

For assisting in ordering more formulary drugs and facilitating access to patient health information, half of those surveyed (50%) acknowledged ‘Practical’ as being associated with the data quality responsible for benefit mentioned above while 40% indicated ‘Relevant’ and 33.3% ‘Accurate’.

Notably, 43.3% also indicated ‘Practical’ as the data quality aspect that facilitated access to suitable clinical guidelines, with 30% indicating ‘Relevant’. However, only 10% indicated ‘Accurate’ as a quality. It is possible that the doctors believed that accuracy might not be a critical factor when accessing appropriate guidelines or that the lack of access to clinical guidelines is due to them being inaccurate.

When identifying patients needing preventive care, the data quality most associated was ‘Relevant’, selected by 43.3% of the doctors surveyed, followed by ‘Practical’, with 33.3%. ‘Accurate’ was only selected by 16.7%, and both ‘Reliable’ and ‘Secure’ only garnered 13.3% response.

Interestingly, for prevention of potential medication errors, ‘Relevant’ was selected by 36.7% of those surveyed. More than half (53.3%) indicated ‘Practical’ as

140 Chapter 6: Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making

the quality why EHR systems were able to assist in the prevention of medication errors. Indeed, this reflected the importance of capturing evidence of actual clinical practice as it would be useful and helpful in ensuring clinical benefits such as this. Furthermore, this was the only aspect of the clinical benefits that had more than 20% of all the data qualities selected by doctors. This indicated the importance of having all quality aspects of EHR data when used to prevent potential medication errors.

Reducing redundant and similar laboratory or radiology testing fared alike. More than 50% selected ‘Practical’ (50% for redundant testing and 53.3% for similar testing) as the quality aspect important in enabling both benefits. More than 30% indicated ‘Relevant’ and 23.3% for ‘Accurate’.

Looking at the overall responses in this particular section of the survey, the qualities of EHR data most associated with perceived clinical benefits gained as indicated by doctors were ‘Practical’, ‘Relevant’ and ‘Accurate’. These qualities were also consistently regarded across the different clinical benefits highlighted in the survey, as illustrated in Figure 32. While the survey was not able to accurately confirm the qualities mentioned above as being directly responsible for individual clinical benefits, it managed to provide us with an understanding of how useful and meaningful EHR data could be and how it had helped in gaining clinical benefits.

Chapter 6: Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making 141

Distribution of Quality of data per Perceived Clinical Benefit

Accuracy Practical Relevant Reliable Secure Valid 80.0%

70.0%

60.0%

50.0%

40.0% % Distibution% 30.0%

20.0%

10.0%

0.0% DQB1 DQB2 DQB3 DQB4 DQB5 DQB6 DQB7 DQB8 DQB9 DQB10 DQB11 Perceived Clinical Benefit

Figure 32. Distribution of the quality of data by doctors using EHR systems in association with perceived clinical benefits gained

142 Chapter 6: Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making

Perceived Clinical Benefits leading to improved decision making The response to perceived clinical benefits and decision making relates to doctors’ perceptions of the EHR system functionalities that best support their decision- making processes.

Distribution for Perceived Clinical Benefits Strongly Agree Agree Neutral Disagree Strongly Disagree 90.0%

80.0%

70.0%

60.0%

50.0%

40.0%

% Distribution% 30.0%

20.0%

10.0%

0.0% PCB1 PCB2 PCB3 PCB4 PCB5 PCB6 PCB7 PCB8 PCB9 PCB10 PCB11 Perceived Clinical Benefits

Figure 33. Distribution of perceived clinical benefits

Based on the survey responses, the majority of the doctors surveyed (82.8%) strongly agreed that the significant influence on decision making was the ability to facilitate access to patient health information (PCB5) when using the EHRs. This was followed by being alerted to critical laboratory test values (PCB1) at 72.4% and preventing potential medication errors (PCB9) at 67.9%.

Perceived usefulness of EHR data and for quality decision making Based on the survey responses, almost all the doctors surveyed agreed that EHR data contained valuable information that could help to improve their decision making. All the doctors surveyed (100%) agreed that when the EHR data captured held information such as clinical practices, they were able to use it to improve their decision making. The EHR data could be utilised to guide the alert of critical laboratory test result values and prescription of appropriate medications However, only slightly more than half of the doctors surveyed agreed with the following. About 55.2% indicated strongly that their decision making could be improved when utilising EHR data to guide in the diagnosis of a patient’s medical condition. About 57.1% reported

Chapter 6: Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making 143

improvements to decision making when ordering appropriate laboratory and radiology tests, 58.6% for the prescription of medications, and 51.7% for the provision of appropriate treatment plans. Further, 75.9% agreed that they could be guided to provide preventive care to patients and 72.4% indicated that they were guided to use appropriate clinical guidelines.

Distribution of EHR data usefulness Strongly Agree Agree Neutral Disagree Strongly Disagree 80.0%

70.0%

60.0%

50.0%

40.0%

30.0% % Distribution%

20.0%

10.0%

0.0% PDU1 PDU2 PDU3 PDU4 PDU5 PDU6 PDU7 Perceived Data Usefulness

Figure 34. Distribution of perceived EHR data usefulness

Availability of information and decision making All the doctors surveyed agreed that the nature of how information was made available could affect the ability to make decisions. Based on the responses, 89.7% strongly agreed that having access to relevant information was important in the ability to improve their decision making. A total of 86.2% indicated that the mere availability of information allowed them to improve their decision making, while 82.8% strongly agreed that organised and timely access to information enabled them to improve their decision making.

144 Chapter 6: Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making

Distribution of Availability of information and decision making

Strongly Agree Agree Neutral Disagree 100.0% 90.0% 80.0% 70.0% 60.0% 50.0% 40.0%

% Distribution% 30.0% 20.0% 10.0% 0.0% DA1 DA2 DA3 DA4 Data Availability

Figure 35. Distribution of availability of information and decision making

6.5.5 Findings and Discussions The descriptive analysis provided us with quantitative information describing the responses of doctors from Singapore’s public hospital and polyclinics regarding the survey constructs. However, this survey study aimed at answering the following research question raised.

“What are the processes required that ensure a PBE approach can assist healthcare professionals in clinical decision making?”

To answer the above question, two objectives were identified that needed to be met. The two objectives were (1) identify the limits to make well-informed decisions and, (2) identify how clinical evidence from electronic health records can be used to assist in decision making. Therefore, a further analysis (inferential) was conducted based on the descriptive analysis of the survey results. In this analysis, we were interested in focusing specifically on aspects of EHR usage trends and the corresponding responses concerning the perceived clinical benefits and perceived usefulness of data for decision making amongst the surveyed doctors. This in turn helped in the evaluation of EHRs as a potential source of evidence for use in our PBE approach to decision making.

Chapter 6: Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making 145

Objective 3.1: Identify the limits of our ability to make well-informed decisions. The survey on the availability of data questioned how the nature of information made available influence the ability to make decisions. Based on the survey results, the following four aspects of data availability, (1) availability of data, (2) organised data, (3) timely access to data, and (4) access to relevant information were indicated by the doctors as being influential in their ability to make decisions. Therefore, in order for healthcare professionals in general to make effective decision making, these four aspects of data availability needed to be addressed. In fact, this is exactly what our PBE approach proposes. The data warehouse architecture, which is one of the key components in our PBE approach, integrates disparate sources of data together, making it available to the healthcare professionals to use. A data warehouse is a central repository that makes data, otherwise unintegrated, integrated and accessible (Ado et al., 2014). In fact, the data warehouse methodology ensures that the integrated data is organised logically so that the information to be used later is correct and comprehensive. Timely access and access to relevant information go to show that disparate sources of information may inhibit healthcare professionals from having these types of access. Therefore, by having a data warehouse, it not only saves time for users to access data but to do it at their convenience (Scheese, 1998).

It is important to note that the survey results did not specifically mention that a data warehouse methodology would solve issues concerning improved decision making. However, the nature of how data availability can be overcome, reflects the solutions that can be provided by implementing a data warehouse.

Objective 3.2: Identify how clinical evidence from electronic health records can be used to assist in decision making. The five aspects of EHR usage trends highlighted were those that facilitated (1) provision of appropriate treatment, (2) diagnosis of medical condition, (3) prescription of appropriate medication, (4) the use of clinical guidelines and (5) provision of preventive care. By aligning the perceived benefits with the adoption of Practice-based evidence for decision making, we were interested to find out how doctors’ perceptions progressed from the use of EHR systems to utilising EHR data, in assisting with decision making. However, we were only focused on opinions of doctors who had indicated to being strongly agreeable.

146 Chapter 6: Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making

In answering the research question above, we first focused on the survey results depicting doctors’ perception of EHR system usage trends (UT). In this section, most of the EHR usage trends had more than 50% of doctors identifying themselves with the use of the listed aspects of EHR systems regularly. Only items UT1, UT3 and UT17 had less than 50% of doctors using them on a regular basis. The three were, using EHR systems to access clinical guidelines (UT1), communicate with patients (UT3) and provide patients with copies of their health information (UT17). The rest of EHR system use questions therefore could be inferred as potentially indicative of the different kinds of data that was documented or captured by EHR systems. About items UT4 to UT22, it could be inferred that the electronic health records contained information such as patients’ clinical notes, demographics, medical diagnosis, medical history, family’s medical history, smoking status, vital signs and other health and healthcare related information based on how EHR systems were being used. Items UT18 to UT22 on the other hand, identified other information that was captured in EHRs. They were, laboratory and radiology tests and results, medication prescribed, and treatments provided. Based on the survey results of doctors’ perception regarding how EHR systems could be used for patient care management, we therefore established that, the EHR systems used in Singapore by the public hospitals and polyclinic could, and potentially captured an extensive level of patients’ healthcare information as mentioned above. This suggested that the EHRs in Singapore contained valuable data that has the potential to be used as evidence that can assist in decision making.

We noticed for the types of EHR usage trends that had a high proportion of doctors being strongly agreeable to, there seemed to be a correlation or similarity in response to the construct of perceived clinical benefits and perceived usefulness of EHR data. Therefore, this made these latter constructs relevant to the establishment of EHRs as evidence for use in PBE approach to decision making. They are further analysed in the following sections.

Following the establishment that EHRs in Singapore’s healthcare organisations contained extensive levels of healthcare information regarding patients, the following analysis turned to examine the clinical benefits gained through EHR systems use. To understand if the EHR system’s use brought clinical benefits to the doctors and in turn was helping them to improve their decision making, the survey results of EHR systems

Chapter 6: Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making 147

Distribution of EHR System Usage Trends Agree, used routinely Agree, used occasionally Agree, but used rarely Agree but never used before Disagree Unsure 100.0%

90.0%

80.0%

70.0%

60.0%

50.0%

% Distribution% 40.0%

30.0%

20.0%

10.0%

0.0% UT1 UT2 UT3 UT4 UT5 UT6 UT7 UT8 UT9 UT10 UT11 UT12 UT13 UT14 UT15 UT16 UT17 UT18 UT19 UT20 UT21 UT22

Usage Trends

Figure 36. Distribution of the usage trends with regards to EHR systems in Singapore

148 Chapter 6: Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making

functionalities leading to clinical benefits (SF), and perceived clinical benefits to decision making (PCB) were reviewed.

Referring to Figure 37, as noted from the survey results, especially for those with more than 50% of doctors strongly in agreement, such as items SF1, SF6 and SF11, there potentially exist correlations with items PCB1, PCB5 and PCB9. SF1 identified with EHR systems providing doctors with the clinical benefit of being alerted to critical laboratory test results and values. PCB1 established that doctors agreed that being alerted to critical laboratory test results and values has the potential to improve their decision making. While the survey result was unable to categorically make claims that perceived clinical benefits gained from EHR system use would lead to improved decision making, however, by correlating the results for SF1 and PCB1 together, we interpreted as, doctors seemingly agreed that the clinical benefit of being alerted to critical laboratory test results and values could potentially improve their decision making. This probably suggest the claim that clinical benefits from EHR system use can actually enhance doctors’ decision making. Further indication from survey results includes item SF6 which identified with EHR system use providing doctors with the clinical benefit of facilitating access to patient health information such as diagnosis, medication, treatment and laboratory test results. This was supported by item PCB5 which was, being facilitated access to patient health information improved doctors’ decision making. Similarly, with item SF11 relating to EHR system providing doctors with the clinical benefit of preventing medical errors and PCB9 explaining that by being able to prevent potential medical errors, it improved their decision making. However, the above analysis only looked at survey results for questions where more than 50% of doctors indicated their strong agreement to. While not dismissing the opinions of doctors who stated other than being in strong agreement, we believed that analysing the views of doctors who indicated their strong agreement showed a stronger conviction and confidence that they had regarding their perceptions.

Chapter 6: Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making 149

Distribution of EHR functionalities leading to Clinical Benefits Strongly Agree Agree Neutral Disagree Strongly Disagree

SF11 SF10 SF9 SF8 SF7 SF6 SF5 SF4 System Funtionalities System SF3 SF2 SF1

0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% % Distribution

Distribution of Perceived Clinical Benefits Strongly Agree Agree Neutral Disagree Strongly Disagree

PCB11 PCB10 PCB9 PCB8 PCB7 PCB6 PCB5 PCB4

Perceived Clinical Perceived Clinical Benefits PCB3 PCB2 PCB1

0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0% % Distribution

Figure 37. Comparing the survey results from EHR functionalities and the corroboration with decision making

The next set of analysis examined the perceived clinical benefits to decision making (PCB) and the perceived EHR data usefulness to decision making (PDU) to identify items that might have supported findings that EHR data could indeed be useful for healthcare professionals in making well-informed decisions. This analysis was evaluated based on the proportion of doctors with high strongly agree opinions for both cases. Based on Figure 38, other than items PDU5 and PDU7, all other items had more than 50% of doctors having strongly agree opinions to how EHR data could be used

150 Chapter 6: Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making

in assisting with decision making. In Figure 39, items PCB1, PCB5, PCB8, PCB9, PCB10 and PCB11 had more than 50% of the proportion of doctors placing strongly agree opinions regarding the clinical benefits of using EHR systems. This analysis therefore studied the relationships between the two constructs.

Figure 38. Items with less than 50% of the doctors in strong agreement

Figure 39. Items with more than 50% of doctors in strong agreement

Chapter 6: Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making 151

PCB2 strong opinions low, PDU6 strong opinions high: Provision of appropriate treatment A part of being able to provide effective care delivery is the ability of healthcare professionals to decide on appropriate treatments for patients based on their individual medical conditions. Based on the survey results, the proportion of doctors who strongly agreed that they gained clinical benefits of being able to assist in creating treatment plans (PCB2) was lower than 50%. While this did not distinctively imply that doctors did not recognise PCB2 as providing clinical benefits, instead it might have just indirectly suggested that not all doctors were strongly convinced that this was a clinical benefit gained. However, when considering the EHR data usefulness in being able to assist doctors in making decisions that would enable the provision of appropriate treatment plans (PDU6), the proportion of doctors strongly agree was actually higher than 50%. This could be taken as an indication that a higher proportion of doctors found that EHR data can be used to assist them in deciding on appropriate treatment plans. Furthermore, all the doctors agreed (100%) that information regarding treatments that was captured in EHR data through the use of EHR systems was useful to them (UI13). Thus this further suggests that EHR data can be used to help with making decisions.

PCB3 strong opinions low, PDU2 strong opinions high: Diagnosing medical condition Being able to diagnose patients’ medical conditions is critical in identifying the medical problems they face. Only then can the right medication be prescribed, treatment be offered, and tests be conducted. Based on the survey results, all the doctors agreed that they had used the EHR system to document and view patients’ medical diagnoses. However, the proportion of doctors who strongly agreed that they found clinical benefits from EHR systems to assist in diagnosing medical conditions (PCB3) was lower than 50%. However, it is important to note that this does not guarantee that the doctors were definite that EHR systems were unable to assist in diagnosing medical conditions. It just mean that doctors did not see that EHR systems were able to assist in reaching a medical diagnosis. Yet, when evaluating the survey results on the usefulness of using EHR data in guiding doctors to diagnose patients’ medical condition (PDU2), doctors actually did show a higher proportion of being strongly agreeable. This may seem to suggest that if EHR systems can make use of EHR data, it may be useful in assisting doctors in making informed decisions. In fact,

152 Chapter 6: Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making

all the doctors surveyed agreed that EHR that contained information regarding a patient’s diagnosis could be useful (UI8) to them. This potentially illustrates another example where doctors believed that EHR data containing valuable information can be used in making well-informed decisions.

PCB4 strong opinions low, PDU4 strong opinions high: Provision of appropriate medication Likewise, the proportion of doctors who strongly agreed was low when assessing if EHR systems provided them with the clinical benefits of ordering more formulary drugs (PCB4). The percentage of doctors who believed that EHR data could be useful in assisting them to prescribe appropriate medication was much higher (PDU4). This seems to suggest the high value of EHR data regarded by doctors, such that its use was considered to be able to guide doctors in improving their decision making through prescribing appropriate medications. In fact, all doctors surveyed also agreed that EHR data that contained information regarding patients’ medication prescribed could be useful (UI10). However, this cannot be taken to imply that not being able to order more formulary drugs will result in prescribing medications inappropriately.

Therefore, based on the analysis of the survey results, the findings above, at the very least, may suggest that EHR data contains valuable information that can be used to assist doctors in providing appropriate treatments, diagnosing medical conditions and prescribing appropriate medication. Thus, we can potentially conclude that EHRs, i.e. evidence of actual clinical practices captured in digital format, have the potential to be the evidence that can be used to assist in decision making.

In addition, we also took into consideration other survey results and analyse how that may relate to how EHR data may benefit in improving decision making.

Perception of the quality of EHR data which is useful Based on the survey results regarding the EHR data quality which makes it useful, the doctors ranked, based on the mean scores, the data quality as follows:

1. Relevant (DQ3) 2. Accurate (DQ1) 3. Valid (DQ6) 4. Practical (DQ2) 5. Secure (DQ5) 6. Reliable (DQ4)

Chapter 6: Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making 153

Summary of data quality with mean rank and standard deviation Data Quality Mean Standard deviation Relevant (DQ3) 1.633 0.765 Accurate (DQ1) 1.800 0.714 Valid (DQ6) 1.867 0.860 Practical (DQ2) 1.900 1.094 Secure (DQ5) 1.933 0.980 Reliable (DQ4) 2.067 0.980

Table 18. Mean rank and standard deviation

However, when analysing the perceived quality of EHR data with associated clinical benefits, the majority of the doctors indicated being accurate (DQB1), practical (DQB2, DQB4-5, DQB8-11) and relevant (DQB3, DQB7) as the quality most associated with. This seems to align with the perceived EHR data quality rank above. Therefore, this can also suggest what makes EHRs useful for secondary uses as well as illustrating the potential of EHR data to be used as practical clinical evidence.

Use of clinical guidelines Clinical guidelines refer to documents that have been systematically developed to help healthcare professionals make clinical decisions based on available evidence (Prior et al., 2008). The guidelines usually contain instructions for healthcare professionals on what tests to be ordered, treatments to be provided as well as other clinical practices (Woolf et al., 1999). Observing these recommendations from such guidelines enables healthcare professionals to improve the care process, outcomes and costs (Prior et al., 2008). This ensures that clinical practices are standardised and where patients get to reap maximum care benefits (Kendall et al., 2009). Therefore, having access to clinical guidelines and applying them to patients represents a way of providing effective care delivery.

Based on the analysis results on EHR system usage trends (UT), about 76% of the doctors surveyed indicated that they have actually used EHR systems to gain access to clinical guidelines. In EHR system functionalities leading to clinical benefits (SF), only 50% of doctors noted that the EHR system actually facilitated the access to suitable guidelines to use. Therefore, it is not surprising to notice that only 40% of the doctors indicated that EHR data contain information regarding clinical guidelines used

154 Chapter 6: Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making

that may be useful to doctors. While exact instructions or recommendations from clinical guidelines are not captured in EHRs, however, the clinical test ordered, medication prescribed or treatments provided are indeed captured in EHR. With more than 90% (96.4%) of the doctors surveyed indicating that by being able to facilitate access to suitable clinical guidelines improves their decision making, and with more than 70% (72.4%) indicating that using EHR data to guide the usage of appropriate clinical guidelines will undoubtedly improve their decision making, this can also imply that if EHR data is able to capture information regarding clinical guidelines, using it can probably improve doctors’ decision making.

The above analysis has managed to answer the research questions that have been identified in this phase of the study. We identified some of the assumed limitations of EHR systems as perceived by the doctors, which were actually survey items measuring the perceived clinical benefits. This perceived limitations however seems to suggest that they can be improved through the use of EHRs which contains data and information useful in assisting with doctors’ decision making. But, as the survey was not able to precisely identify the exact reasons behind the doctors’ opinions, we can only assume based on the results of other survey constructs. However, by being able to observe the differences in opinions with regard to what EHR systems currently offer to doctors and what they benefit from it and then comparing the findings to what doctors perceived can be offered if EHR data is used in accordance to improve their decision making, this potentially illustrate the value of EHR data that should be capitalised. Hence, there is great potential in using EHRs as evidence for our PBE approach to assist and improve healthcare professionals’ decision making.

6.5.6 Limitations As stated, this study was limited to doctors from Singapore’s public hospital and polyclinics, who had experience in using EHR systems. There were no public databases that provided email addresses of doctors in Singapore, other than the exceptions of some of the public hospitals’ website, therefore the CMIO and CMB of the public hospitals in Singapore were initially contacted to request their assistance in getting the doctors from respective healthcare organisations to participate in the survey. However, some requests were denied due to resource and time limitation while other requests were totally ignored, even after repeated emails. One healthcare organisation’s CMIO favourably replied and willingly helped to disseminate the email

Chapter 6: Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making 155

to the doctors from that hospital. The limited number of doctors’ email addresses that were found on the public hospitals’ website was also used to invite them to participate. A decision was later made to get the assistance of collaborating doctors from NUH to help disseminate the survey invitation as well.

The small number of participants may have presented as a limitation of the survey study. This has the potential of introducing biases as studies with small sample size tend to affect the reliability of the survey results (Hackshaw, 2008). However, the doctors who took part in the study were from different departments and clinical disciplines, possibly with varying aspects of EHR system use. The doctors were also from different public hospitals, perhaps with varying EHR systems altogether. Additionally, there was an almost equal distribution of doctors who may be new to EHR systems use and those who have experienced using them for more than a decade. While this may not necessarily compensate the limitations of a study from small sample size, probably, this potentially provide a broader perspective or view from the range of participants. Short description or preamble was used in every section of the survey to explain the meanings of terms used in the survey questions as well as to set the context to the questions asked. However, this may have introduced bias into the survey responses. For example, explaining what “clinical benefits” mean and providing an example of one such benefit to be “prevention of medical errors”. Therefore, when surveying participants’ opinions on clinical benefits, their response may be influenced by the preamble. In our attempt to avoid such biases, the questions that have been asked, have explicitly requested participants to base it on their knowledge and experiences using EHR systems in their respective healthcare organisations. While there is no guarantee that this approach has avoided biases, it has certainly reminded the participants to respond only based on their personal experiences of using the EHR systems. In addition, only participants with EHR systems use experience were invited to participate, further avoiding any presumptions of EHR system use.

Another possible limitation of the survey administration is the lack of balance in the way questions have been asked. The absence of negative questions and the reverse order of ticked responses may also introduce bias to participants’ responses. Therefore, we computed the Cronbach’s alpha value to measure and evaluate the reliability of the surveyed items.

156 Chapter 6: Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making

6.6 CONCLUSIONS

The analysis has provided us with the findings and interpretations based on the survey results obtained.

The analysis may have also exposed some gaps where a far more effective delivery of care can be provided. These gaps, as interpreted based on the survey results, have in fact revealed the potential of utilising information captured in EHR data to better improve healthcare professionals’ clinical decision making. This was reflected in the difference between how the doctors surveyed had described their EHR systems’ usage trends and the perceived benefits where such usage trends or system functionalities could have provided them with improved decision making. For example, only a small proportion of the doctors indicated that their EHR systems are able to assist with the diagnosis of medical conditions, while the majority of the same population of doctors surveyed indicated that, had the EHR system been able to assist with clinical diagnosis, it would have improved their decision making. Thus, this seems to suggest a limitation of the EHR system in providing effective care to patients. However, this limitation could potentially be overcome by utilising the information captured in EHR data, based on the positive perception of doctors’ survey results on the usefulness of EHR data for decision making. Thus, this potentially answers the research question that has been asked.

Based on the overall survey results, doctors in Singapore indicated that the use of EHR systems brought specific clinical benefits and was partly supportive of improved decision making. In addition, the majority of the doctors surveyed agreed that the EHR data can be useful and enables doctors to improve their decision making.

Referring back to our conceptualisation of Practice-based evidence, the approach identifies the need to utilise evidence drawn from clinical practice to make decisions about individual patients. In this approach, we identified electronic health records as being such evidence. Thus, based on the what we can infer from the results of the survey from this study, we acknowledged that there are several doctors that believe the electronic health records has the potential to be used in assisting with clinical decision making. While this number may not be representative of the whole population of doctors in Singapore, it still highlights the potential of EHRs. Therefore, we conclude that EHRs can potentially be utilised in our Practice-based evidence approach to

Chapter 6: Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making 157

decision making because of the information they capture and the findings that have been highlighted in the survey.

However, it is imperative that several limitations of this study are taken into consideration. Firstly, the analysis of the study is based on the final sample size of only 30 participants which is small in comparison to other studies. Nonetheless, as highlighted in Section 6.4.4, the differing doctors in terms of departments and healthcare organisations may provide a more comprehensive representation of the actual doctor population. Secondly, the survey responses may be influenced by the preamble and thus may have introduced biases. However, since participants are those with existing experience in using EHR systems, hopefully, they have not been negatively influenced. Additionally, some of the analysis only examined the opinions of doctors based on their input of being strongly agreeable. A possible limitation is, in not considering other aspects of doctors’ opinions. However, as highlighted, this strong opinion is taken as an indication of doctor’s confidence and conviction that their opinions are true and therefore carry considerable weight and value to be befittingly analysed. Therefore, based on these limitations that have been highlighted, the findings that we uncover may not be entirely accurate to generalise and claim as the opinions and viewpoints of all doctors in Singapore. Instead, it may probably only be true for this small population of participants. However, having raised that point, even though it is based on a small pool of participants, their opinions and viewpoints are still relevant in this study because it helped us to understand if EHR data can genuinely be used to assist in decision making.

158 Chapter 6: Eliciting Doctors’ Perceptions Regarding Benefits of EHR Systems and the Usefulness of EHR Data for Decision Making

Evaluating Our New Practice- Based Evidence Approach

“A truthful evaluation of yourself gives feedback for growth and success” - BRENDA JOHNSON PADGITT

This chapter discusses the evaluation of our Practice-based evidence approach. The chapter begins by highlighting the research study components, which consisted of developing a prototype decision support system, the preparation of the data source for the decision support system and the qualitative evaluation process. This is followed by the discussion and findings from the focus group evaluation.

7.1 OVERVIEW

In the previous chapters, our research began with the conceptualisation of a new Practice-based evidence approach to decision making. This is followed by the uncovering of existing ICT architecture of NUH and perception of doctors regarding the benefits of EHRs. The findings regarding the ICT architecture of NUH revealed the existence of a data warehouse and thus proposing only the need for a data mart design. The survey study eliciting the perception of doctors regarding the clinical benefits of using EHR systems and the usefulness of EHR data for clinical decision making has establish that EHRs are a suitable candidate as the source of evidence for our approach, having captured instances of actual clinical practices. This chapter continues with the final study to evaluate our PBE approach to decision making. The objective of this study is to evaluate qualitatively, our PBE approach using EHRs as the source of evidence of actual clinical practices, and a prototype clinical decision support system as the tool to assist healthcare professionals with decision making

The chapter begins with the outline of the steps taken to design the evaluation of our PBE approach, which is data preparation, a prototype clinical decision support system design and development, and a system evaluation process. The chapter ends with the discussion and findings from the evaluation.

The evaluation was a two-part process, which consisted of a field test followed by a focus group interview and discussion. The field test or system demonstration was

Chapter 7: Evaluating Our New Practice-Based Evidence Approach 159

necessary in describing how our PBE approach to decision making could be realised in a prototype clinical decision support system (CDSS) application. The prototype CDSS was also integral in demonstrating the value of EHRs.

As part of the prototype CDSS demonstration, the collaborating doctors from NUHS Regional Health Strategic (RHS) Planning Office provided anonymised patient records as the data source for the prototype CDSS. The prototype CDSS was able to simulate the routine clinical workflow usually encountered by doctors during patient consultation. To support doctors with assisted clinical decision making, analytics specific to the attended patient were generated by the prototype CDSS.

The motivation to conduct a focus group was influenced by the research aim, which is to evaluate based on the opinions and viewpoints of doctors, our PBE approach implemented in a clinical setting, as a solution that can assist healthcare professionals with clinical decision making. As a result, the research became more of a discovery than a comparative study. In that case, a qualitative method was considered most appropriate. A focus group, as evident by Kitzinger (1995, p. 299), is useful in discovering participants’ thought processes and why they think the way they do. It is also an efficient way for participants to be engaged in a group discussion, asking questions to, “explore and clarify their views” and generally share their thoughts and beliefs. Hence, employing the focus group discussion was vital as it allowed us to gain the understanding of the participants’ observations and views regarding the perceived effectiveness of our PBE approach to decision making, based on participants’ existing knowledge and experiences in using EHR systems.

The outcome of this final study will therefore conclude evaluation of our Practice-based evidence as an approach that is useful in assisting healthcare professionals to make well-informed decisions.

7.2 RESEARCH STUDY COMPONENTS

This chapter consists of four phases of the research study. The first phase involved the preparation of data for the clinical decision support system. The second phase was the design of the prototype CDSS. Thirdly, the development and implementation of the prototype CDSS in NUH and lastly, the evaluation of our PBE approach with collaborative doctors from NUH.

160 Chapter 7: Evaluating Our New Practice-Based Evidence Approach

As part of the evaluation process, NUH had provided us with an anonymised patient data set to be utilised in the prototype CDSS as a form of verifying our PBE approach.

Prototype PBE Approach Data Preparation Prototype Design Development Evaluation

Figure 40. Research study phase

7.3 DATA PREPARATION PHASE

As part of the collaboration with the doctors from National University Hospital, representing NUHS Regional Health Strategic (RHS) Planning Office, anonymised inpatient data set was provided for use with the prototype CDSS as part of the Practice- based evidence approach evaluation. In fact, no new health data or records were collected from patients during this phase of the research study.

The evaluation of our PBE approach had to be performed in NUH as the use and access to the anonymised data set was only provided in-situ (within the confines of NUH) as advised by the collaborating healthcare professionals (see attached Appendix F).

7.3.1 Data source The initial data set planned for the evaluation was supposed to consist of information such as patient demographics, hospital admission and discharge information, medication information and laboratory result information. The intended usage of that particular data set was to identify undiagnosed diabetics from the pool of data. However, the collaborating doctors provided us with data regarding patient demographics, and hospital admission and discharge information instead from the patient registration system, SAP. This required us to change the focus to supporting decision making related to inpatient scenario, such as hospital readmission or length of stay.

The data set provided for use in this phase of the study was NUH’s anonymised administrative inpatient data. It represented inpatient cases of patients, diagnosed with cardiovascular diseases, hypertension and diabetes, discharged from 2010 to 2014. It

Chapter 7: Evaluating Our New Practice-Based Evidence Approach 161

represented a subset of the whole administrative inpatient data set. However, it was difficult to calculate the proportion or percentage of records provided, in comparison to the whole data set available, as RHS were only privileged to data involving patients with chronic diseases such as cardiovascular diseases, diabetes and hypertension. According to national healthcare statistics provided by Ministry of Health Singapore, in 2014, the total hospital readmission rate by age and sex for all acute hospitals was 219,200 patients (Ministry of Health Singapore, 2014). This has increased to 246,900 by 2016 (Ministry of Health Singapore, 2016a).

The data set contained information regarding patient demographics, such as age, gender, nationality, marital status, race, religion, date of birth and residential status. It also featured the inpatient and discharge information. The inpatient and discharge information included patient movement type, ward number, medical department, ward category, patient class category, referral type, referral hospital name, admit reason, treatment category, diagnosis code, diagnosis description, DRG (Diagnosis Related Group) code, patient type, patient case number, case digit, admission date and time, discharge date and time, healthcare cluster and length of hospital stay.

7.3.2 Reviewing data The data set was provided in the form of a single Microsoft Excel file. The file contained 31 data fields or columns with 37,034 rows of entries and one row of data field header. As mentioned, each row entry corresponded to an inpatient case with discharged periods from 2010 to 2014. There were no unique identifiers to identify the number of patients that corresponded to the rows of entries from the data source. The data primarily contained non-identifiable patient demographics such as age, gender, nationality, marital status, residential status, race and religion, as well as hospitalisation admission and discharge information. Hospital admission and discharge information contained data such as patient discharge type, ward number, ward class, outpatient clinic visited, diagnosis code, treatment category and case number to name a few. A detailed list of the data field information and what it entails is provided in Appendix M.

Understanding the data fields required help and assistance from the collaborating doctors. Literature searches were also conducted to supplement further information to understand the nature of the hospitalisation data set, admin or billing data.

162 Chapter 7: Evaluating Our New Practice-Based Evidence Approach

Initial literature searches on hospital data resulted in several interesting findings. Among them were studies related to hospital length of stay and comorbidities. One study in particular, conducted by Freitas et al. (2012), inspired the direction for the application of our PBE approach due to the similarity in the data set used. In that study, the length of stay and length of stay outliers were important factors to consider, since these factors impacted the healthcare resource utilisation and costs (Freitas et al., 2012). However, the study only examined the factors associated with the incidence of both length of stay and length of stay outliers. It did not study how the factors could be used to assist in decision making especially for healthcare professionals in an actual clinical care setting.

Through drawing inspirations from the study by Freitas et al. (2012) and discussions with the collaborating doctors, our Practice-based evidence approach to decision making was to have the prototype CDSS provide analysed patient-centric statistics. As the nature of the data set provided to us was inpatient administrative data, so as a case study which could be useful in evaluating the application of our PBE approach to decision making, we also included prediction of patient’s probable length of hospital stay. Patient-centric statistics would provide an overall overview of patient’s medical conditions with respect to all other existing patient population. For example, given the patient’s medical condition, like the current admitted diagnosis, the statistics on the percentage of hospital readmission with similar admission diagnosis would provide the doctor with an idea of the readmission trends of the current patient and that of similar types of patients. Therefore, a higher readmission percentage would probably indicate to the doctor, the complexity of the condition and should prompt the doctor to consider a much more thorough examination. The prediction of the probable length of hospital stay could serve as a hint to doctors. The doctor could then consider alternative clinical interventions, such as ordering a laboratory test to obtain patients’ latest results, or introducing alternative treatments for patients, especially when the predicted length of stay appears to be higher than the recorded mean or median length of stay.

Improvising the PBE approach due to the nature of the data provided required the computation of additional variables, such as calculating the limit for high length of stay, comorbidity index and hospital readmission types, which were not available from the original data set provided.

Chapter 7: Evaluating Our New Practice-Based Evidence Approach 163

7.3.3 Importing data into MySQL As the inpatient administrative data, which contained a total of 37,034 entries, was provided as an Excel document, the data had to be imported into a database first for easier management before it could be used in the prototype CDSS as part of our PBE approach. In this study, we opted for the use of a MySQL database.

To begin the importing process into a MySQL database, a XAMPP (version 3.2.1) control panel application was used to run the Apache (2.4.10) web server and MySQL (5.6.20) database server. XAMPP is an easy-to-use, packaged tool that assists in the installation of an Apache web server, database server and PHP, a server-side scripting language for web development. The purpose of the Apache web server is to translate the PHP codes, facilitate the querying of SQL (structured query language) statements to MySQL server and display the outcome into web pages through a web browser.

Both the Apache server and MySQL server had to be started before any data could be imported into the database. Once the MySQL server was up and running, a web tool called phpMyAdmin was used to manage the administration of the MySQL database online. The phpMyAdmin tool was accessed directly using a web browser through the http://localhost/phpmyadmin link.

Figure 41. The phpMyAdmin tool used to store, retrieve and update data

A new database was created first before the entries from the Excel file could be loaded into the MySQL database. A database named hospital_data was then added

164 Chapter 7: Evaluating Our New Practice-Based Evidence Approach

using the phpMyAdmin tool. This was done by clicking on the ‘Database’ tab. The database label named ‘hospital_data’ was typed into the form and the ‘Create’ button clicked. Upon the successful creation of the hospital_data database, a table to store all the information from the Excel file needed to be added to the hospital_data database. A patient_record table with 31 fields corresponding to the headers provided in the data set was created. The full detail of the 31 field types created is illustrated in Appendix M.

However, due to the limitation in the maximum upload file size limitation that could be set in phpMyAdmin, the Excel file was divided into eight small individual files. First, the main Excel file was divided into five individual files according to the discharged year, from 2010 to 2014. The resulting files were named accordingly in the format of patient_disch_. After splitting the main data file according to discharged years, the files for discharged years 2012 to 2014 were still too large to be uploaded into phpMyAdmin for automatic importing. These files were subdivided further, as each file could not contain more than 5050 rows to ensure that it met the file upload size limit. After subdividing the files for 2012 to 2014 into smaller chunks, the final eight Excel files all met the upload size limit and were labelled as patient_disch_2010, patient_disch_2011, patient_disch_2012-1, patient_disch_2012- 2, patient_disch_2013-1, patient_disch_2013-2, patient_disch_2014-1 and patient_disch_2014-2. By then, all the eight files had rows less than 5050 entries. These files were then converted from the native Excel file (.xls) to a comma-separated file format (.csv), which was one of the supported file import formats by phpMyAdmin. Although the csv files would be larger in size compared to an xls file after the conversion, the csv files would not contain any hidden scripts, embedded Microsoft Office macros or XML tag headers that could interfere with the uploading process. Furthermore, a csv format also would not alter the content of the original data when being converted, ensuring that the data content was not edited or corrupted. Using the import function available in phpMyAdmin, all eight files were uploaded individually into the patient_record table.

7.3.4 Data Preparation Initial data preparation work was required before the data could be used in the prototype CDSS implementing our PBE approach. Based on the discussions with the collaborative doctors, we were able to identify the types of patient-centric statistics

Chapter 7: Evaluating Our New Practice-Based Evidence Approach 165

that could be provided in the prototype CDSS. The patient-centric statistics identified were (1) % readmission based on diagnosis code, (2) number of readmission cases based on age group, (3) number of cases with diagnosis codes registering high length of stay, (4) calculated mean length of stay days, (5) calculated median length of stay days, and (6) predicted length of stay. However, to be able to provide these statistics as well as the predictive modelling to the length of stay, new data fields had to be introduced, which required a certain level of calculation. The new fields that were introduced into the data set were (1) high length of stay indicator (HIGH_LOS), (2) hospital readmission types (READMISSION), (3) number of days before readmission (READMISSION_DAYS), and (4) Charlson Comorbidity Index (ICD_CCI). Additionally, new field identifiers were also introduced for the purpose of identifying unique patients and row numbers. The additional data fields were (1) IDN (indexing) and (2) PATIENT_ID (patient identifier).

The processes required in preparing the data set to be suitable for use in the prototype CDSS and our PBE approach were accomplished over several steps. These steps began by first identifying unique patients in the anonymised data set, classifying the data set according to three groups of diagnoses, identifying hospital readmission types, identifying high length of stay outliers and lastly calculating the Charlson Comorbidity Index value.

Identifying unique patient from the anonymised data set. An anonymised data set ensures that the data cannot be traced back to an actual person, nor can the data be re-identified back to any particular user. The anonymised data set provided in this study had no patient identifiers that could be used to re-identify an actual patient back. However, this posed a problem in identifying hospital readmission cases. Therefore, a new method had to be employed where we could identify unique patients while still maintaining the anonymity of the data set. The methodology that we adopted in overcoming this limitation was an approach similar to record linkage.

According to Winkler (2006), record linkage has been used to associate information from a variety of electronic files or, in most cases, applied to determine matching information when unique identifiers are non-existent. It functions by comparing and matching fields such as name, address and other non-unique identifiers between files to determine if the records corresponding to the fields could belong to

166 Chapter 7: Evaluating Our New Practice-Based Evidence Approach

the same person, company or business. Therefore, by considering each row of entry in the patient_record table as being different file sources, applying the record linkage approach would possibly enable the identification of unique patients within the data set.

Before we could begin the process of identifying unique patients, we needed to build a “patient profile” first. This “patient profile” refers to data fields from the anonymised data set which could be used to distinguish unique patients. These data fields, when used in combination, would then create a unique “patient profile”, enabling the grouping of inpatient cases belonging to that particular patient. In a multicultural society such as Singapore, race identifiers could be utilised as one of the data fields since they are generally and commonly used to represent the cultural diversity of Singapore’s population. Therefore, the finalised data fields identified were (1) DATE_BIR (date of birth), (2) SEX (sex or gender), (3) RACE (racial profile), (4) RELIGION, (5) NATIONALITY, (6) NON_RESID_STATUS (residential status) and (7) MARITAL_STATUS (marriage status). The data fields corresponding to the date of birth, sex, race, nationality and residential status did not have missing values, which were good candidates to be used in identifying unique patients. Religion and marital status data fields, however, contained some missing entries. The data fields which had missing entries or values were replaced with ‘Unknown’ status instead.

1. UPDATE patient_record SET RELIGION = 'Unknown' WHERE RELIGION = ''; 2. UPDATE patient_record SET MARITAL_STATUS = 'Unknown' WHERE MARITAL_STATUS = '';

Code Snippet 1: Replacing missing items with ‘Unknown’ status

Two new data fields were added to patient_record table. The two data fields were (1) IDN and (2) PATIENT_ID. IDN was used to represent the row entry number for the purpose of faster access and retrieval. An SQL query was executed to add the ‘IDN’ data field as a primary key into the patient_record table with type integer. The SQL query was executed in phpMyAdmin. The query is illustrated below.

1. ALTER TABLE patient_record ADD `IDN` INT NOT NULL AUTO_INCREMENT PRIMARY KEY FIRST;

Code Snippet 2: Adding new ‘IDN’ data field

Chapter 7: Evaluating Our New Practice-Based Evidence Approach 167

A five-digit number was used in the data field PATIENT_ID to identify unique patients. The PATIENT_ID was integral because it enabled the grouping of inpatient cases that belonged to a specific individual or patient. The five-digit number was opted for, as the data set only contained slightly less than 40,000 entries and therefore would not require an identifier of more than 5 digits. An SQL statement was first created to add a new column in the patient_record table with the column name PATIENT_ID (Code Snippet 3). A PHP function was then developed to identify unique patients from the data set and generate the unique identification number for patients. The pseudocode for the function is illustrated below in Code Snippet 4.

1. ALTER TABLE `patient_record` ADD `PATIENT_ID` INT(11) NOT NULL AFTER `IDN`;

Code Snippet 3: Adding new ‘PATIENT_ID’ data field

1. Set patient information as date of birth, sex, race, religion, nationality, resident status & marital status 2. Select row of patient information from patient database 3. 4. while select results is not empty 5. is patient information already exists in patient identity database 6. if Yes 7. skip 8. if No 9. generate new patient id 10. insert patient information and new patient id into patient identity database 11. 12. Set variable patient id as new patient id 13. Select row from patient database matching patient information 14. while select results is not empty 15. if patient id in row is not empty 16. update row with new patient id 17. else 18. skip

Code Snippet 4: Pseudocode to identify unique patients

First, a patient_identifier table was created to store the unique patient profiles. Each patient profile was then provided with a unique patient identification number. After executing the function that matched profiles based on the combination of data fields (DATE_BIR, SEX, RACE, RELIGION, NATIONALITY, NON_RESID_STATUS and MARITAL_STATUS), a total of 21,886 unique patients were identified from the data set. Next, entries from patient_record were examined to

168 Chapter 7: Evaluating Our New Practice-Based Evidence Approach

locate those that matched with the patient profile defined in patient_identifier table. The matched profiles in patient_record were then updated with the corresponding patient identification number. From then, inpatient cases with the same patient identification number now belonged to the same patient.

The data set with the updated patient identification number was further scrutinised to check for accuracy in identifying unique patients. This checking was performed using the admission (ADATE) and discharged (DDATE) date. A function was developed to calculate the difference in the number of days between the last discharged date and the next admission date for all patients’ cases. If the number of days calculated returned a negative value, it meant that the patient was admitted before being discharged. In terms of hospitalisation process or workflow, this was not possible. Therefore, all inpatient cases that were identified to belong to unique patients and calculated to have registered a negative value were not considered in the final data set. Due to the lack in the breadth of patient personal information, such as a home address or postal code, it was possible that during the initial process to identify unique patients, the process might have identified inpatient cases that belonged to different patients, who happened to have the same patient profile. The following was how we performed the checking process.

As highlighted, the data set provided was anonymised, and therefore, it was not possible to identify information or inpatient cases that belonged to the correct individual patients to begin. For this reason, we needed to perform an accuracy check after assigning the patient identification numbers. The accuracy check process started with the creation of two new data fields labelled as ADM_DATE and DIS_DATE. These data fields were created with the date format as ‘YYYY/MM/DD’ to calculate the difference in the number of days between the last discharged date (DIS_DATE) and the next admission date (ADM_DATE). The initial admission and discharge data fields were in a different format than the one to be used in the accuracy check function. The corresponding ADM_DATE and DIS_DATE were extracted from the provided ADATE and DDATE data fields. The date conversion was performed using R. As highlighted earlier, we made use of the admission and discharge date as the accuracy check process with the assumption that a patient cannot be newly admitted into a hospital ward if the patient has not been discharged yet. Therefore, a negative value calculated meant that there was a possibility that this particular patient was assigned

Chapter 7: Evaluating Our New Practice-Based Evidence Approach 169

incorrectly. This was the other limitation identified in this approach to identify unique patients from the anonymised data set. Hence, for all cases where the difference calculated was negative, we discarded such cases entirely from the data set.

1. ALTER TABLE `patient_record` ADD `ADM_DATE` VARCHAR(20) NOT NULL AFTER `ADATE`; 2. ALTER TABLE `patient_record` ADD `DIS_DATE` VARCHAR(20) NOT NULL AFTER `DDATE`;

Code Snippet 5: SQL statements to add new data fields

1. #clear workspace 2. rm(list = ls()) 3. memory.limit(900000) 4. library(RMySQL) 5. 6. #connecting to mysql server 7. mydb = dbConnect(MySQL(), user = 'root', password = '', dbname = 'hospital_data', host = 'localhost') 8. 9. # getting the training sets 10. patient_rs = dbSendQuery(mydb, 'SELECT * FROM patient_record') 11. patient = fetch(patient_rs, n=-1) 12. dbClearResult(patient_rs) 13. 14. patient$ADM_DATE <- as.Date(patient$ADATE, format = "%d/%m/%Y") 15. patient$DIS_DATE <- as.Date(patient$DDATE, format = "%d/%m/%Y") 16. 17. for (x in 1:nrow(patient)) { 18. sql<- sprintf("UPDATE patient_record SET ADM_DATE= '%s', DIS_DATE = '%s' WHERE IDN= %d", pati ent[x,28], patient[x,30], patient[x,1]) 19. print(sql) 20. dbSendQuery(mydb, sql) 21. }

Code Snippet 6: Reformatting date to calculate the difference in days between discharged and next admission date

A total of 442 cases were returned with negative calculated values. The number of patients that were associated with these negative values was 115. In total, 3090 cases from the identified 115 patients were discarded from the data set as they did not meet the accuracy criteria. As a result, 33,944 rows of data remained. Based on the identifier created to identify unique patients, the remaining rows of data corresponded to a total of 21,792 patients.

170 Chapter 7: Evaluating Our New Practice-Based Evidence Approach

Classifying data set into 3 groups The remaining 33,944 rows relates to the number of inpatient cases of patients diagnosed with cardiovascular diseases (CVD), hypertension (HTN) and diabetes (DM) coded in ICD-9 and ICD-10 codes. ICD codes refer to the International Classification of Diseases code which defines and categorises health and medically related conditions such as diseases and disorders (World Health Organization, 2016). In healthcare organisations, ICD codes are used to classify clinical diagnosis of patients, such as ‘E119’ to refer to ‘Type 2 diabetes mellitus without complication’ or ‘E1164’ for ‘Type 2 diabetes mellitus with hypoglycaemia’ (Abdullah et al., 2008). With the collaborative doctors being specialists in the management of diabetes, the evaluation of our Practice-based evidence approach narrowed to focus only on diabetes-related inpatient cases. However, the remaining 33,944 cases were classified according to the three different types of diagnosis mentioned above, coded using the admitting diagnosis description, HOSP_MAIN_DIAG_DESC data field. Therefore, the data set needed to be reclassified again so that all admitting diagnosis code description referring to the parent classes of diabetes, hypertension or cardiovascular diseases could be grouped together. To do that, a GROUP_DIAG data field was created. Codes with descriptions containing ‘diabetes’ were grouped as ‘DM’ in the GROUP_DIAG data field. Accordingly, hypertension cases were coded as ‘HTN’ and cardiovascular diseases as ‘CVD’. A total of 5881 cases were then identified and coded with ‘DM’ to indicate those diagnosed with diabetes.

1. ALTER TABLE `patient_record` ADD `GROUP_DIAG` VARCHAR(50) NOT NULL ; 2. UPDATE patient_record SET GROUP_DIAG = 'DM' WHERE HOSP_MAIN_DIAG_DESC LIKE '%diabete%';

Code Snippet 7. SQL statement to add new data fields and group cases according to ‘DM’

Identifying hospital readmission cases The next process, to prepare the data set for patient-centric statistics and predictive modelling, was to identify hospital readmission cases. Hospital readmissions were considered based on 30 days, 60 days or 90 days period before a patient was readmitted back to the hospital for a recurring condition. It was one of the chosen patient-centric statistics, as a hospital readmission episode was said to cause distress to patients and increase the cost of healthcare systems worldwide (Swain & Kharrazi, 2015). Besides, the evidence of hospital readmission might also indicate

Chapter 7: Evaluating Our New Practice-Based Evidence Approach 171

patient frailty, the progression of chronic diseases, reaction to drug events, being prematurely discharged and even complications or injuries from surgical procedures (Benbassat & Taragin, 2000; Swain & Kharrazi, 2015). Studies have also shown that there is an association between the readmission rates and the inappropriate care provided during hospitalisation, where one in seven readmissions for patients with diabetes were attributed to substandard care (Benbassat & Taragin, 2000). For whatever reasons, it is crucial that hospital readmissions be reduced or prevented from occurring as this is considered both a cost and patient safety issue.

From the remaining 5881 cases involving diabetic patients, we were able to calculate the different readmission types. According to the NUH guideline on calculating readmission rates, cases where hospital discharges were termed as “Death” (130), “Discharged against Advice” (122), “Transferred to Singhealth hospital” (4), “Transferred to NHG Hospital” (7), “Transferred to private hospital” (6) and “Absconded” (5) were not considered for the hospital readmission calculation. Hence, in total, 274 cases involving the above discharge types were discarded from the data set. This made up about 4.67% of the total diabetic cases.

For the remaining 5607 cases (3553 unique patients), a PHP function was created to calculate the four different types of readmission, (1) 30 days readmission, (2) 60 days readmission, (3) 90 days readmission and (4) 120 days readmission. According to Benbassat and Taragin (2000), the cause for hospital readmission can be accounted to the condition of chronic diseases among patients. It is an indication that a patient may not be getting better if the readmission rate is high. Additionally, it can also mean that a healthcare professional may have erred on something during the consultation, such as prescribing wrong or ineffective medication or missing a crucial treatment.

The readmission was calculated based on the number of days between the last discharged date to the next readmission date. According to Fan and Sarfarazi (2014) there are three methodologies that can be adapted to calculate the hospital readmission rates. The three different solutions or methodologies are Data Step (Vertically), Data Step (Horizontally) and PROC SQL SELF-JOIN. The data set provided by NUH contained patient visits from unique patients (identified through the previous process) that were coded with unique case numbers (a combination of CASE_NO and CASE_DIGIT). Each case number had its corresponding admission and discharge dates. Using the unique patient identification number to extract all the case numbers

172 Chapter 7: Evaluating Our New Practice-Based Evidence Approach

belonging to individual patients, the resulting corresponding admission and discharge dates for every patient were identified. Following the Data Step (Horizontal) approach suggested by Fan and Sarfarazi (2014), the number of days before a patient was readmitted were calculated. However, we did not adopt the use of a SAS solution tool as suggested in the study; instead, we made use of both the MySQL database and PHP.

From the 5607 remaining cases, 1240 cases were identified as hospital readmission episodes. A total of 575 cases corresponded to a 30 days readmission, 291 for a 60 days readmission, 210 for 90 days readmission and 164 for 120 days readmission. Two data fields were created, (1) READMISSION to store the type of hospital readmission episodes and (2) READMISSION_DAY to indicate the number of days before being readmitted after discharged.

1. ALTER TABLE `patient_record` ADD `READMISSION` INT(11) NOT NULL; 2. ALTER TABLE `patient_record` ADD `READMISSION_DAY` INT(4) NOT NULL;

Code Snippet 8. Adding 2 data fields, READMISSION AND READMISSION_DAYS

High length of stay (LOS) outliers In statistics, outliers are considered as anomalies; readings which are out of the norm or believed to be erroneous. High outliers in the length of stay (LOS) in hospital admission however reveal the increased need for healthcare resources and ultimately, higher expenditure. A way to improve quality of service can be achieved if factors associated with high outliers of length of stay can be identified. These factors can be addressed and managed to reduce the probability of future likelihood of an outlier.

To identify whether a hospital stay was considered as high outliers, we calculated the geometric mean (gm) and two standard deviations (sd) of the length of stay from the remaining 5607 cases as suggested by Freitas et al. (2012). According to Freitas et al. (2012), the method of using geometric mean and standard deviation was highly useful when applying it to LOS, as a measure of defining extreme costs. However, it also represents a good indicator for healthcare professionals to have, especially when making decisions about patient care. Furthermore, LOS can probably be attributed to the provision of inappropriate treatment to patients, as it has been extensively studied alongside the quality of care provided (Tickoo et al.). The calculation was performed in RStudio using the R language. R “psych” package was installed to calculate the geometric mean and standard deviation. The geometric mean was estimated to be

Chapter 7: Evaluating Our New Practice-Based Evidence Approach 173

4.744 days and the standard deviation was 15.33 days. Therefore, the limit of high LOS calculated was 36 days. As a result, inpatient cases that registered length of stay 36 days and above were considered as high LOS outliers.

= ( ) + 2 ( ) = 4.744 + 2 (15.33) = 35.404

Code Snippet𝐻𝐻𝐻𝐻𝐻𝐻ℎ 9𝐿𝐿𝐿𝐿𝐿𝐿. Calculating𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙𝑙 the𝑔𝑔𝑔𝑔 limit indicating∗ 𝑠𝑠𝑠𝑠 a high LOS outliers∗ or limit 𝑑𝑑𝑑𝑑𝑑𝑑𝑑𝑑

A data field labelled HIGH_LOS was added to the data set to indicate cases that corresponded to high LOS. Any length of stay of 36 days and beyond was indicated with a value of 1 and any cases below the limit were indicated with a value of 0. In the remaining 5607 inpatient cases, 246 cases of high LOS were identified. This made up to be about 31.1% of the total inpatient cases as high LOS outliers. The mean LOS for outliers was calculated to be 62.3 days and the median was 51 days. When compared to all length of stay which were non-outliers, the mean was 6.3 days and the median 4 days.

1. ALTER TABLE `patient_record` ADD `HIGH_LOS` INT(11) NOT NULL;

Code Snippet 10. Adding HIGH_LOS column

Proportion of Outliers and Non-outliers LOS (n=5607) Outliers Non-outliers Cases 246 5361 Mean (days) 62.3 6.3 Median (days) 51 4 Sum length of stay (days) 15,315 33,897 Proportion to total length of stay (%) 31.1 68.9

Table 19. Table of the proportion of high LOS outliers and non-outliers

Calculating the Charlson Comorbidity Index (CCI) Numerous studies have used the Charlson Comorbidity Index (CCI) to determine the presence or absence of comorbidity (Charlson et al., 1987; Elixhauser et al., 1998; Falsetti et al., 2016; Roffman et al., 2016). CCI is one of the most commonly used indexes for calculating patients’ comorbidity when using information from administrative data (Falsetti et al., 2016). While the index is still being used to predict the mortality of patients (Huang et al., 2014; Laor et al., 2016), currently, it has a much

174 Chapter 7: Evaluating Our New Practice-Based Evidence Approach

wider application, such as to predict the outcomes of in-hospital death (Falsetti et al., 2016), hospitalisation charges (Charlson et al., 2014) and length of stay (Freitas et al., 2012). The CCI value of the remaining cases was calculated based on the ICD-9 and ICD-10 codes found in the HOSP_MAIN_DIAG_CODE data field, which corresponded to the patient’s admission diagnosis. Using Wasey (2016) R ‘icd’ package, the calculation of the CCI value was a two-step process. Firstly, a data frame was created in R, which contained the admission ICD codes. The function ‘icd10_comorbid_quan_deyo’, was then used to calculate the values based on the ICD codes. In this function, it calculated the existence of 17 types of comorbidities conditions.

1. # get the patient_icd values 2. patient_icd_deyo <- icd10_comorbid_quan_deyo(patient_icd) 3. write.csv(patient_icd_deyo, "C:/Users/hosop/Documents/Datasets/patient_icd_deyo.csv") 4. patient_icd_values <- data.frame(icd_charlson_from_comorbid(patient_icd_deyo))

Code Snippet 11: icd10_comorbid_quan_deyo

An illustration of the list of 17 comorbidities that is linked to ICD-10 codes can be viewed in Appendix P. The outcome from the function (icd10_comorbid_quan_deyo) was a data frame which listed the Boolean results of each of the 17 comorbidities. The first 10 rows of results is shown in Table 20.

Result of ‘icd10_comorbid_quan_deyo’ function

MI DM HIV PUD PVD CHF Mets Renal DMcx Stroke Cancer Paralysis Dementia LiverMild Rheumatic LiverSever Pulmonary

FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE

FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE

FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE

FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE

FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE

FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE

FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE

FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE

FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE

FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE

Table 20. First 10 rows of result from the application of function on the anonymised data set

For step 2 of the process, ‘icd_charlson_from_comorbid’ function was applied on the data frame containing the outcomes from the earlier process. The function

Chapter 7: Evaluating Our New Practice-Based Evidence Approach 175

calculated the result in absolute numbers, corresponding to the Charlson Comorbidity Index value. The resulting data frame was converted into a csv file format. An R function was created to read the values from the csv file and updated the corresponding ICD_CCI values into the patient_record table. Table 21 is the example of the outcome of the application of the second function.

Calculated CCI Values IDN icd_charlson_from_comorbid.patient_icd_deyo 8230 2 8235 2 8242 1 8250 2 8252 2 8281 2 8287 2 8297 2 8307 1

Table 21. An example of the first 10 rows of ICD_CCI values

The SQL statement used to create an additional column to store the corresponding Charlson Comorbidity Index value is illustrated below.

1. ALTER TABLE `patient_record` ADD `ICD_CCI` INT(11) NOT NULL;

Code Snippet 12. Creating ICD_CCI column in data set

7.3.5 Summary of finalised data The initial data set was provided in a single Excel document with 31 data fields and 37,304 unique inpatient cases with a primary diagnosis of cardiovascular disease, diabetes and hypertension. The diagnoses were coded using either an ICD-9 or ICD- 10 codes in HOSP_MAIN_DIAG_CODE data field. The corresponding description of each diagnosis code was provided in HOSP_MAIN_DIAG_DESC data field.

To enable the calculation of hospital readmission cases, the anonymised data set needed to be updated with an additional patient identification number field. A data field called PATIENT_ID was created, which held the patient identification number

176 Chapter 7: Evaluating Our New Practice-Based Evidence Approach

to group unique patients. A row identifier, IDN, was also created for efficient search and retrieval purposes. A check on the accuracy of the patient identification process recorded 3090 cases that did not pass the accuracy check. These cases were removed entirely from the data set, bringing the total number of eligible cases to 33,944.

The remaining 33,944 cases were grouped into three classes of diagnosis: diabetes (DM), cardiovascular diseases (CVD) and hypertension (HTN). With our focus only on diabetic patients, 5881 cases remained. Following the 5881 cases, we identified the different readmission episodes. The number of inpatient cases that did not meet NUH’s criteria for hospital readmission cases was 274. These cases were discarded leaving a final total of 5607 cases. Out of 5607 cases, only 1240 were identified as having hospital readmission episodes.

Next, the high length of stay outliers was calculated and 246 cases were identified as having high LOS. Finally, the remaining 5607 cases had the Charlson Comorbidity Index values calculated using the R package available and their corresponding values updated in the data set.

7.3.6 Verifying PBE data warehouse architecture with the finalised data set In Chapter 5, we successfully envisioned the ICT architecture of NUH and discovered the existence of a data warehouse architecture. With the discovery of the presence of an enterprise data warehouse, this meant that a new data warehouse architecture was not required. Instead, we needed to design a data mart that would enable us to extract the needed data from this data warehouse. However, at this stage of the research collaboration, we were not provided access to the data warehouse. Therefore, to verify this data warehouse architecture works, we performed a data warehouse simulation instead. The simulation was necessary, as it was the basis of our PBE approach architecture in integrating multiple data sources into one becoming the source of data required in the prototype clinical decision support system. Based on the findings from interviews conducted with IT professionals from IHiS in the first research study, we discovered that the ODS (one of the two data warehouses in NUH) was primarily used to store patient registration, administrative and billing data. This information was what contained in the data set provided to us by the collaborating doctors from NUH. Ideally, it would have been preferred if we could also perform the integration of the multiple data sources into the data warehouse but that was unfortunately not possible. With that above finding, we made the assumption that since

Chapter 7: Evaluating Our New Practice-Based Evidence Approach 177

the anonymised data set provided, contained data that was also described to similarly contain in the ODS, the anonymised data set provided must have been from the data warehouse source. This was further confirmed by the collaborating doctors. Hence, the next step was the data warehouse simulation and the design of a data mart to extract the required information from the data set provided.

Simulating data warehouse with a data mart To simulate the data warehouse, a star-schema data mart was designed with the aim of providing data required for generating patient-centric statistics in the prototype CDSS. Data marts represent the subset of the data in a data warehouse and are used to optimise quick access to the data for analysis. In this process, the data marts were used to extract the data from the data set provided to be analysed for the patient-centric analysis.

In this simulation process, a tool called Pentaho Community Edition (CE) Data- Integration software (version 6.1) (Pentaho, 2016) was utilised. A star-schema data mart, as illustrated in Figure 42, was designed with the purpose of extracting the necessary data to generate patient-centric statistics. The star-schema data mart design consisted of a fact table, with each entry in the table corresponding to a particular dimension. In the design below, the fact table was linked to eight different dimension tables: adate_dim, age_dim, case_dim, ddate_dim, hosp_diag_dim, los_dim, patient_dim and readmission_dim.

Based on this data mart design, the corresponding extraction process was performed using Pentaho. Pentaho allowed for a graphical user interface interaction to perform data warehouse-related activities. Icons that performed specific data extraction and transformation process were dragged into the workspace. Settings within each icon were edited to perform the desired processes.

The fact_record table and each dimension tables were set up to extract the data from the data warehouse sequentially. The new entry into the fact_record table could only be inserted, once all the dimension tables were inserted with their respective entries as the fact_record table required the primary key values from each dimension table. This process however, was made simple using Pentaho by allowing the dependent process to be completed first before proceeding to update respective tables.

178 Chapter 7: Evaluating Our New Practice-Based Evidence Approach

Once extraction and insertion processes were completed, a green tick icon represented the successful execution of that process. Once all the processes were successful and completed, the data mart was updated with the associated information and was ready for analysis. To test the result, the following patient-centric statistics were executed using SQL queries.

Figure 42. Star-schema data mart

Example: Calculating % readmission based on diagnosis code A sample SQL statement was created to query the number of readmissions associated with the diagnosis code ‘E1422’. The query returned an integer value and the percentage of readmission was calculated based on the returned value divided by the total number of cases.

1. SELECT COUNT(*) FROM fact_record, readmission_dim WHERE readmission_dim.readmission > 0 AND fact_record.readmission_key = readmission_dim.readmission_key AND hosp_main_diag_c ode = 'E1422';

Code Snippet 13. Querying from data marts

Chapter 7: Evaluating Our New Practice-Based Evidence Approach 179

To verify that the extraction process using data marts was accurate, an SQL statement was created to query for the number of readmissions associated with the diagnosis code ‘E1422’, from the finalised data set provided by NUH.

1. SELECT count(*) FROM patient_record WHERE readmission > 0 AND hosp_main_diag_code = 'E1 422';

Code Snippet 14. Querying from the finalised main data set

Both SQL statements returned the same result, a value of 268 relating to the number of readmissions, which indicated that the simulation of the data warehousing using star-schema data marts was indeed accurate.

However, for calculating the prediction model of probable length of stay, the data schema used was different from the star-schema data mart design. Instead, it resembled a flat table view that joined the dimension tables and the fact table together. This was necessary to facilitate the use of a regression model to calculate the predicted length of stay.

180 Chapter 7: Evaluating Our New Practice-Based Evidence Approach

Figure 43. Extraction process based on data mart design using Pentaho CE Data Integration

Chapter 7: Evaluating Our New Practice-Based Evidence Approach 181

7.4 PROTOTYPE CDSS DESIGN & DEVELOPMENT

With the completion of the data preparation, the design and development of a Clinical Decision Support System implementing our Practice-based evidence approach were initiated. The prototype CDSS needed to be useful and user-friendly enough when evaluated and at the same time able to assist the collaborating doctors with decision making during the evaluation process.

The feasibility of the prototype clinical decision support system could be influenced by the system’s user-friendliness or ease of use. This bias may then affect the evaluation outcome of the PBE approach to assist in decision making. In order for the prototype to be reasonably effective, the design of the prototype clinical decision support system adopted a user-centered design methodology.

This section of the research study described the development phase of a web- based clinical decision support system implementing our approach of Practice-based evidence. The prototype development strictly adhered to the Software Development Life Cycle (SDLC) methodology, following the fundamental phases of planning, analysis, design and implementation (Balaji & Murugaiyan, 2012). Several models of the SDLC have been created based on these fundamental phases, such as the well- known waterfall model and others like the spiral, rapid prototyping, incremental and agile software development model. However, an important aspect of software development, which does not currently fall within the above fundamental phases, is usability. Usability is essential, as designing and developing user-friendly and easy- to-use systems are critical for their continued use.

This prototype design phase thus adopted the proposed framework from the works of Chamberlain et al. (2006), which integrated the Agile development methodology with User-centered design. In this case, Scenario-based design (SBD) was the design approach adopted. This research study, however, did not intend to include a full-scale development of a decision support system; instead it was solely focused on developing the components that implemented our Practice-based evidence approach to decision making. This approach was defined by the ability to assist decision making by providing patient-centric statistics. The design of the prototype CDSS was described in terms of user scenarios. A series of activities was conducted during the process of creating the prototype system. The process included a mock-up

182 Chapter 7: Evaluating Our New Practice-Based Evidence Approach

design using wireframes, the evaluation of wireframes, prototype development and implementation.

7.4.1 Requirements Analysis (Planning phase) Root concept The root concept in SBD is an initial analysis of the requirements of the system. It is supposed to contain the system’s vision and rationale, initial review of the stakeholders and the system’s assumptions as based on Rosson and Carroll’s work (Rosson & Carroll, 2009). The root concept developed in this phase is depicted in Table 22. Stakeholders represent the organisation or person who would be positively or negatively affected by the system (Checkland, 1981; Muller, 1991 as cited in Rosson & Carroll, 2009). In this study, the stakeholders were the healthcare professionals who would be utilising the prototype CDSS for assistance in making decisions.

Component Contributions to root concept High-level vision A decision support system that is able to assist with clinical decision making which healthcare professionals can trust and is easy to use. Basic rationale There are various clinical decision-support systems available but none which implements the approach of Practice-based evidence and is patient-centred.

Stakeholders: Healthcare professionals Healthcare professionals who manage diabetic patients in their (end-users) clinics. The system provides patient-centred statistics and prediction which prompts them to make well-informed decisions.

Starting assumptions Users have experience in using organisations’ EHR system. Users have the knowledge of how decision support system works. Interview users for requirements Table 22. Root concept of the proposed clinical decision support system

Stakeholders analysis The stakeholder analysis summarised the stakeholders’ general characteristics (Rosson & Carroll, 2002). The following table illustrates the profile of the stakeholders.

Stakeholders General Group Characteristics Healthcare Professionals Background: Clinical experts on managing patients with diabetes. They would want the system to display statistics that are meaningful and assist them in decision making such as in deciding whether to admit or not admit patients to ward.

Expectations: Easy to view and understand statistics displayed. Able to be prompted to make a decision. Table 23. Stakeholder profiles

Chapter 7: Evaluating Our New Practice-Based Evidence Approach 183

Use Case The use case developed focused on the doctor persona, which seeks to decide whether a patient needs prior clinical or medical intervention before being admitted. The decision support system assists in providing statistical information and prediction modelling based on the analytics of the health records.

Use Case Primary Actor: Doctor is deciding on whether a patient needs to be admitted. Goal in context: Doctor wants to find out if patient admission is required or another intervention is preferred. Scope: Supports patient-centred analytics and prediction of probable length of hospital stay. Level: User goal Stakeholders & Doctors are afforded with patient-related information and Interests: analytics that can increase knowledge about the patient. Trigger: Doctor selects the patient for consultation. Precondition: Doctor is logged-in and has access to hospital’s EHR system. Minimal guarantee: Doctor is provided with patient health and medical information. Success guarantee: Doctor is provided with patient-centred analytics and prediction of probable length of stay to make decision about admission.

Main success scenario 1. User : Browse patient brief information and clicks on the “Select” button to select a patient. 2. System : Finds related patient information and displays according to information type; personal, medical and historical. Performs patient-centred analytics and prediction of length of stay. 3. User : Browse through related information, statistics and prediction model. Alerted to statistics, which are highlighted and prompted to engage in the further thought process. User clicks on ‘Admit Patient’ for hospital admission. 4. System : Admission information captured and displays discharge information in a new page. 5. User : Clicks on ‘Continue’ to discharge the patient 6. System : Discharge information is captured and performs a new analysis with updated information. Prediction is remodelled with the addition of the new discharge information. 7. User : Selects a patient by clicking on ‘Select’ 8. System : Finds related patient information and displays according to information type; personal, medical and historical. Performs new patient-centred analytics and prediction of length of stay based on the addition of new discharge information. 9. User : Continues processing until no new patients.

Table 24. Use case

184 Chapter 7: Evaluating Our New Practice-Based Evidence Approach

User Scenarios From the use case developed above, user scenarios were developed to demonstrate the system interaction at a technical level from the viewpoint of the user. It also allowed for the identification of design possibilities for the system. The user scenario created is illustrated below.

Scenario Dr Tan works at the National University Hospital Diabetes Clinic. His daily duties, among others, include attending to patients to review their diabetes conditions, prescribing medication, testing their blood sugar level or admitting patients to hospital wards for further observation or treatment.

The Diabetes Clinic has installed a new decision support system, which is aimed at helping doctors make well-informed decisions. In particular, the decision support system provides patient-centred statistics and the prediction of probable length of hospital stay to help prompt and engage doctors in stimulating further thought processes as to improve their decisions. Dr Tan has been tasked to use the new system as part of his daily routine in the Diabetes Clinic to improve his decision making and the delivery of care to his patients.

Dr Tan starts the decision support system by clicking the system icon. The system opens a web browser and loads the main page that lists the patients he will be attending to for the day. The main content section of the system is separated into two panels, one on the left and the other on the right. The left panel displays the list of patients while the right panel shows the statistics about the collection of analysed health records.

Dr Tan selects the patient he is attending to, by clicking on the ‘Select Patient’ button beside each patient’s information. To prevent him from selecting the wrong patient, the ‘Select Patient’ button changes to dark green when hovered. This informs Dr Tan that this will be the selected patient from the rows of patients.

When the ‘Select Patient’ button is clicked, the system redirects to a new page with similar layout design and loads the selected patient information. On the left panel, the system displays the “Loading Patient Information” message as it goes through the loading process. When all the information regarding the selected patient is completely loaded, the message ends and the page displays detailed patient information. At the top of the left panel, the system shows the Case Number to indicate a unique consultation session with the patient. The system further displays three tables of information regarding the patient for Dr Tan to view. The first table shows the personal information of the patient labelled ‘Patient Personal Information’, the second table displays the medical information labelled ‘Patient

Chapter 7: Evaluating Our New Practice-Based Evidence Approach 185

Medical Information’ and the last table shows patient’s past hospital admission information labelled as ‘Patient Historical Records’.

On the right panel, the system displays the “Loading Stats & Analysis” message. Once the statistics and analysis information are fully loaded on the page, the message disappears and displays corresponding information. The right panel is divided into two columns to display the statistics. Each statistical value is displayed in containers or widgets. On the left column, the system displays the following statistics. (1) Percentage cases of 30- day hospital readmission based on admission diagnosis codes, (2) number of readmission cases based on age group, (3) mean length of stay and (4) median length of stay. On the right column, the system displays (1) limit to high length of stay, (2) recorded highest length of stay, (3) number of cases recording high length of stay based on admission diagnosis code and (4) predicted probable length of stay.

The system assists Dr Tan with decision making by changing the colour of each container to red when each statistic is beyond its prescribed limit. The system also predicts the probable length of stay for the selected patient based on patient collective information and the collection of analysed health records.

To engage our Practice-based evidence approach, Dr Tan discharges the selected patient by clicking on ‘Discharge Patient’ button. The system updates the decision support system by recalculating the statistics and running the analytics. The system returns to the main page and displays an increased reading in the number of cases analysed to indicate the completion of the analysis.

Table 25. Development of a user scenario

186 Chapter 7: Evaluating Our New Practice-Based Evidence Approach

Requirements specifications Based on the use case and user scenario, the functional requirements of the prototype decision support system were identified. The requirements are tabulated in Table 26.

Functional Requirements 1. Main page should provide the list of patients who must be consulted. 2. Provide feedback to user when button is hovered. 3. Provide feedback to user when button is clicked. 4. Select patient for consultation. 5. Provide feedback to user that the patient selection is made. 6. Feedback to user that loading of patient information and statistical analysis are in progress. 7. Enable statistics to show (1) percentage cases of 30 day readmission based on admission diagnosis codes, (2) number of readmission cases based on age group, (3) mean length of stay, (4) limit to high length of stay, (5) recorded highest length of stay, (6) number of cases recording high length of stay based on admission diagnosis code and (7) predicted probable length of stay. 8. Enable change in colour to red when statistics reach limits. 9. Enable patient to be admitted. 10. Provide feedback when patient is admitted. 11. Enable patient to be discharged. 12. Provide feedback when patient is discharged. 13. Enable system to recalculate statistics and analytics. 14. Provide feedback that system is updated with new statistics.

Table 26. List of functional requirements

7.4.2 Wireframes as mock-up design (User interface) A mock-up of the decision support system was created using the Pencil Project tool. Pencil is an open-sourced GUI prototyping tool that is available for all platform users (Evolus, 2016). In this initial design process, wireframes were created as visual mock-ups before any development work could take place. Hence, wireframing was an important activity in the design of any information system as it involved both developers and end users to agree on the final design layout and how the design would meet the functionalities and the requirements of the system. In order to reduce the need for end users to be trained in using the prototype system, the layout design mimicked the design of the Allscripts Sunrise Clinical Manager (SCM) system, which is similar to the CPSS2 system used by NUH healthcare professionals. We had prior experience

Chapter 7: Evaluating Our New Practice-Based Evidence Approach 187

using and managing SCM while working as a Senior Systems Analyst with IHiS in one of the public hospitals in Singapore. Furthermore, the design had to follow that of the SCM system as no CPSS2 documentation was available and, access to the system was not provided as part of the collaboration.

During the mock-up design phase, wireframes were iteratively created based on the user scenarios developed during the requirements analysis phases. First, the wireframes were designed to visually illustrate the skeletal nature of the system. The layout of the system header, content area and footer were sketched during this phase. Placeholders were used to visually allocate their space within the system’s design layout. Mock tables, buttons and other visual elements were used to demonstrate what they would look like in the system. Figure 44 illustrates the initial designs for the prototype decision support system layout.

Figure 44. Mock-up layout

Following the review and feedback for the initial design, the second design iteration phase focused on the actual content design and flow of the system from one functional requirement to the other. Each design layout was created with the content

188 Chapter 7: Evaluating Our New Practice-Based Evidence Approach

in mind. Figure 45 shows the mock-up design of the first functional requirement in the requirements specification list. It represented the first page viewed by the doctors when accessing the decision support system and contained a table illustrating the type of information regarding patients. This helped stakeholders to decide on whether such information should be omitted or if new information needed to be added in.

Figure 45. Design with content in mind

A process flow diagram (Figure 46) was created to provide the logical and functional flow to the system. With each design layout, the design elements mimicked these process flows, progressing from one mock-up design to the other. This allowed stakeholders to visualise the mock-up and the logical flow together as illustrated in Figure 47. By doing so, stakeholders could easily identify missing pages and links between the flows to be rectified by system developers in the next design iteration.

Figure 46. Functional process flow

Chapter 7: Evaluating Our New Practice-Based Evidence Approach 189

Figure 47. Logical flow of system with design layout

Heuristic evaluation (HE) as design evaluation and guidelines Upon the completion of the finalised design mock-ups, the final phase of the design iteration focused on improving the usability aspect of the prototype clinical decision support system. Usability, as defined by the international standard ISO 9241, is “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use” (ISO, 1998). Usability is important, as it affects users’ performance, satisfaction and the acceptance to use (Holzinger, 2005). While the usability of the prototype CDSS was not the core element in the evaluation of our Practice-based evidence approach, however poor usability of the prototype CDSS might negatively influence the evaluation outcome.

Drawing insights from the works of Davis (1993) on the Technology Acceptance Model (TAM), this model highlighted that the motivation to adopt the use of any systems depended on the users’ perceived usefulness and perceived ease of use of the system. With that in mind, it might be possible that end users might misinterpret our PBE approach and considered it as a flaw when the prototype CDSS was deemed not

190 Chapter 7: Evaluating Our New Practice-Based Evidence Approach

user-friendly enough or easy to use. When in fact, it could just have been a design issue while the approach was effective in assisting with decision making.

Thus, using the TAM model as a usability design inspiration, it was therefore necessary to create an easy-to-use and user-friendly prototype that eliminated potential biases when evaluating our PBE approach. Based on the TAM model in Figure 48, the evaluation of our PBE approach to decision making was akin to how users decide on the “perceived usefulness” of a system to encourage its use. Therefore, by designing a user-friendly and easy to use system, this helped in reducing the biases that could be introduced during the evaluation of our PBE approach to decision making. This, in TAM was identified as the “perceived ease of use”. How user-friendly a system is reflected on how easy it is to use. Hence, if our system was not user-friendly enough, it might be misconstrued as being not useful, thus negatively influencing the perception of doctors as they evaluated our PBE approach. Therefore, we defined the gap that bridged the “perceived ease of use” and “perceived usefulness” as “Usability Design Gap”. This gap identified the need to design a user-friendly and easy to use system so that no external forces could negatively influence the evaluation of our PBE approach.

Figure 48. The need for a user-friendly and easy-to-use prototype (Usability design gap) inspired by Davis (1993) Technology Acceptance Model (TAM)

Chapter 7: Evaluating Our New Practice-Based Evidence Approach 191

There are many inspection methods available that can be used to evaluate a system’s usability, such as the cognitive walkthrough, heuristic evaluation and action analysis (Holzinger, 2005). Heuristic evaluation (HE) however, is the most common and inexpensive form of evaluation methodology used (Baker et al., 2002; Paz et al., 2015). Through HE, “usability problems in a user interface design” (Nielsen, 1994, p. 25) that breach design principles can be identified (Nielsen, 1994). A table of heuristics or general rules that describe the characteristic properties of usable design interfaces are used in HE, to help identify user interface problems (Nielsen, 1994). Hence, this research adopted the heuristics evaluation method in order to design a usable prototype CDSS.

However, in this particular study, HE was used as a method of usability evaluation as well as guidelines to design our system user interface. By using the 10 heuristics as listed in Nielsen (1994) HE checklist to identify gaps such as missing design or interaction elements, a usable system could effectively be achieved. The gaps, which we referred to as the ‘usability design gap’, represented the difference between the last iterated design mock-up and the envisioned final interface design. This was also the gap that might potentially introduce the biases and influences of the perceived ease of use of a system identified using TAM. Therefore, the bridging of the ‘usability design gap’ was accomplished by proposing and recommending the design and interaction elements that complied with usability design principles. In HE, the task of evaluating the user interface could be performed by usability specialists, however in this modified adoption, the task of identifying and recommending gap solutions was conducted by the system designer or developer.

Usability design gaps for heuristics H1, H3, H5-H8 and H10 were identified as potential usability design constraints. In the table Identifying Usability Design Gaps below, the Remarks column highlighted what had been done right to fulfil the heuristics as well as any constraints identified. The suitable recommendations were provided in the Recommendations column. For example, H1 remarks provided were “Certain actions can proceed without confirmation. Current status is not clearly indicated.” The recommendation offered was to enable the user to confirm the actions to take when a button was clicked, in order for the user to proceed, such as via a pop- up confirmation window. To indicate a user’s current status clearly within the system, ‘breadcrumbs’ were recommended. These recommendations represented the possible

192 Chapter 7: Evaluating Our New Practice-Based Evidence Approach

gap solutions and were considered in the final design mock-up. Based on the list of recommendations, several design elements and interactions had to be included in the final design iteration such as ‘breadcrumbs’, confirmatory actions, cancellation actions, on-hover actions and help information. This exercise certainly improved the design further and the final design is illustrated in Figure 49.

Figure 49. Finalised mock-up design

Identifying Usability Design Gaps Heuristics Remarks Recommendations H1: Visibility of Feedback and button information is Confirmation is needed to proceed system status displayed. Certain actions can proceed with a button action. without confirmation. Current status to be clearly indicated Current status is not clearly indicated. with breadcrumbs. H3: User control User can cancel any action with the Confirmation button to prompt users and freedom cancel button. to confirm every action. User should be able to cancel any action with the cancel button. H5: Error The need to confirm action prevents Implement confirmation action to prevention user from undesirable occurrence from prevent problems from occurring. happening. H6: Recognition Patient information and case numbers On hover, button reveals the rather than recall are clearly displayed at the top of the summary of the action to remind page to remind users which patient has users its functionality. been selected. Buttons are appropriately

Chapter 7: Evaluating Our New Practice-Based Evidence Approach 193

named to indicate the action that can be performed. However, more information should be provided so that user knows what the buttons are for. H7: Flexibility and No interaction available for expert Implement shortcut buttons to the efficiency of use users. main listing page is available for users to use (breadcrumbs).

H8: Aesthetic and Icons and buttons used are labelled with Provide more information regarding minimalist design unambiguous text to reduce error. the actions of icons and buttons as Correct colour combinations are used to text labelling may not explain much. reflect and grab users’ attention like bright red to indicate important statistics to reflect on. H10: Help and Help icon for novice users on the full documentation explanation of what the icons and buttons represent.

Table 27. Recommendations to usability design gaps based on Nielsen (1994) heuristics

7.4.3 Prototype Development The development of the prototype phase, following the completion of the prototype CDSS design, was done over two sprints, with each sprint being five days long. The functional requirements were broken down into tasks that could be managed during each sprint. The whole development process took two weeks as it was strictly focused on developing the prototype to implement only our PBE approach portion. This included displaying the list of patients to be consulted by the doctor in a routine clinical care set up, providing patient-centric statistics and prediction of length of stay based on the profile of patient selected, discharging patients and improving the prediction and statistics by adding the information of the discharged patient back into the pool of health records for further analysis.

The prototype CDSS was developed using PHP, MySQL and R. The web or user interface, which displayed patient information and statistical analytics, was coded in PHP. The data was stored in a MySQL database while the analytics was done using R.

Tasks were created from the list of requirements. Each task was allocated to one of the sprints. The 3 main tasks identified were (1) List patient, (2) Admit patient and (3) Discharge patient. Each task was further broken down into subtasks. Task 1 and 2 were allocated to the first sprint while task 3 was assigned to the second sprint.

194 Chapter 7: Evaluating Our New Practice-Based Evidence Approach

Two new database tables were created specifically for the prototype CDSS, patient_training and patient_testing, from the finalised data set provided by NUH. Data for both tables were extracted from the anonymised data set provided, with 75% of the data as patient_training and the remaining 25% as patient_testing. The star- schema data mart was used to extract the data used to calculate the patient-centric statistics and prediction model.

Sprint #1: Task 1 – List patient Using an open-sourced UI (User Interface) development kit developed by Dsouza (2016) called ‘Css-mint’, the UI development was fast-tracked as the HTML codes were easily replicated. The UI development kit provided ready templates to create common design elements such as title headings, buttons, tables, widgets and much more. Following the finalised design mock-up, a base template with the header, footer and main content was created.

Next, a PHP file (new_patient.php) to extract patient information for consultation from the MySQL database was created. First, the file initiated a connection to the database (db_connect.php). Once the connection had been established, it went on to create an SQL (Structured Query Language) statement requesting all patient information from the patient_testing table. The patient_testing table held all incoming new patients that needed a doctor’s consultation. Only the required fields from the patient_testing table were selected using the SQL statement. The outcome of the SQL statement was an array of patient information, which was then displayed to the user. A button was inserted at the end of each row of patient information that captured the patient unique identification. When clicked, the button triggered a pop-up box that requested users to confirm the intended action. This was to prevent the unintended errors from happening.

Two functions were created for ‘get total no of cases’ and ‘get highest LOS’ widget. The get_total_case_no() function calculated the total number of cases stored in the patient_training table. This table contained data of all discharged patients that was used to provide the statistical analytics and prediction model. The high_los() function calculated the highest number of days warded in the hospital.

Chapter 7: Evaluating Our New Practice-Based Evidence Approach 195

Breadcrumbs were created to remind users of their current status within the system. The UI kit had a template which automatically generates the breadcrumbs by using the HTML

tag with the class name breadcrumbs.

Figure 50. Prototype clinical decision support system adopting the PBE approach to decision making

Sprint #1: Task 2 – Admit patient Task 2 involved displaying the information of a patient selected by the doctor from the list of patients available for consultation. A PHP file (get_patient.php) was created to get a selected patient’s personal information, medical information and past admission episodes based on the patient’s identification number from patient_testing table using an SQL statement. The returned results from the query were an array of patient information, which was displayed to the user. An ‘Admit patient’ and ‘Cancel’ buttons were inserted at the end of the patient’s medical information table. When clicked, the button triggered a pop-up box, which requested the user to confirm the intended action so as to prevent the unintended errors from happening.

On the right panel, a list of patient-centric statistics was displayed as widgets. A total of seven widgets were created. Six new functions were designed to generate the statistics for ‘get percentage readmission by icd code’, ‘number of readmission cases by age group’, ‘mean los days’, ‘median los days’, ‘high los limit’ and ’number of cases based on icd code with high los’. Another function to calculate the predicted length of hospital stay was also created. A ‘get_percent_by_icd()’ function was created

196 Chapter 7: Evaluating Our New Practice-Based Evidence Approach

to generate the percentage readmission by icd code. A ‘get_cases_by_age_group()’ function calculated the number of cases by age group. A ‘get_mean_median()’ function was created to calculate the mean and median length of stay. A ‘get_los_limit()’ was used to get the high length of stay limit as well as a ‘get_num_high_los_case_by_icd()’ for finding the number of cases based on ICD code with a high length of stay.

Modelling the prediction for length of hospital stay To illustrate how our PBE approach to decision making could be achieved, one of the many patient-centric statistics provided was the prediction of the length of hospital stay. However, we did not consider the algorithm used to derive the prediction modelling of length of stay to be a major contribution of this research study. It was only used as a metric in a case study to show how our PBE approach was able to provide healthcare professionals other avenues to make well-informed decisions through the use of a prototype CDSS. Furthermore, the algorithm that was used not fully formulated by us but was based on studies by other authors.

The function to calculate the predicted length of hospital stay was created using R. The model for the prediction used was Poisson regression as it is generally used to calculate counts, in this case, the probable length of hospital stay. Poisson regression comes from a family of a generalised linear model (GLiM), which “provides accurate results for data sets having binary, ordered categorical, count and time to failure (or success) dependent variables” (Coxe et al., 2009, p. 122). Although there exists many types of algorithms and predictive models that could be adopted, such as decision trees, random forests or support vector machines, the decision to continue using Poisson regression was because the study is on the evaluation of our proposed approach of PBE to assist with decision making rather than the comparative effectiveness of a certain prediction model. Plus, there is no universal standard that governs which algorithms or models is the most suitable to use (Taleb et al., 2017). In fact, Poisson regression was also used in a study that predicts the length of stay for patients with primary total knee replacement (Carter & Potts, 2014). Thus, Poisson regression is a model that has been used and tested before to be able to predict length of stay. Besides, our requirement was to have a model that was relatively simple to create and able to implement it in our PBE approach but still able to provide a significantly accurate prediction. As the prediction was for the length of hospital stay, the variables regressed

Chapter 7: Evaluating Our New Practice-Based Evidence Approach 197

in the prediction model were based on admission attributes such as admission diagnosis codes and admission type. To identify the variables that were most significant in predicting length of stay, we also reviewed other related studies which made use of administrative data in creating the prediction models. For example, in a study conducted by Carter and Potts (2014), the authors identified factors or variables from the data set that had significant effect on length of stay as being the age, gender, consultant, discharge destination, deprivation and ethnicity. In another study by Newman et al. (2018), factors that were found to be associated with longer length of stay were identified as gender, ethnicity, accommodation status and primary diagnosis (Newman et al., 2018). Therefore, taking into consideration, the variables that were used in both studies above plus the limitation of the data set that we were provided with, we were able to come up with the significant variables to be used in our model. The final set of variables used in our regression model therefore comprised of readmission type, CCI value, admission diagnosis codes (HOSP_MAIN_DIAG_CODE), patient type (REF_TYPE), admission reason (ADMIT_REASON), age group (AGE1) and race (RACE). Likewise, these were also the variables that were significantly used in the above two studies. As the Poisson regression was based on binomial values, all the variables used had to be converted into binomials. For example, the readmission types were categorised using four codes, ‘0’ for no readmission, ‘1’ for 30 days readmission, ‘2’ for 60 days readmission, ‘3’ for 90 days readmission and ‘4’ for 120 days readmission. When modelling the prediction in R, the readmission type had to be represented by five new data columns labelled ‘Readmit0’ to ‘Readmit4’. If the readmission type was 1, then ‘Readmit0’, ‘Readmit2’, ‘Readmit3’ and ‘Readmit4’ data columns would contain the value 0 while ‘Readmit1’ would contain the value 1 and likewise for other corresponding readmission types. Once all the regressed variables were converted, the prediction model was created using Poisson regression performed in R. Code snippet 15 represent the finalised algorithm that was used to predict the length of hospital stay.

1. pred_model_poisson <- glm(LOS ~ READMIT0 + READMIT1 + READMIT2 + READMIT3 + READMIT4 + CCI0 + CCI1 + 2. CCI2 +HOSP_MAIN_DIAG_CODE + PATN_DS + PATN_Elective + PATN_Emergency + PATN_New + PATN_ Same + 3. PATN_Tech + ADMIT_4 + ADMIT_6 + ADMIT_13 + ADMIT_51 + ADMIT_52 + ADMIT_53 + ADMIT_54 + ADMIT_EMD +

198 Chapter 7: Evaluating Our New Practice-Based Evidence Approach

4. ADMIT_SOC + ADMIT_WARD + AGE0_19 + AGE20_44 + AGE45_64 + AGE65_84 + AGE85 + RACE_CAUCAS IAN + 5. RACE_CHINESE + RACE_EURASIAN + RACE_INDIAN + RACE_MALAY + RACE_OTHERS + RACE_SIKH, 6. family = "poisson", data = patient_train ) 7. 8. predicted <-predict(pred_model_poisson, patient_test, type = "response") 9. 10. solution <- data.frame("IDN" = patient_test$IDN, "Case NO" = patient_test$CASE_NO, "LOS " = 11. patient_test$LOS, "PRED LOS" = predicted)

Code Snippet 15. Prediction model of probable length of stay using Poisson regression in R

Based on the outcome of the regression model, the predicted value of length of stay for the patient was displayed in the widget.

Figure 51. Prototype CDSS with patient-centric statistics and prediction of length of stay

Relevance of patient-centric statistics and model predicting length of stay The decision to provide patient-centric statistics and to model predicted length of stay as part of our PBE approach to decision making was partly directed by the nature of information captured in the data provided by collaborating doctors. As highlighted, the data provided was administrative in nature and therefore had limited clinical information other than a primary diagnosis. General clinical information such as medication prescribed, laboratory test results or treatment provided were unavailable. This provided our research with a challenge in deciding the most relevant data to present and utilise for our PBE approach. However, initial research investigation into how best to meaningfully use the administrative data for decision

Chapter 7: Evaluating Our New Practice-Based Evidence Approach 199

making, led to the investigation of the relationship of hospital readmission cases and length of stay as significant factors influencing the quality and cost of care.

Hospital readmission has been defined as a “repeated hospitalisation within 1, 2, 4 or 12 months of discharge” (Benbassat & Taragin, 2000). In general, hospital readmission and hospital length of stay are viewed as being an economic concern for both patients and the healthcare systems, as both are focused on the need to utilise hospital resources (Benbassat & Taragin, 2000; Bowers & Cheyne, 2016; Dimick & Ghaferi, 2015; Gilstrap & Joynt, 2014; Kossovsky et al., 2002; Swain & Kharrazi, 2015; Walsh & Hripcsak, 2014). Therefore, the reduction in future hospital readmission cases and length of hospital stay can significantly help to improve the effective usage of hospital resources (and free up hospital beds) and in turn, minimise and even save on health care costs and expenditure. In addition, with the rollout of the Hospital Readmission Reduction Program (HRRP) by Centers for Medicare & Medicaid Services (CMS) in the United States, reducing hospital readmission has been recognised as a way to improve the quality of care management (Dimick & Ghaferi, 2015). Therefore, some of the factors that have been studied and are correlated with hospital readmission include age, gender, the severity of medical condition, comorbidities and interestingly, length of hospital stay (Benbassat & Taragin, 2000). Furthermore, factors such as unpreventable progression of a certain medical condition, drug allergies, untimely hospital discharge or problems arising from a medical or surgical procedure are also major contributors to possible readmission (Swain & Kharrazi, 2015).

Some of the cost-saving steps that have been adopted by healthcare organisations include adopting measures to clinically discharge patients as early as possible. Keeping patients unnecessarily in hospitals increases costs and does not bring any medical benefit, in fact, it increases the risk of these patients getting exposed to infections (Kossovsky et al., 2002). Length of stay has also been considered as a factor associated with quality of care management (Bradywood et al., 2016). Therefore, having patients warded for longer than necessary echoes the inefficiencies in healthcare procedures. However, shortening the length of stay has implications that can be more undesirable than expected. For example, discharging patients early runs the risk of healthcare professionals conducting incomplete evaluations and treatments provided (Kossovsky et al., 2002). Yet again, this demonstrates the inefficiencies in care being provided.

200 Chapter 7: Evaluating Our New Practice-Based Evidence Approach

However, in a study by Bradywood et al. (2016), the authors managed to demonstrate that positive health outcomes could be achieved with a shorter length of stay. Through the implementation of their “provider and patient-focused clinical care pathway”, the authors reported that the improvements in the quality outcomes of lumbar fusion surgery had also reflected a reduction in the length of hospital stay. Hence, for a case study evaluating our Practice-based evidence approach to decision making and plus the fact that the data set provided with, has limited clinical information, length of stay is used as a metric where healthcare professionals can be initiated to make more informed decision making. A predicted long length of stay may be indicative of missing treatments or inappropriate care provision, while a reflection of improved care can possibly be depicted by a shorter stay in the hospital. However, predicted length of stay is not the ultimatum that can precisely indicate the actual state of care for patients. In our PBE approach, LOS represents an indicator that has been inferred from the pool of EHR data representing evidence of actual clinical practices performed by healthcare professionals.

Therefore, in a developed and land-scarce country such as Singapore, with an increasingly ageing population and patients with chronic conditions, the challenge then is meeting the demands to provide effective care. For example, one such demand is providing enough hospital beds for patients who need it most when the supply becomes limited. By being able to continually offer healthcare resources to patients who are likely to be more at risk, healthcare organisations may compel the need to reduce patients’ possibility of hospital readmission and length of hospital stay, whilst not degrading the quality of care provided.

The patient-centric statistics on readmission thus provides healthcare professionals with an overview of the trends of readmission cases based on patient’s unique conditions, such as their primary diagnosis or according to their age group. Similarly, the mean and median length of stay statistics provides a trend or benchmark that can be compared with the length of stay predicted for that particular patient. Healthcare professionals may then choose to interpret this information as a patient needing a new treatment or a change of medication which may reflect an improved length of stay or the reduced possibility of readmission. Therefore, affording healthcare professionals with the opportunity to make more informed decisions, which

Chapter 7: Evaluating Our New Practice-Based Evidence Approach 201

is made possible through our PBE approach, can possibly lead healthcare professionals in delivering improved care.

Sprint #2: Task 3 – Discharge patient A PHP file (get_disch_patient.php) was created to extract patient information for discharge via the use of SQL statements. The information displayed contained patient personal information, medical information, discharge information and historical admission information. A ‘Discharge’ and ‘Cancel’ buttons were inserted at the end of the page to allow the user to discharge selected patients. When clicked, a pop-up box would appear requesting confirmation of action by the user. This helped to prevent any unintended errors from happening.

Upon clicking on the ‘Discharge’ button, the selected patient information was deleted from the patient_testing table and updated into the patient_training table. The patient_training table would now have an additional entry. Based on our PBE approach to decision making, the patient-centric statistics now would need to be updated to take into consideration the additional input inserted. The prediction model would be performed again as the additional row would influence the predicted length of hospital stay for each patient encounter.

A function to process the discharged patient was created to insert the information as a new entry in the patient_training table. A function in R was also created to perform the updated prediction model in order to incorporate the latest entry into the regression.

By the end of task 3, the prototype CDSS development was completed and overall functional testing performed to ensure that it would work as per the requirements.

7.5 PBE APPROACH EVALUATION

In this final phase of the study, our Practice-based evidence approach was qualitatively evaluated through a field test and a focus group session with two diabetic specialists, one doctor from the Public Health System and one health IT system engineer from National University Hospital. The evaluation process involved the demonstration of our PBE approach using our prototype clinical decision support system developed and an anonymised patient data set provided by NUH (field testing), followed by a question and answer session conducted in a focus group to elicit the opinions and viewpoints of participants.

202 Chapter 7: Evaluating Our New Practice-Based Evidence Approach

Using the data set, the prototyped clinical decision support system employed data analytics to provide patient-centric statistics and calculate the probable length of stay. By using the prototype decision support system, the doctors went through a simulated clinical workflow, which mimicked actual clinical consultation when managing diabetic patients in a specialised outpatient clinic (SOC) setting. The doctors then evaluated our PBE approach through the ability of the prototype CDSS to initiate them to further investigate or ask questions as a result of reviewing the statistics provided. In doing so, the doctors may be able to probe whether patients might have missed treatments and so on.

7.5.1 Participants The participants involved in the field testing and focus group were identified as healthcare professionals with the experience of using EHR systems. To evaluate our PBE approach as capable of assisting with decision making, we gathered information through a focus group discussion and interviews conducted with the participants.

In total, four participants were involved in this study. Two of the participants were diabetic specialists who we were already collaborating with us since the beginning of the study. The remaining two participants were, a doctor from Public Health System and Policy, and a healthcare IT system engineer. The initial collaboration plan was to only involve the two diabetic specialists, however they felt that the additional two participants would be adding their valuable inputs as well, during the PBE approach evaluation. We did consider that this may turn out to be a limitation of the study as the extra participants had different backgrounds, resulting in a non-homogeneous group. However, we realised that, all the doctors involved in this study had a common purpose, and that was to make use of the information systems to help manage patient care in the hospital. While the IT personnel was focused on ensuring that the doctors continued to benefit from the use of the information systems. Therefore, we believed that the inputs provided by them were relevant. Plus, they have prior experience in using EHR systems and consequently the views presented were still significant. Furthermore, this also helped in generating multiple viewpoints and opinions which benefitted the research, on top of their willingness to share and exchange ideas.

Chapter 7: Evaluating Our New Practice-Based Evidence Approach 203

Table of participants Designation No. of participant Clinical Director & Senior Consultant (Endocrinology Division) 1 Deputy Head & Associate Consultant 1 Domain Leader (Health System and Policy) 1 Systems Engineer (Health IT System) 1

Table 28. Table of participants in focus group

7.5.2 Instruments A qualitative semi-structured interview and discussion session were conducted during the focus group session to gain the participants’ feedback regarding the implementation of our PBE approach.

7.5.3 Conducting the evaluation process We facilitated the evaluation process ourselves and began with a short presentation on the background of the research to the four participants. They were briefed on the conceptualisation of our Practice-based evidence approach, research problems, questions and objectives. The aim was to ensure that the participants understood our concept of Practice-based evidence in order for them to provide their opinions, viewpoints and perceptions accurately regarding the approach as a means of assisting with clinical decision making.

Next, we started the field testing by demonstrating the application of our Practice-based evidence approach, performed through the use of the prototype clinical decision support system. The demonstration involved utilising the data set provided by NUH so as to verify the practicality of our PBE approach. This was done by simulating a patient visit scenario that ended with a hospital stay. In the simulation scenario, the attending doctor was provided with patient-centric statistics such as (1) percentage of readmission based on diagnosis code, (2) number of readmission cases based on age group, (3) number of cases with the diagnosis code registering high length of stay, (4) calculated mean length of stay days, (5) calculated median length of stay days and a predicted length of stay given the patient’s admission attributes, such as admitting diagnosis, patient type, comorbidity index, hospital readmission type and admit reason, and patient demographics such as age and race. The patient-centric statistics and the predicted length of stay helped to demonstrate the capability of our

204 Chapter 7: Evaluating Our New Practice-Based Evidence Approach

PBE approach. This was done by making use of evidence from the finalised data set and extracting relevant information that assisted healthcare professionals with decision making.

During the demonstration, we selected several different patient profiles to showcase the different patient-centric statistics and predicted length of stay generated based on the individual profile. To generate the statistics above, the finalised data set used in the demonstration was split into two. One was the base data source (patient_training) for calculating the statistics and prediction model and the other (patient_testing), formed the list of patients to be consulted by doctors. The patient- centric statistics and the prediction of the length of stay would differ depending on the different profile of patient selected. To recap, the section on “Modelling the prediction for length of hospital stay” described the algorithm that we employed to predict the length of stay, based on the variables associated with each patient profile. Therefore, based on individual patient’s profile, the variables such as diagnosis codes, patient type, age group or admission reason were regressed to calculate the probable length of hospital stay. Similarly, the patient-centric statistics were calculated based on the differing profile of patients.

Figure 52. Differing patient-centric statistics and prediction of probable length of stay based on patient profile and collection of health records

For example, the profile of a 74-year-old Chinese male with admission diagnosis code of ‘E1422’ was provided with differing statistics and predicted length of stay,

Chapter 7: Evaluating Our New Practice-Based Evidence Approach 205

compared to the profile of a 30-year-old Malay female with admission diagnosis code of ‘E1122’, even though in both cases, they were admitted for diabetic conditions. Therefore, given the patient’s profile, the prototype CDSS implementing our PBE approach calculated each statistic based on this profile and the information stored in the finalised data set provided.

Based on the mean and median length of stay as guides, doctors are able to assess if the predicted length of stay is too long. If the doctors perceive that the predicted length of stay is too long, this may prompt the doctors to investigate the problem further, initiate a treatment protocol or even order a laboratory test. The aim is to be able to assist doctors in making the right and appropriate decisions by providing relevant statistics that are constructive. The statistics on readmission will also help in identifying if this particular patient’s case has a high or low probability of readmission and may even prompt doctors to initiate other care protocols.

The significance of the simulation was to demonstrate the potential of re-using health records in generating statistics and predictive modelling that shows promise in being able to assist doctors with clinical decision making. Following the end of the field-testing demonstration, a focus group session consisting of the semi-structured interview was conducted. In fact, throughout the course of the demonstration, the participants were already engaged in an open-ended, question and answer as well as a discussion session enquiring about the design and practicality of our Practice-based evidence approach.

Therefore, at the end of the demonstration, most of the questions that were intended for discussion had already been answered. Nonetheless, other questions that were not broached earlier were then asked. The demonstration and focus group session were audio-recorded and lasted about 60 minutes.

7.6 ANALYSIS

The content of the demonstration and focus group session was analysed using thematic analysis.

7.6.1 Organising and preparing data Similar to the previous study, which involved the interview session with IT professionals, the primary data collected during this study were recorded using a

206 Chapter 7: Evaluating Our New Practice-Based Evidence Approach

portable digital audio recorder. A copy of the audio was extracted from the recorder and saved in our research laptop as well as on an external hard drive and AARNET CloudStor provided by QUT. No translation work was needed as this study was conducted in English.

NVivo 11 software was used to store and transcribe the data. The study session was transcribed intelligent verbatim as the transcriber was also the facilitator of the group.

7.6.2 Coding Transcripts The content of the transcripts was thoroughly read and analysed. Key concepts were identified and labelled as codes. Coding was done iteratively so that none would be missed. The codes thus represented the concepts and themes that had emerged from the content of the transcripts.

The first iteration of the analysis was done deductively based on the questions prepared prior to the evaluation process. With the coding and theme already developed, the analysis process involved identifying codes that matched the themes. The list of concepts that matched the themes identified is listed below.

Questions Transcript Themes Does the data captured in EHR • …“use existing data to inform..” Assist decision making contain information that can • …“tailor some of these prompts assist healthcare professionals based on clinical data…” make informed decisions? • …“dynamic enough that it can be used in real-time to change how we practice…” • …“the data actually tell you and how this can inform practice…” What is your opinion regarding • …“approach that you have used is PBE approach potential the PBE approach? very good…” • … “totally agree with what XQ said, I think the approach is very valuable..” • … “tool that I think has tremendous potential to be useful…” • …“it (the approach) has potential…” • … “this has potential to help tailor some of these prompts…” • … “I think that is a big potential if we can do the modelling correctly…” • … “it is something exciting…” Can PBE be an effective • … “it is still useful for clinicians in all Yes approach to assist healthcare that, what’s the predicted length of professionals? stay…” • … “the way we use the existing data to inform and how it is displayed and

Chapter 7: Evaluating Our New Practice-Based Evidence Approach 207

everything, I think is it a good approach…” Do you see any value with PBE • … “So it is still valuable. It is…” Applicability to other approach where it can assist in • … “easily extendable to any medical conditions decision making? disease…” • …“the way we use existing data to inform…” • … “what does the data tell you and how this can inform practice…” • …“it’s dynamic enough that it can be used in real-time…” Do you agree that PBE can be • … “absolutely, I think the answer is Patient-centric patient-centric? absolutely yes…” Do you agree that PBE can be • … “it is still useful for clinicians in all Relevance relevant? that, what’s the predicted length of stay…” Do you agree that PBE can assist • …“use existing data to inform..” Assist with decision with decision making? • … “what does the data actually tell making you and how this can inform practice…” In your opinion, is PBE relevant, • … “it is still useful for clinicians in all Relevant reliable, valid? If yes, why? If that, what’s the predicted length of no, why? What is PBE to you stay…” then? • … “I can already think of 2 potential intersection in some places that we do right…” • … “that the tool is most accurate…” Reliable What do you think is lacking? • … “but I know that the model Limitations (prediction) is not refined, because of the data set, limitation in time…” • … “further improvements need to be made like the details of modelling and the accuracy and all…” Table 29. Concepts matching themes developed deductively based on questions asked

However, the evaluation process also involved demonstrating the prototype CDSS. The following iteration of the analysis was done inductively so as to discover new themes that might not have been developed from the questions asked. Thus, codes that represented key concepts relevant to the evaluation of our PBE approach were identified and new themes were developed. The codes and themes from the following iteration are listed below.

Speaker Transcript (with codes) Theme Researcher If a patient would generally spend 9.4 days in hospital, so Useful in initiating this could equate to some, if this can be reduced that actions to improve care means we are improving the quality of care?

208 Chapter 7: Evaluating Our New Practice-Based Evidence Approach

Participant1 Yap (Yes). Participant2 I mean it is still useful for clinicians in all that, what’s the Useful in providing predicted length of stay. information. Participant1 Yes. Value in assisting with decision making. Participant2 So it is still valuable. Participant1 It is. Interviewer So when you discharge the patient, what happens is that Evidence of PBE information is being pushed back into the training sets. approach Participant2 I see, I get it. To improve the model. Participant3 It is easily extendable to any disease. Value in assisting with decision making / applicable across diseases Participant1 So basically an emergency visit that turned into an Relevance of PBE admission is the trajectory, is the segment of the patient approach / Accuracy population that the tool is most accurate. Participant1 Whereas, actually the potential application of this is far PBE has greater greater. application potential Participant2 & Yap (Yes) 3 Participant2 I think what you have done, the approach that you have Opinion of PBE used is very good approach Participant1 Yap (Yes) Participant2 But I know that the model is not refined, because of the Current limitation data set, limitation in time. Participant2 But I think the way we use the existing data to inform Value in assisting and how it is displayed and everything, I think it is a good decision making. approach. Using data to inform, ie decision making Participant3 I am really impressed with the project. Potential of PBE Participant3 I think he has the base, just a matter of data update, it Potential of PBE to be should take care. used across different diseases Participant1 I totally agree with what XQ said, I think the approach PBE approach has value. is very valuable and I think you have done a really good PBE approach has job of demonstrating the concepts put forward as your potential to be useful. background for why you want to do this into a tool that I think has tremendous potential to be useful. Participant1 I guess definitely further improvements need to be made Current limitation like the details of the modelling and the accuracy and all but I think there’s tremendous opportunity to develop that further. Researcher What is your opinion regarding PBE approach as a way to Potential as an approach. improve decision making? Data is key. Participant1 I think like I said, it has potential. I think the quality of Data quality concerns the data set in the variable that is actually captured will be key to ensuring that the application that come out of it will indeed be beneficial Participant2 .. in general now, most of the decision support systems are Patient-centric. very static like checklist, prompts based on standards. So I Tailored to data, think that this has potential to help tailor some of these evidence of PBE prompts based on clinical data that comes in. approach.

Chapter 7: Evaluating Our New Practice-Based Evidence Approach 209

Participant2 With every patient that comes in, it improves so I think Potential that is a big potential if we can do the modelling correctly with I mean proper caveats. Participant3 Right Participant2 So I think it is something exciting if we can improve on the accuracy Participant1 Its dynamic enough that it can be used in real-time to Dynamic, real-time. change how we practice and I can already think of 2 Potential application. potential intersection in some places that we do right Researcher Does it improve the patient-centricness of care for Patient-centric. patients? Complementary Participant1 I mean absolutely, I think the answer is absolutely yes. approach. They are both examples of data-driven approaches, and Data-driven they are complementary approaches right. Participant1 It is almost a ground-up approach where you are mining Ground-up approach the data and letting everything play out as they normally Data inform practice. would and is asking what does the data actually tell you Assist decision making and how this can inform practice Researcher Do you think that we can rely on the evidence to make Data drives decisions decisions? Participant2 Only if the records are coded correctly, etc Researcher Do you think that can we still make use of those records Data reliability issues. Participant2 It is one aspect of you know how reliable it is. I think the Data accuracy issues. accuracy of the records is important but then the other aspect for example if you are doing modelling what is your predictive value. Participant1 I am trying to envision so for example even for hip Initiate decision making fracture patients is it ok for this characteristics, the by considering predicted length of stay is this. And we know the mean alternatives. and median length of stay for hip fracture is as such. Patient-centric The let’s say you, for whatever reason, didn’t initiate a PTOT evaluation, maybe the prompts comes up, ”It is now day 2 of admission, do you want to consider a PTOT evaluation?”. And the clinicians have the right to override it and maybe that’s why it’s no, but let’s say order, then you can see how the predicted length of stay starts to come down. Researcher Starts to change Participant1 That I think would be quite a powerful visual factor Table 30. Coding identified concepts and themes done inductively.

The use of thematic networks helped to generalise the themes emerging from the content of the demonstration and focus group session. At the same time, it provided the general consensus regarding our PBE approach as a tool that can assist healthcare professionals with decision making through the use electronic health records as evidence for the clinical decision support system.

210 Chapter 7: Evaluating Our New Practice-Based Evidence Approach

Figure 53. Themes emerging through thematic networks from evaluation of PBE approach

211 Chapter 7: Evaluating Our New Practice-Based Evidence Approach

7.6.3 Drawing Findings Our Practice-based evidence approach to decision making is described as the approach where “sources of meaningful evidence of clinical practices performed by healthcare professionals as part of their routine practice, such as electronic health records and electronic medical records, are integrated, and this practical and comprehensive evidence used in a decision support system to support and inform clinical decision making towards the care of individual patients”. The implementation of our PBE approach was made possible in this research study through the development of a prototype clinical decision support system that utilised inpatient administrative data as the source of meaningful evidence. As a case study to exemplify how our PBE approach could inform and support healthcare professionals with clinical decision making, our prototype CDSS provided patient-centric statistics and prediction of probable length of stay. The purpose of this study is to evaluate, based on the opinions and viewpoints of doctors elicited from the focus group discussion, our Practice-based evidence approach in assisting healthcare professionals with decision making.

Consequently, in this phase of the study, three research sub-questions identified were answered. This helped in evaluating the adoption of our PBE approach to decision making.

Seven themes emerged from the analysis of the content of the focus group interview and discussion done during the evaluation of our PBE approach. These seven themes were developed both deductively and inductively: deductively, based on the themes in the questions being asked to understand the adoption of our PBE approach. As the field testing or prototype demonstration was a major part of the evaluation process, discussions amongst focus group participants were inevitable and therefore the possible discovery of additional themes had to be done inductively. Thus, the findings relating to our PBE approach to decision making for healthcare professionals were relayed through these themes.

Data-driven The key to our Practice-based evidence approach is in the utilisation of electronic health records as evidence, of actual clinical practices performed, in supporting and informing healthcare professionals with decision making. Therefore, there was a need

Chapter 7: Evaluating Our New Practice-Based Evidence Approach 212

to evidence this during the focus group discussion in order to verify the practicality of our PBE approach.

This was indeed evident based on the theme of being data-driven, emerging from the content of the focus group interview and discussions. Elements of “…using existing data to inform…” and “…tailor some of these prompts based on clinical data that comes in…” suggested the influence that data had on decision making through our PBE approach.

“But I think the way we use the existing data to inform (practice) and how it (data) is displayed and everything, I think it is a good approach.” (Participant2)

“I think the quality of the data set in the variable that is actually capture will be key to ensuring the application that come out of it will indeed be beneficial.” (Participant1)

“So I think that this has potential to help tailor some of these prompts based on clinical data that comes in.” (Participant2)

Assisted decision making The key to evaluating if our PBE approach was capable of assisting healthcare professionals with decision making required the participants to show evidence that the approach was capable of providing assisted decision making.

One of the key findings that emerged for this theme was when one of the participants described how the predicted length of stay could encourage the attending doctor to decide if an order of a particular treatment for the patient should be made. In this particular case, the participant was referring to a hip fracture patient. The decision to order, could potentially result in a shorter period of stay for that patient, or vice- versa.

“I am trying to envision (this) - so for example even for hip fracture patients, is it ok for this characteristics, (that) the predicted length of stay is this? And we know the mean and median length of stay for (a) hip fracture is as such. Then let’s say you, for whatever reason, didn’t initiate a PTOT evaluation, maybe the prompts comes up, ”It is now day 2 of admission, do you want to consider a PTOT evaluation?”. And the clinicians have the right to override

Chapter 7: Evaluating Our New Practice-Based Evidence Approach 213

it and maybe that’s why it’s ‘no’, but let’s say ‘order’, then you can see how the predicted length of stay starts to come down.” (Participant1)

In such a case, a shorter period of stay would therefore mean an improved quality of care had been provided. This opinion, of a shorter length of stay to indicate enhanced quality of care, was echoed by one of the participants when asked by the facilitator if the predicted length of stay was reduced, would that then suggest an improvement in the quality of care provided.

“If a patient would generally spend 9.4 days in hospital, so this could equate to some, if this (length of stay) can be reduced, that means we are improving the quality of care?” (Researcher)

“Yap (Yes).” (Participant1)

In addition, one of the participants also went on to mention that displaying the predicted length of stay was useful for the clinicians, although the participant fell short of elaborating how useful that could be. Therefore, we could assume that the displaying of LOS helped in deciding if a treatment should be ordered, as mentioned above. It could also be understood to mean that presenting LOS helped doctors to determine whether a patient should be warded. Nevertheless, the fact that such statistics can be afforded to clinicians and healthcare professionals is a sign that their decision making can be assisted.

“I mean it is still useful for clinicians; in all that, what’s the predicted length of stay.” (Participant2)

Valuable as an approach In general, the consensus amongst the participants was that our PBE approach to decision making has value. The concept of our PBE approach being applicable to different medical conditions, complementary, dynamic, ground-up, patient-centric, real-time and relevant, was brought up by the participants during the discussion and interview. For example, in the following transcript,

“It’s dynamic enough that it can be used in real-time to change how we practice and I can already think of two potential intersection in some places that we do right.” (Participant1)

214 Chapter 7: Evaluating Our New Practice-Based Evidence Approach

the participant not only suggested that the approach was dynamic and can be used in real-time, but the participant even went further to propose where it would be most applicable for use within the hospital.

Other instances of the value of our PBE approach, such as its applicability to address different medical conditions were iterated twice during the session.

“It is easily extendable to any disease.” (Participant3)

“I think he has the base, just a matter of data update, it should take care.” (Participant3)

However, we had to specifically asked the participants if our PBE approach has value in being patient-centric. “Does it improve the patient-centricness (centeredness) of care for patients?” (Researcher)

“I mean absolutely, I think the answer is absolutely yes.” (Participant1)

Next, relevance was implied as a factor of our PBE approach being valuable. There were no exact instances where the participants voluntarily indicated that our PBE approach was indeed relevant, instead the participants just mentioned how the features of the prototype CDSS that implemented our PBE approach were useful or accurate.

“I mean it is still useful for clinicians in all that, what’s the predicted length of stay.” (Participant2)

“So basically an emergency visit that turned into an admission is the trajectory, is the segment of the patient population that the tool is most accurate (for).” (Participant1)

In one instance, the participant unambiguously mentioned our PBE approach as being very valuable.

“I totally agree with what XQ said, I think the approach is very valuable…” (Participant1)

Visualisation The visualisation theme was not something that we were aware of before the start of the evaluation process. One of the participants highlighted how the use of data

Chapter 7: Evaluating Our New Practice-Based Evidence Approach 215

to inform and how it was displayed to the end-users reflected how good an approach it was, which was actually unexpected.

“But I think the way we use the existing data to inform and how it is displayed and everything, I think it is a good approach.” (Participant2)

The only difference that the prototype CDSS brought to the end users when compared to the EHR systems used by the hospital was the introduction of widgets displaying patient-centric statistics and prediction model. This opinion was again reiterated close to the end of the focus group session.

“That I think would be quite a powerful visual factor.” (Participant1)

Potential During this demonstration and focus group session, our Practice-based evidence had been described as an approach that had excellent application potential and the potential to be useful for healthcare professionals. As for how we had conceptualised our PBE approach, besides utilising data captured in EHRs to direct and assist decision making, our PBE approach also ensured that all new clinical practices performed by healthcare professionals were captured and re-analysed for improved decision making. Therefore, when asked about the approach of PBE as a way to improve decision making, the overall opinions of the participants were in agreement that the approach has potential to be useful.

“What is your opinion regarding PBE approach as a way to improve decision making?” (Researcher) “I think like I said, it has potential. I think the quality of the data set in the variable that is actually captured will be key to ensuring that the application that come out of it will indeed be beneficial.” (Participant1)

The opinion of one participant was that our PBE approach had a higher application potential. The prediction of the length of hospital stay was thought to offer a more significant impact as compared to just using laboratory results to lay claim that a patient had been undiagnosed.

“No I think it is not to say that it is difficult but I don't think it is a very impactful objective in that because then it is just a case of the lab results or

216 Chapter 7: Evaluating Our New Practice-Based Evidence Approach

something, then it is just undiagnosed. Whereas actually the potential application of this is far greater.” (Participant1)

Another participant commented that the procedure of using existing data was recognised as being a good approach adopted to assist healthcare professionals with their decision making.

“I think what you have done, the approach that you have used is very good.” (Participant2)

“But I think the way we use the existing data to inform and how it is displayed and everything, I think it is a good approach.” (Participant2)

“I am really impressed with the project.” (Participant3)

“I totally agree with what XQ said, I think the approach is very valuable and I think you have done a really good job of demonstrating the concepts put forward as your background for why you want to do this into a tool that I think has tremendous potential to be useful.” (Participant1)

However, some of the participants believed that the approach would have potential if some of the limitations and concerns could be addressed.

“With every patient that comes in, it improves so I think that is a big potential if we can do the modelling correctly with I mean proper caveats.” (Participant2)

Concerns & Limitations However, there were indeed concerns and limitations that were highlighted during the focus group. Similar limitations were also highlighted at the beginning of the demonstration. The limitation of time and provided data set, which only contained patient demographics and hospitalisation information, restricted the type of statistics possible and confined the prediction model to only predicting the probable length of stay. This was acknowledged by one of the participants but recognised that it was still valuable to clinicians for decision making.

“I mean it is still useful for clinicians in all that, what’s the predicted length of stay.” (Participant2)

The other concern was regarding data accuracy which was in relation to the limitations highlighted. The predictive modelling technique employed was Poisson

Chapter 7: Evaluating Our New Practice-Based Evidence Approach 217

regression, which was an accepted model for predicting counts such as days or population, without any additional techniques involved such as decision trees, random forests or support vector machines. Although as mentioned, the decision to use Poisson regression was not without any prior evaluation process being done. However, it is also imperative to point out that the evaluation was on the implementation of our PBE approach to assist with decision making through the combined use of EHRs with a prototype CDSS and not on the comparative evaluation of a prediction model’s accuracy. Additionally, we also placed equal effort in designing the prediction model to accurately predict the length of stay. The last concern was regarding the reliability and quality of data used. That was taken into consideration as the participants indicated how the limitation would affect the benefits of adopting our PBE approach.

“I think the quality of the data set in the variable that is actually captured will be key to ensuring that the application that come out of it will indeed be beneficial.” (Participant1)

“I guess definitely further improvements need to be made like the details of the modelling and the accuracy and all but I think there’s tremendous opportunity to develop that further.” (Participant1)

“Only if the records are coded correctly, etc” (Participant 2)

“It is one aspect of you know how reliable it is, I think the accuracy of the records is important but then the other aspect for example if you are doing modelling, what is your predictive value.” (Participant 2)

The seven themes that emerged from the analysis of the content of the focus group interview and discussion enabled us to answer the following research question.

“What are the processes required that ensures a PBE approach can assist healthcare professionals in clinical decision making?”

In answering the above research question, we identified the following three specific objectives that were met during this study. The objectives had helped to answer the above research question identified.

Objective 3.2: Identify how clinical evidence from EHR can be used to assist in decision making. Findings from the focus group discussion and interview revealed the themes of data-driven and assisted decision-making that described the capability of our PBE

218 Chapter 7: Evaluating Our New Practice-Based Evidence Approach

approach to decision making. These themes potentially stemmed from the use of inpatient administrative data set as evidence for our prototype CDSS. This was supported by the findings from the focus group discussion and interview which revealed that our PBE approach enabled the “use of existing data to inform (practice)”. This was be taken to indicate that evidence or data that was captured in the electronic records was capable of assisting or directing healthcare professionals in making well- informed decisions. The use of the provided data set also generated significant patient- centric statistics that helped to encourage healthcare professionals to make appropriate decisions, such as the decision to order a particular treatment or medications.

However, concerns were also raised regarding the quality of data used in generating the patient-centric statistics where doctors could infer from and consequently made their informed decisions with. One of the points raised by the participants when questioned regarding the reliability of evidence to make decisions from was that, it could only be relied on to make decisions “if the records are coded correctly”. This raised the issue of data quality in electronic health records. Similarly, the other point raised was “… how reliable it is, I think the accuracy of the record is important …”. While to a certain extent, in our PBE approach, we demonstrated how data captured in EHRs could be used to assist in decision making. However, the usefulness is dependent upon how the data or information has been captured in an EHR. If it erroneous to begin with, the ensuing information inferred from may result in misinformed decisions.

Objective 3.3: Identify how a clinical decision support system can assist with making well-informed decisions. Clinical decision support systems are tools that help healthcare professionals to make decisions. Similarly, the prototype CDSS implementing our PBE approach aims at supporting healthcare professionals in making well-informed decisions. The findings from the focus group interview and discussion thus revealed how our prototype CDSS assisted with decision making.

Our prototype CDSS enabled the generation of patient-centric statistics and predicted the probable length of stay from the inpatient administrative data set used. In particular, the LOS had encouraged doctors to make better decisions. The LOS provided doctors with the opportunity to decide if treatment was necessary or not. In addition, the use of widgets also added a new dimension to how new CDSS could be

Chapter 7: Evaluating Our New Practice-Based Evidence Approach 219

developed. The widgets not only informed doctors the patient-centric details regarding the patient but also represented as a powerful visual factor that attracted the doctor’s attention and encouraged appropriate response. Our prototype CDSS turned out to be novel based on the doctors’ opinions and viewpoints. When compared to the general CDSS the doctors were currently using, such CDSSs were rather static, and prompted based on standards or rules set. Our prototype CDSS instead has the potential to tailor the prompts based on the data used and unique patient’s profile. In addition, our prototype CDSS is “… dynamic enough that it can be used in real-time to change how we practice...

Objective 3.4: Identify how a PBE approach assists with decision making. Our PBE approach to decision making is not solely based on the use of CDSS to assist healthcare professionals in making decisions. Our PBE approach to decision making is actually made up of four key concepts which have been conceptualised in Chapter 4: Conceptualising our PBE approach and is illustrated in Figure 7, as consisting of a data warehouse architecture, utilisation of evidence such as electronic health records, a clinical decision support system and the provision of decision making capability. In fact, the answers to the two objectives above have in a way, contributed to meeting this current objective as well.

In identifying how our PBE approach can assist with decision making, we must consider the four key concepts mentioned in the earlier paragraph in totality. Therefore, for our PBE approach to enable the aim of assisting healthcare professionals to make well-informed decisions, it starts by integrating disparate sources of digital health data. A data warehouse methodology helps to integrate data generated from multiple clinical information systems found in a healthcare organisation into one. This results in data suitable as a source of evidence for decision making because the resultant data is both integrated and comprehensive. Such comprehensive data which contains evidence of actual clinical practices performed by healthcare professionals is perfect for a clinical decision support system that provides and presents patient-centric statistics along with patient-related health information. Through the use of the CDSS, healthcare professionals are assisted with the capability of making well-informed decisions. As highlighted in the above two sections, the benefit of using electronic health records is the generation of significant patient-centric statistics. The CDSS becomes the tool where the patient-centric statistics are displayed through widgets,

220 Chapter 7: Evaluating Our New Practice-Based Evidence Approach

representing a powerful visual factor that subsequently affords healthcare professionals with relevant information to make more informed decisions. It also provides healthcare professionals with information that is dynamic and be used in real- time.

7.7 LIMITATIONS

The use of inpatient administrative data with hospital episodes represented one of the limitations of this particular study. The data set that was used, which lacked clinical information, threatened to invalidate our Practice-based evidence approach, because it might not contain information that was considered useful and that could assist healthcare professionals with clinical decision making. However, there has been ongoing extensive research that studies the benefits of using administrative data for research (Grimes, 2010; Myers & Steege, 1999; Schneeweiss & Avorn, 2005; Stringer & Bernatsky, 2015). In fact, some of the most common use of administrative data is in length of stay studies. While administrative data may lack clinical information, it was still able to be useful.

In relation to the use of administrative data, another possible limitation was information bias. According to Gavrielov-Yusim and Friger (2014), administrative data suffers from several types of information biases like outcome misclassification and exposure misclassification. In outcome misclassification, it is possible that the medical condition that has been recorded in administrative data may be a result of misdiagnosis or an unsuitable description of the disease a doctor is investigating. Exposure misclassification on the other hand, occurs when information regarding medication use is wrongly documented, such as indicating the drug dosage or if the patient has indeed consumed the medication. Therefore, there is a chance that inferring information from an administrative data set may turn out to be inaccurate if the above biases are not addressed. The recommendation of a data warehouse methodology to be adopted not only integrate disparate data sources into one but integrating it intelligently by ensuring the multiple data sources were first verified for its accuracy and erroneous information cleansed before integrating them.

Another possible limitation of this study was in the algorithm used to predict and calculate length of stay. We acknowledged that there exists many other algorithms or prediction model that can be adopted in this research study. However, we decided on

Chapter 7: Evaluating Our New Practice-Based Evidence Approach 221

using Poisson regression to calculate the predicted length of stay for several reasons. The use of prediction model such as decision trees may yield a more accurate prediction of LOS when compared to using Poisson regression. However, our primary aim was implementing an approach where healthcare professionals could be afforded with the capacity to make well-informed decisions, while implementing the most accurate algorithm to predict LOS was secondary. This is because, it was much more important to evaluate the key components of our PBE approach as being able to assist in decision making. Furthermore, the result of any prediction model is also dependent upon the data set used and the variables that were regressed. So there was also a possibility that employing a more tedious algorithm result in the same performance.

Another possibility of being a limitation was the homogeneity of the participants involved in the focus group discussion and interview. The three doctors and one IT personnel seemed to indicate the lack of a homogeneous group of participants due to the differences in backgrounds. Potentially, there may be conflicting views within this group and finding consensus could be tricky. However, as we discovered during the field test and focus group sessions, the participants provided a broad range of opinions and viewpoints that were not conflicting in nature.

7.8 CONCLUSION

The introduction of our new Practice-based evidence approach to decision making began with the conceptualisation of our approach identifying its key components. In the study that followed, a suitable data warehouse architecture supporting our approach was designed. Next, the survey study on doctors’ perceptions regarding the benefits of EHR systems’ use and the usefulness of EHR data for decision making identified EHRs as being a suitable source of evidence for our PBE approach. Finally, in this last study, we developed a prototype clinical decision support system and simulated a data warehouse extraction process to extract data as the source of evidence that formed components for the evaluation our PBE approach to decision making.

All four studies culminated in the practical implementation and adoption of our PBE approach, which focused on assisting healthcare professionals to make decisions for the care management of diabetic patients from a public hospital in Singapore. In

222 Chapter 7: Evaluating Our New Practice-Based Evidence Approach

particular, it concentrated on using patient-centric statistics and prediction of probable length of stay to determine the next course of action to take.

The prototype CDSS demonstration implementing our Practice-based evidence approach managed to illustrate to the participants how our approach functioned in an actual clinical setting. The findings from the focus group interview and discussion highlighted the capability of our PBE approach to assist healthcare professionals in decision making. This capability was demonstrated to healthcare professionals through the utilisation of data provided (data-driven), in presenting the statistics, predicting the length of stay, and the prototype CDSS visualisation aspects (statistics widgets and prediction model). The length of stay, while it was only used as a metric in a case study to illustrate our PBE approach to decision making, was relevant in helping to determine if treatment was required or if treatment could provide care improvements. However, legitimate concerns were raised especially with regards to the quality of data used as evidence. The understanding was that if data used was of quality, removed from having erroneous data and information captured in them was accurate to begin with, the inference made would then be appropriate and of benefits. If not, most probably decisions made from such data sources would be incorrect.

While certain aspects of our PBE approach were not without its limitation, such as the limitation of the data provided, the doctors who were involved in the evaluation process recognised the potential of the approach to help with their clinical decision making especially in motivating the need to question their next decision or course of action and at the same time acknowledged the potential of implementation our PBE approach to assist with decision making.

Chapter 7: Evaluating Our New Practice-Based Evidence Approach 223

[This page is left intentionally blank.]

224 Chapter 7: Evaluating Our New Practice-Based Evidence Approach

Discussion & Conclusion

“Learning without thought is labour lost; thought without learning is perilous” - CONFUCIUS

This chapter reflects on the findings and highlights the contributions made in this research. The chapter continues by identifying the limitations of the study and proposals for future work.

8.1 DISCUSSIONS OF RESEARCH FINDINGS

The conceptualisation of our Practice-based evidence approach to decision making was a multi-step process. The research began by reviewing literature and analysing the issues surrounding effective clinical decision making among healthcare professionals. This led us to the paradigm of Evidence-based practice or Evidence- based medicine, an approach which directs care and decision making in healthcare. However, earlier studies conducted by other researchers, such as Barkham and Mellor- Clark as well as Horn and Gassaway, highlighted issues regarding the limitations of the evidence emerging from Evidence-based practice being applicable to individual patients. This raised doubts among these researchers, and other academics alike, about EBP’s effectiveness to direct medical practices. This in turn impacted the effectiveness of clinical decision making. As a solution, both Barkham and Mellor-Clark’s, and Horn and Gassaway’s works recommended the approach of Practice-based evidence as a way to generate better quality evidence that is more applicable to current patient populations. This led us to consider how the approach of Practice-based evidence can be applied effectively in an actual clinical practice setting, where healthcare professionals are assisted in making well-informed decisions. Therefore, we adopted a different view of the existing Practice-based evidence approach.

Based on this literature analysis, we developed our own concept of a Practice- based evidence approach to decision making in Chapter 4. The key concepts of the approach were identified as (1) a data warehouse architecture to integrate disparate sources of data, (2) the use of electronic health records as an alternate source of evidence, and (3) a clinical decision support system to assist with decision making. Following this definition, the objectives for the eventual conceptualisation of our PBE

Chapter 8: Discussion & Conclusion 225

approach were established. The conceptualisation continued further in three studies, qualitative research in Chapter 5, quantitative research in Chapter 6, and qualitative research in the concluding chapter (Chapter 7) for the evaluation of our approach.

Our research has asked the fundamental question, “What are the ICT architecture and processes required for a new Practice-based evidence approach to assist healthcare professionals make well-informed decisions?” To answer this fundamental question it was broken down further into sub-questions, as follows.

RQ1: What will be an effective Practice-based evidence approach to decision making in a clinical setting? We introduced and evaluated a new Practice-based evidence approach that assists healthcare professionals in making well-informed decisions. Our studies have shown that our PBE approach has the potential to assisting healthcare professionals with decision making, because it was data-driven, where EHRs as evidence support decision making. This was demonstrated by the themes of ‘Data-driven’ and ‘Assisted decision making’ that emerged from the content of the prototype CDSS demonstration and focus group session. In addition, the prototype CDSS has adopted one of the five challenges to improving effectiveness of CDSS intervention by providing a visualisation capability that differs from other current implementations of CDSSs which appear to be largely static. Based on these findings, an effective PBE approach to decision making in a clinical setting was shown to be one which contains a data warehouse architecture that can integrate multiple sources of digital health data into a single comprehensive repository. This repository then becomes a pool of practical evidence to be used in a clinical decision support system for decision making. The clinical decision support system utilises this integrated data to provide patient-centric statistics and information. These statistics and information in turn assist healthcare professionals to make decisions that are well-informed and relevant to individual patients.

RQ2: What changes to the current state of a healthcare organisation’s ICT architecture are required to adopt a Practice-based evidence approach in assisting healthcare professionals with clinical decision making? To determine the changes required to adopt the PBE approach, it is important to understand the existing enterprise ICT architecture of the healthcare organisation, in this case, Singapore’s National University Hospital, and compare that with key components of our PBE-based approach illustrated in Figure 7.

226 Chapter 8: Discussion & Conclusion

To understand the current existing ICT architecture, we conducted a qualitative study with IT professionals from NUH and IHiS Pte Ltd. From that study, we uncovered the nature of their existing data warehouse. Hence, no new data warehouse implementation was required for this organisation. This is considered typical of healthcare organisations in Singapore as the infrastructure to support ICT healthcare management is managed by IHiS Pte Ltd with the supervision of MOHH.

However, the presence of a data warehouse only indicates the existence of an integrated data repository. Therefore, to extract relevant information from the existing data warehouse to be used as evidence in decision making, a specific data mart design must be developed. The following are the changes required. In this instance, a star schema data mart was designed and used to extract the appropriate information as explained in Section 7.3.6. Next, a new clinical decision support system was developed to utilise the data extracted from the data warehouse as the source of evidence to provide patient-centric statistics and information. Based on these changes, a prototype CDSS demonstration was performed and a focus group session conducted.

The capability to assist healthcare professionals in decision making was demonstrated during the evaluation of the prototype CDSS demonstration and focus group session. In particular, the theme of ‘Assisted decision making’ suggests that the approach was capable of providing support to healthcare professionals in decision making.

RQ3: What are the processes required that ensure a Practice-based evidence approach can assist healthcare professionals in clinical decision making? To ensure that our PBE approach can assist healthcare professionals with clinical decision making, the following processes are required. First, we need to identify a clinical data source suitable to be used as evidence in assisting decision making. To do so, a survey study was conducted to investigate the potential of EHRs as a suitable source of evidence for our PBE approach. The study is significant in eliciting the perceptions of doctors regarding the benefits of EHRs and thus its applicability to assist decision making. We selected EHRs because it contain valuable information that can be analysed, studied and applied to improve healthcare delivery. Therefore, in this survey we focused on evaluating the perceived benefits of using EHR systems and the perceived usefulness of EHR data for decision making. The results indicated that, in general, doctors benefited from the use of EHR systems and that the systems provided

Chapter 8: Discussion & Conclusion 227

effective care delivery and management. Doctors surveyed also identified the potential of utilising information captured in EHR data to assist and further improve their decision making. This suggests that EHRs are a good candidate as practical evidence for our PBE approach.

Next, we designed a star schema data mart based on the structure of EHR data (anonymised patient data) provided. A data mart allows the extraction of relevant and appropriate information from the data warehouse. The conceptualisation of our Practice-based evidence approach to decision making involves the development of a prototype clinical decision support system and the utilisation of EHRs as a source of evidence to assist with decision making. Following the data mart design process, a clinical decision support system was created. The prototype CDSS represented the tool that assists healthcare professionals make well-informed decisions by generating patient-centric statistics and information based on the evidence entered. We employed logistic regression in this study to generate the patient-centric statistics.

To ensure that the processes above can establish our Practice-based evidence approach as capable of assist healthcare professionals in clinical decision making, an evaluation process took place with collaborating doctors from the National University Hospital. The evaluation was a two-part process comprising field testing (prototype CDSS demonstration) and a focus group interview and discussion session. The focus group involved doctors and an IT professional from NUH. Based on the patient-centric statistics and information generated by the evidence (EHR data) used, healthcare professionals were asked regarding the usefulness of them to support decision making. Several themes emerged from the evaluation such as ‘Valuable as an approach’ and ‘visualisation’, where the use of widgets help to inform the doctors. In summary, the evaluation results were fairly favourable with regards to our PBE approach being able to assist with decision making. While there were some minor concerns raised about the quality of the data used and the accuracy of the prediction model, the collective view concerning the approach was positive in that it has value and potential.

Other Findings In addition to the discussions above, there were also other interesting findings to note. There were noticeable differences between the findings gathered from the survey and that of the focus group. In one of the findings from the survey, a small proportion of doctors indicated that EHR systems facilitated access to suitable clinical guidelines

228 Chapter 8: Discussion & Conclusion

which could improve their delivery of care. Another finding showed only half of the doctors surveyed indicating that the EHR systems they were using were able to identify patients who needed preventive care. Therefore, from the overall survey findings, we concluded that this probably represented some of the limitations of the current EHR systems doctors were using and the extent to which these systems were capable of assisting with decision making. In fact, the survey also managed to highlight that these limitations could be overcome easily by utilising information captured in EHRs to improve their decision making. Therefore, in the evaluation phase of our PBE approach, it was necessary and important to investigate this highlighted limitation. We needed to find out if information captured in EHR data was indeed practical in assisting with clinical decision making.

From the resulting demonstration of our PBE approach implemented using a prototype CDSS and anonymised inpatient administrative data, one of the potential benefits proposed by one of the doctors in the focus group was how data used in the prototype CDSS had the potential to enable doctors to make decisions in real time. For example, through the use of the prototype CDSS, doctors would be able to decide on the possible treatments that could be provided to patients and also the probability of how the treatments could affect health outcomes, even though this was reflected based only on the length of stay predicted. In such instances, a predicted reduction in the length of stay could indicate to the doctor that this was positive health outcome.

Furthermore, it could also be concluded that current EHR systems, especially the ones used by the collaborating doctors, were perceived as not being effective in assisting with decision making as compared to the prototype CDSS implementing our PBE approach. This is because patient data presented to doctors in existing EHR systems did not make any references to other patients with similar conditions. This made the information in EHR systems much more static as compared to the patient- centric statistics and prediction model provided which, according to the collaborating doctors, were dynamic and changed accordingly to individual patients. The visualisation aspect and the use of data to inform decisions could thus assist with improving the decision-making capabilities of doctors.

The evaluation of our Practice-based evidence approach to decision making for healthcare professionals marked the end of our research study, which started with the conceptualisation of the PBE approach itself.

Chapter 8: Discussion & Conclusion 229

In conclusion, this thesis study has provided a practical case study that can be adopted to address the issues surrounding the limitations of affording healthcare professionals with reliable evidence in which to make effective decisions through our Practice-based evidence approach. Our conceptualisation of a Practice-based evidence approach has evolved from being an approach to generate practical evidence to bridge the gap created by limitations of evidence used for Evidence-based practice, as indicated by studies conducted by Barkham and Margison (2007), and Horn and Gassaway (2007), to a decision-making approach driven by practical clinical evidence.

8.2 CONTRIBUTIONS

This thesis has provided a number of theoretical and practical contributions in the domain of healthcare ICT following three areas of research work, (1) our concept of a new Practice-based evidence (PBE) approach to assist in decision making, (2) the architecture to implement our PBE approach, and (3) the practical adoption of our PBE approach through an evaluation process.

1. This thesis contributes to the body of knowledge, first by the conceptualisation of our Practice-based evidence approach to decision making for healthcare professionals, and secondly by adopting the approach in an actual clinical care setting, whereas similarly named approaches have adopted a more theoretical stance to generate evidence for use in decision making, such as those by Barkham and Margison (2007), and Horn and Gassaway (2007). We have also noted the existence of other bodies of work, “personalised cohort studies” and “EHR green button”, where the approach is similar to ours. Rather than undermining the novelty of our PBE approach, this reaffirms the significance and relevance of our work in contributing to both the healthcare and information communication technology industries. Furthermore, with similar studies still being in a theoretical phase, our PBE approach has the potential to be among the first to be implemented and evaluated in actual practice.

2. By designing a data mart and simulating a data warehouse, which was designed based on the existing enterprise ICT architecture of Singapore’s National University Hospital, this thesis has contributed to the feasibility of the architecture to support our Practice-based evidence approach.

230 Chapter 8: Discussion & Conclusion

3. Our Practice-based evidence approach to decision making employs the use of a clinical decision support system with anonymised inpatient administrative data from National University Hospital to provide patient-centric statistics such as the rate of patient readmission based on the same diagnosis codes and the prediction of probable length of stay, as tools to assist healthcare professionals in making well-informed decisions. This PBE approach is able to prompt and engage healthcare professionals, potentially improving their clinical decision making. Therefore, by evaluating our PBE approach using actual clinical scenarios, this thesis has provided a case study where there is the potential for implementing the approach as a possible real-world application.

4. The development of the prototype CDSS represents a conceptual framework to support decision making at the point of care, which other similar healthcare organisations could adopt for the effective implementation of health informatics or analytics as part of enhancing care management and services. It is a significant approach that can be adopted without incurring huge investments when compared to utilising third party solutions.

8.3 LIMITATIONS

Even though we noted several limitations of this research, this did not undermine the validity of the research outcomes. By acknowledging the limitations, we are suggesting improvements that can be undertaken in future studies.

Firstly, the study on uncovering the enterprise ICT architecture of NUH relied on the findings from only two participants. The limitation is that the findings from this study might be applicable only to NUH and possibly not extendable to other public hospitals. With healthcare organisations grouped together in healthcare clusters, each cluster have their own unique requirements and needs. As a result, each cluster may have a different ICT architecture in place. Nonetheless, hospitals within the same healthcare clusters tend to adopt similar systems and technologies and thus have the same ICT architecture.

Secondly, the study on perceived clinical benefits and usefulness of electronic health records was based on the survey responses from a small number of participants, 30 to be exact. The risk is that biases could have been introduced because of this small pool of participants, possibly affecting the reliability of the survey results. The use of

Chapter 8: Discussion & Conclusion 231

preamble or short descriptions in every section of the survey might also run the risk of introducing biases into the survey responses. This is because the short descriptions might have indirectly influenced the participants’ perceptions even before they attempted to answer the questions. Next, the lack of balance with how survey questions were being asked, for example not having an ample mixture of negatively and positively asked questions, and the order of ticking responses might have also introduced biases in the participants’ responses. Therefore, a limitation of this particular study is that the findings might only be valid for these particular participants and the findings could not be generalised accurately to the whole population of doctors. However, it is also important to highlight that the participants did not originate from a single healthcare organisation but represented at least five different organisations. For that reason, there is still a possibility that the responses can be taken to represent the perceptions of a large cohort of doctors.

Thirdly, in the final study, evaluating our proposed Practice-based evidence to decision making, there are considerable concerns that require highlighting. First, the patient data used in the study contained only administrative information regarding inpatient hospital episodes. Given that our approach of Practice-based evidence is to assist healthcare professionals with clinical decision making, we would have preferred if more clinical data was made accessible during the study, for example, data which contained information such as medication or laboratory test results. It is anticipated that the availability of a richer data set will contribute to a more significant number of patient-centric statistics and prediction models that can add higher value to the decision-making process. Therefore, it is suggested that any future research into the adoption of our Practice-based evidence approach for decision making considers incorporating clinical information such as medication information, laboratory test results, treatment procedures, etc. Nonetheless, from our evaluation process, the administrative data still proved to be sufficient in demonstrating our Practice-based evidence approach in assisting with decision making. As highlighted in previous chapters, the prediction of length of stay represents a metric that was most suitable and relevant, given the nature of the data set, in providing healthcare professionals with a level of assistance when making decisions. Furthermore, length of stay has been extensively studied alongside quality of care as well as healthcare costs (Bowers & Cheyne, 2016; Kossovsky et al., 2002; Tickoo et al., 2016a).

232 Chapter 8: Discussion & Conclusion

Fourthly, with reference to the collaborating doctors, this study focused on diabetic patients, as the collaborating doctors are diabetes specialists. Therefore, the findings in this study are limited to and in relation to diabetic inpatient cases. However, our Practice-based evidence approach to decision making is not restricted to only work with diabetic data sets. We expect that our PBE approach can accept any given data sets for other medical conditions such as hypertension or asthma. What is required is to design a data mart that can extract relevant information from the data warehouse. Therefore, we foresee that our PBE approach will perform equally well with these data sets.

Fifthly, the regression model adopted for this study to predict the length of stay was Poisson regression. As highlighted, length of stay was used as a metric to illustrate the potential of the PBE approach to assist with decision making. The length of stay predicted varies based on different patient profiles and their medical conditions. While Poisson regression is a valid and sufficient model to use for the prediction of the length of stay, as demonstrated during the PBE evaluation, the prediction result is still dependent on the accuracy of the Poisson regression model and the variables used to model it. In this case, the model used was limited to variables concerning inpatient cases. Therefore, given the limitations of the data set used and the choice of employing Poisson regression, the use of other prediction models such as decision trees could provide better accuracy or results.

8.4 FUTURE WORK

The suggestions for intended future works are predominantly linked to the limitations of the study highlighted in the previous section. The suggestions are not only beneficial for us, but for other researchers who intend to continue from where this study has concluded.

First, the study was conducted in Singapore and in a public hospital. The determinants that influence hospitals in Singapore may be different from those in other parts of the world. Therefore for future work, it is suggested that more research into our approach of Practice-based evidence be carried out in countries that may have different types of clinical information systems and ICT infrastructure in place.

Second, the study focused on the use of administrative data that contains inpatient hospital admissions. For future work, we plan to utilise a clinical data set that

Chapter 8: Discussion & Conclusion 233

contains information such as medication, laboratory test results, treatment information, primary and secondary diagnosis and other clinically related information. Considering that the findings from the use of an administrative data set were useful, it is believed that this future work will contribute even further.

Third, the prediction model adopted in this study was Poisson regression. Although there is nothing wrong with its use in this study, it is foreseen that incorporating other prediction models such as decision trees or random forests may provide equally good or improved prediction accuracy. Therefore, for future work, we will be investigating the use of other prediction models.

Fourth, and more importantly, as a result of the development and evaluation of the prototype CDSS implementing our PBE approach, there are now plans to have the prototype CDSS realised as an actual application in NUH. Discussions have been made with regards to how simple it would be to seamlessly integrate the prototype CDSS with the current EHR system, CPSS2. Although it is still early stages, the potential of a real-world application is promising.

Fifth, as a form of measuring how effective our PBE approach can be, there are plans to conduct a quantitative analysis study that is able to measure improvements to decision making by comparing the use of a CDSS implementing our Practice-based evidence approach with that of a current generally used CDSS.

Sixth, with the development and effective use of clinical guidelines determined by evidence generated from systematic studies, our PBE approach can play a role in reviewing and improving clinical guidelines applicability to actual patient types at large. PBE can be used to examine the effectiveness of guidelines application to already determined patient types as well as to investigate their applicability to other patient types, especially those with comorbidities.

Seventh, in a current context where social media becomes the convenient go-to tool for quick and up-to-date information, this results in increased proliferation of user generated data that has potential use in decision making support. A practical application of Practice-based evidence approach can therefore be adopted, for example in online purchase recommendations, product reviews or trend analyses where consumers get to decide precisely what they need based on other users experiences and producers understand what their customers want.

234 Chapter 8: Discussion & Conclusion

Based on the research study, this list of future works represents both the breadth and depth of the potential of our PBE approach across and beyond the healthcare industry.

8.5 CONCLUSION

This thesis’ development was based on an overarching research question: “What are the ICT architecture and processes required for a new Practice-based evidence approach to assist healthcare professionals make well-informed decisions?” In answering this research question, eight chapters were developed, detailing a systematic process required in the research study.

Chapter 1 introduced the research, the background, research context, aims and objectives of the research, corresponding research questions, hypotheses and the significance of this research. Chapter 2 provided the detailed background of the research study. Chapter 3 consolidated the research methodology and research design for the conceptualisation of our Practice-based evidence approach and three phases of our research studies. Chapter 4 described the conceptualisation of our Practice-based evidence approach to decision making. One of the three research studies, Chapter 5, detailed the first study on the design of a data warehouse architecture and its analysis. Chapter 6 continued with the study on the data component of the research and its analysis. Chapter 7 concluded the research studies and described the research design and analysis in detail. Lastly, Chapter 8 discussed the research findings, the contribution, limitations and future works.

As with the discussion and analysis made in the PBE approach’s conceptualisation and the three research study chapters, this thesis has demonstrated the perceived benefits and the usefulness of utilising electronic health records to assist in the improvements of healthcare professionals’ decision making through the survey findings conducted with doctors from public hospitals and polyclinics. This was evident in the evaluation of our PBE approach, which was illustrated through the use of a prototype clinical decision support system and anonymised patient health data. The findings and analysis of the focus group data revealed that the Practice-based evidence approach to decision making has the potential to be useful for healthcare professionals. The approach is described as being valuable in helping healthcare professionals in their decision-making process. The approach to include patient-centric

Chapter 8: Discussion & Conclusion 235

statistics and prediction modelling is helpful in planning the next care procedure required and in evaluating the quality of care that can be provided. However, the approach has its share of limitations due to the nature of the data set provided as well as the prediction model adopted to predict the length of hospital stay.

The answer to the research question above is a data warehouse architecture, a clinical decision support system and electronic health records are required to implement our PBE approach. The electronic health records are used as the source of evidence for the clinical decision support system to generate patient-centric statistics based on the profile and medical conditions of actual patients that help healthcare professionals make well-informed decisions.

236 Chapter 8: Discussion & Conclusion

References

Abdullah, U., Ahmad, J. & Ahmed, A. (2008, 18-19 Oct. 2008). Analysis of effectiveness of apriori algorithm in medical billing data mining. Paper presented at the 2008 4th International Conference on Emerging Technologies. Adler-Milstein, J., Sarma, N., Woskie, L. R. & Jha, A. K. (2014). A comparison of how four countries use health IT to support care for people with chronic conditions. Health affairs, 33(9), 1559-1566. doi:10.1377/hlthaff.2014.0424 Ado, A., Aliyu, A., Bello, S. A. & Garba, A. u. (2014). Building a Diabetes Data Warehouse to Support Decision making in healthcare industry. International Organisation of Scientific Research Journal of Computer Engineering (IOSR- JCE), 16(2), 138-143. Agha, L. (2014). The effects of health information technology on the costs and quality of medical care. Journal of Health Economics, 34, 19-30. doi:https://doi.org/10.1016/j.jhealeco.2013.12.005 Akyürek, Ç. E., Sawalha, R. & Sina, I. (2015). Factors affecting the decision making process in healthcare institutions. Academy of Strategic Management Journal, 14(1), 1-14. Almutiry, O., Wills, G., Alwabel, A., Crowder, R. & Walters, R. (2013). Towards a framework for data quality in Electonic health records. Paper presented at the IADIS International Conference, e-Society 2013. http://eprints.soton.ac.uk/350302/ Anderson, A. E., Kerr, W. T., Thames, A., Li, T., Xiao, J. & Cohen, M. S. (2016). Electronic health record phenotyping improves detection and screening of type 2 diabetes in the general United States population: A cross-sectional, unselected, retrospective study. Journal of biomedical informatics, 60, 162-168. doi:https://doi.org/10.1016/j.jbi.2015.12.006 Arain, M., Campbell, M. J., Cooper, C. L. & Lancaster, G. A. (2010). What is a pilot or feasibility study? A review of current practice and editorial policy. BMC Medical Research Methodology, 10(1), 67. doi:10.1186/1471-2288-10-67 Attride-Stirling, J. (2001). Thematic networks: an analytic tool for qualitative research. Qualitative Research, 1(3), 385-405. doi:10.1177/146879410100100307 Australian Institute of Health & Welfare. (2014). Australia's Health 2014. Retrieved from http://www.aihw.gov.au/WorkArea/DownloadAsset.aspx?id=60129548150 Australian Institute of Health & Welfare. (2015). Diabetes. Retrieved from http://www.aihw.gov.au/diabetes/ Bagley, S. C. & Altman, R. B. (2016). Computing disease incidence, prevalence and comorbidity from electronic medical records. Journal of biomedical informatics, 63, 108-111. doi:https://doi.org/10.1016/j.jbi.2016.08.005 Baker, K., Greenberg, S. & Gutwin, C. (2002). Empirical development of a heuristic evaluation methodology for shared workspace groupware. Paper presented at the Proceedings of the 2002 ACM conference on Computer supported cooperative work, New Orleans, Louisiana, USA.

K1. Frequency and percentage distribution of EHR systems use and perceived clinical benefits.References 237

Bakken, S. (2001). An Informatics Infrastructure Is Essential for Evidence-based Practice. Journal of the American Medical Informatics Association, 8(3), 199- 201. doi:10.1136/jamia.2001.0080199 Balaji, S. & Murugaiyan, M. S. (2012). Waterfall vs. V-Model vs. Agile: A comparative study on SDLC. International Journal of Information Technology and Business Management, 2(1), 26-30. Barkham, M. & Margison, F. (2007). Practice-based evidence as a complement to evidence-based practice: From dichotomy to chiasmus. In C. Freeman & M. Power (Eds.), Handbook of evidence-based psychotherapies: A guide for research and practice (pp. 443-476): Wiley. Barkham, M. & Mellor‐Clark, J. (2000). The role of practice-based evidence in the psychological therapies. In N. Rowland & S. Goss (Eds.), Evidence-Based Counselling and Psychological Therapies Research and applications. London & Philadelphia: Taylor & Francis Inc. Barkham, M. & Mellor‐Clark, J. (2003). Bridging evidence‐based practice and practice‐based evidence: developing a rigorous and relevant knowledge for the psychological therapies. Clinical Psychology & Psychotherapy, 10(6), 319-327. Barkham, M., Stiles, W. B., Lambert, M. J. & Mellor-Clark, J. (2010). Building a rigorous and relevant knowledge base for the psychological therapies. In Developing and Delivering Practice-Based Evidence: A Guide for the Psychological Therapies (pp. 21-61). Hoboken, NJ, USA: John Wiley & Sons. Bates, D. W., Kuperman, G. J., Wang, S., Gandhi, T., Kittler, A., Volk, L., . . . Middleton, B. (2003). Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. Journal of the American Medical Informatics Association, 10(6), 523-530. doi:10.1197/jamia.M1370 Batini, C., Cappiello, C., Francalanci, C. & Maurino, A. (2009). Methodologies for data quality assessment and improvement. ACM Computing Surveys (CSUR), 41(3), 16. doi:10.1145/1541880.1541883 Baxter, P. & Jack, S. (2008). Qualitative case study methodology: Study design and implementation for novice researchers. The qualitative report, 13(4), 544-559. Bélanger, E., Bartlett, G., Dawes, M., Rodríguez, C. & Hasson-Gidoni, I. (2012). Examining the evidence of the impact of health information technology in primary care: An argument for participatory research with health professionals and patients. International Journal of Medical Informatics, 81(10), 654-661. doi:10.1016/j.ijmedinf.2012.07.008 Benbassat, J. & Taragin, M. (2000). Hospital readmissions as a measure of quality of health care: Advantages and limitations. Archives of internal medicine, 160(8), 1074-1081. doi:10.1001/archinte.160.8.1074 Bennett, C. C., Doub, T. W. & Selove, R. (2012). EHRs connect research and practice: Where predictive modeling, artificial intelligence, and clinical decision support intersect. Health Policy and Technology, 1(2), 105-114. doi:10.1016/j.hlpt.2012.03.001 Botsis, T., Hartvigsen, G., Chen, F. & Weng, C. (2010). Secondary Use of EHR: Data Quality Issues and Informatics Opportunities. Summit on Translational Bioinformatics, 2010, 1-5. Bowen, G. A. (2009). Document Analysis as a Qualitative Research Method. Qualitative Research Journal, 9(2), 27-40. doi:10.3316/QRJ0902027

238 References

Bower, P. & Gilbody, S. (2010). The current view of evidence and evidence-based practice. In Developing and Delivering Practice-Based Evidence: A Guide for the Psychological Therapies: Wiley-Blackwell. Bowers, J. & Cheyne, H. (2016). Reducing the length of postnatal hospital stay: implications for cost and quality of care. BMC Health Services Research, 16(1), 16. doi:10.1186/s12913-015-1214-4 Bradywood, A., Farrokhi, F., Williams, B., Kowalczyk, M. & Blackmore, C. C. (2016). Reduction of Inpatient Hospital Length of Stay in Lumbar Fusion Patients with Implementation of an Evidence Based Clinical Care Pathway. Spine, 42(3), 169-176. doi:10.1097/BRS.0000000000001703 Brown, J. S., Holmes, J. H., Shah, K., Hall, K., Lazarus, R. & Platt, R. (2010). Distributed Health Data Networks: A Practical and Preferred Approach to Multi- Institutional Evaluations of Comparative Effectiveness, Safety, and Quality of Care. Medical care, 48(6), S45-S51. Brubaker, J. R., Lustig, C. & Hayes, G. R. (2010). PatientsLikeMe: empowerment and representation in a patient-centered social network. Paper presented at the CSCW’10; Workshop on research in healthcare: past, present, and future. Buntin, M. B., Burke, M. F., Hoaglin, M. C. & Blumenthal, D. (2011). The benefits of health information technology: a review of the recent literature shows predominantly positive results. Health affairs, 30(3), 464-471. Burns, K. E. A., Duffett, M., Kho, M. E., Meade, M. O., Adhikari, N. K. J., Sinuff, T. & Cook, D. J. (2008). A guide for the design and conduct of self-administered surveys of clinicians. Canadian Medical Association Journal, 179(3), 245-252. doi:10.1503/cmaj.080372 Burnum, J. F. (1989). The misinformation era: the fall of the medical record. Annals Of Internal Medicine, 110(6), 482-484. Carter, E. M. & Potts, H. W. (2014). Predicting length of stay from an electronic patient record system: a primary total knee replacement example. BMC Medical Informatics and Decision Making, 14(1), 26. doi:10.1186/1472-6947-14-26 Castaneda, C., Nalley, K., Mannion, C., Bhattacharyya, P., Blake, P., Pecora, A., . . . Suh, K. S. (2015). Clinical decision support systems for improving diagnostic accuracy and achieving precision medicine. Journal of clinical bioinformatics, 5(1), 4. doi:10.1186/s13336-015-0019-3 Chamberlain, S., Sharp, H. & Maiden, N. (2006). Towards a Framework for Integrating Agile Development and User-Centred Design. In P. Abrahamsson, M. Marchesi & G. Succi (Eds.), Extreme Programming and Agile Processes in Software Engineering: 7th International Conference, XP 2006, Oulu, Finland, June 17-22, 2006. Proceedings (pp. 143-153). Berlin, Heidelberg: Springer Berlin Heidelberg. Chao, W. C., Hu, H., Ung, C. O. L. & Cai, Y. (2013). Benefits and Challenges of Electronic Health Record System on Stakeholders: A Qualitative Study of Outpatient Physicians. Journal of Medical Systems, 37(4), 9960. doi:10.1007/s10916-013-9960-5 Charlson, M., Pompei, P., Ales, K. L. & MacKenzie, C. R. (1987). A new method of classifying prognostic comorbidity in longitudinal studies: Development and validation. Journal of Chronic Diseases, 40(5), 373-383. doi:https://doi.org/10.1016/0021-9681(87)90171-8

K1. Frequency and percentage distribution of EHR systems use and perceived clinical benefits.References 239

Charlson, M., Wells, M. T., Ullman, R., King, F. & Shmukler, C. (2014). The Charlson Comorbidity Index Can Be Used Prospectively to Identify Patients Who Will Incur High Future Costs. PLoS ONE, 9(12), e112479. doi:10.1371/journal.pone.0112479 Chaudhry, B., Wang, J., Wu, S. & et al. (2006). Systematic review: Impact of health information technology on quality, efficiency, and costs of medical care. Annals of Internal Medicine, 144(10), 742-752. doi:10.7326/0003-4819-144-10- 200605160-00125 Chen, E. S., Hripcsak, G., Xu, H., Markatou, M. & Friedman, C. (2008). Automated acquisition of disease–drug knowledge from biomedical and clinical documents: an initial study. Journal of the American Medical Informatics Association, 15(1), 87-98. doi:10.1197/jamia.M2401 Chen, L., Magliano, D. J. & Zimmet, P. Z. (2012). The worldwide epidemiology of type 2 diabetes mellitus—present and future perspectives. Nature Reviews Endocrinology, 8(4), 228-236. doi:http://dx.doi.org/10.1038/nrendo.2011.183 Chen, L., Wang, C.-S., Wang, I.-C., Yu, H.-C., Peng, H.-Y., Chen, H.-C., . . . Lai, F. (2014). Patient-centric Data Warehouse Design - An Empirical Study Applied in Diabetes care. Paper presented at the BIOTECHNO 2014: THe Sixth International Conference on Bioinformatics, Biocomputational Systems and Biotechnologies, Chamonix, France. Chen, M., Hao, Y., Hwang, K., Wang, L. & Wang, L. (2017). Disease Prediction by Machine Learning Over Big Data From Healthcare Communities. IEEE Access, 5, 8869-8879. doi:10.1109/ACCESS.2017.2694446 Chennamsetty, H., Chalasani, S. & Riley, D. (2015, 5-7 March 2015). Predictive analytics on Electronic Health Records (EHRs) using Hadoop and Hive. Paper presented at the 2015 IEEE International Conference on Electrical, Computer and Communication Technologies (ICECCT). Chhabra, R. & Pahwa, P. (2014). Data Mart Designing and Integration Approaches. International Journal of Computer Science and Mobile Computing, 3(4), 74-79. Chignell, M., Yesha, Y. & Lo, J. (2010). New methods for clinical decision support in hospitals. Paper presented at the Proceedings of the 2010 Conference of the Center for Advanced Studies on Collaborative Research. Choi, I. Y., Park, S., Park, B., Chung, B. H., Kim, C.-S., Lee, H. M., . . . Lee, J. Y. (2013). Development of prostate cancer research database with the clinical data warehouse technology for direct linkage with electronic medical record system. Prostate International, 1(2), 59-64. doi:10.12954/PI.12015 Chun, M. & Mooney, J. (2009). CIO roles and responsibilities: Twenty-five years of evolution and change. Information & Management, 46(6), 323-334. doi:https://doi.org/10.1016/j.im.2009.05.005 Claridge, J. A. & Fabian, T. C. (2005). History and Development of Evidence-based Medicine. World Journal of Surgery, 29(5), 547-553. doi:10.1007/s00268-005- 7910-1 Coulter, A., Entwistle, V. A., Eccles, A., Ryan, S., Shepperd, S. & Perera, R. (2013). Personalised care planning for adults with chronic or long-term health conditions. Cochrane Database of Systematic Reviews(5). doi:10.1002/14651858.CD010523

240 References

Cox, A. P., Raluy-Callado, M., Wang, M., Bakheit, A. M., Moore, A. P. & Dinet, J. (2016). Predictive analysis for identifying potentially undiagnosed post-stroke spasticity patients in United Kingdom. Journal of biomedical informatics, 60, 328-333. doi:https://doi.org/10.1016/j.jbi.2016.02.012 Coxe, S., West, S. G. & Aiken, L. S. (2009). The Analysis of Count Data: A Gentle Introduction to Poisson Regression and Its Alternatives. Journal of Personality Assessment, 91(2), 121-136. doi:10.1080/00223890802634175 Cresswell, K. M. & Sheikh, A. (2015). Health information technology in hospitals: current issues and future trends. Future Hospital Journal, 2(1), 50-56. Creswell, J. W. (2007). Qualitative Inquiry & Research Design: Choosing Among Five Approaches (Second ed.). United States: Sage Publications. Crowe, S., Cresswell, K., Robertson, A., Huby, G., Avery, A. & Sheikh, A. (2011). The case study approach. BMC Medical Research Methodology, 11(1), 100. doi:10.1186/1471-2288-11-100 da Rocha Fernandes, J., Ogurtsova, K., Linnenkamp, U., Guariguata, L., Seuring, T., Zhang, P., . . . Makaroff, L. E. (2016). IDF Diabetes Atlas estimates of 2014 global health expenditures on diabetes. Diabetes Research and Clinical Practice, 117, 48-54. doi:10.1016/j.diabres.2016.04.016 Datamark Incorporated. (2013). Unstructured Data in Electronic Health Record (EHR) Systems: Challenges and Solutions. Healthcare Content Management White Paper. Retrieved from Datamark Incorporated website: http://www.datamark.net/uploads/files/unstructured_ehr_data_white_paper.pdf Davis, F. D. (1993). User acceptance of information technology: system characteristics, user perceptions and behavioral impacts. International journal of man-machine studies, 38(3), 475-487. De Mul, M., Alons, P., Van der Velde, P., Konings, I., Bakker, J. & Hazelzet, J. (2012). Development of a clinical data warehouse from an intensive care clinical information system. Computer methods and programs in biomedicine, 105(1), 22- 30. doi:10.1016/j.cmpb.2010.07.002 Department of Statistics Singapore. (2018). Registered Health Personnel. Retrieved from: http://www.tablebuilder.singstat.gov.sg/publicfacing/createDataTable.action?ref Id=15173 DiCicco-Bloom, B. & Crabtree, B. F. (2006). The qualitative research interview. Medical Education, 40(4), 314-321. doi:10.1111/j.1365-2929.2006.02418.x Dimick, J. B. & Ghaferi, A. A. (2015). Hospital readmission as a quality measure in surgery. Jama, 313(5), 512-513. doi:10.1001/jama.2014.14179 Dolin, R. H., Alschuler, L., Beebe, C., Biron, P. V., Boyer, S. L., Essin, D., . . . Mattison, J. E. (2001). The HL7 Clinical Document Architecture. Journal of the American Medical Informatics Association, 8(6), 552-569. doi:10.1136/jamia.2001.0080552 Dsouza, A. M. (2016). Css-mint. Retrieved from http://arunmichaeldsouza.github.io/CSS-Mint/ Elixhauser, A., Steiner, C., Harris, D. R. & Coffey, R. M. (1998). Comorbidity Measures for Use with Administrative Data. Medical care, 36(1), 8-27.

K1. Frequency and percentage distribution of EHR systems use and perceived clinical benefits.References 241

Evans, C., Connell, J., Barkham, M., Marshall, C. & Mellor‐Clark, J. (2003). Practice‐ based evidence: benchmarking NHS primary care counselling services at national and local levels. Clinical Psychology & Psychotherapy, 10(6), 374-388. doi:10.1002/cpp.384 Evolus. (2016). Pencil Project. Retrieved from http://pencil.evolus.vn/ Falsetti, L., Viticchi, G., Tarquinio, N., Silvestrini, M., Capeci, W., Catozzo, V., . . . Pellegrini, F. (2016). Charlson comorbidity index as a predictor of in-hospital death in acute ischemic stroke among very old patients: a single-cohort perspective study. Neurological Sciences, 37(9), 1443-1448. doi:10.1007/s10072-016-2602-1 Fan, W. & Sarfarazi, M. (2014). SAS® Solutions to Identifying Hospital Readmissions. Paper presented at the SAS® Global Forum, Washington DC, USA. Fava, G. A., Guidi, J., Rafanelli, C. & Sonino, N. (2015). The Clinical Inadequacy of Evidence-Based Medicine and the Need for a Conceptual Framework Based on Clinical Judgment. Psychotherapy and Psychosomatics, 84(1), 1-3. Feeny, D. F. & Willcocks, L. P. (1998). Core IS capabilities for exploiting information technology. Sloan management review, 39(3), 9. Feldman, K., Davis, D. & Chawla, N. V. (2015). Scaling and contextualizing personalized healthcare: A case study of disease prediction algorithm integration. Journal of biomedical informatics, 57, 377-385. doi:http://dx.doi.org/10.1016/j.jbi.2015.07.017 Freitas, A., Silva-Costa, T., Lopes, F., Garcia-Lema, I., Teixeira-Pinto, A., Brazdil, P. & Costa-Pereira, A. (2012). Factors influencing hospital high length of stay outliers. BMC Health Services Research, 12(1), 265. doi:10.1186/1472-6963-12- 265 Friesen-Storms, J. H. H. M., Bours, G. J. J. W., van der Weijden, T. & Beurskens, A. J. H. M. (2015). Shared decision making in chronic care in the context of evidence based practice in nursing. International Journal of Nursing Studies, 52(1), 393- 402. doi:http://dx.doi.org/10.1016/j.ijnurstu.2014.06.012 Frost, J. & Massagli, M. (2009). PatientsLikeMe the case for a data-centered patient community and how ALS patients use the community to inform treatment decisions and manage pulmonary health. Chronic Respiratory Disease, 6(4), 225- 229. doi:10.1177/1479972309348655 Gallego, B., Walter, S. R., Day, R. O., Dunn, A. G., Sivaraman, V., Shah, N., . . . Coiera, E. (2015). Bringing cohort studies to the bedside: framework for a ‘green button’ to support clinical decision-making. Journal of Comparative Effectiveness Research, 4(3), 191-197. doi:10.2217/cer.15.12 Gavrielov-Yusim, N. & Friger, M. (2014). Use of administrative medical databases in population-based research. Journal of Epidemiology and Community Health, 68(3), 283-287. doi:10.1136/jech-2013-202744 Gilstrap, L. G. & Joynt, K. E. (2014). Understanding the Relationship Between Readmission and Quality of Hospital Care in Heart Failure. Current Heart Failure Reports, 11(4), 347-353. doi:10.1007/s11897-014-0209-5 Goldzweig, C. L., Towfigh, A., Maglione, M. & Shekelle, P. G. (2009). Costs and benefits of health information technology: new trends from the literature. Health affairs, 28(2), w282-w293.

242 References

Grant, J. S. & Davis, L. L. (1997). Selection and use of content experts for instrument development. Research in Nursing Health, 20(3), 269-274. doi:10.1002/(SICI)1098-240X(199706)20:3<269::AID-NUR9>3.0.CO;2-G Green, D. & Latchford, G. (2012). Maximising the benefits of psychotherapy: A practice-based evidence approach: John Wiley & Sons. Greenhalgh, T., Howick, J. & Maskrey, N. (2014). Evidence based medicine: a movement in crisis? BMJ : British Medical Journal, 348, g3725. doi:10.1136/bmj.g3725 Grimes, D. A. (2010). Epidemiologic research using administrative databases: garbage in, garbage out. Obstetrics & Gynecology, 116(5), 1018-1019. Guyatt, G., Cairns, J., Churchill, D., Cook, D., Haynes, B., Hirsh, J., . . . Tugwell, P. (1992). Evidence-based medicine: A new approach to teaching the practice of medicine. Jama, 268(17), 2420-2425. doi:10.1001/jama.1992.03490170092032 Guyatt, G., Cook, D. & Haynes, B. (2004). Evidence based medicine has come a long way. BMJ: British Medical Journal, 329(7473), 990-991. doi:10.1136/bmj.329.7473.990 Hackshaw, A. (2008). Small studies: strengths and limitations. European Respiratory Journal, 32(5), 1141-1143. doi:10.1183/09031936.00136408 Haluza, D. & Jungwirth, D. (2015). ICT and the future of health care: aspects of health promotion. International Journal of Medical Informatics, 84(1), 48-57. doi:http://dx.doi.org/10.1016/j.ijmedinf.2014.09.005 Hamoud, A. K. & Obaid, T. (2014). Using OLAP with Diseases Registry Warehouse for Clinical Decision Support. doi:https://dx.doi.org/10.2139/ssrn.3061597 Häyrinen, K., Saranto, K. & Nykänen, P. (2008). Definition, structure, content, use and impacts of electronic health records: a review of the research literature. International Journal of Medical Informatics, 77(5), 291-304. doi:10.1016/j.ijmedinf.2007.09.001 Heng, B. H., Sun, Y., Cheah, J. T. & Jong, M. (2010). The Singapore national healthcare group diabetes registry—Descriptive epidemiology of type 2 diabetes mellitus. Annals Academy of Medicine Singapore, 39(5), 348. Henriksson, A., Kvist, M., Dalianis, H. & Duneld, M. (2015). Identifying adverse drug event information in clinical notes with distributional semantic representations of context. Journal of biomedical informatics, 57, 333-349. doi:https://doi.org/10.1016/j.jbi.2015.08.013 Henriques, R., Pina, S. & Antunes, C. (2013). Temporal mining of integrated healthcare data: Methods, revealings and implications. SDM IW on data mining for medicine and healthcare, 52-60. Hersh, W. R., Weiner, M. G., Embi, P. J., Logan, J. R., Payne, P. R., Bernstam, E. V., . . . Cimino, J. J. (2013). Caveats for the use of operational electronic health record data in comparative effectiveness research. Medical care, 51(8 Suppl 3), S30-S37. doi:10.1097/MLR.0b013e31829b1dbd Hillestad, R., Bigelow, J., Bower, A., Girosi, F., Meili, R., Scoville, R. & Taylor, R. (2005). Can Electronic Medical Record Systems Transform Health Care? Potential Health Benefits, Savings, And Costs. Health affairs, 24(5), 1103-1117. doi:10.1377/hlthaff.24.5.1103 Hoerbst, A. & Ammenwerth, E. (2010). Electronic health records. Methods of Information in Medicine, 49(4), 320-336. doi:10.3414/ME10-01-0038

K1. Frequency and percentage distribution of EHR systems use and perceived clinical benefits.References 243

Hoffmann, T., Bennett, S. & Del Mar, C. (2009). Introduction to evidence-based practice. In Evidence-based practice across the health professions (pp. 1-15). Australia: Elsevier Australia. Holden, R. R. (2010). Face Validity. In The Corsini Encyclopedia of Psychology: John Wiley & Sons, Inc. Holzinger, A. (2005). Usability engineering methods for software developers. Commun. ACM, 48(1), 71-74. doi:10.1145/1039539.1039541 Horn, S. D. & Gassaway, J. (2007). Practice-based evidence study design for comparative effectiveness research. Medical care, 45(10), S50-S57. doi:10.1097/MLR.0b013e318070c07b Horn, S. D. & Gassaway, J. (2010). Practice Based Evidence: Incorporating Clinical Heterogeneity and Patient-Reported Outcomes for Comparative Effectiveness Research. Medical care, 48(6), S17-S22. doi:10.1097/MLR.0b013e3181d57473 Horn, S. D., Gassaway, J., Pentz, L. & James, R. (2010). Practice-based evidence for clinical practice improvement: an alternative study design for evidence-based medicine. Studies in health technology and informatics, 151, 446-460. Houser, J. & Oman, K. S. (2010). Evidence-Based Practice in Health Care. In Evidence-Based Practice An Implementation Guide for Healthcare Organisations (pp. 1-20). Sudbury, MA: Jones & Bartlett Publishers. Hoyt, R. & Hersh, W. (2014). Evidence based medicine and clinical practice guidelines. Health Informatics: Practical Guide for Healthcare and Information Technology Professionals. 6th ed. Informatics Education, 323-350. Hsieh, H.-F. & Shannon, S. E. (2005). Three approaches to qualitative content analysis. Qualitative health research, 15(9), 1277-1288. doi:10.1177/1049732305276687 Hu, F. B., Satija, A. & Manson, J. E. (2015). Curbing the diabetes pandemic: The need for global policy solutions. Jama, 313(23), 2319-2320. doi:10.1001/jama.2015.5287 Huang, Y.-Q., Gou, R., Diao, Y.-S., Yin, Q.-H., Fan, W.-X., Liang, Y.-P., . . . Liu, F. (2014). Charlson comorbidity index helps predict the risk of mortality for patients with type 2 diabetic nephropathy. Journal of Zhejiang University SCIENCE B, 15(1), 58-66. doi:10.1631/jzus.B1300109 Hunink, M. M., Weinstein, M. C., Wittenberg, E., Drummond, M. F., Pliskin, J. S., Wong, J. B. & Glasziou, P. P. (2014). Decision making in health and medicine: integrating evidence and values: Cambridge University Press. Ibrahim, H., Saad, A., Abdo, A. & Sharaf Eldin, A. (2016). Mining association patterns of drug-interactions using post marketing FDA’s spontaneous reporting data. Journal of biomedical informatics, 60, 294-308. doi:https://doi.org/10.1016/j.jbi.2016.02.009 IHiS. (2011). Transforming Healthcare with IT Innovation - IHiS Yearbook 2011. Retrieved from https://www.ihis.com.sg/MediaCentre/publications/Documents/IHiS_Yearbook. pdf IHiS. (2015). Integrated Health Information Systems. Retrieved from https://www.ihis.com.sg/Pages/IHIS.aspx

244 References

IHiS. (2016). Latest IT Systems at Public Healthcare Institutions in Singapore. Retrieved from https://www.ihis.com.sg/MediaCentre/Documents/Latest%20IT%20Systems%2 0at%20PHIs%20in%20Singapore.pdf Infor. (2016). Infor Cloverleaf Integration Suite. Retrieved from http://www.infor.com/product-summary/healthcare/cloverleaf-integration-suite/ International Diabetes Federation. (2015). IDF Diabetes Atlas 7th Edition, Country Report Singapore. Retrieved from http://reports.instantatlas.com/report/view/846e76122b5f476fa6ef09471965aedd /SGP International Diabetes Federation. (2017). IDF Western Pacific members. Retrieved from https://www.idf.org/our-network/regions-members/western- pacific/members/113-singapore.html ISO. (1998). Ergonomic requirements for office work with visual display terminals (vdts) - Part 11: Guidance on usability. In Part 11: Guidance on usability (Vol. ISO 9241-11:1998). Geneva, Switzerland. ISO. (2018). ISO/TC 215. Retrieved from https://www.iso.org/committee/54960/x/catalogue/p/1/u/0/w/0/d/0 Jamshed, S. (2014). Qualitative research method-interviewing and observation. Journal of Basic and Clinical Pharmacy, 5(4), 87-88. doi:10.4103/0976- 0105.141942 Jang, D., Lee, S., Lee, J., Kim, K. & Lee, D. (2016). Inferring new drug indications using the complementarity between clinical disease signatures and drug effects. Journal of biomedical informatics, 59, 248-257. doi:https://doi.org/10.1016/j.jbi.2015.12.003 Jao, C. S., Hier, D. B. & Galanter, W. L. (2008, 12-15 Oct. 2008). Using clinical decision support to maintain medication and problem lists A pilot study to yield higher patient safety. Paper presented at the Systems, Man and Cybernetics, 2008. SMC 2008. IEEE International Conference on. Jha, A. K., Doolan, D., Grandt, D., Scott, T. & Bates, D. W. (2008). The use of health information technology in seven nations. International Journal of Medical Informatics, 77(12), 848-854. doi:10.1016/j.ijmedinf.2008.06.007 Johnson, S. G., Speedie, S., Simon, G., Kumar, V. & Westra, B. L. (2015). A Data Quality Ontology for the Secondary Use of EHR Data. AMIA Annual Symposium Proceedings, 2015, 1937-1946. Jonnagaddala, J., Liaw, S.-T., Ray, P., Kumar, M., Chang, N.-W. & Dai, H.-J. (2015). Coronary artery disease risk assessment from unstructured electronic health records using text mining. Journal of biomedical informatics, 58, S203-S210. doi:https://doi.org/10.1016/j.jbi.2015.08.003 Kahn, M. G., Callahan, T. J., Barnard, J., Bauck, A. E., Brown, J., Davidson, B. N., . . . Schilling, L. (2016). A Harmonized Data Quality Assessment Terminology and Framework for the Secondary Use of Electronic Health Record Data. eGEMs, 4(1), 1244. doi:10.13063/2327-9214.1244 Kaplan, B. & Maxwell, J. A. (2005). Qualitative research methods for evaluating computer information systems. Evaluating the Organizational Impact of Healthcare Information Systems, 30-55.

K1. Frequency and percentage distribution of EHR systems use and perceived clinical benefits.References 245

Karystianis, G., Dehghan, A., Kovacevic, A., Keane, J. A. & Nenadic, G. (2015). Using local lexicalized rules to identify heart disease risk factors in clinical notes. Journal of biomedical informatics, 58, S183-S188. doi:https://doi.org/10.1016/j.jbi.2015.06.013 Kendall, E., Sunderland, N., Muenchberger, H. & Armstrong, K. (2009). When guidelines need guidance: considerations and strategies for improving the adoption of chronic disease evidence by general practitioners. Journal of Evaluation in Clinical Practice, 15(6), 1082-1090. doi:10.1111/j.1365- 2753.2009.01147.x Khoo, H. S., Lim, Y. W. & Vrijhoef, H. J. (2014). Primary healthcare system and practice characteristics in Singapore. Asia Pacific family medicine, 13(1), 8-8. doi:10.1186/s12930-014-0008-x King, J., Patel, V., Jamoom, E. W. & Furukawa, M. F. (2014). Clinical Benefits of Electronic Health Record Use: National Findings. Health services research, 49(1pt2), 392-404. doi:10.1111/1475-6773.12135 Kitzinger, J. (1995). Qualitative research. Introducing focus groups. BMJ : British Medical Journal, 311(7000), 299-302. Kossovsky, M. P., Sarasin, F. P., Chopard, P., Louis-Simonet, M., Sigaud, P., Perneger, T. V. & Gaspoz, J.-M. (2002). Relationship between hospital length of stay and quality of care in patients with congestive heart failure. Quality and Safety in Health Care, 11(3), 219-223. doi:10.1136/qhc.11.3.219 Krishnan, K. (2013). Data Warehousing in the Age of Big Data: Newnes. Kwon, O., Lee, N. & Shin, B. (2014). Data quality management, data usage experience and acquisition intention of big data analytics. International Journal of Information Management, 34(3), 387-394. doi:http://dx.doi.org/10.1016/j.ijinfomgt.2014.02.002 Laor, A., Tal, S., Guller, V., Zbar, A. P. & Mavor, E. (2016). The Charlson Comorbidity Index (CCI) as a Mortality Predictor after Surgery in Elderly Patients. The American Surgeon, 82(1), 22-27. Leitheiser, R. L. (2001). Data quality in health care data warehouse environments. Paper presented at the 34th Annual Hawaii International Conference on System Sciences 2001. Longhurst, C. A., Harrington, R. A. & Shah, N. H. (2014). A ‘Green Button’ For Using Aggregate Patient Data At The Point Of Care. Health affairs, 33(7), 1229-1235. doi:10.1377/hlthaff.2014.0099 Lu, J. & Keech, M. (2015, 1-4 Sept. 2015). Emerging Technologies for Health Data Analytics Research: A Conceptual Architecture. Paper presented at the 2015 26th International Workshop on Database and Expert Systems Applications (DEXA). Lyman, J. A., Scully, K. & Harrison Jr, J. H. (2008). The development of health care data warehouses to support data mining. Clinics in Laboratory Medicine, 28(1), 55-71. doi:10.1016/j.cll.2007.10.003 Martin-Sanchez, F. J., Aguiar-Pulido, V., Lopez-Campos, G. H., Peek, N. & Sacchi, L. (2017). Secondary Use and Analysis of Big Data Collected for Patient Care. Yearb Med Inform, 26(01), 28-37. doi:10.15265/IY-2017-008 McKibbon, K. (1998). Evidence-based practice. Bulletin of the Medical Library Association, 86(3), 396.

246 References

Ministry of Health Singapore. (2010). National Health Survey 2010. Retrieved from http://www.moh.gov.sg/content/dam/moh_web/Publications/Reports/2011/NHS 2010%20-%20low%20res.pdf Ministry of Health Singapore. (2014). Hospital Admission Rates* By Age And Sex, 2014. Retrieved from: https://www.moh.gov.sg/resources-statistics/healthcare- institution-statistics/hospital-admission-rates-by-age-and-sex/hospital- admission-rates-by-age-and-sex-2014 Ministry of Health Singapore. (2016a). Hospital Admission Rates* by Age and Sex, 2016p. Retrieved from: https://www.moh.gov.sg/resources-statistics/healthcare- institution-statistics/hospital-admission-rates-by-age-and-sex/hospital- admission-rates-by-age-and-sex-2016p Ministry of Health Singapore. (2016b). Hospital Services. Retrieved from https://www.moh.gov.sg/content/moh_web/home/our_healthcare_system/Health care_Services/Hospitals.html Ministry of Health Singapore. (2017). Reorganisation of healthcare system into three integrated clusters to better meet future healthcare needs [Press release]. Retrieved from https://corp.nhg.com.sg/Media%20Releases/Press%20Release%20on%20reorga nisation%20of%20healthcare%20system%20(without%20embargo).pdf Ministry of Health Singapore. (2018). Primary Healthcare Services. Retrieved from https://www.moh.gov.sg/our-healthcare-system/healthcare-services-and- facilities/primary-healthcare-services Mishra, D., Yazici, A. & Basaran, B. P. (2008, 4-6 Aug. 2008). A casestudy of data models in data warehousing. Paper presented at the 2008 First International Conference on the Applications of Digital Information and Web Technologies (ICADIWT). Musen, M. A., Middleton, B. & Greenes, R. A. (2014). Clinical decision-support systems. Biomedical informatics, 643-674. Myers, E. R. & Steege, J. F. (1999). Risk adjustment for complications of hysterectomy: Limitations of routinely collected administrative data. American Journal of Obstetrics & Gynecology, 181(3), 567-575. doi:10.1016/S0002- 9378(99)70494-1 Myers, M. D. & Avison, D. (1997). Qualitative research in information systems. Management Information Systems Quarterly, 21, 241-242. Newman, L., Harris, V., Evans, L. J. & Beck, A. (2018). Factors Associated with Length of Stay in Psychiatric Inpatient Services in London, UK. Psychiatric Quarterly, 89(1), 33-43. doi:10.1007/s11126-017-9498-7 Ng, S. C., Toh, M. P. H. S., Ko, Y. & Lee, Y.-C. J. (2015). Direct Medical Cost of Type 2 Diabetes in Singapore. PLOS ONE, 10(3), e0122795. doi:10.1371/journal.pone.0122795 Nguyen, L., Bellucci, E. & Nguyen, L. T. (2014). Electronic health records implementation: An evaluation of information system impact and contingency factors. International Journal of Medical Informatics, 83(11), 779-796. doi:10.1016/j.ijmedinf.2014.06.011 Nielsen, J. (1994). Heuristic evaluation. Usability inspection methods, 17(1), 25-62. NUHS. (2016). Corporate Profile. Retrieved from http://www.nuhs.edu.sg/about- us/corporate-profile.html#

K1. Frequency and percentage distribution of EHR systems use and perceived clinical benefits.References 247

Onitilo, A. A., Stankowski, R. V., Berg, R. L., Engel, J. M., Williams, G. M. & Doi, S. A. (2014). A novel method for studying the temporal relationship between type 2 diabetes mellitus and cancer using the electronic medical record. BMC Medical Informatics and Decision Making, 14(1), 38. doi:10.1186/1472-6947-14-38 Orfanidis, L., Bamidis, P. D. & Eaglestone, B. (2004). Data quality issues in electronic health records: an adaptation framework for the Greek health system. Health informatics journal, 10(1), 23-36. doi:10.1177/1460458204040665 Osop, H. & Sahama, T. (2015). Data driven and practice-based evidence: Design and development of efficient and effective clinical decision support system. In Improving Health Management through Clinical Decision Support Systems. Hershey, PA: IGI Global Publishing. Osop, H. & Sahama, T. (2016a). Electronic Health Records: Improvement to Healthcare Decision-Making. Paper presented at the 18th International Conference on E-Health Networking, Application & Services (IEEE Healthcom'16), Munich, Germany. Osop, H. & Sahama, T. (2016b). Quality Evidence, Quality Decisions: Ways to Improve Security and Privacy of EHR Systems. Paper presented at the 18th International Conference on E-Health Networking, Application & Services (IEEE Healthcom'16), Munich, Germany. Parshutin, S. & Kirshners, A. (2013). Research on clinical decision support systems development for atrophic gastritis screening. Expert Systems with Applications, 40(15), 6041-6046. Paz, F., Paz, F. A., Villanueva, D. & Pow-Sang, J. A. (2015, 13-15 April 2015). Heuristic Evaluation as a Complement to Usability Testing: A Case Study in Web Domain. Paper presented at the 2015 12th International Conference on Information Technology - New Generations. Pedersen, T. B. & Jensen, C. S. (1998). Research issues in clinical data warehousing. Paper presented at the Tenth International Conference on Scientific and Statistical Database Management, 1998. Pentaho. (2016). Pentaho - Data Integration. Retrieved from http://www.pentaho.com/product/data-integration Poon Chian, H. (2018, 18 January 2017). Public healthcare sector to be reorganised into 3 integrated clusters, new polyclinic group to be formed. The Straits Times Singapore. Retrieved from https://www.straitstimes.com/singapore/health/public-healthcare-sector-to-be- reorganised-into-3-integrated-clusters-new Poon, E. G., Wright, A., Simon, S. R., Jenter, C. A., Kaushal, R., Volk, L. A., . . . Bates, D. W. (2010). Relationship Between Use of Electronic Health Record Features and Health Care Quality: Results of a Statewide Survey. Medical care, 48(3), 203-209. doi:10.1097/MLR.0b013e3181c16203 Porter, S. R. & Whitcomb, M. E. (2003). The Impact of Contact Type on Web Survey Response Rates. The Public Opinion Quarterly, 67(4), 579-588. Prather, J. C., Lobach, D. F., Goodwin, L. K., Hales, J. W., Hage, M. L. & Hammond, W. E. (1997). Medical data mining: knowledge discovery in a clinical data warehouse. Paper presented at the American Medical Informatics Association annual fall symposium.

248 References

Prior, M., Guerin, M. & Grimmer-Somers, K. (2008). The effectiveness of clinical guideline implementation strategies – a synthesis of systematic review findings. Journal of Evaluation in Clinical Practice, 14(5), 888-897. doi:10.1111/j.1365- 2753.2008.01014.x Qureshi, S. R. & Gupta, A. (2014, 8-9 March 2014). Towards efficient Big Data and data analytics: A review. Paper presented at the 2014 Conference on IT in Business, Industry and Government (CSIBIG). Raghupathi, W. & Raghupathi, V. (2013). An overview of health analytics. Journal of Health & Medical Informatics, 2013. Rieman, J., Franzke, M. & Redmiles, D. (1995). Usability evaluation with the cognitive walkthrough. Paper presented at the Conference Companion on Human Factors in Computing Systems, Denver, Colorado, USA. Robelia, P. M., Kashiwagi, D. T., Jenkins, S. M., Newman, J. S. & Sorita, A. (2017). Information Transfer and the Hospital Discharge Summary: National Primary Care Provider Perspectives of Challenges and Opportunities. The Journal of the American Board of Family Medicine, 30(6), 758-765. doi:10.3122/jabfm.2017.06.170194 Roelofs, E., Persoon, L., Nijsten, S., Wiessler, W., Dekker, A. & Lambin, P. (2013). Benefits of a clinical data warehouse with data mining tools to collect data for a radiotherapy trial. Radiotherapy and Oncology, 108(1), 174-179. doi:10.1016/j.radonc.2012.09.019 Roffman, C. E., Buchanan, J. & Allison, G. T. (2016). Charlson Comorbidities Index. Journal of Physiotherapy, 62(3), 171. doi:10.1016/j.jphys.2016.05.008 Rosson, M. B. & Carroll, J. M. (2002). Usability engineering: scenario-based development of human-computer interaction: Morgan Kaufmann. Rosson, M. B. & Carroll, J. M. (2009). Scenario based design. Human‐computer interaction. Boca Raton, FL, 145-162. Rousidis, D., Garoufallou, E., Balatsoukas, P. & Sicilia, M.-A. (2014). Data Quality Issues and Content Analysis for Research Data Repositories: The Case of Dryad. Paper presented at the Let’s Put Data to Use: Digital Scholarship for the Next Generation, 18th International Conference on Electronic Publishing, Thessaloniki, Greece. Rousseau, N., McColl, E., Newton, J., Grimshaw, J. & Eccles, M. (2003). Practice based, longitudinal, qualitative interview study of computerised evidence based guidelines in primary care. Bmj, 326(7384), 314. Rubio, D. M., Berg-Weger, M., Tebb, S. S., Lee, E. S. & Rauch, S. (2003). Objectifying content validity: Conducting a content validity study in social work research. Social Work Research, 27(2), 94-104. doi:10.1093/swr/27.2.94 Sackett, D. L., Rosenberg, W., Gray, J., Haynes, R. B. & Richardson, W. S. (1996). Evidence based medicine: what it is and what it isn't. British Medical Journal, 312(7023), 71-72. Sahama, T. & Croll, P. (2007). A data warehouse architecture for clinical data warehousing. Paper presented at the Australasian Workshop on Health Knowledge Management and Discovery (HKMD 2007) CRPIT, Ballarat, Victoria.

K1. Frequency and percentage distribution of EHR systems use and perceived clinical benefits.References 249

Sahama, T., Simpson, L. & Lane, B. (2013). Security and privacy in eHealth: is it possible? A sociotechnical analysis. Paper presented at the 15th International Conference on e-Health Networking, Applications and Services, Lisbon, Portugal. Salive, M. E. (2013). Multimorbidity in older adults. Epidemiologic reviews, 35(1), 75-83. doi:10.1093/epirev/mxs009 Scheese, R. (1998). Data warehousing as a healthcare business solution. Healthcare financial management: journal of the Healthcare Financial Management Association, 52(2), 56-59. Schneeweiss, S. & Avorn, J. (2005). A review of uses of health care utilization databases for epidemiologic research on therapeutics. Journal of Clinical Epidemiology, 58(4), 323-337. doi:https://doi.org/10.1016/j.jclinepi.2004.10.012 Schnipper, L. E., Smith, T. J., Raghavan, D., Blayney, D. W., Ganz, P. A., Mulvey, T. M. & Wollins, D. S. (2012). American Society of Clinical Oncology identifies five key opportunities to improve care and reduce costs: The top five list for oncology. Journal of Clinical Oncology, 30(14), 1715-1724. doi:10.1200/JCO.2012.42.8375 Secginli, S., Erdogan, S. & Monsen, K. A. (2014). Attitudes of health professionals towards electronic health records in primary health care settings: a questionnaire survey. Informatics for Health and Social Care, 39(1), 15-32. doi:10.3109/17538157.2013.834342 Sen, A. & Sinha, A. P. (2005). A comparison of data warehousing methodologies. Communications of the ACM, 48(3), 79-84. doi:10.1145/1047671.1047673 Shastri, A., Bangar, S., Waldman, S., Esfahani, E. & Brindle, N. (2014). Content and timing of inpatient discharge summaries at the mount. British Journal of Medical Practitioners, 7(3), 37-44. Simon, S. R., Soran, C. S., Kaushal, R., Jenter, C. A., Volk, L. A., Burdick, E., . . . Bates, D. W. (2009). Physicians' Use of Key Functions in Electronic Health Records from 2005 to 2007: A Statewide Survey. Journal of the American Medical Informatics Association, 16(4), 465-470. doi:http://dx.doi.org/10.1197/jamia.M3081 Sinha, P., Sunder, G., Bendale, P., Mantri, M. & Dande, A. (2012). Singapore's NEHR. Electronic Health Record: Standards, Coding Systems, Frameworks, and Infrastructures, 259-266. Sittig, D. F., Wright, A., Osheroff, J. A., Middleton, B., Teich, J. M., Ash, J. S., . . . Bates, D. W. (2008). Grand challenges in clinical decision support. Journal of biomedical informatics, 41(2), 387-392. doi:https://doi.org/10.1016/j.jbi.2007.09.003 Smith, M., Higgs, J. & Ellis, E. (2008). Factors influencing clinical decision making. Clinical reasoning in the health professions. 3rd ed. Amsterdam: Butterworth Heinemann, 89-100. Straus, S. E. & McAlister, F. A. (2000). Evidence-based medicine: a commentary on common criticisms. Canadian Medical Association Journal, 163(7), 837-841. Stringer, E. & Bernatsky, S. (2015). Validity of juvenile idiopathic arthritis diagnoses using administrative health data. Rheumatology International, 35(3), 575-579. doi:10.1007/s00296-014-3142-8

250 References

Su, C.-H., Chiu, H.-C., Hsieh, H.-M., Yen, J.-Y., Lee, M.-H., Li, C.-Y., . . . Huang, C.-J. (2016). Healthcare Utilization and Expenditures for Persons with Diabetes Comorbid with Mental Illnesses. Psychiatric Quarterly, 87(3), 545-557. doi:10.1007/s11126-015-9408-9 Sujansky, W. (2001). Heterogeneous Database Integration in Biomedicine. Journal of biomedical informatics, 34(4), 285-298. doi:http://dx.doi.org/10.1006/jbin.2001.1024 Swain, M. J. & Kharrazi, H. (2015). Feasibility of 30-day hospital readmission prediction modeling based on health information exchange data. International Journal of Medical Informatics, 84(12), 1048-1056. doi:http://dx.doi.org/10.1016/j.ijmedinf.2015.09.003 Taleb, A. R. A., Hoque, M., Hasanat, A. & Khan, M. B. (2017, 21-23 Feb. 2017). Application of data mining techniques to predict length of stay of stroke patients. Paper presented at the 2017 International Conference on Informatics, Health & Technology (ICIHT). Tan, P. & Ong, L. S. (2009). Singapore's National Electronic Health Record Architecture. Paper presented at the SOA in Healthcare Conference, Chicago, IL USA. Thabane, L., Ma, J., Chu, R., Cheng, J., Ismaila, A., Rios, L. P., . . . Goldsmith, C. H. (2010). A tutorial on pilot studies: the what, why and how. BMC Medical Research Methodology, 10(1), 1. doi:10.1186/1471-2288-10-1 The Australian Hospital Healthcare Bulletin. (2018). A Framework For Success. Retrieved from https://www.hospitalhealth.com.au/content/facility- admin/article/a-framework-for-success-1319110333#axzz5WiA4XDgA Tickoo, S., Bhardwaj, A., Fonarow, G. C., Liang, L., Bhatt, D. L. & Cannon, C. P. (2016a). Relation Between Hospital Length of Stay and Quality of Care in Patients With Acute Coronary Syndromes (from the American Heart Association's Get With the Guidelines—Coronary Artery Disease Data Set). The American Journal of Cardiology, 117(2), 201-205. doi:http://dx.doi.org/10.1016/j.amjcard.2015.10.027 Tickoo, S., Bhardwaj, A., Fonarow, G. C., Liang, L., Bhatt, D. L. & Cannon, C. P. (2016b). Relation Between Hospital Length of Stay and Quality of Care in Patients With Acute Coronary Syndromes (from the American Heart Association's Get With the Guidelines—Coronary Artery Disease Data Set). American Journal of Cardiology, 117(2), 201-205. doi:10.1016/j.amjcard.2015.10.027 Tunis, S. R., Stryer, D. B. & Clancy, C. M. (2003). Practical clinical trials: Increasing the value of clinical research for decision making in clinical and health policy. Jama, 290(12), 1624-1632. doi:10.1001/jama.290.12.1624 Turner, D. W. (2010). Qualitative interview design: A practical guide for novice investigators. The qualitative report, 15(3), 754-760. van Ginneken, A. M. (2002). The computerized patient record: balancing effort and benefit. International Journal of Medical Informatics, 65(2), 97-119. doi:10.1016/S1386-5056(02)00007-2 Walsh, C. & Hripcsak, G. (2014). The effects of data sources, cohort selection, and outcome definition on a predictive model of risk of thirty-day hospital readmissions. Journal of biomedical informatics, 52, 418-426. doi:http://dx.doi.org/10.1016/j.jbi.2014.08.006

K1. Frequency and percentage distribution of EHR systems use and perceived clinical benefits.References 251

Wasey, J. O. (2016). icd: Tools for Working with ICD-9 and ICD-10 Codes, and Finding Comorbidities. R package version 2.0.1. Retrieved from http://CRAN.R- project.org/package=icd Weiskopf, N. G. & Weng, C. (2013). Methods and dimensions of electronic health record data quality assessment: enabling reuse for clinical research. Journal of the American Medical Informatics Association, 20(1), 144-151. doi:10.1136/amiajnl- 2011-000681 Welsh, E. (2002). Dealing with Data: Using NVivo in the Qualitative Data Analysis Process. Forum Qualitative Sozialforscgung / Forum: Qualitative Social Research, 3(2). White, R. W., Wang, S., Pant, A., Harpaz, R., Shukla, P., Sun, W., . . . Horvitz, E. (2016). Early identification of adverse drug reactions from search log data. Journal of biomedical informatics, 59, 42-48. doi:https://doi.org/10.1016/j.jbi.2015.11.005 Wickham, M. & Woods, M. (2005). Reflecting on the Strategic Use of CAQDAS to Manage and Report on the Qualitative Research Process. The qualitative report, 10(4), 687-702. Wicks, P., Massagli, M., Frost, J., Brownstein, C., Okun, S., Vaughan, T., . . . Heywood, J. (2010). Sharing Health Data for Better Outcomes on PatientsLikeMe. Journal of Medical Internet Research, 12(2), e19. doi:10.2196/jmir.1549 Wills, M. J. (2014). Decisions Through Data: Analytics in Healthcare. Journal of Healthcare Management, 59(4), 254-262. Winkler, W. E. (2006). Overview of record linkage and current research directions. Paper presented at the Bureau of the Census. Wisniewski, M. F., Kieszkowski, P., Zagorski, B. M., Trick, W. E., Sommers, M., Weinstein, R. A. & Project, C. A. R. (2003). Development of a clinical data warehouse for hospital infection control. Journal of the American Medical Informatics Association, 10(5), 454-462. doi:10.1197/jamia.M1299. Woolf, S. H., Grol, R., Hutchinson, A., Eccles, M. & Grimshaw, J. (1999). Potential benefits, limitations, and harms of clinical guidelines. British Medical Journal, 318(7182), 527. Woon, B. (2014). [Singapore]Health Database in an IT Society(). Japan Medical Association Journal : JMAJ, 57(4), 212-215. World Health Organization. (2015). Noncommunicable diseases Country Profile 2014. Retrieved from http://apps.who.int/iris/bitstream/10665/128038/1/9789241507509_eng.pdf?ua= 1 World Health Organization. (2016). Classification of Diseases (ICD). Retrieved from http://www.who.int/classifications/icd/en/ World Health Organization. (2018). Global Health Observatory (GHO) data. Retrieved from: http://www.who.int/gho/ncd/mortality_morbidity/en/ Yang, Y., Kankanhalli, A. & Chandran, S. (2015). A stage model of information technology in healthcare. Health and Technology, 5(1), 1-11. doi:10.1007/s12553-015-0097-z Yeo, S. Q., Harris, M. & Majeed, F. A. (2012). Integrated care for diabetes—a Singapore approach. International journal of integrated care, 12(1).

252 References

Zakaria, N., Yusof, S. A. M. & Zakaria, N. (2010). Managing ICT in Healthcare Organization: Culture, Challenges, and Issues of Technology Adoption and Implementation. In Handbook of Research on Advances in Health Informatics and Electronic Healthcare Applications: Global Adoption and Impact of Information Communication Technologies (pp. 153). Hershey, PA: IGI Global. Zhang, Y. & Wildemuth, B. M. (2016). Qualitative Analysis of Content. In Applications of Social Research Methods to Questions in Information and Library Science (Second ed., pp. 318).

K1. Frequency and percentage distribution of EHR systems use and perceived clinical benefits.References 253

Appendices

Appendix A. Table of direct medical costs of diabetes mellitus paid by the hospital (Rieman et al., 1995)

254 Appendices

Appendix B. Table of prevalence of Diabetes-related comorbidities and complications in Singapore from 2005 to 2008 (ISO, 1998)

Appendices 255

Appendix C. Paradigm of EBP

Figure explains the integration of information from four sources of clinical expertise, research evidence, practice context and client’s values in the paradigm of EBP by Hoffmann et al. (2009, p. 3).

256 Appendices

Appendix D. Evidence Pyramid

Figure illustrates the evidence pyramid by Hoyt & Hersh (2014).

Appendices 257

Appendix E. Interview questions for IT Professionals Relating to systems administration :

1. What are the clinical/EHR systems currently being used in the hospital? a. Are there multiple systems supporting a single clinical department or specialty?

2. Which clinical/EHR systems are used by healthcare professionals to manage chronic disease? a. Are these systems also used by doctors who do not manage patients with chronic disease?

3. Is there any interface systems which consolidate all information from various systems to display it for healthcare professionals?

Relating to database administration :

1. How many database systems are used to store the healthcare and medical records of patients? a. Are the data stored in a single or multiple databases? b. Are these systems cross-platform compatible?

2. What is the database structure of the clinical systems that stores patient data?

3. Is there any prior implementation of a data warehouse? a. What is the use of the data warehouse? b. Can a new data mart be created easily to extract patient health and healthcare record?

Relating to networks :

1. Is there any prior implementation of a data warehouse? a. How are data being extracted, transformed and loaded into a data warehouse?

258 Appendices

Appendix F. Data access and use agreement

Appendices 259

260 Appendices

Appendix G. Finalised Survey Question

Appendices 261

262 Appendices

Appendices 263

264 Appendices

Appendices 265

266 Appendices

Appendices 267

Appendix H. Survey Email Invitation Template

268 Appendices

Appendix I. Ethics Application

Appendices 269

270 Appendices

Appendices 271

272 Appendices

Appendices 273

274 Appendices

Appendices 275

276 Appendices

Appendix J. Ethics Clearance Certification

Appendices 277

278 Appendices

Appendix K. Frequency and Distribution K1. Frequency and percentage distribution of EHR systems use and perceived clinical benefits.

Table of EHR System Usage Trends Agree, used Agree, but used Agree, but never Agree, used routinely. Disagree Unsure occasionally rarely used before freq. % freq. % freq. % freq. % freq. % freq. % Mean SD UT1 10 33.3 6 20 7 23.3 3 10 4 13.3 0 0 2.500 1.408 UT2 20 66.7 3 10 3 10 1 3.3 3 10 0 0 1.800 1.349 UT3 1 3.4 1 6.9 3 10.3 5 17.2 18 62.1 0 0 4.276 1.131 UT4 29 96.7 0 0 1 3.3 0 0 0 0 0 0 1.067 0.365 UT5 22 73.3 3 10 3 10 1 3.3 1 3.3 0 0 1.533 1.042 UT6 19 63.3 3 10 4 13.3 2 6.7 2 6.7 0 0 1.833 1.289 UT7 25 83.3 2 6.7 2 6.7 1 3.3 0 0 0 0 1.300 0.750 UT8 30 100 0 0 0 0 0 0 0 0 0 0 1.000 0.000 UT9 30 100 0 0 0 0 0 0 0 0 0 0 1.000 0.000 UT10 16 53.3 4 13.3 5 16.7 3 10 2 6.7 0 0 2.033 1.326 UT11 22 75.9 3 10.3 1 3.4 1 3.4 2 6.9 0 0 1.552 1.183 UT12 28 93.3 2 6.7 0 0 0 0 0 0 0 0 1.067 0.254 UT13 29 96.7 1 3.3 0 0 0 0 0 0 0 0 1.033 0.183 UT14 28 93.3 1 3.3 1 3.3 0 0 0 0 0 0 1.100 0.403 UT15 27 93.1 1 3.4 1 3.4 0 0 0 0 0 0 1.103 0.409

Appendices 279

UT16 16 53.3 3 10 4 13.3 4 13.3 3 10 0 0 2.167 1.464 UT17 13 46.4 10 35.7 1 3.6 2 7.1 2 7.1 0 0 1.929 1.215 UT18 30 100 0 0 0 0 0 0 0 0 0 0 1.000 0.000 UT19 30 100 0 0 0 0 0 0 0 0 0 0 1.000 0.000 Table of EHR System Usage Trends Agree, used Agree, but used Agree, but never Agree, used routinely. Disagree Unsure occasionally rarely used before freq. % freq. % freq. % freq. % freq. % freq. % Mean SD UT20 30 100 0 0 0 0 0 0 0 0 0 0 1.000 0.000 UT21 28 93.3 1 3.3 1 3.3 0 0 0 0 0 0 1.100 0.403 UT22 30 100 0 0 0 0 0 0 0 0 0 0 1.000 0.000 Total 1.518 0.644

280 Appendices

K2. Frequency and percentage distribution of EHR systems use and perceived clinical benefits.

EHR System Use and Clinical Benefits Strongly Strongly agree Agree Neutral Disagree disagree freq. % freq. % freq. % freq. % freq. % Mean SD SF1 20 66.7 10 33.3 0 0.0 0 0.0 0 0.0 1.333 0.479 SF2 5 16.7 14 46.7 7 23.3 4 13.3 0 0.0 2.333 0.922 SF3 6 20.7 7 24.1 9 31.0 7 24.1 0 0.0 2.586 1.086 SF4 11 36.7 14 46.7 2 6.7 3 10.0 0 0.0 1.900 0.923 SF5 6 20.0 9 30.0 9 30.0 5 16.7 1 3.3 2.533 1.106 SF6 22 73.3 7 23.3 1 3.3 0 0.0 0 0.0 1.300 0.535 SF7 9 30.0 14 46.7 4 13.3 3 10.0 0 0.0 2.033 0.928 SF8 5 16.7 11 36.7 11 36.7 3 10.0 0 0.0 2.400 0.894 SF9 12 40.0 11 36.7 3 10.0 3 10.0 1 3.3 2.000 1.114 SF10 13 43.3 10 33.3 3 10.0 3 10.0 1 3.3 1.967 1.129 SF11 16 53.3 10 33.3 2 6.7 2 6.7 0 0.0 1.667 0.884 Total 2.005 0.909

Appendices 281

K3. Frequency and distribution of information believed to be contained in EHR data

EHR data Strongly Strongly agree Agree Neutral Disagree disagree freq. % freq. % freq. % freq. % freq. % Mean SD UI1 20 66.7 6 20.0 1 3.3 2 6.7 1 3.3 1.600 1.070 UI2 4 13.3 8 26.7 12 40.0 4 13.3 2 6.7 2.733 1.081 UI3 22 73.3 5 16.7 2 6.7 1 3.3 0 0.0 1.400 0.770 UI4 21 70.0 7 23.3 1 3.3 1 3.3 0 0.0 1.400 0.724 UI5 15 50.0 7 23.3 4 13.3 3 10.0 1 3.3 1.933 1.172 UI6 22 75.9 6 20.7 1 3.4 0 0.0 0 0.0 1.276 0.528 UI7 26 86.7 4 13.3 0 0.0 0 0.0 0 0.0 1.133 0.346 UI8 24 80.0 6 20.0 0 0.0 0 0.0 0 0.0 1.200 0.407 UI9 24 80.0 6 20.0 0 0.0 0 0.0 0 0.0 1.200 0.407 UI10 26 86.7 4 13.3 0 0.0 0 0.0 0 0.0 1.133 0.346 UI11 22 75.9 5 17.2 0 0.0 1 3.4 1 3.4 1.414 0.946 UI12 12 41.4 9 31.0 4 13.8 2 6.9 2 6.9 2.069 1.223 UI13 23 76.7 7 23.3 0 0.0 0 0.0 0 0.0 1.233 0.430 UI14 16 53.3 12 40.0 0 0.0 2 6.7 0 0.0 1.600 0.814 Total 1.523 0.733

K4. Frequency and percentage distribution of perceived EHR data quality

EHR data quality Strongly Strongly agree Agree Neutral Disagree disagree freq freq freq freq freq . % . % . % . % . % Mean SD DQU1 11 36.7 14 46.7 5 16.7 0 0.0 0 0.0 1.800 0.714 DQU2 14 46.7 9 30.0 4 13.3 2 6.7 1 3.3 1.900 1.094 DQU3 15 50.0 12 40.0 2 6.7 1 3.3 0 0.0 1.633 0.765 DQU4 10 33.3 11 36.7 6 20.0 3 10.0 0 0.0 2.067 0.980 DQU5 11 36.7 13 43.3 4 13.3 1 3.3 1 3.3 1.933 0.980 DQU6 12 40.0 11 36.7 6 20.0 1 3.3 0 0.0 1.867 0.860 Total 1.867 0.899

282 Appendices

K5. EHR data quality associated with clinical benefits

EHR data quality and Clinical Benefits Accurate Practical Relevant Reliable Secure Valid freq. % freq. % freq. % freq. % freq. % freq. % DQB1 21 70.0 14 46.7 10 33.3 11 36.7 8 26.7 7 23.3

DQB2 7 23.3 14 46.7 10 33.3 6 20.0 6 20.0 2 6.7

DQB3 10 33.3 10 33.3 12 40.0 6 20.0 5 16.7 5 16.7

DQB4 10 33.3 15 50.0 12 40.0 5 16.7 5 16.7 6 20.0

DQB5 10 33.3 15 50.0 12 40.0 5 16.7 5 16.7 6 20.0

DQB6 3 10.0 13 43.3 9 30.0 2 6.7 3 10.0 4 13.3

DQB7 5 16.7 10 33.3 13 43.3 4 13.3 4 13.3 1 3.3

DQB8 8 26.7 13 43.3 11 36.7 3 10.0 5 16.7 1 3.3

DQB9 10 33.3 16 53.3 11 36.7 8 26.7 6 20.0 8 26.7

DQB10 7 23.3 15 50.0 10 33.3 6 20.0 5 16.7 7 23.3

DQB11 7 23.3 16 53.3 9 30.0 8 26.7 5 16.7 7 23.3

Total Frequency 98 18.1 151 27.8 119 21.9 64 11.8 57 10.5 54 9.9

Appendices 283

K6. Perceived Clinical Benefits and Decision Making

Perceived Clinical Benefits & Decision -making Strongly Strongly agree Agree Neutral Disagree disagree freq. % freq. % freq. % freq. % freq. % Mean SD PCB1 21 72.4 6 20.7 2 6.9 0 0.0 0 0.0 1.345 0.614 PCB2 11 37.9 14 48.3 3 10.3 0 0.0 1 3.4 1.828 0.889 PCB3 12 41.4 11 37.9 5 17.2 0 0.0 1 3.4 1.862 0.953 PCB4 12 41.4 13 44.8 4 13.8 0 0.0 0 0.0 1.724 0.702 PCB5 24 82.8 4 13.8 1 3.4 0 0.0 0 0.0 1.207 0.491 PCB6 13 46.4 14 50.0 1 3.6 0 0.0 0 0.0 1.571 0.573 PCB7 11 37.9 11 37.9 6 20.7 0 0.0 1 3.4 1.931 0.961 PCB8 15 51.7 10 34.5 4 13.8 0 0.0 0 0.0 1.621 0.728 PCB9 19 67.9 9 32.1 0 0.0 0 0.0 0 0.0 1.321 0.476 PCB10 18 62.1 9 31.0 2 6.9 0 0.0 0 0.0 1.448 0.632 PCB11 18 62.1 9 31.0 2 6.9 0 0.0 0 0.0 1.448 0.632 Total 1.573 0.695

K7. Frequency and percentage distribution of Perceived Usefulness of EHR data with Decision Making

Perceived Usefulness of EHR Data & Decision -making Strongly Strongly agree Agree Neutral Disagree disagree freq. % freq. % freq. % freq. % freq. % Mean SD PDU1 21 72.4 8 27.6 0 0.0 0 0.0 0 0.0 1.276 0.455 PDU2 16 55.2 12 41.4 1 3.4 0 0.0 0 0.0 1.483 0.574 PDU3 16 57.1 11 39.3 1 3.6 0 0.0 0 0.0 1.464 0.576 PDU4 17 58.6 12 41.4 0 0.0 0 0.0 0 0.0 1.414 0.501 PDU5 8 27.6 14 48.3 6 20.7 0 0.0 1 3.4 2.034 0.906 PDU6 15 51.7 11 37.9 3 10.3 0 0.0 0 0.0 1.586 0.682 PDU7 9 31.0 12 41.4 7 24.1 0 0.0 1 3.4 2.034 0.944 Total 1.613 0.663

284 Appendices

K8. Frequency and percentage distribution of availability of information and decision making

Nature of availability of information and Decision making Strongly Strongly agree Agree Neutral Disagree disagree freq. % freq. % freq. % freq. % freq. % Mean SD DA1 25 86.2 4 13.8 0 0.0 0 0.0 0 0.0 1.138 0.351 DA2 24 82.8 5 17.2 0 0.0 0 0.0 0 0.0 1.172 0.384 DA3 24 82.8 5 17.2 0 0.0 0 0.0 0 0.0 1.172 0.384 DA4 26 89.7 3 10.3 0 0.0 0 0.0 0 0.0 1.103 0.310 Total 1.147 0.357

Appendices 285

Appendix L. Cronbach’s alpha results EHR System Functionalities leading to clinical benefits

Reliability Statistics Cronbach's Alpha Based Cronbach's Alpha on Standardized Items N of Items .869 .863 11

Inter-Item Correlation Matrix SF1 SF2 SF3 SF4 SF5 SF6 SF7 SF8 SF9 SF10 SF11 SF1 1.000 SF2 .344 1.000 SF3 .077 .712 1.000 SF4 .295 .375 .469 1.000 SF5 .493 .633 .429 .487 1.000 SF6 .122 .189 .226 .330 .120 1.000 SF7 -.055 .610 .742 .215 .445 .170 1.000 SF8 .314 .543 .615 .504 .505 .456 .477 1.000 SF9 .308 .624 .306 .314 .441 -.018 .344 .338 1.000 SF10 .326 .628 .319 .341 .450 0.000 .375 .313 .986 1.000 SF11 .340 .240 .269 .538 .142 .133 .157 .386 .298 .318 1.000

286 Appendices

Item-Total Statistics

Scale Mean if Scale Variance if Corrected Item- Squared Multiple Cronbach's Alpha if Item Deleted Item Deleted Total Correlation Correlation Item Deleted SF1 20.9655 43.106 .386 .528 .869 SF2 19.9310 36.352 .792 .770 .842 SF3 19.7241 36.135 .649 .793 .851 SF4 20.3793 38.315 .580 .658 .856 SF5 19.7586 36.047 .631 .690 .853 SF6 21.0000 43.786 .239 .367 .874 SF7 20.2414 38.475 .565 .736 .858 SF8 19.8966 37.525 .670 .673 .850 SF9 20.2759 35.993 .638 .981 .852 SF10 20.3103 35.650 .654 .982 .851 SF11 20.6207 40.387 .408 .481 .868

Appendices 287

Useful EHR Data

Reliability Statistics

Cronbach's Alpha Based on Cronbach's Alpha Standardized Items N of Items .878 .924 14

Inter-Item Correlation Matrix SF12 SF13 SF14 SF15 SF16 SF17 SF18 SF19 SF20 SF21 SF22 SF23 SF24 SF25 SF12 1.000 SF13 .204 1.000 SF14 .670 .282 1.000 SF15 .415 .106 .384 1.000 SF16 .516 .355 .558 .424 1.000 SF17 .235 .032 .482 .798 .352 1.000 SF18 .415 -.124 .361 .597 .264 .552 1.000 SF19 .655 .094 .830 .645 .526 .708 .780 1.000 SF20 .655 .094 .830 .645 .526 .708 .780 1.000 1.000 SF21 .415 -.124 .361 .597 .264 .552 1.000 .780 .780 1.000 SF22 .774 .370 .744 .428 .622 .274 .478 .706 .706 .478 1.000 SF23 .258 .325 .601 .212 .342 .411 .176 .451 .451 .176 .459 1.000 SF24 .409 -.097 .544 .747 .321 .810 .875 .892 .892 .875 .397 .241 1.000 SF25 .317 .134 .105 .515 .038 .335 .441 .349 .349 .441 .287 .190 .446 1.000

288 Appendices

Item-Total Statistics Scale Mean if Scale Variance if Corrected Item- Squared Multiple Cronbach's Alpha if Item Deleted Item Deleted Total Correlation Correlation Item Deleted SF12 19.8889 39.026 .660 .864 SF13 18.7407 44.430 .267 .889 SF14 20.1852 41.464 .774 .859 SF15 20.1111 42.564 .641 .865 SF16 19.5556 39.026 .591 .870 SF17 20.2593 44.738 .604 .869 SF18 20.4074 46.328 .599 .873 SF19 20.3333 44.308 .873 .864 SF20 20.3333 44.308 .873 .864 SF21 20.4074 46.328 .599 .873 SF22 20.1481 38.900 .790 .855 SF23 19.5556 40.487 .485 .878 SF24 20.3704 45.627 .678 .870 SF25 19.9259 44.533 .373 .878

Appendices 289

EHR data & clinical benefits

Reliability Statistics Cronbach's Alpha Based on N of Cronbach's Alpha Standardized Items Items .921 .924 6

Inter-Item Correlation Matrix DQ12 DQ13 DQ14 DQ15 DQ16 DQ17 DQU1 1.000 DQU2 .503 1.000 DQU3 .618 .738 1.000 DQU4 .561 .746 .678 1.000 DQU5 .571 .765 .472 .794 1.000 DQU6 .628 .755 .866 .747 .602 1.000

Item-Total Statistics

Scale Mean if Scale Variance if Corrected Item- Squared Multiple Cronbach's Alpha if Item Deleted Item Deleted Total Correlation Correlation Item Deleted DQU1 9.4000 17.007 .648 .538 .923 DQU2 9.3000 13.459 .833 .791 .901 DQU3 9.5667 15.978 .781 .832 .908 DQU4 9.1333 14.189 .838 .761 .898 DQU5 9.2667 14.685 .758 .804 .910 DQU6 9.3333 14.989 .842 .809 .899

290 Appendices

EHR Data Availability

Reliability Statistics

Cronbach's Cronbach's Alpha Based on N of Alpha Standardized Items Items .957 .957 4 Inter-Item Correlation Matrix DA1 DA2 DA3 DA4 DA1 1.000 DA2 .876 1.000 DA3 .876 1.000 1.000 DA4 .849 .744 .744 1.000

Item-Total Statistics Scale Mean if Scale Variance Corrected Item- Squared Multiple Cronbach's Alpha if Item Deleted if Item Deleted Total Correlation Correlation Item Deleted DA1 3.4483 1.042 .918 .936 DA2 3.4138 .966 .939 .930 DA3 3.4138 .966 .939 .930 DA4 3.4828 1.187 .799 .971

Appendices 291

Perceived clinical benefits improve decision-making Reliability Statistics Cronbach's Cronbach's Alpha Based on N of Alpha Standardized Items Items .966 .968 9

Inter-Item Correlation Matrix PCB1 PCB2 PCB3 PCB4 PCB5 PCB6 PCB7 PCB8 PCB9 PCB1 1.000 PCB2 .661 1.000 PCB3 .738 .898 1.000 PCB4 .680 .960 .938 1.000 PCB5 .750 .654 .691 .665 1.000 PCB6 .657 .841 .860 .883 .493 1.000 PCB7 .756 .915 .941 .887 .634 .825 1.000 PCB8 .746 .861 .857 .893 .710 .830 .824 1.000 PCB9 .791 .659 .751 .685 .593 .660 .767 .774 1.000

Item-Total Statistics Scale Mean if Scale Variance Corrected Item- Squared Multiple Cronbach's Alpha if Item Deleted if Item Deleted Total Correlation Correlation Item Deleted PCB1 12.5714 20.550 .786 .801 .965 PCB2 12.1429 18.942 .919 .966 .959 PCB3 12.1071 18.025 .949 .964 .958 PCB4 12.1786 18.745 .940 .974 .958 PCB5 12.7143 22.138 .710 .761 .969 PCB6 12.2857 19.915 .861 .846 .962 PCB7 12.0357 17.962 .930 .958 .959 PCB8 12.2857 18.730 .911 .873 .959 PCB9 12.5357 20.999 .785 .755 .965

292 Appendices

Perceived usefulness of EHR data for decision-making

Reliability Statistics Cronbach's Cronbach's Alpha Based on N of Alpha Standardized Items Items .956 .963 7

Inter-Item Correlation Matrix PDU1 PDU2 PDU3 PDU4 PDU5 PDU6 PDU7 PDU1 1.000 PDU2 .667 1.000 PDU3 .692 .947 1.000 PDU4 .718 .929 .891 1.000 PDU5 .762 .703 .711 .705 1.000 PDU6 .692 .873 .911 .831 .736 1.000 PDU7 .822 .753 .749 .757 .935 .764 1.000

Item-Total Statistics

Scale Mean if Scale Variance if Corrected Item- Squared Multiple Cronbach's Alpha if Item Deleted Item Deleted Total Correlation Correlation Item Deleted PDU1 9.6786 11.485 .799 .717 .956 PDU2 9.5000 10.852 .892 .937 .948 PDU3 9.4643 10.406 .896 .929 .946 PDU4 9.5357 10.925 .881 .880 .949 PDU5 9.0000 9.778 .843 .877 .952 PDU6 9.3929 10.099 .880 .847 .947 PDU7 9.0000 9.333 .886 .912 .950

Appendices 293

294 Appendices

Appendix M. Data fields of data source provided

Data Field Name Data Type Description AGE1 VarChar Age group: 0-4, 5-9, 10-14, 15-19, 20-24, 25-29, 30-34, 35-39, 40-44, 45-49, 50-54, 55-59, 60-64, 65-69, 70-74, 75-79, 80-84, 85 & above. SEX Int Representation of patient gender. 1 = Male, 2 = Female NATIONALITY Text 2 alphabet code to represent country. MARITAL_STATUS Text Patient marital status: Divorced, Married, Separated, Single, Unknown, Widowed. RACE Text Patient racial representation: Caucasian, Chinese, Eurasian, Indian, Malay, Others, Sikh. RELIGION Text Patient religious background: Buddhism, Christianity, Free Thinker, Hinduism, Islam, Others, Sikh, Taoism. NON_RESID_STATUS Text Singapore residential status: Resident or Non- Resident. MOVEMENT_TYPE Text Type of patient discharge. TREATMENT_OU VarChar Hospital Ward No. DEPT_OU Text Outpatient clinic type ACCOM_CAT VarChar Class of hospital ward admitted. PATIENT_CLASS VarChar Minimum ward class of patient REF_TYPE Text Type of inpatient referral REF_HOSP_NAME Text Department or hospital referred from ADMIT_REASON Text Reason for admission TREATMENT_CAT VarChar Category of treatment provided HOSP_MAIN_DIAG_CODE VarChar ICD9 and ICD10 codes of admission diagnosis HOSP_MAIN_DIAG_DESC Text Description of ICD9 and ICD10 codes DRG_DISCH_CODE VarChar Diagnosis Related Group (DRG) Discharge Code. Used to group discharge diagnosis. Used predominantly for administrative billing purpose, Casemix. CASE_NO Int 10 digit integer. CASE_NO is not unique. Patient unique inpatient encounter is identified using the combination of CASE_NO and CASE_DIGIT. CASE_DIGIT VarChar A single alphabet used together with CASE_NO to represent a unique patient encounter. PATIENT_TYPE Text Type of patient admitted AGE Int Age of patient at the point of admission DATE_BIR Date Patient’s date of birth ADATE Date Admission Date DDATE Date Discharge Date ATIME Time Admission Time DTIME Time Discharge Time CLUSTER VarChar Refers to department or cluster of medical types LOS Int No of days stayed in ward; length of stay MM Int Month of admission; Month of case

Appendices 295

Appendix N. Structure of data set (presented in R) $ AGE1 : Factor w/ 18 levels "0-4","05-09",..: 17 8 13 14 15 13 17 13 14 4 ... $ SEX : Factor w/ 2 levels "1","2": 1 1 1 1 1 2 1 1 1 1 ... $ NATIONALITY : Factor w/ 63 levels "AE","AM","AS",..: 52 12 52 52 52 52 52 52 52 52 ... $ MARITAL_STATUS :Factor w/ 7 levels "","Divorced",..: 1 1 3 3 1 1 1 3 1 1 ... $ RACE : Factor w/ 7 levels "Caucasian","Chinese",..: 2 2 2 2 2 5 2 2 2 2 ... $ RELIGION : Factor w/ 10 levels "","Buddhism",..: 1 1 2 2 1 1 1 1 1 1 ... $ NON_RESID_STATUS : Factor w/ 2 levels "Non-Resident",..: 2 1 2 2 2 2 2 2 2 2 ... $ MOVEMENT_TYPE : Factor w/ 20 levels "13 Months Auto Case End (SOC)",..: 5 16 16 7 10 4 16 7 4 16 ... $ TREATMENT_OU : Factor w/ 60 levels "NW12","NW20",..: 56 56 30 3 37 12 56 25 2 38 ... $ DEPT_OU : Factor w/ 61 levels "Anaesthesia",..: 6 39 38 39 6 6 6 38 6 6 ... $ ACCOM_CAT : Factor w/ 11 levels "A1","A1+","A2",..: 9 6 6 6 6 9 6 6 9 6 ... $ PATIENT_CLASS : Factor w/ 14 levels "A","AP","ARF",..: 10 12 10 10 10 10 10 10 10 7 ... $ REF_TYPE : Factor w/ 12 levels "","21","25","Intra-Hospital (A&E)",..: 4 4 4 4 5 6 4 4 6 5 ... $ REF_HOSP_NAME : Factor w/ 25 levels "","Alexandra Hospital",..: 7 7 7 7 10 11 7 7 12 10 ... $ ADMIT_REASON : Factor w/ 11 levels "","13","4","51",..: 9 9 9 9 10 9 9 9 9 10 ... $ TREATMENT_CAT : Factor w/ 41 levels "","A","A1+","A2",..: 18 10 10 10 10 18 10 10 18 6 ... $ HOSP_MAIN_DIAG_CODE : Factor w/ 475 levels "3940","3941",..: 116 94 100 93 31 31 116 100 115 12 ... $ HOSP_MAIN_DIAG_DESC : chr "Abdominal Aneurysm, Ruptured" "Unspecified Intracranial Hemorrhage" "Cerebral Artery Occlusion, Unspecified" "Subdural Hemorrhage" ... $ DRG_DISCH_CODE : Factor w/ 444 levels "","10","128",..: 91 91 105 91 91 91 91 1 91 86 ... $ CASE_NO : int 1590616986 1590643502 1590662035 1590667251 1590670926 1590672283 1590687190 1590689621 1590691515 1590708448 ... $ CASE_DIGIT : chr "G" "H" "F" "H" ... $ PATIENT_TYPE : Factor w/ 6 levels "DS Turn IN","Elective inpat.",..: 3 3 3 3 2 3 3 3 3 2 ... $ AGE : int 82 35 60 69 73 61 80 61 65 19 ... $ DATE_BIR : chr "14/02/1927" "10/03/1974" "11/05/1949" "13/03/1940" ... $ ADATE : chr "06/10/2009" "20/10/2009" "03/11/2009" "05/11/2009" ... $ DDATE : chr "13/05/2010" "27/04/2010" "08/01/2010" "18/05/2010" ... $ ATIME : chr "21:15:45" "18:10:00" "14:39:24" "12:34:04" ... $ DTIME : chr "14:30:00" "3:43:23" "11:24:44" "14:16:27" ... $ CLUSTER : Factor w/ 13 levels "(01) NCIS - National Uni Cancer Institute Singapore",..: 3 5 4 5 3 3 3 4 3 3 ... $ LOS : int 219 189 66 194 204 57 47 70 89 7 ... $ MM : int 5 4 1 5 6 1 1 1 2 1 ...

296 Appendices

Appendix O. Summary of data sets (presented in R)

AGE1 SEX NON_RESID_STATUS 0-19 : 119 1 : 3102 Non-Resident : 204

20-44 : 561 2 : 2505 Resident : 5403 45-64 : 2685 65-84 : 2066 85 & : 1176

above

NATIONALITY MARITAL_STATUS PATIENT_TYPE SG : 5232 Divorced : 29 DS Turn IN : 27 MY : 165 Married : 3625 Elective inpat. : 440 ID : 58 Separated : 1 Emergency : 5057 IN : 54 Single : 482 Same Day Admission : 83 ZO : 15 Unknown : 1453 Cardiac Thoracic : 451

BD : 13 Widowed : 17 Cardiac : 320

(Other) : 70 Other : 952

RACE RELIGION MOVEMENT_TYPE Caucasian : 4 Unknown : 2252 Patient Discharged : 5213 Chinese : 2829 Islam : 1462 Follow-up at SOC : 160 Eurasian : 14 Buddhism : 1118 Discharge to Nursing : 128 Indian : 781 Hinduism : 378 Home Malay : 1427 Christianity : 180 Discharge to : 83 Others : 525 None : 130 Community Hospital Sikh : 27 (Other) : 87 Department : 7

Discharge to Hospices : 6

Other : 10

HIGH_LOS ICD_CCI READMISSION 0 : 5361 1 : 1858 0 : 4367

1 : 246 2 : 3749 1 : 575 2 : 291 3 : 210

4 : 164

Appendices 297

Appendix P. Deyo Quan list of 17 comorbidities for CCI values

Translation of Charlson comorbidity index components into ICD-10-CM codes Comorbidities ICD-10 Myocardial Infarction 121.x, 122.x, 125.2 Congestive heart failure 109.9, 111.0, 113.0, 113.2, 125.5, 142.0, 142.5- 142.9, 143.x, 150.x, P29.0 Peripheral vascular disease 170.x, 171.x, 173.1, 173.8, 173.9, 177.1, 179.0, 179.2, K55.1, K55.8, K55.9, Z95.8, Z95.9 Cerebrovascular disease G45.x, G46.x, H34.0, I60.x-I69.x Dementia F00.x-F03.x, F05.1, G30.x, G31.1 Chronic pulmonary disease 127.8, 127.9, J40.x-J47.x, J60.x-J67.x, J68.4, J70.1, J70.3 Rheumatic disease M05.x, M06.x, M31.5, M32.x-M34.x, M35.1, M35.3, M36.0 Peptic ulcer disease K25.x-K28.x Mild liver disease B18.x, K70.0-K70.3, K70.9, K71.3-K71.5, K71.7, K73.x, K74.x, K76.0, K76.2-K76.4, K76.8, K76.9, Z94.4 Diabetes without chronic complication E10.0, E10.1, E10.6, E10.8, E10.9, E11.0, E11.1, E11.6, E11.8, E11.9, E12.0, E12.1, E12.6, E12.8, E12.9, E13.0, E13.1, E13.6, E13.8, E13.9, E14.0, E14.1, E14.6, E14.8, E14.9 Diabetes with chronic complication E10.2-E10.5, El0.7, E11.2-Ell11.5, E11.7, E12.2-E12.5, E12.7, E13.2- E13.5, E13.7, E14.2-E14.5, E14.7 Hemiplegia or paraplegia G04.1, G11.4, G80.1, G80.2, G81.x, G82.x, G83.0-G83.4, G83.9 Renal disease 112.0, I113.1, N03.2-N03.7, N05.2- N05.7, N18.x, N19.x, N25.0, Z49.0- Z49.2, Z94.0, Z99.2 Any malignancy, including lymphoma and COO.x-C26.x, C30.x-C34.x, C37.x- C41.x, leukaemia, except malignant neoplasm of skin C43.x, C45.x-C58.x, C60.x- C76.x, C81.x- C85.x, C88.x, C90.x-C97.x Moderate of severe liver disease 185.0, I185.9, I186.4, I198.2, K70.4, K71.1, K72.1, K72.9, K76.5, K76.6, K76.7 Metastatic solid tumour C77.x-C80.x AIDS/HIV B20.x-B22.x, B24.x

298 Appendices

Appendix Q. Nielsen’s (1994) Heuristic Evaluation Checklist

Usability Inspection Methods Heuristics Description H1: Visibility of system The system should always keep users informed about what is going on, status through appropriate feedback within reasonable time H2: Match between The system should speak the user’s language, with words, phrases, and system and real world concepts familiar to the user, rather than system-oriented terms. Follow real-world conventions, making information appear in a natural and logical order H3: User control and Users often choose system functions by mistake and will need a clearly freedom marked “emergency exit” to leave the unwanted state without having to go through an extended dialogue. Support undo and redo. H4: Consistency and Users should not have to wonder whether different words, situations, or standards actions mean the same thing. Follow platform conventions. H5: Error prevention Even better than good error messages is a careful design which prevents a problem from occurring in the first place. H6: Recognition rather Make objects, actions, and options visible. The user should not have to than recall. remember information from one part of the dialogue to another. Instructions for use of the systems should be visible of easily retrievable whenever appropriate. H7: Flexibility and Accelerators – unseen by the novice users – may often speed up the efficiency of use interaction for the expert user to such an extent that the system can cater to both inexperienced and experienced users. Allow users to tailor frequent actions. H8: Aesthetic and Dialogues should not contain information which is irrelevant or rarely minimalist design needed. Every extra unit of information in a dialogue competes with the relevant units of information and diminishes their relative visibility. H9: Help users recognise, Error messages should be expressed in plain language (no codes), diagnose, and recover precisely indicate the problem, and constructively suggest a solution. from errors H10: Help and Even though it is better if the system can be used without documentation, documentation it may be necessary to provide help and documentation. Any such information should be easy to search, focused on the user’s task, list concrete steps to be carried out, and not be too large.

Appendices 299

Appendix R. Prototype design checklist based on Nielsen’s list of heuristics

Heuristics Remarks Recommendation H1: Visibility Feedback and button information are Confirmation is needed to proceed of system status displayed. Certain actions can proceed within a button action. without confirmation. Current status to be clearly indicated Current status is not clearly indicated. with breadcrumbs. H2: Match The layout is similar to current between system CPSS2 and Allscripts SCM systems and real world used by healthcare professionals in the hospital. All buttons, tables, fields and labels are clearly indicated to reduce ambiguity. H3: User User can cancel any action with the Confirmation button to prompt users to control and cancel button. confirm every action. freedom User should be able to cancel any action with the cancel button. H4: Similar icons, naming conventions are Consistency used as per current information and standards systems used by users to maintain consistency and accordance to standards. H5: Error The need to confirm action prevent Implement confirmation action to prevention user from undesirable occurrence prevent problems from occurring. from happening. H6: Patient information and case numbers On hover, button reveals the summary Recognition are clearly displayed at the top of the of the action to remind users. rather than page to remind users which patient recall. has been selected. Buttons are appropriately named to indicate the action that can be performed. However, more information should be provided so that user knows what the buttons are for. H7: Flexibility No interaction available for expert Implement shortcut buttons to main and efficiency users. listing page is available for users to of use use.

H8: Aesthetic Icons and buttons used are labelled Provide more information regarding and minimalist with unambiguous text to reduce the actions of icons and buttons as text design error. labelling may not explain much. Correct colour combinations are used to reflect and grab user’s attention like bright red to indicate important statistics to reflect on. H9: Help users Error messages are clearly displayed recognise, on the screen. diagnose, and recover from errors H10: Help and Help icon for novice users on the full documentation explanation of what the icons and buttons represent.

300 Appendices

Appendix S. Usability Design Gap Solutions

Identifying Usability Design Gaps Heuristics Design changes H1: Confirmation is needed to proceed within a button action. H3: Confirmation button to prompt users to confirm every action. H5: Implement confirmation action to prevent problems from occurring.

H1: Current status to be clearly indicated with breadcrumbs. H7: Implement shortcut buttons to main listing page is available for users to use (breadcrumbs).

H3: User should be able to (Confirmation action allowing user to cancel actions) cancel any action with the cancel button.

(Cancel buttons when viewing patient information)

H6: On hover, button reveals the summary of the action to remind users its functionality. H8: Provide more information regarding the actions of icons and buttons as text labelling may not explain much.

Appendices 301

H10: Help icon for novice users on the full explanation of what the icons and buttons represent.

302 Appendices

Appendix T. Interview Transcript CIO

Appendices 303

304 Appendices

Appendices 305

Appendix U. Interview Transcript Principal Systems Specialist

306 Appendices

Appendices 307

Appendix V. Focus Group Transcript on the Evaluation of PBE Approach

308 Appendices

Appendices 309

310 Appendices

Appendices 311

312 Appendices

Appendices 313

314 Appendices

Appendices 315

316 Appendices

Appendices 317

318 Appendices

Appendices 319

320 Appendices

Appendices 321

322 Appendices

Appendices 323

Appendix W. PBE Evaluation Participant Sign Off

324 Appendices

Appendices 325

Appendix X. List of Healthcare Institutions under the new cluster

Central: Eastern: Western: NHG and AHS SingHealth and EHA NUHS and JHS

Acute Hospitals • Khoo Teck Phuat Hospital • Changi General Hospital • National University Hospital • Tan Tock Seng Hospital • Singapore General Hospital • Ng Teng Fong General Hospital • Woodlands General Hospital • Sengkang General Hospital (name tbc)

Community • Yishun Community Hospital • Bright Vision Community Hospital • Jurong Community Hospital Hospitals • Woodlands Community Hospital • Outram Community Hospital (name tbc) • Sengkang Community Hospital

Primary Care • Ang Mo Kio Polyclinic • Bedok Polyclinic • Bukit Batok Polyclinic • Geylang Polyclinic • Bukit Merah Polyclinic • Choa Chu Kang Polyclinic • Hougang Polyclinic • Marine Parade Polyclinic • Clementi Polyclinic • Toa Payoh Polyclinic • Outram Polyclinic • Jurong Polyclinic • Woodlands Polyclinic • Pasir Ris Polyclinic • Queenstown Polyclinic • Yishun Polyclinic • Sengkang Polyclinic • Bukit Panjang Polyclinic • Sembawang Primary Care Centre • Tampines Polyclinic • Pioneer Polyclinic • Eunos Polyclinic • Punggol Polyclinic

Appendices 327