Promise and peril: How artificial intelligence is transforming health care COVERAGE SUPPORTED BY Promise and peril: How artificial intelligence is transforming health care AI has enormous potential to improve the quality of health care, enable early diagnosis of diseases, and reduce costs. But if implemented incautiously, AI can exacerbate health disparities, endanger patient privacy, and perpetuate bias. STAT, with support from the Commonwealth Fund, explored these possibilities and pitfalls during the past year and a half, illuminating best practices while identifying concerns and regulatory gaps. This report includes many of the articles we published and summarizes our findings, as well as recommendations we heard from caregivers, health care executives, academic experts, patient advocates, and others. It was written by STAT’s Erin Brodwin, a California-based health tech correspondent, and Casey Ross, national technology correspondent. PROMISE AND PERIL: HOW AI IS TRANSFORMING HEALTH CARE Preface The Commonwealth Fund is a national philanthropy committed to improving our health care system. When STAT approached us with the idea of a series on artificial intelligence (AI) in health care, we saw an important opportunity to shed light on a trend of huge significance, despite the fact that we don’t have a formal program on health information technology (HIT) or the related topic of AI. It didn’t hurt, of course, that as a former national coordinator for health information technology under the Obama administration, I had personally continued to follow develop- ments in HIT with more than passing interest. Information is the lifeblood of medicine and thus effective information management is fundamental to everything our health system does. Data drives the diagnosis and treatment of illness but also the allocation of health resources, payment for clinical services, and, indeed, effective responses to public health emergencies, like pandemics. AI offers opportunities to make better sense of the massive streams of data flooding doctors’ offices, nursing stations in hospitals, community clinics, insurance companies and every other nook and cranny of the vast $3 trillion enterprise that we call our health care system. Making sense of data, in turn, could mean better quality of care and better health at lower cost. Nevertheless, our health care system has a fascination with new technologies that leads, not infrequently, to an overestimation of their value and an underestimation of their weaknesses and dangers. AI’s promise is great, but we know with a certainty born of hard experience that at some point it will disappoint us in ways that might have been predicted and prevented if only we had examined it more critically at the outset. PROMISE AND PERIL: HOW AI IS TRANSFORMING HEALTH CARE PREFACE With this in mind, we are especially appreciative of STAT’s reporting on the equity implications of AI. One story found that analytics software, although not intentionally biased, often reinforces deeply rooted inequities in the American health care system. Another found that AI tools could worsen disparities in breast cancer, a disease which is 46% more likely to be fatal for Black women. Further, when a machine learning model is developed using data skewed by sex, the algorithm furthers biases instead of reducing them. We were also pleased to see that STAT’s coverage reached a broad audience, attracting over 900,000 unique visitors and over a million page views. A knowledgeable and informed public is critical to accomplishing the Common- wealth Fund’s mission of making high-quality, affordable health care available to all Americans. Partnerships with strong media outlets like STAT to explore emerging issues like artificial intelligence are one powerful way to move our mission forward. — David Blumenthal, M.D., President, The Commonwealth Fund PROMISE AND PERIL: HOW AI IS TRANSFORMING HEALTH CARE INTRODUCTIONPREFACE Table of contents 01 INTRODUCTION 70 CHAPTER 3: THE EMERGING SCIENCE OF IMPLEMENTING AI 04 WHAT IS AI? 73 ‘It’s really on them to learn’: How the 08 CHAPTER 1: THE RISE OF AI IN rapid rollout of AI tools has fueled HEALTH CARE frustration among clinicians 12 At Mayo Clinic, AI engineers face an 79 With ‘nutrition labels’ and an anthro- ‘acid test’: Will their algorithms help pologist’s eye, Duke pioneers a new real patients? approach to AI in medicine 21 An experiment in end-of-life care: Tapping AI’s cold calculus to nudge 86 CHAPTER 4: BIAS AND HEALTH the most human of conversations INEQUITY 32 Hospitals deploy AI to send patients 89 From a small town in North Carolina to home earlier — and keep them there big-city hospitals, how software infuses racism nto U.S. health care 37 IVF can be a painstaking process. Could AI make it more precise? 103 Could AI tools for breast cancer worsen disparities? Patchy public data in FDA filings fuel concern 42 Can AI improve cancer diagnosis in real patients? The U.S. military has enlisted Google to find out 111 Medical AI systems are disproportion- ately built with data from just three states, new research finds 46 CHAPTER 2: AN UNEXPECTED TURN — AI TAKES ON COVID-19 114 AI systems are worse at diagnosing disease when training data is skewed by sex 49 Health systems are using AI to predict severe Covid-19 cases.But limited data could produce unreliable results 119 Could training AI to learn from patients give doctors a better understanding of the causes of pain? 55 Debate flares over using AI to detect Covid-19 in lung scans 62 STAT’s guide to how hospitals are using AI to fight Covid-19 PROMISE AND PERIL: HOW AI IS TRANSFORMING HEALTH CARE TABLE OF CONTENTS Table of contents 126 CHAPTER 5: COMMODITIZATION 196 CHAPTER 7: RECOMMENDATIONS OF PATIENT DATA AND PRIVACY CONCERNS 201 4 steps for AI developers to build trust in their clinical tools 129 Hospitals turn to big tech companies to store and analyze their data — 204 Ten steps to ethics-based governance leaving patients in the dark on privacy of AI in health care protections 210 Health care needs AI. It also needs the 140 At Mayo Clinic, sharing patient data human touch with companies fuels AI innovation — and concerns about consent 215 Machine learning for clinical decision- making: pay attention to what you 149 Google research suggests efforts to don’t see anonymize patient data aren’t foolproof 153 Backed by big hospitals, a former 220 OTHER RESOURCES Microsoft executive wades into the messy business of selling patient data 158 CHAPTER 6: THE ROLE OF REGULATORS: HOW TO ENSURE SAFETY AND TRANSPARENCY 162 An invisible hand: Patients aren’t being told about the AI systems advising their care 174 As the FDA clears a flood of AI tools, missing data raise troubling questions on safety and fairness 185 Can you sue an algorithm for mal- practice? It depends 190 How should the FDA regulate adaptive AI, software that designs itself? Cover and illustrations by Mike Reddy PROMISE AND PERIL: HOW AI IS TRANSFORMING HEALTH CARE TABLE OF CONTENTS Introduction I watched with a feeling somewhere between amusement and awe as Sam Asirvatham, an electrophysiologist at the Mayo Clinic, toggled rapidly through electrocardiogram tracings projected on a wall-sized screen in a conference room packed with his colleagues. He was straining to find even a hint of the heart wave abnormality an artificial intelligence algorithm could see in a patient’s EKGs but he could not. After it was fed tracings going back 30 years, the AI had concluded — correctly — that the patient’s risk of atrial fibrillation had dramatically increased at a certain point. Sure enough, seven years later, the individual had suffered a stroke. To the cardiologists gathered in the room, the implications hit like a thunderclap: With advanced warning from the AI, they could intervene far earlier in the course of disease, prescribing changes in diet, in exercise, in lifestyle. Perhaps, the suggestion was made, the patient could have been put on blood thinners to head off the stroke. This is the dream, as yet unrealized, now driving investment in AI all over the world. Physicians and entrepreneurs have seen evidence of an untapped dimension of medical knowledge, and they are racing to discover it and show that AI’s insights can be used to handcuff cancer before it invades cells, or spot a sputtering heart before it’s too late. During the past year, my colleagues and I have followed their work on this frontier. We wanted to understand how this new generation of AI works, to explain its true potential, and to expose the difficulties and dangers of deploying it in medicine, a field that so intrinsically involves humans caring for humans. PROMISE AND PERIL: HOW AI IS TRANSFORMING HEALTH CARE INTRODUCTION | 01 Hospitals and huge technology companies are pressing forward, but there are still so many obstacles and unanswered questions. In most cases, we simply don’t know how much we can trust AI systems used in health care. Developers of these tools promise they will detect illnesses that doctors miss, flag medical crises before they happen, and save patients’ lives, clinicians’ time, and hospitals’ money. But proof that these outcomes can be achieved is so far as scarce as machine learning models are opaque. Clinicians don’t know how the Mayo AI, for example, reached its conclusion about the patient’s risk of atrial fibrillation. They don’t know whether earlier intervention would have caused more harm — blood thinners can cause serious side effects and impact patients’ quality of life — than good. And they don’t know if the AI, when applied to diverse groups of patients in different communities and clinical settings, would be just as accurate as it was for this one patient in this one instance.