Indexed in MEDLINE, PubMed, and PubMed Central National Library of Medicine

Spring 2021 Volume 25 No. 2

A peer-reviewed journal of medical science, social science in medicine, and medical humanities

ORIGINAL RESEARCH ARTICLES 1 Trends in Infl uenza Vaccine Uptake and Severe Infl uenza-Related Outcomes at Kaiser Permanente Southern California, 2007-2017 9 Ten-year Thyroid Cancer Incidence in an Integrated Healthcare Delivery System 15 Using Real-world Data for Decision Support: Recommendations from a Primary Care Provider Survey 21 Surgeon-Therapist Communication Must Be Improved in Rotator Cuff Repair Rehabilitation: An Electronic Survey of Physical Therapists on Postoperative Rehabilitation Protocols and Communication with Treating Surgeons 34 “Headache” Online Information: An Evaluation of Readability, Quality, Credibility, and Content 50 Insurance Barriers, Gendering, and Access: Interviews with Central North Carolinian Women About Their Health Care Experiences 90 Changes in Emergency Department Patient Volume and Acuity Associated with Early Stages of the COVID-19 Pandemic in a Unique Environment 112 Adverse Events of Sodium–Glucose Cotransporter-2 Inhibitors in Chronic Kidney Disease: A Retrospective Chart Review REVIEW ARTICLE

144 What Is Genitourinary Syndrome of Menopause and Why Should We Care? CASE REPORTS

154 Utilization of Nivolumab in Adenoid Cystic Carcinoma After Progression on Platinum-Based Chemotherapy 159 Deadly Neuroinvasive Mosquito-Borne Virus: A Case of Eastern Equine Encephalitis 167 Rapid Response of a BRCA2/TP53/PTEN-Deleted Metastatic Uterine Leiomyosarcoma to Olaparib: A Case Report COMMENTARIES

175 Unconscious Bias Is a Human Condition 179 The Ally Book Club: A Tool for Challenging Racism

Find us online at www.thepermanentejournal.org Spring 2021/Volume 25 No. 2 TABLE OF CONTENTS

– Sponsored by the 8 Permanente Medical Groups ORIGINAL RESEARCH ARTICLES 21 Surgeon Therapist Communication Must Be Improved in Rotator Cuff Repair fl Mission: The Permanente Journal advances 1 Trends in In uenza Vaccine Uptake and Severe Rehabilitation: An Electronic Survey of Influenza-Related Outcomes at Kaiser knowledge in scientifc research, clinical medicine, Physical Therapists on Postoperative Permanente Southern California, 2007-2017. and innovative health care delivery. Rehabilitation Protocols and Communication In-Lu Amy Liu, MS; Hilary C Tanenbaum, PhD, with Treating Surgeons. Mark Schultzel, MD; MS, MPH; Lei Qian, PhD; Lina S Sy, MPH; Wansu Karl B Scheidt, MD; Brian McNeill, DPT; Circulation: 2 million page views of TPJ articles Chen, PhD, MS; Steven J Jacobsen, MD, PhD Christopher M Klein, MS; Colin Blout, BS; John M in PubMed from a broad international readership Major efforts to increase influenza vaccine uptake Itamura, MD among Kaiser Permanente Southern California There is no consensus on post-operative rotator (KPSC) members have been undertaken in recent cuff repair protocols in orthopedic or physical years. However, whether these improvements therapy literature. Despite surgical management, fl ················ translate to a decline in severe in uenza-related the frequency of rotator cuff re-tears continues to outcomes has not been examined. We aimed to be high. Most rehabilitation programs follow fl understand the impact of the in uenza vaccination protocols developed by surgeons and physical fl program at KPSC by examining in uenza vaccine therapists. Tissue quality, size of tear, and repair fl uptake and three severe in uenza-related type are usually documented in the operative outcomes. report, which are rarely conveyed to the therapist. 9 Ten-year Thyroid Cancer Incidence in an This study highlights the lack of communication Integrated Healthcare Delivery System. between the physician and the therapist. Improved Stephanie J. Kim, MD, MPH; Megan L. Durr, MD; communication regarding the findings at surgery, Jeanne A Darbinian, MPH; Lori C. Sakoda, PhD; open lines of communication, and making Charles J. Meltzer, MD; Hasmik Arzumanyan, MD; alterations to the protocol may improve patient Kevin H. Wang, MD; Jonathan K. Lin, MD; Deepak outcomes. Gurushanthaiah, MD; Joan C. Lo, MD 27 Trends in Endoscopic Sinonasal Debridement The incidence of papillary thyroid cancer (PTC) in the Medicare Population. Mulin Xiong, BA; has increased in recent decades, but data from Rijul S Kshirsagar, MD; Jonathan Liang, MD, community-based settings are limited. This study FACS, FARS characterizes PTC trends in a large integrated Nasal/sinus endoscopy with biopsy/polypectomy/ healthcare system over ten years. Increasing PTC debridement or Current Procedure Terminology incidence over ten years was most evident for (CPT) code 31237 is one of the top 10 most photograph ≤ tumors 4 cm and Stage I disease. While these frequent and highest billed otolaryngology fi The Permanente Federation / Jan Sonnemair ndings may be attributable to greater PTC procedures among Medicare patients. We This photo captures the challenge of caring for patients during the detection, the increase across a range of tumor analyzed temporal and geographic trends in COVID-19 pandemic in the spring of 2020. sizes suggests that PTC burden might also have endoscopic debridement and correlated these with increased. sinus surgery and balloon sinuplasty trends. 15 Using Real-world Data for Decision Support: Otolaryngologists continue to perform increasing Recommendations from a Primary Care numbers of endoscopic debridement and receive Provider Survey. Patricia A Arean,´ PhD; Emily C increasing payments. There is some geographic Friedman, MID, CPE; Abhishek Pratap, PhD; variation in these trends. Amongst individual EDITORIAL & PUBLISHING OFFICE Ryan Allred, BA; Jaden Duffy, BA; Sara Gille, providers, there was a positive correlation The Permanente Journal MPH; Shelley Reetz, BS; Erin Keast, MPH; between the number of endoscopic debridement c/o Laura Fegraus Gregory Clarke, PhD procedures and both the number of balloon 1 Kaiser Plaza, 27th Floor sinuplasty and sinus surgery procedures. Oakland, CA 94612 The use of data from wearable sensors, Email: [email protected] smartphones, and apps holds promise as a clinical 34 “Headache” Online Information: An Evaluation INSTRUCTIONS FOR SUBMISSION decision-making tool in health and mental health in of Readability, Quality, Credibility, and Instructions for Authors are available along with a link to our manuscript primary care medicine. The aim of this study is to Content. Abdulrahman Bukhari, MBBS; Saad submission center at www.thepermanentejournal.org/authors.html determine provider perspectives about the utility of Alshihri, MBBS; Mohammed Abualenain, MBBS; PERMISSIONS AND REPRINTS this data for building digitally-based decision- Jordan Barton, BA; Genevieve Kupsky, BS; Jesse Reprint Permission Form available at: fi M Pines, MD; Ali Pourmand, MD, MPH, RDMS www.thepermanentejournal.org/about-us/5818-reprint-permissions.html making tools. Primary care providers nd value in collecting real-world patient data to assist in The Editorial Staff have disclosed that they have no personal, professional, Headache is experienced by more than half of the clinical decision-making provided such information world population each year. In this study, we or fnancial involvement in any of the manuscripts they might judge. Should a fl confict arise in the future, the Editorial Staff have agreed to recuse themselves does not interfere with provider work ow or evaluate the content, quality, and health literacy regarding any specifc manuscripts. The Editorial Staff also will not use the impose undue burden on patients. Additionally, required to understand online information for information attained through working with manuscripts for private gain. digital tools will need to demonstrate high patients with headaches. We found variable accuracy, be able to integrate into current clinical content and quality in online headache websites fl work ows, and maintain privacy and security of for patients. Many of these websites failed to The Permanente Journal (ISSN 1552-5775) is a quarterly publication patients’ data. of articles from the online journal of record, which is available at: disclose information about authorship, conflicts of www.thepermanentejournal.org. interest, and details on the prognosis or prevention Copyright © 2021 The Permanente Journal of headaches. Readability, credibility, completeness, and quality of information were lacking in the most websites.

Contents continued on next page

The Permanente Journal·Spring 2021 39 The Utility of Brain Magnetic Resonance Imaging/ 64 Better Together: Long-term Behaviors and 90 Changes in Emergency Department Patient Angiography and Neck Magnetic Resonance Perspectives after a Practitioner–Family Writing Volume and Acuity Associated with Early Stages Angiography in Patients with Suspected Acute Intervention in Clinical Practice. Madelaine of the COVID-19 Pandemic in a Unique Stroke. Mark Harris; Alyssa Finger; Emily Nishimura; Schaufel, MS, RD; Douglas Moss, BS; Ramona Environment. Brent Lorenzen, MD; Adam Schwartz, Blake Watabe; Hyo-Chun Yoon, MD, PhD Donovan, MS, RD, CCRC; Yi Li, MS; David G Thoele, MD, MS MD In our health maintenance organization, we have seen Hospitals and emergency departments faced a trend among our referring physicians to order a Excessive stress has been shown to be detrimental to profound uncertainty during the COVID-19 pandemic. simultaneous brain MRI, head MRA, and neck MRA in human health in many ways. Therefore, interventions Early concerns regarding demand far exceeding the evaluation of patients for acute stroke. However, that either reduce stress or improve coping are of capacity were balanced by anecdotal reports of there is little data to demonstrate any improvement in considerable interest. One method used to help cope decreased patient visits, including those for specific patient care resulting from ordering this triple study. with stress is expressive writing (EW), defined as high acuity conditions. This study seeks to identify The objective of this study was to analyze the utility of therapeutic writing that involves disclosure of personal changes in Emergency Department (ED) volume and this triple study for patients who experience stroke-like information, thoughts, or feelings. 2 EW has been acuity, within a specific managed care environment, symptoms. associated with reduced stress, improved health, associated with the onset of the pandemic increased disease-related quality of life scores, and 44 The ATTAIN Solution Tested: Initial Pilot Results 94 Colorectal Cancer and Return to Work: A Pilot reductions in physical symptoms in a variety of patient of an Automated, Web-based Screening Tool for Study of Recruiting Cancer Survivors and Their populations. 3-16 EW interventions also have the Unhealthy Drinking Behaviors. Jiseung Yoon, MD; Employers. Inga Gruß, PhD; Cathy J Bradley, PhD, potential to be low-cost, low-risk, and generally well- Emma Fredua, MPH, CHES; Shahriar B Davari, MD; MPA; Matthew P Banegas, PhD, MPH accepted by patients. Mohamed H Ismail, MD, DrPH In this study we assessed the feasibility of recruiting 71 Multistage Adolescent Depression Screening: A AuTomaTed Alcohol misuse Interventions (ATTAIN) is colorectal cancer survivors and their employers to Comparison of 11-Year-Olds to 12-Year-Olds. Alan an automated, computer-based tool that screens participate in research on return-to-work (RTW) after B Cortez, MD; Julia Wilkins, BS; Eric Handler, MD; people for unhealthy alcohol use and offers web- cancer diagnosis. Marc A Lerner, MD; Raoul Burchette, MS; Lawrence S based counseling with minimal involvement of Wissow, MD 100 Time-varying Reproduction Numbers of COVID-19 healthcare personnel. ATTAIN was well accepted by in Georgia, USA, March 2, 2020 to November 20, eligible adults, appeared non-inferior to office-based Adolescent depression screening is recommended 2020. Kamalich Muniz-Rodriguez, DrPH; Gerardo screening, and added several potential advantages in starting at age 12-years, but younger children Chowell, PhD; Jessica S Schwind, PhD; Randall terms of screening for alcohol use disorders and experience depression as well. Our objective was to Ford, DDS; Sylvia K Ofori, MPH; Chigozie A Ogwara, readiness to change. determine if screening for depression at age 11-years BS; Margaret R Davies, BS; Terrence Jacobs, BS; yields similar results to screening at age 12-years. 50 Insurance Barriers, Gendering, and Access: Chi-Hin Cheung, MS; Logan T Cowan, PhD; Andrew Multi-stage depression screening in 11-year-olds can Interviews with Central North Carolinian Women R Hansen, DrPH; Isaac Chun-Hai Fung, PhD be successfully applied in clinical practice with most About Their Health Care Experiences. Lena JP cases identifying youth without a prior mental health In 2020 SARS-CoV-2 impacted Georgia, USA. Cardoso, MPP; Anna Gassman-Pines, PhD; Nathan diagnosis. Georgia announced state-wide shelter-in-place on A Boucher, DrPH, PA, MS, MPA, CPHQ April 2 and partially lifted restrictions on April 27. We 79 Validation Study of Kids Hearing Game: A Self- Women face unique logistical and financial barriers to estimated the time-varying reproduction numbers, Rt, Administered Pediatric Audiology Application. healthcare access. They also have higher healthcare of COVID-19 in Georgia, Metro Atlanta, and Brian Kung, MD; Larisa Kunda, MD; Sarah Groff, expenditures and higher rates of morbidity. Women’s Dougherty County and environs, from March 2 AuD; Erica Miele, AuD; Marion Loyd, AuD; Diane M. experiences while utilizing healthcare is historically through November 20, 2020. Carpenter, MPH less well researched and warrants exploration. Future 112 Adverse Events of Sodium–Glucose clinical and research efforts should include 1) Conduct a comparison study between conventional Cotransporter-2 Inhibitors in Chronic Kidney increasing awareness of and facilitating access to audiometry and a tablet-based hearing screening Disease: A Retrospective Chart Review. Hanul affordable post-partum care; 2) easing burdens application, Kids Hearing Game (KHG). If KHG Choi, PharmD; Leigh-Anh Nguyen, PharmD; Jenny around scheduling appointments and improved care measures hearing at levels comparable with Wan, PharmD, BCPS; Hooman Milani, PharmD, coordination; and 3) more research exploring conventional audiometry, it could be used to screen MBA; Kristine McGill, PharmD, BCPS; Jong Park, MD women’s experiences during in-person healthcare hearing in children. KHG is comparable to encounters. Concerns and barriers that women conventional audiometry and may be used as a The renal benefits of SGLT2 inhibitors are now well- described may be due to systems-level requirements screening tool for children. established, and these agents are recommended by and constraints. the ADA and KDIGO guidelines for patients with type 86 Effects of Implementing a Higher Threshold for 2 diabetes and CKD. However, the safety profile of 57 Does a Recumbent Lateral Stability Trainer Recommending Thyroid Biopsies on Malignancy SGLT2 inhibitors in CKD is not as clear. This study Improve Balance Scores Among Older Adults Rates. Kori Higashiya, BS; Liam Delgesso, BS; describes the adverse event rates of SGLT2 Within 4 Weeks? Andrew Shim, EdD; Samantha Hyo-Chun Yoon, MD, PhD inhibitors, primarily empagliflozin, in KP SCAL Prichard, MHS; David Newman, PhD; Carly Lara, We implemented a new thyroid nodule classification members with diabetic kidney disease. Mike Waller, PhD; Maureen Hoppe, EdD, OTR/L system where biopsy was recommended for thyroid 118 Visit Content Analysis: Doctor-Patient Past literature has shown balance and strength are lesions ≥1 cm with at least two or more suspicious Communication in Patients with Type 2 Diabetes. important in preventing falls, but few studies have features. In a community setting performing less than Dana A Abdelgadir, BA; Laurie M Rodriguez, BS; focused on developing strength and power in a lateral 200 biopsies per year, the utilization of more stringent Maruta A Blatchins, BS; Pranita Mishra, MPP; Anjali plane. A recumbent lateral stability device can requirements for thyroid biopsy are necessary to Gopalan, MD, MS; Richard W Grant, MD, MPH improve balance scores among older adults within 4 achieve malignancy rates comparable to the weeks of training. published literature. The primary care visit is an important opportunity to discuss and modify diabetes management. Our results highlight two potential strategies (preparing patients for their visits through identifying priorities and learning how to ask more questions during visits) for improving diabetes primary care.

Contents continued on next page

The Permanente Journal·Spring 2021 124 Regionalization of Acute Myeloid Leukemia 154 Utilization of Nivolumab in Adenoid Cystic COMMENTARIES Treatment in a Community-Based Population: Carcinoma After Progression on Platinum-Based Implementation and Early Results. Lisa Y Law, MD; Chemotherapy. Diana V Maslov, MD, MS; Katharine 175 Unconscious Bias Is a Human Condition. Mihal Stephen P Uong, MS; Hyma T Vempaty, MD; Vu H Thomas, MD, MS; Marc Matrana, MD, MS, FACP Emberton, MD, MPH, MS Nguyen, MD; David Baer, MD; Vincent X Liu, MD; Lisa Immunotherapy has clinical activity for those with The unconscious bias that plagues our police J Herrinton, PhD metastatic head and neck cancers (HNC) who departments is the same unconscious bias that Regionalization of care for acute myeloid leukemia progress on proton therapy6. This case reviews the plagues business in the form of employee oppression (AML) has not been described for community-based use of immunotherapy in a patient with ACC. and burnout, that plagues academia in the form of fi settings. In 2015, we shifted AML induction from 21 Immunotherapy has been utilized for HNC patients social promotion and xed mindsets, that plagues law local centers to three regional centers. that have progressed. No study has evaluated the in the form of poetic injustice and that plagues politics efficacy of immunotherapy in ACC. Our patient in the form of disenfranchisement and voter 132 Unhealthy Drinking Behavior and the ATTAIN survived three months following administration of suppression. Each industry has also attempted to Solution: Web-based Automated Alcohol Misuse Nivolumab. Nivolumab and other PD-L1 inhibitors correct the negative effects of unconscious bias with Interventions. Jennifer Chevinsky, MD, MPH; Emma may be considered in patients with ACC. leadership development in business, Montessori Fredua, MPH, CHES; Ebonie M Vazquez, MD; method and active learning in academia, restorative Mohamed H Ismail, MD, DrPH 159 Deadly Neuroinvasive Mosquito-Borne Virus: A justice in law, and voter participation and town halls in Case of Eastern Equine Encephalitis. Natalie Millet, Context: Up to 30% of American adults may have politics. The reason that each of these efforts is still in DO; Saif Faiek, MD; Daniel Gurrieri, DO; Karanvir unhealthy drinking behavior, but only 17% get its infancy is that our understanding of the rules of Kals, DO; William Adams, DO; Edward Hamaty, DO; screened. There is promise in improving screening via unconscious bias and the behaviors that stem from Manish Trivedi, MD; David Zeidwerg, DO technology, but a lack of published evidence that unconscious bias have been incompletely supporting these efforts. Objective: To describe the In this report, we describe a 42 years old man who understood until now. Good people like yourself and development of ATTAIN, an automated, web-based worked primarily in wooded areas, presented to a like me have unconscious bias; having unconscious process to screen for and manage adults with hospital in South Jersey with an intractable headache bias is a human condition. But those of us who learn to unhealthy drinking behavior with minimal involvement and global facial paraesthesia. He reported multiple recognize and overcome our unconscious bias of healthcare personnel. tick bites in the weeks prior to his presentation. Based become more impactful and powerful stewards of on high clinical suspicion, cerebrospinal samples society. REVIEW ARTICLES were sent to the CDC, which confirmed the diagnosis 179 The Ally Book Club: A Tool for Challenging 138 The Role of Cannabidiol in Neurological of eastern equine encephalitis. Racism. Jeffrey B Ritterman, MD; Miranda Ritterman Disorders. Sirichai Chayasirisobhon, MD, FAAN 163 Jejunal Gastrointestinal Stromal Tumor as a Weintraub, PhD, MPH CBD is a non-psychotropic chemical and therefore Source of Small Bowel Bleeding: A Case Report. The Black Lives Matter movement has encouraged has become a compound of interest for clinical Jacob Burch, DO; Iftiker Ahmad, MD many of us to challenge our personal racism and to researchers to study its therapeutic potential. This We present the case of a 76-year-old male who uproot the racism within our institutions. This is a fi article is intended to review the ef cacy and safety of presented with 2 weeks of melena which began after national project, long overdue. While we have CBD in various neurological disorders in humans. starting dual antiplatelet therapy with aspirin and watched with interest the toppling of Confederate clopidogrel after undergoing coronary artery stenting. statues, we were surprised to learn that we, in 144 What Is Genitourinary Syndrome of Menopause medicine, have statues of our own that need removal. and Why Should We Care? Kelly Jo Peters, DO After EGD and colonoscopy failed to identify the culprit, the patient underwent VCE which identified a The marble statue of Dr. James Marion Sims that ’ Genitourinary syndrome of menopause (previously suspicious area concerning for intussusception. once graced New York City s Central Park no longer known as vulvovaginal atrophy or atrophic vaginitis) Computed tomography enterography (CTE) was then stands. For years, it proudly faced the New York involves symptoms of vaginal dryness, burning, performed and showed a short segment of bowel wall Academy of Medicine. Other statues of Dr. Sims can ’ itching as well as dyspareunia, dysuria, urinary thickening. The patient underwent laparoscopic small be found at South Carolina s Statehouse and on the urgency and recurrent urinary tract infections. This bowel resection and was found to have a GIST. Alabama State Capitol grounds. article is intended to address the signs, symptoms and – fi 167 Rapid Response of a BRCA2/TP53/PTEN-Deleted 181 Etiology of Belief behavior Systems and signi cant impact this condition can have for women Hierarchies. Mihal Emberton, MD, MPH, MS and help healthcare providers be more comfortable Metastatic Uterine Leiomyosarcoma to Olaparib: knowing how to ask about this, diagnosis it and review A Case Report. Minggui Pan, MD, PhD; Kristen In order to understand the well documented patterns various treatment options that are available about this Ganjoo, MD; Amer Karam, MD of mental health, human learning, human behavior, condition. Here we report a patient whose metastatic uLMS and the mechanics of hierarchies such as academic institutions, political systems, and business CASE REPORTS contains a BRCA2 deep deletion, as well as TP53 and PTEN deep deletion, responded rapidly to olaparib, a organizations, one must discover the rules and 150 Bringing Down the Door-to-needle Time: Patient poly (ADP-ribose) polymerase (PARP) inhibitor, after pathways that cause those patterns. My Belief- fi Thrombolysed in 6 Minutes—A Case Report. progressing on gemcitabine-docetaxel, doxorubicin, Behavior Systems archetype is the rst of its kind to Ankur Verma, MBBS, MEM; Sanjay Jaiswal, MBBS, and temozolomide regimens. This case report shall reconcile the theories and insights from social MEM be helpful to the treatment of other patients with sciences, political science, psychiatry, and evolutionary biology into a unifying paradigm which It has been clearly established that thrombolysis using metastatic uLMS that harbors a BRCA2 mutation or deletion. explains how socialization and human interactions recombinant tissue plasminogen activator (rTPA) is evolved into the patterns we recognize today. More strongly beneficial for acute stroke patients. The 170 A Case Report of Leptomeningeal importantly is that this new contribution to our sensitivity of brain tissue to ischemia causes this time Carcinomatosis Secondary to Recurrent Merkel understanding of human behavior within hierarchies dependence of the effectiveness of rTPA. Early Cell Carcinoma after Avelumab. Pedro Mendoza, provides the key insights to guide the restoration and recognition and activation of a stroke alert/code MD; Kathy Lin Chuang, MD repair of our dysfunctional hierarchies which, becomes imperative to effectively treat acute stroke Merkel cell carcinoma is a rare and aggressive unfortunately, all too often oppress, manipulate and patients and have positive outcomes. Here we exploit our humanity. describe the case and our rapid thrombolysis protocol neuroendocrine cancer with a high mortality rate of 33% - 46%. 1 Merkel cell is a type of epidermis cell which helped us achieve this door to needle time. A receptor responsible for contact sensitivity and is structured protocol is recommended to reduce the door to needle times for thrombolysis in acute known to have neuroendocrine properties. Treatment ischemic strokes. of Merkel cell carcinoma with avelumab has been promising, but its rarity and poor prognosis necessitates close follow up.

The Permanente Journal·Spring 2021 EDITOR-IN-CHIEF: Stephen L. Tarnoff, MD

SENIOR EDITORS

James J. Annesi, PhD, FAAHB, FTOS, FAPA Gus M. Garmel, MD, FACEP, FAAEM Professor, School of Health Professions Clinical Professor of EM (Affiliate) University of Alabama at Birmingham Stanford University Senior Emergency Physician Philip I. Haigh, MD, MSc, FRCSC, FACS Kaiser Permanente Santa Clara Medical Center Assistant Chief, Department of Surgery Santa Clara, CA Kaiser Permanente Los Angeles Medical Center Los Angeles, CA Eric Macy, MD, MS, FAAAAI Department of Allergy David Riley, MD Kaiser Permanente San Diego Medical Center Director, CARE - health research reporting guidelines for case reports Southern California Permanente Medical Group fi Founder, Scienti c Writing in Health and Medicine and CARE-writer San Diego, CA Network Director, HSCaseRepRN case report preprint server (Elsevier) Adjunct Professor, Maryland University of Integrative Health H. Nicole Tran, MD, PhD Portland, OR Internal Medicine Physician, Department of Adult and Family Medicine Director for Quality Improvement and Patient Safety, Internal Medicine Residency Kaiser Permanente Oakland Medical Center Oakland, CA ASSOCIATE EDITORS

Joshua Barzilay, MD Raina Phillips, MD, FACP, FAAP Endocrinologist, The Southeast Permanente Medical Group Internal Medicine and Pediatrics, The Southeast Permanente Professor of Medicine, Emory University School of Medicine Medical Group Adjunct Investigator, Center for Research and Evaluation, KPGA Adjunct Clinical Professor, Department of Internal Medicine, Emory University Atlanta, GA Atlanta, GA Somjot (Sam) Brar, MD MPH Chief/Director, Regional Department of Cardiac Catheterization Chunyuan Qiu, MD, MS Kaiser Permanente Medical Center, Los Angeles, CA Director, Perioperative Service Assistant Clinical Professor of Medicine, UCLA Chief, Department of Anesthesiology and Perioperative Medicine Los Angeles, CA Kaiser Permanente Baldwin Park Medical Center Carrie Davino-Ramaya, MD Baldwin Park, CA Practice Leader and Methodologist of Guidelines and Evidence-Based Medicine Department of Quality Management and Systems Calvin Weisberger, MD, FACC, FACP Northwest Permanente, P.C. Cardiologist Portland, OR Partner Emeritus Kimberly L. Ferrante, MD, MAS Southern California Permanente Medical Group Division of Urogynecology Pasadena, CA Department of Obstetrics and Gynecology Chairman, Southern California Regional Product Council Southern California Kaiser Permanente Medical Group Los Angeles, CA San Diego, CA Lisa J. Herrinton, PhD Scott S. Young, MD Research Scientist, Division of Research Associate Executive Director, Clinical Care and Innovation Kaiser Permanente Northern California Senior Quality Director Oakland, CA The Permanente Federation Oakland, CA Tom M. Judd, MS, CPHIMS, CPHQ, CCE, FACCE, FHIMSS, FAIMBE Senior Medical Director and Executive Director, Care Management Institute Information Technology and Quality Oakland, CA Former National Project Director Kaiser Permanente Clinical Technology Marietta, GA Pat Zrelak, RN, PhD, FAHA, NEA-bc, CNRN, SCRN Health Technology Advisor Clinical Practice Consultant World Health Organization Clinical Education, Practice, & Informatics Washington, DC Kaiser Permanente Board Chair, Global Clinical Engineering Federation Sacramento, CA

EDITORIAL & PUBLISHING OFFICE The Permanente Press Monica Leigh: Managing Editor The Permanente Journal is published KnowledgeWorks Global Ltd.: Composition Services by The Permanente Press The Technology Group, TPMG: Web Developer

The Permanente Journal·Spring 2021 n ORIGINAL RESEARCH ARTICLE Trends in Influenza Vaccine Uptake and Severe Influenza-Related Outcomes at Kaiser Permanente Southern California, 2007-2017

In-Lu Amy Liu, MS1; Hilary C Tanenbaum, PhD, MS, MPH1; Lei Qian, PhD1; Lina S Sy, MPH1; Wansu Chen, PhD, MS1; Steven J Jacobsen, MD, PhD1 Perm J 2021;25:20.154 E-pub: 03/17/2021 https://doi.org/10.7812/TPP/20.154

ABSTRACT assessment is difficult. Although randomized, placebo- Introduction: Major efforts to increase influenza vaccine up- control studies are considered the ideal study design for take among Kaiser Permanente Southern California (KPSC) mem- evaluating vaccine protection and minimizing bias, they are bers have been undertaken in recent years. However, whether typically conducted prelicensure for influenza vaccine rather fl these improvements translate to a decline in severe in uenza- than for postlicensure annual assessments of VE.3,4 Ob- related outcomes has not been examined. We aimed to under- servational cohort studies are perhaps the most common stand the impact of the influenza vaccination program at KPSC by approach used to assess influenza VE, but the findings have examining influenza vaccine uptake and 3 severe influenza- related outcomes. been rather controversial. Researchers have detected sub- Methods: We conducted an ecologic trend analysis to un- stantial selection biases, resulting in an overestimation of fi 5-9 derstand influenza vaccine uptake and influenza-related hospi- vaccine bene ts. For example, a study conducted among talization, intensive care unit (ICU) admission, and mortality for elderly subjects reported that influenza vaccine reduced each influenza season (2007-2017). The same cohort was followed all-cause mortality by more than 50%10; however, this from the influenza season to the noninfluenza season immediately number was subsequently questioned by Simonsen et al,9 afterward while using the noninfluenza season as the comparison who found it difficult to reconcile that the vaccination group. We also assessed the within-season correlation between could prevent more than 10-fold the number of deaths fl fl in uenza vaccine uptake and in uenza-related outcomes. attributed to the virus. Some observational cohort studies fl Results: In uenza vaccine uptake rose from 23.9% to 45.5%, have used laboratory-confirmed influenza cases to evaluate and all 3 influenza-related outcome rates declined (hospitaliza- influenza-related deaths,11,12 but they are likely to provide tion: 35.4-26.8/10,000 patients; ICU: 5.9-5.2/10,000 patients; and 13 mortality: 3.4-2.3/10,000 patients). Influenza vaccine uptake was underestimates of VE. negatively correlated with hospitalization (−0.32, p < 0.001) and Another approach is a test-negative study, a special type mortality (−0.29, p = 0.001). However, once we adjusted for the of case-control study in which all individuals presenting noninfluenza season, the results of the correlation analysis were with an influenza-like illness (ILI) are tested for influenza; no longer statistically significant. cases are those who test positive, and comparison subjects Conclusion: Although we could not establish a statistically are those who test negative.14,15 Although this study de- significant inverse relationship between influenza vaccination sign reduces confounding due to differences in health care and severe influenza-related outcomes over the study period, our seeking behavior, data are sparser on the use of test-negative fi fl ndings indicate an overall decline in in uenza-related outcomes studies to capture severe influenza-related outcomes.4 over the study period, suggesting improvements in both pre- Given the limitations of the aforementioned designs, ventive and acute care quality at KPSC. ecologic trend studies provide another option for assessing the relationship between influenza vaccination and out- INTRODUCTION comes for large study populations. Although trend studies do not take into account individual vaccination status, they Routine annual influenza vaccination is recommended are also not vulnerable to selection bias and can provide for the prevention of influenza among all persons aged valuable information on group-level effects.9 Because ≥ 6 months who do not have contraindications.1 Attention focused on improving influenza vaccine uptake across the ff United States has been relatively e ective, with the Centers Author Affiliation for Disease Control and Prevention reporting that the 1Department of Research and Evaluation, Kaiser Permanente Southern California, Pasadena, CA percentage of vaccinated individuals increased from 43% in the 2010-2011 season to 49% in the 2018-2019 season.2 Corresponding Author In-Lu Amy Liu, MS, ([email protected]) Efforts to increase vaccination rates are undertaken with the assumption that they will translate to decreases in influenza- Keywords: influenza vaccine, hospitalization, intensive care unit (ICU) admission, mortality, ecologic trend related hospitalizations and death. analysis, within-season correlation Different study designs have been used to assess influenza ff Abbreviations: CI, confidence intervals; EHR, electronic health records; ILI, influenza-like illness; IRB, vaccine e ectiveness (VE), but achieving a truly accurate Institutional Review Board; ICU, intensive care unit; KPSC, Kaiser Permanente Southern California; P&I, pneumonia and influenza; RT-PCR, reverse transcription-polymerase chain reaction; VE, vaccine effectiveness. The Permanente Journal·https://doi.org/10.7812/TPP/20.154 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 1 ORIGINAL RESEARCH ARTICLE Trends in Influenza Vaccine Uptake and Severe Influenza-Related Outcomes at Kaiser Permanente Southern California, 2007-2017

influenza is a seasonal infection, the true protective effect of identified using multiple cause-of-death data (P&I ICD vaccination should be limited to the influenza season. codes) from the state of California and supplemented with Analyses comparing outcome rates in the influenza season information from EHR.19 Patients with multiple encounters versus the noninfluenza season assume that the rate for 1 outcome measure (eg, more than 1 hospitalization) in difference can be attributed to influenza16,17;thus,the each season were only counted once. Patients with more than purpose of comparing season and nonseason rates is to 1 outcome measure (eg, hospitalization followed by a transfer adjust for secular confounding. Despite their limitations, to the ICU) were counted once for each. ecologic trend studies have proven valuable in epidemiological Potential Confounders research and for establishing important medical guidelines.18 We identified age category (0-4 years, 5-17 years, 18-49 years, To understand the potential impact of the vaccination 50-64 years, and ≥ 65 years), sex, race/ethnicity (Hispanic, non- program at Kaiser Permanente Southern California (KPSC), Hispanic Asian, non-Hispanic Black, non-Hispanic White, and we examined the association between influenza vaccination Multiple/Other/Unknown race/ethnicity), and place of and severe influenza-related outcomes. e availability of many service (ie, medical center) as potential confounders and years of data from KPSC’s extensive electronic health records adjusted for them in the multivariable analysis. (EHR) and diverse patient population provide a solid footing for using an ecologic trend study to evaluate associations. Analysis Ecologic Trend Analysis METHODS Influenza vaccine uptake and the 3 outcomes (influenza- Study Setting related hospitalization, ICU admission, and death) were We assessed influenza vaccination rates and 3 outcomes evaluated during each influenza season from 2007-2008 (influenza-related hospitalization, intensive care unit [ICU] through 2016-2017. Overall rates were age-, sex-, and race/ admission, and death) from 2007 to 2017 using data from ethnicity-standardized (direct standardization) using the KPSC, which serves more than 4.6 million members across population from the final season (2016-2017) to account for 15 medical centers. KPSC maintains comprehensive demographic changes over the study period. Standardized EHR, which include information on vaccines adminis- rates and rates stratified by age group, sex, and race/ 2 tration, diagnosis and procedure codes, encounter records, ethnicity were plotted. We used χ tests to evaluate pharmacy data, and demographics. group total rate differences for demographic variables. Total rates and 95% confidence intervals (CIs) were estimated Subjects using the binomial proportions exact method. Annual In this study, the annual sample of study subjects was percentage change and 95% CIs were estimated using formed based on the number of KPSC health plan members Poisson regression models adjusting for age, sex, race/ who were actively enrolled on September 1 of each year. ethnicity, and medical center. e same cohort was fol- lowed from the influenza season to the noninfluenza season Measures occurring immediately afterward, using noninfluenza sea- Exposure son data as the comparison group. Influenza vaccine uptake was the main exposure measure. Within-Season Correlation Vaccination uptake was determined using records of influenza In addition to the ecologic trend analysis, we assessed vaccines administered at a KPSC or a partner facility as well as the correlation between influenza vaccine uptake and from patient-reported receipt of the vaccine outside the sys- influenza-related outcome rates in the same season and tem. eannualinfluenza vaccination period was defined as medical center. e correlation was weighted by the September 1 of each year to April 30 of the following year. population size of each medical center. We compared the Outcomes correlation in the influenza season versus the noninfluenza We assessed the rates of 3 influenza-related events: hos- season. enoninfluenza season was used to remove the pitalizations, ICU admissions, and deaths based on pneu- baseline differences in the underlying characteristics of the monia and influenza (P&I) ICD diagnosis codes or population in various medical centers. p Values were underlying cause of death codes (ICD-9: 480-488, ICD- calculated for the Pearson’s correlation coefficients. All 10: J09-J18) in both the influenza season (October-May) analyses were conducted using SAS version 9.4 (SAS and noninfluenza season (the following June-September). Institute Inc, Cary, NC). We used place of service codes (the location where medical care was provided) and event type information to identify Institutional Review Board Approval hospitalizations and department specialty information to Ethical approval for this study was obtained from the identify ICU admissions. Influenza-related deaths were Institutional Review Board of KPSC. Waivers for informed

2 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.154 ORIGINAL RESEARCH ARTICLE Trends in Influenza Vaccine Uptake and Severe Influenza-Related Outcomes at Kaiser Permanente Southern California, 2007-2017

Table 1. Demographic characteristics of study population in Years 1, 5 and 10, 2007-2017, Kaiser Permanente Southern California Year 1a Year 5 Year 10 Season 2007-2008 Season 2011-2012 Season 2016-2017 n%n%n% Ageb 0-4 years 146,309 4.6 152,342 4.5 171,323 4.2 5-17 years 663,634 20.9 668,357 19.7 712,292 17.5 18-49 years 1,390,351 43.8 1,463,049 43.1 1,792,994 44.0 50-64 years 624,385 19.7 692,448 20.4 808,648 19.8 ≥65 years 346,539 10.9 420,877 12.4 588,851 14.5 Sex Female 1,629,829 51.4 1,757,241 51.7 2,098,389 51.5 Male 1,541,389 48.6 1,639,832 48.3 1,975,719 48.5 Race/ethnicity Hispanic 850,279 26.8 1,036,975 31.9 1,492,413 36.6 Multiple/Other/Unknown 681,045 21.5 467,246 14.4 490,279 12.0 Non-Hispanic Asian 223,993 7.1 270,308 8.3 386,146 9.5 Non-Hispanic Black 292,081 9.2 299,728 9.2 334,773 8.2 Non-Hispanic White 1,123,820 35.4 1,171,652 36.1 1,370,497 33.6 Total 3,171,218 100.0 3,397,073 100.0 4,074,108 100.0 a.Cohort for each influenza season was defined as KPSC members on September 1st of each year. b.Subjects’ age for each influenza season was calculated on September 1st of each year.

consent and HIPAA authorization were granted by the Institutional Review Board.

RESULTS General Characteristics of the Study Population KPSC membership steadily increased over the study period, from 3.2 million in the 2007-2008 season to 4.1 million in the 2016-2017 season (Table 1). eper- centage of members age ≥ 65 years increased by approxi- mately 3.6% (from 10.9% to 14.5%), and the percentage of female subjects remained consistent (51.4%-51.5%). ere were notable shifts in race/ethnicity over the study period, with Hispanic membership increasing from Figure 1. Influenza vaccine uptake, 2007-2017§, Kaiser Permanente Southern 26.8% to 36.6% and the Multiple/Other/Unknown California. § Influenza vaccination period defined as September-April.*Overall rates  were standardized for age, sex, and race/ethnicity using data from the last season group decreasing from 21.5% to 12.0%. esubstan- (2016-17). tial changes in Multiple/Other/Unknown race/ethnicity were mainly due to a KPSC initiative implemented in 2011 to collect self-reported race/ethnicity to meet was 6.1% (95% CI = 6.1-6.1%) and 5.4% (95% CI = 5.4- stipulations of meaningful use introduced as part of the 5.4%) (Table 2), respectively. AffordableCareAct. Influenza vaccine uptake improved in all subgroups (see Figures S1 and S2 and Table S1 in the Supplemental Influenza Vaccine Uptake Material) during the study period. Members age ≥ 65 years ere was a notable increase in influenza vaccine uptake consistently had the highest uptake (10-season total rate = over the 10 influenza seasons (Figure 1 and see Table S1 in 69.5%, 95% CI = 69.5-69.6%), and members age 18-49 years the Supplemental Material), from 23.9% in 2007-2008 to had the lowest uptake (25.3%, 95% CI = (25.3% [25.3%, 45.5% in 2016-2017. e total rate of influenza vaccine 25.3%]); p-value <0.001 for five age groups. e total vac- uptake for the 10 seasons was 37.4% (95% CI = 37.4- cination uptake rate was 40.4% (95% CI = 40.4-40.4%) 37.4%). e unadjusted and adjusted annual percent change among women and 34.2% (95% CI = 34.2-34.2%) among

The Permanente Journal·https://doi.org/10.7812/TPP/20.154 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 3 ORIGINAL RESEARCH ARTICLE Trends in Influenza Vaccine Uptake and Severe Influenza-Related Outcomes at Kaiser Permanente Southern California, 2007-2017

Table 2. Adjusted annual percent change of influenza vaccination and influenza-related outcomes, 2007-2017, Kaiser Permanente Southern California Influenza season (October-May) Noninfluenza season (June-September) Adjusted annual Adjusted annual changea (%) 95% CI changea (%) 95% CI Influenza vaccination 5.4 5.4 to 5.4 Influenza-related hospitalization −4.7 −4.9 to −4.5 −5.0 −5.3 to −4.7 Influenza-related ICU admission −3.1 −3.6 to −2.6 −4.1 −4.9 to −3.4 Influenza-related death −5.5 −6.2 to −4.8 −5.7 −6.9 to −4.6 a.Adjusted for age, sex, race/ethnicity, and medical center. ICU = intensive care unit.

men (p < 0.001). Non-Hispanic Asian members had the shifting from 12.5/10,000 patients (2007-2008) to 10.4/ highest mean vaccination rate (48.8%, 95% CI = 48.8- 10,000 patients (2016-2017). e unadjusted annual per- 48.9%), and non-Hispanic White members ranked second cent change was −3.1% (95% CI = −3.4 to −2.7%). and the (42.5%, 95% CI = 42.5-42.5%), followed by Hispanic adjusted annual percent change was −5.0% (95% CI = −5.3 members (36.7%, 95% CI = 36.7-36.8%) and non-Hispanic to −4.7%) (Table 2). Black members (30.8%, 95% CI = 30.7-30.8%). Members Influenza-Related ICU Admissions in the Multiple/Other/Unknown race/ethnicity cate- Influenza Season gory had the lowest uptake (23.1%, 95% CI = 23.1-23.1%; e rate of influenza-related ICU admissions during p < 0.001). the influenza season decreased over the study period, from 5.9 per 10,000 patients in the 2007-2008 season to 5.2 per Outcomes 10,000 patients in the 2016-2017 season (Figure 2 and Influenza-Related Hospitalization see Table S3 in the Supplemental Material). eun- Influenza Season adjusted annual percent change was −1.2% (95% CI = e rate of influenza-related hospitalization during the −1.7 to −0.7%), and the adjusted annual percent change influenza season decreased over the study period, from 35.4/ was −3.1% (95% CI = −3.6 to −2.6%) (Table 2). 10,000 patients in 2007-2008 to 26.8/10,000 patients in Members age ≥ 65 years had the highest mean influenza- 2016-2017 (Figure 2 and see Table S2 in the Supplemental related ICU admission rate during the influenza season Material). e unadjusted annual percent change was −2.8 % (25.8, 95% CI = 25.3-26.3/10,000 patients), whereas (95% CI = −3.0 to −2.6%), and the adjusted annual percent members age 5 to 17 years had the lowest rate (0.9, 95% change was −4.7% (95% CI = −4.9 to −4.5%) (Table 2). CI = 0.8 to 0.9/10,000 patients; p < 0.001). Men had a Members age ≥ 65 years had the highest rate of influenza- higher influenza-related ICU admission rate during the related hospitalization during all influenza seasons (total influenza season (6.2, 95% CI = 6.1-6.3/10,000 patients) rate = 153.0, 95% CI = 151.9-154.1/10,000 patients), compared with women (4.8, 95% CI, 4.7-4.9/10,000 pa- whereas those age 5 to 17 years had the lowest (5.1, 95% CI = tients; p < 0.001). 4.9-5.3/10,000 patients; p < 0.001). Men had a higher mean einfluenza-related ICU admission rate varied by race/ influenza-related hospitalization rate during the influenza ethnicity (see Figure S3 and Table S3 in the Supplemental season (33.0, 95% CI = 32.7-33.3/10,000 patients) com- Material). During the study period, non-Hispanic Black pared with women (29.6, 95% CI = 29.3-29.8/10,000 pa- members had the highest rate (9.1, 95% CI = 8.7-9.4/ tients; p < 0.001). 10,000 patients), and members in the Multiple/Other/ Differences in influenza-related hospitalization rates were Unknown race/ethnicity category again had the lowest also observed by race/ethnicity (see Figure S3 and Table S2 rate (2.0, 95% CI = 1.9-2.1/10,000 patients; p < 0.001). in the Supplemental Material). Non-Hispanic White mem- Noninfluenza Season bers had the highest rate (50.0, 95% CI = 49.6-50.4/10,000 In the noninfluenza season, the influenza-related ICU patients), and members in the Multiple/Other/Unknown admission rate also decreased during the study period, from category had the lowest (12.9, 95% CI = 12.6-13.3/10,000 2.3/10,000 patients in the 2007-2008 season to 2.0/10,000 patients; p < 0.001). patients in the 2016-2017 season. e unadjusted annual Noninfluenza Season percent change was −2.3% (95% CI = −3.0 to −1.5%), and During the noninfluenza seasons, the influenza-related the adjusted annual percent change was −4.1% (95% CI = hospitalization rate also declined during the study period, −4.9 to −3.4%) (Table 2).

4 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.154 ORIGINAL RESEARCH ARTICLE Trends in Influenza Vaccine Uptake and Severe Influenza-Related Outcomes at Kaiser Permanente Southern California, 2007-2017

Figure 2. Rates of influenza-related outcomes per 10,000 patients, 2007-2017, Kaiser Permanente Southern California.

The Permanente Journal·https://doi.org/10.7812/TPP/20.154 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 5 ORIGINAL RESEARCH ARTICLE Trends in Influenza Vaccine Uptake and Severe Influenza-Related Outcomes at Kaiser Permanente Southern California, 2007-2017

 − Table 3. Correlation between influenza vaccine uptake and e unadjusted annual percent change was 3.5% (95% CI = − − influenza-related outcomes, 2007-2017, Kaiser Permanente 4.6 to 2.4%), and the adjusted annual percent change Southern California was −5.7% (95% CI = −6.9 to −4.6%) (Table 2). fl Influenza vaccine uptake Correlation Between In uenza Vaccine Uptake and Influenza-related outcomes weighted correlation (%)a p valueb Influenza-Related Outcomes Influenza season ratesc Table 3 shows the weighted correlation between influ- Hospitalization −0.323 < 0.001 enza vaccine uptake and each influenza-related outcome in ICU admission −0.145 0.089 both the influenza season and noninfluenza season as well Death −0.285 0.001 as the influenza-related outcomes adjusted for the non- Noninfluenza season ratesc influenza season rates. Without controlling for the non- Hospitalization −0.381 < 0.001 influenza season, influenza vaccine uptake was negatively − ICU admission −0.230 0.006 correlated with hospitalization ( 0.32; p < 0.001) and − Death −0.205 0.015 mortality ( 0.29; p < 0.001), but these associations were no fi Adjusted ratesd longer signi cant after adjustment. Hospitalization −0.012 0.891 ICU admission 0.133 0.118 DISCUSSION fl Death −0.095 0.263 Overall, in uenza vaccine uptake rose steadily over the ff a.Weight = subject count in each medical center in each season. study period, suggesting that targeted e orts to improve b.p value calculated using Pearson’s correlation. coverage at KPSC have been successful. At the same time, c. Influenza season defined as October-May; noninfluenza season defined as the June- we found declining rates of severe outcomes attributed to September immediately following the influenza season. d.Correlation between influenza vaccine uptake and influenza-related outcomes, adjusted ILI, with the exception of an uptick during the H1N1 for the following noninfluenza season rate. pandemic in 2009-2010. Similarly, our within-season ICU = intensive care unit. correlation analysis showed a significant inverse associ- ation between vaccination and severe influenza-related Influenza-Related Death outcomes. Despite these encouraging findings, once we Influenza Season adjusted for the noninfluenza season, the results of the Rates of influenza-related deaths occurring within the in- correlation analysis were no longer statistically signifi- fluenza season (Figure 2 and see Table S4 in the Supplemental cant. Although general improvements in care (whether Material) decreased over the study period, from 3.4 per concomitant or through specific interventions) may ex- 10,000 patients (2007-2008) to 2.3/10,000 patients (2016- plain the decrease in severe outcomes, they could be 2017). e unadjusted annual percent change was −3.3% completely confounded and difficult to tease out at the (95% CI = −4.0 to −2.6%), and the adjusted annual percent ecologic level. change was −5.5% (95% CI = −6.2 to −4.8%) (Table 2). Although a number of observational studies have reported Members age ≥ 65 years had the highest mean influenza- powerful protective effects of influenza vaccine against related death rate during the influenza season (16.9, 95% severe outcomes,11,12 our findings are aligned with other CI = 16.5-17.2/10,000 patients), whereas members age 5 to trend analyses. An ecologic trend study of 14 European 17 years had the lowest rate (0.04, 95% CI = 0.03-0.06/ nations identified a significant inverse relationship between 10,000 patients; p < 0.001). Men had a slightly higher influenza vaccination and ILI incidence in only 1 country.20 influenza-related death rate during the influenza season Another study used a cyclical regression model with over 30 (2.8, 95% CI = 2.7-2.9/10,000 patients) than women (2.4, years of US data to examine influenza vaccination and 95% CI = 2.3-2.4/10,000 patients; p < 0.001). mortality (identified with P&I cause-of-death codes) and Differences in influenza-related mortality during the in- could not find a significant association.8 An ecologic trend fluenza season were also observed by race/ethnicity (see study used P&I codes from death certificates and reported Figure S3 and Table S4 in the Supplemental Material). We no decline in mortality despite increasing rates of influenza found that non-Hispanic White members had the highest vaccination coverage.21 death rate (4.8, 95% CI = 4.6-4.9/10,000 patients), whereas Despite some similar findings, not all ecologic studies Hispanic members had the lowest (0.8, 95% CI = 0.8-0.9/ have reported nonsignificant results. A Brazilian study using 10,000 patients; p < 0.001). 14 years of data observed a negative correlation (−0.001; NonInfluenza Season p < 0.001) between vaccination coverage and hospitalization e rates of influenza-related deaths in the noninfluenza rates in the elderly population.22 However, this study did season decreased over the study period, from 1.1/10,000 not provide a comparison group to adjust for secular trends patients (2007-2008) to 0.8/10,000 patients (2016-2017). in care improvement over time. An ecologic trend study

6 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.154 ORIGINAL RESEARCH ARTICLE Trends in Influenza Vaccine Uptake and Severe Influenza-Related Outcomes at Kaiser Permanente Southern California, 2007-2017

using all-cause mortality data identified an inverse rela- the influenza season (November-April) showed that our tionship between influenza vaccination and death.23 It may analysis estimates were robust to changes in the definition of be that the selected outcome measure contributes to dis- influenza season. crepancies found among studies with similar designs. Studies ere are some strengths of our study to note as well, related evaluating influenza-related deaths typically use either P&I to both the study design and setting. First, an advantage of cause-of-death or all-cause mortality data. Due to the using an ecologic study design is that it is relatively quick to broader classification of the latter, estimates obtained with conduct, allowing data to be incorporated as soon as it be- this approach appear to overestimate influenza deaths.24 comes available. In addition, this type of analysis provides a For studies investigating severe outcomes other than high-level overview of trends, which can be helpful in making mortality, laboratory confirmation with reverse transcription- future projections. We also were able to link information from polymerase chain reaction testing is considered the gold KPSC’s EHR with death data from California. standard.25 However, obtaining population-wide estimates Although we could not establish a statistically significant with this approach is infeasible because influenza is not inverse relationship between influenza vaccination and routinely verified with this method. Additionally, the severe influenza-related outcomes over the study period, our onset of secondary complicationsisoftenresponsiblefor findings suggest improvements in both preventive and acute influenza-related hospitalization or death, but the virus care quality at KPSC. is is aligned with a previous in- may have cleared by that point.11,26 Given these limitations, vestigation of mortality trends, which reported declining ICD codes are often used instead, but there are challenges rates of all-cause mortality among KPSC members from with this as well. Using the parent category of P&I codes, 2001 to 2016.19 At the same time, our study highlights influenza cases have reportedly been overestimated,27 and some of the challenges and tradeoffs of examining the using codes for influenza only may result in underestimates. relationship between influenza vaccination and severe out- Furthermore, the nonspecific clinical presentation of influ- comes. ese tradeoffs suggest that no single approach is enza likely corresponds with nonspecific diagnosis coding, likely to provide a definitive estimate of effectiveness and which can lead to inaccurate conclusions about the benefits of that truth may best be determined through the triangulation vaccination.28 of multiple approaches, while taking the necessary factors In addition to the complexities noted above, some specific into account. v limitations of our study should be considered when Supplemental Material interpreting the results. Because ecologic trend studies use aSupplemental Material is available at: www.thepermanentejournal.org/files/ aggregated group-level data, exposure, confounders, and 2021/20.154supp.pdf outcomes are based on population averages. As such, an- alyses were not done on individual-level data, and we cannot Disclosure Statement fl be certain if vaccination exposure preceded our examined In-Lu Amy Liu, MS, reports no con icts of interest. Hilary C Tanenbaum, PhD, MS, MPH, received research contract grants from companies of GlaxoSmithKline outcomes; thus, causal inferences cannot be made. Al- and Modulated Imaging. Lei Qian, PhD, received a research contract grant from though influenza vaccination rates steadily improved at GlaxoSmithKline. Lina S Sy, MPH, received research contract grants from KPSC over the study period from 23.9% to 45.5%, it is companies of GlaxoSmithKline, Dynavax, and Novavax. Wansu Chen, PhD, MS, fl fl possible that our vaccination uptake was too low to detect reports no con icts of interest. Steven J Jacobsen, MD, PhD, reports no con icts of interest xist. significant correlations. Other studies have found that fl higher in uenza vaccination coverage was associated with Authors’ Contributions decreased rates of hospitalization and mortality.29,30 e In-Lu Amy Liu, MS, participated in acquisition and analysis of data and in association between vaccination and influenza-related drafting, critical review, and submission of the final manuscript. Hilary C Tanenbaum, PhD, MS, MPH, participated in drafting, critical review, and outcomes can also vary from year to year based on the fi fl submission of the nal manuscript. Lei Qian, PhD, participated in the study design severity of the circulating in uenza virus subtypes and and critical review of the final manuscript. Lina S Sy, MPH, participated in critical potential mismatch between the vaccine contents and cir- review and submission of the final manuscript. Wansu Chen, PhD, MS, and Steven culating viruses. In addition, because the activity of influ- J Jacobsen, MD, PhD, participated in critical review of the final manuscript. fi enza virus and ILI may be influenced by weather, the true All authors have given nal approval to the manuscript. influenza seasons in Southern California could be different 31 Funding than September to April. Furthermore, there was po- This study obtained financial and material support from Kaiser Permanente tential misclassification because the exact start and end of Southern California (KPSC) internal research funds. In-Lu Amy Liu, MS, received the influenza season each year may have differed from research funding from KPSC internal research funds. Hilary C Tanenbaum, PhD, October and May, which could affect comparisons between MS, MPH, received research contract grants from companies of GlaxoSmithKline fl fl and Modulated Imaging and received research funding from KPSC internal the in uenza season and nonin uenza season. However, research funds. Lei Qian, PhD, received a research contract grant from sensitivity analyses conducted using a narrower period for GlaxoSmithKline and received research funding from KPSC internal research

The Permanente Journal·https://doi.org/10.7812/TPP/20.154 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 7 ORIGINAL RESEARCH ARTICLE Trends in Influenza Vaccine Uptake and Severe Influenza-Related Outcomes at Kaiser Permanente Southern California, 2007-2017

funds. Lina S Sy, MPH, received research contract grants from companies of 15. Sullivan SG, Feng S, Cowling BJ. Potential of the test-negative design for measuring GlaxoSmithKline, Dynavax, Novavax, and received research funding KPSC influenza vaccine effectiveness: A systematic review. Expert Rev Vaccines 2014 Dec; internal research funds. Wansu Chen, PhD, MS, received research funding from 13(12):1571. DOI: https://doi.org/10.1586/14760584.2014.966695, PMID:25348015 fi fl fl KPSC internal research funds. Steven J Jacobsen, MD, PhD, received research 16. Jackson LA. Bene ts of examining in uenza vaccine associations outside of in uenza season. Am J Respir Crit Care Med 2008 Sep;178(5):439-40. DOI: https://doi.org/10. funding from KPSC internal research funds. 1164/rccm.200805-805ED, PMID:18713848 17. Jackson ML. Confounding by season in ecologic studies of seasonal exposures and outcomes: Examples from estimates of mortality due to influenza. Ann Epidemiol 2009 References Oct;19(10):681-91. DOI: https://doi.org/10.1016/j.annepidem.2009.06.009 1. Grohskopf LA, Sokolow LZ, Broder KR, et al. Prevention and control of seasonal influenza 18. Morgenstern H. Uses of ecologic analysis in epidemiologic research. Am J Public Health 1982 with vaccines. MMWR Recomm Rep 2016 Aug;65(5):1-54. DOI: https://doi.org/10.15585/ Dec;72(12):1336-44. DOI: https://doi.org/10.2105/ajph.72.12.1336, PMID:7137430 mmwr.rr6505a1 19. Chen W, Yao J, Liang Z, et al. Temporal trends in mortality rates among Kaiser 2. Lin MH, Wood JR, Mittelman SD, Freyer DR. Institutional adherence to cardiovascular risk Permanente Southern California health plan enrollees, 2001–2016. Perm J 2019 April;23: factor screening guidelines for young survivors of acute lymphoblastic leukemia. J Pediatr 18-213. DOI: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6499114/pdf/18-213.pdf Hematol Oncol 2015 May;37(4):e253-7. DOI: https://doi.org/10.1097/MPH. 20. Spruijt IT, de Lange MM, Dijkstra F, Donker GA, van der Hoek W. Long-term correlation 0000000000000320, PMID:25757021 between influenza vaccination coverage and incidence of influenza-like illness in 14 fl 3. Fukushima W, Hirota Y. Basic principles of test-negative design in evaluating in uenza European countries. PloS One 2016 Sept;11(9):e0163508. DOI: https://doi.org/10.1371/ vaccine effectiveness. Vaccine 2017 Aug;35(36):4796-800. DOI: https://doi.org/10.1016/j. journal.pone.0163508, PMID:27684558 vaccine.2017.07.003, PMID:28818471 21. Rizzo C, Viboud C, Montomoli E, Simonsen L, Miller MA. Influenza-related mortality in the fl 4. World Health Organization. Evaluation of in uenza vaccine effectiveness: A guide to the Italian elderly: No decline associated with increasing vaccination coverage. Vaccine 2006 design and interpretation of observational studies.Geneva: World Health Organization; Oct;24(42-3):6468-75. DOI: https://doi.org/10.1016/j.vaccine.2006.06.052, PMID:16876293 2017. Report No.: 9241512121. 22. Cruzeta AP, Schneider IJ, Traebert J. Impact of seasonality and annual immunization of fl 5. Fireman B, Lee J, Lewis N, Bembom O, van der Laan M, Baxter R. In uenza vaccination elderly people upon influenza-related hospitalization rates. Int J Infect Dis 2013 Dec; and mortality: Differentiating vaccine effects from bias. Am J Epidemiol 2009 Sep;170(5): 17(12):e1194-7. DOI: https://doi.org/10.1016/j.ijid.2013.07.013, PMID:24084246 650-6. DOI: https://doi.org/10.1093/aje/kwp173, PMID:19625341 23. Jansen AG, Sanders EA, Nichol KL, van Loon AM, Hoes AW, Hak E. Decline in influenza- 6. Jackson LA, Jackson ML, Nelson JC, Neuzil KM, Weiss NS. Evidence of bias in estimates associated mortality among Dutch elderly following the introduction of a nationwide of influenza vaccine effectiveness in seniors. Int J Epidemiol 2006 Apr;35(2):337-44. DOI: vaccination program. Vaccine 2008 Oct;26(44):5567-74. DOI: https://doi.org/10.1016/j. https://doi.org/10.1093/ije/dyi274, PMID:16368725 vaccine.2008.08.003, PMID:18722492 7. Jackson ML, Yu O, Nelson JC, et al. Further evidence for bias in observational studies of 24. Li L, Wong JY, Wu P, et al. Heterogeneity in estimates of the impact of influenza on influenza vaccine effectiveness: The 2009 influenza A(H1N1) pandemic. Am J Epidemiol population mortality: A systematic review. Am J Epidemiol 2018 Feb;187(2):378-88. 2013 Oct;178(8):1327-36. DOI: https://doi.org/10.1093/aje/kwt124, PMID:23978527 DOI: https://doi.org/10.1093/aje/kwx270 8. Simonsen L, Reichert TA, Viboud C, Blackwelder WC, Taylor RJ, Miller MA. Impact of 25. Uyeki TM. Influenza diagnosis and treatment in children: A review of studies on influenza vaccination on seasonal mortality in the US elderly population. Arch Intern Med clinically useful tests and antiviral treatment for influenza. Pediatr Infect Dis J 2003 Feb;22(2): 2005 Feb;165(3):265-72. DOI: https://doi.org/10.1001/archinte.165.3.265, PMID: 164-77. DOI: https://doi.org/10.1097/01.inf.0000050458.35010.b6,PMID:12586981 15710788 26. Jhung MA, Swerdlow D, Olsen SJ, et al. Epidemiology of 2009 pandemic influenza A 9. Simonsen L, Taylor RJ, Viboud C, Miller MA, Jackson LA. Mortality benefits of (H1N1) in the United States. Clin Infect Dis 2011 Jan;52(Suppl 1):S13-26. DOI: https://doi. influenza vaccination in elderly people: An ongoing controversy. Lancet Infect Dis org/10.1093/cid/ciq008, PMID:21342884 2007 Oct;7(10):658-66. DOI: https://doi.org/10.1016/S1473-3099(07)70236-0,PMID: 27. Thompson WW, Moore MR, Weintraub E, et al. Estimating influenza-associated deaths in 17897608 the United States. Am J Public Health 2009 Oct;99(Suppl 2):S225-30. DOI: https://doi.org/ 10. Vu T, Farish S, Jenkins M, Kelly H. A meta-analysis of effectiveness of influenza vaccine 10.2105/AJPH.2008.151944, PMID:19797736 in persons aged 65 years and over living in the community. Vaccine 2002 Mar;20(13-4): 28. Vd Hoeven A, Scholing M, Wever P, Fijnheer R, Hermans M, Schneeberger P. Lack of 1831-6.DOI: https://doi.org/10.1016/s0264-410x(02)00041-5, PMID:11906772 discriminating signs and symptoms in clinical diagnosis of influenza of patients admitted to 11. Shang M, Blanton L, Brammer L, Olsen SJ, Fry AM. Influenza-associated pediatric deaths the hospital. Infection 2007 April;35:65. DOI: https://www.ncbi.nlm.nih.gov/pmc/articles/ in the United States, 2010-2016. Pediatrics 2018 Apr;141(4):e20172918. DOI: https://doi. PMC2778620/pdf/15010_2007_Article_6112.pdf. org/10.1542/peds.2017-2918, PMID:29440502 29. Ridenhour BJ, Campitelli MA, Kwong JC, et al. Effectiveness of inactivated influenza 12. Flannery B, Reynolds SB, Blanton L, et al. Influenza vaccine effectiveness against vaccines in preventing influenza-associated deaths and hospitalizations among Ontario pediatric deaths: 2010-2014. Pediatrics 2017 May;139(5): e20164244. DOI: https://doi. residents aged ≥ 65 years: Estimates with generalized linear models accounting for org/10.1542/peds.2016-4244, PMID:28557757 healthy vaccinee effects. PloS One 2013;8(10):e76318. DOI: https://doi.org/10.1371/ 13. Wong KK, Cheng P, Foppa I, Jain S, Fry AM, Finelli L. Estimated paediatric mortality journal.pone.0076318, PMID:24146855 associated with influenza virus infections, United States, 2003-2010. Epidemiol Infect 30. Bonmarin I, Belchior E, Lévy-Bruhl D. Impact of influenza vaccination on mortality in 2015 Feb;143(3):640-7. DOI: https://doi.org/10.1017/S0950268814001198, PMID: the French elderly population during the 2000-2009 period. Vaccine 2015 Feb;33(9): 24831613 1099-101. DOI: https://doi.org/10.1016/j.vaccine.2015.01.023, PMID:25604800 14. Jackson ML, Nelson JC. The test-negative design for estimating influenza vaccine 31. Van Noort SP, Aguas´ R, Ballesteros S, Gomes MG. The role of weather on the relation effectiveness. Vaccine 2013 Apr;31(17):2165-8. DOI: https://doi.org/10.1016/j.vaccine. between influenza and influenza-like illness. J Theor Biol 2012 Apr;298:131-7. DOI: 2013.02.053, PMID:23499601 https://doi.org/10.1016/j.jtbi.2011.12.020, PMID:22214751

8 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.154 n ORIGINAL RESEARCH ARTICLE Ten-year Thyroid Cancer Incidence in an Integrated Healthcare Delivery System

Stephanie J. Kim, MD, MPH1; Megan L. Durr, MD2,3; Jeanne A Darbinian, MPH4; Lori C. Sakoda, PhD3,4; Charles J. Meltzer, MD3,5; Hasmik Arzumanyan, MD3,6; Kevin H. Wang, MD2,3; Jonathan K. Lin, MD2,3; Deepak Gurushanthaiah, MD2,3; Joan C. Lo, MD1,3,4,6 Perm J 2021;25:20.209 E-pub: 3/10/2021 https://doi.org/10.7812/TPP/20.209

ABSTRACT large increases in thyroid cancer incidence from 1993 to 2011, Introduction: The incidence of papillary thyroid cancer (PTC) whereas thyroid cancer-related mortality rates remained stable; has increased in recent decades, but data from community-based the authors attributed the rising incidence to increased settings are limited. This study characterizes PTC trends in a large, screening, including thyroid sonography, rather than an actual integrated healthcare system over 10 years. increase in thyroid disease.5 Other US studies, using data Methods: The annual incidence of PTC (2006-2015) was ex- from the Veterans Affairs Health System6 and private and amined among Kaiser Permanente Northern California adults public insurance claims databases,7 also reported an in- aged 21 to 84 years using Cancer Registry data, including tumor crease in thyroid cancer incidence, accompanied by an size and stage. Incidence estimates were age-adjusted using the fi 2010 US Census. increase in thyroid ultrasound and/or thyroid ne needle Results: Of 2990 individuals newly diagnosed with PTC aspiration (FNA) procedures. (76.8% female, 52.7% non-Hispanic White), 38.5% and 61.5% New considerations have arisen following recent ana- wereaged<45and<55years,respectively.Atdiagnosis, lyses of SEER-9 data from 1974 to 2013 that demonstrated 60.9% had PTC tumors ≤ 2 cm, 9.2% had tumors > 4 cm, and concomitant increases in the annual incidence and mortality 66.1% had Stage I disease. The annual age-adjusted incidence of rate of thyroid cancer for all sex, race, and age groups as well PTC increased from 9.4 (95% confidence interval [CI] = 8.1-10.7) to as for every stage and tumor size category at diagnosis.8 14.5 (95% CI = 13.1-16.0) per 100,000 person-years and was higher Notably, this included an increase in thyroid cancer inci- for female patients than for male patients. Incidence tended to be dence for advanced-stage PTC, suggesting that the rise in higher in Asian/Pacific Islanders and lower in Black individuals. incidence may reflect a true increase in disease occurrence Increasing incidence was notable for Stage I disease (especially 2006-2012) and evident across a range of tumor sizes (3.0-4.6 and not just overdiagnosis from greater use of ultrasound and 8 for ≤ 1 cm, 2.5-3.5 for 1-2 cm, and 2.4-4.7 for 2-4 cm) but was modest other imaging modalities. Others have similarly reported an for large tumors (0.9-1.5 for > 4 cm) per 100,000 person-years. increased trend in the incidence of larger tumors, including data Discussion: Increasing PTC incidence over 10 years was most from areas of high and low socioeconomic status, which would evident for tumors ≤ 4 cm and Stage I disease. Although these be less likely to reflect the increased screening of asymptomatic findings may be attributable to greater PTC detection, the in- cases.1 However, systematic data pertaining to thyroid cancer crease across a range of tumor sizes suggests that PTC burden incidence from single community-based healthcare systems, might also have increased. where care delivery and case ascertainment practices are also more consistent, have not yet been reported. For the current study, we used data from the Kaiser Permanente Northern California (KPNC) Cancer Registry INTRODUCTION to characterize recent trends in PTC incidence, overall and  e overall incidence of thyroid cancer has increased, by tumor size, from 2006 to 2015 within a single large, 1 especially over the last 3 decades. Based on data from the integrated healthcare delivery system. e selected time National Cancer Institute Surveillance, Epidemiology and End window represented a period of contemporary endocrine Results (SEER-9) Registry, thyroid cancer incidence nearly tripled from 1975 to 2009, largely attributed to an increase in the detection of papillary thyroid carcinoma (PTC), the most ffi 2 Author A liations common histologic variant. A follow-up study using SEER 1Department of Medicine, Kaiser Permanente Oakland Medical Center, Oakland, CA data from 1983 to 20123 showed relative stabilization in the 2Department of Head and Neck Surgery, Kaiser Permanente Oakland Medical Center, Oakland, CA incidence of all types of thyroid cancer between 2010 and 2012, 3The Permanente Medical Group, Oakland, CA fi fl 4Division of Research, Kaiser Permanente Northern California, Oakland, CA suggesting that these ndings may in part re ect changes 5Department of Head and Neck Surgery, Kaiser Permanente Santa Rosa Medical Center, Santa Rosa, CA in clinical practice, including implementation of the 2009 6Department of Endocrinology, Kaiser Permanente Oakland Medical Center, Oakland, CA American yroid Association (ATA) thyroid nodule guidelines4 that refined criteria for which nodules should Corresponding Author Joan C Lo, MD, ([email protected]) be biopsied.3 Researchers in South Korea also reported Keywords: cancer, carcinoma, incidence, papillary, trends, thyroid The Permanente Journal·https://doi.org/10.7812/TPP/20.209 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 9 ORIGINAL RESEARCH ARTICLE Ten-year Thyroid Cancer Incidence in an Integrated Healthcare Delivery System

practice within our health system that included office-based with the eighth edition,14 raising the age cutoff from 45 to 55 thyroid sonography and was consistent with the clinical years for Stage I-II in younger patients, but this change does approach set forth in the 2009 ATA thyroid nodule not affect the time period of our study. guidelines.4 is time period allowed us to examine trends For population denominators used to calculate disease over time prior to the 2016 implementation of new incidence, we obtained KPNC membership data for adults evidence-based workflows for thyroid nodules and thyroid aged 21 to 84 years using membership counts at the midpoint cancer within KPNC9 that included the revised national (ie, July 1) of each calendar year of the study. Race/ethnicity ATA guidelines released at the end of 2015.10 We hypoth- was determined using administrative data and classified esized that, for the study period of 2006-2015, detection of as non-Hispanic White, Black, Hispanic, Asian/Pacific early stage disease would contribute to rising PTC incidence, Islander, and other or unknown race/ethnicity. e study possibly attributable to identification of thyroid nodules by was approved by the KPNC Institutional Review Board, ultrasound that would otherwise not have been detected and a waiver of consent was obtained due to the nature during routine neck examination. However, the incidence of the study. would likely plateau over time if the true burden of disease was not increased. Statistical Analyses e annual incidence of PTC with 95% confidence intervals METHODS (CI) was calculated overall, by sex, and by race/ethnicity Study Population and Setting using KPNC health plan membership denominators, with KPNC is a large, integrated healthcare delivery system age-adjusted rates standardized to the 2010 US Census. providing care to more than 4 million members. emem- Subgroup-specific rates based on tumor size and cancer bership population is racially and ethnically diverse, with stage were also examined. All analyses were conducted using demographic and health-associated characteristics similar SAS version 9.3 (SAS Institute, Cary, NC). to those of the general northern California population.11 Each of the 21 KPNC medical centers is staffed by clinical RESULTS endocrinologists, head and neck surgeons, and/or general Among the adult population aged 21 to 84 years, we surgeons with expertise in the medical and surgical man- identified 2990 new cases of PTC diagnosed during the agement of differentiated thyroid carcinoma. 10-year period from 2006 to 2015. Two-thirds of PTC Incident cases of PTC diagnosed from 2006 to 2015 in cases (66.1%) were classified as Stage I disease based on AJCC adults aged 21 to 84 years were identified using the KPNC sixth and seventh edition criteria. e average age (± standard Cancer Registry. Following SEER Program standards, the deviation) at PTC diagnosis was 49.6 ± 14.5 years, and KPNC Cancer Registry includes data on all patients di- 61.5% and 38.5% of diagnosed individuals were aged < 55 agnosed or treated with any primary cancer (except non- and < 45 years, respectively, at diagnosis. Overall, 52.7% melanoma skin cancer) at its medical centers since 1988. individuals were non-Hispanic; 76.8% of patients were Histologic types of thyroid cancer were defined using mor- female. Moreover, 60.9% and 9.2% of individuals with PTC phology codes as proposed by Lim et al8: PTC (8050, 8260, had tumor sizes ≤ 2 and > 4 cm, respectively. Demographic 8340-8344, 8350, 8450-8460), follicular thyroid cancer (8290, and tumor characteristics of incident PTC cases, overall and 8330-8335), medullary thyroid cancer (8345, 8510-8513), stratified by age at diagnosis, are shown in Table 1. and anaplastic thyroid cancer (8020-8035). Identified PTC e overall age-adjusted incidence of PTC increased by cases were characterized in the Cancer Registry with respect 54.3% from 9.4 (95% CI = 8.1-10.7) to 14.5 (95% CI = to tumor size and stage, the latter informed by tumor size, 13.1-16.0) per 100,000 person-years between 2006 and evidence of extrathyroidal extension (metastatic disease), and 2015 (Figure 1A). Age-adjusted incidence was higher for the extent and level of lymph node involvement. Tumor stage female patients, increasing by 60.9% from 12.8 (95% CI = at diagnosis was classified following the American Joint 10.9-15.1) to 20.6 (95% CI = 18.3-23.2) per 100,000 Commission on Cancer (AJCC) Tumor-Node-Metastasis person-years over the 10-year period. Incidence in female Staging System (sixth edition for tumors diagnosed from patients reached a peak of 21.8 (95% CI = 19.3-24.1) in 2006 to 2009 and seventh edition for tumors diagnosed from 2012 and stabilized over the subsequent 3 years, with an 2010 to 2015). No major changes in staging for PTC oc- incidence of 20.6 (95% CI = 18.3-23.2) per 100,000 curred between the sixth and seventh editions that would person-years in 2015. e age-adjusted incidence in male have affected classification in this study. Consistent with the patients was 7.9 (95% CI = 6.5-9.6) per 100,000 person- AJCC/TMN criteria for staging of thyroid cancer,12,13 we years in 2015. Figure 1B shows the variation in PTC in- used an age threshold of 45 years to compare subgroups. We cidence by race/ethnicity over time. e incidence of PTC note that the AJCC PTC staging guidelines changed in 2018 tended to be slightly higher in Asian/Pacific Islander than

10 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.209 ORIGINAL RESEARCH ARTICLE Ten-year Thyroid Cancer Incidence in an Integrated Healthcare Delivery System

Table 1. Baseline demographic and clinical characteristics of adults with papillary thyroid cancer, 2006-2015 Age < 45 y Age 45-84 y Total population (N = 2990) (n = 1150) (n = 1840) Age in years, mean ± SD 49.6 ± 14.5 34.6 ± 6.4 59.0 ± 9.4 Female sex, % 76.8 80.9 74.2 Race/ethnicity, % Non-Hispanic White 52.7 46.4 56.7 Black 4.7 4.2 5.1 Hispanic 17.4 22.5 14.1 Asian/Pacific Islander 22.7 25.0 21.3 Multiple/other/unknown 2.5 1.9 2.8 Tumor size, % 0-1.0 cm 33.0 26.2 37.3 1.1-2.0 cm 27.9 29.0 27.2 2.1-4.0 cm 26.1 31.0 22.9 ≥ 4.1 cm 9.2 10.7 8.2 Unknown 3.9 3.0 4.4 AJCC/TNM stage, % I 66.1 97.9 46.2 II 8.2 1.6 12.4 III 11.8 0 19.2 IV 9.1 0 14.8 Unknown 4.8 0.5 7.4 AJCC = American Joint Commission on Cancer; TNM = tumor node metastasis.

in non-Hispanic White and Hispanic individuals for most for higher stage tumors were modest and variable. A small years and lower among Black individuals. decline in the number of cases with undefined tumor stage Figure 1C illustrates the incidence of PTC by tumor in the KPNC Cancer Registry was also observed, especially size each year. e annual age-adjusted incidence of PTC in the latter years. increased for tumors ≤ 4 cm between 2006 and 2015. For tumors ≤ 1 cm, incidence increased from 3.0 (95% CI = DISCUSSION 2.3-3.8) to 4.6 (95% CI = 3.8-5.5) during the period Within our integrated healthcare system, we found an studied, with a peak of 5.2 (95% CI = 4.3-6.2) per 100,000 overall increase in PTC incidence from 2006 to 2015. is person-years in 2012. e incidence of tumors 1.1 to 2.0 cm period was characterized by relatively consistent clinical increased from 2.5 (95% CI = 1.9-3.2) to 3.5 (95% CI = practice pertaining to thyroid cancer screening and man- 2.8-4.3), with a peak of 4.8 (95% CI = 3.9-5.8) per 100,000 agement in our organization, during which thyroid so- person-years in 2011. For tumors 2.1 to 4.0 cm, incidence nography (including the availability of office-based thyroid increased from 2.4 (95% CI = 1.8-3.1) to 4.7 (95% CI = sonography) increased. Others have also observed national 3.9-5.6) per 100,000 person-years in 2006-2015. In con- increases in PTC incidence using data from SEER, in- trast, for tumors > 4.0 cm, changes in incidence were cluding data from KPNC that have contributed to the larger more modest, increasing from 0.9 (95% CI = 0.5-1.4) to 1.5 SEER-91-3,8 and SEER-1315 datasets. Although the SEER (95% CI = 1.1-2.0) per 100,000 person-years. Approxi- program uses race/ethnicity- and sex-specificpopulation mately 4% of cases had tumor size undefined in the KPNC denominators based on county population estimates that Cancer Registry across all years, and the proportion with have been aggregated,16 an important contribution of our undefined sizes declined over the observation period. study is the examination of findings from a single integrated Figure 1D shows annual age-adjusted incidence of PTC healthcare delivery system serving a diverse northern California by tumor stage, which was most notable for Stage 1 tumors, patient population, with population denominators derived which increased from 6.4 (95% CI = 5.4-7.6) in 2006 to a from the same health plan membership from which the peak of 10.7 (95% CI = 9.4-12.2) per 100,000 person-years PTC cases were identified. in 2012 and then decreased to 9.1 (95% CI = 8.0-10.3) per A national study of the US Veterans Affairs Health Care 100,000 person-years in 2015. Changes in PTC incidence System found that thyroid cancer incidence doubled from

The Permanente Journal·https://doi.org/10.7812/TPP/20.209 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 11 ORIGINAL RESEARCH ARTICLE Ten-year Thyroid Cancer Incidence in an Integrated Healthcare Delivery System

Figure 1. Incidence of papillary thyroid cancer, 2006-2015. (A) Age-adjusted incidence overall and by sex. (B) Age-adjusted incidence by race/ethnicity. Figure 1. (Continued). (C) Age-adjusted incidence by tumor size. (D) Age-adjusted incidence by stage at diagnosis. adults except for 2011-2012, when the incidence was similar between Black and non-Hispanic White adults. In 2000 to 2012, corresponding to a 4.6-fold increase in comparison to national observations, our study population thyroid ultrasound use and a 6.6-fold increase in thyroid reflects the diversity of the northern California population FNA procedures.6 Data from the Healthcare Cost and and contributes a larger proportion of Asian/Pacific Is- Utilization Project Nationwide Inpatient Sample, the landers and immigrants than other regions of the US. omson Reuters MarketScan Outpatient View data- Furthermore, all KPNC health plan members have access to base, and the American Cancer Society similarly in- healthcare and preventive exams, which may explain the dicated that the incidence of thyroid cancer increased relatively high incidence of PTC among those of non-White by 59% from 2006 to 2011, accompanied by more than race/ethnicity. Also consistent with published SEER-9 re- double the number of thyroid FNAs performed and sults from 1974 to 2013 by Lim et al8, the rise in new cases of a 31% increase in number of thyroid nodule-related PTC over the 10-year observation period was reflected across operations.7 Our results support the increase in thyroid most tumor sizes ≤ 4 cm, although large increases in inci- cancer trend reported by others, with additional obser- dence were seen primarily for Stage I disease. e findings vations about tumor size, stage, and demographic factors from SEER-98 also demonstrated an increase in identi- that complement existing reports. fication of localized PTC disease. Investigation of thyroid cancer incidence by race/ethnicity We postulate that the rising incidence of PTC is largely using SEER-13 data from 1992 to 2010 found an increase in attributable to higher rates of PTC detection but similar observed thyroid cancer incidence in non-Hispanic compared to others,1,8 we cannot exclude the possibility that PTC with Hispanic adults and in White adults compared with burden has also increased due to other contributing factors. Asian/Pacific Islander adults, with the highest overall in- e availability of thyroid sonography, integration of office- cidence in White adults.15 In contrast, we observed a based thyroid sonography in current endocrine practice, and slightly higher incidence of PTC among Asian/Pacific greater incidental thyroid screening would be expected to Islander, followed by non-Hispanic White and Hispanic contribute to greater detection of thyroid nodules and sub- adults, with a generally lower incidence of PTC among Black sequent thyroid FNA biopsies, as demonstrated in multiple

12 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.209 ORIGINAL RESEARCH ARTICLE Ten-year Thyroid Cancer Incidence in an Integrated Healthcare Delivery System

prior published reports.6,7 ese clinical management factors before 2012. Finally, the number of PTC cases identified likely explain the increased detection of early-stage thyroid within KPNC was relatively small compared with large cancer and small tumors, with peak incidence evident in epidemiologic reports based on data from multiple SEER 2012, after which the detection of small tumors appears to regions; therefore, we did not attempt to calculate the annual have declined. In analyses of more than 77,000 patients with percentage change by tumor size or stage for comparison. thyroid cancer, Lim et al8 observed that the increase in thyroid e primary goal of this study was to examine population- cancer incidence is accompanied by a concomitant increase in level PTC trends within a single health system, where the thyroid cancer mortality, possibly reflecting an actual increase strengths of our data include use of a comprehensive cancer in thyroid disease burden, and cited risk factors such as the registry coupled with known health plan population de- increasing burden of obesity, exposure to ionizing radiation, nominators to calculate disease incidence, as opposed to and endocrine-disrupting chemicals as potential contributors. determining population denominators from US Census Indeed, recent data regarding environmental exposures report data at the local or county level (as implemented for SEER a potential association of endocrine disruptors and PTC risk.17 Registry-based studies). Additionally, we conducted our Individuals with increased exposure to selected flame- study in a single integrated healthcare delivery system over retardant chemicals previously shown to be associated with a period of relatively consistent clinical practice related to thyroid disease in women18 were twice as likely to develop thyroid cancer screening, whereas most previous studies PTC, and specifictypesofflame retardants have also been focused primarily on data from multiple health systems associated with tumor size and severity.17 with variable practices. Furthermore, the KPNC population Although the burden of newly identified PTC cases has was large and racially and ethnically diverse, allowing for clearly increased, the revised 2015 ATA guidelines currently comparisons across ethnic subgroups. support a less aggressive surgical approach and surveillance for low-risk PTC and allow for safe monitoring of small, CONCLUSIONS incidentally detected thyroid nodules with low-risk imaging In summary, we found an overall increase in PTC in- phenotypes.10 Several recent studies demonstrate that active cidence from 2006 to 2015 within our large, integrated surveillance of low-risk papillary microcarcinoma for se- healthcare delivery system, supporting findings from other lected patients appears to be feasible and may even become studies. e increase in PTC incidence was reflected largely the standard of care in certain countries.19 Data from Japan for tumors ≤ 4 cm. At diagnosis, the majority of PTC cases suggest that active surveillance of these select tumors had tumors ≤ 2 cm and Stage I disease and, relevant to the (without thyroidectomy) could be a safe and cost-effective age thresholds used in the 2018 AJCC staging criteria, alternative in health systems where patients are able to be occurred among adults aged < 55 years. Although these carefully monitored.19 Future studies should be conducted findings may be attributable to higher rates of PTC de- to examine trends in PTC incidence and healthcare ex- tection, the modest increases observed in the incidence penditures as practice patterns shift toward less aggressive of larger tumors raise the possibility that the true burden intervention for low risk thyroid nodules, especially PTC of PTC may also have increased. Additionally, higher tumors ≤ 1 cm in size. PTC incidence was noted in Asian/Pacific Islander adults, Our study has several limitations to consider. First, we compared with non-Hispanic White, Hispanic, and Black report data on a population of northern California adults adults. is study provides an important benchmark for who have access to healthcare, which may limit the gen- future studies following implementation of the revised 2015 eralizability of our findings; however, the large size and ATA guidelines for the management of thyroid nodules and consistency of practice within our region allows more differentiated thyroid cancer.10 v specific examination of incidence trends not influenced by practice variation and access to care, as would be the case with national data. Second, we did not examine the con- Disclosure Statement The author(s) have no conflicts of interest to disclose. comitant frequency of thyroid sonography and FNA biopsy and are thus unable to determine the extent to which our fi Acknowledgments ndings are due to increased diagnostic activity, an im- The authors thank Jennifer Green for editorial assistance. portant speculation. During the latter part of the obser- vation period, the proportion of PTC cases with unknown Authors’ Contributions tumor size/stage in the KPNC Cancer Registry declined, Stephanie Kim, MD, MPH; Joan C Lo, MD; Hasmik Arzumanyan, MD; Jeanne A Darbinian, MPH; Megan Durr, MD; and Lori C Sakoda, PhD, conceived the study. potentially contributing to a slightly greater incidence of Charles Meltzer, MD; Kevin Wang, MD; Jonathan Lin, MD; and Deepak PTC by tumor size and stage, although this would not Gurushanthaiah MD, provided input on the design. Jeanne A Darbinian, MPH; explain the large increase in prevalence of Stage I disease led the acquisition and analysis of data. Stephanie Kim, MPH, drafted the

The Permanente Journal·https://doi.org/10.7812/TPP/20.209 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 13 ORIGINAL RESEARCH ARTICLE Ten-year Thyroid Cancer Incidence in an Integrated Healthcare Delivery System

manuscript in collaboration with Joan C Lo, MD; Jeanne A Darbinian, MPH; Lori C 7. Sosa JA, Hanna JW, Robinson KA, Lanman RB. Increases in thyroid nodule fine-needle Sakoda, PhD; Hasmik Arzumanyan, MD; Charles Melzer, MD; and Megan Durr, aspirations, operations, and diagnoses of thyroid cancer in the United States. Surgery MD. All authors provided important input, revised the manuscript for important 2013 Dec;154(6):1420-7. DOI: https://doi.org/10.1016/j.surg.2013.07.006, PMID: 24094448 intellectual content, and approved the final manuscript for submission. 8. Lim H, Devesa SS, Sosa JA, Check D, Kitahara CM. Trends in thyroid cancer incidence and mortality in the United States, 1974-2013. J Am Med Assoc 2017 Apr;317(13): Funding 1338-48. DOI: https://doi.org/10.1001/jama.2017.2719, PMID:28362912 This study was supported by Kaiser Permanente Northern California Graduate 9. Meltzer C, Budayr A, Chavez A, et al. Evidence-based workflows for thyroid and Medical Education, funded by the KPNC Community Benefit Program. The sponsor parathyroid surgery. Perm J 2016 Summer; 20(3):16-035. DOI: https://doi.org/10.7812/ TPP/16-035, PMID:27479948 had no role in the study design, data collection, analysis, and interpretation; writing 10. Haugen BR, Alexander EK, Bible KC, et al. 2015 American thyroid association of the report; and the decision to submit for publication. management guidelines for adult patients with thyroid nodules and differentiated thyroid cancer: The American thyroid association guidelines task force on thyroid nodules and Abbreviations differentiated thyroid cancer. Thyroid. 2016 Jan;26(1):1-133. DOI: https://doi.org/10.1089/ thy.2015.0020, PMID:26462967 AJCC, American Joint Commission on Cancer; ATA, American Thyroid 11. Gordon N, Lin T. The Kaiser Permanente Northern California adult member health survey. Association; CI, Confidence intervals; FNA, Fine-needle aspiration; KPNC, Perm J 2016 Aug;20(4):15-225. DOI: https://doi.org/10.7812/TPP/15-225 Kaiser Permanente Northern California; PTC, Papillary thyroid cancer; SEER-9, 12. Greene FL PD, Fritz AG, Balch CM, Haller DG, Morrow M. AJCC cancer staging manual. Surveillance, Epidemiology and End Results; TNM, Tumor-node-metastasis Chicago: American College of Surgeons; 2001. 13. Edge SB BD, Compton CC, Fritz AG, Greene FL, Trotti A. AJCC cancer staging manual. References Chicago: American College of Surgeons; 2010. 1. Li N, Du XL, Reitzel LR, Xu L, Sturgis EM. Impact of enhanced detection on the increase in 14. Tuttle RM, Haugen B, Perrier ND. Updated American Joint Committee on Cancer/Tumor- thyroid cancer incidence in the United States: Review of incidence trends by socioeconomic Node-Metastasis Staging System for differentiated and anaplastic thyroid cancer (eighth status within the surveillance, epidemiology, and end results registry, 1980-2008. Thyroid edition): What changed and why? Thyroid 2017 Jun;27(6):751-6. DOI: https://doi.org/10. 2013 Jan;23(1):103-10. DOI: https://doi.org/10.1089/thy.2012.0392, PMID:23043274 1089/thy.2017.0102, PMID:28463585 2. Davies L, Welch HG. Current thyroid cancer trends in the United States. JAMA 15. Magreni A, Bann DV, Schubart JR, Goldenberg D. The effects of race and ethnicity on Otolaryngol Head Neck Surg 2014 Apr;140(4):317-22.DOI: https://doi.org/10.1001/ thyroid cancer incidence. JAMA Otolaryngol Head Neck Surg 2015 Apr;141(4):319-23. jamaoto.2014.1, PMID:24557566 DOI: https://doi.org/10.1001/jamaoto.2014.3740, PMID:25654447 3. Morris LG, Tuttle RM, Davies L. Changing trends in the incidence of thyroid cancer in the 16. Centers for Disease Control and Prevention. United States cancer Statistics (USCS): United States. JAMA Otolaryngol Head Neck Surg 2016 Jul;142(7):709-11. DOI: https:// Population denominator data Sources. Accessed May 14, 2019. https://www.cdc.gov/ doi.org/10.1001/jamaoto.2016.0230, PMID:27078686 cancer/npcr/uscs/2011/technical_notes/data_sources/population.htm. 4. Cooper DS, Doherty GM, Haugen BR, et al. Revised American Thyroid Association 17. Hoffman K., Lorenzo A., Butt C. M., et al. Exposure to flame retardant chemicals and management guidelines for patients with thyroid nodules and differentiated thyroid cancer. occurrence and severity of papillary thyroid cancer: A case-control study. Environ Int 2017 Thyroid 2009 Nov;19(11):1167-214. DOI: https://doi.org/10.1089/thy.2009.0110 Oct;107:235-42. DOI: https://doi.org/10.1016/j.envint.2017.06.021 5. Ahn HS, Kim HJ, Welch HG. Korea’s thyroid-cancer "epidemic"--screening and 18. Allen JG, Gale S, Zoeller RT, Spengler JD, Birnbaum L, McNeely E. PBDE overdiagnosis. N Engl J Med 2014 Nov;371(19):1765. DOI: https://doi.org/10.1056/ flame retardants, thyroid disease, and menopausal status in U.S. women. Environ NEJMp1409841, PMID:25372084 Health 2016 May;15(1):60. DOI: https://doi.org/10.1186/s12940-016-0141-0, 6. Zevallos JP, Hartman CM, Kramer JR, Sturgis EM, Chiao EY. Increased thyroid cancer PMID:27215290 incidence corresponds to increased use of thyroid ultrasound and fine-needle aspiration: 19. Ito Y, Miyauchi A, Oda H. Low-risk papillary microcarcinoma of the thyroid: A review of A study of the Veterans Affairs health care system. Cancer 2015 Mar;121(5):741-6. DOI: active surveillance trials. Eur J Surg Oncol 2018 Mar;44(3):307-15. DOI: https://doi.org/ https://doi.org/10.1002/cncr.29122, PMID:25376872 10.1016/j.ejso.2017.03.004, PMID:28343733

14 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.209 n ORIGINAL RESEARCH ARTICLE Using Real-world Data for Decision Support: Recommendations from a Primary Care Provider Survey

Patricia A Arean,´ PhD1,2; Emily C Friedman, MID, CPE2; Abhishek Pratap, PhD3; Ryan Allred, BA1; Jaden Duffy, BA1; Sara Gille, MPH4; Shelley Reetz, BS4; Erin Keast, MPH4; Gregory Clarke, PhD4 Perm J 2021;25:20.213 E-pub: 03/01/2021 https://doi.org/10.7812/TPP/20.213

ABSTRACT as the Patient Health Questionnaire 9,3 which is based on Introduction: The use of data from wearable sensors, smart- retrospective self-report of symptoms and is collected only phones, and apps holds promise as a clinical decision-making tool in sporadically. Indeed, there is a marked decrease in the health and mental health in primary care medicine. The aim of this number of follow-up depression assessments in primary care study was to determine provider perspectives about the utility of medicine in people who screen positive for depression and these data for building digitally based decision-making tools. receive treatment for it.4 Self-reports also are not infor- Methods: This mixed quantitative and qualitative cross- mative about when treatment should be augmented or sectional survey of a convenience sample of primary-care clini- switched, or if a patient needs to be seen immediately for cians at Kaiser Permanente Northwest was conducted between fi April and July 2019 online via Institute for Translational Health emergency reasons. Although patients nd these measures fi Sciences’ Research Electronic Data Capture. Study outcomes were somewhat informative, they also nd that the questions 1) attitudes toward digital data, 2) willingness to use digital data asked do not assess important measures of improvement, to support clinical decision making, and 3) concerns and rec- such as activity, social connectedness, and work productivity.5 ommendations about implementing a digital tool for clinical Decision support and access to expert opinion on the de- decision making. livery of depression care is limited and impacts the quality of Results: This sample of 131 clinicians was largely white (n = 98) care substantially.6 is problem is recognized by many female (n = 91) physicians (n = 86). Although respondents (75.7%, large health care systems that want to support the use of n = 87) had a positive attitude toward using digital tools in their decision support tools.7,8 practice, 88 respondents (67.3%) voiced concerns about the ff ffi To mitigate this problem, recent e orts have turned out possible lack of clinical utility, suspected di culty in integration — with clinical workflows, and worried about the potential burden the use of Clinical Decision Support Systems (CDSS) placed on patients. Participants indicated that the accuracy of the data analytic tools embedded in electronic health records data in detecting the need for treatment adjustments would need that compile patient information, and synthesize and vi- to be high and the tool should be clinically tested. sualize the information to support clinicians in making Conclusions: Primary care providers find value in collecting treatment decisions. Preliminary evidence suggests these real-world patient data to assist in clinical decision making, tools can be effective in supporting integrated mental health provided such information does not interfere with provider programs.9,10 A growing interest in the informatics field workflow or impose undue burden on patients. In addition, digital is the addition of data streams from ubiquitous sensing tools will need to demonstrate high accuracy, be able to integrate technologies and smartphone applications (patient-com- fl into current clinical work ows, and maintain the privacy and pleted apps as a means of informing and improving clinical security of patients’ data. decision making, with the intention of embedding these data into CDSS).11 e large scale and frequent use of INTRODUCTION smartphones have the potential to capture changes in mood, activity, and function in real time, with minimal burden to Primary care medicine is the first stop for the assessment the patient. A growing number of electronic health record and treatment of mental health problems. As a result, the apps that are both patient- and provider-facing tools are integration of mental health services into primary care being deployed for health-care reasons,12 but the application medicine is now the subject of substantial dissemination to mental health care is in a nascent stage. An additional and implementation supported through state, regional, and national efforts, including the implementation of specific billing codes for Medicare insurance recipients.1,2 A criti- Author Affiliations cal feature of good integrated care is continuous outcome 1Department of Psychiatry and Behavioral Sciences, University of Washington, Seattle, WA 2ALACRITY Center, University of Washington, Seattle, WA measurement to monitor treatment response and access to 3Sage Bionetworks, Seattle, WA expert opinion when patients fail to respond as anticipated. 4Kaiser Permanente Center for Health Research, Portland, OR However, the monitoring of treatment outcomes and me- diation at scale is challenging. Depression outcome mea- Corresponding Author Patricia A Arean,´ PhD ([email protected]) surement relies on the use of patient-reported outcomes, such Keywords: decision support, digital health, mental health, primary care, real-world data, return of information The Permanente Journal·https://doi.org/10.7812/TPP/20.213 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 15 ORIGINAL RESEARCH ARTICLE Using Real-world Data for Decision Support: Recommendations from a Primary Care Provider Survey

challenge to the integration of these data into CDSS tools is Capture survey was distributed to Kaiser Permanente that clinician uptake is low, resulting from the perceived Northwest providers via letter and email by study staff at the burden and poor clinical utility of the information pro- Kaiser Permanente Center for Health Research. Upon vided.13-15 According to the Agency for Healthcare Re- clicking the link, participants were asked to review the search and Quality’s 2010 report on Clinical Decision study consent form and decide whether they would like Supports,16 poor uptake is a result of CDSS developers’ lack to continue to the full survey. e survey questions were of familiarity with clinician workflows and how clinicians designed to elicit the opinions of mental health care pro- make decisions, and limited clinician involvement during viders on using digital tools to help inform clinical decision CDSS design and development. For CDSS that integrate making (see eAppendix 3a in the Supplement for the full real-world data from ubiquitous technology to be usable and survey, including the consent form). It included questions effective for providers delivering depression treatment in that asked for both multiple-choice responses (quantitative) busy primary care clinics, these tools must be designed to as well as free-text responses (qualitative). e survey took account for clinician workflows and meaningfulness in approximately 10 minutes to complete. decision making. e purpose of this study was to determine what Survey Development and Content primary care providers feel would help them with clinical e survey items were codeveloped by the research team, decision making; whether real-world data from digital and initial tests with pilot survey respondents were con- tools are seen as helpful in decision making, and how the ducted to ascertain survey clarity, completeness, and burden. data should be shared and integrated into their practice; Modifications were made to the initial survey to account for and whether they have any concerns regarding collecting survey burden and content. e aim was to create a face- and accessing such information. Collection of such data is valid survey that could be completed within a 10- to meant to guide the development and deployment of 15-minute timeframe. CDSS in the future. Although the final survey was a mixed-methods survey that included some forced-choice answers to questions with METHODS an opportunity to provide further comments in an open-text Study Design field, the majority of the survey relied on the use of open- is is a mixed-methods quantitative and qualitative ended questions. ese decisions were made to elicit the cross-sectional survey of a convenience sample consisting of greatest range of unanticipated input without limiting or clinicians working in primary care medicine. Clinician par- influencing the participants with predefined answers, which ticipants were recruited from Kaiser Permanente Northwest, is particularly important when trying to understand atti- an integrated delivery system, to participate in an online tudes and perceptions. Participants were asked to complete survey regarding attitudes toward the use of digital infor- demographic questions that included gender, age, years in mation for clinical decision making. Recruitment took place practice, ethnicity, and highest degree obtained. e survey between April and July 2019. Every eligible provider for the probed 3 areas regarding mental health apps: general at- selected clinic locations was sent a preparticipation incentive titudes toward digital tools for clinical practice, current use (a quality chocolate bar) and a study recruitment letter with of such tools, and acceptability of a future tool to track instructions for how to complete the online study survey patient progress and inform clinical care. Each forced- (see eAppendix 1a in the Supplement for the recruitment choice question was followed by an open-ended query letter). A week later, we sent an initial follow-up email to about the reason for their response and any concerns they each of the providers who had not yet completed the survey had regarding the use of these tools for clinical purposes. (see eAppendix 2a in the Supplement for the follow-up email). In total, we sent 373 letters to providers, 358 initial Data Analysis follow-up emails, and 142 subsequent follow-up emails. SPSS data analysis software (IBM version 27 2020) was is study was reviewed and approved by the institutional used to tabulate demographic information as well as the review boards at the University of Washington and Kaiser answers to the multiple-choice questions on topics such as Permanente Northwest. behavior and attitudinal ratings. Most of the survey contained open-ended questions. To analyze the respondents’ quali- Survey Procedures tative responses, the answers were imported into Miro,17 a e survey was administered via Research Electronic Data visual collaborationplatform.Usinganaffinity diagram Capture, a secure, online database and survey platform hosted method,18 the data were organized into meaningful cate- at the University of Washington Institute of Translational gories based on problems, common themes, and patterns that Health Sciences. A link to the Research Electronic Data emerged or evolved naturally from the research questions.

16 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.213 ORIGINAL RESEARCH ARTICLE Using Real-world Data for Decision Support: Recommendations from a Primary Care Provider Survey

Table 1. Demographic characteristics of study participants Table 2. Main quantitative survey results Demographic n %a Survey questions n % Gender In general, would you describe your attitude toward using digital tools, such as Man 39 30.0 an app or a monitor, in your practice as Woman 91 70.0 Negative 3 2.6 Race/ethnicityb Slightly negative 5 4.3 Native American or Alaskan native 0 0.0 Neutral 20 17.4 Asian 29 22.1 Slightly positive 30 26.1 Black or African American 1 0.8 Positive 57 49.6 Hispanic/Latino 5 3.8 Yes (n) Yes (%) Native Hawaiian or Pacific Islander 0 0.0 Do you currently use any apps to help with your 48 42.5 clinical decision making? Other 1 0.8 Do you encourage your patients to use any particular 67 58.3 White 98 74.8 apps that are available now? b Degree(s) held Would you encourage patients to use an app that 96 84.2 MD 86 65.6 tracked their progress and helped you stay PhD 3 2.3 informed about their treatment? LCSW 11 8.4 Would you have any concerns about such an app? 76 67.3 PA 6 4.6 If an app existed that could collect information on a 47 42.7 moment-by-moment basis about your patients’ MA 5 3.8 state of mind, would you think this was useful? MS 1 0.8 If an app could tell you accurately that one of your 101 90.2 NP 7 5.3 patients will need a change in treatment, would you Other (includes RN, PsyD, LPC, LMFT, DO) 13 9.9 want that information? Mean (SD) Range n% Age (y) 46 (9.0) 27–65 If you could receive information from a patient-used app that could augment Years in practice 13.8 (8.6) 1–37 your clinical decision making, how/where would you want that information delivered? a.Percentages exclude missing responses. b.Racial categories and degrees held are not mutually exclusive. Clinical notes 80 61.1 N = 131. Epic staff message 41 31.3 DO = Doctor of Osteopathy; LCSW = licensed clinical social workers; LMFT = licensed marriage and family therapist; LPC = licensed professional counselor; MA = master of arts; Outlook email message 4 3.1 MD = doctor of medicine; MS = master of science; NP = nurse practitioner; PA = physician Other 9 6.9 assistant; PsyD = doctor of psychology; PhD = doctor of philosophy; RN = registered nurse; N = 131. SD = standard deviation.

RESULTS attitudes and 26.1% (n = 30) indicating slightly positive Participants attitudes. Forty-eight providers (42.5%) reported they Of the 131 clinicians who agreed to complete the survey, currently use apps to help with decision making; 67 re- 70% (n = 91) of the respondents were female and a majority spondents (58.3%) reported they encouraged patients to use (74.8%, n = 98) were white. A total of 65.6% (n = 86) were mental health apps to augment treatment. Respondents doctors (Doctor of Medicine) and 5.3% (n = 7) were nurse indicated that the apps they commonly used were tools for practitioners, with the remainder being health-care pro- clinical support and risk assessment. Although respondents viders from other disciplines (doctors of philosophy, li- found these existing tools to be helpful in enhancing and censed clinical social workers, physician assistants, medical expediting clinical decision making, they found these tools assistants, and psychologists with a master’s of science to be time-consuming to use and incompatible with existing degree). All providers worked in primary care medicine. e technology systems they currently use (eg, the integrated average age was 49 and the average years in practice was electronic health record system). Apps that respondents 13.8. See Table 1 for participant demographics. tended to recommend to patients were mood trackers, meditation apps, and Kaiser Permanente’s health tools. Re- Attitudes Toward and Use of Digital Tools for Clinical spondents indicated that these tools were useful for patient Decision Making support and education, but were concerned that the tech- A majority of respondents reported positive attitudes toward nology felt too robotic/impersonal and that the information the use of apps or sensors for gathering patient’s real-world on these patient-facing apps may not be accurate or evidence data in clinical practice, with 49.6% (n = 57) indicating positive based. See Table 2 for the main quantitative survey results.

The Permanente Journal·https://doi.org/10.7812/TPP/20.213 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 17 ORIGINAL RESEARCH ARTICLE Using Real-world Data for Decision Support: Recommendations from a Primary Care Provider Survey

Acceptability of an Outcome Tracking App for Clinician workload. We have so many streams of information and inbox Decision Making management has become overwhelming. ...What would Ninety-six respondents (84.2%) reported they would happen if there were concerns such as [suicidal ideation] and I encourage patients to use a future app that could monitor am not available to respond to them. What if nobody responds to treatment outcomes and could be shared with the provider them?” See eTable 1a in the Supplement for additional for clinical decision making. Qualitative data indicated that, participant quotes that illustrate these areas of concern. in addition to the information they already collect at each visit, such as symptom rating scales and identification of Recommendations stressors, useful information from digital tools would be Respondents offered recommendations in 2 major areas: any data related to wellness, such as physical and social the evidence providers need to have before using a clinical activity, sleep, symptom tracking, nutrition, heart rate, decision support tool and how data collected from a clinical blood pressure, and any recent external life stressors. As one decision support tool should be integrated into their provider said, “[I would like to have information about] self- practice. e 3 most prominent themes that respondents care behaviors as they truly are (social activity, exercise, sleep felt would need to be addressed before the future use of real- habits, alcohol intake, diet), but in a quick, usable format.” world data in a clinical decision support tool were 1) the Respondents were less likely to use a digital tool that accuracy of the data in detecting the need for treatment tracked moment-to-moment information about patient adjustments, 2) the requirement for the tool be clinically state of mind (42.7%, n = 47), but were more likely to use a tested and vetted, and 3) the need for the tool to be evidence tool that could accurately inform them if any of their pa- based (see eTable 2a in the Supplement for participant tients needed a change in treatment (90.2%, n = 101). quotes that illustrate these recommendations). For the Eighty respondents (61.1%) reported they would prefer to information to be readily useful to providers, respondents have this information integrated with their electronic indicated that the information in clinical notes or messaging clinical notes; 41 respondents (31.3%) preferred the in- services and alerts should be based on an assessment of formation be provided in their electronic health record patients’ needs and current status. messaging platform. One provider explained, “. . . as long as the patient is comfortable, and it doesn’t add too much to my DISCUSSION workload. In Primary Care we have so many streams of in- Real-time acquisition of information from smart devices formation and the inbox management with electronic messages has the potential to transform how quickly clinicians can and results has become overwhelming. More data isn’t neces- intervene in treatment and may enhance clinical outcomes. sarily better.” See Table 2 for the main quantitative survey According to the results of our survey, many providers results. currently use clinical decision support apps to inform practice, and encourage their patients to use self-guided Concerns mental health tools to support and augment their care. We Qualitative analysis surfaced 3 areas of major concern also found there is considerable interest and perceived utility about information from patient-facing apps and wearable of real-world data for clinical decision making, but before sensors. First, respondents reported their patients may be these tools can be adopted into practice, a number of overwhelmed by too much information and may have concerns should be addressed, and recommendations of trouble interpreting the data from digital health tools. providers put into action. Examples of such comments include the following: First and foremost, the accuracy of the high-frequency “Sometimes too much information is more anxiety provoking” real-world acquired data to inform clinical decisions would and “Information must be clear & easily presented so patient need to be demonstrated before providers would use rec- doesn’t need my interpretation to benefit.” Second, data ommendations from a digital health app or CDSS using concerns were also prominently reported, particularly with such data. is is a concern that has been voiced in other regard to data security, privacy, and accuracy. One comment studies of patient opinion regarding such tools19,20 and by included, “Accuracy of data, plus amount of data would have to critics in the digital mental health field.21,22 Indeed, many be summarized well. I don’t want to be scrolling thru days of existing decision support tools that rely on data from sensors info.” Finally, respondents were concerned about the impact and apps have not been properly vetted in their accuracy.23 such information would have on their workflow and time, Only support for research to develop and test the accuracy of and potential responsibility and liability if providers were these tools will mitigate provider and patient concerns around not able to respond right away to a high-risk situation (eg, accuracy, and, importantly, this research must follow open suicidal ideation). As one provider said, “As long as the science policies to ensure customer trust in the results from such patient is comfortable and it doesn’t add too much to my research,23 rather than being held as proprietary information.

18 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.213 ORIGINAL RESEARCH ARTICLE Using Real-world Data for Decision Support: Recommendations from a Primary Care Provider Survey

A second and related concern was the potential liability group health model and, as such, the opinions are limited to with which providers could be faced if they were not able to people in these practices. In addition, these providers are provide timely care. Providers were less likely to be inter- potentially more interested in the use of technology than ested in moment-to-moment changes in mood and activity, typical primary care providers. Kaiser Permanente has but instead preferred to have an overall general well-being adopted a number of digital tools for health care, and thus score beyond what self-report measures would provide. these respondents may have a more positive attitude toward ey were particularly concerned with having access to such tools and devices. Second, although we were able to alerts about adverse events, such as suicide risk. Providers acquire additional qualitative opinions, we were not able to indicated that the current system of care in primary care have respondents interact with any tools to provide a more medicine was not nimble enough to respond to emergencies nuanced opinion into such tools. Future research will need such as suicide. is particular feature, the potential for to employ user-centered design methods to develop any end these tools to result in proximal risk detection of adverse product integrating real-world data in CDSS. events, presents an important challenge in creating alert tools. e first, as was indicated in this survey, is the ability CONCLUSION for health-care systems, like primary care, to respond ap- e major takeaway from this provider survey of the use propriately to these emergencies. For these tools to be of data from personal digital technology to inform clinical useful, systems will need to provide resources to providers to decision making is that primary care providers who manage respond in a timely manner. e second problem is one of mental health problems are open to the use of such data, accuracy, reflected to some degree in the first concern raised. provided certain caveats are addressed. Primarily, ease of If a tool is too sensitive to emergencies, in that there are too use, data security, data accuracy, and synthesized data are all many false-positive alerts, providers and health-care systems important characteristics that need to be considered when will be expending limited resources unnecessarily. If the tool developing these tools. We recommend more research be is insensitive, with too many false negatives, providers and conducted to validate the signal from these tools, and to in- health-care systems will be less likely to use this feature clude providers and patients in the design of the information because of the dangers associated with false-negative re- to be collected, and how it is compiled and used in care. v ports. Given the inability of any past effort24 to identify the proximal risk of suicide, this may be a feature that should be Supplemental Material a fi omitted from the design of any CDSS or mental health app Supplemental Material is available at: www.thepermanentejournal.org/ les/ 2021/20.213supp.pdf until better detection and a follow-up system is available. In sum, providers were more interested in information that Disclosure Statement could offer an overall response to treatment, rather than The authors have no conflicts of interest to disclose. detailed functioning or emergency alerts. ird, and as important a concern as the first, is the issue Authors’ Contributions regarding the secure transfer of data from apps to CDSS in Emily Friedman, MID, CPE; Abhishek Pratap, PhD; and Patricia Arean,´ PhD; developed the survey and survey methods for this study. This team also analyzed health records. Other studies have found that patients are data. Gregory Clarke, PhD; Sara Gille, MPH; Shelley Reetz, BS; and Erin Keast, very concerned about who would have access to such data MPH; were responsible for participant recruitment. All parties contributed to writing and if very personal information could be shared acciden- the manuscript. tally with organizations that would use the data for other purposes.25 Data security and sharing problems have been Funding 26,27 This project was supported by award nos. UL1 TR002319, KL2 TR002317, and elucidated in other studies, indicating these issues are TL1 TR002318 from the National Center for Advancing Translational Sciences/ valid. More methods to protect sensitive data are needed National Institutes of Health. Funding was also provided by a grant from the Kaiser before tools of this nature are implemented. Permanente Center for Health Research to Dr Clarke. Finally, providers do not want any tool to place addi- tional burdens on the patient in terms of information References 1. World Health Organization Regional Office for Europe. Integrated care models: An overload or interference with daily activities, nor do pro- overview working document. Copenhagen, Denmark. 2016. August 2020. www.euro.who.int/ viders want a tool that interferes with their workflow. ese pubrequest 2. Centers for Medicare & Medicaid Services. Behavioral Health Integration Services booklet concerns have also been identified as potential reasons for 2018. August 2020. www.cms.gov/Outreach-and-Education/Medicare-Learning-Network- why mental health apps have such poor uptake by patients, MLN/MLNProducts/Downloads/BehavioralHealthIntegration.pdf 3. Siu AL, Bibbins-Domingo K, Grossman DC, et al. Screening for depression in adults. because of their poor design and limited accounting for J Am Med Assoc 2016 Jan;315(4):380–7. DOI: https://doi.org/10.1001/jama.2015. patient burden.28 18392.  4. Schaeffer AM, Jolles D. Not missing the opportunity: Improving depression screening and is study has some limitations that require mention. follow-up in a multicultural community. Joint Comm J Qual Patient Saf 2019 Jan;45(1): First, this is a convenience sample survey of providers in a 31–9. DOI: https://doi.org/10.1016/j.jcjq.2018.06.002, PMID:30139563.

The Permanente Journal·https://doi.org/10.7812/TPP/20.213 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 19 ORIGINAL RESEARCH ARTICLE Using Real-world Data for Decision Support: Recommendations from a Primary Care Provider Survey

5. Malpass A, Shaw A, Kessler D, Sharp D. Concordance between PHQ-9 scores and 17. Miro Software v1.0 (1995). Miro. San Francisco, CA; 2019. https://miro.com/ patients’ experiences of depression: A mixed methods study. Br J Gen Pract 2010 Jun; 18. Harboe G, Huang EM. Real-world affinity diagramming practices: Bridging the paper- 60(575):e231. DOI: https://doi.org/10.3399/bjgp10X502119, PMID:20529486. digital gap. In: Conference on human factors in computing systems: Proceedings. 6. Mancini AD, Moser LL, Whitley R, et al. Assertive community treatment: Facilitators Association for Computing Machinery: 2015; p 95–104. and barriers to implementation in routine mental health settings. Psychiatr Serv 19. Pifer R. Patient use of digital health tools lags behind hype, poll finds. Healthcare Dive. 2009 Feb;60(2):189–95. DOI: https://doi.org/10.1176/ps.2009.60.2.189, 2019. August 2020. www.healthcaredive.com/news/patient-use-of-digital-health-tools- PMID:19176412. lags-behind-hype-poll-finds/562778/ 7. Ranmuthugala G, Plumb JJ, Cunningham FC, Georgiou A, Westbrook JI, Braithwaite J. 20. Renn BN, Hoeft TJ, Lee HS, Bauer AM, Arean´ PA. Preference for in-person How and why are communities of practice established in the healthcare sector? A psychotherapy versus digital psychotherapy options for depression: Survey of systematic review of the literature. BMC Health Serv Res 2011 Oct;11, 273. DOI: https:// adults in the U.S. NPJ Digit Med 2019 Dec;2(1). DOI: https://doi.org/10.1038/s41746- doi.org/10.1186/1472-6963-11-273, PMID:21999305. 019-0077-1. 8. Ranmuthugala G, Cunningham FC, Plumb JJ, et al. A realist evaluation of the role of 21. Hatch A, Hoffman JE, Ross R, Docherty JP. Expert consensus survey on digital health communities of practice in changing healthcare practice. Implement Sci 2011 May;6(1): tools for patients with serious mental illness: Optimizing for user characteristics and user 49. DOI: https://doi.org/10.1186/1748-5908-6-49, PMID:21600057. support. JMIR Ment Health 2018 Jun;5(2):e46. DOI: https://doi.org/10.2196/mental.9777. 9. Brown GS, Simon A, Cameron J, Minami T. A Collaborative Outcome Resource Network 22. Minen MT, Stieglitz EJ, Sciortino R, Torous J. Privacy issues in smartphone applications: (ACORN): Tools for increasing the value of psychotherapy. Psychotherapy 2015 Dec; An analysis of headache/migraine applications. Headache 2018 Jul;58(7):1014–27. DOI: 52(4):412–21. DOI: https://doi.org/10.1037/pst0000033, PMID:26641371. https://doi.org/10.1111/head.13341, PMID:29974470. 10. Lutz W, de Jong K, Rubel J. Patient-focused and feedback research in psychotherapy: 23. Torous J, Andersson G, Bertagnoli A, et al. Towards a consensus around standards for Where are we and where do we want to go? Psychother Res 2015 Sep;25(6):625–32. smartphone apps and digital mental health. World Psychiatry 2019 Feb;18:97–8. DOI: DOI: https://doi.org/10.1080/10503307.2015.1079661, PMID:26376225. https://doi.org/10.1002/wps.20592. 11. Ng A, Kornfield R, Schueller SM, Zalta AK, Brennan M, Reddy M. Provider perspectives 24. Franklin JC, Ribeiro JD, Fox KR, et al. Risk factors for suicidal thoughts and behaviors: A on integrating sensor-captured patient-generated data in mental health care. In: meta-analysis of 50 years of research. Psychol Bull 2017 Feb;143(2):187–232. DOI: Proceedings of the ACM on human–computer interaction. Association for Computing https://doi.org/10.1037/bul0000084, PMID:27841450. Machinery: 2019 Nov; Galsgow Scotland. Article number 115. p 25. 25. Pratap A, Allred R, Duffy J, et al. Contemporary views of research participant willingness 12. Pusic M, Ansermino JM. Clinical decision support systems. British Columbia Med J 2004; to participate and share digital data in biomedical research. JAMA Netw Open 2019 Nov; 46(5):236–9. www.bcmj.org/articles/clinical-decision-support-systems 2(11):e1915717. DOI: https://doi.org/10.1001/jamanetworkopen.2019.15717,PMID: 13. Lurio J, Morrison FP, Pichardo M, et al. Using electronic health record alerts to provide 31747031. public health situational awareness to clinicians. J Am Med Inf Assoc 2010 Mar–Apr; 26. Huckvale K, Torous J, Larsen ME. Assessment of the data sharing and privacy practices 17(2):217–9. DOI: https://doi.org/10.1136/jamia.2009.000539, PMID:20190067. of smartphone apps for depression and smoking cessation. JAMA Netw Open 2019 Apr 5; 14. Singh H, Spitzmueller C, Petersen NJ, et al. Primary care practitioners’ views on test 2(4):e192542. DOI: https://doi.org/10.1001/jamanetworkopen.2019.2542, PMID: result management in EHR-enabled health systems: A national survey. J Am Med Inf 31002321. Assoc 2013 Jul;20(4):727–35. DOI: https://doi.org/10.1136/amiajnl-2012-001267. 27. Rosenfeld L, Torous J, Vahia IV. Data security and privacy in apps for dementia: An 15. Singh H, Spitzmueller C, Petersen NJ, Sawhney MK, Sittig DF. Information overload and analysis of existing privacy policies. Am J Geriatr Psychiatry 2017 Aug;25(8):873–7. DOI: missed test results in electronic health record-based settings. JAMA Intern Med 2013; https://doi.org/10.1016/j.jagp.2017.04.009, PMID:28645535. 173(8):702–4. DOI: https://doi.org/10.1001/2013.jamainternmed.61. 28. Ledel Solem IK, Varsi C, Eide H, et al. A user-centered approach to an evidence-based 16. Eichner J, Das M. Challenges and barriers to clinical decision support (CDS) design and electronic health pain management intervention for people with chronic pain: Design and implementation experienced in the Agency for Healthcare Research and Quality CDS development of EPIO. J Med Internet Res. 2020 Jan;22(1):e15889. DOI: https://doi.org/ demonstrations. 2010. August 2020 www.ahrq.gov 10.2196/15889, PMID:31961331.

20 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.213 n ORIGINAL RESEARCH ARTICLE Surgeon–Therapist Communication Must Be Improved in Rotator Cuff Repair Rehabilitation: An Electronic Survey of Physical Therapists on Postoperative Rehabilitation Protocols and Communication with Treating Surgeons

Mark Schultzel, MD1; Karl B Scheidt, MD2; Brian McNeill, DPT3; Christopher M Klein, MS4; Colin Blout, BS4; John M Itamura, MD4 Perm J 2021;25:20.088 E-pub: 03/01/2021 https://doi.org/10.7812/TPP/20.088

ABSTRACT from 4% to nearly 100%, the outcomes continues to be Background: There is no consensus on postoperative rotator variable and inconsistent.1-6 Studies have shown that rotator cuff repair protocols in orthopedic or physical therapy literature. cuff retears occur primarily during weeks 6 to 26 after Despite surgical management, the frequency of rotator cuff shoulder surgery.3,5 With the majority of postoperative retears continues to be high. patients undergoing some form of formal rehabilitation, the Objectives: This study is designed to investigate the current timelines for these retears occur while under the care of both concepts of postoperative rehabilitation and to evaluate the state physical therapist and surgeon. Currently, the most com- of communication between referring surgeons and treating physical therapists. mon tool in managing patients is a postoperative protocol; Methods: A survey was conducted over a 2-year period, however, rehabilitation protocols tend to vary considerably performed by an online survey company. among health-care providers and are frequently based on Results: Six hundred responses were obtained from physical clinical experience and expert opinion rather than a stan- therapists. Most rehab protocols were based on size of tear, tissue dardized and universally accepted protocol. e goal of this quality, and open versus arthroscopic repair. Current intervention study was to investigate the current concepts regarding concepts and professional experience guided protocol devel- postoperative rotator cuff protocols and to assess the state of opment. Thirty-three percent of therapists receive operative communication between the referring surgeon and the ≤ notes 25% of the time. Sixteen percent reported not receiving physical therapist. operative notes and not having access to the physician >50% of the time. Most patients were seen within 2 weeks, with passive MATERIALS AND METHODS range of motion started in 83% of cases. Sixty percent started ≤ Online surveys were conducted over a 2-year period of active-assist range of motion at 4 weeks. Sixty-four percent of ff therapy was continued for 12 to 16 weeks. Patient compliance, physical therapists who commonly treat rotator cu in-  poor tissue quality, and rapid rehab progression were reported as juries. ese surveys were distributed via email lists obtained common causes of failure. through various physical therapy networks across the United Conclusion: Most rehabilitation programs follow protocols States. Individual physical therapist participation was vol- developed by surgeons and physical therapists. Tissue quality, untary. e questionnaires consisted of demographic ques- size of tear, and repair type are usually documented in the tions regarding the education level of the physical operative report, and are rarely conveyed to the therapist. therapist, and which orthopedic and physical therapy This study highlights the lack of communication between the journals they read, as well as knowledge-based questions physician and the therapist. Improving communication re- about rotator cuff tears and protocols they use. Participants garding the findings at surgery, opening lines of communi- were also asked about how they communicated with the cation, and making alterations to the protocol may improve patient outcomes. treating surgeon and about their thoughts on causes of retear. e surveys were performed by a commercially INTRODUCTION ff Rotator cu pathology is one of the most common Author Affiliations musculoskeletal dysfunctions seen today. Rotator cuff 1Synergy Orthopedic Specialists Medical Group, Southern California Permanente Medical Group, San Diego, CA dysfunction associated with pain and weakness directly 2Clement J. Zablocki VA Medical Center, Milwaukee, WI ff 3Department of Physical Therapy, University of Southern California, Los Angeles, CA diminishes a patient functional ability and can a ect a 4 ’ Kerlan-Jobe Orthopaedic Clinic, White Memorial Medical Center, Cedars-Sinai Medical Center, Keck School of patient s quality of life substantially. Whether the treat- Medicine, Los Angeles, CA ment intervention is surgical or conservative, many surgeons may have differing opinions on the type of rehabilitation Corresponding Author Mark Schultzel, MD ([email protected]) protocol. With the rate of postoperative retears ranging Keywords: orthopedic, physical therapy, rehabilitation, rotator cuff, shoulder, surgery The Permanente Journal·https://doi.org/10.7812/TPP/20.088 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 21 ORIGINAL RESEARCH ARTICLE Surgeon–Therapist Communication Must Be Improved in Rotator Cuff Repair Rehabilitation

Figure 1. Physical therapist beliefs on cuff repair technique.

Figure 2. Determining therapy progression protocols. available online survey company (Survey Monkey), which provided data acquisition and analysis. Therapist Opinions Regarding Rotator Cuff Repairs RESULTS erapists were questioned about the types of rotator cuff Demographics repairs and what they felt were the important factors in the A total of 600 responses were obtained. e study was success of the repairs. Forty-two percent (245 of 588) did distributed nationally: Northeast, 15.2% (91 of 598); from not know whether a single-row repair or a double-row repair the Mid Atlantic, 8.4% (50 of 598); Southeast, 14.7% (88 was stronger, 41% (242 of 588) believed the double-row of 598); Midwest, 22.1% (132 of 598); Southwest, 9.9% (59 repair was stronger, 14.8% (14 of 588) believed there was no of 598); West, 22.9% (137 of 598); and Northwest, 6.7% difference, and 2.4% (87 of 588) believed the single-row (41 of 598). Of the respondents, 55.6% (332 of 597) did not repair was stronger (Figure 1). Eighty percent (461 of 572) claim specialty certification, 27.3% (163 of 597) were or- did not alter their rehab program based on the type of repair, thopedic clinical specialist certified, and 19.9% (119 of 597) but 77% (445 of 578) did alter their protocol based on open were sports clinical specialist certified. Sixty-four percent versus mini-open versus arthroscopic repair techniques. (385 of 600) of the respondents were doctors of physical Ninety-three percent (434 of 533) reported they would alter therapy, 17.7% (106 of 600) had a master’sdegreeinphysical the protocol based on tear size, and 95% (503 of 530) would therapy), 19% (114 of 600) had a bachelor’s degree, 10.5% (63 alter based on tissue quality. e therapists ranked the of 600) were in accredited clinical residency programs, 3.8% following factors used in determining the rate of rehab (23 of 600) were in accredited clinical fellowship programs, progression from most to least important (from 1–7, with 7 and 7.3% (44 of 600) were described as “other.” Forty-six being least important): For 549 respondents, the size of the percent (273 of 593) of the respondents worked in a private tear ranked 5.85 on average; tissue quality, 5.62; age, 3.85; practice outpatient setting; 30% (175 of 593), in an outpatient location of tear, 3.76; concomitant surgery, 3.71; patient hospital-based practice; 13% (78 of 593), in outpatient cor- activity level, 3.08; and type of repair, 2.4 (Figure 2). porate practices; 6% (39 of 593), in physician-owned prac- tices; and 4% (26 of 593), in educational/research institutions. Rehabilitation Program Development e focus for scholarly research for nearly all therapists Eighty-seven percent (465 of 533) of therapists use re- was in reviewing physical therapy journals such as the habilitation guidelines/protocols, with 63% (299 of 477) of Journal of Orthopaedic and Sports Physical erapy (95.5%, these developed between the surgeon and therapist, 16% (75 571 of 598) and the Physical erapy Journal (56%, 355 of of 477) completely physician directed, and 13% (62 of 477) 598). ere was less focus on orthopedic surgery journals; developed by consensus guidelines. Seventy-three percent only 37% (220 of 598) reviewed the American Journal of (340 of 467) of the protocols were evidence/literature based, Sports Medicine; 15% (87 of 598), the Journal of Bone and but 27% (127 of 467) were not. Of the literature-based Joint Surgery; and 11% (65 of 598), the Journal of Shoulder responses, 46% (116 of 254) were based on citations from and Elbow Surgery. the Journal of Orthopaedic and Sports Physical erapy; 18%

22 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.088 ORIGINAL RESEARCH ARTICLE Surgeon–Therapist Communication Must Be Improved in Rotator Cuff Repair Rehabilitation

Table 1. Rehabilitation program development Therapist Best Practices Yes No Rehab guidelines/protocol used 87.2% 12.8% Based on current evidence/literature 72.8% 27.2% Protocol developed Physical therapist + physician 62.7% Physician only 15.7% Physical therapist only 1.0% Consensus guidelines 13.0% Other 7.6% Treatment interventions influenced by Current intervention concepts 94.2% Physical therapist’s experience 94.3% Animal studies on tissue healing 34.0%

(46 of 254), from the American Journal of Sports Medicine; 11% (28 of 254), from Journal of Bone and Joint Surgery; and 9% (23 of 254), from Journal of Shoulder and Elbow Surgery. erapeutic interventions were influenced pri- marily by current intervention concepts and professional Figure 3. Treatment protocol determination method. experience as opposed to animal model studies of rotator cuff healing (Table 1).

Physician Communication e study also examined the level of communication that exists between therapists and physicians, and how treatment progression decisions are made. One-third (171 of 520) of respondent therapists receive operative notes 25% of the time or less (Figure 3). Sixteen percent (83 of 506) of therapists reported not receiving operative notes or having access to the physician more than 50% of the time. More than 75% (372 of 469) of the therapists reported contacting the treating surgeon by phone or email with treatment progress and planning, and 12% (54 of 469) ask the patient about treatment progression and planning (Figure 3 and Figure 4). Current literature (69.7%, 327 of 469) is used more commonly than physician communications (physician phone call, 54.16%, 254 of 469; physician email, 25.16%, 118 of 469) to make determinations to progress the re- habilitation. Physician communication did not differ overall by type of practice when comparing private practice settings to hospital-owned or corporate practices. Figure 4. Rates of receipt of operative note from surgeon. Therapy Timing As for when therapists start their rehabilitation protocols, therapists continue therapy for 12 to 16 weeks. Submaximal 74% (402 of 539) of therapists see patients within 2 weeks isometric exercises were started at 2 weeks in 13% (70 of postoperatively, with passive range of motion started in 83% 533) of the respondents, at 4 weeks in 28% (151 of 533), (445 of 536) of cases. Sixty percent (322 of 533) of ther- and at 6 weeks in 28% (149 of 533). Isotonic strengthening apists started patients on active-assist range of motion at was started at 8 weeks or earlier 55% (290 of 530) of the 4 weeks or less, and 56% (302 of 536) started active range of time. Scapular strengthening was started usually at or before motion at 6 weeks or less. Sixty-four percent (322 of 500) of 6 weeks (432 of 534) (Table 2).

The Permanente Journal·https://doi.org/10.7812/TPP/20.088 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 23 ORIGINAL RESEARCH ARTICLE Surgeon–Therapist Communication Must Be Improved in Rotator Cuff Repair Rehabilitation

Table 2. Reported timeline in initial evaluation and rehabilitation protocols Clinical Metrics POD1 1 wk 2 wk 3 wk 4 wk Initial evaluation 8.4% 39.9% 26.4% 8.7% 16.7% < 2 wk 2 wk 4 wk 6 wk 8 wk 10 wk 12 wk > 12 wk 16 wk 20 wk 24 wk Other PROM 56.2% 26.9% 13.1% 3.4% 0.4% 0% 0.2% 0 AAROM 2.8% 9.1% 48.6% 34.0% 4.3% 0.8% 0.6% (> 10 wk) AROM 0.6% 9.0% 46.8% 30.6% 7.5% 5.6% (> 10 wk) Scapular strengthening 22.5% 17.4% 21.5% 19.5% 10.5% 2.4% 2.6% 0.6% 3.0% Submaximal isometrics 13.1% 28.3% 28.0% 13.5% 5.1% 4.5% 0.3% 7.1% Isotonic strengthening 3.2% 18.5% 33.0% 14.3% 25.3% 5.7% How long treated 0.0% 0.0% 0.0% 0.0% 5.4% 28.0% 36.4% 13.6% 10.4% 6.2% AAROM = active-assist range of motion; AROM = active range of motion; POD1 = postoperative day 1; PROM = passive range of motion.

difference in end range of motion and retear rates.10 Human studies revealed no significant difference in some studies,9 whereas others showed increased forward flexion in the early-motion groups, with an increased risk of retear for cuff tears > 3 cm, regardless of repair technique.12 Recent meta- analyses of multiple level I and level II studies revealed overall no difference in functional outcomes between groups or difference in retear rates.13-15 It is important to note that most of these studies did not control for medical comor- bidities or intraoperative tissue quality that might have affected postoperative rehabilitation. Although most meta-analyses concluded that early range of motion was of little detriment to patient outcome, in practice there is considerable debate among surgeons re- garding the length of immobilization and speed of reha- fl ff Figure 5. Physical therapist opinions on reasons for retear and failure. bilitation. Considering the con icting data and di ering opinions, it is unclear how physical therapists plan or adjust their rehabilitation protocols. e common postoperative Opinions on Causes of Failures protocol is for a physician to prescribe a rehabilitation Physical therapists report that patient compliance (44%, program for the patient, and for that program to be executed 218 of 498), poor tissue quality (39%, 194 of 498), and rapid by the therapist. With retears occurring primarily during the rehabilitation progression (6%, 28 of 498) are felt to be the 6- to 26-week postoperative period, improvements in major causes of failure (Figure 5). Residual pain (64%, 320 postoperative rehabilitation must be made to increase the of 502) and stiffness (57%, 286 of 502) are the most success rate of rotator cuff surgery.3,5 common complaints therapists have after rotator cuff repair. Communication between surgeon and therapist seems to be a “weak link” in the process. Although rehabilitation DISCUSSION protocols streamline the referral process to physical therapy, Rotator cuff repair surgery has the potential to improve the protocols may be generic and may not consider the patient function and decrease pain. Despite this, the risk of variables that may affect greatly the success of the surgery. retear is a significant concern among orthopedic surgeons, is study demonstrated that therapists often do not have as demonstrated in the literature.1-10 ere are numerous access to patients’ operative notes and are not able to potential reasons why surgery may fail, as poor tissue communicate with the surgeon, as one-third of respondent quality, tear size, and advanced patient age are variables not therapists received operative notes 25% or less, and 16% of well controlled by the surgeon. therapists reported not receiving operative notes and not Discordance exists in the literature regarding whether having access to the physician more than 50% of the time. early versus delayed range of motion has any benefitin ese results reflect a significant number of cases when functional range of motion and whether retear rates are the specifics of the patient are not communicated to the affected.11 In animal models, recent studies have shown no therapists. In these circumstances, there may be cases

24 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.088 ORIGINAL RESEARCH ARTICLE Surgeon–Therapist Communication Must Be Improved in Rotator Cuff Repair Rehabilitation

were repairs may be tenuous or less likely to heal, such increase the risk of inappropriate therapy protocols, especially that a “standard” therapy protocol may not be appro- in the case when therapists may not have orthopedic clinical priate. Most therapists alter their rehab protocol based on specialist or sports clinical specialist certification, increasing tear size, tissue quality, and type of repair (based on their the chance they are not familiar with commonly used re- responses), so having access to the surgeon’soperative habilitation protocols or risk factors for cuff retear. notes and recommendations on therapy may improve Physicians and physical therapists both strive to obtain patient outcome. the best results for their patients. Unfortunately, highly An effective method for improving the communication technical surgical repairs with excellent prognoses are at risk process can be found in the physical therapy prescription. for retear as a result of patient noncompliance, prematurely ese prescriptions may vary from being patient-specific or overly aggressive rehabilitation programs, and lack of instructions to generic prescriptions that read “evaluate and communication between surgeon and therapist. is study treat” or “rotator cuff repair protocol,” without guidance raises the question of whether improved communication with regard to range of motion and weight-bearing limi- and individualized patient-specific rehabilitation is possible, tation, or suggestions for what modalities might benefit and, if so, whether it would improve patient outcomes in the patient’s rehabilitation. Communication of the findings rotator cuff repair. In the study by Miller et al,5 7 of 9 retears at surgery (tissue quality, size of the tear, type of repair, occurred within 3 months of surgery, and the other 2 within and concurrent procedures) and specificchangestobe 6 months of surgery. Similarly, in the multicenter study of made in the standard protocol could be included in Iannotti et al,3 there was a 17% retear rate at a mean time of the rehab prescription. More specific guidelines may be 19.2 weeks. In our study, 16% of therapists reported they helpful in optimizing a patient’s rehabilitation potential did not receive communication from a surgeon or surgical and may decrease the rates of retear via overly aggressive operative notes. erefore, in at least 16% of cases, there therapy exercises. then exists the possibility of premature or inappropriate Another method to improve communication and patient rehabilitation, which put the surgical repair at risk and outcomes is for the operating surgeon to provide more hamper good patient outcomes. descriptive information in their operative notes regarding A strength of this study is that it includes 600 respon- the quality of the rotator cuff, the size of the tear, and the dents from across the country, in varied clinical settings and type of repair performed. e operative notes could then be at different levels of training, which likely reflects the broad used as a way of transmitting these operative findings to range of rehabilitation facilities available for patients. A both patient and therapist. Forty-seven percent of therapists limitation of this study is that it reports the knowledge, in this study stated they receive operative notes 50% or less opinions, and attitudes of professionals on a condition for of the time. Without operative notes, there may be a dis- which there is no gold standard or consensus rehabilitation cordance between physician goals, therapist goals, and the protocol. Future studies on the root causes of lack of timeline of therapy. communication may shed light on ways to improve col- e commonly accepted variables affecting rotator cuff laboration between surgeons and therapists. Given the repair discussed earlier can alter substantially the speed at findings of this study, we recommend increased commu- which therapy is progressed. is study discovered that 87% nication between physical therapists and orthopedic sur- of the therapists used rehabilitation guidelines or protocols, geons when treating rotator cuff repair, with the hope of with 74% of patients being seen within 2 weeks of surgery, decreasing the risk of iatrogenic rotator cuff retear and and 83% of patients started passive range of motion within improving overall patient outcome. v that 2-week time frame. is may not be appropriate for all patients, especially those with larger tears or poor tissue Disclosure Statement Mark Schultzel, MD; Karl B Scheidt, MD; Brian McNeill, DPT; Christopher M quality. Because there is no consensus in the literature about Klein, MS; and Colin Blout, BS; declare that no conflict of interest exists. These rehabilitation protocols and because each patient presents a authors, their immediate family, and any research foundation with whom they are unique circumstance with regard to the specifics of his or affiliated did not receive any financial payments or other benefits from any her cuff tear and repair, there is an increased need for commercial entity related to the subject of this article. John M Itamura, MD, declares he has a history of receiving paid consulting fees from Acumed and Wright communication between surgeon and therapist. Most Medical. Dr Itamura, his immediate family, and any research foundation with whom therapists altered the treatment plan based on the patient’s he is affiliated with did not receive any financial payments or other benefits from any tear size and tissue quality, and whether the repair was open, commercial entity related to the subject of this article. mini-open, or arthroscopic. Failure to receive operative notes Authors’ Contributions and the frequent inaccessibility of the surgeons thus pre- Mark Schultzel, MD, and John M Itamura, MD, participated in the study design, vented therapists from making informed decisions regarding acquisition and analysis of data, critical review, drafting and submission of the final the rehabilitation plan. Lack of communication may also manuscript. Karl B Scheidt, MD, and Brian McNeill, DPT, participated in the study

The Permanente Journal·https://doi.org/10.7812/TPP/20.088 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 25 ORIGINAL RESEARCH ARTICLE Surgeon–Therapist Communication Must Be Improved in Rotator Cuff Repair Rehabilitation

design, acquisition and analysis of data, critical review, and drafting of the final 6. Millett PJ, Horan MP, Maland KE, Hawkins RJ. Long-term survivorship and outcomes after manuscript. Christopher M Klein, MS, and Colin Blout, BS, participated in the study surgical repair of full-thickness rotator cuff tears. J Shoulder Elbow Surg 2011 Jun;20(4): – design, acquisition and analysis of data, and drafting of the final manuscript. All 591 7. DOI: https://doi.org/10.1016/j.jse.2010.11.019 fl authors have given final approval to the manuscript. 7. Ahmad S, Haber M, Bokor DJ. The in uence of intraoperative factors and postoperative rehabilitation compliance on the integrity of the rotator cuff after arthroscopic repair. J Shoulder Elbow Surg 2015 Feb;24(2):229–35. DOI: https://doi.org/10.1016/j.jse.2014. STROBE Declaration 06.050 The STROBE Statement’s guidelines have been adopted for this manuscript. 8. Mall NA, Tanaka MJ, Choi LS, Paletta GA. Factors affecting rotator cuff healing. J Bone Joint Surg Am 2014 May;96(9):778–88. DOI: https://doi.org/10.2106/ JBJS.M.00583 Funding 9. Shen C, Tang ZH, Hu JZ, Zou GY, Xiao RC, Yan DX. Does immobilization after The authors received research funding for this project from the Kerlan Jobe arthroscopic rotator cuff repair increase tendon healing? A systematic review and meta- – Institute. The authors have no proprietary interests in the materials described in the analysis. Arch Orthop Trauma Surg 2014 Sep;134(9):1279 85. DOI: https://doi.org/10. 1007/s00402-014-2028-2, PMID:25027677. article. No external sources of funding were used. 10. Zhang S, Li H, Tao H, et al. Delayed early passive motion is harmless to shoulder rotator cuff healing in a rabbit model. Am J Sports Med 2013 Aug;41(8):1885–92. DOI: https://doi. References org/10.1177/0363546513493251, PMID:23845402. 1. Bishop J, Klepps S, Lo IK, Bird J, Gladstone JN, Flatow EL. Cuff integrity after 11. Thomson S, Jukes C, Lewis J. Rehabilitation following surgical repair of the rotator cuff: A arthroscopic versus open rotator cuff repair: A prospective study. J Shoulder Elbow Surg systematic review. Physiotherapy 2016 Mar;102(1):20–8. DOI: https://doi.org/10.1016/j. 2006 May–Jun;15(3):290–9. DOI: https://doi.org/10.1016/j.jse.2005.09.017 physio.2015.08.003, PMID:26510584. 2. Harryman DT, Mack LA, Wang KY, Jackins SE, Richardson ML, Matsen FA. Repairs of the 12. Kluczynski MA, Isenburg MM, Marzo JM, Bisson LJ. Does early versus delayed active rotator cuff: Correlation of functional results with integrity of the cuff. J Bone Joint Surg Am range of motion affect rotator cuff healing after surgical repair? A systematic review and 1991 Aug;73(7):982–9. DOI: https://doi.org/10.2106/00004623-199173070-00004, PMID: meta-analysis. Am J Sports Med 2016 Mar;44(3):785–91. DOI: https://doi.org/10.1177/ 1874784. 0363546515582032, PMID:25943112. 3. Iannotti JP, Deutsch A, Green A, et al. Time to failure after rotator cuff repair: A 13. Chan K, MacDermid JC, Hoppe DJ, et al. Delayed versus early motion after arthroscopic prospective imaging study. J Bone Joint Surg Am 2013 Jun;95(11):965–71. DOI: https:// rotator cuff repair: A meta-analysis. J Shoulder Elbow Surg 2014 Nov;23(11):1631–9. doi.org/10.2106/JBJS.L.00708, PMID:23780533. DOI: https://doi.org/10.1016/j.jse.2014.05.021 4. Liem D, Bartl C, Lichtenberg S, Magosch P, Habermeyer P. Clinical outcome and tendon 14. Chang KV, Hung CY, Han DS, Chen WS, Wang TG, Chien KL. Early versus delayed integrity of arthroscopic versus mini-open supraspinatus tendon repair: A magnetic passive range of motion exercise for arthroscopic rotator cuff repair: A meta-analysis of resonance imaging-controlled matched-pair analysis. Arthrosc J Arthrosc Relat Surg 2007 randomized controlled trials. Am J Sports Med 2015 May;43(5):1265–73. DOI: https://doi. May;23(5):514–21. DOI: https://doi.org/10.1016/j.arthro.2006.12.028 org/10.1177/0363546514544698, PMID:25143489. 5. Miller BS, Downie BK, Kohen RB, et al. When do rotator cuff repairs fail? Serial ultrasound 15. Hsu JE, Horneff JG, Gee AO. Immobilization after rotator cuff repair: What evidence do examination after arthroscopic repair of large and massive rotator cuff tears. Am J Sports Med we have now? Orthop Clin North Am 2016 Jan;47(1):169–77. DOI: https://doi.org/10. 2011 Oct;39(10):2064–70. DOI: https://doi.org/10.1177/0363546511413372, PMID:21737833. 1016/j.ocl.2015.08.017, PMID:26614931.

26 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.088 n ORIGINAL RESEARCH ARTICLE Trends in Endoscopic Sinonasal Debridement in the Medicare Population

Mulin Xiong, BA1; Rijul S Kshirsagar, MD2; Jonathan Liang, MD, FACS, FARS2 Perm J 2021;25:20.110 E-pub: 03/01/2021 https://doi.org/10.7812/TPP/20.110

ABSTRACT associated with potential postoperative complications, Introduction: Nasal/sinus endoscopy with biopsy/polypectomy/ including a low risk (< 1%) of major complications such as debridement, or Current Procedure Terminology code 31237, is cerebrospinal fluid leak, meningitis, hemorrhage, and one of the top 10 most frequent and highest billed otolaryn- orbital injuries.5 Furthermore, despite a high initial success gology procedures among Medicare patients. We analyzed rate, up to 10% to 15% of patients will require revision temporal and geographic trends in endoscopic debridement, surgery.5 and correlated them with sinus surgery and balloon sinuplasty Controversy surrounds the ideal postoperative care reg- trends. Methods: Medicare Part-B National Summary Data Files were imen after FESS and largely falls to provider preference. In analyzed from 2000 to 2016 for temporal trends of endoscopic particular, recommendations for the timing and extent of debridement. Medicare Physician and Other Supplier Public Use postoperative debridement after FESS varies. Some suggest Files detailing provider information were collected and analyzed initial debridement days after the procedure followed by from 2012 to 2016. Individual providers performing a reportable weekly or monthly follow-up visits.6 Regular debridement number of procedures were included. Linear regression was used after sinus surgery has been shown to reduce adhesions to correlate endoscopic debridement, sinus surgery, and balloon when compared to saline irrigation alone.7 However, sinuplasty procedures. depending on debridement frequency, issues arise regarding Results: Between 2000 and 2016, the number of endoscopic patient compliance and comfort without substantial pro- debridement procedures increased from 31,579 to 79,762 (6.0% healing benefits.8 Some providers have implemented al- average annual growth). The annual total payments increased ternative postoperative care strategies in adult patients to from $5,944,582 to $19,438,956 (8.4% average annual growth), with average allowed charge per procedure increasing from minimize debridement while still preventing adhesions and 9,10 $188.24 to $243.71. The greatest and least number of debride- crusting, and maintaining rates of symptom improvement. ment procedures occurred in the Southeast (12,703) and New As such, although aggressive debridement may aid in England (1810) regions, respectively. There was a positive cor- remucosalization and improve surgical success, patient relation between providers (n = 752) performing endoscopic discomfort and associated procedural costs also need to be debridement and sinus surgery (r = 0.31, p < 0.001), which was taken into account. In a recent systematic review, Green similar to providers performing endoscopic debridement and et al11 analyzed 6 randomized controlled trials comparing balloon sinuplasty (r = 0.29, P < 0.001). postoperative debridement to no debridement. eir work Conclusion: Otolaryngologists continue to perform increasing demonstrated no clear evidence in long-term benefits based numbers of endoscopic debridements and receive increasing on symptom scores. Additional studies have revealed evi- payments. There is some geographic variation in these trends. dence for other postoperative care involving saline irriga- Among individual providers, there was a positive correlation between the number of endoscopic debridement procedures tion, topical and systemic corticosteroids, and antibiotics to and both the number of balloon sinuplasty and sinus surgery alleviate short-term symptoms, prevent nasal polyp recur- procedures. rence, and improve ease of surgery and postoperative appearance.12-15 First introduced in the early 2000s, balloon sinuplasty INTRODUCTION (BSP) was approved for the treatment of CRS in 2005 Chronic rhinosinusitis (CRS) is an inflammatory disease by the Food and Drug Administration. Compared to of the paranasal sinuses that affects up to 15% of the US traditional FESS, BSP is considered a minimally invasive population.1 CRS disease burden is high, with a national annual health-care cost estimated between $8 billion and $64.5 billion dollars, largely arising from outpatient services, Author Affiliations 1Michigan State University College of Human Medicine, Lansing, MI prescription medication, ambulatory care, and emergency 2Department of Head and Neck Surgery, Kaiser Permanente Oakland & Richmond Medical Centers, Oakland, CA room treatment.2 For patients with CRS refractory to medical treatment, conventional functional endoscopic sinus sur- Corresponding Author gery (FESS) has provided immediate and long-term relief Rijul Kshirsagar, MD ([email protected]) 3,4 in as many as 85% of patients. However, the procedure is Keywords: balloon sinuplasty, chronic rhinosinusitis, endoscopic debridement, functional endoscopic sinus surgery, Medicare The Permanente Journal·https://doi.org/10.7812/TPP/20.110 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 27 ORIGINAL RESEARCH ARTICLE Trends in Endoscopic Sinonasal Debridement in the Medicare Population

Table 1. Top 10 ear, nose, and throat procedures for Medicare patients in 2015 Rank HCPCS description (HCPCS code) Total allowed charges ($) Total nonfacility volume 1 Nasal endoscopy dx (31231) 94,472,734 454,324 2 Diagnostic laryngoscopy (31575) 58,471,364 489,057 3 Remove impacted ear wax unilateral (69210) 37,124,601 726,043 4 Antigen therapy services (95165) 24,996,095 2,072,867 5 Nasal/sinus endoscopy surgery (31237) 24,409,484 80,453 6 Sinus endoscopy with balloon dilation of frontal sinus ostium (31296) 21,342,912 17,381 7 Sinus endoscopy with balloon dilation of maxillary sinus ostium (31295) 14,370,228 10,967 8 Comprehensive hearing test (92557) 14,208,909 333,378 9 Laryngoscopy telescopic (31579) 11,041,916 52,893 10 Create eardrum opening (69433) 7,977,471 36,688 dx = diagnostic; HCPCS = Healthcare Common Procedure Coding System.

procedure with the potential to preserve the nasal mucosa, Table 2. Current procedural terminology (CPT) codes for shorten postoperative recovery, and minimize complications.16 postoperative endoscopic sinonasal debridement, functional Calitxo et al17 demonstrated the rapid replacement of tra- endoscopic sinus surgery (FESS), and balloon sinuplasty ditional FESS with BSP procedures since the introduction procedures of Current Procedure Terminology (CPT) codes specific for Type Procedure name CPT code BSP in 2011. Given the less-invasive and mucosa-sparing Postoperative endoscopy/debridement 31237 nature of this new procedure, one would expect a lower need FESS Endoscopic maxillary antrostomy + tissue 31267 for postoperative debridement. However, rates of debride- FESS Endoscopic frontal ± tissue 31276 ment utilization given the rise in BSP remain uncharacterized. FESS Endoscopic sphenoid 31287 Since 2014, the Centers for Medicare and Medicaid FESS Endoscopic sphenoid + tissue 31288 Services has made available data on utilization, payment, FESS Endoscopic anterior ethmoid 31254 CPT codes, and place of service information for fee-for- FESS Endoscopic total ethmoid 31255 18 service beneficiaries enrolled in Medicare Part B. is FESS Endoscopic maxillary antrostomy 31256 information has made it possible to analyze the rates and Balloon Balloon maxillary 31295 utilization of procedures, and provide insight into epide- Balloon Balloon frontal 31296 17, 19-21 miologic and financial burdens of various diseases. e Balloon Balloon sphenoid 31297 primary objective of this study was to analyze temporal and geographic trends in endoscopic debridement from 2000 to 2016 in the Medicare population. In addition, we aimed to years 2012 to 2016, which were the only years available at correlate the use of endoscopic debridement with FESS and the time of analysis. Geographic information was obtained BSP given temporal shifts in procedure utilization. from 2016 and was summarized from provider information. Geographic results were organized using US Census METHODS Bureau designated regions: the West (including Mountain Medicare Part B National Summary data files22 were and Pacific states), Northeast (Mid-Atlantic and New analyzed from 2000 to 2016 for temporal trends of en- England), Midwest (East North Central, West North doscopic debridement, FESS, and BSP (see Table 1 for Central), and South (West South Central, East South CPT codes). ese data files include information about the Central, South Atlantic). e West and East South Central number of services performed for a given CPT code as well regions were combined. e 2015 US Census population as allowed charges and payments. Allowed charge is defined estimate23 was used to obtain regional populations of as the “amount Medicare determines to be reasonable payment individuals ≥ 65 years old. ese values were used to es- for a provider or service covered under Part B [which] includes timate procedures per 100,000 Medicare beneficiaries. the coinsurance and deductible amounts.” Payment is the Linear regression was used to correlate endoscopic de- amount Medicare reimburses to providers. bridement, sinus surgery, and BSP procedures (Table 2). e Medicare Physician and Other Supplier Public Use Providers who performed no procedures, including no bal- data files18 provide information on services and procedures loon dilation and no endoscopic sinus surgery, were ex- provided to Medicare beneficiaries by physicians and other cluded from the regression analysis. We performed ordinary health-care professionals. Data files were collected for the least-squares linear regression analysis, with the test of

28 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.110 ORIGINAL RESEARCH ARTICLE Trends in Endoscopic Sinonasal Debridement in the Medicare Population

Figure 1. The number of endoscopic sinus debridements performed has doubled Figure 2. Average Medicare-allowed charges and reimbursements for endoscopic during the past 16 years. BSP = balloon sinuplasty procedures; FESS = functional debridements have increased during the past 16 years. endoscopic sinus surgery.

Pearson’s r for bivariate correlations. Statistical analysis was performed using RStudio version 0.98.1091 (RStudio, Boston, MA) in R 3.1.2 (e R Foundation for Statistical Computing, Vienna, Austria). is study was exempt from institutional review board review at our institution.

RESULTS Between 2000 and 2016, the number of allowed endo- scopic debridement procedures increased from 31,579 to 79,762, with an annual average growth rate of +6.0% Figure 3. Endoscopic debridement procedures in 2016 denoted as total number (Figure 1). During this time frame, annual growth rate and number per 100,000 population members. There is substantial geographic variation in the number of endoscopic debridements performed in the United trended downward from +13.3% in 2000 to 2001 to +5.5% States. in 2015 to 2016 (range, +1.0%–+14.5%). During this pe- riod, the number of allowed FESS procedures increased e greatest and least numbers of total debridement pro- from 66,559 to 116,008, with an average annual growth rate cedures occurred in the South East (12,703) and New En- of +3.6%. Upon the introduction of BSP CPT codes in gland (1810) regions, respectively (Figure 3). enumberof 2011, the number of allowed BSP procedures from 2011 to procedures nationally totaled 47,841 and averaged 6834 per 2016 increased from 7604 to 42,949, with an average annual region. When adjusted for procedures per 100,000 Medicare growth rate of +44.2%. beneficiaries, the South Atlantic had the greatest number at e annual total reimbursement increased from $5,944,582 137 procedures per 100,000 beneficiaries whereas the Midwest to $19,438,956, with an average annual increase of +8.4%. is had the least at 70 procedures per 100,000 beneficiaries. annual change fluctuated over time without a notable trend, ere was a positive correlation between the number of ranging from –16.9% in 2013 to 2014 to +50.3% in 2003 to sinus surgeries performed by providers (n = 752) and the 2004. e annual total allowed charges trended similarly, number of endoscopic debridements performed (r =0.31,P < increasing from $7,596,575 to $25,443,892, with an average 0.001; Figure 4A). Similarly, there is also a positive corre- annual increase of +8.6%. Likewise, the annual change in lation between the number of BSP procedures performed and allowed charges tracked similarly with annual change in the number of endoscopic debridements (r =0.29,P <0.001; total reimbursement, ranging from –16.5% from 2013 to Figure 4B). When the number of sinus surgeries and BSP 2014 to +49.8% from 2003 to 2004. e average allowed procedures performed are combined, there is also a positive charge per procedure increased from $240.56 to $319.00, correlation to the number of endoscopic debridements whereas the average payment per procedure increased performed (r =0.4,P <0.001;Figure 4C). from $188.24 to $243.71 (Figure 2). Of note, although the average allowed charges and payment per procedure in- DISCUSSION creased over time, the proportion remained the same at is is the first study to report trends of endoscopic 75% to 78% reimbursement of allowed charge. debridement after sinus surgery in the Medicare population.

The Permanente Journal·https://doi.org/10.7812/TPP/20.110 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 29 ORIGINAL RESEARCH ARTICLE Trends in Endoscopic Sinonasal Debridement in the Medicare Population

source. By drawing on the evidence from previous work, this study seeks to add to the body of clinical knowledge, and describe postoperative care and practice following sinus surgery. BSP is generally reserved for patients without extensive or complex sinus disease and is less invasive in nature, leading to improved recovery and a decreased need for postop- erative debridement. Previous studies have demonstrated the safety and efficacy of BSP in treating a wide range of sinusitis patients with comparatively fewer complications and revisions, and greater symptom improvement and patient satisfaction rates compared to traditional FESS.26-28 Additional advantages of provider comfort level, ease of procedure, and increased reimbursement rates have likely contributed to the plateauing effect of non-BSP FESS procedures, with total sinus procedure numbers remaining relatively static whereas BSP procedures in- creased dramatically.17 Previous studies have demonstrated a lower rate of postoperative debridement in patients who underwent office BSP compared to FESS.26,29 As such, the expectation is that, as BSP practice expands and reduces the use of traditional FESS, associated postoperative de- bridement procedures should also decline in tandem. However, nasal/sinus endoscopy with biopsy/polypectomy/ debridement, or CPT code 31237, remains one of the top 10 most frequent and highest billed otolaryngology pro- cedures among Medicare patients (Table 1). Our study shows that the number of allowed endoscopic debridement procedures has increased somewhat steadily over time. Although the rate of growth shows little correlation with the transition timeline for BSP—namely, remaining static despite the dramatically increased use of BSP after 2011 (Figure 1),17 the consistent positive trend seems incon- gruent with the shift in sinus procedure distribution. Furthermore, although the number of endoscopic debride- ment procedures correlates positively with the number of Figure 4. Correlation analysis of the number of endoscopic debridement FESS procedures, there is also a positive correlation with procedures and functional endoscopic sinus surgery procedures and balloon the number of BSP procedures completed by physicians. sinuplasty procedures performed by providers demonstrating (A) a positive correlation between functional endoscopic sinus surgery and endoscopic is does not align with a key advantage of BSP needing debridements, (B) a positive correlation between balloon sinuplasty and fewer postoperative debridement procedures when endoscopic debridements, and (C) a positive correlation between functional compared to FESS.29 ebodyofevidencecomparing endoscopic sinus surgery and balloon sinuplasty combined, and endoscopic ffi debridements. the relative safety and e cacy of FESS and BSP, par- ticularly in determining superiority as far as recovery and e primary goal was to use publicly available Medicare Part revision rate, is ample yet complex, with many sources B data to analyze temporal and geographic trends in en- citing strong yet opposing perspectives.1,26,30 Further- doscopic debridement from 2000 to 2016. Furthermore, more, the evidence directly associating improved recovery this study served to correlate the use of endoscopic de- with fewer postoperative endoscopic debridement re- bridement with FESS and BSP in light of changes in mains unclear. procedure utilization. Previous studies have assessed the In our study, the Midwest region had the greatest overall shift from open to endoscopic approaches in sinus surgery,24 Medicare population but the least number of endoscopic geographic variations in endoscopic sinus surgery utilization,25 debridement procedures per 100,000 beneficiaries. Mean- and the transition from FESS to BSP17 using the same data while, the South Central and South Atlantic regions had

30 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.110 ORIGINAL RESEARCH ARTICLE Trends in Endoscopic Sinonasal Debridement in the Medicare Population

similar-size populations, but the greatest number of total increase in procedure reimbursement has the potential procedures, resulting in the highest number of procedures to serve as an incentive for providers when determining per 100,000 beneficiaries. is correlates with previous postoperative care and may be a possible explanation for the studies that demonstrated that the South had the greatest trends observed. overall increase in BSP procedures,17 the greatest total When BSP was initially introduced, controversy around number of endoscopic procedures,24 the greatest number of the new technique and technology focused on safety and endoscopic dacryocystorhinostomies,31 and the highest efficacy, as well as on operating time, associated costs with number of diagnoses of chronic rhinosinusitis.25 In par- disposable balloon catheter equipment, and complications ticular, Texas, Florida, and Louisiana had notably the of fluoroscopy or transillumination wire usage.30 e BSP highest total BSP procedures and highest total BSP pro- device from Acclarent costs approximately $1200 for a cedures per 100,000 beneficiaries.17 However, the definitive single-use device system, which initially served as a de- connection between diagnosis of CRS and performance of terrent because reimbursement rates would have called for endoscopic sinus surgeries is unclear, as Venkatraman et al25 the treatment of multiple sinuses or adjunctive procedures notably found no correlation between number of pro- per patient to cover equipment costs.40 As rates increased cedures performed and diagnosed Medicare beneficiaries, over time, with BSP procedures reimbursement ranging per capita number of otolaryngologists, or population density from 4 to 10 times greater than non-BSP sinus surgeries, of beneficiaries. Instead, rates of sinus surgeries may be and providers became more accustomed to the technique, more guided by physician practice patterns, perhaps a utilization increased dramatically.17 However, the overall result of the controversy regarding the comparative efficacy comparative health-care cost to patients remains unclear of medical versus surgical interventions.32 Still, other given variations in relative operating time, recovery time, studies assessing prevalence of total knee arthroplasty and and postoperative care between BSP with FESS.30,40,41 In other high-cost procedures such as bypass surgery and hip addition, the associated growing rate and cost of postop- replacement in the Medicare population have demon- erative endoscopic debridement procedures may contribute strated a direct correlation between increased number to overall increased health-care burden on patients un- of enrolled beneficiaries and procedure utilization.33,34 dergoing sinus surgery, regardless of method. Further as- Although the exact cause is still to be determined, variation sessment of cost-effectiveness and procedural efficacy is in health-care access between US regions as defined by needed to prevent overuse of postoperative debridement, number of specialists per capita, distance to major medical and to optimize patient-centered and evidence-based care. centers, and population density may contribute to differ- Given the age restriction of Medicare beneficiaries, this ences in utilization of postoperative procedures such as study has limited generalizability. e study focused on the endoscopic debridement. patient population older than 65 years, which does not fully Recent reviews have found no clear evidence for frequent represent the overall patient population requiring surgical postoperative debridement and called for additional ran- treatment of CRS.42 Given the potential for the elderly domized controlled trials to establish benefit, optimal fre- population to present with disproportionately advanced quency, extent, and timing of debridement.11,35,36 Despite sinus disease requiring greater postoperative care and de- the lack of evidence, our study has demonstrated a con- bridement, the findings of this study may not be repre- sistent upward trend of endoscopic debridement procedures sentative of patients undergoing sinus surgery at all ages. from 2000 to 2016 that is out of proportion with the overall However, studies have shown that, although patient age increase in sinus procedures, with an average annual growth may affect postoperative outcomes following endoscopic rate of +6.0% versus the +3.1% to +3.6% annual growth rate sinus surgery, patients in the over-65 age group typically of total sinus procedures.17 Meanwhile, the annual total have comparable or even superior postoperative outcomes allowed charges and reimbursement also increased during when compared to pediatric and adult groups.43-45 Further this period, with an average annual increase of +8.4%. is analysis of postoperative debridement utilization relative to escalation is not entirely accounted for by the national rate BSP utilization in the general population is needed to of inflation, which averaged +2.16% from 2000 to 2016,37 determine trends and efficacy of health-care use. Likewise, and even outpaces the annual growth rate of overall Medicare age-related considerations in procedural type and utilization and Medicaid health-care costs, ranging from +3% to +6%.38 as well as postoperative outcomes following BSP are nec- e overall number of qualified Medicare beneficiaries has essary to optimize care and to guide clinical decision also increased dramatically over time, increasing by +21.1% making. Despite these limitations, the Centers for Medi- from 2000 to 2010.39 Such factors do not fully account for care and Medicaid Services remains the single largest payer the overall positive trend in average allowed charge and for health care in the United States,46 making Medicare Part reimbursement per procedure observed in our study. Overall B information a valuable source of publicly available data

The Permanente Journal·https://doi.org/10.7812/TPP/20.110 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 31 ORIGINAL RESEARCH ARTICLE Trends in Endoscopic Sinonasal Debridement in the Medicare Population

that can provide insight into reimbursement incentives and 4. Soler ZM, Mace J, Smith TL. Symptom-based presentation of chronic rhinosinusitis and symptom-specific outcomes after endoscopic sinus surgery. Am J Rhinol 2008 health-care utilization trends. May–Jun;22(3):297–301. DOI: https://doi.org/10.2500/ajr.2008.22.3172,PMID: 18588763. 5. Krings JG, Kallogjeri D, Wineland A, Nepple KG, Piccirillo JF, Getz AE. Complications of CONCLUSION primary and revision functional endoscopic sinus surgery for chronic rhinosinusitis. Laryngoscope 2014 Apr;124(4):838–45. DOI: https://doi.org/10.1002/lary.24401, PMID: Timing and frequency of endoscopic debridement re- 24122737. mains a controversial part of postoperative care for sinus 6. Thaler ER. Postoperative care after endoscopic sinus surgery. Arch Otolaryngol Head Neck Surg 2002 Oct;128(10):1204. DOI: https://doi.org/10.1001/archotol.128.10.1204, procedures in the absence of standard recommendations or PMID:12365896. evidence of long-term benefit. Although BSP procedures 7. Bugten V, Nordgard˚ S, Steinsvag˚ S. The effects of debridement after endoscopic sinus surgery. Laryngoscope 2006 Nov;116(11):2037–43. DOI: https://doi.org/10.1097/01.mlg. are associated with a reduced need for postoperative de- 0000241362.06072.83 bridement, general trends toward BSP procedures over 8. Lee JY, Byun JY. Relationship between the frequency of postoperative debridement and patient discomfort, healing period, surgical outcomes, and compliance after endoscopic FESS have not decreased the practice of postoperative sinus surgery. Laryngoscope 2008 Oct;118(10):1868–72. DOI: https://doi.org/10.1097/ debridement from 2000 to 2016. Instead, both procedure MLG.0b013e31817f93d3, PMID:18641526. use and reimbursement have increased over time at rates 9. Fernandes SV. Postoperative care in functional endoscopic sinus surgery? Laryngoscope 1999 Jun;109(6):945–8. DOI: https://doi.org/10.1097/00005537-199906000-00020, disproportionate to the overall increase in total sinus pro- PMID:10369288. cedures and BSP utilization. e cause of the observed 10. Brennan LG. Minimizing postoperative care and adhesions following endoscopic sinus surgery. Ear Nose Throat J 1996 Jan;75(1):45–8. DOI: https://doi.org/10.1177/ increase remains unclear, but could be attributed to changes 014556139607500111, PMID:8608747. in practice as physicians adapt to the relatively novel BSP 11. Green R, Banigo A, Hathorn I. Postoperative nasal debridement following functional endoscopic sinus surgery: A systematic review of the literature. Clin Otolaryngol 2015 technique, adjustment in patient preferences and expec- Feb;40(1):2–8. DOI: https://doi.org/10.1111/coa.12330 tations of postoperative outcomes, or financial incentives 12. Freeman SRM, Sivayoham ESG, Jepson K, de Carpentier J. A preliminary randomised controlled trial evaluating the efficacy of saline douching following endoscopic sinus from increasing reimbursements. Given the lack of con- surgery. Clin Otolaryngol 2008 Oct;33(5):462–5. DOI: https://doi.org/10.1111/j.1749-4486. sistent clinical evidence and associated procedural dis- 2008.01806.x comfort and complications, overuse of endoscopic debridement 13. Grzeskowiak B, Wierzchowska M, Walorek R, Seredyka-Burduk M, Wawrzyniak K, Burduk PK. Steroid vs. antibiotic impregnated absorbable nasal packing for wound may represent a negotiable health and financial burden that healing after endoscopic sinus surgery: A randomized, double blind, placebo-controlled calls for greater scrutiny. Further analysis is needed to study. Braz J Otorhinolaryngol 2019 Jul–Aug;85(4):473–80. DOI: https://doi.org/10.1016/j. bjorl.2018.04.002, PMID:29807811. determine current debridement utilization in the general 14. Stjarne¨ P, Olsson P, Alenius˚ M. Use of mometasone furoate to prevent polyp relapse after population, evidence-based guidance for postoperative care endoscopic sinus surgery. Arch Otolaryngol Head Neck Surg 2009 Mar;135(3):296. DOI: ff https://doi.org/10.1001/archoto.2009.2 after sinus surgery, and cost-e ectiveness of sinus treatment 15. Albu S, Lucaciu R. Prophylactic antibiotics in endoscopic sinus surgery: A short follow-up regimens. v study. Am J Rhinol Allergy 2010 Jul–Aug;24(4):306–9. DOI: https://doi.org/10.2500/ajra. 2010.24.3475, PMID:20819471. 16. Catalano PJ. Balloon dilation technology: Let the truth be told. Curr Allergy Asthma Rep 2013 Apr;13(2):250–4. DOI: https://doi.org/10.1007/s11882-012-0337-5, PMID:23354529. Disclosure Statement 17. Calixto NE, Gregg-Jaymes T, Liang J, Jiang N. Sinus procedures in the Medicare The authors have no conflicts of interest to disclose. population from 2000 to 2014: A recent balloon sinuplasty explosion. Laryngoscope 2017 Sep;127(9):1976–82. DOI: https://doi.org/10.1002/lary.26597, PMID:28397270. 18. Centers for Medicare and Medicaid Services. Provider utilization and payment data: Authors’ Contributions Physician and other supplier. Accessed November 17, 2019. www.cms.gov/Research- Statistics-Data-and-Systems/Statistics-Trends-and-Reports/Medicare-Provider-Charge- Mulin Xiong, BA, and Rijul S Kshirsagar, MD, contributed equally to this article. Data/Physician-and-Other-Supplier Rijul S Kshirsagar, MD, and Jonathan Liang MD, FACS, FARS, participated in the 19. Yang EL, Macy TM, Wang KH, Durr ML. Economic and demographic characteristics study design, acquisition and analysis of data, and drafting and critical review of the of cerumen extraction claims to Medicare. JAMA Otolaryngol Head Neck Surg final manuscript. Mulin Xiong, BA, participated in the analysis of data, drafting, 2016 Feb;142(2):157. DOI: https://doi.org/10.1001/jamaoto.2015.3129, critical review, and submission of the final manuscript. All authors have given final PMID:26720764. approval to the manuscript. 20. Huntington SF, Keshishian A, Xie L, Baser O, McGuire M. Evaluating the economic burden and health care utilization following first-line therapy for diffuse large B-cell lymphoma patients in the US Medicare population. Blood 2016;128(22):3574. DOI: https:// doi.org/10.1182/blood.V128.22.3574.3574 Funding 21. Nussbaum SR, Carter MJ, Fife CE, et al. An economic evaluation of the impact, cost, and The authors have no funding or financial relationships to disclose. Medicare policy implications of chronic nonhealing wounds. Value Health 2018 Jan;21(1): 27–32. DOI: https://doi.org/10.1016/J.JVAL.2017.07.007 22. Centers for Medicare and Medicaid Services. Part B National summary data file References (previously known as BESS). Accessed November 17, 2019. www.cms.gov/research- fi 1. Thottam PJ, Haupert M, Saraiya S, Dworkin J, Sirigiri R, Belenky WM. Functional statistics-data-and-systems/downloadable-public-use- les/part-b-national-summary- fi endoscopic sinus surgery (FESS) alone versus balloon catheter sinuplasty (BCS) and data- le/overview ethmoidectomy: A comparative outcome analysis in pediatric chronic rhinosinusitis. Int J 23. American FactFinder. US Census Bureau. Accessed November 17, 2019. https:// Pediatr Otorhinolaryngol 2012 Sep;76(9):1355–60. DOI: https://doi.org/10.1016/J. factfinder.census.gov/faces/nav/jsf/pages/index.xhtml IJPORL.2012.06.006, PMID:22770596. 24. Svider PF, Sekhsaria V, Cohen DS, Eloy JA, Setzen M, Folbe AJ. Geographic and 2. Bhattacharyya N, Villeneuve S, Joish VN, et al. Cost burden and resource utilization in temporal trends in frontal sinus surgery. Int Forum Allergy Rhinol 2015 Jan;5(1):46–54. patients with chronic rhinosinusitis and nasal polyps. Laryngoscope 2019 Sep;129(9): DOI: https://doi.org/10.1002/alr.21425, PMID:25367305. 1969–75. DOI: https://doi.org/10.1002/lary.27852 25. Venkatraman G, Likosky DS, Morrison D, Zhou W, Finlayson SRG, Goodman DC. Small 3. Damm M, Quante G, Jungehuelsing M, Stennert E. Impact of functional endoscopic sinus area variation in endoscopic sinus surgery rates among the Medicare population. Arch surgery on symptoms and quality of life in chronic rhinosinusitis. Laryngoscope 2002 Feb; Otolaryngol Head Neck Surg 2011 Mar;137(3):253. DOI: https://doi.org/10.1001/archoto. 112(2):310–5. DOI: https://doi.org/10.1097/00005537-200202000-00020 2011.17

32 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.110 ORIGINAL RESEARCH ARTICLE Trends in Endoscopic Sinonasal Debridement in the Medicare Population

26. Levine HL, Sertich AP, Hoisington DR, Weiss RL, Pritikin J. PatiENT Registry Study 36. Tzelnick S, Alkan U, Leshno M, Hwang P, Soudry E. Sinonasal debridement versus no Group. Multicenter registry of balloon catheter sinusotomy outcomes for 1,036 patients. debridement for the postoperative care of patients undergoing endoscopic sinus surgery. Ann Otol Rhinol Laryngol 2008 Apr;117(4):263–70. DOI: https://doi.org/10.1177/ Cochrane Database Syst Rev 2018 Nov;(11):CD011988. DOI: https://doi.org/10.1002/ 000348940811700405 14651858.CD011988.pub2 27. Ramadan HH. Safety and feasibility of balloon sinuplasty for treatment of chronic 37. US Bureau of Labor Statistics. Consumer price index (CPI) databases. Accessed rhinosinusitis in children. Ann Otol Rhinol Laryngol 2009 Mar;118(3):161–5. DOI: https:// November 17, 2019. www.bls.gov/cpi/data.htm doi.org/10.1177/000348940911800301 38. Centers for Medicare and Medicaid Services. National health expenditure data. Accessed 28. Batra PS., Ryan MW. Sindwani R, Marple BF. Balloon catheter technology in rhinology: November 17, 2019. www.cms.gov/Research-Statistics-Data-and-Systems/Statistics- Reviewing the evidence. Laryngoscope 2011 Jan;121(1):226–32. DOI: https://doi.org/10. Trends-and-Reports/NationalHealthExpendData/index 1002/lary.21114 39. Howden LM, Meyer JA. Age and sex composition: 2010. 2011. DOI: https://doi.org/10. 29. Cutler J, Bikhazi N, Light J, Truitt T, Schwartz M Standalone balloon dilation versus sinus 1017/cbo9780511781001.010 surgery for chronic rhinosinusitis: A prospective, multicenter, randomized, controlled trial. 40. Stewart AE, Vaughan WC. Balloon sinuplasty versus surgical management of chronic Am J Rhinol Allergy 2013 Sept–Oct;27(5):416–22. DOI: https://doi.org/10.2500/ajra.2013. rhinosinusitis. Curr Allergy Asthma Rep 2010 May;10(3):181–7. DOI: https://doi.org/10. 27.3970, PMID:23920419. 1007/s11882-010-0105-3, PMID:20425008. 30. Koskinen A, Myller J, Mattila P, et al. Long-term follow-up after ESS and balloon sinuplasty: 41. Hopkins C, Noon E, Bray D, Roberts D. Balloon sinuplasty: Our first year. J Laryngol Otol Comparison of symptom reduction and patient satisfaction. Acta Otolaryngol 2016 May; 2011 Jan;125(1):43–52. DOI: https://doi.org/10.1017/S0022215110001520 136(5):532–6. DOI: https://doi.org/10.3109/00016489.2015.1129553 42. Won HK, Kim YC, Kang MG, et al. Age-related prevalence of chronic rhinosinusitis and 31. Kshirsagar RS, Vu PQ, Liang J. Endoscopic versus external dacryocystorhinostomy: nasal polyps and their relationships with asthma onset. Ann Allergy Asthma Immunol Temporal and regional trends in the United States Medicare population. Orbit 2019 2018 Apr;120(4):389–94. DOI: https://doi.org/10.1016/j.anai.2018.02.005, PMID: Dec;38(6):453–60. DOI: https://doi.org/10.1080/01676830.2019.1572767,PMID: 29432969. 30712428. 43. Lee JY, Lee SW. Influence of age on the surgical outcome after endoscopic sinus 32. Khalil H, Nunez DA. Functional endoscopic sinus surgery for chronic rhinosinusitis. surgery for chronic rhinosinusitis with nasal polyposis. Laryngoscope 2007 Jun; Cochrane Database Syst Rev 2006 Jul;(3):CD004458. DOI: https://doi.org/10.1002/ 117(6):1084–9. DOI: https://doi.org/10.1097/MLG.0b013e318058197a,PMID: 14651858.CD004458.pub2 17545871. 33. Cram P, Lu X, Kates SL, Singh JA, Li Y, Wolf BR. Total knee arthroplasty volume, 44. Colclasure JC, Gross CW, Kountakis SE. Endoscopic sinus surgery in patients older than utilization, and outcomes among Medicare beneficiaries, 1991–2010. J Am Med Assoc sixty. Otolaryngol Head Neck Surg 2004 Dec;131(6):946–9. DOI: https://doi.org/10.1016/j. 2012 Sep;308(12):1227. DOI: https://doi.org/10.1001/2012.jama.11153 otohns.2004.06.710 34. Card D, Dobkin C, Maestas N. The impact of nearly universal insurance coverage on 45. Jiang RS, Hsu CY. Endoscopic sinus surgery for the treatment of chronic sinusitis in health care utilization: Evidence from Medicare. Am Econ Rev 2008 Dec;98(5):2242–58. geriatric patients. Ear Nose Throat J 2001 Apr;80(4):230–2. DOI: https://doi.org/10.1177/ DOI: https://doi.org/10.1257/aer.98.5.2242, PMID:19079738. 014556130108000411, PMID:11338647. 35. Eloy P, Andrews P, Poirrier AL. Postoperative care in endoscopic sinus surgery: A critical 46. Centers for Medicare and Medicaid Services. CMS roadmaps for the traditional review. Curr Opin Otolaryngol Head Neck Surg 2017 Feb;25(1):35–42. DOI: https://doi. fee-for-service (FFS) program: Overview. Health care (Don Mills). 2015. Accessed org/10.1097/MOO.0000000000000332, PMID:27846022. January 7, 2018

The Permanente Journal·https://doi.org/10.7812/TPP/20.110 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 33 n ORIGINAL RESEARCH ARTICLE “Headache” Online Information: An Evaluation of Readability, Quality, Credibility, and Content

Abdulrahman Bukhari, MBBS1; Saad Alshihri, MBBS1; Mohammed Abualenain, MBBS1; Jordan Barton, BA1; Genevieve Kupsky, BS1; Jesse M Pines, MD1; Ali Pourmand, MD, MPH, RDMS1 Perm J 2021;25:20.185 E-pub: 03/01/2021 https://doi.org/10.7812/TPP/20.185

ABSTRACT Despite its prevalence, headache accounts for just 5% of Introduction: Headache is experienced by more than half of all general practice consultations,3 and only 41% of people the world population each year. In this study, we evaluate the with chronic migraines have consulted a health-care pro- content, quality, and health literacy required to understand online vider about their condition.4 With increasing frequency, information for patients with headaches. patients are turning to the internet as a source of health Methods: We selected 4 commonly used search engines (Google, information. In one study, nearly 59% of adult Americans Yahoo, Bing, and Ask.com) and searched using the term “headache.” reported searching for health information online.5 e The 30 top hits on each site were selected for review. After ex- clusions, we examined the websites for completeness of content, discrepancy in the amount of consultations and the amount readability, credibility, and quality. of internet searches suggests that individuals are seeking Results: A total of 28 websites were included. None of the information on headache from the internet rather than from websites met our criteria for completeness. Using 2 standard health-care providers. measures of readability, most websites required reading skills at Internet users searching “headache” online encounter a the 10th-grade level or greater. Only 4 of the 28 websites were variety of websites, including personal blogs, education readable below the eighth-grade level. Only 3 websites fulfilled portals, peer-reviewed articles, and patient information from all 4 credibility criteria of authorship, currency, citations, and professional societies. ese sources of online information disclosure. Most websites did not list authorship, and only 17% also come with varying degrees of content as well as reli- fl reported disclosures of con icts of interest. When assessing ability. Prior studies have shown that online health infor- quality of treatment information using the DISCERN tool, scores mation is highly variable in content and quality, raising ranged from 23 to 59, with a mean score of 41, which could be interpreted as “fair” quality. concerns about the accuracy and trustworthiness of consumer- 6,7 Conclusions: We found variable content and quality in online oriented websites. headache websites for patients. Many of these websites failed To be effective, online information should be clear, ac- to disclose information about authorship, conflicts of interest, and cessible, and consistent with patient education materials details on the prognosis or prevention of headaches. Readability, provided by clinicians. Two important components of this credibility, completeness, and quality of information were lacking are required health literacy and readability. Health literacy is in most websites. the ability of individuals and communities to obtain and understand basic health information and services needed to 8,9 INTRODUCTION make appropriate health decisions. Readability is the ease of reading a written text.10 To our knowledge, no studies Headache is one of the most common neurologic disorders have examined the content, quality, or health literacy re- worldwide. According to the World Health Organization, quired to read patient-oriented websites on headache. In almost half of the adult population has had a headache at this study, we aimed to describe the content in patient least one time during the past year.1 e most common websites for headache, as well as the required health literacy, types of primary headaches are tension-type headaches, readability, and quality. migraines, and cluster headaches.2 Secondary headaches— — originating from alternative causes may be the result of METHODS serious underlying diseases such as subarachnoid hemor- In this study, we used 4 popular search engines used rhage, intracranial tumors, or meningitis. (Google, Yahoo, Bing, and Ask.com) to identify patient- oriented headache websites to examine the content, read- ability, and quality of the material. We searched the term Author Affiliation “headache” across these 4 search engines and examined the 1Emergency Medicine Department, George Washington University School of Medicine and Health Sciences, first 30 results from each engine. We limited our websites to Washington, DC the top 30 results because prior studies have found that ffi Corresponding Author about 91% of internet tra c in an average search is con- Ali Pourmand, MD, MPH, RDMS ([email protected]) centrated on the first page.11

Keywords: credibility, headache, online, quality, readability 34 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.185 ORIGINAL RESEARCH ARTICLE “Headache” Online Information: An Evaluation of Readability, Quality, Credibility, and Content

Inclusion and Exclusion Criteria comprised of approximately 16 questions, with a rating scale We identified the 30 top websites each from the 4 search for each question of 1 to 5, where 1 means the article did not engines, for a total of 120 websites. Websites that consisted fulfill any elements of the question and 5 means the article of scientific research articles only, those that did not provide fulfilled all elements of the question. Questions 1 through 8 information about headache, advertisements, sponsored web- assess reliability and whether the sources of information can sites, medical dictionaries, websites that provided information be trusted, questions 9 through 15 focus on specificdetailsof on only 1 type of headache, and duplicate sites within and across the information about the treatment choices, and question 16 the search engines were excluded. After applying exclusion assesses the overall quality of information. e total DIS- criteria, 28 websites were evaluated. For each of the following CERN score is calculated by summing the scores, with scores scales, we tabulated the data using descriptive statistics. is that can range from 15 to 80. study was determined to be not human subject research and did not warrant an institutional review board review. RESULTS Content Assessment Content Assessment Across the 28 included websites, none fulfilled all 11 Our team used 11 criteria (description, types, causes, as- content criteria in our checklist. Only 7 (25%) websites sociated symptoms, triggers, red-flag symptoms, diagnosis, fulfilled 10 of the 11 criteria. Only a minority of websites treatment, alternative treatment, prevention, and prognosis) had information on prevention and prognosis, with 12 to quantify completeness of information for each website. (43%) and 4 (14%) containing that information, respec- ese criteria were developed based on the team’s assessment tively. Most websites described the causes and types of of the information about headache that websites for patients headaches, with 27 (96%) and 24 (85%) providing that should contain. An initial list was developed by the study information, respectively (Figure 1 and Table 1). team, and was then refined based on feedback from a group of 18 physicians. Health Literacy and Readability e readability scores of online health information about Readability Assessment headache ranged from 5.6 to 17.3 using the SMOG tool, e readability of websites was assessed using the SMOG with a mean of 10.2. Readability scores using the Flesch- index tool and the Flesch-Kincaid tool. Both tools estimate Kincaid tool ranged from 6.3 to 22.1, with a mean of 11.5. the years of education needed to comprehend an article. For Values generated using these tools can be interpreted as, to example, a readability score of 7 means an average student in understand most of the material, one needs the reading the sixth grade can understand the text. e Flesch-Kincaid ability of at least a 10th grader. Only 4 of the 28 websites tool measures the average number of syllables per word and had a readability score less than the eighth-grade level. the number of words per sentence, whereas the SMOG index tool uses a formula developed by McLaughlin, which Reliability generates a score based on the number of words with 3 or more e reliability of headache-related websites was assessed syllables at the beginning, middle, and end of a passage.12 using the JAMA Benchmark Criteria (authorship, currency, attribution, and disclosure); only 3 websites fulfilled all 4 Credibility Assessment criteria. e majority of the websites (n = 23, 82%) provided e JAMA benchmark tool consists of 4 criteria (au- the date on which the material was posted or updated. thorship, attribution, disclosure, and currency) developed by the Journal of the American Medical Association to assess credibility. Authorship refers to mention of the authors’ names, affiliations, and credentials. Attribution ensures that references are mentioned appropriately. Disclosure iden- tifies whether the ownership and any conflict of interest are properly declared. Last, currency establishes whether the content and the resources are properly dated and updated. Each website is given a score of 1 for each element that is present and a 0 for each element that is absent.

Quality Assessment e DISCERN instrument is a questionnaire used to assess the quality of health information for consumers.13 It is Figure 1. Percentage of websites for each checklist item.

The Permanente Journal·https://doi.org/10.7812/TPP/20.185 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 35 ORIGINAL RESEARCH ARTICLE “Headache” Online Information: An Evaluation of Readability, Quality, Credibility, and Content

Table 1. Number and percentage of websites for each checklist item Checklist items No. of websites Percentage of websites Causes 27 96 Types 24 86 Triggers 22 79 Associated symptoms 21 75 Red-flag symptoms 21 75 Treatment 21 75 Alternative treatment 20 71 Diagnosis 19 68 Description 18 64 Prevention 12 43 Prognosis 4 14 Figure 2. Average DISCERN score for each item.

Authorship information was provided in 17 websites (61%); attribution and disclosure were the least common criteria simpler terms in order to be accessible to all patients. In ad- noted, mentioned in 14 (54%) and 11 (39%) websites, dition, an alternative solution might be to use more visual respectively. media or diagrams, which may help both low- and high-literacy individuals both absorb and understand the information. Quality When assessing the reliability of the websites, we found e quality of the headache-related websites was assessed that that currency was the most reported element, with using the DISCERN tool, with an emphasis on treatment. around 82% of sites providing dates for when the content Websites were rated as “excellent” (63–80), “good” (51–62), was posted or updated. Authorship information was present “fair” (39–50), “poor” (27–38), and “very poor” (15–26).14-16 in only 60% of websites. Authorship can add assurance and e mean score was 41 (“fair”), with a range of 23 to 59. authenticity to the material, enabling readers to know who Only 7 websites were classified as “good,” and 3 were “very wrote the content. Attribution was only included in ap- poor.” None of the websites fell in the excellent range proximately half of the websites, which allows readers to (Figure 2 and Table 2). know the source of the information, can increase confidence in the material, as well as provide additional literature sources. DISCUSSION Disclosure of possible conflicts of interest was reported the In this study, we identified 28 websites that patients least and was only mentioned in 39% of sites, which may have commonly encounter when searching for online informa- an impact on the overall balance and bias of information. tion about headache and found that the type of content When it came to quality, the average website was “fair,” most frequently included were causes of headache, followed mostly because of the absence of clear aims. Lack of a clear by types and triggers. Information about associated symp- aim can make it difficult for readers to know whether the toms, available treatment options, and alternative treatment website can answer particular questions or whether it is were observed in more than 50% of the websites. By contrast, applicable to their problem. Furthermore, most websites did information regarding prevention strategies and prognosis not emphasize the importance of treating headaches, in- was present in a lower proportion of websites. erefore, cluding how headaches can affect daily life and the con- patients may read some websites that may not answer all sequences of not treating headaches. their potential questions about headache and may have to Based on the checklist assessment of the 28 websites, the search multiple sites for answers. top 3 websites in terms of content completeness (fulfilling 10 e average readability of the websites was well above the of the 11 checklist items) were www.medicinenet.com, www. recommended level of sixth to eighth grade.17 is could emedinicehealth.com,andwww.drugs.com. ewebsiteswith be attributed to the fact that health-related information the lowest checklist scores (4 of 11) included www.sciencedaily. contains many polysyllabic terms and lengthy sentences. com, www.nhsinform.scot,andwww.who.int (Table 3). Another contributing factor may be that these websites are mostly written by physicians who may not trained to write in LIMITATIONS simple terms for a lay audience. Writers of websites for is study has several limitations. We only evaluated 28 patients should consider shortening sentences and using websites, and it is possible that our results would be different

36 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.185 ORIGINAL RESEARCH ARTICLE “Headache” Online Information: An Evaluation of Readability, Quality, Credibility, and Content

Table 2. Total DISCERN score for each website Table 3. Checklist score for each website Websites Total DISCERN score Websites Checklist score www.drugs.com/cg/acute-headache.html 59 www.medicinenet.com/headache/article.htm 10 www.webmd.com/migraines-headaches/migraines- 56 www.emedicinehealth.com/mild_headache/article_em. 10 headaches-basics#1 htm www.buoyhealth.com/current/23-headache- 56 www.drugs.com/cg/acute-headache.html 10 remedies-comprehensive-list/ http://healthlibrary.brighamandwomens.org/Search/ 10 www.emedicinehealth.com/mild_headache/ 55 85,P00784 article_em.htm www.hopkinsmedicine.org/health/conditions-and- 10 https://en.wikipedia.org/wiki/Headache 55 diseases/headache www.medicalnewstoday.com/articles/320767.php 54 https://en.wikipedia.org/wiki/Headache 10 https://familydoctor.org/condition/headaches/ 52 https://familydoctor.org/condition/headaches/ 10 https://medlineplus.gov/headache.html 50 www.webmd.com/migraines-headaches/migraines- 9 www.migrainetrust.org/about-migraine/types-of- 49 headaches-basics#1 migraine/other-headache-disorders/headache/ www.health.harvard.edu/pain/headache-when-to-worry- 9 www.medicinenet.com/headache/article.htm 48 what-to-do www.healthline.com/health/headache 47 www.healthline.com/health/headache 9 www.wikem.org/wiki/Headache 46 www.migrainetrust.org/about-migraine/types-of- 9 migraine/other-headache-disorders/headache/ https://patient.info/brain-nerves/headache-leaflet 45 www.medicalnewstoday.com/articles/320767.php 8 https://my.clevelandclinic.org/health/diseases/9639- 39 headaches-in-adults www.wikem.org/wiki/Headache 8 www.ninds.nih.gov/Disorders/All-Disorders/ 39 https://my.clevelandclinic.org/health/diseases/9639- 7 Headache-Information-Page headaches-in-adults emedicine.medscape.com/article/1048596-overview 39 www.health.com/health/condition-article/ 7 0,,20327041,00.html www.health.com/health/condition-article/ 38 0,,20327041,00.html emedicine.medscape.com/article/1048596-overview 7 www.hopkinsmedicine.org/health/conditions-and- 37 www.merckmanuals.com/professional/neurologic- 7 diseases/headache disorders/headache/approach-to-the-patient-with- headache www.mayoclinic.org/symptoms/headache/basics/ 34 definition/sym-20050800 www.buoyhealth.com/current/23-headache-remedies- 7 comprehensive-list/ www.radiologyinfo.org/en/info.cfm?pg=headache 34 www.radiologyinfo.org/en/info.cfm?pg=headache 6 www.nhsinform.scot/illnesses-and-conditions/brain- 34 nerves-and-spinal-cord/headaches www.ninds.nih.gov/Disorders/All-Disorders/Headache- 6 Information-Page www.merckmanuals.com/professional/neurologic- 33 disorders/headache/approach-to-the-patient-with- http://chemocare.com/chemotherapy/side-effects/ 6 headache headache.aspx fl http://healthlibrary.brighamandwomens.org/Search/ 32 https://patient.info/brain-nerves/headache-lea et 6 85,P00784 www.mayoclinic.org/symptoms/headache/basics/ 5 fi https://kidshealth.org/en/kids/headache.html 31 de nition/sym-20050800 www.health.harvard.edu/pain/headache-when-to- 27 https://medlineplus.gov/headache.html 5 worry-what-to-do https://kidshealth.org/en/kids/headache.html 5 www.sciencedaily.com/terms/headache.htm 23 www.sciencedaily.com/terms/headache.htm 4 http://chemocare.com/chemotherapy/side-effects/ 23 www.nhsinform.scot/illnesses-and-conditions/brain- 4 headache.aspx nerves-and-spinal-cord/headaches www.who.int/news-room/fact-sheets/detail/ 23 www.who.int/news-room/fact-sheets/detail/headache- 4 headache-disorders disorders

if we evaluated additional or different websites. Also, we SMOG and Flesch-Kincaid readability assessment tools only evaluated websites written in English. ere is some were computed objectively for written text. Nevertheless, degree of subjectivity involved when using tools like the these tools do not assess other components of websites such JAMA Benchmark and DISCERN instruments. Also, as photos or videos, both of which can impact patient there was also subjectivity in assessing content using an understanding. Furthermore, these tools do not take into internally generated checklist. Scores generated using the consideration the complexity of medical terminology or

The Permanente Journal·https://doi.org/10.7812/TPP/20.185 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 37 ORIGINAL RESEARCH ARTICLE “Headache” Online Information: An Evaluation of Readability, Quality, Credibility, and Content

reader familiarity with such words. Because the content and 2. Olesen J, Gobel¨ H. The international classification of headache disorders 3rd edition. IHS Classification ICHD-3. International Headache Society; 2019. Accessed popularity of websites found online is ever-changing, our January 12, 2020. https://ichd-3.org/ results cannot be generalized beyond the date of publication. 3. Frese T, Druckrey H, Sandholzer H. Headache in general practice: Frequency, management, and results of encounter. Int Sch Res Notices 2014 Oct 29;2014:1–6. DOI: https://doi.org/10.1155/2014/169428 CONCLUSIONS 4. Dodick DW. Migraine. Lancet 2018 Mar;391(10127):1315–30. DOI: https://doi.org/10. 1016/s0140-6736(18)30478-1 Our study showed that health information on headache 5. Fox S. Health topics. Pew Research Center: Internet, Science & Tech: 2019. Accessed had low readability, mixed credibility, fair quality, and January 12, 2020. www.pewresearch.org/internet/2011/02/01/health-topics-2/ 6. Eysenbach G, Powell J, Kuss O, Sa ER. Empirical studies assessing the quality of health variable content. In general, most websites did not provide information for consumers on the world wide web: A systematic review. J Am Med Assoc information on prognosis or prevention of headaches. 2002 May;287(20):2691. DOI: https://doi.org/10.1001/jama.287.20.2691, PMID: 12020305. Measures should be taken to ensure readers have access to 7. Friedman DB, Hoffman-Goetz L. A systematic review of readability and comprehension reliable information online. v instruments used for print and web-based cancer information. Health Educ Behav 2006 Jun;33(3):352–73. DOI: https://doi.org/10.1177/1090198105277329 8. Nielson-Bohlman L, Panzer A, Kindig D, eds. Health literacy: A prescription to end Disclosure Statement confusion. Washington, DC: National Academies Press; 2004. fl The authors have no con icts of interest to disclose. 9. World Health Organization. Track 2: Health literacy and health behaviour. World Health Organization; 2010. Accessed January 12, 2020. www.who.int/healthpromotion/ Authors’ Contributions conferences/7gchp/track2/en/ Abdulrahman Bukhari, MBBS; Saad Alshihri, MBBS; Mohammed Abualenain, 10. DuBay W. The principles of readability. Costa Mesa, CA: Impact Information; 2004. 11. Chitika, Inc. The value of Google result positioning: Chitika Online Advertising Network. MBBS; Jordan Barton, BA; Ali Pourmand, MD, MPH, RDMS; and Jesse M Pines, Chitika, Inc; 2013. Accessed January 12, 2020. https://web.archive.org/web/ MD; designed the study. Abdulrahman Bukhari, MBBS; Saad Alshihri, MBBS; 20130610125944/https://chitika.com/google-positioning-value Mohammed Abualenain, MBBS; Jordan Barton, BA; and Genevieve Kupsky, BS; 12. The SMOG readability formula, a simple measure of Gobbledygook. 2020. Accessed performed the analysis, drafted the manuscript, and designed the tables and January 12, 2020. https://readabilityformulas.com/smog-readability-formula.php figures. All authors discussed the results and contributed to the final manuscript. 13. General instructions: The DISCERN instrument. Accessed January 12, 2020. www. Ali Pourmand, MD, MPH, RDMS, and Jesse M Pines, MD, supervised the discern.org.uk/general_instructions.php project. 14. Weiss BD. Health literacy: A manual for clinicians. Chicago, IL: American Medical Association, American Medical Foundation; 2003. 15. Cotugna N, Vickery CE, Carpenter-Haefele KM. Evaluation of literacy level of patient Financial Support education pages in health-related journals. J Community Health 2005 Jun;30(3):213–9. This is a nonfunded study, with no compensation or honoraria for conducting the DOI: https://doi.org/10.1007/s10900-004-1959-x, PMID:15847246. study. 16. National Institutes of Health. How to write easy to read health materials: National Library of Medicine website. Accessed February 1, 2009. www.nlm.nih.gov/ medlineplus/etr.html References 17. Eltorai AE, Ghanian S, Adams CA Jr, Born CT, Daniels AH. Readability of patient education 1. Pietrasik T. Headache disorders. World Health Organization; 2020. Accessed January 12, materials on the American Association for Surgery of Trauma website. Arch Trauma Res 2020. www.who.int/news-room/fact-sheets/detail/headache-disorders 2014 Jun;3(2):e18161. DOI: https://doi.org/10.5812/atr.18161, PMID:25147778.

38 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.185 n ORIGINAL RESEARCH ARTICLE The Utility of Brain Magnetic Resonance Imaging/Angiography and Neck Magnetic Resonance Angiography in Patients with Suspected Acute Stroke

Mark Harris1; Alyssa Finger1; Emily Nishimura1; Blake Watabe1; Hyo-Chun Yoon, MD, PhD1 Perm J 2021;25:20.214 E-pub: 03/01/2021 https://doi.org/10.7812/TPP/20.214

ABSTRACT to patients. We hypothesized that the use of this triple study Background: In our health maintenance organization, we have should result in a substantial (> 10%) proportion of patients seen a trend among our referring physicians to order simulta- undergoing some type of open surgical or percutaneous neous brain magnetic resonance imaging (MRI), head magnetic endovascular intervention on the intracranial or cervical resonance angiography (MRA), and neck MRA in the evaluation of arteries within the following 90 days to justify performing all patients for acute stroke. However, there are little data to dem- 3 studies simultaneously rather than order them sequentially onstrate any improvement in patient care resulting from ordering as the clinical need arose. this triple study. The objective of this study was to analyze the utility of the triple MRI/MRA study for patients who experience stroke-like symptoms. METHODS Methods: We reviewed all triple-study cases between January Our institutional review board approved this study with a 1, 2013 and December 31, 2016. We recorded whether or not an waiver of consent. is retrospective analysis was per- acute stroke occurred, the presence or absence of a major ste- formed on patients who all belonged to a geographically nosis in the intracranial and/or neck arteries, subsequent per- isolated HMO where all imaging is provided within the cutaneous endovascular or open surgical intervention within HMO and where all inpatient and outpatient information 90 days, and any follow-up computed tomography angiography is available on an electronic medical record (EMR). We or carotid ultrasound studies within 30 days. reviewed the results from all adult patients who under- Results: During the studied period, 591 triple studies were went brain MRI, brain magnetic resonance angiography ordered, and 162 patients (27.4%) were found to have moderate (MRA), and neck MRA simultaneously for the assessment or severe stenosis. Of the patients who had an acute stroke, 100 (48.3%) also had a major stenosis. Of 591 patients, only 15 (2.5%) of acute stroke from the emergency department or inpa- underwent percutaneous endovascular or open surgical inter- tient setting. All information was obtained from the EMR vention within 90 days. Of these, 4 patients had an intervention in for triple studies performed between January 1, 2013 and less than a week; in all of the cases, the triple study did not need to December 31, 2016. be ordered simultaneously to achieve the same clinical outcome. Data collected at the time of the triple MRI/MRA ex- Conclusion: Brain MRI, head MRA, and neck MRA studies amination were as follows: age, sex, presence or absence of should not be ordered simultaneously as a generalized response acute stroke, presence or absence of a major stenosis in the to patients presenting with acute stroke-like symptoms. intracranial and/or cervical arteries, subsequent percu- taneous endovascular or open surgical intervention within INTRODUCTION 90 days, and any follow-up computed tomography angiog- raphy (CTA) or carotid ultrasound studies within 30 days. Magnetic resonance imaging (MRI) has been shown to Children who presented stroke-like symptoms were ex- be an excellent method for the detection of acute stroke cluded, as they are unlikely to have atherosclerotic disease of as well as for the evaluation of intracranial and cervical the cervical or intracranial arteries. arterial anatomy.1-3 In our health maintenance organization All imaging was performed on 1 of 3 MRI scanners: GE (HMO), we have seen a trend among our referring phy- Signa 1.5T (GE Medical Systems, Milwaukee, WI), Philips sicians to order all 3 studies simultaneously in the evaluation Intera 1.5T (Philips Medical Systems, Andover, MA), or of patients for acute stroke. e reason for this trend is Philips Ingenia 3T. Intracranial MRA was performed using unclear but may be related to a perceived benefit in pro- a 3-dimensional time-of-flight (TOF) technique without viding timely interventions or reducing patient inconve- nience associated with having to return for multiple studies. However, there are little data to demonstrate any im- Author Affiliation provement in patient care resulting from ordering this triple 1Department of Diagnostic Imaging, Kaiser Permanente Moanalua Medical Center, Honolulu, HI MRI study. At a time when health care costs are under increasing scrutiny, the use of multiple expensive imaging Corresponding Author Alyssa Finger ([email protected]) studies is not warranted unless there is clear clinical benefit Keywords: brain MRI, EMR, head MRA, neck MRA, radiology, stroke, utilization The Permanente Journal·https://doi.org/10.7812/TPP/20.214 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 39 ORIGINAL RESEARCH ARTICLE The Utility of Brain Magnetic Resonance Imaging/Angiography and Neck Magnetic Resonance Angiography in Patients with Suspected Acute Stroke

Table 1. Comparison of patients determined to have had or not had a stroke by magnetic resonance imaging Total (n = 591) No stroke (n = 384) Acute stroke (n = 207) No stenosis 429 (72.6) 322 (83.9) 107 (51.7) Stenosis 162 (27.4) 62 (16.1) 100 (48.3) Intracranial 58 (35.5) 21 (33.9) 37 (37.0) Extracranial 95 (64.5) 39 (62.9) 56 (56.0) Both 9 (1.5) 2 (3.2) 7 (7.0) No intervention 571 (96.6) 378 (98.4) 193 (93.2) Intervention 20 (3.2) 6 (1.6) 14 (6.8) Values are presented as n (%).

intravenous contrast. Neck MRA was performed using a Table 2. Comparison of the sexes and average ages of patients 3-dimensional TOF technique with intravenous contrast. who did or did not have a stenosis All magnetic resonance images of patients who presented No stenosis (n = 429) Stenosis (n = 162) with major intracranial arterial disease or cervical arterial Age, y 62.1 ± 15.8 68.1 ± 15.3 disease were reviewed. Patients were considered to have Sex major arterial disease when specifically reported by the “ ” Men 203 (47.3) 85 (52.5) radiologist as a stenosis of greater than 50% or as severe. Women 226 (52.7) 77 (47.5) Furthermore, one of the authors reviewed the images that Values are presented as averages ± SD or n (%). were reported as “moderate” or “mild” stenosis and mea- sured the stenosis severity. ose stenoses greater than 50% were considered major. e EMRs of patients with major while 429 patients (72.6%) did not demonstrate any stenosis stenoses were reviewed to determine whether percutaneous on MRI studies. e average ± SD age was 68.1 ± 15.3 years endovascular or open surgical intervention occurred within for patients with stenosis and 62.1 ± 15.8 years for the group 90 days. without stenosis (Table 2). is age difference was statis- Any CTA or carotid ultrasound study performed tically significant (Student t-test, p < 0.001). Among those within 30 days of the triple study was reviewed and patients with stenosis, there were 77 women and 85 men; results correlated the finding on the magnetic resonance among those without a major stenosis, there were 226 angiograms. women and 203 men, which was not significantly different Continuous variables were analyzed through a Student (χ2 test, p > 0.05) (Table 2). t-test. Nominal variables were analyzed with a χ2 test. As shown in Figure 1, 207 (35.0%) of the 591 patients Statistical significance was determined to be p ≤ 0.05. who underwent the triple study had an acute stroke as determined by MRI. Of the patients who had a stroke, RESULTS 100 (48.3%) also had some major stenosis. Only 62 patients Between January 1, 2013 and December 31, 2016, a (16.1%) who did not have a stroke had a major stenosis. total of 591 adult patients received a triple MRI/MRA Patients who had a stroke were significantly more likely to study of the head and neck simultaneously. Overall, only have a stenosis (χ2 test, p < 0.001). Of the patients who had a 15 patients in this cohort (2.5%) had an intervention stroke, 37 (37.0%) presented with intracranial stenosis only, within 90 days of the examinations. is rate of inter- 56 (56.0%) presented with extracranial stenosis only, and vention was substantially lower than our hypothesized 7 (7.0%) presented with both (Table 1). Of the patients 10% rate of intervention. who did not have a stroke, 21 stenoses (33.9%) were In our cohort of patients who underwent intervention, the intracranial, 39 (62.9%) were extracranial, and 2 (3.2%) median number of days between the triple study and in- presented with both (Table 1). ere was no significant tervention was 51 days (interquartile range, 2.5-56.5 days). difference in the distribution of intracranial versus ex- erewere303womenand288men,withanaverage tracranial stenoses among patients who did or did not age±SDof63.7±15.9years.eaverageageofthe have a stroke (χ2 test, p > 0.05). women (62.7 ± 17.4 years) was not significantly different None of the patients in this cohort who presented with from that of the men (64.8 ± 14.0 years) (Student t-test, intracranial stenosis underwent a percutaneous endovas- p > 0.05). cular or open surgical procedure. Of 63 patients with a As shown in Table 1, 162 patients (27.4%) were found to major extracranial stenosis who also had an acute stroke, have moderate or severe stenosis based on MRI studies, 9 (14.2%) underwent surgical intervention within 90 days.

40 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.214 ORIGINAL RESEARCH ARTICLE The Utility of Brain Magnetic Resonance Imaging/Angiography and Neck Magnetic Resonance Angiography in Patients with Suspected Acute Stroke

Figure 1. A flow chart description of triple-study patients who did or did not have an acute stroke, did or did not have a major stenosis, and did or did not have an intervention and how long the intervention took.

e remaining 54 patients (85.7%) did not undergo surgical urgent carotid endarterectomy for crescendo transient intervention within 90 days. ere were 41 patients with a ischemic attack and another underwent urgent carotid major extracranial stenosis who did not have an acute stroke, endarterectomy for continued symptoms 2 days after acute 5 of whom (12.2%) underwent surgical intervention within stroke. e fourth patient underwent carotid endarterectomy 90 days. e remaining 36 patients (87.8%) did not undergo within a week of imaging because of ongoing drug abuse, surgical intervention. Intervention rates between patients which the operating surgeon felt made the patient to be with a major cervical stenosis were not significantly different unreliable for discharge without surgical intervention. (χ2 test, p > 0.05) between patients who did or did not have Of the remaining 11 patients, surgical intervention an acute stroke. occurred more than a week after the triple study. In ad- A total of 68 patients (11.5%) also received follow-up dition, 3 patients had an intervention that occurred more imaging within 30 days (±1 week) of the simultaneous triple than 90 days after their triple study. ese patients were study. Of these, 52 patients received an ultrasound of the not included in our analysis because of the long time carotid arteries (76.5%). One patient underwent a cerebral interval between the surgical intervention and their triple angiography (1.5%) and the remaining patients received study. either MRA or CTA. During 2013, 120 triple studies were completed. In 2014, Twelve patients (17.7%) had a substantial difference in there were 129 triple studies. is number increased in 2015 their imaging findings in follow-up studies compared to the to 163 triple studies. Finally, in 2016, 187 of these triple original triple study. Of these, no patients underwent in- studies were completed. is high utilization continues in tervention. Eleven of the 12 patients showed no major our HMO, as the data from 2019 demonstrated 119 such stenosis on carotid ultrasound compared to the original neck triple studies in patients with suspected acute stroke in just MRA findings at the time of the triple study because the the first 6 months. stenosis was not substantial enough to meet criteria for major stenosis on an ultrasound. One patient showed a DISCUSSION stenosis in the high neck that was not seen on the carotid Each year, more than 795,000 people experience a stroke ultrasound. However, this patient also had prior catheter and more than 130,000 people die from stroke in the US.4,5 angiography performed 5 years earlier, which demon- To assess strokes, brain MRI, intracranial MRA, and neck strated the same high-grade stenosis of the extracranial MRA are frequently used because of their high sensitivity in right internal carotid artery above the region imaged by the detection of acute stroke and stenoses.1-3 e American ultrasound. College of Radiology appropriateness criteria currently state Of the 15 patients who underwent surgical intervention that for patients with suspected stroke with symptoms within 90 days of examination, 4 patients had an inter- lasting longer than 6 hours, “Further evaluation is ideally vention within a week. One of these patients underwent performed with a contrast-enhanced brain MRI along carotid endarterectomy the day after examination, for the with TOF-MRA of the circle of Willis and CE-MRA of patient’s convenience. One of the patients underwent the cervical vessels.”6 is may be one reason why many

The Permanente Journal·https://doi.org/10.7812/TPP/20.214 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 41 ORIGINAL RESEARCH ARTICLE The Utility of Brain Magnetic Resonance Imaging/Angiography and Neck Magnetic Resonance Angiography in Patients with Suspected Acute Stroke

physicians in our HMO are ordering a simultaneous triple None of the 591 patients in this study had surgical inter- study of brain MRI, head MRA, and neck MRA when vention for intracranial stenosis. Some patients may benefit patients experience stroke-like symptoms. However, we from imaging of their cervical arteries. is can be done could find no studies published in the medical literature that with carotid ultrasound, CTA, or MRA depending on the demonstrate a substantial improvement in clinical outcomes clinical scenario. In our HMO, ultrasound is considered for patients who undergo simultaneous imaging of the brain the standard imaging modality for suspected cervical ca- with intracranial and neck MRA compared to those who do rotid stenosis. not. Our study found that the number of simultaneous triple In our patient population, less than 3% of patients un- studies increased from 120 to 187, annually, from 2013 to dergoing a triple study had a subsequent surgical inter- 2016. During this period, membership within the HMO vention within 90 days; the majority of these patients did so increased by less than 2%, marking an approximate 50% more than a week later. Based on our data, 48.3% of patients increase in the number of these studies. with an acute stroke had a major stenosis. ese patients Medicare reimbursement is often used as a marker of may benefit from additional vascular imaging during their imaging cost.7,8 According to Medicare Part B, the global admission or shortly thereafter if they are candidates for fees in our region are $250.10 for brain MRI, $277.46 for intervention. Only 16.1% of patients who did not have an head MRA without dye, and $278.70 for neck MRA.9 acute stroke had a major stenosis. Less than 2% of patients us, the average Medicare reimbursement of this triple who had a stroke underwent intervention within 90 days. study is approximately $806.26. In addition, ultrasound erefore, the vast majority of these patients did not require imaging is used as indication for surgery and has a re- immediate vascular imaging. e workup for these patients imbursement of $227.43, which brings the total fees to could have been performed at their convenience as clinically $1033.69. At a time when there is increasing scrutiny of appropriate. health expenditures, this is an expensive combination Only 4 patients out of the entire cohort had surgical of studies. intervention within a week of their triple study. One of these Furthermore, even within our HMO, many of our pa- patients elected to have that intervention within that week tients have substantial copayment requirements. Depending for the patient’s own convenience. Two other patients had on their health care copayment requirements, some patients ongoing or worsening symptoms and therefore required may be required to pay up to $300 per imaging study. In a urgent surgical intervention, which was performed 2 days time when health care costs are an increasing burden and after their triple study. e fourth patient was unreliable leading cause of bankruptcy, it is important to consider for follow-up and it was decided to perform the procedure the financial burden of all procedures.10 Finally, there are before the patient was discharged. considerable wait times for MRI studies within our HMO Our study suggests that only patients with ongoing or because of the high demand for all MRI studies. We un- worsening symptoms after admission may require urgent dertook this investigation because we suspected that there vascular imaging to determine whether surgical revascu- were very few urgent surgical interventions associated with larization is necessary. Even these patients do not require these triple MRI studies, which also require increased setup simultaneous brain MRI and MRA at their initial pre- and imaging times for our radiology technologists and sentation because they can and do undergo urgent imaging therefore limit the availability of these magnets for other with carotid ultrasound, CTA, or MRA. Since they rep- imaging studies. resent a very small proportion of our cohort (<1%), these For patients presenting within 3 to 4.5 hours of stroke studies would rarely be necessary on an emergent basis. In onset, intravenous alteplase may be used to dissolve clots fact, 2 of the 3 patients in this study requiring urgent and improve blood flow.11,12 Such patients can also undergo surgical revascularization also had a carotid ultrasound prior an endovascular procedure to remove clots if presenting to their surgical intervention. within 6 hours of symptom onset.11,12 ese patients were Limitations of this study include its retrospective nature, not included in this study, as our acute stroke protocol which precludes us from understanding why physicians dictates that such patients undergo emergent computed simultaneously order these triple studies and why some tomography of the head with CTA of the neck and brain patients underwent urgent revascularization without worsen- rather than MRI. ing symptoms while other patients’ surgeries were delayed 90 Most other patients who experience a stroke do not undergo or more days. Review of the medical records at the time of surgical interventions. e standard treatment for patients who admission for a small sample of the patients in this study experience an acute stroke involves 4 preventative therapies: an revealed that very few had a recorded National Institutes of antiplatelet agent, a statin, an angiotensin-converting enzyme Health stroke scale score included in their admission his- inhibitor, and a diuretic.13 tory and physical examination. As a National Institutes of

42 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.214 ORIGINAL RESEARCH ARTICLE The Utility of Brain Magnetic Resonance Imaging/Angiography and Neck Magnetic Resonance Angiography in Patients with Suspected Acute Stroke

Health stroke scale score greater than 10 has been shown to Financial Support be associated with a greater likelihood of large vessel No financial support was received for this work. occlusion, requiring a similar threshold to order a triple study in patients not eligible for acute thrombolytic References 14 1. Chalela JA, Kidwell CS, Nentwich LM, et al. Magnetic resonance imaging and computed therapy may reduce the number of these studies. Fi- tomography in emergency assessment of patients with suspected acute stroke: A nally, the findings of our study should not be applied to prospective comparison. Lancet 2007 Jan;369(9558):293-8. DOI: https://doi.org/10.1016/ S0140-6736(07)60151-2, PMID:17258669 those patients presenting with acute stroke who may be 2. Heiserman JE, Drayer BP, Keller PJ, Fram EK. Intracranial vascular stenosis and eligible for thrombolytic therapy. In our HMO, those occlusion: Evaluation with three-dimensional time-of-flight MR angiography. Radiology ff 1992 Dec;185(3):667-73. DOI: https://doi.org/10.1148/radiology.185.3.1438743, PMID: patients undergo a di erent care pathway with emergent 1438743 brain computed tomography with CTA and perfusion 3. Birenbaum D, Bancroft LW, Felsberg GJ. Imaging in acute stroke. West J Emerg Med imaging. 2011 Feb;12(1):67-76. PMID:21694755 4. Mozzafarian D, Benjamin EJ, Go AS, et al; American Heart Association Statistics In summary, only 3% of patients in our cohort un- Committee; Stroke Statistics Subcommittee. Heart disease and stroke statistics—2016 derwent subsequent surgical intervention. Furthermore, update: A report from the American Heart Association. Circulation 2016 Jan;133(4): e38-360. DOI: https://doi.org/10.1161/CIR.0000000000000350 approximately 1% of these patients underwent urgent re- 5. National Center for Health Statistics. Underlying cause of death 1999-2019 on CDC vascularization within a week of their MRI studies. Even WONDER online database. https://wonder.cdc.gov/deaths-by-underlying-cause.html Accessed 6/25/2020 within this small group, only 2 patients had worsening or 6. American College of Radiology. ACR appropriateness criteria: Cerebrovascular disease. continued symptoms that required urgent intervention. Published 2016. Accessed September 21, 2020. https://www.acr.org/ erefore, it is readily apparent that the intracranial MRA 7. Sharma A, Viets R, Parsons MS, Reis M, Chrisinger J, Wippold FJ 2nd. A two-tiered approach to MRI for hearing loss: Incremental cost of a comprehensive MRI over high- or neck MRA need not be performed concurrently with resolution T2-weighted imaging. AJR Am J Roentgenol 2014 Jan;202(1):136-44. DOI: brain MRI in patients with stroke-like symptoms. https://doi.org/10.2214/AJR.13.10610, PMID:24370138 8. Suh CH, Kim KW, Park SH, et al. A cost-effectiveness analysis of the diagnostic strategies for differentiating focal nodular hyperplasia from hepatocellular adenoma. Eur Radiol 2018 Conclusion Jan;28(1):214-25. DOI: https://doi.org/10.1007/s00330-017-4967-9, PMID:28726119 9. Physician fee schedule search. Accessed June 25, 2020. www.cms.gov/apps/physician- Physicians should not simultaneously order brain MRI, fee-schedule/ head MRA, and neck MRA for patients presenting with 10. Himmelstein DU, Thorne D, Warren E, Woolhandler S. Medical bankruptcy in the United States, 2007: Results of a national study. Am J Med 2009 Aug;122(8):741-6. DOI: https:// stroke-like symptoms and instead should order each study doi.org/10.1016/j.amjmed.2009.04.012 individually as clinically indicated. v 11. Filho J, Mullen M. Initial assessment and management of acute stroke. UpToDate. Published 2017. https://www.uptodate.com/contents/initial-assessment-and-management-of-acute-stroke 12. Powers WJ, Derdeyn CP, Biller J, et al. 2015 American Heart Association/American Disclosure Statement Stroke Association focused update of the 2013 guidelines for the early management of The author(s) have no conflicts of interest to disclose. patients with acute ischemic stroke regarding endovascular treatment: A guideline for healthcare professionals from the American Heart Association/American Stroke Association. Stroke 2015 Oct;46(10):3020-35. DOI: https://doi.org/10.1161/STR.0000000000000074. Authors’ Contributions 13. Wardlaw J, Brazzelli M, Miranda H, et al. An assessment of the cost-effectiveness of Mark Harris and Hyo-Chun Yoon, MD, PhD, participated in the study design, magnetic resonance, including diffusion-weighted imaging, in patients with transient acquisition and analysis of data, and drafting of the final manuscript. Alyssa Finger ischaemic attack and minor stroke: A systematic review, meta-analysis and economic fi evaluation. Health Technol Assess 2014 Apr;18(27):1-368. DOI: https://doi.org/10.3310/ participated in the critical review, drafting, and submission of the nal manuscript. hta18270, PMID:24791949 fi Emily Nishimura participated in the analysis of data and drafting of the nal 14. Cooray C, Fekete K, Mikulik R, Lees KR, Wahlgren N, Ahmed N. Threshold for NIH stroke manuscript. Blake Watabe participated in the study design. All authors have given scale in predicting vessel occlusion and functional outcome after stroke thrombolysis. Int J final approval to the manuscript. Stroke 2015 Aug;10(6):822-9. DOI:https://doi.org/10.1111/ijs.12451, PMID:25588617

The Permanente Journal·https://doi.org/10.7812/TPP/20.214 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 43 n ORIGINAL RESEARCH ARTICLE The ATTAIN Solution Tested: Initial Pilot Results of an Automated, Web-based Screening Tool for Unhealthy Drinking Behaviors

Jiseung Yoon, MD1; Emma Fredua, MPH, CHES2; Shahriar B Davari, MD2; Mohamed H Ismail, MD, DrPH2 Perm J 2021;25:20.143 E-pub: 03/17/2021 https://doi.org/10.7812/TPP/20.143

ABSTRACT unhealthy drinking behaviors, but have focused on hospitals, Background: Automated Alcohol Misuse Interventions (ATTAIN) community-based settings, and smaller population subsets is an automated, computer-based tool that screens people for (such as veterans or college students).5-29 ere is a gap in unhealthy alcohol use and offers web-based counseling with research covering computer-based programs that address minimal involvement of health-care personnel. We describe the unhealthy drinking behavior screening and intervention in outcomes of the initial quality improvement pilot tests done to integrated, managed-care organizations. examine the screening component of ATTAIN. In a separate, related article, we describe our development Methods: This is a data-only study that reports the results from the ATTAIN–quality improvement database. A detailed descrip- of an automated, computer-based tool that screens people tion of the quality improvement methods is presented in a related for unhealthy drinking behaviors and alcohol use disorders, ff article. The primary outcome was the ATTAIN questionnaire re- and o ers appropriate web-based counseling with minimal 30 sponse rate. Secondary outcomes included the percentage of involvement of health-care personnel. is tool was in- individuals who screened positive for unhealthy drinking be- tegrated into the electronic medical record–patient portal haviors and for alcohol use disorders, respondents’ comfort level interface of the integrated, managed-care health care or- with ATTAIN, and the cost of ATTAIN. ganization (Kaiser Permanente, Southern California). It Results: A total of 301 members were included in the pilot was developed by the Southern California Permanente tests. The ATTAIN response rate was 46%. The prevalence of Medical Group (SCPMG), which serves more than 2.68 unhealthy drinking behaviors and alcohol use disorders was 16% million adult Kaiser Permanente health plan members, who and 5%, respectively. These rates were noninferior to the Healthcare were eligible for unhealthy drinking behavior screening. We Effectiveness Data and Information Set 2019 rates submitted from the same organization for the Unhealthy Alcohol Use Screening and named this tool ATTAIN, AuTomaTed Alcohol misuse Follow-up measure. In the post-ATTAIN paper surveys, 100% agreed INterventions. We planned a series of quality improvement fi to the statement: “I felt comfortable answering [ATTAIN] questions . (QI) pilots to test and re ne ATTAIN so as to optimize the ...” The cost of ATTAIN screening was estimated to be one-tenth the process of screening members for unhealthy drinking be- cost of office-based screening. haviors and alcohol use disorders, and offering brief in- Conclusion: ATTAIN was well accepted by eligible adults, terventions when appropriate. appeared noninferior to office-based screening, and added several Here we describe the outcomes of the initial QI pilots potential advantages in terms of screening for alcohol use disorders done to test the feasibility of screening members for un- and readiness to change. healthy drinking behaviors and alcohol use disorders using the ATTAIN tool. INTRODUCTION Excessive use of alcohol contributes to negative health, METHOD social, and financial outcomes for all age groups.1 us, the Study Type and Procedures  Centers for Medicare and Medicaid Services and the is is a data-only study that reports results from an National Committee for Quality Assurance have recom- existing QI database. It includes all the data in the QI  mended that health-care organizations incorporate alcohol database without exclusion. e institutional review board screening as a quality measure [Healthcare Effectiveness at the institution where the QI project took place reviewed  Data and Information Set (HEDIS)].2-4 Some studies have and approved this study. e specialty practice was a pre- tested digital or computer-based interventions to address ventive medicine/health education-based clinic that re- ceived referrals for health plan members mainly looking to lose weight or improve their metabolic health (not related Author Affiliations to alcohol). 1Loma Linda University Medical Center, Loma Linda, CA 2Southern California Permanente Medical Group, Pasadena, CA QI Workflow fi Corresponding Author Eligible participants for the QI project were identi ed by Mohamed H Ismail, MD, DrPH ([email protected]) scanning upcoming visits for 1 SCPMG provider in a specialty practice (using ATTAIN version 1.0 and, later, Keywords: alcohol misuse, automated screening, HEDIS measures, screening and brief intervention, unhealthy drinking behavior 44 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.143 ORIGINAL RESEARCH ARTICLE The ATTAIN Solution Tested: Initial Pilot Results of an Automated, Web-based Screening Tool for Unhealthy Drinking Behaviors

1.1) and for 1 SCPMG provider in a primary care practice percentage who screened positive for alcohol use disorders; (using ATTAIN version 1.0). any differences in response rates based on the number of Inclusion criteria were age ≥ 18 years and an active ac- days before an appointment that the invitation was sent count with the health plan’s internet-based electronic (specialty clinic only); and the responses to questions re- medical record patient portal (www.kp.org). An invitation garding the interest in cutting down, interest in medication, message with the attached ATTAIN questionnaire link was and the interest in referral to a specialty clinic. We also sent to these member accounts. As with all messages sent from noted demographic data (average age, gender, and lan- the health plan, the message triggered an email to members’ guage), which were recorded in the QI database for the personal email account, alerting them of a care-related message specialty clinic group (but not recorded for the primary on their www.kp.org account. ey could access their account clinic group). on a computer or a portable electronic device. Messages were e ATTAIN questionnaire response rate and the per- sent 1 day, 3 days, 7 days, or 14 days prior to their appointment centage of patients that screened positive for unhealthy date. Members who followed the link in the alert email were drinking behaviors were compared to HEDIS 2019 rates routed to their portal, where they could log in to their account. submitted from the SCPMG to the National Committee Once logged in, they could see the message (as well as a for Quality Assurance for the Unhealthy Alcohol Use 30-second video) describing the purpose behind ATTAIN Screening and Follow-Up measure.4 and asking them to complete the questionnaire by clicking e cost of ATTAIN screening was estimated to be 10 alink.Patients’ responses became a part of their medical cents × the total number of the population. e cost of record and were returned to the sending provider’sin-basket office-based screening was estimated based on the following (a unique electronic medical record inbox for each provider). assumptions and calculations: [(Cost per person assuming 1 minute of nurse time to screen and $60/h cost per nurse) × QI Survey Questions (No. of members screened in the office in 2019)] + [(Cost In ATTAIN version 1.0, the initial question asked, “How per person assuming 2 minutes of physician time and often did you have a drink containing alcohol in the last $180/h cost per clinician) × (No. of members who screened year?” For those who answered “Never,” the questionnaire positive in 2019 and would require further screening by the ended. For all others, 2 additional follow-up questions were physician)].32 asked to screen for unhealthy drinking behaviors relating to We reported descriptive statistics from the QI database drinking beyond the recommended daily and weekly limits. with the responses to the paper survey about the ATTAIN ese questions were age and gender specific. ose who process. is paper survey was given in the office after the screened positive on either of these 2 questions were given specialty clinic pilot (ATTAIN version 1.0) and results were additional questions (based on the Diagnostic and Statistical reported in Excel graphs. We described whether partici- Manual of Mental Disorders, 5th edition criteria) to screen pants reported receiving the email invitation, whether they for alcohol use disorders.31 Patients were also asked whether completed the survey (for those who received it), their they were interested in cutting down (either on their own or frequency of checking their email, whether they felt via help) and whether they were interested in receiving comfortable answering the ATTAIN questions, whether medication or a specialty clinic referral. In ATTAIN ver- they felt they answered the questions honestly, whether they sion 1.1, the survey branching was modified, based on found the survey helpful, whether if they found it easy to National Committee for Quality Assurance feedback, so complete, and whether they felt it was an invasion of privacy. that all participants were asked at least the first 3 questions (the initial question and the 2 additional questions just Statistical Analysis described for version 1.0). e χ2 test was used for analysis, with an alpha of 0.05 or In the specialty clinic pilot (ATTAIN version 1.0), a paper less.33 survey was given to all the invitees at their in-person visit to solicit feedback about their experience with the online survey. RESULTS ATTAIN Total Results: Combined Specialty and Primary Care Clinics Outcome Measures A total of 301 patients met the inclusion criteria for the e primary outcome was the ATTAIN questionnaire ATTAIN pilots (Table 1 and Figure 1). e average age response rate (number of respondents who completed the was 47 years and 73% of the respondents were female. questionnaire/number who were invited). We reviewed the Ninety-five percent preferred English as their first language. existing QI database to determine this. e ATTAIN questionnaire response rate was 46% (137 of e secondary outcomes included the percentage who 301). e 2019 HEDIS reported screening rate by the screened positive for unhealthy drinking behaviors; the SCPMG was 43% (SCPMG unpublished report).

The Permanente Journal·https://doi.org/10.7812/TPP/20.143 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 45 ORIGINAL RESEARCH ARTICLE The ATTAIN Solution Tested: Initial Pilot Results of an Automated, Web-based Screening Tool for Unhealthy Drinking Behaviors

Table 1. Demographic characteristics of Automated Alcohol Misuse Interventions invitees in the primary and specialty clinics (N = 301) Characteristics Value Mean age, y 47 Gender, n (%) Male 80 (27) Female 221 (73) Language, n (%) English 286 (95) Spanish 13 (4.3) Other 2 (0.6)

e prevalence of unhealthy drinking behaviors among the ATTAIN group was 16% (22 of 137). e 2019 HEDIS-reported positive screen rate by the SCPMG was 11.5% (SCPMG unpublished report). e prevalence of alcohol use disorders among ATTAIN responders was 5% (7 of 137). ree participants indicated interest in further help, 9 indicated they would cut back Figure 1. Results flowchart of primary and secondary outcomes. ATTAIN, on their own, and 6 participants indicated they were not Automated Alcohol Misuse Interventions. interested in cutting back.

ATTAIN Results: Specialty Clinic severe alcohol use disorder and was interested in medica- A total of 252 members (213 with ATTAIN version 1.0 tions, learning more about what we can offer, and in a and 39 with ATTAIN version 1.1) met the inclusion referral to a specialist. Seven participants indicated they criteria in the specialty clinic pilots. e average age of would cut back on their own, including 2 of those who participants was 45 years and 80% (201 of 252) were female. screened positive for a moderate alcohol use disorder. ree Ninety-four percent (238 of 252) preferred English as their participants indicated they were not interested in cutting first language. e ATTAIN questionnaire response rate back, including one who screened positive for a mild alcohol was 44% (112 of 252) (Table 2). use disorder. e response rates by messaging timing in reference to During follow-up for those who screened positive (for their upcoming appointment were 48% (32 of 66) for 1 day either unhealthy drinking or an alcohol use disorder), 4 before, 48% (45 of 94) for 3 days before, 33% (23 of 69) for subjects stated they had quit, 2 said they would quit right 1 week before, and 52% (12 of 23) for 2 weeks before away, 3 stated they cut down markedly, and 1 said they (p = 0.18). e response rates by gender were 46% (92 of would cut down to low risk levels. 201) for females and 39% (20 of 51) for males (p = 0.19). Of the 6% (13 of 252) of members in the QI database who ATTAIN Results: Primary Care reported Spanish rather than English as their preferred Forty-nine members (ATTAIN version 1.0) met the language, 15% (2 of 13) completed the ATTAIN ques- inclusion criteria in the primary care clinic pilot. e average tionnaire survey and both screened negative for unhealthy age of participants was 55 years and 40% (20 of 49) were drinking behaviors (p = 0.29). female. Forty-eight listed English and 1 listed either En- Among those who completed the survey, the prevalence glish or Spanish as their first language. e ATTAIN of unhealthy drinking behaviors and alcohol use disorders response rate was 51% (25 of 49). e response rates by was 11.6% (13 of 112) and 3.5% (4 of 112), respectively. e gender were 45% (9 of 20) for females and 55% (16 of 29) breakdown by gender for unhealthy drinking behaviors was for males (p = 0.7). 10% (9 of 92) for females and 20% (4 of 20) for males. Among those who completed the survey, the prevalence Overall, 2 participants indicated interest in further help. of unhealthy drinking behaviors and alcohol use disorders One of these screened positive for unhealthy drinking only was 36% (9 of 25) and 12% (3 of 25), respectively. e and was interested in medications and in learning more breakdown by gender for unhealthy drinking behaviors was about what we can offer. e other screened positive for a 33% (3 of 9) for females (2 of whom did not complete the

46 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.143 ORIGINAL RESEARCH ARTICLE The ATTAIN Solution Tested: Initial Pilot Results of an Automated, Web-based Screening Tool for Unhealthy Drinking Behaviors

Table 2. Gender, language, and message timing in respondents vs nonrespondents in the specialty clinic (N = 252) Variable Responded to survey, n (%) Did not responded to survey, n (%) p Value Gender 0.19 Male 92 (45) 109 (55) Female 20 (39) 31 (61) Preferred language 0.029a English 110 (46.2) 128 (53.7) Spanish 2 (15.4) 11 (84.6) Messaging timing 0.18 1 d before 32 (48) 34 (52) 3 d before 45 (48) 49 (52) 1 wk before 23 (33) 46 (66.6) 2 wk before 12 (52) 11 (48) aSignificant.

secondary questions for alcohol use disorders) and 37% (6 of Table 3. Automated Alcohol Misuse Interventions feedback 16) for males. survey in the specialty clinic (N = 36) Overall, 1 participant was interested in learning more Questions n (%) about what we can offer. Two participants (1 screened Received email survey? positive for unhealthy drinking and another for an alcohol Yes 28 (77.8) use disorder) indicated they would cut back on their own. No 8 (22.2)  ree participants indicated they were not interested in Survey completed? cutting back, and 2 did not complete that part of the Yes 21 (75.0) questionnaire. No 7 (25.0) How often do you check your email? Participant Perception Survey Results about the ATTAIN Process: Daily 21 (70.0) Specialty Clinic Every 2–3 d 1 (3.3)  irty-six participants (of the 213 from ATTAIN version Weekly 4 (13.3) 1.0) completed paper surveys about their perception of the > Weekly 4 (13.3) ATTAIN questionnaire process (Table 3 and Figure 2). Seventy-eight (28 of 36) indicated they received the email invitation to do the survey and, of those, 75% (21 of 28) indicated they completed it. Of 30 participants who an- swered questions about their email use, 73% (22 of 30) said they check email at least every 3 days and 27% (8 of 30) said they check their email once a week or less frequently. Of those who completed the survey, 100% (21 of 21) agreed to the statement “I felt comfortable answering questions on the survey,” 57% (12 of 21) agreed to the statement “I found the survey to be helpful,” 95% (20 of 21) agreed to the statements “It was easy to complete the survey” and “I answered the questions honestly,” and 5% (1 of 21) agreed to the statement “I felt like this survey was an invasion of Figure 2. Participants’ perceptions of Automated Alcohol Misuse Interventions privacy.” (ATTAIN) in the specialty clinic (N = 21). NA, not applicable.

Cost Estimates of Screening DISCUSSION e cost of screening via ATTAIN is estimated to be 10 e burden of disease from alcohol in the US is very high. cents per person whereas the costs of screening done in the Although the US Preventive Services Task Force concludes traditional primary care office is estimated to be about $1 with moderate certainty that screening and brief behav- (assuming 1 minute of nurse time to screen and $60/h cost ioral counseling interventions for unhealthy alcohol use is per nurse). of moderate net benefit, many physicians do not perform

The Permanente Journal·https://doi.org/10.7812/TPP/20.143 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 47 ORIGINAL RESEARCH ARTICLE The ATTAIN Solution Tested: Initial Pilot Results of an Automated, Web-based Screening Tool for Unhealthy Drinking Behaviors

this screening consistently.34 Some reported barriers to screening in an integrated health-care organization. In screening include time constraints and a lack of knowledge addition, it added several potential advantages in terms of about best practices.34 screening for alcohol use disorders and for indications of ATTAIN addresses these barriers, providing a cost- readiness to change. Further testing of ATTAIN on a larger effective way to screen and counsel adults. e rate sample of the population and with integrated automated achieved by ATTAIN in screening for unhealthy drinking counseling is needed. v behaviors (the primary outcome of this study) was non- inferior to the rate of traditional primary care office Disclosure Statement The authors have no conflicts of interest to disclose. screening for this condition reported in the health plan’s  HEDIS measure submissions. e lower cost of ATTAIN, Authors’ Contributions estimated to be one-tenth the cost of office screening, makes Mohamed H Ismail, MD, DrPH, participated in the study design, data collection, this a particularly encouraging result. e secondary out- statistical analysis, critical review, drafting of the final manuscript, and submission comes suggested that ATTAIN 1) might detect more of the final manuscript. Jiseung Yoon, MD, participated in the study design, data ffi collection, statistical analysis, drafting of and critical review of the final manuscript. unhealthy drinkers compared to o ce-based screening, Emma Fredua, MPH, CHES, and Shahriar Davari, MD, participated in the study 2) included further screening for alcohol use disorders, 3) intervention, data collection, and critical review of the final manuscript. All authors screened people for interest in medication or specialty have given final approval to the manuscript. services to help reducing drinking, 4) performed well in both the primary and specialty care settings, and 5) was well ac- Funding The authors did not receive funding for this work. cepted by the target population. Furthermore, internal pre- sentations and discussions of the data have led to an interest in Related Article expanding the process to automate other screenings (eg, for Unhealthy Drinking Behavior and the ATTAIN Solution: Web-based, Automated drug abuse, tobacco use, and obesity-related concerns), thus Alcohol Misuse Interventions expanding ATTAIN’s potential beyond alcohol use. References 1. National Institute on Alcohol Abuse and Alcoholism. Alcohol facts and statistics (updated Limitations March 2021). Accessed February 6, 2018. www.niaaa.nih.gov/alcohol-health/overview- ese pilots tested only the first goal of ATTAIN: to alcohol-consumption/alcohol-facts-and-statistics 2. U.S. Preventive Services Task Force. Final recommendation statement. May 15, 2013 screen. ey did not include integration of automated Accessed February 6, 2018. www.uspreventiveservicestaskforce.org/Page/Document/ counseling (goal 2). Second, these pilots relied on subjects RecommendationStatementFinal/alcohol-misuse-screening-and-behavioral-counseling- ’ ffi interventions-in-primary-care with upcoming appointments to a doctor so ce. It is not 3. National Institute on Alcohol Abuse and Alcoholism. Alcohol use disorder (Updated known whether ATTAIN would work similarly if sent to December 2020). Accessed February 6, 2018. www.niaaa.nih.gov/alcohol-health/ overview-alcohol-consumption/alcohol-use-disorders random health plan members (an important option because 4. National Committee on Quality Assurance. HEDIS® measure: Unhealthy alcohol use not all eligible adults see a provider each year). It is also not screening and follow-up. Accessed March 9, 2021. www.ncqa.org/hedis/reports-and- known whether it can be replicated in less integrated health- research/hedis-measure-unhealthy-alcohol-use-screening-and-follow-up/ 5. Freyer-Adam J, Baumann S, Haberecht K, et al. In-person alcohol counseling versus care organizations. Also, 6% of the participants in the QI computer-generated feedback: Results from a randomized controlled trial. Health Psychol database indicated Spanish rather than English as their 2018 Jan;37(1):70–80. DOI: https://doi.org/10.1037/hea0000556, PMID:28967769. 6. Baumann S, Gaertner B, Haberecht K, Bischof G, John U, Freyer-Adam J. How alcohol preferred language, but still received the (English) ATTAIN use problem severity affects the outcome of brief intervention delivered in-person versus questionnaire (a Spanish version was not available then). It is through computer-generated feedback letters. Drug Alcohol Depend 2018 Feb;183:82–8. ff DOI: https://doi.org/10.1016/j.drugalcdep.2017.10.032, PMID:29241105. not known whether a language barrier may have a ected the 7. Cadigan JM, Haeny AM, Martens MP, Weaver CC, Takamatsu SK, Arterberry BJ. accuracy of these member’s responses as well as their low Personalized drinking feedback: A meta-analysis of in-person versus computer-delivered interventions. J Consult Clin Psychol 2015 Apr;83(2):430–7. DOI: https://doi.org/10.1037/ response rate. In addition, these results only prove ATTAIN a0038394, PMID:25486373. as a supplement to office-based screening. If it was to replace 8. Carey KB, Scott-Sheldon LA, Elliott JC, Garey L, Carey MP. Face-to-face versus ffi computer-delivered alcohol interventions for college drinkers: A meta-analytic review, o ce-based screening adequately, it should yield a completed 1998 to 2010. Clin Psychol Rev 2012 Dec;32(8):69–703. DOI: https://doi.org/10.1016/j. questionnaire rate much greater than 44%. Last, the small cpr.2012.08.001, PMID:23022767. sample size in the ATTAIN pilots limited its power in terms 9. Freyer-Adam J, Baumann S, Haberecht K, et al. In-person and computer-based alcohol interventions at general hospitals: Reach and retention. Eur J Public Health 2016 Oct; of comparison to the control group. Testing ATTAIN on a 26(5):844–9. DOI: https://doi.org/10.1093/eurpub/ckv238, PMID:26748101. larger sample can show whether it is more accurate in 10. Khadjesari Z, Murray E, Hewitt C, Hartley S, Godfrey C. Can stand-alone computer-based interventions reduce alcohol consumption? A systematic review. Addiction 2011 Feb; detecting those with unhealthy drinking behaviors. 106(2):267–82. DOI: https://doi.org/10.1111/j.1360-0443.2010.03214.x, PMID:21083832. 11. Nair NK, Newton NC, Shakeshaft A, Wallace P, Teesson M. A systematic review of digital and computer-based alcohol intervention programs in primary care. Curr Drug Abuse Rev CONCLUSION 2015 Sep;8(2):111–8. DOI: https://doi.org/10.2174/1874473708666150916113538, ATTAIN, an automated, web-based screening tool for PMID:26373848. , 12. Sundstrom¨ C, Blankers M, Khadjesari Z. Computer-based interventions for problematic unhealthy drinking behaviors was well accepted by eligible alcohol use: A review of systematic reviews. Int J Behav Med 2017 Oct;24(5):646–58. adults and proved noninferior to traditional office-based DOI: https://doi.org/10.1007/s12529-016-9601-8

48 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.143 ORIGINAL RESEARCH ARTICLE The ATTAIN Solution Tested: Initial Pilot Results of an Automated, Web-based Screening Tool for Unhealthy Drinking Behaviors

13. Bewick BM, Trusler K, Barkham M, Hill AJ, Cahill J, Mulhern B. The effectiveness of web- 24. Mullen J, Ryan SR, Mathias CW, Dougherty DM. Feasibility of a computer-assisted based interventions designed to decrease alcohol consumption: A systematic review. alcohol screening, brief intervention and referral to treatment program for DWI Prev Med 2008 Jul;47(1):17–26. DOI: https://doi.org/10.1016/j.ypmed.2008.01.005, offenders. Addict Sci Clin Pract 2015 Dec;10(1):25. DOI: https://doi.org/10.1186/ PMID:18302970. s13722-015-0046-1 14. Rooke S, Thorsteinsson E, Karpin A, Copeland J, Allsop D. Computer-delivered 25. Merrill JE, Wardell JD, Read JP. Is readiness to change drinking related to interventions for alcohol and tobacco use: A meta-analysis. Addiction 2010 Aug;105(8): reductions in alcohol use and consequences? A week-to-week analysis. J Stud 1381–90. DOI: https://doi.org/10.1111/j.1360-0443.2010.02975.x, PMID:20528806. Alcohol Drugs 2015 Sep;76(5):790–8. DOI: https://doi.org/10.15288/jsad.2015.76. 15. Kaner EF, Beyer FR, Garnett C, et al. Personalised digital interventions for reducing 790, PMID:26402360. hazardous and harmful alcohol consumption in community-dwelling populations. 26. Lettow BV, Vries HD, Burdorf A, Boon B, Empelen PV. Drinker prototype alteration and Cochrane Database Syst Rev 2017 Sep;9(9):CD011479. DOI: https://doi.org/10.1002/ cue reminders as strategies in a tailored web-based intervention reducing adults’ alcohol 14651858.cd011479.pub2 consumption: Randomized controlled trial. J Med Internet Res 2015 Feb;17(2):e35. DOI: 16. Guillemont J, Cogordan C, Nalpas B, Nguyen-Thanh V, Richard JB, Arwidson P. https://doi.org/10.2196/jmir.3551 Effectiveness of a web-based intervention to reduce alcohol consumption among French 27. Sinadinovic K, Wennberg P, Johansson M, Berman AH. Targeting individuals with hazardous drinkers: A randomized controlled trial. Health Educ Res 2017 Aug;32(4): problematic alcohol use via web-based cognitive–behavioral self-help modules, 332–42. DOI: https://doi.org/10.1093/her/cyx052, PMID:28854571. personalized screening feedback or assessment only: A randomized controlled trial. Eur 17. Brendryen H, Johansen A, Duckert F, Nesvag˚ S. A pilot randomized controlled trial Addiction Res 2014 Oct;20(6):305–18. DOI: https://doi.org/10.1159/000362406, PMID: of an internet-based alcohol intervention in a workplace setting. Int J Behav Med 25300885. 2017 Oct;24(5):768–77. DOI: https://doi.org/10.1007/s12529-017-9665-0,PMID: 28. Johnson NA, Kypri K, Attia J. Development of an electronic alcohol screening and brief 28755326. intervention program for hospital outpatients with unhealthy alcohol use. JMIR Res Protoc 18. Baumann S, Gaertner B, Haberecht K, Bischof G, John U, Freyer-Adam J. Who benefits 2013 Sep;2(2):e36. DOI: https://doi.org/10.2196/resprot.2697, PMID:24055787. from computer-based brief alcohol intervention? Day-to-day drinking patterns as a 29. Schulz DN, Candel MJ, Kremers SP, Reinwand DA, Jander A, de Vries H. Effects of a moderator of intervention efficacy. Drug Alcohol Depend 2017 Jun;175:119–26. DOI: web-based tailored intervention to reduce alcohol consumption in adults: Randomized https://doi.org/10.1016/j.drugalcdep.2017.01.040, PMID:28412302. controlled trial. J Med Internet Res 2013 Sep;15(9):e206. DOI: https://doi.org/10.2196/ 19. Pedersen ER, Parast L, Marshall GN, Schell TL, Neighbors C. A randomized controlled jmir.2568, PMID:24045005. trial of a web-based, personalized normative feedback alcohol intervention for young-adult 30. Chevinsky J, Fredua E, Vasquez EM, Ismail MH. Unhealthy drinking behavior and the veterans. J Consult Clin Psychol 2017 May;85(5):459–70. DOI: https://doi.org/10.1037/ ATTAIN solution: Web-based, AuTomaTed alcohol misuse interventions: An ccp0000187, PMID:28287799. automated, web-based screening and brief intervention tool for unhealth drinking 20. Johansson M, Sinadinovic K, Hammarberg A, et al. Web-based self-help for problematic behavior in an integrated health-care organization. Perm J 2021;25:20.141. DOI: alcohol use: A large naturalistic study. Int J Behav Med 2017 Oct;24(5):749–59. DOI: https://doi.org/10.7812/TPP/20.141 https://doi.org/10.1007/s12529-016-9618-z 31. American Psychiatric Association. (2013). Diagnostic and statistical manual of mental 21. Tebb KP, Erenrich RK, Jasik CB, Berna MS, Lester JC, Ozer EM. Use of theory in disorders (5th ed.). https://doi.org/10.1176/appi.books.9780890425596 computer-based interventions to reduce alcohol use among adolescents and young 32. U.S. Prevention Services Task Force. Specific recommendation: Unhealthy alcohol use in adults: A systematic review. BMC Public Health 2016 Jun;16(1):517. DOI: https://doi.org/ adolescents and adults: Screening and behavioral counseling interventions: Adults 18 10.1186/s12889-016-3183-x, PMID:27317330. years or older, including pregnant women. Agency for Healthcare Research and Quality. 22. Pedersen ER, Marshall GN, Schell TL. Study protocol for a web-based personalized https://epss.ahrq.gov/ePSS/RecomDetail.do?method=rades&tab=0&sid=370 normative feedback alcohol intervention for young adult veterans. Addict Sci Clin Pract 33. Preacher, KJ. Calculation for the chi-square test: An interactive calculation tool for chi- 2016 Dec;11(1):6. DOI: https://doi.org/10.1186/s13722-016-0055-8 square tests of goodness of fit and independence [Computer software]; April 2001. 23. Steers ML, Coffman AD, Wickham RE, Bryan JL, Caraway L, Neighbors C. Evaluation of Accessed March 1, 2020. http://quantpsy.org/chisq/chisq.htm alcohol-related personalized normative feedback with and without an injunctive message. 34. Bray JW, Zarkin GA, Hinde JM, Mills MJ. Costs of alcohol screening and brief intervention J Stud Alcohol Drugs 2016 Mar;77(2):337–42. DOI: https://doi.org/10.15288/jsad.2016. in medical settings: A review of the literature. J Stud Alcohol Drugs 2012 Nov;73(6):911–9. 77.337, PMID:26997192. DOI: https://doi.org/10.15288/jsad.2012.73.911, PMID:23036208

The Permanente Journal·https://doi.org/10.7812/TPP/20.143 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 49 n ORIGINAL RESEARCH ARTICLE Insurance Barriers, Gendering, and Access: Interviews with Central North Carolinian Women About Their Health Care Experiences

Lena JP Cardoso, MPP1; Anna Gassman-Pines, PhD1; Nathan A Boucher, DrPH, PA, MS, MPA, CPHQ1,2,3,4 Perm J 2021;25:20.176 E-pub: 03/01/2021 https://doi.org/10.7812/TPP/20.176

ABSTRACT are attributed to women’s longer life expectancy, in- Background: Women face unique logistical and financial creased morbidity, and health costs related to maternity barriers to health care access. They also have higher health care care and menopause.4-6 Women utilize preventative care expenditures and higher rates of morbidity. Women’s experiences at substantially higher rates than men,7 with one 2005 US while utilizing health care are historically less well researched and study finding that women on average had a preventative warrant exploration. care visit rate of 74.4 visits per 100 persons compared to Methods: We conducted 14 semistructured interviews about 44.8 per 100 persons for men.8 women’s health care experiences with 11 women health con- sumers and 3 women health care practitioners in central North While women in the US have longer life expectancies 9 Carolina. than men (81.1 compared to 76.3 years ), women face 10-14 Results: When discussing their experiences, participants noted higher rates of chronic illness and morbidity. In scheduling challenges, barriers related to insurance and cost, 2015, 38% of women suffered from 1 or more chronic and dismissive or negative in-person encounters. Participants illnesses compared to 30% of men.11 Women are more frequently discussed lack of resources for care postpartum. likely to experience depression and anxiety12 and report Practitioners noted lack of knowledge of disease burden, over- higher rates of chronic stress and minor daily stress.13 medicalization of women’s care, lack of care postpartum, and Women are also more likely to experience domestic, trends around changes in primary care.Women health consumers physical, and sexual violence, which can lead to adverse in this study faced challenges related to access and in-person mental and physical health.15 experience of care delivery, which were echoed by the clinician ere are well-documented gender discrepancies in health interviewees. Barriers to optimal women’s health care exist even care costs and insurance coverage. Prior to the 2010 imple- for those with insurance coverage and point to systemwide ff constraints as well as deficits in organizational culture. mentation of the A ordable Care Act, women were estimated Conclusion: Future clinical and research efforts should include to be paying between 50% and 80% more for monthly 16, 17 1) increasing awareness of and facilitating access to affordable premiums than men on the individual market. Despite postpartum care, 2) easing burdens around scheduling appointments the Affordable Care Act’s 2017 mandate that preventative and improved care coordination, and 3) more research exploring care services be included at no cost, many women with women’s experiences during in-person health care encounters. insurance reported paying out of pocket for critical preven- Concerns and barriers that women described may be due to tative screening examinations, including pap smears (20%), systems-level requirements and constraints. mammograms (13%), and colonoscopies (7%).1 Although previous research has identified gendered dif- INTRODUCTION ferences in health care and health outcomes, less is known about women’s experiences and feelings about their inter- Women face unique obstacles when seeking health care, fi including logistical barriers, often due to women’s roles as actions with the health system. Research speci cally exam- fi ining women’s experiences has focused on narrow subsets caregivers, and nancial barriers, as women on average earn 18 less than men.1,2 In 2010, women on average spent $7860 of women, such as lesbian women in rural settings, lesbian women becoming mothers,19 Somali immigrant women,20 per capita on health care costs compared to $6313 per capita 21 for men.3 Many of the differences in health expenditures deaf women, and women who have experienced intimate partner violence.22 While research on gender discrepancies exists, there is Author Affiliations less qualitative research on women’s experiences, espe- 1Duke University Sanford School of Public Policy, Durham, NC  2Center of Innovation to Accelerate Discovery and Practice Transformation, Durham Veterans Affairs Health cially outside of narrow subgroups of women. is study System Health Services Research and Development, Durham, NC provides exploratory research to help address the gap in 3Duke-Margolis Center for Health Policy, Durham, NC qualitative analysis around how women experience health 4Departments of Medicine (Geriatrics) and Population Health Sciences, Duke University School of Medicine, care encounters, approach their general health care, and Durham, NC advocate for their health care needs and to gain under- Corresponding Author standing into barriers that keep women from receiving Nathan A Boucher, DrPH, PA, MS, MPA, CPHQ ([email protected]) optimal care.

Keywords: gender, health care delivery, insurance, primary care, women 50 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.176 ORIGINAL RESEARCH ARTICLE Insurance Barriers, Gendering, and Access: Interviews with Central North Carolinian Women About Their Health Care Experiences

METHODS began early, with the first interview. e study team dis- Recruitment Method for Women Participants continued the data collection once thematic saturation was Following Duke University Institutional Review Board reached and no further new ideas were being seen in the approval, participants were recruited using convenience and data. Audio recordings were transcribed verbatim using the snowball sampling methods in Durham, Wake, and Orange Temi audio transcription service (www.temi.com, 2019). counties of central North Carolina. Snowball sampling is NVivo 12 Pro software (QSR International Pty Ltd., 2018) a recruitment method in which participants are asked to was used to manage the transcription data and to develop identify other potential participants withing their network.23 codes to segment participants’ narratives into conceptual e research team made contact via phone or email after categories (eg, all text describing a similar concept). Content- confirming participants’ interest and willingness to partic- driven emergent codes were identified and applied to the text ipate. Participants completed a phone screening to confirm for each of the conceptual categories (eg, potential themes they met the following study inclusion criteria: ability to related to responsibilities as caregivers). ree interview complete a one-time interview in English, willingness to transcriptions were randomly selected and the 2 coders be audio recorded, age 18 years or older, self-identified as separately identified themes. After reconciling on emer- a woman, and residing in Durham, Wake, or Orange gent themes, researchers separately coded the next 3 in- counties. Upon completion of the interview, participants terviews. Application of coding was then discussed for received an electronic $15 gift card to honor their time. A those 3 interviews. total of 11 women were screened; all 11 women were Discrepancies in coding were resolved through iterative interviewed. discussions between researchers. e initial 3 transcripts used to identify themes were recoded, and the codebook Recruitment Method for Practitioners was revised accordingly. Coding was then completed on Practitioners were identified through colleagues and the remaining 8 transcripts. Code frequencies were ex- internet searches and were contacted directly via email with amined across transcripts to identify salient factors for institutional review board approval. Six practitioners were participants. For example, when coding for content around contacted; 3 were interviewed. Practitioners did not receive exploration of gender, coders looked for explicit mentions compensation. of participants’ gender affecting their care, mentions of health concerns that mainly affect women such as birth Data Collection control or pregnancy, and mentions of caregiver duties Participants engaged in an initial phone screening, an affecting care or access. To conclude analysis, illustrative informed consent, a demographic survey, and an hour-long, quotations were taken and used to demonstrate common in-person audio-recorded interview. e semistructured themes. Women’sthoughtswerereconciledwiththe interview guide was designed by the research team through perspectives given by the 3 practitioners interviewed. an iterative process, based on previous research, and in- Since there were only 3 practitioner interviews, content cluded questions pertaining to the following: daily stress, analysis was more simplistic. One open-ended question was recent health care experiences, advocacy, social and sup- asked exploring each of the following: observed gendering port networks, motherhood, health care during and after in care delivery (defined as treatment specifictostereo- pregnancy, preventative care experiences, and health care typical gender norms25), women as caregivers and that utilization. impact on their health, observed stressors for women, e research activities were similar for the 3 practitioners, gender rating by insurance providers, and observed except that the practitioners only completed an informed barriers to care for women. consent and semistructured interview. An audio recording was taken for only 1 of the 3 interviews due to background RESULTS noise in some of the available interview locations. esem- Table 1 presents participants’ demographic characteris- istructured interview was designed to obtain practitioners’ tics. Of the 11 women interviewed, 7 had a master’s degree perspectives on questions asked of the participants and or higher. e 4 other women had a bachelor’s degree. e covered opinions on women’s health care, gender discrimi- educational attainment for the study sample is higher than nation, postpartum care, and insurance practices. the average for the tricounty area from which the sample was drawn, in which 26% of the population has a bachelor’s Data Analysis degree or higher and 17% has some postcollege education.26 Data were analyzed using a 2-coder, descriptive content With an average age of 34 years (range, 25-52), the women analysis with an inductive approach24 relying on what the interviewed were slightly younger than the average age of participants discussed and not using a priori themes. Analysis 37 years for the tricounty area. All of the women had

The Permanente Journal·https://doi.org/10.7812/TPP/20.176 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 51 ORIGINAL RESEARCH ARTICLE Insurance Barriers, Gendering, and Access: Interviews with Central North Carolinian Women About Their Health Care Experiences

Table 1. Characteristics of women participants (n = 11) Table 2. Themes and subthemes Characteristic Frequency Percent Theme Frequency Percent Age, y Barriers to an optimal encounter 20-29 3 27 Scheduling logistics 8 73 30-39 5 45 Insurance/cost barriers 10 91 40-49 2 18 Dismissive or negative in-person encounters 8 73 50-59 1 9 Care postpartum 4 80a Insurance Exploration of gender 11 100 Blue Cross Blue Shield 5 45 a.Indicates the percentage who mentioned this theme among mothers. United HealthCare 3 27 Tricare 1 9 mentioned as a barrier by 73% of participants. Difficulties Aetna 1 9 and confusion around scheduling resulted in switching Medicaid 1 9 practitioners, delays in seeking care, decreased engagement Insurance through employer (yes) 9 82 in preventative care, or forgoing care altogether. Regular care provided (yes) 7 64 e few [small] attempts I’ve made have been just a little Mother (yes) 5 45 frustrating trying to find someone that accepts new patients Educational attainment with my health insurance. I live in Durham, work in High school 0 0 Raleigh, so trying to figure out where I should go.… And so, I Bachelor’s degree 4 36 just haven’t found a general practitioner. Like when I’ve Master’s degree or higher 7 64 tried before, I don’t know if I was making the right choice and Hospitalization in the past year (yes) 2 18 then I was getting conflicting information from what I saw online. (Patient in late 20s) Insurance/Cost Barriers insurance. Insurance providers included Blue Cross Blue Insurance barriers and costs were mentioned by 91% of Shield,UnitedHealthCare,Tricare,andAetna.Twowomen the participants. Insurance costs affected medication choices had Medicaid or pregnancy Medicaid. Most women had a and resulted in delays in seeking necessarily or recom- regular care practitioner (63%) and about half of the women mended care. interviewed had been pregnant and are mothers (45%). I had birth control. I used [contraception ring] … I had e 3 practitioners were all women physicians, including been using it and then at a certain point it no longer allows 2 current medical residents. While all 3 practitioners are you to pick it up at your local pharmacist, … you had to do it internists, 2 practice primary care while 1 plans to focus through mail. However, because I travel so much for work, in cardiology. Two of the 3 were specifically focused on the [contraception ring] has to be refrigerated, so it would be women’s care issues. All 3 practitioners have large segments dropped off at my door. And if I am on a trip for 3 days or of their patient panels who use Medicaid. something, it spoils after 24 hours. So, it kept being sent to my Salient themes that emerged across the participant in- door when I was away despite me telling them that I was not terviews (Table 2) can be separated into the following able to have this service. If it was at a pharmacist, it would categories: barriers to an optimal clinical encounter, care have cost $175. If it had been sent to my house it would be postpartum, and exploration of gender. free, but that just wasn’t an option. So, I ended up just going off of [contraception ring] because of the stress.… [It] probably Theme 1: Barriers to an Optimal Clinical Encounter took me like 5 to 10 phone calls between my provider, In response to questions around a recent health care [pharmacy], and my insurance provider … a huge pain. experience, a time they had to advocate regarding their care, (Patient in mid-20s) and a time they had delayed seeking care, every woman What’s a barrier now, particularly with arthritis. is is interviewed discussed barriers to either accessing care or hard because physical therapy they don’t cover very much. And receiving care that left them feeling upset or frustrated. so that really would be what they would recommend as kind of Barriers can be further segmented into 3 categories: my long term … go into physical therapy. And you know, the scheduling difficulties, insurance barriers and costs, and copays, they’re horrible. ey’re over $50 and they want you frustrating or dismissive in-person encounters. to go 2 or 3 times a week. (Patient in early 50s) Scheduling Logistics Dismissive or Negative In-Person Encounters e scheduling process, including general logistics, practi- Most women (73%) mentioned in-person encounters where tioner turnover, and timing due to insurance restrictions, was they felt dismissed or unheard. ese negative encounters

52 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.176 ORIGINAL RESEARCH ARTICLE Insurance Barriers, Gendering, and Access: Interviews with Central North Carolinian Women About Their Health Care Experiences

resulted in diminished patient trust, caused patients to where women felt patronized or unheard in appointments. switch practitioners, and affected some women’soverall Every woman interviewed had experiences that fit into this interaction with the health care system. category. … And while we were there, I had an additional lump So when the endocrinologist was like, this is something that [breast] that I wanted to be checked out and one of the doctors we see really frequently in women and not so much in men just said like, “Oh, it’s probably because you bumped yourself.” and we’re not really sure why, that makes me think like, well And I was like, I never had that happen before.… e a lack of investment in researching medical conditions that supervising doctor came in and I decided to ask again and affect women more than men, which is just like a frustrating asked if I could also get another lump [breast] x-rayed … but part of the patriarchy, but even more insidious is … is it it was like mental gymnastics to ask a third time to get that because it’s like on the throat and there’s generations of [a] checked out. (Patient in late 20s) pattern of women’s voices not being heard in the world. And … When I was in grad school I didn’t have a primary care so, there’s literately a big obstruction growing around my physician and I was like I have insurance why don’t I go see a throat for no reason as like inherited trauma. (Patient in doctor and have a wellness check, get my cholesterol checked … early 30s) and I felt like the guy was so patronizing and he’s like, “You I already felt dumb as a 23-year-old asking for birth do triathlons? at’s really unhealthy for your body.” And I control for the first time. I’m kind of covering up my reasons was like what are you taking about? Like I’m doing the just why I hadn’t gotten it before. And then just to be told like, above the minimum recommendations. You don’t know who I “No, why are you even asking us about that?” Like that was am. (Patient in early 30s) the tone I was getting.… I didn’tknowifit’s because I am a woman or because I was asking dumb questions or maybe I Theme 2: Care Postpartum wasn’t asking dumb questions.… But I felt like this didn’t For the 5 participants who are mothers, when asked about make me feel good and I want to be free to ask these questions their health throughout pregnancy and their approach to and it’s not my fault that I don’t know these things. And so, I health since becoming a mother, 80% mentioned frustra- don’t want to be made to feel dumb. Even if a lot of it is me tions around the care they received postpartum. is in- projecting, I can find someone who can explain things nicer. cluded not feeling taken seriously by medical practitioners, (Patient in late 20s) difficulty in accessing care with an infant, and being unsure I think people are very quick to dismiss women of a certain of symptoms of common health concerns postpartum. age as, oh, hey, you’re just premenopausal. You’re getting fat. I think the pediatricians were always more like, let’s make You’re not, you know 22, and, or they assume that you’re sure that the girl’s healthy and as long as she’s [baby] healthy, eating poorly, you know, as opposed to your body is just not let’s help mom focus on that. But not really ask what else is metabolizing properly. (Patient in early 40s) going on with mom. (Patient in early 30s) I was seeing a physical therapist for pelvic floor issues and Practitioner Perspectives she actually just left the [health care system] … so now I’d Reported practitioner perspectives are based on notes have to get re-established with someone else.… at’s going from in-depth interviews. Practitioner discussions centered to fall to the bottom of my list because I just don’t have that around knowledge of disease burden, medicalization and luxury of dropping him off at daycare or depending on a sister coordination of women’s care, lack of care postpartum, and or someone that could come over and watch him. And it’s just, trends in primary care. All 3 practitioners interviewed for it’s just going to fall to the bottom. I mean, you feel like you’re this study had large panels of Medicaid patients, which may going to the spa just to get your basic medical care needs. not be representative and may affect their perspectives. (Patient in early 40s) Knowledge of Disease Burden Two practitioners noted that cardiovascular disease is the Theme 3: Exploration of Gender number 1 killer of women in America, yet cardiovascular is category captures general impressions and aspects of disease is not discussed as much in the general women’s care that feel inherently gendered. Responses in this cat- population. Focus often centers around issues that on the egory came from questions around a recent health care surface seem more specific to women, like breast cancer. experience (within the past year), a time they have been is is an example of a knowledge gap between the general concerned about their health, a time they had to advocate population of women and what practitioners consider major regarding their care, and if they ever felt they were treated issues. differently because of their gender. ese instances captured Knowledge also pertains to where we prioritize training women’s feelings regarding poor practitioner rapport when and research. One practitioner noted the lack of training dealing with typically female health issues and instances when talking to women about domestic violence or sexual

The Permanente Journal·https://doi.org/10.7812/TPP/20.176 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 53 ORIGINAL RESEARCH ARTICLE Insurance Barriers, Gendering, and Access: Interviews with Central North Carolinian Women About Their Health Care Experiences

assault. She specifically mentioned feeling there is a lack of solely for continued reproductive health questions, high- clear protocol for next steps should a woman present with lighting concerns addressed by practitioners around how that type of trauma. women interact with the medical system frequently but not Medicalization and Coordination of Women’s Care optimally. Two practitioners discussed how women inherently have Participants also echoed what was discussed by practi- more contact with the health system compared to men, tioners regarding insufficient postpartum care and the need mainly due to contraceptive and reproductive health from for better care coordination. When discussing postpartum a young age. Women are also subject to more frequent care, participants noted feeling dismissed when bringing up preventative screenings. Practitioners discussed how increased concerns in encounters or they had issues accessing care contact can be positive as it frequently brings women in for altogether mainly due to their new roles as mothers. e care, but they emphasized that increased contact can also feelings of dismissal and difficulties with access noted by burden patients. For example, contraceptive needs, such as participants may point to larger systems issues and highlight an intrauterine device, require separate appointments or a the importance of care coordination. When access is dif- visit with a subspecialist, which logistically can be chal- ficult, and time is a barrier, it may be difficult for patients to lenging and often a more expensive copay. Practitioners seek out the best type of practitioner for their specific needs. emphasized that if we are going to ask more of patients is idea is captured well in the example of when a new (another appointment and copay), we need to make the mother was in a pediatric appointment and felt dismissed logistics of coordinating this separate visit easier on patients. when she brought up her struggles adapting to motherhood. Care Postpartum e pediatrician probably did not have the expertise to deal While 2 practitioners noted concerns over care post- with the mother’s issues, which is possibly why she said they partum, 1 practitioner discussed how she feels that as a should focus on the child in that encounter. Recognizing in system we care about women only up until they have that type of encounter that the mother needs a referral to a children. Another echoed these sentiments, discussing how different type of practitioner takes increased care coordi- she feels opportunities and resources for prenatal care are nation and improvement in transitions of care, which can be abundant but that care for mothers diminishes once the difficult to implement. baby is born. More broadly, participants also reported feeling rushed Primary Care and dismissed in appointments, which relates to concerns Practitioners discussed some noticeable trends in primary noted by practitioners around increased demands in well- care and wellness visits. First, practitioners noted that ness visits. One participant specifically noted feeling like more women are utilizing obstetrics and gynecology as their their practitioner was going through a checklist in their primary care. One practitioner noted that for young, healthy visit and that if her concern did not fit nicely into one of women with few risk factors, seeing an OB/GYN as their the checkboxes, then it was not addressed or considered primary care practitioner is fine, but it is possible that other important. is example demonstrates how the burden on concerns, like hypertension or diabetes management, can go practitioners for increased screening and documentation is without sufficient treatment if women replace primary care being passed down to patients, affecting the care they re- altogether. Practitioners also discussed the increased system ceive and their comfort with practitioners. burden in wellness visits, citing increased documentation as the main culprit. One practitioner indicated that thorough DISCUSSION documentation is currently both the best and worst thing is study sought to broaden our understanding around about care. women’s experiences with health care. Specifically, this study examined how women experience in-person health Overlap Between Women Practitioners and Women Participants care encounters, their overall approach to health care, how they Many concepts addressed during practitioner interviews advocate around health care needs, and barriers that exist for overlapped with themes from participant interviews. Ideas women to receive optimal care. Additionally, this study cap- discussed by practitioners around how women substitute tures impressions of care that feel inherently gendered.27-29 reproductive check-ins for primary care and how women’s Although the women in this study had insurance and care is inherently medicalized due to reproductive health higher education levels, they still had trouble navigating serve as an example. Most participants were taking birth scheduling logistics, had stories of feeling dismissed, and were control and checked in with some type of medical practi- not always receiving optimal care. While prior research tioner regularly to ensure continued access and reproductive highlights the need for increased cultural competency for health. For the most part, the check-ins described by vulnerable subgroups of women,18-22 this study demonstrates participants were not traditional primary care visits and were the need for an improved approach for treating all women.

54 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.176 ORIGINAL RESEARCH ARTICLE Insurance Barriers, Gendering, and Access: Interviews with Central North Carolinian Women About Their Health Care Experiences

Adaptation of practitioners’ approach to in-person en- appointments, how to ensure patients feel like more than a counters is needed. emes addressed by participants around checklist, and how to ease burdens around access and cost. feeling dismissed or unheard could be rectified in part by Special attention should be given to mothers and postpartum increasing visit lengths.30 Encouraging longer in-person care. Future efforts should focus on increasing awareness appointments fits into current shifts in health care reform and facilitating access to affordable postpartum care, easing away from fee-for-service models and toward value-based burdens around scheduling appointments and care transi- payment reform.31 tions, and conducting more research regarding women’s It will also be important to continue work to decrease experiences during in-person encounters. burdens around scheduling and logistics. In recent years, We should also consider training standards to make sure thehealthsystemhasworkedtobemoreadaptiveand women-specific health care is covered more robustly in user-friendly, with initiatives around patient-centered pri- practitioners’ training. Examples may include training in mary care involving more flexibility and transparency for responding to domestic violence,sexualassault,andwomen’s patients.32 Results from this study suggest the need for mental health and also how implicit biases can affect care continued work to expand the use of remote communi- toward women. v cation options and remote health check-ins or nontradi- tional clinic hours to reduce barriers for patients to access Disclosure Statement care. Indeed, the utility of remote care has been placed in The author(s) have no conflicts of interest to disclose. the spotlight with the recent COVID-19 coronavirus Acknowledgments pandemic. The authors thank all of the women participants for their vulnerability in sharing From discussions with mothers and practitioners, it is their experiences. clear there is a need for improved care postpartum. Most of the mothers interviewed discussed instances of feeling Authors’ Contributions ignored or dismissed or having trouble accessing proper Lena JP Cardoso, MPP, designed the study, primarily conducted the data collection, collaborated on the data analysis, and prepared the manuscript. Anna care after birth. In addition to concerns around inadequate Gassman-Pines, PhD, designed the study and prepared the manuscript. Nathan A care postpartum, some mothers mentioned childcare with Boucher, DrPH, PA, MS, MPA, CPHQ, designed the study, collaborated on the data an infant being a substantial barrier for them making it to analysis, and prepared the manuscript. All authors have given final approval to the an appointment. One option to reduce the burden on new manuscript. mothers and ensure their care needs are met is to combine ’  Financial Support care for new mothers with their babies wellness visits. is This work was supported by the Duke University Sanford School of Public Policy. may involve housing OB/GYN practices in spaces with pediatricians to link care involving mother and child. References A strength of this study is the in-depth line of inquiry we 1. Kaiser Family Foundation. Kaiser Women’s Health Survey. San Francisco: Kaiser Family Foundation; 2017. were able to explore. However, study participants were not 2. Kullgren JT, McLaughlin CG, Mitra N, Armstrong K. Nonfinancial Barriers and Access to representative of the broader population in central North Care for U.S. Adults. Health Serv Res. 2012;47(1 Pt 2):462-85.  3. Lassman D, Hartman M, Washington B, Andrews K, Catlin A. US health spending trends Carolina due to the small sample size. ey were, in general, by age and gender: Selected years 2002-10. Health Aff (Millwood) 2014 May;33(5): more educated and not as racially and ethnically diverse as 815-22. DOI: https://doi.org/10.1377/hlthaff.2013.1224, PMID:24799579 4. Cylus J., Hartman M., Washington B, Andrews K, Catlin A. Pronounced Gender and Age the larger population. Even in this group of women who had Differences are Evident in Personal Health Care Spending per Person. Health Aff a high level of education, access, and insurance, dismissive (Millwood). 2011;30(1). care still served as a major barrier to optimal treatment. 5. Alemayehu B, Warner K E. The Lifetime Distribution of Health Care Costs. Health Serv Res. 2004 Jun;39(3):627-42. Additionally, this study was advertised as aiming to examine 6. Owens GM. Gender Differences in Health Care Expenditures, Resource Utilization, and women’s unique experiences and barriers to care, which Quality of Care. Suppl J Manag Care Pharm. 2008;14(3):S2-6. 7. Vaidya V, Partha G, Karmakar M. Gender Differences in Utilization of Preventive Care may partially explain the largely negative sentiment expressed Services in the United States. J Womens Health. 2012;21(2). by participants. Concerns discussed by this group are likely 8. Pinkhasov RM, Wong J, Kashanian J, et al. Are men shortchanged on health? fi Perspective on health care utilization and health risk behavior in men and women in the ampli ed for women of color and women from lower-income United States. Int J Clin Pract 2010 Mar;64(4):475-87. DOI: https://doi.org/10.1111/j.1742- backgrounds and lower-resource settings.33 Future research 1241.2009.02290.x, PMID:20456194 should include a more diverse range of women to gain a 9. Center for Disease Control and Prevention. Life Expectancy at birth and at age 65, by sex: Organization for Economic Co-operation and Development (OECD) Countries, selected more complete scope of the problem. years 1980-2015. Natl Cent Health Stat; 2017. Experiences with dismissive care likely reflect systems 10. Case A, Paxson CH. Sex Differences in Morbidity and Mortality. Demography. 2005;42(2): 189-214. issues around increased burden on practitioners in en- 11. Goodwin K. Improving women’s health state policy options. Washington, DC: National counters and historical norms around women’s treatment, Conference of State Legislatures; 2015. 12. Arons J. Women and Obamacare: What’s at stake for women if the Supreme Court rather than practitioner intent. Future research should ex- strikes down the Affordable Care Act? Washington, DC: Center for American amine how to ensure women feel heard and acknowledged in Progress; 2012.

The Permanente Journal·https://doi.org/10.7812/TPP/20.176 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 55 ORIGINAL RESEARCH ARTICLE Insurance Barriers, Gendering, and Access: Interviews with Central North Carolinian Women About Their Health Care Experiences

13. Matud MP. Gender differences in stress and coping styles. Pers Individ Differ 2004 Nov; 23. Browne K. Snowball sampling: Using social networks to research non-heterosexual 37(7):1401-15. DOI: https://doi.org/10.1016/j.paid.2004.01.010 women. Int J Soc Res Methodol 2005 Feb;8(1):47-60. DOI: https://doi.org/10.1080/ 14. Wingard DL, Cohn BA, Kaplan GA. Sex differentials in morbidity and mortality risks 1364557032000081663 examined by age and cause in the same cohort. Am J Epidemiol. 1989;130(3): 24. Thomas DR. A general inductive approach for analyzing qualitative evaluation data. Am J 601-10. Eval 2006 Jun;27(2):237-46. DOI: https://doi.org/10.1177/1098214005283748 15. National Coalition Against Domestic Violence. National Statistics Domestic Violence Fact 25. Celik H, Lagro-Janssen TA, Widdershoven GG, Abma TA. Bringing gender sensitivity into Sheet [Internet]. 2019 [cited 2019 Aug 13]. Available from: https://ncadv.org/statistics. healthcare practice: A systematic review. Patient Educ Couns 2011 Aug;84(2):143-9. DOI: 16. Cuellar A, Simmons A, Finegold K. The Affordable Care Act and women. Washington, https://doi.org/10.1016/j.pec.2010.07.016, PMID:20719461 DC: US Department of Health and Human Services Assistant Secretary for Planning and 26. US Census Bureau. Quick facts: North Carolina. Published 2017. Accessed September 5, Evaluation; 2012. 2019. https://www.census.gov/quickfacts/NC 17. Garrett D, Greenberger M, Waxman J, Benyo A, Dickerson K, Gallagher-Robbins K, 27. Morgan R, Mangwi Ayiasi R, Barman D, Buzuzi S, Ssemugabo C, Ezumah N, et al. Moore R, Trumble S. Turning to fairness: Insurance discrimination against women today Gendered Health Systems: evidence from low- and middle- income countries. Health Res and the Affordable Care Act. NWLC. Published March 2012. Accessed March 2019. Policy Syst. 2018;16(58). https://www.nwlc.org/sites/default/files/pdfs/nwlc_2012_turningtofairness_report.pdf 28. Doyal L. Sex, gender, and health: the need for a new approach. BMJ 2001;323(7320):1061-3. 18. Barefoot KN, Warren JC, Smalley KB. Women’s health care: The experiences and 29. Samulowitz A, Gremyr I, Eriksson E, Hensing G. “Brave Men” and “Emotional Women”:A behaviors of rural and urban lesbians in the USA. Rural Rem Health 2017 Jan-Mar;17(1): Theory-Guided Literature Review on the Gender Bias in Health Care and Gendered 3875. DOI: https://doi.org/10.22605/rrh3875, PMID:28248528 Norms towards Patients with Chronic Pain. Pain Res Manag 2018. 19. Gregg I. The health care experiences of lesbian women becoming mothers. Nurs 30. Linzer M, Bitton A, Tu SP, et al. The end of the 15-20 minute primary care visit. J Gen Womens Health 2018 Feb;22(1):40-50. DOI: https://doi.org/10.1016/j.nwh.2017.12.003 Intern Med 2015 Nov;30(11):1584-6. DOI: https://doi.org/10.1007/s11606-015-3341-3, 20. Hill N, Hunt E, Hyrkas¨ K. Somali immigrant women’s health care experiences and beliefs PMID:25900539 regarding pregnancy and birth in the United States. J Transcult Nurs 2012 Jan;23(1): 31. Robert Wood Johnson Foundation. Value-based payment reform. Published 2018. 72-81. DOI: https://doi.org/10.1177/1043659611423828, PMID:22052095 Accessed August 13, 2019. https://www.rwjf.org/en/library/collections/health-care- 21. Steinberg AG, Wiggins EA, Barmada CH, Sullivan VJ. Deaf women: Experiences and payment-reform.html perceptions of healthcare system access. J Wom Health 2002 Oct;11(8):729-41. DOI: 32. Davis K, Schoenbaum SC, Audet A-M. A 2020 vision of patient-centered primary care. J Gen https://doi.org/10.1089/15409990260363689 Intern Med 2005 Oct;20(10):953-7. DOI: https://doi.org/10.1111/j.1525-1497.2005.0178.x. 22. Rodriguez MA, Quiroga SS, Bauer HM. Breaking the silence. Battered women’s 33. Geronimus AT, Hicken M, Keene D, Bound J. “Weathering” and age patterns of allostatic perspectives on medical care. Arch Fam Med 1996 Mar;5(3):153-8.DOI: https://doi.org/ load scores among blacks and whites in the United States. Am J Public Health 2006 May; 10.1001/archfami.5.3.153, PMID:8620256 96(5):826-33. DOI: https://doi.org/10.2105/AJPH.2004.060749, PMID:16380565

56 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.176 n ORIGINAL RESEARCH ARTICLE Does a Recumbent Lateral Stability Trainer Improve Balance Scores Among Older Adults Within 4 Weeks?

Andrew Shim, EdD1; Samantha Prichard, MHS1; David Newman, PhD2; Carly Lara1; Mike Waller, PhD3; Maureen Hoppe, EdD, OTR/L1 Perm J 2021;25:20.100 E-pub: 3/3/2021 https://doi.org/10.7812/TPP/20.100

ABSTRACT fractures are caused from falls. e estimated medical costs Backgroud: Past literature has shown that balance and of fatal and nonfatal falls in adults age 65 and older was over strength are important in preventing falls, but few studies have $50 billion in the United States. From fatal falls alone, the focused on developing strength and power in a lateral plane. The medical expenditure was estimated to be $754 million.5 As purpose of this study was to determine if a lateral pedal re- the population ages, an increase in costs is expected for older cumbent training device can improve balance scores among adults, and these negative outcomes may diminish the older adults in 4 weeks. quality of life in victims as well as create burdensome Methods: A 2-group experimental-control multivariate design 6 (43 women, 13 men; age, 77.4 ± 3 years; weight, 78.91 ± 0.2 kg; healthcare costs. height, 167.13 ± 0.8 cm; body mass index, 28.7 ± 0.5 kg/m) was Falling can result in a decrease of self-esteem and as- selected for the study. Participants (n = 56) were divided into 2 surance and therefore in eventual withdrawal from com- groups and were pretested and posttested on a computerized munal activities and participation. Research has begun to posturography plate to determine center of pressure scores with consider factors that may diminish a person’s ability to eyes opened with stable surface (EOSS), with eyes closed with perform activities of daily living after falling.7 Activities of stable surface (ECSS), with eyes open with perturbed surface daily living are described as tasks that an individual must (EOPS), and with eyes closed with perturbed surface (ECPS). The perform on an everyday basis, such as movement or eating experimental group used the lateral trainer for 15 minutes, 3 times on their own. e ability to perform these tasks is greatly per week, for 4 consecutive weeks; the control group maintained diminished once a fall has occurred. e risk factors for a sedentary lifestyle. A mixed-effects repeated measures multiple fi falling increase the likelihood for subsequent falls. Tinetti analysis of variance was used to determine signi cance. 8 Results: There were statistically significant differences over and Kumar showed that a person has a 78% chance of time for EOPS (p = 0.047) and ECPS (p = 0.047). Likewise, there falling if 4 risk factors are present. Other risk factors listed were statistically significant differences for each univariate out- include use of psychoactive medications, polypharmacy come with EOSS (p = 0.045), ECSS (p = 0.033), EOPS (p = 0.010), (taking too many medications that could contraindicate and ECPS (p = 0.026). with others), poor visual acuity, vertigo, cognitive impair- Conclusion: A recumbent lateral stability device can improve ment, orthostasis (decreased blood pressure shortly after balance scores among older adults within 4 weeks of training. standing), and the female sex.9 Previous falls, balance impairments, muscle strength, and gait are the strongest risk factors for subsequent falls, all of INTRODUCTION 8 10 According to the United States Census Bureau, it is which can be improved. Hess and Woollacott showed the predicted that there will be approximately 78 million people importance of muscle strength and balance for preventing 65 years of age or older by the year 2035, compared with the future falls in the elderly; however, few research studies have population of 76.7 million people under the age of 18.1 put precedence on the importance of muscular power Assessing fall risk among the older population would output. likely help sustain their quality of life. All individuals over Muscular power output is the ability to exert maximal force at a fast rate in a short period of time, such as accel- the age of 65 ought to be asked at least once a year by their 7 11 health care provider if they have fallen, based upon a neutral erating and jumping. Skelton, Kennedy, and Rutherford opinion from health care providers associated with this noted that women with a history of falls were 24% less article. Falls are the number one cause of fatal and nonfatal injuries in adults ages 65 and older.2 e National Council Author Affiliations on Aging3 reports that falls are the leading cause of trauma- 1College of Saint Mary, Omaha, NE 2Florida Atlantic University, Boca Raton, FL related visits to the hospital in elderly adults. Patel and 3Arkansas Tech University, Russellville, AR Ackermann4 reported that approximately 2.5 million falls among adults age 65 and older are treated in the emergency Corresponding Author room. Of those treated in the emergency room, 87% of Andrew Shim, EdD ([email protected]) Keywords: center of pressure, falls, older adults, recumbent bicycle The Permanente Journal·https://doi.org/10.7812/TPP/20.100 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 57 ORIGINAL RESEARCH ARTICLE Does a Recumbent Lateral Stability Trainer Improve Balance Scores Among Older Adults Within 4 Weeks?

powerful than their peers who had not sustained falls by measuring their lower limb explosive power, isometric strength of the quadriceps and hamstrings, and isokinetic concentric strength of the lower limbs. Initial studies fo- cusing on generational differences of men also have dis- played a decrease of 8.3% in maximal anaerobic power per decade.12 A faster decrease in muscle power in comparison to strength has been identified in individuals between the ages of 65 and 89.13 e evidence suggests that low power output in patients may influence falls and should be con- sidered in addition to strength training. Comparable results were noted in a study performed on nonelderly adults, identifying muscle power as a factor with gait variability.14 Findings from that study showed that muscle quality and Figure 1. Group means of eyes open stable surface with regard to time. quantity in the quadriceps influence how frail the elderly person is. If there are additional higher-density muscle be over the age of 55 years, independent, and determined fibers in the quadriceps, the elderly person performed healthy by answering a health questionnaire. Seven subjects significantly better in step time variability, gait performance, who were experiencing musculoskeletal pain or unable to and velocity.14 perform the balance assessments or adhere to the 4-week Modifying exercise programs to address improving lower training were excluded. Group assignments for experi- body strength and power would vastly improve stability and mental and control groups were based on location of the functional movement. e study by Shim et al7 demon- facility and nonrandomized. Informed consent was signed strated a strong correlation between posterior limit of by all participants before the start of the study. e In- stability plane among 13 seniors over the age of 65 and stitutional Review Board of the sponsoring university ap- relative peak power by using a 15-foot ramp to measure proved the investigation before the start date. One local power output in seniors (n = 17) and observing that lower senior residential site was designated as the experimental body power output does have a relationship with balance group, and the other independent senior center was des- scores. Based on these findings, use of an exercise device ignated as the control group. that addresses both lower body strength and power de- e primary pieces of equipment the researchers used for velopment safely for older adults could reduce the risks of this study were the SCIFIT Latitude Lateral Stability falling in older adults. Trainer (Brunswick Corporation, Rosemont, IL) and a Recently, a seated bicycle called a SCIFIT Latitude BalanceCheck computerized posturography plate (Bertec Lateral Stability Trainer was developed by a fitness man- Corporation, Columbus, OH). e experimental group ufacturer that proposed pedaling in a lateral direction could used the SCIFIT Latitude Lateral Stability Trainer 3 times possibly prevent falls by strengthening additional prime per week for a minimum of 15 minutes for 4 consecutive movers, such as the hip adductors and abductors, that a weeks. Safety and user directions were given to the par- traditional recumbent exercise bike was not designed to do. ticipants before each session as well explaining the training Based on direct information from SCIFIT, there has been curriculum. e experimental participants were instructed no clinical data collection to prove the manufacturer’s claims to perform light calisthenics such as walking quickly for of improving balance in older adults by using this novel approximately 5 to 10 minutes to increase blood flow and device (Figure 1). erefore, the purpose of this study was to pressure, heart rate, and body temperature gradually before determine if a recumbent bicycle with pedals that allowed using the SCIFIT Latitude Lateral Stability Trainer. e lateral movement would improve balance scores in older experimental group followed the directions on the exercise adults within a 4-week period. machine’s display to completion during each training ses- sion, which lasted 15 minutes. Instead of pedaling back and METHODS forth in the anterior/posterior plane, each pedal allowed A 2-group pretest/posttest experimental/control design motion in a horizontal plane, requiring effort to maintain was selected for this 4-week study. All subjects (n = 56) 59 the lateral movement of each pedal. Each exercise session years of age or older were recruited from senior independent performed was noncompetitive in nature among the ex- living centers from the vicinity (43 women, 13 men; age, perimental group participants. A Borg intensity chart 77.4 ± 3 years; weight, 78.91 ± 0.2 kg; height, 167.13 ± numbered from 1 to 10 was used to determine the pedaling 0.8 cm; body mass index, 28.7 ± 0.5 kg/m). Subjects had to intensity during the exercise session, with 1 representing an

58 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.100 ORIGINAL RESEARCH ARTICLE Does a Recumbent Lateral Stability Trainer Improve Balance Scores Among Older Adults Within 4 Weeks?

Table 1. Descriptive statistics for pretest/posttest by treatment groups Pretest Posttest Group 1 Group 2 Group 1 Group 2 control (experimental) control (experimental) Tests Mean SD Mean SD Mean SD Mean SD EOSS 87.8 8.6 90.5 4.8 88.5 7.7 90.1 4.5 ECSS 85.5 8.5 88.0 5.6 85.9 8.7 87.2 5.4 EOPS 83.2 11.0 87.1 5.5 85.1 10.4 86.9 5.7 ECPS 79.6 11.3 79.1 7.8 82.9 11.3 79.5 8.2 ECPS = eyes closed perturbed surface; ECSS, eyes closed stable surface; EOPS = eyes open perturbed surface; EOSS = eyes opened stable surface.

Figure 2. Group means of eyes closed stable surface with regard to time. including gestures, turning, or talking, until all 4 balance extremely easy pedaling intensity and 10 representing ex- measurements were taken. tremely difficult. e Borg chart was mounted on the wall and was visible to all participants while pedaling on the RESULTS SCIFIT Latitude Lateral Stability Trainer. Once a general Data were collected and analyzed with SPSS Version 26 warm-up was performed that consisted of pedaling on the (IBM Corp, Armonk, NY). e descriptive statistics for the recumbent bicycle without much resistance, each rider pretest and posttest from both groups can be found in pedaled at a level of a 6 during the first several minutes of a Table 1. When comparing Group 1 numbers with Group 2, 15-minute ride and eventually worked at pedaling rapidly to noticeable differences are seen regarding the static balance a level of 8 during the last minute for 15 seconds before scores during the 4 weeks of training. Postural sway (CoP) slowing down and resuming back to a level of a 6 to scores have decreased over time in the experimental group. complete their 15-minute ride. Once the subject completed However, significance is not recognized until empirical data the displayed curriculum from the screen, a cool-down are revealed in Table 2. Table 2 illustrates the mixed-effect, exercise, such as easy walking and light stretching, was repeated-measures multiple analysis of variance results for completed for approximately 5 to 10 minutes to reduce body the linear combination of EOSS, ECSS, EOPS, and temperature, decrease heart rate, and reduce blood pressure. ECPS. ere were no significant main treatment effects for e control group was given strict written and verbal in- either group (p = 0.221) observed in Table 2 or Figure 3 for structions to maintain their sedentary lifestyle. ere were EOSS. However, Figures 4 and 5 show there were sta- no requirements to enter an activity log or survey during the tistically significant differences over time for EOPS (p = 4-week investigation. 0.047) and ECPS (p = 0.047). Likewise, there were sta- e Bertec computerized posturography plate was used to tistically significant differences (Figure 6) for each uni- calculate body mass index and center of pressure (CoP) variate outcome with EOSS (p = 0.045), ECSS (p = 0.033), scores in all participants. CoP is the precise measurement of EOPS (p = 0.010), and ECPS (p = 0.026). Statistical power postural sway in the static position, measuring the amount (ie, the ability to detect a true difference when one is of movement occurring on the computerized force plate present) was achieved (> 0.98) for both univariate and while standing erect. Each subject was tested on the multivariate measures. It is generally accepted that the computerized posturography plate before the first week and power of a study will be > 0.80, indicating an 80% greater after the fourth week (Figure 2). chance of rejecting the null hypothesis. e multivariate Each participant was given 4 balance tests to measure outcome for the group × time interaction accounted for CoP scores on the computerized posturography plate before 15.2% more variance than time alone (28.1%) for the ex- the first week and after the completion of the fourth week: perimental group (Group 2). eyes open with stable surface (EOSS), eyes closed with stable surface (ECSS), eyes open with perturbed surface DISCUSSION (EOPS), and eyes closed with perturbed surface (ECPS). is current investigation demonstrated among elderly Subjects were asked to stand on the plate barefoot for 10 s adults that an exercise protocol consisting of pedaling in the for each specific balance test while the computer program lateral plane on a SCIFIT Latitude Lateral Stability Trainer measured postural sway. Subjects were coached to stand on can improve CoP scores within 4 weeks compared with the the plate relaxed and to minimize unnecessary movements, control group. In their investigation of over 700 studies of

The Permanente Journal·https://doi.org/10.7812/TPP/20.100 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 59 ORIGINAL RESEARCH ARTICLE Does a Recumbent Lateral Stability Trainer Improve Balance Scores Among Older Adults Within 4 Weeks? 2 η 2 F(1,54) p value η

Figure 3. Group means of eyes open perturbed surface with regard to time.

various exercise modalities on older adults, ompson et al15 demonstrated that a varied exercise intensity 2 to 3 times per week could improve muscle force after several weeks of training. When observing body sway in the elderly, in- 2 F(1,54) p value

η creases occur when an individual does not have postural control due to decreased muscle mass and strength.16 In- dividuals who have large body sway displacements increase demands for large postural corrections to maintain their balance. Low’s16 investigation showed that proper corrective exercise training can decrease postural sway by improving muscular strength and power of the lower extremities. e current investigation promoted not only muscular strength but also power production based on varying the bicycle

2 F(1,54) p value pedaling intensities that required the participant to pedal η with a higher velocity for a short period of time. Rouffet and Hautier’s17 study on cyclists demonstrated improvements in peak EMG amplitude in 6 major muscles (gluteus max- imus, vastus lateralis, rectus femoris, bicep femoris long head, gastrocnemius medialis, and soleus) while cycling under 15 different submaximal and maximal conditions for 30-second intervals. eir study demonstrated peak EMG F(1,54) p value 2 η 0.208 0.2 0.657 0.004 0.45 0.506 0.008 4.12 0.047 0.071 4.81 0.033 0.082 .017 Multivariate EOSS ECSS EOPS ECPS F(4,54) p value Table 2. Multivariate and univariate results for the linear combination of EOSS, ECSS, EOPS, and ECPS ECPS = eyes closed perturbed surface; ECSS, eyes closed stable surface; EOPS = eyes open perturbed surface; EOSS = eyes opened stable surface. InterceptGroupTime 2634.6Time × group < 0.001 1.089 7.18 0.995 3.34 0.372 10,299 0.000 0.079 0 0.360 < 0.001Figure 1.531 0.995 4.23 4. 8178.7 0.221 Group 0.045 0.028

60 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.100 ORIGINAL RESEARCH ARTICLE Does a Recumbent Lateral Stability Trainer Improve Balance Scores Among Older Adults Within 4 Weeks?

Figure 5. Sci-fit latitude recumbent bicycle.

values with muscle activation under maximal pedaling. is resulted in improved neural responses from the 6 selected prime movers of the lower limbs. Moreover, activating muscles used for stability through the lateral movements and producing greater energy output bursts for a short period of time may translate to improved balance. Even though this study did not measure lower body power production or EMG neural activity, the exercise protocol provided to the experimental group should have encouraged developing lower body power production, resulting in ac- tivation of additional adductors and abductor prime movers. e study protocol allowed each participant to quickly increase intensity by pedaling faster for a shorter time near the end of the exercise session. e researchers also spec- ulated that the bidirectional lateral motion activated dif- ferent neuromuscular firing patterns that standard upright or recumbent bicycles do not promote. is bicycle training protocol, involving a short burst of intensity, could have promoted and enhanced increased power production in the lateral plane, creating additional benefits of motor unit recruitment and lower body power production. Shim’s7 investigation demonstrated the importance of developing or maintaining lower body power as a major contributor toward improving stability among the elderly. e control group was not physically or visually super- vised during their 4-week period besides the pretesting Figure 6. Bertec posturography plate. and posttesting dates for balance assessments. Table 1 shows that there were mean increases to postural sway in the control group compared with the experimental group.

The Permanente Journal·https://doi.org/10.7812/TPP/20.100 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 61 ORIGINAL RESEARCH ARTICLE Does a Recumbent Lateral Stability Trainer Improve Balance Scores Among Older Adults Within 4 Weeks?

e investigators did hypothesize that stability improve- stability exercises in the upright body position compared ments would not exist within this particular group based on with being in a recumbent position. Also, measuring lower nonparticipation of any physical activity. Past studies10,12,18 body power output as an additional dependent variable support these findings from the inactive control group because would determine that this variable was a major factor toward they did not exhibit any other balance improvements, including balance score improvement. v the perturbed surface testing. esignificant improvement with the experimental group was likely due to improved lateral Disclosure Statement fl stability due to the SCIFIT Latitude Lateral Stability Trainer The author(s) have no con icts of interest to disclose. design and from participation in the exercise protocol. Pro- Authors’ Contributions prioception mechanoreceptors would be the main rationale Andrew Shim, EdD, is the Program Director for Kinesiology & Exercise Science of these balance improvements, probably based on the lateral at College of Saint Mary, Omaha, NE. He proposed and developed the study, rotation of the SCIFIT Latitude Lateral Stability Trainer performed the data collection, and wrote 45% of the final manuscript. Samantha Prichard, MHS, is a Physician Assistant Studies student at Stephens College in pedals increasing the ability to pay attention to changes of Columbia, MO. She wrote 20% of the final manuscript and assisted with the editing. direction while visually cued to watch the screen. David Newman, PhD, is the Statistician and Associate Professor for the Christine E Although most studies have focused on combining various Lynn College of Nursing at Florida Atlantic University, Boca Raton, FL. He modes of mobility and exercise programs for the elderly, the performed the data analysis and assisted with 15% of the final manuscript. Ms. Carly 8,16 18 Lara is a KES major at College of Saint Mary, Omaha, NE. She assisted with the results were not as successful. Foldvari et al used proposal, on-site data collection, and 5% of the final manuscript. Mike Waller, PhD, is an combined exercise modalities with a self-reported functional Associate Professor in Health & Physical Education at Arkansas Tech University. He status, and the only strong correlation was one that combined assisted with the proposal and wrote 10% of the final manuscript. Maureen Hoppe, the self-reported functioning status with leg power. e EdD, OTR/L, is an Assistant Professor for the Occupational Therapy Department at College of Saint Mary. She performed the editing, proofing, and wrote 5% of current study focused on the connection between lower leg the final manuscript. All authors have given final approval to the manuscript muscle power output and CoP using the SCIFIT Latitude There was no funding source for the study to declare. Lateral Stability Trainer to improve balance stability scores in elderly women. Yearly physical examinations and educating Abbreviations older adults on how they can prevent future or subsequent CoP = center of pressure; EOSS = eyes open stable surface; ECSS = eyes falls are important to the rising population of aging adults. By closed stable surface; EOPS = eyes open perturbed surface; ECPS = eyes closed perturbed surface. focusing on strengthening the lower leg muscles (increasing fi the muscle power output), this will signi cantly prevent References future or subsequent falls as reported in Table 1. 1. United States Census Bureau. Older people projected to outnumber children for first time in U.S. history; 2018. Accessed September 16, 2018. https://www.census.gov/newsroom/ press-releases/2018/cb18-41-population-projections.html. CONCLUSION 2. Boyé ND, Van Lieshout EM, Van Beeck EF, Hartholt KA, Van Der Cammen TJ, Patka P. e purpose of this study was to determine if the use of a The impact of falls in the elderly. Trauma 2013 Jan;15(1):29-35. DOI: https://doi.org/10. 1177/1460408612463145 SCIFIT Latitude Lateral Stability Trainer could improve 3. National Council on Aging. Falls prevention; 2019. Accessed August 5, 2019. https:// balance scores in elderly adults within 4 weeks of an exercise www.ncoa.org/healthy-aging/falls-prevention. 4. Patel D, Ackermann R. Issues in geriatric care: Falls. FP Essent 2018 May;468:18-25. protocol compared with an inactivecontrolgroup.Balance PMID:29714993 https://www.ncbi.nlm.nih.gov/pubmed/29714993. exercise training can help improve postural sway by making the 5. Florence CS, Bergen G, Atherly A, Burns E, Stevens J, Drake C. Medical costs of sway area smaller, decreasing the need for corrections.16 e fatal and nonfatal falls in older adults. J Am Geriatr Soc 2018 Apr;66(4):693-8. DOI: https://doi.org/10.1111/jgs.15304. experimental group had significant changes using the SCIFIT 6. Polinder S, Boyé ND, Mattace-Raso FU, et al. Cost-utility of medication withdrawal in Latitude Lateral Stability Trainer over 4 weeks with “Per- older fallers: Results from the improving medication prescribing to reduce risk of FALLs (IMPROveFALL) trial. BMC Geriatr 2016 Nov;16(1):179. DOI: https://doi.org/10.1186/ turbed Surface – Eyes Open and Eyes Closed” assessments. s12877-016-0354-7, PMID:27809792. Proprioception mechanoreceptors would be the main rationale 7. Shim A, Harr B, Waller M. Does a relationship exist between lower body power and balance scores among older adults? Perm J 2018;22:17-096. DOI: https://doi.org/ of improvements, probably based on the lateral rotation of the 10.7812/TPP/17-096, PMID:29616904. SCIFIT Latitude|Lateral Stability Trainer’spedalsincreasing 8. Tinetti ME, Kumar C. The patient who falls: “It’s always a trade-off”. J Am Med Assoc 2010 the ability to pay attention to changes of direction while vi- Jan;303(3):258-66. DOI: https://doi.org/10.1001/JAMA.2009.2024. 9. Lee A, Lee KW, Khang P. Preventing falls in the geriatric population. Perm J 2013;17(4): sually cued to watch the screen. Limitations of this investi- 37-9. DOI: https://doi.org/10.7812/TPP/12-119, PMID:24361019. gation included the time of day of the training sessions, specific 10. Hess JA, Woollacott M. Effect of high-intensity strength-training on functional measures of balance ability in balance-impaired older adults. J Manipulative Physiol Therapeut 2005 type of curriculum proposed for this investigation, sample size Oct 28(8):582-90. DOI: https://doi.org/10.1016/j.jmpt.2005.08.013. of the study, and gender inequity of participants. 11. Skelton DA, Kennedy J, Rutherford O. Explosive power and asymmetry in leg muscle function in frequent fallers and non-fallers aged over 65. Age Ageing 2002 Mar;31(2): 119-25. DOI: https://doi.org/10.1093/ageing/31.2.119 Future Research 12. Aagaard P, Suetta C, Caserotti P, Magnusson SP, Kjaer M. Role of the nervous system in sarcopenia and muscle atrophy with aging: Strength training as a countermeasure. Scand It would be recommended for future studies to increase J Med Sci Sports 2010 Feb;20(1):49-64. DOI: https://doi.org/10.1111/j.1600-0838.2009. the sample size and combine multiple modalities of lateral 01084.x

62 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.100 ORIGINAL RESEARCH ARTICLE Does a Recumbent Lateral Stability Trainer Improve Balance Scores Among Older Adults Within 4 Weeks?

13. Li Z, Liang Y-Y, Wang L, Sheng J, Ma S-J. Reliability and validity of center of pressure 16. Low DC, Walsh GS, Arkesteijn M. Effectiveness of exercise interventions to improve measures for balance assessment in older adults. J Phys Ther Sci 2016 Apr;28(4): postural control in older adults: A systematic review and meta-analyses of centre of 1364–7. DOI: https://doi.org/10.1589/jpts.28.1364. pressure measurements. Sports Med 2017 Jan;47(1):101-12. DOI: https://doi.org/10. 14. Martinikorena I, Martínez-Ramírez A, Gómez M, et al. Gait variability related 1007/s40279-016-0559-0. to muscle quality and muscle power output in frail nonagenarian older adults. 17. Ruhe A, Fejer R, Walker B. Center of pressure excursion as a measure of balance J Am Med Dir Assoc 2016 Feb;17(2):162-7. DOI: https://doi.org/10.1016/ performance in patients with non-specific low back pain compared to healthy controls: A j.jamda.2015.09.015. systematic review of the literature. Eur Spine J 2011 Mar;20(3):358-68. DOI: https://doi. 15. Thompson BJ, Ryan ED, Herda TJ, Costa PB, Herda AA, Cramer JT. Age-related org/10.1007/s00586-010-1543-2, PMID:20721676. changes in the rate of muscle activation and rapid force characteristics. 18. Foldvari M, Clark M, Laviolette LC, et al. Association of muscle power with functional Age (Dordr) 2014 Apr;36(2):839–49. DOI: https://doi.org/10.1007/ status in community-dwelling elderly women. J Gerontol A Biol Sci Med Sci 2000 Apr; s11357-013-9605-0. 55(4):M192-M199. DOI: https://doi.org/10.1093/gerona/55.4.m192.

The Permanente Journal·https://doi.org/10.7812/TPP/20.100 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 63 n ORIGINAL RESEARCH ARTICLE Better Together: Long-term Behaviors and Perspectives after a Practitioner–Family Writing Intervention in Clinical Practice

Madelaine Schaufel, MS, RD1; Douglas Moss, BS1; Ramona Donovan, MS, RD, CCRC2; Yi Li, MS3; David G Thoele, MD2 Perm J 2021;25:20.250 E-pub: 03/01/2021 https://doi.org/10.7812/TPP/20.250

ABSTRACT writing (EW), defined as therapeutic writing that involves Background: An expressive writing intervention, the Three- disclosure of personal information, thoughts, or feelings.2 minute Mental Makeover (3MMM), was previously associated with EW has been associated with reduced stress, improved reduced stress for practitioners, patients, and families. The goal of health, increased disease-related quality-of-life scores, and this follow-up study was to evaluate long-term use of writing and reductions in physical symptoms in a variety of patient perspectives after participation in the 3MMM. populations.3-16 EW interventions also have the potential to Methods: The original study involved patients and families in be low-cost, low-risk, and are generally well-accepted by the neonatal and pediatric Intensive Care Units, inpatient units, 9,10,13-15,17-21 and outpatient clinics of a children’s hospital. Health-care prac- patients. titioners led the intervention, writing concurrently with patients Although many EW interventions have been associated fi 7,9-12,14,15,22 and families using the 3MMM intervention. Follow-up contact by with short-term health bene ts, few studies have 23-25 phone was attempted for all original patient/family participants examined the long-term effects of EW. One EW in- 12 to 18 months after completing the exercise. Practitioners were tervention was associated with long-term benefits and contacted via email 24 months after the original study. Partici- behavior changes in resident physicians who participated pants were surveyed about the 3MMM and continued use of in a 2-day writing workshop.25 To our knowledge, there are writing to cope with stressful situations. Original and follow-up no studies showing long-term behavior changes in patients survey responses were compared using the Mann-Whitney test. or family members following a brief EW intervention in Results: Of the 96 original patient/family members, 61 indi- clinical practice. viduals were reached, remembered the 3MMM, and agreed to We recently described a brief, novel, EW intervention for participate in the follow-up study. Among the 61 participants, 52  (85%) agreed that the 3MMM had been helpful. Thirty-six (59%) use in routine clinical practice, titled the ree-minute reported using writing to help cope with stress at follow-up, Mental Makeover (3MMM), during which the practitioner  compared to only 23 (38%) at baseline (P = 0.005). The majority of and patient/family members write and share together. e clinicians (87%) also continued to use the 3MMM in clinical 3MMM intervention was shown to be associated with practice following the original 3MMM study period and ranked it reduced stress for patient/family members and practitioners as both feasible (75%) and worth the time investment (75%). immediately after completing the intervention.26 is follow- Conclusion: The 3MMM demonstrated long-term perceived up study examines long-term practitioner and patient/family benefits and behavior changes. Findings provide preliminary member perspectives regarding the 3MMM intervention, as support for using the 3MMM in routine clinical practice. well as the long-term use of writing to cope with stress. e 3MMM intervention prompts are presented in Figure 1.

METHODS INTRODUCTION is follow-up study evaluated long-term perspectives e medical environment features many stressors for related to the 3MMM intervention and writing behavior patients, their families, and health-care practitioners. Ex- changes in patient/family members and practitioners after cessive stress has been shown to be detrimental to human participation in the original 3MMM study. health in many ways.1 erefore, interventions that either Patients and family participants from the original study reduce stress or improve coping are of considerable interest. were recruited using convenience sampling. ese patients One method used to help cope with stress is expressive and family members were cared for in clinical settings by 1 of the 8 participating practitioners, who sensed the patient/ Author Affiliations family members were experiencing stress and therefore were 1Rosalind Franklin University, North Chicago, IL invited to complete the 3MMM activity. Health-care prac- 2Advocate Children’s Hospital, Park Ridge, IL titioners included 2 pediatric residents, 3 pediatric nurses 3 Advocate Children’s Hospital, Oak Lawn, IL [1 neonatal Intensive Care Unit(ICU),1pediatricICU,and Corresponding Author 1 general inpatient pediatric unit], a developmental psychologist, David G Thoele, MD ([email protected]) a developmental educator, and a pediatric cardiologist. Patient/

Keywords: communication, expressive writing, mindfulness, narrative medicine, stress reduction, writing together 64 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.250 ORIGINAL RESEARCH ARTICLE Better Together: Long-term Behaviors and Perspectives after a Practitioner–Family Writing Intervention in Clinical Practice

Table 1. Three-minute Mental Makeover follow-up survey for patients and family members 1. Do you remember doing the 3MMM writing activity? □ Yes (if yes, proceed to question 2) □ No 2. We are doing a short follow-up survey that will take approximately 10 minutes. Would you be willing to answer a few questions about the 3MMM? Figure 1. Three-minute Mental Makeover (3MMM) writing prompts. □ Yes (if yes, proceed to question 3) □ No 3. Thinking back to when you originally participated in the 3MMM activity, which family members were recruited from the neonatal ICU, pe- statement would best describe your experience: diatric ICU, inpatient pediatric unit, and outpatient pediatric a) I agree the 3MMM was helpful in reducing stress at the time. clinics at a Chicago-area children’s hospital. Clinicians selected b) I disagree that participating in the 3MMM was helpful in reducing stress patient/family participants who they determined subjectively at the time. to be experiencing emotional stress. c) I neither agree nor disagree that the 3MMM was helpful in reducing stress Twelve to 18 months after completing the original study, at the time. follow-up contact was attempted for all 96 original patient/ 4. Was it important that you and your provider completed the activity at the same time? family members. Participants were called and asked to □ Yes □ No complete a 9-question survey (Table 1) as a phone inter- 5. Was it important that you both shared your responses with one another? view. Calls were conducted by a medical student researcher □ Yes □ No not involved in facilitating the initial writing exercise. e 6. Have you experienced similar stressful times since you were here last year and phone survey took less than 10 minutes to complete. Patient/ completed the 3MMM activity? family members were also given the option to complete the □ Yes □ No survey electronically using the Google Forms online survey 7. How often have you used writing to help cope with stress? platform. If participants were unable to be contacted by a) Never (proceed to question 8) phone, they were mailed a paper copy of the survey with a b) Occasionally (< 1× per month) (proceed to question 7) return addressed envelope. c) Often (> 1× per month) (proceed to question 7) e 8 practitioners who participated in the original 8. Which writing technique have you used? 3MMM research study were contacted by email at least a) 3MMM 2 years after the initial study was completed. Practitioners b) Writing/journaling were sent a 14-question online Google Forms survey (Table 2). c) Both Up to 3 email/text reminders were sent to practitioners to  9. What about the writing exercise did you find useful? What was not useful? complete the survey. eonlinesurveywasmanagedbya (open ended) medical student researcher not involved in the initial study. 3MMM = Three-minute Mental Makeover. Data were summarized in counts and percentages. Likert-type survey responses, from the original and follow- survey completed it (N = 61). Among the 8 practitioners up surveys, were compared within individual respondents who participated in the original study, all responded to the using the Mann-Whitney test. Dichotomous response follow-up survey. Table 3 presents participant demo- categories, when available from both time points, were graphics as well as physical location of clinical contact. compared using McNemar change tests. Data were ana- lyzed using SPSS (version 25.0 for Windows; IBM Corp, Patient/Family Findings Armonk, NY). Statistical significance was determined by a First, we asked a few questions about patient/family par- P value of less than 0.05. ticipant perspectives related to the design of the 3MMM is research study was approved by the facility’s insti- intervention. Of the 61 participants, 52 (85%) agreed that tutional review board. the 3MMM had been helpful. Fifty-four (89%) agreed it was important that practitioners and participants wrote RESULTS together, and 56 (92%) indicated that it was important that Of the 96 patient/family member participants enrolled in both practitioner and patient shared what they wrote. the original study, 65 were reached for follow-up (68%). irty-five (57%) reported experiencing similar stressful Sixty-one of these participants remembered completing the events since the hospitalization or outpatient visit during 3MMM intervention and agreed to respond to the follow- which the 3MMM intervention was completed. e change up survey. Two participants of the original study did not perceived helpfulness of the intervention was similar across recall doing the activity and 2 declined survey participation. all subgroups regardless of unit or relationship to the All participants who agreed to respond to the follow-up patient.

The Permanente Journal·https://doi.org/10.7812/TPP/20.250 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 65 66 ARTICLE RESEARCH ORIGINAL h emnneJournal Permanente The

Table 2. Three-minute Mental Makeover follow-up survey for practitioners 1. Do you think it was important that you and the patient/family member completed the 3MMM activity at the same time? (yes/no) 2. Do you think it was important that you both shared your responses with one another after completing the 3MMM activity? (yes/no) · o esnlueol.N te sswtotpriso.C permission. without uses other No only. use personal For Personal use of journaling/writing 3. Before doing the 3MMM study, I used writing or journaling to help cope with difficult situations in my own life. (yes/no) 4. As a result of participating in the 3MMM study, I have used writing or journaling to help cope with difficult situations in my own life. (yes/no) 5. In the past year, how often have you used writing to help cope with stress in your own life? (Never; Occasionally, < 1× per month; Often, >1× per month; Very often, > 1× per week) 6. If you have used writing in the past year to cope with stress, which writing technique(s) have you used? (3MMM, Writing/journaling, Both, I have not used writing to cope with stress) Clinical use and feasibility 7. Prior to participating in 3MMM research, did you use any type of writing intervention to help reduce stress patients/family member? (yes/no) 8. Since participating in the original 3MMM study, have you used the 3MMM activity with patients/families? (yes/no)

9. In the past year, approximately how frequently have you used the 3MMM activity with patients/families? (Never or rarely; Occasionally, < 1× per month; Often, > 1× per month; Very often, > 1× per week) Practitioner a after Perspectives and Behaviors Long-term Together: Better 10. Weighing benefits vs barriers of utilizing the 3MMM activity with patient/family members, is the time investment justified? (Strongly agree, Agree, Neutral, Disagree, Strongly disagree) 11. How feasible would you rank the 3MMM activity for completion with patients as a regular part of clinical practice? (Very feasible, Somewhat feasible, Neutral, Somewhat unfeasible, Very unfeasible)

prgt©22 h emnnePes l ihsreserved. rights All Press. Permanente The 2021 © opyright Overall practitioner perspectives on use of 3MMM and writing 12. Doing the 3MMM is beneficial to the practitioner facilitating the activity. (Strongly agree, Agree, Neutral, Disagree, Strongly disagree) 13. In your experience completing the 3MMM with patients, what do you think is helpful to practitioners about the activity? (choose all that apply) Improves patient communication Reduces practitioner stress Saves time during patient/family interactions Increases trust in patient–practitioner relationship Helps the patient and practitioner get to know one another better Allows one to feel closer/more connected to patients Helps the patient see the practitioner as a person Helps the practitioner to understand the patient better Helps the practitioner focus on overall well-being of patient and family Other ______None. I do not think the 3MMM was helpful to patients. 14. What are barriers to completing the 3MMM activity for practitioners? (choose all that apply) h emnneJournal Permanente The Takes too much time – aiyWiigItreto nCiia Practice Clinical in Intervention Writing Family Seems unrelated to the purpose of the visit Discomfort with sharing feelings Practitioner does not like writing Other ______. · https://doi.org/10.7812/TPP/20.250 None. I do not think there are any barriers to completing the 3MMM. 3MMM = Three-minute Mental Makeover. ORIGINAL RESEARCH ARTICLE Better Together: Long-term Behaviors and Perspectives after a Practitioner–Family Writing Intervention in Clinical Practice

Table 3. Demographics of survey participants Patient/family member participants n=61 Patient 12 (20%) Family 49 (80%) Mother 37 Father 10 Other Relative 2 Practitioner Participants n=8 Clinical location Outpatient 24 Pediatric Developmental Clinic 5 (21%) Pediatric Cardiology Clinic 19 (79%) Figure 2. Participant use of writing to cope with stress. 3MMM = Three-minute Inpatient 37 Mental Makeover. NICU 23 (62%) PICU 8 (22%) In addition, practitioners reported their use of the General Pediatrics 6 (16%) 3MMM intervention with patients/families following the original study. Prior to participation, 2 of 8 practitioners reported using a writing intervention to help reduce stress in We then investigated patient/family participants’ per- patients/family members. Subsequently, the 2 practitioners sonal use of writing to cope with difficult situations after continued to use writing in their practice and an additional their participation in the 3MMM study. At follow-up, 36 5 practitioners adopted use of the 3MMM with patients/ (59%) reported using writing to help cope with stress, families (Figure 3). compared to 23 (38%) at baseline (P = 0.005) (Figure 2). Of Last, we investigated practitioner perspectives regarding the 61 participants, 25 (41%) reported using writing/ barriers and benefits related to using the 3MMM inter- journaling, 2 (3%) reported using only the 3MMM, and vention with patients/families and the feasibility of its use in 10 (16%) reported using both writing/journaling and the a clinical setting. e top barrier identified to completing 3MMM. e change of use in writing was similar across all the 3MMM activity with patients/families was the time subgroups regardless of unit or relationship to the patient. required. However, the majority of practitioners endorsed Patient/family participants also responded qualitatively that the benefits of doing the activity with patients/families regarding what they found useful about the 3MMM in- justified the time investment. In addition, 7 of the 8 tervention itself, the impact of self-reflection, and the in- practitioners agreed that the 3MMM activity is beneficial to teraction with the practitioner or family members. A the facilitator. Top benefits reported included reduced stress summary of the responses is presented in Table 4. and improved communication, understanding, and trust in practitioner–patient or practitioner–family relationships. Practitioner Findings Seventy-five percent of practitioners also endorsed the Follow-up survey results are described for the 8 practi- feasibility of the 3MMM intervention for use as a regular tioners who facilitated the 3MMM intervention during the part of clinical practice. original study. A series of questions was asked related to practitioner perspectives. DISCUSSION First, related to the design of the 3MMM intervention, e 3MMM intervention demonstrated long-term per- all 8 agreed it was important that practitioners and par- ceived benefits and was associated with lasting behavior ticipants wrote together, and all indicated it was important changes in both patient/family participants and practitioners. that both parties shared what they wrote. Both groups reported increased personal use of writing to Next, practitioners reported personal use of writing to cope with stress after participating in the 3MMM inter- cope with difficult situations after their participation in the vention. In addition, there was a trend toward increased and original study. Prior to participating, 3 of the 8 practitioners sustained practitioner use of writing as a therapeutic tool reported the use of journaling or writing to cope with with patients/families. Compared to the current litera- difficult situations in their own life. ese practitioners ture, both our intervention design and the study structure reported continued use of writing at follow-up and 2 ad- feature unique aspects. ditional practitioners began using writing to cope with stress One unique feature is the facilitated format of the after participation in the original study (Figure 2). 3MMM intervention. A recent systematic review of EW

The Permanente Journal·https://doi.org/10.7812/TPP/20.250 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 67 ORIGINAL RESEARCH ARTICLE Better Together: Long-term Behaviors and Perspectives after a Practitioner–Family Writing Intervention in Clinical Practice ect on our life story, which was fl

Figure 3. Practitioner use of the Three-minute Mental Makeover with patients/ ” ” families. EW = expressive writing; 3MMM = Three-minute Mental Makeover. limit it to 6, but also to re ” interventions by Nyssen et al23 found that most are non- facilitated, where a written prompt is suggested and participants are instructed to write independently. Facilitated interventions involve direct interaction with a clinician and appear to hold more therapeutic potential. e authors of the review suggested that the interpersonal component may be an important con- ” tributor to perceived benefits related to the intervention.23 EW is a form of writing characterized by self-disclosure, and this self-disclosure appears to be an important mech- fi 27-29

” anism of bene t related to EW use. Practitioner self- disclosure has been associated with increased perceived – ” trust, rapport, and satisfaction in patient practitioner re- lationships.30-33 With the 3MMM, the practitioner par- ticipates in and models completion of the exercise. is

cial because it deepens the sense of connection within the group. fi

fi method of engagement appears bene cial not only for

” patients and families, but also for practitioners. 3MMM participants consistently agreed it was important to write ”

” together with their practitioner and share what they wrote. Many 3MMM participants also shared comments en- dorsing the importance of emotional expression as a helpful

” aspect of the 3MMM activity (Table 4). e 3MMM is the first writing intervention described in ” the literature for use in medical settings by nonbehavioral ” health professionals in a variety of clinical disciplines. Other EW interventions may be time prohibitive for on-the-spot ” use within the clinical environment. Although many of these tools are multisession in structure (eg, writing for 15 minutes on 3 or 4 consecutive days), the 3MMM is brief (< 10 minutes) and requires no formal writing or behavioral health training.34-36 ective, helped remind you of what to be thankful for, reminds you what good things occurred that you may have forgot happened. fl e busy medical environment presents multiple barriers to successful implementation of an EW intervention as part ection and expression through writing fl of regular patient care. Despite the challenges of conducting m really down, I do the 3MMM and it refreshes me. ’ a writing exercise in clinical practice, the 3MMM was ranked as both feasible and beneficial by the majority of m not a big writer, but the sharing part was helpful. ’ When I It was useful because it made sure my wife andI I were on the same page in terms of our stress level. It was the verbal sharing that relieved the stress. . . . [My physician] modeling helped me do the activity with my family and was very important. I enjoyed the group aspect of the activity and thought it was most useful and bene I enjoyed the entire writing activity. It was a reminder to continue to use the activity more often. I liked sharing my feelings with my family. “ “ “ “ “ “ “ With the 3MMM, what I liked about it in particular were the 3 prompts. The story of your life in 6 words was a clever way to force us to really think in order to Putting thoughts down on paper helped destress. It was useful to getIt your was thoughts nice on to the take page a and few share minutes how to you focus feel. on better things and not stress. [The 3MMM] helped me work through and talk about my emotions, and most importantly to look forward to the future. [The 3MMM was] re “ kind of fun and helped shift the focus from some stressors to a little lighter fare. “ “ “ “ “ practitioners, in addition to being a tool that they reported Table 4. Selected family/patient participantThe 3MMM written exercise comments itself 3MMM = Three-minute Mental Makeover. Impact of self-re Interaction with the practitioner or family members to continue to use with patients after the original study.

68 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.250 ORIGINAL RESEARCH ARTICLE Better Together: Long-term Behaviors and Perspectives after a Practitioner–Family Writing Intervention in Clinical Practice

 is study was conducted an extended period of time after Disclosure Statement the original 3MMM study, providing insight into possible The authors have no conflicts of interest to disclose. long-term behavior changes. We are not aware of any prior EW studies involving patients with follow-up duration Authors’ Contributions greater than 1 year. In addition, no previous research ex- Madelaine Schaufel, MS, RD, and Douglass Moss, BS, assisted with study and amines study participants’ long-term changes in writing survey design, collected data, and assisted with manuscript development. Romana ’ Donovan, MS, RD, CCRC, assisted with study design and implementation, and behaviors or practitioners use of writing interventions. contributed to manuscript development. Yi Li, MS, assisted with study design, Sustained behavior change may be an important metric. guided statistical analysis, and contributed to manuscript development. David G Although EW interventions have been associated with mul- Thoele, MD, conceptualized and developed the 3MMM writing tool, designed and fi ff implemented the study, and directed manuscript development. All authors tiple bene ts in the short-term, most e ects faded after a few fi 13,28 14,37-39 approved the nal manuscript as submitted and agree to be accountable for all months, or changed over time. We, however, found aspects of the work. that many of the original 3MMM study participants continued to use EW to deal with stress after this single intervention. Funding The authors have no financial relationships relevant to this article. Limitations EW appears to be potentially beneficial. However, there References is no consensus regarding optimal use of it, including in- 1. Yaribeygi H, Panahi Y, Sahraei H, Johnston TP, Sahebkar A. The impact of stress on body function: A review. EXCLI J 2017 Jul;16:1057–72. DOI: https://doi.org/10.17179/ tervention design, dosing, frequency, and delivery method. excli2017-480, PMID:28900385. ere also appear to be multiple moderators of EW benefit, 2. Lepore SJ, Kliewer W. Expressive writing and health. In: Encyclopedia of behavioral medicine. Gellman MD, Turner JR, editors. New York, NY: Springer; including gender, culture, personality, writing content, 2013; p 735–41. nature of trauma/stressor, health condition, disease severity, 3. Sayer NA, Noorbaloochi S, Frazier PA, et al. Randomized controlled trial of online 7,12,23,24,28,29,38,40-47 expressive writing to address readjustment difficulties among U.S. Afghanistan and Iraq and level of available social support. Re- war veterans. J Trauma Stress 2015 Oct;28(5):381–90. DOI: https://doi.org/10.1002/jts. search related to the 3MMM likely faces similar challenges, 22047, PMID:26467326. although the current study did not evaluate these factors. 4. Milbury K, Spelman A, Wood C, et al. Randomized controlled trial of expressive writing for patients with renal cell carcinoma. J Clin Oncol 2014 Mar;32(7):663–70. DOI: https://doi. It is possible that clinical outcomes may have influenced org/10.1200/jco.2013.50.3532 the perception of care and affected the findings of perceived 5. Hevey D, Wilczkiewicz E. Changes in language use mediate expressive writing’s benefits on health-related quality of life following myocardial infarction. Health Psychol benefits of the 3MMM. However, we are unable to explore Behav Med 2014 Jan;2(1):1053–66. DOI: https://doi.org/10.1080/21642850.2014. this possibility because clinical outcomes were not evaluated 971801, PMID:25750834. 6. Craft MA, Davis GC, Paulson RM. Expressive writing in early breast cancer survivors. in the initial or follow-up studies. J Adv Nurs 2013 Feb;69(2):305–15. DOI: https://doi.org/10.1111/j.1365-2648.2012. As with any study of this type, possible limitations include 06008.x, PMID:22494086. 7. Lu Q, Zheng D, Young L, Kagawa-Singer M, Loh A. A pilot study of expressive writing convenience sampling and response bias. We sought to intervention among Chinese-speaking breast cancer survivors. Health Psychol 2012 Sep; minimize response bias by having all follow-up calls be 31(5):548–51. DOI: https://doi.org/10.1037/a0026834 conducted by a medical student researcher not involved 8. Williamson TJ, Stanton AL, Austin JE, et al. Helping yourself by offering help: Mediators of expressive helping in survivors of hematopoietic stem cell transplant. Ann Behav Med in the care of the patient. Other limitations include lack 2017 Oct;51(5):683–93. DOI: https://doi.org/10.1007/s12160-017-9892-2 of a control group and subjective measurement methods. 9. Ayers S, Crawley R, Button S, et al. Evaluation of expressive writing for postpartum health: A randomised controlled trial. J Behav Med 2018 Oct;41(5):614–26. DOI: https:// It is possible that the findings for this sample may not doi.org/10.1007/s10865-018-9970-3, PMID:30291538. be generalizable to other populations. 10. Horsch A, Tolsa JF, Gilbert L, du Cheneˆ LJ, Muller-Nix¨ C, Bickle Graz M. Improving maternal mental health following preterm birth using an expressive writing intervention: A randomized controlled trial. Child Psychiatry Hum Dev 2016 Oct;47(5):780–91. DOI: Future Directions https://doi.org/10.1007/s10578-015-0611-6, PMID:26659113. fi 11. Rini C, Austin J, Wu LM, et al. Harnessing benefits of helping others: A randomized Based on the ndings in this study, additional studies of controlled trial testing expressive helping to address survivorship problems after the 3MMM are warranted. Future areas of research might hematopoietic stem cell transplant. Health Psychol 2014 Dec;33(12):1541–51. DOI: include controlled studies comparing the 3MMM to other https://doi.org/10.1037/hea0000024 12. Ironson G, O’Cleirigh C, Leserman J, et al. Gender-specific effects of an augmented writing interventions. It may also be of interest to inves- written emotional disclosure intervention on posttraumatic, depressive, and HIV-disease- tigate the impact of 3MMM use on patient satisfaction, related outcomes: A randomized, controlled trial. J Consult Clin Psychol 2013 Apr;81(2): 284–98. DOI: https://doi.org/10.1037/a0030814, PMID:23244367. rapport, and trust in patient/family–practitioner relationships. 13. Henry EA, Schlegel RJ, Talley AE, Molix LA, Bettencourt BA. The feasibility and effectiveness of expressive writing for rural and urban breast cancer survivors. Oncol Nurs Forum 2010 Nov;37(6):749–57. DOI: https://doi.org/10.1188/10.ONF.749-757, CONCLUSION PMID:21059586. e 3MMM is a brief, guided EW intervention that 14. Carmack CL, Basen-Engquist K, Yuan Y, et al. Feasibility of an expressive-disclosure fi group intervention for post-treatment colorectal cancer patients: Results of the healthy appears to have short- and long-term bene ts for patients, expressions study. Cancer 2011 Nov;117(21):4993–5002. DOI: https://doi.org/10.1002/ families, and practitioners. Additional studies may be cncr.26110, PMID:21480203. 15. Willmott L, Harris P, Gellaitry G, Cooper V, Horne R. The effects of expressive writing helpful in determining how the 3MMM can best be used by following first myocardial infarction: A randomized controlled trial. Health Psychol 2011 practitioners with their patients. v Sep;30(5):642–50. DOI: https://doi.org/10.1037/a0023519

The Permanente Journal·https://doi.org/10.7812/TPP/20.250 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 69 ORIGINAL RESEARCH ARTICLE Better Together: Long-term Behaviors and Perspectives after a Practitioner–Family Writing Intervention in Clinical Practice

16. Frisina PG, Borod JC, Lepore SJ. A meta-analysis of the effects of written emotional 32. Zink KL, Perry M, London K, et al. “Let me tell you about my . . .”: Provider self-disclosure disclosure on the health outcomes of clinical populations. J Nerv Ment Dis 2004 Sep;192(9): in the emergency department builds patient rapport. West J Emerg Med 2017 Jan;18(1): 629–34. DOI: https://doi.org/10.1097/01.nmd.0000138317.30764.63, PMID:15348980. 43–9. DOI: https://doi.org/10.5811/westjem.2016.10.31014, PMID:28116007. 17. Gellaitry G, Peters K, Bloomfield D, Horne R. Narrowing the gap: The effects of an 33. Hill CE, Knox S, Pinto-Coelho KG. Therapist self-disclosure and immediacy: A qualitative expressive writing intervention on perceptions of actual and ideal emotional support in meta-analysis. Psychotherapy 2018 Dec;55(4):445–60. DOI: https://doi.org/10.1037/ women who have completed treatment for early stage breast cancer. Psycho Oncol 2010 pst0000182 Jan;19(1):77–84. DOI: https://doi.org/10.1002/pon.1532 34. Pennebaker JW, Beall SK. Confronting a traumatic event: Toward an understanding of 18. Meston CM, Lorenz TA, Stephenson KR. Effects of expressive writing on sexual inhibition and disease. J Abnorm Psychol 1986 Aug;95(3):274–81. DOI: https://doi.org/10. dysfunction, depression, and PTSD in women with a history of childhood sexual abuse: 1037//0021-843x.95.3.274, PMID:3745650. Results from a randomized clinical trial. J Sex Med 2013 Sep;10(9):2177–89. DOI: https:// 35. Pennebaker JW, Kiecolt-Glaser JK, Glaser R. Disclosure of traumas and immune doi.org/10.1111/jsm.12247, PMID:23875721. function: Health implications for psychotherapy. J Consult Clin Psychol 1988 Apr;56(2): 19. Broderick JE, Stone AA, Smyth JM, Kaell AT. The feasibility and effectiveness of an 239–45. DOI: https://doi.org/10.1037//0022-006x.56.2.239, PMID:3372832. expressive writing intervention for rheumatoid arthritis via home-based videotaped 36. Pennebaker JW, Seagal JD. Forming a story: The health benefits of narrative. J Clin instructions. Ann Behav Med 2004 Feb;27(1):50–9. DOI: https://doi.org/10.1207/ Psychol 1999 Oct;55(10):1243–54. DOI: https://doi.org/10.1002/(sici)1097-4679(199910) s15324796abm2701_7 55:10<1243::aid-jclp6>3.0.co;2-n, PMID:11045774. 20. Lepore SJ, Revenson TA, Roberts KJ, Pranikoff JR, Davey A. Randomised controlled trial 37. McGuire KM, Greenberg MA, Gevirtz R. Autonomic effects of expressive writing in of expressive writing and quality of life in men and women treated for colon or rectal individuals with elevated blood pressure. J Health Psychol 2005 Mar;10(2):197–209. DOI: cancer. Psychol Health 2015 Mar;30(3):284–300. DOI: https://doi.org/10.1080/08870446. https://doi.org/10.1177/1359105305049767, PMID:15723890. 2014.971798 38. Baddeley JL, Pennebaker JW. A postdeployment expressive writing intervention for 21. Lepore SJ, Greenberg MA, Bruno M, et al. Expressive writing and health: Self-regulation of military couples: A randomized controlled trial. J Trauma Stress 2011 Oct;24(5):581–5. emotion-related experience, physiology, and behavior. In: Lepore, SJ, Smyth JM, eds. The DOI: https://doi.org/10.1002/jts.20679, PMID:21887713. writing cure: How expressive writing promotes health and emotional well-being (p. 99–117). 39. Baikie KA, Geerligs L, Wilhelm K. Expressive writing and positive writing for American Psychological Association. 2002. p 99-117. DOI: https://doi.org/10.1037/10451-005. participants with mood disorders: An online randomized controlled trial. J Affect 22. Zhou C, Wu Y, An S, Li X. Effect of expressive writing intervention on health outcomes in Disord 2012 Feb;136(3):310–9. DOI: https://doi.org/10.1016/j.jad.2011.11.032, breast cancer patients: A systematic review and meta-analysis of randomized controlled PMID:22209127. trials. PLoS One 2015 Jul;10(7):e0131802. DOI: https://doi.org/10.1371/journal.pone.0131802 40. Frederiksen Y, O’Toole MS, Mehlsen MY, et al. The effect of expressive writing 23. Nyssen OP, Taylor SJ, Wong G, et al. Does therapeutic writing help people with long-term intervention for infertile couples: A randomized controlled trial. Hum Reprod 2017 Feb; conditions? Systematic review, realist synthesis and economic considerations. Health 32(2):391–402. DOI: https://doi.org/10.1093/humrep/dew320 – Technol Assess 2016 Apr;20(27):17 192. DOI: https://doi.org/10.3310/hta20270 41. Zakowski SG, Herzer M, Barrett SD, Milligan JG, Beckman N. Who benefits from 24. Smith HE, Jones CJ, Hankins M, et al. The effects of expressive writing on lung function, emotional expression? An examination of personality differences among gynaecological quality of life, medication use, and symptoms in adults with asthma: A randomized cancer patients participating in a randomized controlled emotional disclosure intervention controlled trial. Psychosom Med 2015 May;77(4):429–37. DOI: https://doi.org/10.1097/ trial. Br J Psychol 2011 Aug;102(3):355–72. DOI: https://doi.org/10.1348/ PSY.0000000000000166, PMID:25939030. 000712610X524949 25. Lemay M, Encandela J, Sanders L, Reisman A. Writing well: The long-term effect on empathy, 42. Lu Q, Stanton AL. How benefits of expressive writing vary as a function of writing observation, and physician writing through a residency writers’ workshop. J Grad Med Educ instructions, ethnicity and ambivalence over emotional expression. Psychol Health 2010 2017 Jun;9(3):357–60. DOI: https://doi.org/10.4300/JGME-D-16-00366.1, PMID:28638517. Jul;25(6):669–84. DOI: https://doi.org/10.1080/08870440902883196 26. Thoele DG, Gunalp C, Baran D, et al. Health care practitioners and families writing 43. Milbury K, Lopez G, Spelman A, et al. Examination of moderators of expressive writing together: The Three-minute Mental Makeover. Perm J 2020 Nov;24:19.056. DOI: https:// in patients with renal cell carcinoma: The role of depression and social support. doi.org/10.7812/TPP/19.056, PMID:31852046. Psycho Oncol 2017 Sep;26(9):1361–8. DOI: https://doi.org/10.1002/pon.4148, PMID: 27. Laccetti M. Expressive writing in women with advanced breast cancer. Oncol Nurs Forum 2007 27145447. Sep;34(5):1019–24. DOI: https://doi.org/10.1188/07.ONF.1019-1024, PMID:17878130. 44. Jensen-Johansen MB, O’Toole MS, Christensen S, et al. Expressive writing intervention 28. Averill AJ, Kasarskis EJ, Segerstrom SC. Expressive disclosure to improve well-being in and self-reported physical health outcomes: Results from a nationwide randomized patients with amyotrophic lateral sclerosis: A randomised, controlled trial. Psychol Health controlled trial with breast cancer patients. PLoS One. 2018 Feb;13(2):e0192729. DOI: 2013 Jun;28(6):701–13. DOI: https://doi.org/10.1080/08870446.2012.754891 https://doi.org/10.1371/journal.pone.0192729, PMID:29474441. 29. Niles AN, Haltom KE, Mulvenna CM, Lieberman MD, Stanton AL. Randomized controlled 45. Chu Q, Wong CCY, Lu Q. Acculturation moderates the effects of expressive writing on trial of expressive writing for psychological and physical health: The moderating role of post-traumatic stress symptoms among Chinese American breast cancer survivors. Int J emotional expressivity. Hist Philos Logic 2014 Jan;27(1):1–17. DOI: https://doi.org/10. Behav Med 2019 Apr;26(2):185–94. DOI: https://doi.org/10.1007/s12529-019-09769-4, 1080/10615806.2013.802308, PMID:23742666. PMID:30656609. 30. Unhjem JV, Vatne S, Hem MH. Transforming nurse–patient relationships: A qualitative 46. Hoyt MA, Austenfeld J, Stanton AL. Processing coping methods in expressive study of nurse self-disclosure in mental health care. J Clin Nurs 2018 Mar;27(5–6): essays about stressful experiences: Predictors of health benefit. J Health Psychol e798–807. DOI: https://doi.org/10.1111/jocn.14191, PMID:29193417. 2016 Jun;21(6):1183–93. DOI: https://doi.org/10.1177/1359105314550347, PMID: 31. Nazione S, Perrault EK, Keating DM. Finding common ground: Can provider–patient race 25266296. concordance and self-disclosure bolster patient trust, perceptions, and intentions? 47. Schmidt S, Hahm S, Freitag S. Writing interventions in older adults and former children of J Racial Ethn Health Disparities 2019 Oct;6(5):962–72. DOI: https://doi.org/10.1007/ the World War II: Impact on quality of life and depression. Aging Ment Health 2018 Aug; s40615-019-00597-6, PMID:31119610. 22(8):1017–24. DOI: https://doi.org/10.1080/13607863.2017.1334036

70 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.250 n ORIGINAL RESEARCH ARTICLE Multistage Adolescent Depression Screening: A Comparison of 11-Year-Olds to 12-Year-Olds

Alan B Cortez, MD1; Julia Wilkins, BS2; Eric Handler, MD3; Marc A Lerner, MD4; Raoul Burchette, MS5; Lawrence S Wissow, MD6 Perm J 2021;25:20.233 E-pub: 3/3/2021 https://doi.org/10.7812/TPP/20.233

ABSTRACT about depression and suicide should inform universal Introduction: Adolescent depression screening is recommended screening strategies.6,7 eUSPreventiveServicesTask starting at age 12 years, but younger children experience depression Force found insufficient data to make a determination to as well. Our objective was to determine whether screening for perform 11-year-old screening, prompting the need to validate depression at age 11 years yields similar results to screening at tools and perform outcomes research for this age group.3 age 12 years. e medical group in this report (Southern California Methods: We conducted a retrospective chart review of 1000 Permanente Medical Group) is associated with a large health 11- and 12-year-olds in multiple pediatric offices of a large-group practice associated with a health maintenance organization in maintenance organization (Kaiser Foundation Health Plan) Southern California. All offices used a multistage depression and has screened thousands of 11- and 12-year-olds for de-  fi screening process during well-child visits using the Patient pression since June 2015. especi caimofthisstudyisto Health Questionnaire for Adolescents, the global depression determine whether there are clinically important differences in inquiry within a parent questionnaire, a chart-based review of the process, results, and outcomes of depression screening in mental health history, and brief patient/parent interview in- 11-year-olds compared to 12-year-olds. We hypothesized that formed by the first 3 elements. all observed differences in study parameters would be clinically Results: The 11- and 12-year-old cohorts had similar com- and statistically insignificant. pletion rates for the Patient Health Questionnaire for Adolescents (99.2% vs 97.8%, P = 0.06), with similar mean total Patient Health METHODS P Questionnaire for Adolescents scores (2.12 vs 2.22, = 0.48). There Study Population was no significant difference for positive screenings determined P We performed chart reviews on patients of 60 pediatricians by the pediatrician (12.0% vs 16.0%, = 0.07), but parents of ffi 12-year-olds were more likely have concerns for their child’s who practiced in 14 medical o ces of a large-group practice. mood (6.8% vs 10.5%, P = 0.04). There were similar percentages of e pediatricians had received brief training and written referrals (6.2% vs 8.8%, P = 0.12), beneficial conversations related information on depression screening methods and inter- to depression and anxiety, (4.5% vs 4.8%, P = 0.85), and new pretation of results. ere was no expected difference in mental health diagnoses (2.0% vs 2.3%, P = 0.79). the content, approach, or documentation for well-visits of Discussion: The process, results, and outcomes of screenings 11- and 12-year-olds in the workflow of this medical group. are similar for 11- and 12-year-olds, with a tendency toward more Data were extracted from electronic health records (EHRs) fi positive ndings in 12-year-olds. of well-visits of 11- and 12-year-olds occurring from July 1, Conclusion: Multistage depression screening in 11-year-olds 2016, through March 28, 2017. Visits were chosen ran- can be applied successfully in clinical practice, with most cases domly within each age group, aiming for the final study identifying youths without a prior mental health diagnosis. sample with 60% of the charts (n = 600) from 11-year-olds and 40% (n = 400) from 12-year-olds. e total sample size fi INTRODUCTION was chosen as a signi cant but manageable amount given the need for manual chart reviews on all subjects, and e prevalence of major depressive episodes increases through adolescence, peaking at 17%, but is already 5% by age 12 years.1 A recent analysis of 5- to 18-year-olds showed ffi that suicidal thoughts or attempts accounted for 3.5% of Author A liations 1Department of Pediatrics, Southern California Permanente Medical Group, Inc, Tustin, CA all their emergency department visits in the US (more than 2School of Medicine, New York Medical College, Valhalla, NY 1 million visits/y). ese visits doubled from 2007 to 2015, 3Orange County Health Care Agency, Santa Ana, CA and more than 40% were in the 5- to 11-year-old subgroup.2 4Department of Pediatrics, University of California at Irvine, Irvine, CA 5Department of Research and Evaluation, Southern California Permanente Medical Group, Inc, Pasadena, CA Screening for adolescent depression starting at age 12 years 6Division of Child and Adolescent Psychiatry, Department of Psychiatry and Behavioral Sciences, University of has been recommended by organizations such as the US Washington, Seattle, WA Preventive Services Task Force,3 the National Committee for Quality Assurance,4 and the American Academy of Corresponding Author Alan B Cortez, MD ([email protected]) Pediatrics,5 but it is not known how early-in-life concerns Keywords: adolescent, depression, screening The Permanente Journal·https://doi.org/10.7812/TPP/20.233 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 71 ORIGINAL RESEARCH ARTICLE Multistage Adolescent Depression Screening: A Comparison of 11-Year-Olds to 12-Year-Olds

the age distribution matched the frequency of well-visits byageinthispractice(approximately 3500 well-visits at age 11 years and 2500 well-visits at age 12 years). Twelve- year-olds were excluded if they had a previous screening as an 11-year-old, so only initial screenings were compared.

Clinical Depression Screening Process e EHRs supported the screening by increasing the visibility of important entries in the medical record, pro- viding templates for documentation, and prompting nurses and providers to enter results prior to closing the chart. e screening process was designed to improve accuracy by using several tools simultaneously and synthesizing the information with human interpretation. With the published sensitivity (75%–90%) and specificity (85%–90%) of the Patient Health Questionnaire for Adolescents (PHQ-A),8-11 Figure 1. English language wording on form used for Patient Health Questionnaire for Adolescents. On questions 1 through 9, “not at all” is scored as 0, “several days” the core initial step in the process, and an expected prevalence is scored as 1, “more than half the days” is scored as 2, and “nearly every day” is 1 of depression in the screened population of ≤ 5%, there likely scored as 3. The last 4 questions are referred to in this article as questions 10 would have been a positive predictive value in the range through 13, but they are not numbered in the actual questionnaire and do not contribute to the total score. To analyze the data, questions 10, 12, and 13 were of only 10% to 30% using this test alone. Furthermore, scored as 0 for “no” and 1 for “yes”, and the 4 responses to question 11 were depression screening tools without suicidality and parent scored from 0 to 3. assessment, although helpful, miss cases of adolescent depression.12 e screening thus consisted of 4 components: 1. e PHQ-A tool administered on paper to the adolescent data fields. Reading notes was necessary to determine in English or Spanish. e PHQ-A includes the Patient whether the provider considered the screening process to Health Questionnaire-9 plus 4 additional questions re- be “positive” and developed a treatment plan. To ensure lated to duration of depression, severity/functionality, and consistent methodology between the 2 reviewers, they suicide risk (Figure 1).6-11,13 eparentwasinstructedto extracted data independently from the first 25 charts, avoid giving any assistance and was told that results would then reviewed their results mutually. During subsequent be discussed later with the provider. reviews, the primary reviewer for a chart consulted with 2. Question administered to parent: “Does your child often the other reviewer for all positive or questionable results. appear sad, depressed, or anxious?” is question (given Screening steps were only considered “positive” if both in parent’s preferred language when available) was part of a reviewers agreed. larger questionnaire adapted from the American Academy After comparing the age groups for demographic simi- of Pediatrics and administered routinely at well-visits.14 larity, we sought data on frequency of performing each 3. Review of the patient’s mental health history over 3 years component of the screening process. For the PHQ-A, we (recent history) and evidence of treatment within 6 months also evaluated whether it was performed correctly (specif- (active history). is information came from the EHR and ically, were all questions answered, documented, and scored included dates of appointments, mental health di- correctly, and was there evidence the provider addressed the agnoses and problem listings, and prescribed psychotropic results). medications. We extracted and compared the 11- and 12-year-old age 4. Interview with patient and parent using the previously groups for the total PHQ-A score (range, 0–27), specific noted data to clarify potential mental health dysfunction answers to each of the 13 PHQ-A questions (scored 0–3), and to develop a plan of action. For example, clarifying a result of the parent question, and whether a recent or active strongly positive response to a single PHQ-A question mental health history existed. A positive screen based solely (especially when it was not associated with a high total on these objective data was defined as a yes response to any PHQ-A score) might provide enhanced information on of the following categories: psychosocial function.9,10 1. Total score of the PHQ-A≥ 10, matching the cutoff currently used by the National Committee on Quality Data Extraction Assurance for ages 12 years and older.4 Data extraction required manual chart reviews because 2. PHQ-A questions 12 or 13 > 0 (suicidal ideation and some steps in screening are not recorded in extractable plan),

72 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.233 ORIGINAL RESEARCH ARTICLE Multistage Adolescent Depression Screening: A Comparison of 11-Year-Olds to 12-Year-Olds

Table 1. Demographics by age Characteristic Total (N = 1000) Age 11 (n = 600) Age 12 (n = 400) n (%) n (%) n (%) P value Mean age, y 11.72 (0.58)a 11.32 (0.30)a 12.33 (0.28)a Gender 0.06 Male 466 (46.6) 294 (49.0) 172 (43.0) Female 534 (53.4) 306 (51.0) 228 (57.0) Gender of pediatrician 0.22 Male 340 (34.0) 213 (35.5) 127 (31.8) Female 660 (66.0) 387 (64.5) 273 (68.3) Ethnicity 0.91 White 286 (28.6) 175 (29.2) 111 (27.8) Hispanic 441 (44.1) 259 (43.2) 182 (45.5) Asian 176 (17.6) 109 (18.2) 67 (16.8) African American 19 (1.9) 12 (2.0) 7 (1.8) Other 78 (7.8) 45 (7.5) 33 (8.3) Payor 0.36 Private 688 (68.8) 423 (70.5) 265 (66.3) Affordable Care Act 86 (8.6) 48 (8.0) 38 (9.5) Medicaid 226 (22.6) 129 (21.5) 97 (24.3) aMean (standard deviation).

3. PHQ-A responses≥ 2 to questions 1, 2, 9, or 11, or yes 4. Advice or planning for a psychosocial issue that did on question 10. ese questions are more focused on not meet criteria for a positive determination of depression than questions 3 through 8, which deal with depression less-specific adolescent symptoms.10,11 4. Positive answer to parent question Statistical Analysis 5. Active mental health history present Pearson χ2 tests were used for comparisons between 6. Recent mental health history present groups, and the Mann-Whitney test was used for PHQ-A 7. In some cases when the PHQ-A results were not score totals. High P values indicated a lack of statistical recorded, but the provider note stated it was positive difference between 11- and 12-year-olds. We also recorded and performed an age group com- parison on the provider’s comprehensive determination of RESULTS possible depression, which could differ from the results of e mean ages (Table 1) of the 11- and 12-year-old the instruments and chart review, and whether a mental populations were 11.3 and 12.3 years, respectively. ere health diagnosis had been assigned at the end of the visit. was no significant difference between 11- and 12-year-olds Treatment plans extracted from the medical record and in the other demographic characteristics we studied, al- compared by age groups included emergency interven- though patient gender and pediatrician gender for the whole tions, documentation of potentially beneficial mental group tended toward more girls and women, respectively. health conversations related to both positive and negative e study population reflected the ethnic distribution in screens, and mental health referrals made (and if com- this practice as a whole, although it differed from the local pleted). e finding that a beneficial conversation had population, which has more whites (44%) compared with occurred required both reviewers to agree and were de- Latinos (28%).15 From the EHR data or interview, 3% of fined as documentation of conversations resulting in one theseadolescentswereknowntobeinactivetreatment or more of the following: for a mental health diagnosis other than Attention Deficit 1. A positive determination when the objective data were Hyperactivity Disorder or Autism Spectrum Disorder at the negative time of screening, including 0.8% who were being treated 2. Mental health referral actively for depression. 3. Advice or planning related to depression, anxiety, or e PHQ-A was administered 98.6% of the time. adjustment disorder diagnoses that did not lead to a ere was only one documented parental refusal (of an referral 11-year-old). e PHQ-A was also not performed on

The Permanente Journal·https://doi.org/10.7812/TPP/20.233 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 73 ORIGINAL RESEARCH ARTICLE Multistage Adolescent Depression Screening: A Comparison of 11-Year-Olds to 12-Year-Olds

Table 2. Total Patient Health Questionnaire for Adolescents score and positive responses (1, 2, or 3) for the screened population on each Patient Health Questionnaire for Adolescents question compared by age Total (N = 888)a Age 11 (n = 537) Age 12 (n = 351) PHQ-A result n (%) n (%) n (%) P value PHQ-A mean total score 2.16 (2.98)b 2.12 (2.95)b 2.22 (3.02)b 0.48 Total score of 0 379 (42.7) 239 (44.5) 140 (39.9) 0.44 Total score range of 1–4 361 (40.7) 209 (38.9) 152 (43.3) 0.09 Total score range of 5–9 117 (13.2) 73 (13.6) 44 (12.5) 0.21 Total score range ≥ 10 31 (3.5) 16 (3.0) 15 (4.3) 0.14 All above scoring ranges compared N = 877c n = 528 n = 349 0.35 PHQ-A question #1 112 (12.8) 62 (11.7) 50 (14.3) 0.26 PHQ-A question #2 179 (20.4) 104 (19.7) 75 (21.5) 0.52 PHQ-A question #3 253 (28.8) 140 (26.5) 113 (32.4) 0.06 PHQ-A question #4 138 (15.7) 79 (15.0) 59 (16.9) 0.44 PHQ-A question #5 210 (23.9) 116 (22.0) 94 (26.9) 0.09 PHQ-A question #6 108 (12.3) 59 (11.2) 49 (14.0) 0.21 PHQ-A question #7 196 (22.3) 127 (24.1) 69 (19.8) 0.14 PHQ-A question #8 84 (9.6) 51 (9.7) 33 (9.5) 0.92 PHQ-A question #9 36 (4.1) 24 (4.5) 12 (3.4) 0.42 PHQ-A question #10 83 (9.5) 42 (8.0) 41 (11.7) 0.06 PHQ-A question #11 140 (16.0) 81 (15.3) 59 (16.9) 0.54 PHQ-A question #12 10 (1.1) 5 (0.9) 5 (1.4) 0.51 PHQ-A question #13 2 (0.2) 2 (0.4) 0 (0.0) 0.25 a. Calculations exclude adolescents with undocumented total scores. b. Mean (standard deviation). c. Calculations exclude adolescents with undocumented answers to the questions. PHQ-A = Patient Health Questionnaire for Adolescents.

13 others because of intellectual disability (5 adolescents) Figure 2 summarizes the information used by the pro- or staff error (8 adolescents). Overall, 76.3% of the time viders to make their determination of a positive or nega- (data not shown) there was documentation the PHQ-A was tive screen, and provides data for the combined 11- and performed accurately by staff (recording all 13 answers 12-year-old screening outcomes. We found that 13.6% andscoringcorrectly)andanalyzedbythepediatrician. of screens (11.5% for adolescents not currently in treatment) e parent question was performed and documented were deemed positive, although a smaller percentage of successfully in both age groups 99.7% of the time. We adolescents (7.2%) was referred formally to a mental health could not determine the frequency of successful perfor- specialist. Of note, for 9 of 30 adolescents with an active mance of the mental health history and the interview mental health history in the EHR, the provider did not becausetheywerefrequentlynotdocumentedifthere- document its presence and the screen was deemed negative sults were negative. ere was no observed difference by the provider. (data not shown) between 11- and 12-year-olds for the We also compared by age the various ways the screening process of performing the PHQ-A screening (99.2% vs process could lead the provider to determine an ado- 97.8%, P = 0.06), documenting accurate completion of all lescent had possible depression (Table 3). Only the PHQ-A elements (76.0% vs 76.8%, P =0.79),and parent question showed a higher degree of positivity in documenting accurate completion of the entire multi- 12-year-olds (P = 0.04). We also found no age differ- stage screening process except for the interview (75.2% vs ence with combinations of categories we analyzed and 75.5%, P =0.91). no age difference when looking at those found positive by ere was no statistical difference between 11- and interview. 12-year-old results on the PHQ-A (Table 2) for the average Table 4 shows data on treatment plans that resulted from total score (2.12 vs 2.22, P = 0.48); percent of responses with the screening. Only 1 of 136 children (a 12-year-old) with the total score grouped by ranges of 0, 1 through 4, 5 through a positive screen was sent for an emergency intervention 9, or ≥ 10 (P = 0.35); and individual responses to each of the because of suicidal concerns. Most positive screens (115 of 13 questions (P =0.06–0.92). 136) represented new findings; active mental health history

74 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.233 ORIGINAL RESEARCH ARTICLE Multistage Adolescent Depression Screening: A Comparison of 11-Year-Olds to 12-Year-Olds

Figure 2. Provider actions within the depression screen at 1000 well-visits for 11- and 12-year-olds.

Table 3. Results of each component of the depression screen compared by age Total (N = 1000) Age 11 (n = 600) Age 12 (n = 400) Screen components n (%) n (%) n (%) P value Any positive from all 8 screen categories 243 (24.3) 144 (24.0) 99 (24.8) 0.79 Any positive from the 7 objective screen categories 217 (21.7) 128 (21.3) 89 (22.3) 0.73 Any positive from the 4 PHQ-A categories 151 (15.1) 87 (14.5) 64 (16.0) 0.52 PHQ-A score ≥ 10 31 (3.1) 16 (2.7) 15 (3.8) 0.33 PHQ-A by questions 1, 2, 9, 10, or 11 positive 127 (12.7) 74 (12.3) 53 (13.3) 0.67 PHQ-A by questions 12 or 13 positive 11 (1.1) 6 (1.0) 5 (1.3) 0.71 PHQ-A positive by provider but not recorded 16 (1.6) 8 (1.3) 8 (2.0) 0.41 Parent screen positive 83 (8.3) 41 (6.8) 42 (10.5) 0.04 Active mental health history positive 30 (3.0) 18 (3.0) 12 (3.0) 1.00 Recent mental health history positive 53 (5.3) 34 (5.7) 19 (4.8) 0.53 Interview positive when objective screen negative 26 (2.6) 16 (2.7) 10 (2.5) 0.87 PHQ-A = Patient Health Questionnaire for Adolescents.

was not present in 85% of positive screens. ere were no without referrals (8.5% in 11-year-olds, 9.1% in 12-year-olds). age differences in the number of mental health referrals e discussions concerned depression, anxiety, and various generated, mental health appointments completed, mental psychosocial issues that were revealed in both positive and health diagnoses, and specific depression diagnoses. In- negative screens. cluding the cases diagnosed previously, we found a prevalence of 1.7% for depression and 5.1% for mental DISCUSSION health diagnoses overall (excluding Attention Deficit Screening 11-year-olds for depression using a multistep Hyperactivity Disorder or Autism Spectrum Disorder). approach that included the PHQ-A was feasible in this Some adolescents with recent but not active mental health real-world clinical practice. e processes and outcomes history (8 of 23) were referred back for further treatment, and for 11- and 12-year-olds were very similar, although parents of 1 child in active treatment for an adjustment disorder was 12-year-olds were more likely to report concerns about their referred specifically to evaluate for depression. e 2 age child. We did not investigate potential reasons for this dis- groups had a similar number of beneficial conversations parity on the parental question. Still, this singular difference

The Permanente Journal·https://doi.org/10.7812/TPP/20.233 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 75 ORIGINAL RESEARCH ARTICLE Multistage Adolescent Depression Screening: A Comparison of 11-Year-Olds to 12-Year-Olds

Table 4. Outcomes of the screening program for the entire study population compared by age Total (N = 1000) Age 11 (n = 600) Age 12 (n = 400) Outcome % (95% CI) % (95% CI) % (95% CI) P value Positive screen per pediatrician 136 (13.6) 72 (12.0) 64 (16.0) 0.07 New referrals sent 72 (7.2) 37 (6.2) 35 (8.8) 0.12 Referrals completed successfully 26 (2.6) 13 (2.2) 13 (3.3) 0.29 Beneficial conversation with positive screena 46 (4.6) 27 (4.5) 19 (4.8) 0.85 Beneficial conversation with negative screen 39 (3.9) 22 (3.7) 17 (4.3) 0.64 New mental health diagnosis 21 (2.1) 12 (2.0) 9 (2.3) 0.79 Preexisting mental health diagnosis 30 (3.0) 18 (3.0) 12 (3.0) 1.00 New depression diagnosis 9 (0.9) 6 (1.0) 3 (0.8) 0.68 Preexisting depression diagnosis 8 (0.8) 3 (0.5) 5 (1.3) 0.19 a.Excludes cases with mental health referrals. CI = confidence interval.

did not affect the clinical decision making (provider deter- with the family that included psychoeducation, and often minations and interventions) for the parameters we measured. plans for follow-up care. In addition, the PHQ-A questions that were closest to a Beneficial conversations (n = 2) and referrals (n = 1) were significant difference involved sleep, fatigue, and depression, not commonly seen among the 21 cases with recognized all of which are expected to increase with age in adolescence.1,16 active mental health history, suggesting the providers e PHQ-A was chosen as the initial step in the screening generally did not invest additional time when they believed process because of its extensive use in clinical settings and as a the adolescent was already receiving care. e decreased public health quality indicator, its ease of administration in an mental health engagement with these adolescents may have office setting, and its focus on depression, rather than a wider also been compounded by an EHR function that blocks array of psychosocial issues that were already screened for content of most notes from mental health professionals. during well-visits with a comprehensive parent question- Conversations with potential benefit also occurred in naire and interview.4,6,8-10,11,13 Despite its widespread use, visits with negative depression screens when the process the PHQ-A has not been validated formally for use in uncovered other concerns. However, we also found cases in children younger than 12 years old.4,6 Based on our expec- which mental health histories apparently were not noticed by tations of developmental capabilities at age 11,17,18 we were providers and the screening process was considered negative. not surprised that this age group performed in an equivalent ese cases may represent failures to integrate somatic and fashion to 12-year-olds, with similar responses to specific mental health care or to support adherence to mental health questions, similar mean scores, and similar lack of ex- treatment. It is possible that future modifications to the treme results (only 3 recorded scores > 14 in both groups EHR, such as more visibility of therapist notes and notifi- combined). cations about mental health history during well-visits, could Depression screening for 11-year-olds has been in make it less likely that this data type would be missed. existence for several years in the study practice and has Importantly, when referrals were made, about one-third now expanded to other pediatrics and family medicine (36%) were completed, even in this highly integrated system. offices in this medical group, in which approximately We do not know the extent to which this reflects real or 25,000 11-year-olds are now screened annually with this perceived barriers, or whether families felt the referrals were not process throughout Southern California. We did not in- necessary either because they doubted the diagnosis or because vestigate formally the possible harm to these adolescents or they found the pediatrician’scounselingsufficiently helpful. their families from the screening process or from conse- Validations of screening tools have centered on a score quences of incorrect screen determinations. Still, we did not for that tool in relation to a depression diagnosis, but not come across any evidence of harm within the chart reviews, whether a provider has mental health concerns about a nor did we learn about evidence of harm from the providers particular adolescent.6,8-10,11,13 e concept that a PHQ-A in this medical practice with whom we discussed it. score ≥ 10 by itself is inadequate, because a single screening We found that pediatricians did not refer many of the tool was supported by our data finding a PHQ-A score ≥ 10 adolescents they newly identified (44 of 115) via the in only 3.1% of the study group, whereas providers de- screening. is proportion did not differ by age. In all these termined positive screens in 13.6% and referred 7.2%. In nonreferred cases, the EHR documented a conversation addition, of the 31 cases of PHQ-A scores ≥ 10, only

76 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.233 ORIGINAL RESEARCH ARTICLE Multistage Adolescent Depression Screening: A Comparison of 11-Year-Olds to 12-Year-Olds

3 patients were diagnosed with depression and 4 were be introduced readily into clinical practice, with similar diagnosed with anxiety or an adjustment disorder. A more processes, results, and outcomes as for 12-year-olds. Better appropriate paradigm for 11- and 12-year-olds may be to identification of depression, leading to treatment even 1 year validate the concern of the pediatrician who should “cast a earlier, may be beneficial toward the goal of preventing suicide wider net,” rather than seek only depression diagnoses and attempts and emergency department visits at age 11 years. recognize suicidal ideation. Secondary benefits include making other mental health di- e lower prevalence of depression and suicidal ideation agnoses, providing beneficial conversations on a variety of in younger adolescents compared to older teens was con- mental health issues, reactivating mental health care, and firmed by the low volumes we found for diagnoses of de- identifying children for whom closer monitoring may be pression and emergency interventions.1 Along with the high appropriate. We hope these findings will lead other medical frequency of beneficial conversations, these findings support practices to consider adopting a depression screening protocol the concept that a major purpose of screening younger at 11-year-old well-visits, leading to prospective studies on populations is to determine mental health dysfunction at screening 11-year-olds to delineate benefits and harms further, earlier stages, rather than finding only those patients with and eventually leading to public health recommendations to established depression or suicidal ideation.6 In this regard, screen all 11-year-olds for depression. v positive screens that do not lead to an immediate depression diagnosis, or referrals of patients who do not attend mental Disclosure Statement fl health appointments, are not necessarily a failing of de- The authors have no con icts of interest to disclose. pression screening programs and may actually constitute a ’ fi Authors Contributions long-term bene t as these children are followed over time. Alan B Cortez, MD, conceptualized and designed the study, created the data ese data also demonstrate the success of the screening collection instruments, reviewed the medical records, collected the data, drafted the in identifying young adolescents who were not previously initial manuscript, and maintained full control of the database. Julia Wilkins, BS, known to have potential mental health problems. In ad- participated in the design of the study, reviewed the medical records, collected the fi data, drafted the initial manuscript, and maintained full control of the database. The dition, a possible added bene t was the ability to identify first two authors performed all chart reviews. Raoul Burchett, MS, participated in the patients with past treatment and what now appeared to be design of the study and the data collection instruments, maintained full control of persistent or recurring problems, leading to reactivation of the database, performed all statistical analyses, and drafted part of the initial manuscript. Eric Handler, MD; Marc A Lerner, MD; and Lawrence S Wissow, MD; their mental health care. participated in the conceptualization and design of the study, interpretation of the Limitations of this study relate primarily to interpretation data, and revision of the manuscript. All authors approved the final manuscript as of data, obtained in large part by manual chart reviews, submitted and agree to be accountable for all aspects of the work. from a retrospective, real-world clinical setting not designed prospectively for research. is study population may differ Funding No external funding or sponsorship was secured for this study. from other regional populations regarding gender of ado- lescent, gender of pediatrician, distribution of ethnicities, References and distribution of payors. It is possible that the parent and 1. National Survey on Drug Use and Health. Major depression. National Institute of Mental child filled out the tool together, and individualized Health; 2016. Accessed November 2018, www.nimh.nih.gov/health/statistics/major- depression.shtml nursing approaches for introducing the PHQ-A may have 2. Burstein B, Agostino H, Greenfield B. Suicidal attempts and ideation among children and affected responses. ere was absent documentation of some adolescents in US emergency departments, 2007–2015. JAMA Pediatr 2019 Jun;173(6): fi fi 598–600. DOI: https://doi.org/10.1001/jamapediatrics.2019.0464, PMID:30958529. ndings we sought; we could not determine all bene cial 3. Siu AL and others on behalf of the US Preventive Services Task Force. Screening for conversations or whether there may have been an undoc- depression in children and adolescents: US Preventive Services Task Force  ffi recommendation statement. Pediatrics 2016 Mar;137(3):e20154467. DOI: https://doi.org/ umented mental health referral. ere was insu cient 10.1542/peds.2015-4467 documentation and power in the sample to allow for the 4. National Committee for Quality Assurance. The health effectiveness information and data set. (2018) Accessed November 2018, www.ncqa.org/hedis-quality-measurement/hedis- correlation of screening outcomes with known depression measures/hedis-2018 risk factors. It is possible that borderline P values may have 5. Committee on Practice and Ambulatory Medicine, Bright Futures Periodicity Schedule shown a different level of significance with a larger sample Workgroup. 2106 Recommendations for Preventive Pediatric Health Care. Pediatrics Jan 2016, 137(1) e20153908 size or a prospective study design. 6. Zuckerbrot RA, Cheung A, Jensen PS, et al. Guidelines for Adolescent Depression in Primary Care (GLAD-PC): Part I: Practice preparation, identification, assessment, and initial management. Pediatrics 2018 Mar;141(3):e20174081. DOI: https://doi.org/10.1542/ CONCLUSIONS peds.2017-4081 Current public health policy supports initiating universal 7. Zuckerbrot RA, Cheung AH, Jensen PS, Stein RE, Laraque D. Guidelines for Adolescent 3-5 Depression in Primary Care (GLAD-PC): I. Identification, assessment, and initial annual depression screening at age 12 years, but de- management. Pediatrics 2007 Nov;120(5):e1299–312. DOI: https://doi.org/10.1542/ pression and suicidal ideation are prevalent in children peds.2007-1144, PMID:17974723. 1,2 8. Kroenke K, Spitzer RL, Williams JB. The PHQ-9: Validity of a brief depression severity younger than 12 years old. We have shown that an ad- measure. J Gen Intern Med 2001 Sep;16(9):606–13. DOI: https://doi.org/10.1046/j.1525- olescent depression screening protocol for 11-year-olds can 1497.2001.016009606.x, PMID:11556941.

The Permanente Journal·https://doi.org/10.7812/TPP/20.233 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 77 ORIGINAL RESEARCH ARTICLE Multistage Adolescent Depression Screening: A Comparison of 11-Year-Olds to 12-Year-Olds

9. Allgaier AK, Pietsch K, Fruhe¨ B, Sigl-Glockner¨ J, Schulte-Korne¨ G. Screening for depression 14. Hagan, JF, Bright Futures Tool and Reosurce KIt. American Academy of Pediatrics 2007. in adolescents: Validity of the Patient Health Questionnaire in pediatric care. Depress Anxiety 15. Population Division, US Census Bureau. 2010 Demographic Analysis. (December 6, 2010 2012 Oct;29(10):906–13. DOI: https://doi.org/10.1002/da.21971, PMID:22753313. release). Accessed November 2018. census.gov/library/publications/2011/compendia/ 10. Johnson JG, Harris ES, Spitzer RL, Williams JBW. The Patient Health Questionnaire for usa- counties-2011.html. adolescents. J Adolesc Health 2002 Mar;30(3):196–204. DOI: https://doi.org/10.1016/ 16. Thorleifsdottir B, Bjornsson¨ JK, Benediktsdottir B, Gislason T, Kristbjarnarson H. Sleep and sleep s1054-139x(01)00333-0 habits from childhood to young adulthood over a 10-year period. J Psychosom Res 2002 Jul; 11. Richardson LP, McCauley E, Grossman DC, et al. Evaluation of the Patient Health 53(1):529–37. DOI: https://doi.org/10.1016/s0022-3999(02)00444-0, PMID:12127168. Questionnaire-9 item for detecting major depression among adolescents. Pediatrics 2010 17. Smarr KL, Keefer AL. Measures of depression and depressive symptoms: Beck Dec;126(6):1117–23. DOI: https://doi.org/10.1542/peds.2010-0852, PMID:21041282. Depression Inventory-II (BDI-II), Center for Epidemiologic Studies Depression Scale 12. Richardson LP, Rockhill C, Russo JE, et al. Evaluation of the PHQ-2 as a brief screen for (CES-D), Geriatric Depression Scale (GDS), Hospital Anxiety and Depression Scale detecting major depression among adolescents. Pediatrics 2010 May;125(5):e1097. DOI: (HADS), and Patient Health Questionnaire. Arthritis Care Res 2011 Nov;63(S11): https://doi.org/10.1542/peds.2009-2712, PMID:20368315. S454–66. DOI: https://doi.org/10.1002/acr.20556 13. Lewandowski RE, O’Connor B, Bertagnolli A, et al. Screening for and diagnosis of 18. Edelbrock C, Costello AJ, Dulcan MK, Kalas R, Conover NC. Age differences in the depression among adolescents in a large health maintenance organization. Psychiatr Serv reliability of the psychiatric interview of the child. Child Dev 1985 Feb;56(1):265–75.DOI: 2016 Jun;67(6):636–41. DOI: https://doi.org/10.1176/appi.ps.201400465, PMID:26876655. https://doi.org/10.2307/1130193, PMID:3987406.

78 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.233 n ORIGINAL RESEARCH ARTICLE Validation Study of Kids Hearing Game: A Self-Administered Pediatric Audiology Application

Brian Kung, MD1; Larisa Kunda, MD1; Sarah Groff, AuD2; Erica Miele, AuD2; Marion Loyd, AuD2; Diane M. Carpenter, MPH3 Perm J 2021;25:20.157 E-pub: 3/10/2021 https://doi.org/10.7812/TPP/20.157

ABSTRACT About 11% of school-age children have mild hearing Objective: Conduct a comparison study between conventional loss.3 It has also come to light that, just as amblyopia exists audiometry and a tablet-based hearing screening application, in childhood visual development, “amblyaudia” can be a Kids Hearing Game (KHG). If KHG measures hearing at levels lasting consequence of asymmetric hearing loss,4 even in comparable with conventional audiometry, it could be used to children who have temporary hearing loss from effusions/ screen hearing in children. infection.5 In some of these cases, central connections in the Methods: Prospective equivalence study where measurements of auditory system can be permanently altered through ab- pure tone hearing via KHG are compared with measurements of normal development, thus leading to permanent potentially pure tone hearing via conventional audiometry in patients aged ff ’ 6-11 years over a 4-month period. Eighteen patients completed detrimental change in the a ected ear s central auditory 4 the study. Results from 36 ears were included in the data for connections. Some researchers have even noted an associ- analysis. Decibel measurements from each frequency measured ation between hearing loss and a variety of psychological or with KHG for each ear were compared with conventional audi- psychiatric disorders.6 ometry. Mean measurements were calculated for each ear and ere is a relative shortage of audiologists (4.1 per frequency as well as mean differences in measurements at each 100,000 population, ranging from 7.3/100,000 in Colorado ear and frequency. Tests of equivalence were used to assess mean to 1.9/100,000 in California as of 20197) compared with the ff within-subject di erences in decibel measurements using a 10-dB number of patients who need assessment and treatment, ff zone of indi erence. and this shortage is projected to get worse over the next Results: Mean decibel measurements using KHG for the right decade.8 After birth, children in the US generally get screening ear at 500, 1000, 2000, 4000, and 8000 Hz and the left ear at 1000, tests at their schools at certain intervals. It is hoped that many 2000, 4000, and 8000 Hz were found equivalent to conventional ’ ffi audiometry (p < 0.050). The mean decibel measurement using will also get an annual screening at their pediatrician so ce. KHG for the left ear at 500 Hz was found not equivalent (p = 0.101). ese annual screenings usually consist of some form of pure However, when left and right ear data were analyzed together, tone air conduction audiometry through headphones. In the KHG was found to be equivalent to conventional audiometry US alone, school screening protocols are variable,9,10 and even across all frequencies. Eight patients having hearing loss greater in the most proactive districts they may not be done often than 25 dB on conventional audiometry were also identified by enough to pick up cases of hearing loss. January 2020 data KHG to have hearing loss. from the American Speech-Language-Hearing Association Conclusion: KHG is comparable to conventional audiometry states that 8 states (Alabama, Hawaii, Idaho, Iowa, New and may be used as a screening tool for children. Mexico, North Dakota, South Dakota, and Wyoming) do not have required school hearing screening, and in 5 states INTRODUCTION (Missouri, Montana, New Hampshire, South Carolina, and Wisconsin) school hearing screening is suggested but not Hearing loss can have a detrimental effect on childhood mandated.11 Generally, children only are referred for formal development and academic performance. Children with audiograms if the occasional hearing screening picks up a untreated hearing loss may experience mild to moderate problem or if the child’s pediatrician or parents believe there learning difficulties, inattention, behavioral problems, and is a problem. Because of these deficiencies and the variability poor social interaction. Children having more severe to profound hearing loss may experience severe learning dif- fi 1 culties, stigmatization, and social isolation. Most people Author Affiliations are aware of the effects of severe to profound hearing loss in 1Department of Head and Neck Surgery,Kaiser Permanente, Union City, CA childhood, and often these patients are easily identified. 2Department of Audiology, Kaiser Permanente, Union City, CA 3Division of Research, Kaiser Permanente, Oakland, CA However, the correlation between mild hearing loss and Present address: Brian Kung, MD, Department of Otolaryngology, Kaiser Permanente, 201, 16th Ave E, Seattle, 2 poor academic performance along with the aforementioned WA 98112. potential sequelae make screening for hearing loss a very important part of the medical care for children not just in Corresponding Author Brian Kung, MD ([email protected]) the US but also around the world. Keywords: amblyaudia, audiology, hearing loss, hearing screening, pediatrics The Permanente Journal·https://doi.org/10.7812/TPP/20.157 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 79 ORIGINAL RESEARCH ARTICLE Validation Study of Kids Hearing Game: A Self-Administered Pediatric Audiology Application

in screening, it is likely that many children have undiag- audiometry in an audiometry booth and the tablet screening nosed hearing loss for several months or longer (in cases of applicationinaquietroomoftheoffice with low ambient temporary hearing loss) during critical developmental ages. noise (< 30 dB). e order of testing was random via coin flip. ere has been a push to develop a more efficient means of Subjects were presented with an online gift card of nominal testing children for hearing loss. value as a thank-you for enrolling. With the advent of tablet computing and the huge growth Conventional audiometry was performed in a sound- of tablet applications, there is a tremendous opportunity to treated booth in the following way. e subject wore apply this technology to improve access to hearing screening headphones, and the audiologist presented tones via air in children. Children have been found as early as 12 months conduction at various frequencies and varying intensities to of age to have at least moderate ability to use a tablet, and in both the right and left ears. resholds were recorded when 1 study 90% of subjects age 24 months showed at least the subject properly responded to the presented tone 50% of moderate ability.12 Although there are already self-administered the time. Bone conduction thresholds, speech reception audiometric software applications on tablets or smartphones thresholds (via air conduction), speech discrimination thresh- that have been shown to adequately measure hearing olds (via air conduction), and tympanometry were performed as thresholds in patients,13-15 developers (Hearing.Games, part of the standard audiogram. Speech reception thresholds LLC) have designed a new software application, Kids were used to help determine audiogram reliability: if the speech Hearing Game (KHG), which is essentially an audiogram reception thresholds were within 10 dB of the pure tone inagameformat.Becausethisapplicationisagameand average, the audiogram was determined to be reliable. For may appeal to children, we anticipated that a child may be the purposes of the study, only pure tone air conduction able to maintain focus and concentration to adequately thresholds were used for comparison with air conduction complete the test. e animation, graphics, and interactive thresholds measured by the tablet application. gameplay make this application different from other hearing Tablet application hearing screening was performed in test applications. It should be noted that this application is the following way. Amazon Kindle Fire 7 tablets were not meant to replace a formal audiogram or the medical loaded with the KHG via the Amazon App Store. esubject personnel needed to confirm and treat hearing loss; nor will played the game in a quiet room (not an audiology booth) in it serve to mitigate inconsistencies and deficiencies in state- the clinic. egame’s sound presentation level was calibrated mandated school screening. However, it may be useful as a to Ausdom F01 Wired Over-Ear Headphones. Each subject screening tool for concerned and motivated parents to see wore the headphones and played the game on the tablet. e if a formal audiogram is warranted for their children. We game enacts a first-person view of the player in a boat moving aimed to assess whether a tablet application under condi- down a river, aiming to catch fireflies. e subject is told to tions similar to real-world conditions performs at a level listen carefully for the presented tones and to touch the screen that is equivalent to audiometry in a clinical setting among when he or she hears the tone. When the subject responds pediatric patients belonging to Kaiser Permanente Northern appropriately, he or she “catches” a firefly. ese pure tone California. sounds are presented via air conduction to the subject at varying frequencies and intensities to the left and right ears. MATERIALS AND METHODS e game sustains the subject’s interest by continuing to is study was reviewed and approved by the Institutional provide visual and spoken auditory feedback to the subject Review Board of Kaiser Foundation Research Institute. during game play. One hundred twenty tones are presented Pediatric patients ages 6-11 years who were scheduled to over 5 rounds, or 24 tones per round. ere is a built-in have audiograms done for any reason at one Kaiser Per- mechanism in the game that help to ensure that each subject manente Northern California medical center between July 1, is properly responding: if the subject touches the screen more 2018 and November 30, 2018 were recruited to participate than 36 times in 1 round, the subject does not advance to the in the study. Patients were excluded from consideration for next round and needs to re-do the round, in which case the the study if they met any of the following criteria: cognitive series of tones is presented again in a different order. After or developmental dysfunction, non-English speaking, con- each subject completes 5 rounds, the game concludes, and the genital ear malformation, known congenital binaural severe results are recorded. Pictures of gameplay screenshots can be deafness, and disruptive or uncooperative behavior. Written seen in Figures 1 and 2. informed consent was obtained from each patient’s parent or Data from each ear for each subject were recorded on an guardian, and written assent was obtained from each patient in Excel spreadsheet. Because the lowest sound threshold level order to participate in the study. e ear canal was examined that could be recorded by the tablet was 20 dB, any values before testing to ensure that there was no excessive cerumen or under 20 dB recorded via conventional audiometry were other blockage. Each patient underwent both conventional entered as 20 dB.

80 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.157 ORIGINAL RESEARCH ARTICLE Validation Study of Kids Hearing Game: A Self-Administered Pediatric Audiology Application

Figure 1. Screenshot of gameplay - the player has a first person perspective while sitting in a boat moving down a calm river. The yellow lights represent fireflies that are “caught” when a sound is successfully responded to.

Univariate statistics were used to summarize categorical RESULTS data as counts and frequencies; for continuous variables, During the enrollment period, 19 patients were enrolled means and standard deviations or medians and interquartile and consented, and 18 patients were able to successfully ranges were calculated. e decibel measurement from each complete both the conventional audiogram and the KHG frequency (500, 1000, 2000, 4000, and 8000 Hz) measured tablet audiometry application. Data were thus obtained for with the tablet for each ear was compared with corresponding 36 ears: 18 left ears and 18 right ears. Of the 18 subjects, measurements obtained by conventional audiometry within 12 (66.7%) were male and 6 (33.3%) were female; median the same subject. Mean measurements were then calculated age was 7 years (interquartile range: 6-9). for each ear and each frequency, as well as mean differences in Eight patients were found to have hearing loss with a measurement at each ear and frequency. Tests of equivalence hearing threshold in any ear measured at ≥ 25 dB at any were performed to assess the mean within-subject difference tested frequency by conventional audiometry. Of these in decibel measurements at 500, 1000, 2000, 4000, and patients, all 8 were found to have at least 1 threshold in any 8000 Hz. e zone of indifference for these comparisons was ear measured at 25 dB or above when tested by the KHG 10 dB. Analyses were performed using SAS 9.4 (Cary, NC). tablet application.

The Permanente Journal·https://doi.org/10.7812/TPP/20.157 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 81 ORIGINAL RESEARCH ARTICLE Validation Study of Kids Hearing Game: A Self-Administered Pediatric Audiology Application

Figure 2. Screenshot of gameplay taken when a sound is successfully responded to. The stars are seen here, giving that indication. A sound will also accompany this during gameplay.

Mean dB measurements for each frequency for each ear regardless of referring department, 7 subjects had pressure were measured for both the KHG tablet application and equalizing tubes or previously extruded pressure equalizing conventional audiometry (Table 1). Based on these mea- tubes (1 with a tympanic membrane perforation from a tube), surements, the KHG tablet application was found to be 1 subject had bilateral retractions, 3 had recent findings of equivalent to conventional audiometry in all frequencies serous otitis media, and 8 were found to have normal tested (p < 0.050) except for 500 Hz in the left ear (p = 0.101) tympanic membranes. with a zone of indifference of 10 dB. When left ear and right ear data were combined, the DISCUSSION mean dB measurements for the KHG tablet application and is equivalence study comparing hearing measurements conventional audiometry were found to be equivalent at all taken with the KHG tablet audiometry application and frequencies (p < 0.050, Table 2). edifference between the conventional audiometry shows that equivalence was largely means was particularly small at 2000 Hz at 0.28 dB, whereas achieved. is shows that KHG could be a useful screening the difference was larger at 500 Hz at 5.00 dB. e mean dB tool for children thought to have some degree of hearing measurements for the tablet were higher than for con- loss. ventional audiometry across all frequencies. ere are potential variables that may make KHG sub- Of all enrolled subjects, 6 were referred by pediatrics optimal for screening outside of the doctor’soffice and can subsequent to failed hearing screenings. irteen were re- lead to inaccuracies. First, the KHG tablet application was ferred by the otolaryngology department. Of all subjects, run using ordinary headphones (under $20) in a relatively

82 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.157 aiainSuyo isHaigGm:ASl-diitrdPdarcAdooyApplication Audiology Pediatric Self-Administered A Game: Hearing Kids of Study Validation h emnneJournal Permanente The · tp:/o.r/071/P/017TePraet Journal Permanente The https://doi.org/10.7812/TPP/20.157 Table 1. Equivalence test, comparing tablet to audiometry, by ear (left vs right)

Hz Ear Mean value, tablet (± SD) (dB) Mean value, audiometry (± SD) (dB) Difference in means (± SD) (dB) 90% confidence intervala Equivalence p value 500 (n = 18) Left 28.89 (± 11.19) 23.61 (± 8.37) −5.28 (± 15.10) −11.47, 0.91 Not equivalent 0.101 500 (n = 18) Right 27.50 (± 9.89) 22.78 (± 8.26) −4.72 (± 11.56) −9.46, 0.02 Equivalent 0.035 1000 (n = 18) Left 24.72 (± 6.96) 21.39 (± 4.79) −3.33 (± 8.40) −6.78, 0.11 Equivalent 0.002 1000 (n = 18) Right 25.00 (± 5.94) 21.67 (± 4.85) −3.33 (± 6.42) −5.96, −0.70 Equivalent < 0.001 2000 (n = 18) Left 22.22 (± 3.92) 21.67 (± 5.14) −0.55 (± 6.16) −3.08, 1.97 Equivalent < 0.001 2000 (n = 18) Right 22.50 (± 3.93) 22.50 (± 6.48) 0.00 (± 6.64) −2.72, 2.72 Equivalent < 0.001 4000 (n = 18) Left 23.06 (± 6.67) 20.56 (± 2.36) −2.50 (± 7.33) −6.14, 1.14 Equivalent < 0.001 4000 (n = 18) Right 25.28 (± 8.13) 21.94 (± 5.18) −3.33 (± 8.57) −6.85, 0.18 Equivalent 0.002 8000 (n = 18) Left 24.17 (± 6.47) 21.11 (± 2.74) −3.06 (± 7.50) −6.13, 0.02 Equivalent < 0.001 8000 (n = 18) Right 25.28 (± 8.82) 21.67 (± 5.14) −3.61 (± 6.82) −6.41, 0.82 Equivalent < 0.001 aThe zone of indifference for the test of equivalence was ± 10 decibels (dB). To establish equivalence, the 90% confidence interval needs to fall within the limits of (−10.00, 10.00). · o esnlueol.N te sswtotpriso.Cprgt©22 h emnnePes l ihsreserved. rights All Press. Permanente The 2021 © Copyright permission. without uses other No only. use personal For

Table 2. Equivalence test, comparing tablet to audiometry, both ears (left and right combined)

Hz Ear Mean value, tablet (± SD) (dB) Mean value, audiometry (± SD) (dB) Difference in means (± SD) (dB) 90% confidence intervala Equivalence p value 500 (n = 36) Both 28.19 (± 10.43) 23.19 (± 8.21) −5.00 (± 13.26) −8.73, −1.27 Equivalent 0.015

1000 (n = 36) Both 24.86 (± 6.38) 21.53 (± 4.75) −3.33 (± 7.37) −5.41, −1.26 Equivalent < 0.001 ARTICLE RESEARCH ORIGINAL 2000 (n = 36) Both 22.36 (± 3.87) 22.08 (± 5.78) −0.28 (± 6.32) −2.06, 1.50 Equivalent < 0.001 4000 (n = 36) Both 24.17 (± 7.42) 21.26 (± 4.03) −2.92 (± 7.87) −5.13, −0.70 Equivalent < 0.001 8000 (n = 36) Both 24.72 (± 7.65) 21.39 (± 4.07) −3.33 (± 7.07) −5.32, −1.34 Equivalent < 0.001 aThe zone of indifference for the test of equivalence was ± 10 decibels (dB). To establish equivalence, the 90% confidence interval needs to fall within the limits of (−10.00, 10.00). 83 ORIGINAL RESEARCH ARTICLE Validation Study of Kids Hearing Game: A Self-Administered Pediatric Audiology Application

quiet room (under 30 dB ambient noise), whereas con- With some modifications to KHG, including changes to ventional audiometry uses more sophisticated headphones, significantly shorten gameplay without compromising the which are calibrated on a routine and regular basis, in a accuracy of testing (which may help younger subjects with sound-treated audiometry booth. ere is more potential shorter attention spans) and the development of versions in for some competing noise to make it more difficult for different languages, we believe it could become more ac- the subject to hear the tones being presented on the tablet curate with different patient populations. Larger studies application. It is also possible that in real-life use outside of than ours, using subjects who primarily communicate in the the clinic an environment quiet enough to test with the same language as the language used in KHG, should be tablet would not be able to be achieved; this could affect conducted to confirm our findings. accuracy. Second, although the calibration procedure in KHG has been optimized to adjust the volume of the tablet CONCLUSION based on the type of tablet and the type of headphones, there Pediatric hearing loss, whether mild to severe, progres- could be inconsistencies that could affect the accuracy of sive, permanent, or even temporary, can lead to lasting sound measurements in real-life use. Finally, children having detrimental developmental and academic consequences. any of the conditions for which we excluded (cognitive or Tablet screening audiometry applications such as KHG can developmental dysfunction, non-English speaking, congen- be used to test and screen for hearing loss in children who ital ear malformation, binaural severe deafness, and behavioral may not have adequate access to resources for traditional problems) would likely have a suboptimal experience playing hearing screening, let alone conventional audiometry. this game. ese applications may also be used by concerned family Although our study was adequately powered to show members to screen their children for hearing loss without statistical equivalence between mean thresholds measured, a needingtogotothepediatrician’soraudiologist’soffice. larger number of patients studied could further validate is may be particularly beneficial during the years when these findings. A study was performed by the developers of traditional school hearing screening is not offered or in the application using 516 pediatric subjects in China.16 It cases in which school screening is not available. Although demonstrated 85% agreement of KHG hearing measure- these screening tools do not replace conventional audi- ments with conventional audiometry at 500, 1000, and ometry, our results suggest that they may help to detect 2000 Hz with decreased levels of agreement at 4000 Hz and hearing loss in children. is may allow affected children 8000 Hz.16 It also showed a sensitivity of 91% in detecting to be identified earlier and may allow them to undergo hearing loss 20 dB or greater and a specificity of 74% in conventional audiometry earlier than they otherwise would subjects age 4 years or older, with the specificity increasing have to confirm the presence of hearing loss. If screening to 90% in their subjects age 7 years or older.16 e language tools such as KHG allowed hearing loss to be detected discrepancy with the Chinese-speaking subjects playing the and treated earlier, they could be of tremendous benefit English-only KHG application may have contributed to to neurocognitive development, social development, and some decreased levels of correlation of hearing measure- academic performance in these patients. v ments in that study.16 ere were also remarks from subjects in that study that the gameplay took too long and could Disclosure Statement become tedious, leading to potentially less accurate results.16 The author(s) have no conflicts of interest to disclose. We also noticed similar sentiments from several subjects in  Acknowledgments this study. ere was 1 patient in our study (a potential Funding for this study was provided by the KPNC Community Benefit program. subject who was excluded from the final cohort) who turned ff the tablet o twice during gameplay. Results from that Authors’ Contributions patient were not able to be recorded. Brian Kung, MD: study design, data analysis, manuscript preparation. Larisa Another potential limitation to tablet and phone-based Kunda, MD: study design, data analysis, manuscript preparation. Sarah Groff, AuD: data collection, manuscript preparation. Erica Miele, AuD: data collection, audiometry applications is that they require an internet manuscript preparation. Marion Loyd, AuD: data collection, manuscript preparation. connection and compatible hardware/equipment. ere- Diane Carpenter, MPH: study design, data analysis, manuscript preparation. fore, it may only be available to motivated, at least somewhat technologically sophisticated, and financially stable families Funding Statement at this time; this would be even more of a problem outside of Funding for this study was provided by the Kaiser Permanente Northern California Community Benefit program. developed countries. However, as access to technology continues to improve and education about the existence of Abbreviations these applications becomes more widespread, these barriers dB = decibel; Hz = hertz; KHG = Kids Hearing Game; SAS = Statistical Analysis should become reduced over time. System

84 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.157 ORIGINAL RESEARCH ARTICLE Validation Study of Kids Hearing Game: A Self-Administered Pediatric Audiology Application

9. American Speech-Language-Hearing Association. State licensure trends and quarterly updates. References Accessed March 28, 2017. http://www.asha.org/advocacy/state/StateLicensureTrends/. 1. Olusanya BO, Newton VE. Global burden of childhood hearing impairment and disease 10. Sehkar DL, Zalewski TR, Paul IM. Variability of state school-based hearing screening control priorities for developing countries. Lancet 2007 Apr;369(9569):1314-7. DOI: protocols in the United States. J Community Health 2013 Jun;38:569-74. DOI: https://doi. https://doi.org/10.1016/s0140-6736(07)60602-3 org/10.1007/s10900-013-9652-6 2. Daud M, Noor R, Rahman N, Sidek D, Mohamad A. The effect of mild hearing loss on 11. American Speech-Language-Hearing Association. School-Age Hearing Screening. academic performance in primary school children. Int J Pediatr Otorhinolaryngol 2010 Accessed January 16, 2020. https://www.asha.org/Advocacy/state/School-Age-Hearing- Jan;74:67-70. DOI: https://doi.org/10.1016/j.ijporl.2009.10.013 Screening/. 3. Bess FH, Dodd-Murphy J, Parker RA. Children with minimal sensorineural hearing loss: 12. Hourcade JP, MascherSL, WuD, PantojaL. Look, my baby is using an iPad! an analysis of Prevalence, educational performance, and functional status. Ear Hear 1998 Oct;19(5): YouTube videos of infants and toddlers using tablets. In Proceedings of the 33rd annual 339-54. DOI: https://doi.org/10.1097/00003446-199810000-00001 ACM conference on human factors in computing systems, Seoul, Republic of Korea; 4. Kaplan A, Kozin E, Remenschneider A, et al. Amblyaudia: Review of pathophysiology, 2015, pp 1915-24. clinical presentation, and treatment of a new diagnosis. Otolaryngol Head Neck Surg 2016 13. Yeung J, Javidnia H, Heley S, Beauregard Y, Champagne S, Bromwich M. The Feb;154(2):247-55. DOI: https://doi.org/10.1177/0194599815615871, PMID:26556464 new age of play audiometry: Prospective validation testing of an iPad-based play 5. Uclés P, Alonso MF, Aznar E, Lapresta C. The importance of right otitis media in childhood audiometer. J Otolaryngol Head Neck Surg 2013 Mar;42:21. DOI: https://doi.org/ language disorders. Int J Otolaryngol 2012;2012:818927. DOI: https://doi.org/10.1155/ 10.1186/1916-0216-42-21 2012/818927 14. Yeung J, Heley S, Beauregard Y, Champagne S, Bromwich M. Self-administered hearing 6. Carvill S. Sensory impairments, intellectual disability and psychiatry. J Intellect Disabil Res loss screening using an interactive, tablet play audiometer with ear bud headphones. Int J 2001 Dec;45(Pt 6):467-83. DOI: https://doi.org/10.1046/j.1365-2788.2001.00366.x, PMID: Pediatr Otorhinolaryngol 2015 Aug;79(8):1248-52. DOI: https://doi.org/10.1016/j.ijporl. 11737534 2015.05.021, PMID:26055197 7. Brook G. Annual workforce data: 2019 ASHA-certified audiologist- and speech-language 15. Whitton J, Hancock K, Shannon J, Polley D. Validation of a self-administered audiometry pathologist-to-population ratios; 2019. https://www.asha.org/siteassets/uploadedfiles/ application: An equivalence study. Laryngoscope 2016 Oct;126(10):2382-8. DOI: https:// 2019-Audiologist-and-SLP-to-Population-Ratios.pdf doi.org/10.1002/lary.25988, PMID:27140227 8. Windmill I, Freeman BA. Demand for audiology services: 30-yr projections and impact on 16. Xiao L, Zou B, Gao L, et al.. A novel tablet-based approach for hearing screening of the academic programs. J Am Acad Audiol 2013 May;24(5):407-16. DOI: https://doi.org/10. pediatric population, 516-patient study. Laryngoscope 2020 Sep;130:2245-51. DOI: 3766/jaaa.24.5.7, PMID:23739060 https://doi.org/10.1002/lary.28329.

The Permanente Journal·https://doi.org/10.7812/TPP/20.157 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 85 n ORIGINAL RESEARCH ARTICLE Effects of Implementing a Higher Threshold for Recommending Thyroid Biopsies on Malignancy Rates

Kori Higashiya, BS1; Liam Delgesso, BS2; Hyo-Chun Yoon, MD, PhD3 Perm J 2021;25:20.240 E-pub: 3/10/2021 https://doi.org/10.7812/TPP/20.240

ABSTRACT e 2015 American yroid Association (ATA) guide- Introduction: We implemented a new thyroid nodule lines suggest that all solid thyroid nodules ≥ 1 cm should classification system in which a biopsy was recommended undergo a biopsy because of a risk of malignancy of 10% to for thyroid lesions ≥ 1 cm with at least 2 or more suspicious 20% for nodules with no suspicious characteristics, and > 70% features. for nodules with at least 1 suspicious characteristic.2 e Methods: Three consecutive years of thyroid biopsies using American College of Radiology–yroid Imaging Reporting the new classification system were reviewed for patient demo- and Data Systems (ACR-TIRADS) guidelines recommend graphics, biopsy recommendation based on nodule size and ≥ imaging characteristics, and pathology results. The primary biopsy for TR5 lesions that are 1 cm plus > 7 points based outcome was malignancy rates for thyroid biopsies. These results on suspicious characteristics due to an estimated risk of 3 were compared to a 3-year historical data set. malignancy of > 20%. Results: Review of thyroid biopsies from 2010 to 2012, prior At our institution, we previously recommended biopsy for to the implementation of current recommendations, revealed all lesions ≥ 1 cm with at least 1 suspicious characteristic. 996 thyroid biopsies with a malignancy rate of 12.8%. Sub- After getting agreement from the Departments of Endo- sequent to the new classification system in 2017, 483 thyroid crinology and Head and Neck Surgery, we instituted a more biopsies were performed over the next 3 years with a ma- stringent classification system for thyroid ultrasounds lignancy rate of 21.9%. similar to ACR-TIRADS. e classification system sim- fi Discussion: Implementation of the new classi cation system plified ACR-TIRADS into 4 categories: #THY1 through with a higher threshold for biopsy reduced our yearly biopsy #THY4. #THY1 recommends no further imaging unless volume by approximately 50% while also increasing our malig- nancy rate from 12.8% to 21.9%, which is more in line with clinical symptoms change for thyroid glands with no published rates of malignancy. nodules, nodules with benign imaging characteristics, or Conclusion: In a community setting performing less than 200 nodules that have been stable for 1 year. #THY2 recom- biopsies per year, the use of more stringent requirements for mends 12-month follow-up imaging for thyroid nodules thyroid biopsy are necessary to achieve malignancy rates com- with no or 1 suspicious imaging feature. #THY3 recom- parable to the published literature. mends fine-needle aspiration (FNA)/biopsy for lesions with 2 or more suspicious features, and #THY4 recommends an endocrinology or head and neck surgery consult for ab- normalities that do not fit into any of the other 3 categories. INTRODUCTION We compared the volume of biopsy procedures and the e use of nodule size and suspicious characteristics to prevalence of malignancy in the biopsy specimens for a predict malignancy is well documented. Suspicious charac- 3-year period before and after the implementation of the teristics such as microcalcifications, irregular margins, marked more stringent biopsy threshold. hypoechogenicity, and taller than wide shape have been fi METHODS shown to have 87% sensitivity and 86.5% speci city for  thyroid lesion malignancy.1 To standardize recommendations e Institutional Review Board approved this study with and reduce unnecessary biopsies, several professional societies a waiver of informed consent because this is a data-only have formed guidelines for the classification of thyroid retrospective investigation with no patient interaction. nodules with corresponding targets for malignancy rates. We reviewed the records of all patients who underwent a thyroid biopsy performed in our geographically isolated health maintenance organization that serves approximately Author Affiliations 250,000 members, where all nonemergent care is provided 1John A. Burns School of Medicine, University of Hawaii, Honolulu, HI within the organization and all patient encounters are included 2University of British Columbia, Vancouver, Canada in a comprehensive electronic medical record. We reviewed 3Hawaii Permanente Medical Group, Honolulu, HI the patient charts for all patients who underwent a thyroid Corresponding Author biopsy for a full 3-year period before (January 1, 2010– Kori Higashiya, BS ([email protected]) December 31, 2012) and after (January 1, 2017–December

Keywords: malignancy rate, thyroid, thyroid biopsy, thyroid cancer, thyroid malignancy 86 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.240 ORIGINAL RESEARCH ARTICLE Effects of Implementing a Higher Threshold for Recommending Thyroid Biopsies on Malignancy Rates

31, 2019) the implementation of the more stringent biopsy or surgical excision. is resulted in a combined biopsy threshold. We recorded the age and gender of each prevalence of 12.8%, with a surgically confirmed malig- patient as well as the pathology results of the biopsy nancy rate of 7.5%. specimens. For those patients who underwent subsequent In the 3 years after the institution of the more stringent surgical excision, the pathology results of the surgical conditions for thyroid biopsy, 483 biopsies were performed. specimen were used to determine whether the lesion was Of the 483 biopsies, 373 (77.2%) were the result of a prior cancerous. For the latter study period, we also recorded thyroid ultrasound that was classified as #THY3 or #THY4. thyroid classification (#THY). A total of 297 (79.6%) were classified as #THY3 (ie, a Malignancy rates were determined based on the pa- nodule ≥ 1 cm and with 2 or more suspicious character- thology results of the excised surgical specimens, or the istics), and 76 (20.3%) were classified as #THY4 (a lesion biopsy results if the patient did not undergo surgical ex- for which endocrinologist or head and neck surgeon input cision. Among patients who underwent a biopsy of multiple was requested). lesions, only the most suspicious pathological result was e remaining 110 (22.7%) biopsies were performed on included in the analysis. For example, if a patient underwent patients who previously had a thyroid ultrasound that was 2 biopsies and 1 result was reported as benign follicular classified as #THY1 (11 patients) or #THY2 (99 patients). lesion and the other was a lesion suspicious for follicular Of the 483 thyroid biopsies conducted from 2017 neoplasm, then only the lesion suspicious for follicular through 2019, 106 were found to be cancer, follicular neoplasm was included in the analysis for malignancy rate. neoplasm, or suspicious for follicular neoplasm on biopsy. Pathology results reported as suspicious for follicular Twenty-three lesions reported as follicular neoplasm or neoplasm were included as cancer if the patient declined suspicious for follicular neoplasm on biopsy did not undergo surgical excision, because the usual practice among our head surgical excision for definitive confirmation but were in- and neck surgeons is to remove such lesions surgically. cluded in the total number of cancers. Eighty-three were Lesions reported as follicular lesion of undetermined sig- confirmed papillary thyroid carcinoma or other cancer nificance (FLUS) were not included in our calculation of confirmed by biopsy or surgical excision. Other cancers malignancy rates because these lesions are usually followed included 1 anaplastic thyroid cancer, 4 cases of lymphoma, by imaging or a repeat biopsy. Statistical analysis was and 3 metastases from a different primary malignancy. performed on nominal variables with the χ2 test and con- Collectively, the malignancy rate for this 3-year period tinuous variables with 2-tailed t-tests assuming equal was 21.9%, with a surgically confirmed malignancy rate variance. of 17.1%. ere was a significant increase in the rate of cancers reported between 2010–2012 and 2017–2019 RESULTS (χ2 for trends, P < 0.00001). A total of 996 thyroid biopsies were performed during the For both time periods, we did not include FLUS because 3-year period of 2010 through 2012 (Table 1). Of these 996 it is unknown whether these lesions have oncological sig- biopsies, 127 were found to be cancer or suspicious for nificance. However, there were 53 cases of FLUS in 2010– follicular neoplasia on biopsy. Fifty-two lesions suspicious 2012 and 17 cases of FLUS in 2017–2019. ere was no for follicular neoplasm on biopsy did not undergo surgical significant difference in the number of FLUS cases between excision for definitive confirmation but were included in the the 2 time periods (χ2 for trends, P = 0.056). None of FLUS total number of cancers. Seventy-five were confirmed samples from 2017–2019 underwent additional molecular papillary thyroid carcinoma or other cancer confirmed by testing. ree of the 17 underwent repeat biopsy and were

Table 1. Comparison of thyroid biopsies and cancers for 2010 through 2012 and 2017 through 2019 Study Parameter 2010–2012 2017–2019 Total biopsies 996 483 Average age 55.6 55.7 Female 825 (82.8%) 389 (80.5%) Male 171 (17.2%) 94 (19.4%) Suspicious for follicular neoplasm, biopsy only 52 (5.2%) 23 (4.7%) Papillary thyroid cancer or other cancer, biopsy or surgical excision 75 (7.5%) 83 (17.2%) Total cancera 127 (12.8%) 106 (21.9%) Follicular lesion of undetermined significance 53 (5.3%) 15 (3.1%) aTotal cancer includes all biopsy-proved or surgically proved cancers and all suspected follicular neoplasms that did not undergo surgical confirmation.

The Permanente Journal·https://doi.org/10.7812/TPP/20.240 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 87 ORIGINAL RESEARCH ARTICLE Effects of Implementing a Higher Threshold for Recommending Thyroid Biopsies on Malignancy Rates

reported as benign. One patient had open surgery with these estimates are derived from high-volume centers; the benign pathology. e remaining 13 patients elected to have overall risk of malignancy may be lower given the inter- clinical and imaging follow-up. observer variability in sonography.2 From 2010–2012 we Compared to the historical volume of 332 thyroid bi- recommended thyroid biopsies for lesions > 1 cm with at opsies per year in 2010–2012, the implementation of the least 1 suspicious characteristic. is resulted in a malig- new classification system resulted in 161 thyroid biopsies nancy rate of 12.8%, which is very much lower the 70% to per year in 2017–2019. is represents an approximate 50% 90% chance of malignancy that the ATA cites. reduction in number of biopsies and also achieved a target Although our thyroid classification system is similar to prevalence of malignancy similar to that of the ACR- ACR-TIRADS, we do not use a point system to determine TIRADS recommendations. ere was no significant into which category the lesion falls. Instead, we base our difference for gender (χ for trends, P = 0.28) or age (t-test, recommendations on the size of the lesion and the number P = 0.90) between the 2 time periods. of suspicious characteristics. With ACR-TIRADS, TR5 Despite having the classification of THY#1, when no (score ≥ 7), which is assigned to highly suspicious lesions, further follow-up is recommended, there were 11 patients has an estimated malignancy rate of > 20%.3 Similarly, in who underwent biopsy. On chart review, common reasons our classification system, we achieved a malignancy rate of for biopsy were patient preference or head and neck surgery/ 17.1% for surgically documented cancer. endocrinology recommendation based on clinical history. Despite having a higher threshold to conduct a biopsy, we One patient had a biopsy that reported a lesion suspicious were not able to reproduce the > 70% positivity rate cited by for follicular neoplasm. Because of the family history of the ATA guidelines for lesions ≥ 1 cm with at least 1 thyroid neoplasia and its large size, the patient elected to suspicious characteristic. erefore, we recommend other have the lesion excised surgically. It proved to be a papillary centers that are not high-volume centers for thyroid biopsy thyroid cancer. Another patient also had a biopsy report to review their malignancy rates among their thyroid bi- suspicious for follicular neoplasm, but elected to have opsies to ensure they are achieving the expected prevalence clinical and imaging follow-up. of malignancy in their biopsy specimens. Similarly, there were 99 thyroid biopsies performed on Molecular tests such as the Afirma Gene Expression patients with THY#2 classification, which recommends Classifier (Veracyte Inc, South San Francisco, California) ultrasound follow-up in 12 months. Again, the decision to on FNA specimens reported as FLUS are available to our perform a biopsy was usually made by the clinician based on clinicians. According to our institutional policy, a molecular the patient’s clinical history and patient preference. Of these test on thyroid FNA samples is generally reserved for pa- 99 patients, biopsy revealed papillary thyroid cancer in 3, tients who have FLUS on 2 consecutive FNA specimens. other malignancy in 1, follicular neoplasm in 7, and sus- However, for patients with a diagnosis of FLUS on FNA, picious for neoplasm in 5. Of the 12 patients with follicular our endocrinologists generally prefer to monitor these pa- neoplasm or lesion suspicious for neoplasm, 7 underwent tients or obtain a core biopsy. erefore, none of the pa- surgical excision. One patient with a suspicious lesion was tients with FLUS in our study underwent molecular testing. found to have a papillary thyroid cancer. Another patient Last, it is important to note that in our patient pop- with a follicular lesion was found to have a 0.1-cm ulation, approximately 20% of biopsies was performed on micropapillary thyroid cancer separate from the follicular patients in whom a biopsy was not recommended by the lesion. radiologist. In most cases, the biopsy was performed on the recommendation of the endocrinologist or head and neck DISCUSSION surgeon, who based their decision on additional informa- e 2015 ATA guidelines recommend conducting a tion obtained from the patient’s medical history and clinical biopsy of any solid hypoechoic nodule or solid hypoechoic presentation. ere were also a few biopsies performed at component of a partially cystic nodule ≥ 1 cm with 1 or more the patient’s insistence. Based on our 3-year experience with suspicious characteristics due to an estimated > 70% risk of our new classification system, approximately 20% of bi- malignancy. ese characteristics include irregular margins opsies may be requested based on clinical factors rather than (infiltrative, microlobulated); microcalcifications; taller than on imaging features alone. wide shape; rim calcifications with a small, extrusive soft tissue component; or evidence of extrathyroidal extension. Limitations In addition, they cite a 10% to 20% estimated risk of ere is operator variability associated with the devices malignancy for hypoechoic solid nodule ≥ 1cmwith used to obtain images of the thyroid, the sonographers’ smooth margins without any other suspicious characteris- experience with thyroid imaging, and the radiologists’ in- tics. However, as noted in a footnote within the guidelines, terpretation of the images. However, we did not review any

88 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.240 ORIGINAL RESEARCH ARTICLE Effects of Implementing a Higher Threshold for Recommending Thyroid Biopsies on Malignancy Rates

images as part of our study, nor did we record which Disclosure Statement sonographer or radiologist was involved in the cases that led The authors have no conflicts of interest to disclose. to the decision for biopsy, because the purpose of our study was to assess overall rates of malignancy in patients un- Authors’ Contributions dergoing thyroid biopsy. Kori Higashiya, BS, participated in the acquisition and analysis of data, and fi drafting, review, and submission of the final manuscript. Liam Delgesso, BS, We did not look at the speci c features of lesions that participated in the acquisition and analysis of data. Hyo-Chun Yoon, MD, PhD, determined the need for a biopsy in each time period. participated in the study design, acquisition of data, and drafting and review of the erefore, no regression analysis was performed on which final manuscript. factors among the generally accepted suspicious features were most associated with the likelihood of malignancy. Funding The authors did not receive funding for this study. ere have been numerous published studies that have tried to address this issue, including a meta-analysis that favored References 4 using a combination of features rather than a single feature. 1. Tae HJ, Lim DJ, Baek KH, et al. Diagnostic value of ultrasonography to distinguish between benign and malignant lesions in the management of thyroid nodules. Thyroid Last, we assume there was no change in the underlying 2007 May;17(5):461–6. DOI: https://doi.org/10.1089/thy.2006.0337, PMID:17542676. prevalence of thyroid cancer in our patients between the 2 2. Haugen BR, Alexander EK, Bible KC, et al. 2015 American Thyroid Association study periods. A recent meta-analysis by Furuya-Kanamori management guidelines for adult patients with thyroid nodules and differentiated thyroid 5 cancer: The American Thyroid Association Guidelines Task Force on thyroid nodules and et al reports that the baseline prevalence of incidental differentiated thyroid cancer. Thyroid 2016 Jan;26(1):1–133. DOI: https://doi.org/10.1089/ differentiated thyroid cancer in autopsy specimens has thy.2015.0020, PMID:26462967. 3. Middleton WD, Teefey SA, Reading CC, et al. Multiinstitutional analysis of thyroid nodule remained stable at about 11% since 1970. risk stratification using the American College of Radiology Thyroid Imaging Reporting and Data System. AJR Am J Roentgenol 2017 Jun;208(6):1331–41. DOI: https://doi.org/10. 2214/AJR.16.17613, PMID:28402167. CONCLUSION 4. Remonti LR, Kramer CK, Leitão CB, Pinto LC, Gross JL. Thyroid ultrasound features Using a 1-cm cutoff with 2 suspicious characteristics as and risk of carcinoma: A systematic review and meta-analysis of observational studies. Thyroid 2015 May;25(5):538–50. DOI: https://doi.org/10.1089/thy.2014.0353,PMID: the threshold for thyroid biopsy resulted in a 25% preva- 25747526. lence of malignancy in our biopsy specimens more than 5. Furuya-Kanamori L, Bell KJL, Clark J, Glasziou P, Doi SAR. Prevalence of differentiated thyroid cancer in autopsy studies over six decades: A meta-analysis. halved the volume of yearly biopsies compared to historical J Clin Oncol 2016 Oct;34(30):3672–9. DOI: https://doi.org/10.1200/JCO.2016.67. numbers. v 7419, PMID:27601555.

The Permanente Journal·https://doi.org/10.7812/TPP/20.240 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 89 n ORIGINAL RESEARCH ARTICLE Changes in Emergency Department Patient Volume and Acuity Associated with Early Stages of the COVID-19 Pandemic in a Unique Environment

Brent Lorenzen, MD1; Adam Schwartz, MD, MS1 Perm J 2021;25:20.212 E-pub: 3/10/2021 https://doi.org/10.7812/TPP/20.212

ABSTRACT reports of increased demand for emergency and intensive Introduction: Hospitals and emergency departments (EDs) care far exceeding capacity began to emerge.2 Reports of faced profound uncertainty during the COVID-19 pandemic. Early decreased admissions for specific emergent conditions such concerns regarding demand far exceeding capacity were bal- as acute coronary syndrome have been reported in Italy.3 A anced by anecdotal reports of decreased patient visits, including single tertiary care center in New Jersey reported decreased rates fi those for speci c high-acuity conditions. This study sought to of diagnosis of acute stroke during the pandemic.4 Anecdotally, identify changes in ED volume and acuity, within a specific physicians reported that daily emergency department (ED) managed care environment, associated with the onset of the pandemic. volumes had dropped to 40% to 60% of normal during late 5  Methods: Data from patient visits to 2 San Diego, California, March and early April. e Centers for Disease Control and EDs—within an integrated health-care system—were extracted Prevention (CDC) reported that nationwide ED visits during from the electronic health record. Daily patient visits, hospital the period from March 29 to April 25, 2020, declined 42% admissions from the ED, Emergency Severity Index scores, and compared to a comparable 4-week period in 2019.6 Here we mode of arrival were compared between two 28-day periods, with describe selected changes in our departments over a 28-day the 28 days following a “stay at home” order issued by the period following the “stay at home” order for all of California governor of California and a control period of the same dates in issued by Governor Gavin Newsom on March 19, 2020. is 2019. study sought to identify changes in ED volume and acuity, fi Results: These EDs observed a signi cant decrease in daily within a specific managed care environment, associated with visits (42% compared to the previous year) associated with the the onset of the pandemic. pandemic. An increased rate of hospital admissions (16.6%– 21.6%) was suggestive of an overall increase in acuity; however, changes in the distribution of Emergency Severity Index scores METHODS were less pronounced. The overall number of admissions declined Kaiser Permanente (KP) operates 2 acute care hospitals in significantly. Although overall ambulance traffic decreased, the San Diego County that operate under 1 license. San Diego proportion of patients arriving by ambulance was unchanged. county currently has a population of about 3.3 million Conclusion: Patient volume in 2 EDs dropped significantly in people. KP serves more than 630,000 members in San association with a statewide response to the COVID-19 pandemic. Diego. Annual ED volume was 133,088 in 2019. Ninety- There was also a shift in acuity as measured by the proportion of two percent of patients were members of KP. Patients do patients admitted to the hospital, but overall admissions declined, occasionally seek care at EDs outside of the KP system, suggesting sicker patients also did not seek care. and data regarding this use by members were not assessed. Before the study period, 2020 volumes were 6% INTRODUCTION greater than in 2019. A 28-day study period was selected to coincide with the “stay at home” order issued on March e onset of the novel coronavirus (COVID-19) pan- 19, 2020 (March 19–April 15, 2020). A corresponding demic in the US resulted in profound uncertainty regarding control period was selected for comparison, the equiva- hospital utilization. e first US case was confirmed on lent 28-day period from 2019 (March 19–April 15, January 20, 2020.1 However, spread of the disease was 2019). A 28-day period was chosen to control for known initially slow and there was minimal public response. As the variability that occurs based on day of the week. edate disease spread more rapidly in other parts of the world, of the “stay at home” order was chosen as an objective marker of public recognition of the pandemic and as- Author Affiliation sociated behavioral changes. Data from patient visits to 1Department of Emergency Medicine, Kaiser Permanente/Southern California Permanente Medical Group, EDs in our system are stored automatically in an elec- San Diego, CA tronic health record nonrelational database (Chronicles/  Corresponding Author Epic). e data are then transferred and stored in a Brent Lorenzen, MD ([email protected]) distinct relational database (Clarity/Oracle) daily. is

Keywords: administration, COVID-19, emergency, emergency medicine, infectious disease 90 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.212 ORIGINAL RESEARCH ARTICLE Changes in Emergency Department Patient Volume and Acuity Associated with Early Stages of the COVID-19 Pandemic in a Unique Environment

database was queried for records of daily patient volume, Table 1. Mean daily emergency department patient volumes and ED disposition, Emergency Severity Index (ESI) score, admissions fi and arrival mode for the prespeci ed time periods de- March 19–April 15, March 19–April 15, scribed. e ESI triage system is a 5-level system that 2020 2019 assigns a score based on acuity of illness and expected Daily volume (mean ± SD) 224 ± 30 388 ± 25 (P < 0.0001) 7 resource use. ESI scores of 1 and 5 were not included Daily admissions (mean ± SD) 48 ± 10 64 ± 8 (P < 0.0001) because they occur so infrequently that any comparison Daily admission, 21.6 ± 4.4 16.6 ± 1.6 (P < 0.0001) during the study time period would not be meaningful. % (mean ± SD) Each sample was assessed for normality using the SD = standard deviation. Shapiro-Wilk test, with a P value of 0.05. All samples for daily patient visits, absolute number of admissions, Table 2. Mean proportion of daily emergency department visits admission percentages, absolute number of arrivals by by Emergency Severity Index Emergency Medical Services (EMS), and percentage of March 19–April 15, March 19–April 15, arrivals by EMS were found to be distributed normally. 2020 2019 Samples of ESI scores were noted not to represent ESI-2, % (mean ± SD) 23.5 ± 4.2 24.9 ± 3.1 (P = 0.01) normally distributed populations using this test. Means ESI-3, % (mean ± SD) 67.7 ± 4.6 60.6 ± 2.9 (P < 0.0001) and standard deviations for each time period were cal- ESI-4, % (mean ± SD) 7.5 ± 3.3 14.0 ± 2.0 (P < 0.0001) culated using Excel (Microsoft). For the normally dis- ESI = Emergency Severity Index; SD = standard deviation. tributed data, comparisons between time periods were made using a 2-sample t-test for independent samples. For comparisons of the proportion of ESI scores, the Table 3. Mean daily emergency department arrival by emergency Mann-Whitney test was used. For all tests, P <0.05was medical services considered statistically significant. March 19–April 15, March 19–April 15, 2020 2019 RESULTS n (mean ± SD) 41 ± 8.7 66 ± 11.3 (P < 0.0001) Compared to the equivalent 28-day period from the Emergency department 18.2 ± 2.9 17.0 ± 2.8 (P = 0.14) visits, % (mean ± SD) previous year, daily ED visits decreased by 42%. e ab- SD = standard deviation. solute number of admissions decreased by 25%; however, the proportion of admitted patients increased by 30% (Table 1). weakness in the ESI system in the ability to discriminate ere was a small, but statistically significant decrease in patient acuity accurately. Previous research has suggested the proportion of ESI-2 visits as well as a significant in- poor accuracy and high variability with use of the ESI crease in ESI-3 visits. ese were offset by a significant system.8 Our department does not assign ESI-1 or ESI- decrease in ESI-4 visits (Table 2). 5scoreswithsufficient frequency to make any valid e absolute number of arrivals by EMS was significantly comparisons between time periods. less; however, the proportion of patients arriving by EMS It is very interesting that local changes in volume closely was not significantly different (Table 3). matched nationwide data reported by the CDC. More than 90% of our patient visits are from insured patients within DISCUSSION the KP system. It might be expected that because this Our study shows significant changes in ED patient population has access to many options for lower acuity volume and acuity within an integrated health-care delivery care,theimpactofthepandemiconEDvolumewould system in San Diego, California. e decrease in daily be blunted in comparison if patients are not as dependent patient presentations is consistent with nationwide data just on the ED to access care on a regular basis. Many of our reported by the CDC. Although the absolute number of outpatient clinics closed or decreased services dramati- hospital admissions decreased, the proportion of ad- cally during this time period, so it might be expected that mitted patients increased by 30%. is finding supports patients would actually increase use of the ED when concerns that many patients with emergent medical other venues for care were less available. ese trends conditions were not seeking care during this phase of the would suggest that other factors were likely more in- pandemic.Italsosuggestsashifttowardoverallhigher fluential in patients’ decisions to seek urgent or emergent acuity among patients seeking ED care. Changes in care. admission rates appeared to be greater in magnitude than Defining the study period and comparison periods was shifts in ESI distribution. is likely represents inherent challenging because the pandemic does not have a well-

The Permanente Journal·https://doi.org/10.7812/TPP/20.212 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 91 ORIGINAL RESEARCH ARTICLE Changes in Emergency Department Patient Volume and Acuity Associated with Early Stages of the COVID-19 Pandemic in a Unique Environment

Figure 1.

defined start date. We chose the California “stay at home” care. Consequently, routine use of the ED for lower acuity order as an objective measure. is was 6 days after a na- conditions might be expected to be less than in other envi- tional state of emergency was declared on March 13. ronments. However, the observed decline in patient volume However, California was the first state in the country to associated with the pandemic was in line with national averages. issue a “stay at home” order. Behavioral changes, including We did observe a significant decrease in the small proportion of medical care delivery and decisions to seek care, likely patients triaged as clearly lower acuity as ESI-4. is does happened progressively over a poorly defined time frame. suggestthatasignificant number of patients, who might have e true decrease in patient volume is likely underestimated used the ED for nonemergent care, chose to defer care or by this analysis, as January and February ED volumes were receive it in an alternate setting in response to the pandemic. greater in 2020 when compared to 2019. Compared to many other areas of the country and state, Limitations San Diego County has been less impacted by severe acute Our data are from EDs that function as part of an respiratory syndrome-associated corona virus 2. As of June integrated health-care delivery system that sees mostly well- 2, 2020, there were 7674 reported cases and 276 deaths in a insured patients, so the findings may not be generalizable to county with a population of approximately 3.3 million people.9 other environments. However, overall changes in patient Our data demonstrate that, even in a geographic area with a volume are in line with national data reported by the CDC. low prevalence of disease, patients did not seek medical care in Defining an appropriate study period was challenging given our EDs at a rate consistent with historical norms. Further the gradual evolution of the pandemic, and subsequent re- research is needed to identify more accurately the reasons for sponse by federal and local authorities. Comparisons may these behavioral changes. Fear of exposure to disease in the have been different depending on the selected time frames. acute care environment is one possible explanation. Our data use ESI level as a marker of acuity, which may not Approximately 92% of our ED patients are members of an be a reliable or accurate indicator of actual acuity. Similarly, integrated health-care delivery system with consistent access to although an increased rate of hospital admissions might

92 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.212 ORIGINAL RESEARCH ARTICLE Changes in Emergency Department Patient Volume and Acuity Associated with Early Stages of the COVID-19 Pandemic in a Unique Environment

suggest increased acuity, other factors may have contributed References to decisions regarding admission to the hospital. 1. Holshue ML, DeBolt C, Lindquist S, et al. First case of 2019 novel coronavirus in the United States. N Engl J Med 2020 Mar;382(10):929–36. DOI: https://doi.org/10.1056/ NEJMoa2001191, PMID:32004427. CONCLUSIONS 2. Grasselli G, Pesenti A, Cecconi M. Critical care utilization for the COVID-19 outbreak in Lombardy, Italy: Early experience and forecast during an emergency response. J Am Med Our data show a large decrease in patient visits to the Assoc 2020 Apr;323(16):1545–6. DOI: https://doi.org/10.1001/jama.2020.4031, PMID: ED within an integrated health-care delivery system in 32167538. association with a statewide response to the severe acute 3. De Filippo O, D’Ascenzo F, Angelini F, et al. Reduced rate of hospital admissions for ACS during COVID-19 outbreak in northern Italy. N Engl J Med 2020 Jul;383:88–9. DOI: https:// respiratory syndrome-associated corona virus 2 pan- doi.org/10.1056/NEJMc2009166 demic. Surrogate markers of ESI distribution and 4. Siegler JE, Heslin ME, Thau L, Smith A, Jovin TG. Falling stroke rates during COVID-19 pandemic at a comprehensive stroke center. J Stroke Cerebrovasc Dis 2020 Aug;29(8): percentage of patients admitted to the hospital suggest a 104953. DOI: https://doi.org/10.1016/j.jstrokecerebrovasdis.2020.104953, PMID: shift toward higher acuity and a decrease in low-acuity 32689621. 5. Stone W, Yu E. Eerie emptiness of ERs worries doctors: Where are the heart attacks and visits. Overall arrivals via EMS decreased similarly to strokes? NPR. Accessed May 7, 2020. www.npr.org/sections/health-shots/2020/05/06/ all patient visits without a proportional increase in EMS 850454989/eerie-emptiness-of-ers-worries-doctors-where-are-the-heart-attacks-and- ffi v strokes?fbclid=IwAR22eNo0RCcteZcN6LWCnp764Hht8A8Tg-aSbi9rpD4uVsnwfVl- tra c. 3RrLMns 6. Hartnett KP, Kite-Powell A, DeVies J, et al. Impact of the COVID-19 pandemic on emergency department visits—United States, January 1, 2019–May 30, 2020. MMWR Disclosure Statement Morb Mortal Wkly Rep 2020 Jun;69(23):699–704. DOI: https://doi.org/10.15585/mmwr. The authors have no conflicts of interest to disclose. mm6923e1 7. Gilboy N, Tanabe T, Travers D, Rosenau AM. Emergency Severity Index (ESI): A triage tool for emergency department care, version 4.Implementation handbook 2012 edition. Agency Authors’ Contributions for Healthcare Research and Quality: 2011. Brent Lorenzen, MD, and Adan Schwartz, MD, MS, contributed equally to the 8. Mistry B, de Ramirez S, Kelen G, et al. Accuracy and reliability of emergency department study design, data collection, data analysis, and manuscript preparation. They were triage using the Emergency Severity Index: An international multicenter assessment. Ann not assisted. Emerg Med 2018 May;71(5):581–e3. DOI: https://doi.org/10.1016/j.annemergmed.2017. 09.036, PMID:29174836. 9. SanDiegoCounty.gov. Coronavirus in San Diego County. Local situation. Accessed June 4, Funding 2020. www.sandiegocounty.gov/content/sdc/hhsa/programs/phs/community_ The authors did not receive financial contributions for this endeavor. epidemiology/dc/2019-nCoV/status.html

The Permanente Journal·https://doi.org/10.7812/TPP/20.212 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 93 n ORIGINAL RESEARCH ARTICLE Colorectal Cancer and Return to Work: A Pilot Study of Recruiting Cancer Survivors and Their Employers

Inga Gruß, PhD1; Cathy J Bradley, PhD, MPA2; Matthew P Banegas, PhD, MPH1 Perm J 2021;25:20.198 E-pub: 03/17/2021 https://doi.org/10.7812/TPP/20.198

ABSTRACT INTRODUCTION Introduction: In this study we assessed the feasibility of e number of adults in the US living with a history of recruiting colorectal cancer survivors and their employers to cancer is on the rise. In 2019, an estimated 16.9 million participate in research on returning to work after a cancer individuals with a history of cancer were alive, and this diagnosis. number is projected to increase to more than 22.1 million by Methods: Semistructured qualitative interviews were con- 2030.1 Sixty-three percent of cancer survivors continue to ducted with 6 survivors and 4 employers to collect feedback on 2 work through treatment or return to work (RTW) after recruitment approaches: 1) an employee/cancer survivor-focused 2 ff approach whereby researchers obtained employer contact in- treatment. However, the short- and long-term e ects of ff ’ formation directly from survivors and 2) an employer-focused cancer and its treatment can a ect a survivor s ability to 3-5 approach whereby researchers interviewed employers to identify maintain employment. Limitations in physical and pathways to recruiting survivor–supervisor dyads. Recordings of cognitive functioning, as well as emotional and psychosocial all interviews were transcribed and analyzed. effects, may impair patients’ and survivors’ abilities to Results: Of the 6 survivor participants, 3 (50%) consented to complete work-related tasks.6,7 Patients with colorectal follow up with their supervisors. One of the 3 supervisors cancer (CRC) may also have unique physiological symp- responded but declined participation in the study. The 4 par- toms that affect their ability to work, such as frequent or fi ticipating employers included 2 compensation/bene ts man- irregular bowel movements.8 Acknowledging and addressing fi agers and 2 human resources managers. Employers identi ed these side effects is critical for supporting CRC survivors’ specific avenues for potential recruitment of survivor–supervisor fi employment needs and goals. Maintaining employment is dyads, including rst obtaining organizational support for par- fi ticipating in this type of research. often critical for nancial security and health insurance ff 9,10 Discussion: While challenges and opportunities exist with coverage, and o ers cancer survivors a sense of normalcy. both the employee- and the employer-focused recruitment ap- A growing body of research has highlighted the need to proaches, our findings suggest that an employer-focused ap- obtain multiple stakeholder perspectives to illuminate the proach, whereby researchers obtain organizational support for barriers and facilitators of continued employment for cancer the research first, may be more promising for recruiting survivor– survivors.11 ese perspectives are critical for developing supervisor dyads to studies on cancer and employment. Results timely and well-managed RTW processes for individuals from this study underscore the importance of gaining support with cancer.12-14 Employers are a stakeholder group that — from all stakeholders from administrators to employees who are have a key role in ensuring employment of cancer sur- cancer survivors. vivors, but their perspectives are largely absent from the Conclusion: Our study informs recruitment strategies that literature.3,4 Cancer survivors have consistently identified a bring together cancer survivors and employers to improve our understanding of the barriers and facilitators of returning to work supportive work environment, ongoing communication, ’ after cancer, in effort to develop interventions that mitigate and realistic employer expectations about employees abil- employment challenges for cancer survivors. ities after RTW as crucial components for enabling a successful RTW process.15,16 Although limited, existing research on the perspective of employers has identified open and ongoing communication and RTW policies as potential facilitators to RTW, and lack of knowledge about cancer and competing interests as central barriers.10,17-19 ese documented barriers and facilitators to successful RTW point to the importance of obtaining the perspectives of both cancer survivors and their employers to identify Author Affiliations common areas of concern and priorities for improvement. 1Kaiser Permanente Center for Health Research, Portland, OR 2Colorado School of Public Health, University of Colorado Denver, Aurora, CO Moreover, for future trials that will test interventions to improve RTW after cancer, participation of dyads will be Corresponding Author required, and exploring ways to recruit them effectively is Matthew P Banegas, MPH, PhD ([email protected]) warranted. Only a few research projects, however, have – Keywords: colorectal cancer survivors, employer perspective, employment, recruitment methods, return to recruited cancer survivor employer dyads for research, using work 94 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.198 ORIGINAL RESEARCH ARTICLE Colorectal Cancer and Return to Work: A Pilot Study of Recruiting Cancer Survivors and Their Employers

snowball sampling as well as asking cancer survivors for Northwest Workforce Health—a team that develops em- consent to contact and enroll their supervisors.20,21 ese ployee wellness engagement strategies and programs with studies documented certain challenges recruiting employers organizations that offer health insurance plans through for research on cancer survivorship and employment, in- Kaiser Permanente. Eligibility for the employer participants cluding the small number of survivors within a single included a willingness to discuss the challenges and needs of employer, and confidentiality issues about health and work survivors in maintaining employment and RTW. Kaiser that make it challenging for employers to discuss individual Permanente Northwest Workforce Health contacted and cases of cancer survivors.10 As such, more research is needed sent study information to 10 organizations. Five organi- to assess the feasibility of recruiting cancer survivor– zations responded to the initial recruitment inquiry. Study employer dyads to enable a comprehensive stakeholder staff contacted these organizations by phone or email to engagement program. describe the study in detail and to assess their interest in As part of a study assessing the challenges and needs of participating. Of the 5 organizations that responded, 1 CRC survivors in maintaining employment and RTW from declined to participate because of time constraints; 4 agreed both the survivor and employer perspective,22 we assessed to participate. All 4 participating organizations also con- the feasibility of enrolling cancer survivor–employer dyads sented to make a staff member available for a 30- to 45- using 2 different approaches: 1) a survivor/employee-based minute phone interview. We enrolled 2 compensation and approach that identifies the supervisors/employers through benefits managers and 2 human resources managers. participating CRC survivors and 2) an employer-based approach that identifies supervisors through administrative- Data Collection and Analysis level access from the employer. We conducted qualitative We conducted all interviews by phone and recorded these interviews with survivors, human resource managers, and conversations after obtaining participants’ permission. benefits managers to understand better the feasibility of Recordings were transcribed and analyzed using NVivo 10 these recruitment strategies, and to identify policies and (QSR International). All participants were offered a $25 context that could facilitate recruitment of survivor– gift card for their participation. e Kaiser Permanente employer dyads. Northwest Institutional Review Board approved this study. For survivors, part of the interviews focused on their METHODS willingness to contact about participating in the study the Study Population direct or immediate supervisors with whom they worked at We recruited CRC survivors through the Patient Out- the time of diagnosis and their willingness to share the contact comes Research to Advance Learning (PORTAL) Net- details of these supervisors with researchers so that the em- work’s CRC cohort. is established cohort includes CRC ployers could be recruited for study participation. efol- survivors from 6 health-care systems who were diagnosed lowing questions were asked of all employee participants: between 2010 and 2014.23 Survivors were eligible for re- · We are exploring ways to hear from employee and cruitment if they were between 18 and 70 years of age, fluent supervisor/manager pairs about their experiences with in English, employed at the time of their cancer diagnosis, colorectal cancer. If you were asked to provide your and enrolled at Kaiser Permanente Northwest (1 of the 6 supervisor’s contact information so that study staff could PORTAL CRC sites). Forty survivors who met these invite him or her to participate in a confidential survey criteria were selected randomly and sent recruitment letters. about cancer and the workplace, would you feel com- Eight of these 40 eligible participants reported to be either fortable doing this? If not, what reservations do you have? retired or not working at the time of the diagnosis, 10 · Would you be willing to give us permission to contact declined participation, and 9 could not be reached. Of the your current/former supervisor to ask whether he or she 13 who agreed to participate, 10 survivors provided consent would be willing to do an interview (just like the one we and completed a 30- to 45-minute telephone interview. are doing now) about his or her experience supervising an Four of the 10 consenting survivor participants were employee with cancer? Follow-up note: Make sure we employed in a position, at the time of diagnosis, that did not emphasize that the focus of the interview is not about the have a direct supervisor (self-employed [n = 3]) or no direct employee personally. supervisor [n = 1]) and were excluded from the analyses. If the interviewee provided employer contact informa- e final analytic sample included 6 participants. tion, the study team attempted to contact the supervisor to To assess the employer-based approach, from electronic request a 30- to 45-minute semistructured interview. health record data we generated a list of companies and For employers, interviews focused on their willingness to organizations that employed PORTAL CRC cohort sur- help recruit both cancer survivors and their supervisors for a vivors, then submitted the list to Kaiser Permanente study on cancer and employment, and their suggestions for

The Permanente Journal·https://doi.org/10.7812/TPP/20.198 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 95 ORIGINAL RESEARCH ARTICLE Colorectal Cancer and Return to Work: A Pilot Study of Recruiting Cancer Survivors and Their Employers

how best to recruit supervisors for a study. e following think their supervisor from the time of diagnosis would be questions were asked of all employer participants: willing to participate. · We are especially interested in studying employee and Of the supervisors for whom survivor participants pro- supervisor/manager pairs. Do you have any thoughts vided contact information, 1 was contacted by email about how we might reach out to employee and whereas the other 2 supervisors were mailed letters to their supervisor/manager pairs in a way that would allow both place of work. e supervisor who was contacted by email to feel comfortable? Follow-up prompts: What privacy or responded, but declined participation in the study because legal considerations do we have to be aware off? How can we of time constraints. Neither supervisor who was contacted respect the privacy of both parties? by mail responded to our letters. · How comfortable would you feel with your employee giving us your contact information so that we could invite Employer Approach you to participate in a confidential survey about your Employer participants included 2 compensation and experience managing employees experiencing cancer? benefits managers who worked in public administration Follow-up prompts: How comfortable do you think your and 2 human resources managers who worked in the ed- employee might feel with regard to providing us with your ucational sector (secondary and tertiary education sector). contact information? Do you have any thoughts about how Participants were primarily women (75%). All employer we could make employees who are cancer survivors feel more participants recognized the importance of the study and the comfortable participating in research? interest in collecting data from both CRC survivors and e transcripts for both survivor and employer interviews their supervisors to understand better and to improve were analyzed by 2 researchers (Inga Gruß, PhD, and processes related to cancer survivorship and employment. Matthew P Banegas, PhD, MPH), who combined de- When asked about the approach they felt would be most ductive and inductive approaches to content analysis.24 appropriate for recruiting supervisors for a research study, First, the researchers developed an initial code list for they identified 2 potential pathways: 1) obtain consent from the interviews, including predefined themes and addi- survivor participants to contact their employer, then initiate tional codes identified during initial transcript review. contact with the employer organization administrator (ie, rough an iterative process, each researcher coded 2 human resources manager) to help identify and recruit interview transcripts using the initial code list, after which supervisors; or 2) contact administrators at the employer we met to compare and discuss the coded transcripts, organizations and then work with the organization to re- revise the codes and code definitions, and generate a final cruit supervisors within the organization who have expe- code list. e final code list was then used to analyze all rience with employees who are cancer survivors: “But I think interview transcripts. you could do it both ways. You could do it through the . . . patient to the supervisor. And do they think that their supervisor would RESULTS be interested in participating, because they could at least ask Details of the study population have been reported in them. And then you go through the employer, through a contact detail previously.22 person, and say, ‘is employee . . . needs to connect with us,’ so we know it’s okay. [Compensation and Benefits Manager]” Participant/CRC Survivor Approach All the employer participants emphasized the critical role Of the 6 survivor participants included in the study, 5 had of organizational consent and support. Some felt that a direct supervisor at the time of diagnosis and returned to human resource departments could be helpful in identifying work; 1 participant had retired since receiving the cancer potential legal concerns (related to the Health Insurance diagnosis. Of the 5 participants with a direct supervisor, 3 Portability and Accountability Act) associated with rep- agreed to share their supervisor’s contact information, al- resentatives of the employer participating in research, and to though 1 asked us to withhold their name when contacting ensure that any concerns are appropriately addressed: “But their former employer. None of the 3 survivor participants in terms of whether there would be a legal requirement for us as who agreed to share their supervisor’s contact information a public institution, for that kind of conversation ...I’d was in the same job they held at the time of their diagnosis probably go to my legal department and say, ‘You know . . . this (2 had been laid off after RTW and 1 had retired from the question has come up. Do you see any problems?’ [Human position at the time of diagnosis, although all had moved to Resources Manager]” Employer participants also noted a new employer). Of the 2 survivor participants who re- that, if approval is sought at the organizational level, the ported they did not feel comfortable or willing to share their research or compliance (legal) departments of these orga- supervisor’s contact information, 1 declined to participate nizations should be integrated into the process to ensure all because of privacy issues and the other reported they did not steps of the research process comply with organizational

96 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.198 ORIGINAL RESEARCH ARTICLE Colorectal Cancer and Return to Work: A Pilot Study of Recruiting Cancer Survivors and Their Employers

policies and federal, state, and local laws: “We have our suggest that an employer-focused approach, whereby re- institutional research department. So, if I were to want to do a searchers obtain organizational support for the research survey, I would work with them to make sure that it was a study first, may be a more promising approach for future legitimate survey in terms of processes and . . . legal require- studies on cancer and employment, rather than asking for ments and whatnot. So, I’m assuming that there are some kind help from cancer survivors to recruit their supervisors (the of standards, you know, especially in the medical area with survivor-/employee-focused approach). By collaborating confidentiality issues, that there would be . . . things that may with the organizational administration, the employer- need to be signed or . . . you know, kind of disclaimers and focused approach may address potential barriers to re- whatnot. [Human Resources Manager]” search participation, including concerns about compliance e employer participants described other advantages to with organizational policies. e survivor-/employee- contacting employers first, including: 1) the employer or- focused approach may not be practical or feasible because ganization can encourage both employees who are cancer of changes in job roles/positions of both the survivor and the survivors and their supervisors to participate, 2) the em- supervisor, policy concerns, reluctance to participate on the ployer organization can endorse organizational interest in part of survivors, or lack of supervisor interest. improving RTW processes, and 3) the employer organi- We found a benefit to including human resources zation can convey to employee participants that information managers or benefits administrators in conversations about resulting from the research will or will not be shared with study recruitment. ese stakeholders can foster recruit- the organization: “So, I think it would be most appropriate if ment of supervisor participants by facilitating commu- an email came from me or my office. And it said something like, nication, connecting to specificstaff, approving study ‘You know, we have . . . a study related to cancer patients and . . participation across the employer organization, and en- . for those [employees] who have gone through [a cancer di- suring organizational policies are upheld. is finding agnosis] or are continuing to deal with or go through cancer supports the results of previous work conducted by Tiedtke treatment, if you feel comfortable . . . in wanting to participate et al,19 who found that human resources managers are likely in the survey with your supervisor, then you could contact us.’ to be knowledgeable about employer policies and legal [Compensation and Benefits Manager]” restrictions, whereas departmental managers who main- e 2 employer participants who were human resource tain direct communication with their employees who are managers also identified practical considerations that could cancer survivors may lack this knowledge. Such organiza- improve recruitment, such as conducting web-based surveys tional knowledge and support may improve the involvement rather than qualitative interviews to improve feasibility, of various administrative-level stakeholders during the allow participants flexibility to complete the surveys on their recruitment process, and may empower supervisors to own time, and protect the privacy of the participants. participate. Employer participants also identified possible barriers to When recruiting employers to cancer studies, we found recruitment, including the inability of public service that 1 limiting factor to successful recruitment of sufficient agencies to accept gift cards or incentives for participating in numbers of employers may be the low number of cancer research, limited knowledge about privacy regulations survivors in any given organization and, thus, few employer among supervisors, and fear of violating these regulations: representatives who have experience with an employee/ “[Protected Health Information is] a really big deal. And we are direct report who has been diagnosed with cancer. is an employer, so we are completely connected to their [employees’] may be particularly true among small organizations that livelihood. And nobody [among supervisors] would even talk to have relatively few employees. Survivors employed in small me about [the employee] ....AndI’m like, ‘Look, I’m not organizations may need the most assistance with RTW looking for [Protected Health Information], guys. You know, because these organizations may not have established tell me about the people that you know that the supervisor knows policies or legal protections, nor the experience with other there’sa[cancer] diagnosis. So that I can reach out to that employees with a cancer history, to develop procedures and supervisor and ask, Hey, here’s what’s going on. It’d be great to accommodations that best support RTW after cancer.3 get your input.’ And they . . . just completely shut down. Prior studies on RTW after cancer have recruited em- [Compensation and Benefits Manager]” ployer participants using approaches such as recruiting through professional conferences and societies, insurance DISCUSSION companies, LinkedIn, and disability management pro- rough interviews with cancer survivors and employers grams, and by using snowball techniques.14,17,19,21,25 An- of cancer survivors, we identified key challenges and op- other approach may be for researchers to develop effective portunities of 2 approaches to recruiting survivor–supervisor collaborations with large employer organizations, such as dyads for research on RTW after cancer. Our findings Chambers of Commerce or labor unions, that could provide

The Permanente Journal·https://doi.org/10.7812/TPP/20.198 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 97 ORIGINAL RESEARCH ARTICLE Colorectal Cancer and Return to Work: A Pilot Study of Recruiting Cancer Survivors and Their Employers

large sampling frames of employer organizations from Last, our sampling frame included employers who were which to engage and recruit participants for studies on recruited through their connection with Kaiser Permanente cancer and employment. Northwest, which may lead to selection bias, as these or- Participants identified several challenges for recruiting ganizations may be more likely to participate in research or cancer survivor–supervisor dyads. Fear of violating privacy differ in other characteristics. regulations, as well as limitations on accepting study in- centives, may discourage participation. Several participants CONCLUSIONS had also changed jobs, which may further affect the interest Our study highlights the challenges and opportunities to in participating in a study, and poses challenges for both survivor-focused and employer-focused approaches to recruiting dyads. Other challenging factors identified in recruitment of survivor–supervisor dyads, and suggests that previous research include confidentiality and legal issues employer-focused approaches may be the most feasible for surrounding employee health and workplace accommoda- overcoming barriers to supervisor participation. Employer tions, as well as fear of job loss.10,22,25,26 participants recognized the importance of participating in Two previous studies that successfully enrolled cancer research of RTW after cancer and suggested different av- survivor–employer dyads relied on the snowballing method enues of recruiting eligible survivor–supervisor dyads. Re- and participant-driven (ie, employee-focused) recruitment, sults from this study underscore the importance of gaining respectively. Yagil et al21 recruited 12 dyads through support from all levels of the employer organizations—from Facebook and through human resource managers who administrators to employees who are cancer survivors. Our contacted survivors and their supervisors by phone. e findings help inform the development of recruitment authors did not provide any information how they identified strategies that bring together survivors and employers, cancer survivors, or the number of dyads they contacted to improving our understanding of the barriers and facilitators enroll 12 successfully. Eskilsson et al20 recruited cancer of RTW after cancer, and leading to interventions that survivors through an outpatient rehabilitation program and mitigate employment challenges for cancer survivors. v then contacted their supervisors; these researchers were able to enroll 16 of 22 dyads. eir success in pursuing a cancer Disclosure Statement survivor-/employee-focused approach may be a result, in Matthew P Banegas, PhD, MPH, has received funding from AstraZeneca, paid ff to the institution, for research outside the scope of this work. Inga Gruß, PhD, and part, of the di erent social and cultural context within Cathy J Bradley, PhD, MPA, have no conflicts of interests to declare. which their study was conducted. In Sweden, attitudes among cancer survivors and employers about participating Authors’ Contributions in RTW research may be different. Matthew P Banegas, PhD, MPH, and Inga Gruß, PhD, conceived and designed the study and collected the qualitative data. All authors wrote, reviewed, and Limitations revised the manuscript. e focus of this work was to assess the feasibility of recruiting cancer survivor–supervisor dyads to research the Funding fi This study used the infrastructure developed by the PORTAL Network, a impact of cancer on employment. Our ndings are based consortium of 3 integrated delivery systems (Kaiser Permanente, HealthPartners, on a small sample of survivors and employers. Study par- and Denver Health) and their affiliated research centers. Research reported in this ticipants included a limited number of employer represen- article was funded in part through a Patient-Centered Outcomes Research Institute Award (CDRN-1306-04681 Phase II), and the Kaiser Permanente Northwest tatives, all of whom were either human resource managers or Center for Health Research. benefits managers, and may not reflect the views of su- pervisors or employer representatives. Furthermore, we did References not collect information about the timing of their experience 1. American Cancer Society. Cancer treatment & survivorship facts & figures 2019–2021. with a cancer survivor employee, which may have been Atlanta, GA: American Cancer Society; 2019. 2. Stone DS, Ganz PA, Pavlish C, Robbins WA. Young adult cancer survivors and work: A several years ago (ie, recall bias) and may not reflect current systematic review. J Cancer Surviv 2017 Dec;11(6):765–81. DOI: https://doi.org/10.1007/ policies or legislation. Self-employed cancer survivors and s11764-017-0614-3, PMID:28478587. 3. Bradley CJ, Brown KL, Haan M, et al. Cancer survivorship and employment: those without a direct supervisor, which we excluded from Intersection of oral agents, changing workforce dynamics, and employers’ our analysis, may face different employment obstacles al- perspectives. J Natl Cancer Inst 2018 Dec;110(12):1292–9. DOI: https://doi.org/ 10.1093/jnci/djy172, PMID:30346557. together. For example, self-employed cancer survivors have 4. de Moor JS, Alfano CM, Kent EE, et al. Recommendations for research and practice to unique needs and, if they have employees, they may not improve work outcomes among cancer survivors. J Natl Cancer Inst 2018 Oct;110(10): 1041–7. DOI: https://doi.org/10.1093/jnci/djy154, PMID:30252079. want to share their diagnosis with them. Accordingly, 5. Kent EE, Davidoff A, de Moor JS, et al. Impact of sociodemographic characteristics on developing research approaches that are inclusive of dif- underemployment in a longitudinal, nationally representative study of cancer survivors: Evidence for the importance of gender and marital status. J Psychosoc Oncol 2018 May– ferent types of employment and supervisory structures is Jun;36(3):287–303. DOI: https://doi.org/10.1080/07347332.2018.1440274, PMID: essential for future research on cancer and employment. 29634413.

98 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.198 ORIGINAL RESEARCH ARTICLE Colorectal Cancer and Return to Work: A Pilot Study of Recruiting Cancer Survivors and Their Employers

6. Ekwueme DU, Yabroff KR, Guy GP, Jr., et al. Medical costs and productivity losses 16. Yarker J, Munir F, Bains M, Kalawsky K, Haslam C. The role of communication and of cancer survivors—United States, 2008–2011. MMWR Morb Mortal Wkly Rep support in return to work following cancer-related absence. Psycho Oncol 2010 Oct; 2014 Jun;63(23):505–10. https://www.cdc.gov/mmwr/preview/mmwrhtml/mm6323a2. 19(10):1078–85. DOI: https://doi.org/10.1002/pon.1662 htm?s_cid=mm6323a2_w. PMID:24918485. 17. McKay G, Knott V, Delfabbro P. Return to work and cancer: The Australian 7. Zheng Z, Yabroff KR, Guy GP, Jr., et al. Annual medical expenditure and productivity loss experience. J Occup Rehabil 2013 Mar;23(1):93–105. DOI: https://doi.org/10. among colorectal, female breast, and prostate cancer survivors in the United States. 1007/s10926-012-9386-9 J Natl Cancer Inst 2016 May;108(5):djv382. DOI: https://doi.org/10.1093/jnci/djv382 18. Stergiou-Kita M, Pritlove C, van Eerd D, et al. The provision of workplace 8. Arndt V, Merx H, Stegmaier C, Ziegler H, Brenner H. Quality of life in patients with accommodations following cancer: Survivor, provider, and employer perspectives. colorectal cancer 1 year after diagnosis compared with the general population: A J Cancer Surviv 2016 Jun;10(3):489–504. DOI: https://doi.org/10.1007/s11764-015-0492- population-based study. J Clin Oncol 2004 Dec;22(23):4829–36. DOI: https://doi.org/10. 5, PMID:26521166. 1200/JCO.2004.02.018, PMID:15570086. 19. Tiedtke C, Donceel P, de Rijk A, Dierckx de Casterlé B. Return to work following breast 9. Veenstra CM, Wallner LP, Jagsi R, et al. Long-term economic and employment outcomes cancer treatment: The employers’ side. J Occup Rehabil 2014 Sep;24(3):399–409. DOI: among partners of women with early-stage breast cancer. J Oncol Pract 2017 Nov;13(11): https://doi.org/10.1007/s10926-013-9465-6, PMID:23982854. e916–26. DOI: https://doi.org/10.1200/JOP.2017.023606, PMID:28880714. 20. Eskilsson T, Norlund S, Lehti A, Wiklund M. Enhanced capacity to act: Managers’ 10. Greidanus MA, de Boer AGEM, de Rijk AE, et al. Perceived employer-related barriers and perspectives when participating in a dialogue-based workplace intervention for facilitators for work participation of cancer survivors: A systematic review of employers’ employee return to work. J Occup Rehabil 2020. DOI: https://doi.org/10.1007/s10926- and survivors’ perspectives. Psycho Oncol 2018 Mar;27(3):725–33. DOI: https://doi.org/ 020-09914-x 10.1002/pon.4514, PMID:28753741. 21. Yagil D, Goldblatt H, Cohen M. Dyadic resources in the return to work of cancer 11. Fitch MI, Nicoll I. Returning to work after cancer: Survivors’, caregivers’, and employers’ survivors: Exploring supervisor–employee perspectives. Disabil Rehabil 2019 Sep; perspectives. Psycho Oncol 2019 Apr;28(4):792–8. DOI: https://doi.org/10.1002/pon. 41(18):2151–8. DOI: https://doi.org/10.1080/09638288.2018.1459885,PMID: 5021, PMID:30720242. 29631449. 12. Dorland HF, Abma FI, Roelen CAM, Smink JG, Ranchor AV, Bultmann¨ U. Factors 22. Gruß I, Hanson G, Bradley C, et al. Colorectal cancer survivors’ challenges to returning to influencing work functioning after cancer diagnosis: A focus group study with cancer work: A qualitative study. Eur J Cancer Care 2019 Jul;28(4):e13044. DOI: https://doi.org/ survivors and occupational health professionals. Support Care Cancer 2016 Jan;24(1): 10.1111/ecc.13044 261–6. DOI: https://doi.org/10.1007/s00520-015-2764-z, PMID:26022706. 23. Feigelson HS, McMullen CK, Madrid S, et al. Optimizing patient-reported outcome and 13. Tiedtke C, Donceel P, Knops L, Désiron H, Dierckx de Casterlé B, de Rijk A. Supporting risk factor reporting from cancer survivors: A randomized trial of four different survey return-to-work in the face of legislation: Stakeholders’ experiences with return-to-work methods among colorectal cancer survivors. J Cancer Surviv 2017 Jun;11(3):393–400. after breast cancer in Belgium. J Occup Rehabil 2012 Jun;22(2):241–51. DOI: https://doi. DOI: https://doi.org/10.1007/s11764-017-0596-1, PMID:28084606. org/10.1007/s10926-011-9342-0, PMID:22105670. 24. Elo S, Kyngas¨ H. The qualitative content analysis process. J Adv Nurs 2008 Apr;62(1): 14. Stergiou-Kita M, Grigorovich A, Tseung V, et al. Qualitative meta-synthesis of survivors’ 107–15. DOI: https://doi.org/10.1111/j.1365-2648.2007.04569.x, PMID:18352969. work experiences and the development of strategies to facilitate return to work. J Cancer 25. Stergiou-Kita M, Pritlove C, Kirsh B. The “Big C”-stigma, cancer, and workplace Surviv 2014 Dec;8(4):657–70. DOI: https://doi.org/10.1007/s11764-014-0377-z, PMID: discrimination. J Cancer Surviv 2016 Dec;10(6):1035–50. DOI: https://doi.org/10.1007/ 24993807. s11764-016-0547-2, PMID:27170116. 15. Kennedy F, Haslam C, Munir F, Pryce J. Returning to work following cancer: A qualitative 26. Brown RF, Owens M, Bradley C. Employee to employer communication skills: Balancing exploratory study into the experience of returning to work following cancer. Eur J Cancer cancer treatment and employment. Psycho Oncol 2013 Feb;22(2):426–33. DOI: https:// Care 2007 Jan;16(1):17–25. DOI: https://doi.org/10.1111/j.1365-2354.2007.00729.x doi.org/10.1002/pon.2107, PMID:22162192.

The Permanente Journal·https://doi.org/10.7812/TPP/20.198 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 99 n ORIGINAL RESEARCH ARTICLE Time-varying Reproduction Numbers of COVID-19 in Georgia, USA, March 2, 2020 to November 20, 2020

Kamalich Muniz-Rodriguez, DrPH1; Gerardo Chowell, PhD2; Jessica S Schwind, PhD1; Randall Ford, DDS3; Sylvia K Ofori, MPH1; Chigozie A Ogwara, BS1; Margaret R Davies, BS1; Terrence Jacobs, BS1; Chi-Hin Cheung, MS4; Logan T Cowan, PhD1; Andrew R Hansen, DrPH3; Isaac Chun-Hai Fung, PhD1 Perm J 2021;25:20.232 E-pub: 03/17/2021 https://doi.org/10.7812/TPP/20.232

ABSTRACT impacted the state of Georgia as well as other jurisdictions Background: In 2020, Severe Acute Respiratory Syndrome within the US. Within Georgia, Metro Atlanta counties Coronavirus 2 impacted Georgia, USA. Georgia announced a have been the hardest hit by the virus, with thousands of state-wide shelter-in-place on April 2 and partially lifted restric- confirmed cases cumulatively: 37,238 in Fulton, 36,407 in tions on April 27. We estimated the time-varying reproduc- Gwinnett, 25,853 in Dekalb, 26,255 in Cobb, and 12,314 R tion numbers ( t) of COVID-19 in Georgia, Metro Atlanta, in Hall County as of November 20, 2020.1 Dougherty and Dougherty County and environs from March 2, 2020, to County, with Albany as the county seat, was an early November 20, 2020. Methods: We analyzed the daily incidence of confirmed COVID-19 hotspot in southeastern Georgia and reported COVID-19 cases in Georgia, Metro Atlanta, and Dougherty County a large number of cases (as of November 20, 2020: cu- and its surrounding counties, and estimated Rt using the R mulative number, 3431; incidence rate, 3816 per 100,000 1 package EpiEstim. We used a 9-day correction for the date of individuals). In Georgia, every county government had report to analyze the data by assumed date of infection. the power to impose preventive measures to reduce viral Results: The median Rt estimate in Georgia dropped from transmission as they see fit, before the state imposed a between 2 and 4 in mid-March to < 2 in late March to around 1 state-wide emergency that overrode the autonomy of county from mid-April to November. Regarding Metro Atlanta, Rt fluc- governments (Table 1). On March 23, 2020, the Georgia tuated above 1.5 in March and around 1 since April. In Dougherty state government issued an executive order requesting citizens R County, the median t declined from around 2 in late March to with underlying conditions and those with a COVID-19 0.32 on April 26. Then, Rt fluctuated around 1 in May through diagnosis to shelter in place.2 Certain businesses were to November. Counties surrounding Dougherty County registered remain closed and no more than 10 individuals could gather in an increase in Rt estimates days after a superspreading event  occurred in the area. a location without maintaining a distance of at least 6 feet. e ff Conclusions: In Spring 2020, Severe Acute Respiratory Syn- order also called for restaurants to o er curbside pickup or 2 drome Coronavirus 2 transmission in Georgia declined likely delivery only. On April 2, 2020, a state-wide shelter-in-place because of social distancing measures. However, because re- ordinance was enacted by the governor, allowing only essential strictions were relaxed in late April and elections were conducted services to operate (implemented on April 3).3 eGeorgia R fl in November, community transmission continued, with t uc- state government announced on April 27, 2020, during a tuating around 1 across Georgia, Metro Atlanta, and Dougherty press conference, that services such as beauty salons, barber County as of November 2020. The superspreading event in shops, stores, and restaurants can reopen if they follow ff Dougherty County a ected surrounding areas, indicating the pertinent social distancing measures specified by the state.4 possibility of local transmission in neighboring counties. On May 12, 2020, the state government recommended residents and visitors to the state wear face coverings, practice INTRODUCTION social distance, and limit gatherings. On July 28, 2020, and In 2020, the COVID-19 pandemic caused by severe with a renewal on November 1, 2020, all individuals in the acute respiratory syndrome coronavirus 2 (SARS-CoV-2) state of Georgia with a positive or suspected COVID-19 diagnosis should isolate until their infectious period is over, and those exposed to the virus should comply with a 14-day 5 Author Affiliations quarantine. As the COVID-19 epidemic in Georgia con- 1Department of Biostatistics, Epidemiology and Environmental Health Sciences, Jiann-Ping Hsu College of Public tinues, it is important to quantify the epidemiologic char- Health, Georgia Southern University, Statesboro, GA acteristics of COVID-19 so that we may formulate policies 2Department of Population Health Sciences, School of Public Health, Georgia State University, Atlanta, GA 3Department of Community Health and Health Policy, Jiann-Ping Hsu College of Public Health, Georgia and implement interventions to minimize transmission and Southern University, Statesboro, GA mortality. 4Independent researcher, Hong Kong Special Administrative Region To characterize the transmission potential of an epi- demic, it is necessary to calculate the reproduction number Corresponding Author 6  Isaac Chun-Hai Fung, PhD ([email protected]) based on the trajectory of the incidence curve. e basic

Keywords: coronavirus, COVID-19, reproductive number, SARS-CoV-2 100 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.232 ORIGINAL RESEARCH ARTICLE Time-varying Reproduction Numbers of COVID-19 in Georgia, USA, March 2, 2020 to November 20, 2020

Table 1. Control measures announced and implemented by state and local government agencies in the state of Georgia, in Metro Atlanta counties and Dougherty County Location Date (mo/d/y) Implemented measures 3/23/20 Executive order declared to limit physical interactions, including shelter-in- place if diagnosed with underlying conditions, bars closed, and no gathering with more than 10 individuals. 4/2/20 State-wide shelter-in-place order declared. 4/15/20 State-wide testing required for all symptomatic individuals (referral is still needed). State of Georgia5 4/27/20 Businesses in the state allowed to start opening according to social distancing and prevention measures. 7/28/20 All positive or suspected COVID-19 cases should isolate until their infectious period is over, and those exposed to the virus should comply with a 14-d quarantine. 11/1/20 Positive cases should isolate until their infectious period is over. Suspected cases should comply with a 14-d quarantine. 3/20/20 State of emergency declared by county officials. Bartow County35 3/26/20 Follow-up on the state government order to limit public gatherings to 10 people: restaurants available only for takeout or delivery, and establishments that require physical contact will be closed. Butts County36 3/24/20 County declared under state of emergency. Limit public gatherings to 10 people, restaurants available for takeout or delivery only, and establishments that require physical contact will be closed. 3/22/20 Shelter-in-place order for the county. No public gatherings of more than 10 Carroll County people, and food will be sold as takeout or delivery.37 3/26/20 Citizens should stay home, and gatherings of any size are prohibited.38 Catoosa County39 3/23/20 Follow-up on state government order to limit public gatherings to 10 people: restaurants available for takeout or delivery only, and establishments that require physical contact will be closed. Chattooga County40 3/16/20 Recommendation made to cancel events and large gatherings of more than 50 people, and to maintain physical distance of 6 feet. Cherokee County41 3/25/20 Local state of emergency declared. Limit public gatherings to 10 people. Restaurants available for takeout or delivery only. Individuals of at-risk groups should shelter-in-place; those with a positive diagnosis will stay quarantined in their house. 3/13/20 Citizens encouraged to avoid public gatherings and public events. Clayton County42 3/24/20 State of emergency amended to include limiting gatherings to 10 individuals. Cobb County43 3/24/20 State of emergency declared according to the state-level ordinance to emphasize social distancing. Essential business will open from 6 am to 9 pm; no dine-in services will be available. Coweta County44 3/26/20 State of emergency declared according to the state-level ordinance to emphasize social distancing and voluntary shelter at home. No dine-in services will be available. DeKalb County45 3/23/20 Gatherings of 10 or more people prohibited; citizens will shelter-in-place; curfew established from 9 pm to 6 am; playgrounds, parks, and gyms will remain closed; and food services will be delivery or takeout only. Dougherty County46 3/21/20 Shelter-in-place order declared. Nonessential activities will be canceled. Restaurants will provide drive-through, pickup/curbside, and delivery services. Douglas County47 3/23/20 Shelter-in-place order declared for county residents. Public gatherings of more than 10 are prohibited, and restaurants will be available for takeout or delivery only. 4/1/20 Public gatherings of 10 or more are prohibited. Restaurants will only Fayette County48 provide curbside, takeout, or delivery services. 4/8/20 Shelter-in-place declared in compliance with state-level ordinance. (continued on following page)

The Permanente Journal·https://doi.org/10.7812/TPP/20.232 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 101 ORIGINAL RESEARCH ARTICLE Time-varying Reproduction Numbers of COVID-19 in Georgia, USA, March 2, 2020 to November 20, 2020

Table 1. Control measures announced and implemented by state and local government agencies in the state of Georgia, in Metro Atlanta counties and Dougherty County (continued) Location Date (mo/d/y) Implemented measures Forsyth County49 3/18/20 Public gatherings of 10 or more are prohibited. 3/18/20 Senior centers, art centers, and libraries will be closed to the public. 3/24/20 All libraries will be closed to the public. Fulton County50 3/24/20 Maintain less than 10 people in 1 location, restaurants will not have dine-in services, and residents must stay at home. 4/8/2020 Follow-up on statewide judicial emergency: services and hearings suspended. 3/13/20 Libraries and parks in the county will be closed. Hall County51 3/19/20 Commissioners place a stay-at-home mandate. 3/24/020 Closure of dine-in services mandated. Haralson County52 3/25/20 Shelter-in-place order declared and public gatherings prohibited. Travel for essential activities only. Heard County53 3/26/20 Shelter-in-place order made official. Henry County54 4/1/20 Shelter-in-place order declared for county residents. Public gatherings of more than 10 are prohibited. Restaurants will be available for takeout or delivery only. Jasper County55 3/26/20 Curfew mandated from 10 pm to 6 am. Public gatherings of 10 or more are prohibited. Restaurants will provide curbside, takeout, or delivery only. Lamar County56 4/3/20 Shelter-in-place order mandated by county government. 3/20/20 State of emergency declared by county officials. Curfew from 9 pm to 6 am. Meriwether County57 3/24/20 Public gatherings of 10 or more are prohibited. Restaurants will provide curbside, takeout, or delivery services only. Businesses for which physical interaction is needed will be closed. Curfew is imposed from 9 pm to 6 am. Newton County58 3/31/20 Shelter-in-place order mandated for county residents. Public gatherings of more than 10 are prohibited. Restaurants available for takeout or delivery services only. Paulding County59 3/26/20 Persons showing symptoms shall refrain from entering public buildings. Restaurants will refrain from providing dine-in services. Nonessential businesses will remain closed. No gatherings of more than 10 people are permitted. Pickens County60 3/24/20 Residents must shelter in their homes. All gatherings are suspended. All travel is limited to essential travel needs. Walton County61 3/26/20 Public gatherings of 10 or more are prohibited. Restaurants will provide curbside, takeout, or delivery services only. Note that most of these control measures began to be implemented the day after the announcement. For example, Georgia’s state-wide shelter-in-place was announced on April 2, 2020 to be implemented on April 3, 2020.

reproduction number, R0, is the average number of sec- In contrast, the time-varying reproduction number, Rt,is ondary cases that 1 primary case can generate in a com- a time-dependent estimate of the secondary cases that arise pletely susceptible population in the absence of behavioral from 1 case at time t, when depletion of the susceptible changes or public health interventions.6 e estimated population, behavioral changes, and measures to control 10,11 values of R0 for SARS-CoV-2 vary across geographic lo- transmission of disease have taken place. As with R0,if cations. An early study of the epidemic in Wuhan reported Rt > 1, it indicates there is sustainable transmission in the 7 an R0 value of 2.2, assuming a serial interval of 7.5. Amore population. When Rt < 1, disease transmission cannot be recent study of the epidemic in China, adjusted for the sustained, and it is used as an indication of the effectiveness fi 6,10 changing case de nition, estimated an R0 value of 1.8 to 2.0 of infection control measures. (assuming a serial interval of 7.5) or 1.4 to 1.5 (assuming a Various statistical methods have been proposed to esti- 8  serial interval of 4.7). Assuming a serial interval of 4.4, our mate Rt. eir strengths and weaknesses have been recently analysis of confirmed COVID-19 cases in Iran estimated assessed by researchers who compared the performance of ff 12,13 the mean R0 value as 3.5 or 4.4, depending on the statistical di erent methods using synthetic epidemic data and method chosen.9 observed COVID-19 incidence data.14 We chose to use

102 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.232 ORIGINAL RESEARCH ARTICLE Time-varying Reproduction Numbers of COVID-19 in Georgia, USA, March 2, 2020 to November 20, 2020

an oft-used method, known as the instantaneous re- the Georgia Department of Public Health.1 If any in- production number method, as implemented in the R consistencies were found, the numbers from the Georgia package EpiEstim version 2.2-3 (R Version 1.2.5033 Department of Public Health were used as the standard. Windows NT 10.0; R Core Team, Vienna, Austria).10,11 We searched the local government web pages to verify is Bayesian method provides an estimate of the average whether any control measures were established. Such fi Rt over a short time window speci ed by the user (in our information is presented in Table 1. study, a 7-day window that ends at time t). It treats the fluctuation in incidence data as signals of an increasing or Statistical Analyses   decreasing reproduction number. is method has been e Rt value was estimated using the R package EpiEstim.  used to estimate COVID-19 Rt values in jurisdictions e Rt estimate is also known as an instantaneous repro- 15 16 17 11 such as mainland China, Hong Kong, Iran, South duction number. For this analysis, we implemented the Rt Korea,18 Italy,19 Nigeria,20 and Switzerland.21 estimate measure as defined by Cori et al11 as the ratio Λ Our study aimed to estimate Rt for COVID-19 in Georgia, between It, the number of incident cases at time t,to t, the urban Metro Atlanta counties, and rural Dougherty County total infectiousness of all the infected individuals at time t and its surrounding counties, analyzing data from March 2, (see Supplemental Materialsa for details).  2020, through November 20, 2020, as the state incrementally e Rt estimate reported here is an average of the estimate implemented and then relaxed social distancing interventions over 7 days before time t. Using a Bayesian framework with (Table 1). a gamma-distributed prior for Rt,τ, Cori et al derived an analytical expression of the posterior distribution of Rt and METHODS thus estimated its median, the variance, and the 95% is study uses data from the COVID-19 pandemic, credible interval (CrI).11 In our study, the data were ana- March 2, 2020 to November 20, 2020, in the state of lyzed using EpiEstim version 2.2-310,11 (R Core Team). Georgia, all Metro Atlanta counties (Supplemental Tables 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, and 12a), and Dougherty Sensitivity Analysis County and its surrounding counties. Metro Atlanta is We performed a sensitivity analysis, assuming that asymp- defined by the US Office of Management and Budget as the tomatic infections account for 10% of all infections respectively. “Atlanta-Sandy Springs-Alpharetta, Georgia Metropolitan is is the lower bound estimate presented in the Centers Statistical Area.”22 e list of Metro Atlanta counties is for Disease Control and Prevention’s pandemic planning provided in Table 2. scenario web page,24 and it is an approximate of the lower 95% confidence interval bound estimated by Byambasuren et al.25 Data Acquisition We multiplied the daily case count by 0.11 and then repeated fi  We downloaded the cumulative data of con rmed cases the Rt estimation. e sensitivity analysis results are discussed on November 21, 2020. eanalyzeddatasetincludes in the Supplemental Materials.a the cumulative incidence reports from March 2, 2020, to November 20, 2020, for the entire state of Georgia and Ethics itscountiesfromtheNew York Times GitHub data e Georgia Southern University Institutional Review repository.23 New York Times GitHub data are published Board made a nonhuman subjects determination for this by date of report. To account for the median number of project (H20364) under the G8 exemption category. days from SARS-CoV-2 symptom onset to the day of testing among positive cases (approximately 3 days), RESULTS andtimefromexposuretothevirustosymptomonset Community transmission of SARS-CoV-2 remained (approximately 6 days), we corrected the day of report by ongoing in Georgia based on incidence data by assumed a total of 9 days to estimate the assumed date of infection date of infection from February 22, 2020, to November 11, for every jurisdiction included in this study (Supplemental 2020 (date of report: March 2, 2020–November 20, 2020). Tables 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, As of November 11, 2020, the median EpiEstim Rt estimate 18, 19, 20, 21, and 22a).12,24 e first case in Georgia was was 1.03 (95% CrI: 1.03, 1.03). e same results were reported on March 2, 2020, andtheassumeddateof observed for Metro Atlanta, with an Rt estimate equal to infection was estimated as February 22, 2020.23 Our 1.03 (95% CrI: 1.03, 1.03). e transmission may have been ff cuto point for all jurisdictions was the date of report of under control for Dougherty County, with the Rt estimate November 20, 2020, 6 days before anksgiving Day and being 1.01 (95% CrI: 1.00, 1.02) (Table 2). 18 days after the presidential elections in the US. We As social distancing measures unfolded and then verified the numbers with official statistical reports from relaxed in Georgia during our study period, the median

The Permanente Journal·https://doi.org/10.7812/TPP/20.232 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 103 ORIGINAL RESEARCH ARTICLE Time-varying Reproduction Numbers of COVID-19 in Georgia, USA, March 2, 2020 to November 20, 2020

Table 2. Estimates for the time-varying reproduction number, Rt, for the state of Georgia, Metro Atlanta counties, Dougherty County, and counties surrounding Dougherty County, using the instantaneous reproduction number method as implemented in the R package EpiEstim Assumed date of As of 6/5/20 As of 11/11/20 infection of the first reported Median Rt Mean Rt Median Rt Mean Rt Location case (mo/d/y) (2.5%, 97.5% quantiles) (standard deviation) (2.5%, 97.5% quantiles) (standard deviation) Georgia 2/22/20 1.14 (1.11, 1.17) 1.14 (0.02) 1.03 (1.03, 1.03) 1.03 (0.001) Metro Atlanta 2/22/20 1.02 (0.98, 1.06) 1.02 (0.02) 1.03 (1.03, 1.03) 1.03 (0.001) Dougherty 3/6/20 1.03 (0.72, 1.41) 1.04 (0.18) 1.01 (1.00, 1.02) 1.01 (0.01) Baker 3/15/20 1.00 (0.88, 1.13) 1.00 (0.06) 1.02 (0.95, 1.09) 1.02 (0.04) Bartow 3/2/20 0.73 (0.46, 1.08) 0.74 (0.16) 1.04 (1.03, 1.05) 1.04 (0.01) Butts 3/13/20 0.47 (0.11, 1.27) 0.53 (0.30) 1.03 (1.01, 1.06) 1.03 (0.01) Calhoun 3/17/20 1.05 (0.99, 1.11) 1.05 (0.03) 0.99 (0.95, 1.04) 0.99 (0.02) Carroll 3/11/20 0.89 (0.63, 1.22) 0.90 (0.15) 1.04 (1.03, 1.06) 1.04 (0.01) Cherokee 2/28/20 1.06 (0.83, 1.33) 1.07 (0.13) 1.05 (1.04, 1.06) 1.05 (0.004) Clayton 3/6/20 1.06 (0.79, 1.16) 1.07 (0.10) 1.03 (1.02, 1.03) 1.03 (0.003) Cobb 2/27/20 1.15 (1.03, 1.27) 1.15 (0.06) 1.03 (1.03, 1.04) 1.03 (0.002) Coweta 3/5/20 1.00 (0.75, 1.31) 1.01 (0.14) 1.04 (1.03, 1.05) 1.04 (0.01) Dawson 3/11/20 0.94 (0.36, 1.93) 0.99 (0.41) 1.03 (1.01, 1.06) 1.03 (0.01) DeKalb 2/29/20 0.97 (0.86, 1.08) 0.97 (0.06) 1.01 (1.00, 1.02) 1.01 (0.006) Douglas 3/11/20 1.10 (0.85, 1.40) 1.11 (0.14) 1.03 (1.02, 1.04) 1.03 (0.005) Fayette 2/29/20 1.15 (0.68, 1.79) 1.17 (0.28) 1.04 (1.02, 1.05) 1.04 (0.007) Forsyth 3/7/20 1.10 (0.82, 1.43) 1.10 (0.16) 1.04 (1.03, 1.05) 1.04 (0.005) Fulton 2/22/20 0.94 (0.83, 1.05) 0.94 (0.06) 1.03 (1.02, 1.03) 1.03 (0.002) Gwinnett 2/27/20 1.05 (0.97, 1.12) 1.05 (0.04) 1.03 (1.02, 1.03) 1.03 (0.002) Hall 3/7/20 0.96 (0.81, 1.13) 0.96 (0.08) 1.02 (1.01, 1.03) 1.02 (0.004) Haralson 3/17/20 0.90 (0.35, 1.85) 0.95 (0.39) 1.08 (1.06, 1.10) 1.08 (0.02) Heard 3/11/20 1.44 (0.65, 2.71) 1.50 (0.53) 1.04 (1.00, 1.09) 1.04 (0.02) Henry 3/4/20 1.16 (0.93, 1.43) 1.17 (0.13) 1.03 (1.03, 1.04) 1.03 (0.004) Jasper 3/15/20 0.89 (0.42, 1.62) 0.93 (0.31) 1.05 (1.01, 1.08) 1.05 (0.02) Lamar 3/11/20 0.68 (0.20, 1.63) 0.74 (0.37) 1.04 (1.01, 1.07) 1.04 (0.02) Lee 3/2/20 1.01 (0.97, 1.05) 1.01 (0.02) 1.02 (1.00, 1.05) 1.02 (0.01) Meriwether 3/15/20 1.07 (0.66, 1.61) 1.08 (0.24) 1.01 (0.99, 1.04) 1.01 (0.01) Mitchell 3/15/20 1.01 (0.98, 1.05) 1.01 (0.02) 1.01 (0.99, 1.04) 1.01 (0.01) Morgan 3/14/20 1.05 (0.15, 3.50) 1.25 (0.89) 1.03 (1.00, 1.06) 1.03 (0.01) Newton 3/6/20 0.91 (0.61, 1.31) 0.92 (0.18) 1.03 (1.02, 1.04) 1.03 (0.01) Paulding 3/7/20 1.15 (0.87, 1.49) 1.16 (0.16) 1.04 (1.03, 1.05) 1.04 (0.01) Pickens 3/11/20 1.48 (0.83, 2.40) 1.51 (0.40) 1.06 (1.04, 1.08) 1.06 (0.01) Pike 3/19/20 1.11 (0.52, 2.01) 1.15 (0.38) 1.04 (1.02, 1.07) 1.04 (0.01) Rockdale 3/10/20 1.18 (0.82, 1.63) 1.19 (0.21) 1.03 (1.01, 1.04) 1.03 (0.01) Spalding 3/11/20 1.04 (0.66, 1.56) 1.06 (0.23) 1.03 (1.01, 1.05) 1.03 (0.01) Terrell 3/11/20 1.00 (0.95, 1.05) 1.00 (0.03) 1.01 (0.98, 1.04) 1.01 (0.02) Walton 3/19/20 0.81 (0.54, 1.17) 0.82 (0.16) 1.02 (1.00, 1.05) 1.02 (0.01) Worth 3/11/20 1.06 (1.01, 1.11) 1.06 (0.02) 1.10 (1.03, 1.17) 1.10 (0.03) The analysis used a serial interval following a gamma distribution with a mean of 4.60 days and a standard deviation of 5.55 days, with α = 0.05. Data were analyzed with 2 cutoff points: the dates of report of June 14, 2020, and November 20, 2020 (ie, the assumed date of infection of June 5, 2020, and November 11, 2020).

 fl EpiEstim Rt estimate in Georgia dropped from 1.14 assumed date of infection. e median Rt estimate uc- (95% CrI: 1.11, 1.17) on June 14, 2020 to 1.03 tuated around 1 from mid-March to November 11, 2020 (95% CrI: 1.03, 1.03) until November 11, 2020, as the (Figure 1).

104 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.232 ORIGINAL RESEARCH ARTICLE Time-varying Reproduction Numbers of COVID-19 in Georgia, USA, March 2, 2020 to November 20, 2020

Figure 1. Comparison between incidence by assumed date of infection and time-varying reproduction numbers (Rt) for Georgia, Metro Atlanta, and Dougherty County, USA, February 22, 2020 to November 11, 2020 (assumed date of infection), estimated using the instantaneous reproduction number method implemented in the EpiEstim package.

Regarding Metro Atlanta (Figure 1), the EpiEstim For Dougherty County (Figure 1), we observed a speedy fl Rt estimate uctuated above 1.5 before the end of decline in EpiEstim Rt estimates from around 2 in mid- March and gradually decreased to around 1 by May March to around 1 in mid-April; these values were  through November 11. e Rt estimates for each of the maintained around 1 up to November 11, 2020, when the fl Metro Atlanta counties uctuated around 1 during our mean Rt estimate was observed at 1.01 (95% CrI: 1.00, study period (Table 2, Supplemental Figures 2, 3, 4, 5, 1.02). is finding was driven primarily by the early epi- and 6a). demic observed in Dougherty County, where large clusters

The Permanente Journal·https://doi.org/10.7812/TPP/20.232 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 105 ORIGINAL RESEARCH ARTICLE Time-varying Reproduction Numbers of COVID-19 in Georgia, USA, March 2, 2020 to November 20, 2020

of cases were infected via 2 funerals that happened to be slight increase near August and then continuing to fluctuate 26 superspreading events. On March 13, 2020, the mean Rt near 1 up to November 11, 2020. estimate for Dougherty County was 2.63 (CrI: 2.27, 3.02). e relaxation of social distancing measures should be Counties surrounding Dougherty (Baker, Calhoun, Lee, implemented with an abundance of caution because of the ’ Mitchell, Terrell, and Worth counties) presented mean population s vulnerability. With the mean Rt estimates in EpiEstim Rt estimates around 2.00, as its outbreak devel- Georgia and almost all counties included in our study oped (Figure 2, Supplemental Figure 1a). It was observed remaining near 1 for more than 6 months, we believe that that the surrounding counties around Dougherty also ob- mandating nonpharmaceutical interventions, such as wearing 29 tained mean Rt estimates reaching 2 in all of them, except facemasks when outdoors, could help decrease the mean Rt for Baker County, which had the greatest Rt median es- estimates even more. Another important factor for consid- timate of 1.78 (CrI: 1.14, 2.56) for March 26, 2020 eration is access to health care and surge capacity in hospitals, (Figure 2). A week after the first case was reported in especially in rural Georgia. ehealth-caresystemin Dougherty County, Calhoun, Mitchell, Terrell, and Worth DoughertyCountywasimpactedheavilybythesurgeof counties presented estimates greater than 2 (Table 3). e COVID-19 patients driven by superspreading events.26 ff Rt median estimates decreased to near 1.00 up to November Our study evidences the negative e ects the superspreading 11, with the exception of Calhoun County, which presented event in Dougherty County caused in surrounding counties. a median point estimate of 0.99 (95% CrI: 0.95, 1.04) OneweekaftertheincreaseincasesinDoughertyCounty, (Table 2). neighboring areas showed an increase in their mean Rt estimates. ese results reflect local transmission of SARS- DISCUSSION CoV-2 in rural areas in Georgia as the epidemic spread from Community transmission of SARS-CoV-2 remained Dougherty County to neighboring counties. ongoing in Georgia as of November 11, 2020 (ie, ap- e resumption of economics activities, mobility of young proximately 3 weeks after the presidential election). On adults, and reopening of educational institutions led to the April 27, after implementing strict social distancing resurgence of COVID-19 cases in Georgia, as observed in measures, Georgia reopened some sectors of the econ- July and August.30 Further research into the spatiotemporal omy, with specific guidelines pertinent to social variation of SARS-CoV-2 transmission potential, and its distancing.4 As the economy slowly reopened and un- association with economic and medical vulnerability will shed protected social mixing increased, and events such as the light on the disease and economic burden of COVID-19 in presidential election occurred, an increase in the daily Georgia. fi number of new con rmed cases was observed starting in Our study estimated Rt values using the instantaneous June and continuing until November (study period) as reproduction number method implemented in the R package SARS-CoV-2 transmission continued unabated.1 Our EpiEstim.10,11 e EpiEstim estimate is sensitive to fluctuation study documents the decrease in Rt following social dis- in daily incident case counts as the instantaneous reproduction tancing interventions in Georgia and provides further ev- number method treats such changes as meaningful signals idence that social distancing measures remained important reflecting genuine increases or decreases in transmission to keep COVID-19 under control. Our findings are sup- potential. e instantaneous reproduction number method ported by the analysis of Lau et al (2020), in which they also in the EpiEstim package can be used if the purpose is to ff registered a decreased in the e ective reproductive number identify time-dependent changes in the Rt estimate that for Dougherty County after the shelter-in-place order was reflect the implementation or relaxation of social distancing mandated, with estimates decreasing from 5.19 (95% CrI: measures over time. However, cautious interpretation is 5.01, 5.31) to less than 1, and then fluctuating around needed, especially at the beginning of the outbreak, as the 27 1 weeks later. case count was small and the Rt estimate was unstable. Furthermore, many residents in both rural and urban Regarding the time window chosen for EpiEstim,we Georgia are medically vulnerable. A recent analysis by e used a window of 7 days in our analysis. We did not use a Surgo Foundation28 estimated the COVID-19 community window of < 7 days, because a weekend effect was observed vulnerability index for Dougherty County, by combining in the data (ie, the daily number of cases reported during the epidemiologic risk factors for infection and sociodemo- weekend was consistently less than those reported during graphic factors, at very high levels (COVID-19 community the weekdays before or after the weekend). vulnerability index = 0.87) when compared with counties in Metro Atlanta (Fulton county’s COVID-19 community Limitations  vulnerability index = 0.42). e EpiEstim Rt estimates for Our study is limited by several factors. First, we used the Dougherty County fluctuated near 1 since April, with a New York Times data set, in which data were recorded by

106 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.232 ORIGINAL RESEARCH ARTICLE Time-varying Reproduction Numbers of COVID-19 in Georgia, USA, March 2, 2020 to November 20, 2020

Figure 2. Comparison between time-varying reproduction numbers (Rt) for counties surrounding Dougherty County, USA, February 22, 2020 to November 11, 2020 (assumed date of infection), estimated using the instantaneous reproduction number method implemented in the EpiEstim package.

ff reported date and not by day of symptom onset. However, such distinction in our data does not a ect our Rt estimate we implemented a date correction of 9 days to account for substantially since April. the period of date of infection and date of testing.24 ird, the data used here are an aggregated number of Second, our data do not differentiate between imported reported cases that do not distinguish different types of and community transmission cases. Although this dis- local transmission. Transmission in congregate facilities, tinction was important during the early stage of the epi- such as long-term care facilities,31 correctional facilities,32 demic, community transmission has been responsible for and factories,33 may show dynamics that are different from the majority of cases since April, and thus this absence of community transmission in noncongregate settings.

The Permanente Journal·https://doi.org/10.7812/TPP/20.232 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 107 ORIGINAL RESEARCH ARTICLE Time-varying Reproduction Numbers of COVID-19 in Georgia, USA, March 2, 2020 to November 20, 2020

Table 3. Estimates for the time-varying reproduction number, Rt, for the first weeks of the pandemic for Dougherty County and its surrounding counties, using the instantaneous reproduction number method as implemented in the R package EpiEstim

Start date by assumed date End date by assumed date Median Rt County of infection (mo/d/y) of infection (mo/d/y) (2.5%, 97.5% quantiles) 3/7/20 3/13/20 2.63 (2.27, 3.02) 3/8/20 3/14/20 2.45 (2.15, 2.76) 3/9/20 3/15/20 2.44 (2.19, 2.71) 3/10/20 3/16/20 2.35 (2.14, 2.57) 3/11/20 3/17/20 2.26 (2.08, 2.44) 3/12/20 3/18/20 2.14 (1.99, 2.29) Dougherty 3/13/20 3/19/20 2.03 (1.90, 2.16) 3/14/20 3/20/20 1.94 (1.82, 2.05) 3/15/20 3/21/20 1.85 (1.75, 1.95) 3/16/20 3/22/20 1.94 (1.85, 2.03) 3/17/20 3/23/20 1.89 (1.81, 1.98) 3/18/20 3/24/20 1.80 (1.73, 1.87) 3/16/20 3/22/20 1.64 (0.88, 2.65) 3/17/20 3/23/20 1.58 (0.86, 2.50) 3/18/20 3/24/20 1.73 (1.01, 2.64) 3/19/20 3/25/20 1.75 (1.07, 2.59) 3/20/20 3/26/20 1.78 (1.14, 2.56) Baker 3/21/20 3/27/20 1.75 (1.16, 2.46) 3/22/20 3/28/20 1.70 (1.16, 2.33) 3/23/20 3/29/20 1.73 (1.23, 2.32) 3/24/20 3/30/20 1.66 (1.20, 2.20) 3/25/20 3/31/20 1.67 (1.24, 2.17) 3/18/20 3/24/20 2.39 (1.74, 3.15) 3/19/20 3/25/20 2.12 (1.59, 2.73) 3/20/20 3/26/20 2.04 (1.57, 2.56) 3/21/20 3/27/20 1.97 (1.56, 2.43) Calhoun 3/22/20 3/28/20 1.87 (1.51, 2.26) 3/23/20 3/29/20 1.9 (1.57, 2.26) 3/24/20 3/30/20 1.88 (1.59, 2.21) 3/25/20 3/31/20 1.83 (1.57, 2.12) 3/3/20 3/9/20 1.96 (1.01, 3.21) 3/4/20 3/10/20 2.04 (1.14, 3.19) 3/5/20 3/11/20 2.97 (1.96, 4.19) 3/6/20 3/12/20 2.85 (2.04, 3.80) 3/7/20 3/13/20 2.54 (1.91, 3.27) 3/8/20 3/14/20 2.37 (1.85, 2.95) Lee 3/9/20 3/15/20 2.28 (1.84, 2.77) 3/10/20 3/16/20 2.13 (1.76, 2.53) 3/11/20 3/17/20 2.02 (1.70, 2.36) 3/12/20 3/18/20 1.87 (1.60, 2.17) 3/13/20 3/19/20 1.77 (1.53, 2.03) 3/14/20 3/20/20 1.70 (1.48, 1.94) (continued on following page)

108 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.232 ORIGINAL RESEARCH ARTICLE Time-varying Reproduction Numbers of COVID-19 in Georgia, USA, March 2, 2020 to November 20, 2020

Table 3. Estimates for the time-varying reproduction number, Rt, for the first weeks of the pandemic for Dougherty County and its surrounding counties, using the instantaneous reproduction number method as implemented in the R package EpiEstim (continued)

Start date by assumed date End date by assumed date Median Rt County of infection (mo/d/y) of infection (mo/d/y) (2.5%, 97.5% quantiles) 3/16/20 3/22/20 2.38 (1.96, 2.85) 3/17/20 3/23/20 2.31 (1.94, 2.70) 3/18/20 3/24/20 2.07 (1.77, 2.39) 3/19/20 3/25/20 1.92 (1.67, 2.19) 3/20/20 3/26/20 1.85 (1.63, 2.09) Mitchell 3/21/20 3/27/20 1.78 (1.58, 2.00) 3/22/20 3/28/20 1.72 (1.54, 1.91) 3/23/20 3/29/20 1.76 (1.59, 1.93) 3/24/20 3/30/20 1.75 (1.60, 1.92) 3/25/20 3/31/20 1.72 (1.58, 1.86) 3/12/20 3/18/20 2.12 (1.37, 3.03) 3/13/20 3/19/20 2.20 (1.52, 3.02) 3/14/20 3/20/20 2.10 (1.51, 2.79) 3/15/20 3/21/20 2.36 (1.80, 3.00) 3/16/20 3/22/20 2.82 (2.28, 3.41) 3/17/20 3/23/20 2.68 (2.25, 3.15) Terrell 3/18/20 3/24/20 2.38 (2.05, 2.74) 3/19/20 3/25/20 2.16 (1.89, 2.45) 3/20/20 3/26/20 2.01 (1.78, 2.26) 3/21/20 3/27/20 1.88 (1.68, 2.09) 3/22/20 3/28/20 1.77 (1.59, 1.95) 3/23/20 3/29/20 1.70 (1.54, 1.87) 3/12/20 3/19/20 2.17 (1.45, 3.02) 3/13/20 3/20/20 2.03 (1.41, 2.76) 3/14/20 3/21/20 2.00 (1.44, 2.63) 3/15/20 3/22/20 1.94 (1.45, 2.50) 3/16/20 3/23/20 2.30 (1.81, 2.84) 3/17/20 3/24/20 2.24 (1.82, 2.70) Worth 3/18/20 3/25/20 2.05 (1.70, 2.43) 3/19/20 3/26/20 1.95 (1.65, 2.28) 3/20/20 3/27/20 1.87 (1.60, 2.16) 3/21/20 3/28/20 1.77 (1.54, 2.02) 3/22/20 3/29/20 1.69 (1.48, 1.91) 3/23/20 3/30/20 1.65 (1.46, 1.85) The analysis used a serial interval following a gamma distribution with a mean of 4.60 days and a standard deviation of 5.55 days, with α = 0.05.

Fourth, cases may be underreported as a result of limited Fifth, our analysis is right-censored by November 20, testing capacity, or they may be mild or asymptomatic cases. 2020 (date of report), and sixth, the observed fluctuations in Testing capacity was expanded in March and April, and it has the Rt estimates, could be a result of low case numbers  been stable since then. us, variation in case numbers and Rt reported that could result in unstable estimates. Future estimates should reflect changing transmission dynamics and studies can extend the analysis further as the pandemic not changes in testing capacity. Meanwhile, the degree to progresses. which asymptomatic transmission has changed over time Seventh, in addition to the method used here, there are 12,13 cannot be estimated using our data. Age distribution of cases other statistical methods that estimate Rt (eg, the case has changed over time and may reflect a changing fraction of reproduction number method as proposed by Wallinga and asymptomatic cases among all infections. Teunis34). However, the case reproduction number method

The Permanente Journal·https://doi.org/10.7812/TPP/20.232 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 109 ORIGINAL RESEARCH ARTICLE Time-varying Reproduction Numbers of COVID-19 in Georgia, USA, March 2, 2020 to November 20, 2020

estimates the transmission potential of time t using the Disclaimer number of cases observed after time t and does not meet the The opinions expressed in this paper do not necessarily represent the fi need of this study because we attempted to estimate Rt up to of cial positions of the Centers for Disease Control and Prevention or the the nearest possible time. US government.

References CONCLUSION 1. Georgia Department of Public Health. Georgia Department of Public Health COVID-19  fl daily status report; 2020. Accessed July 24, 2020. https://dph.georgia.gov/covid-19-daily- e Rt estimate of SARS-CoV-2 has been uctuating status-report around 1 for Georgia, Metro Atlanta, and Dougherty 2. State of Georgia government. Gov. Kemp issues new executive orders, provides COVID- 19 update (March 23, 2020); 2020. Accessed June 30, 2020. https://gov.georgia.gov/ County and its neighboring counties since the Georgia press-releases/2020-03-23/gov-kemp-issues-new-executive-orders-provides-covid-19- economy reopened in late-April 2020. Social distancing update and other personal protective behavior (such as face 3. State of Georgia government. Governor Kemp Issues Shelter in Place Order (April 2, 2020); 2020. Accessed June 30, 2020. https://gov.georgia.gov/press-releases/2020-04- coverings) appear to keep the SARS-CoV-2 transmission 02/governor-kemp-issues-shelter-place-order potential at a reduced level. Government agencies should 4. State of Georgia government. Governor Kemp gives COVID-19 update (April 27, 2020); 2020. Accessed June 30, 2020. https://gov.georgia.gov/press-releases/2020-04-27/ weigh carefully the next steps of their COVID-19 re- governor-kemp-gives-covid-19-update sponse plans for their communities, considering ongoing 5. State of Georgia government. News: Press releases; 2020. Accessed June 30, 2020. https://gov.georgia.gov/press-releases transmission across Georgia and the potential surge after 6. Vynnycky E, White R. An introduction to infectious disease modelling. Oxford: Oxford the holiday season. v University Press; 2010. 7. Nishiura H, Linton NM, Akhmetzhanov AR. Serial interval of novel coronavirus (COVID- 19) infections. Int J Infect Dis 2020 Apr;93:284–6. DOI: https://doi.org/10.1016/j.ijid.2020. Supplemental Material 02.060, PMID:32145466. a fi Supplemental Material is available at: www.thepermanentejournal.org/ les/ 8. Tsang TK, Wu P, Lin Y, Lau EHY, Leung GM, Cowling BJ. Effect of changing case 2021/20.232supp.pdf definitions for COVID-19 on the epidemic curve and transmission parameters in mainland China: A modelling study. Lancet Public Health 2020 May;5:e289–96. DOI: https://doi.org/ 10.1016/S2468-2667(20)30089-X, PMID:32330458 Disclosure Statement fl 9. Muniz-Rodriguez K, Fung IC, Ferdosi SR, et al. Severe Acute Respiratory Syndrome The authors have no con icts of interest to disclose. Coronavirus 2 transmission potential, Iran, 2020. Emerg Infect Dis 2020 Aug;26(8): 1915–7. DOI: https://doi.org/10.3201/eid2608.200536, PMID:32320641. Acknowledgments 10. Thompson RN, Stockwin JE, van Gaalen RD, et al. Improved inference of time- We acknowledge Bryan O. Sepulveda-Bahamundi, MS, for his contribution to varying reproduction numbers during infectious disease outbreaks. Epidemics 2019 Dec;29:100356. DOI: https://doi.org/10.1016/j.epidem.2019.100356,PMID: data collection for this project. 31624039. 11. Cori A, Ferguson NM, Fraser C, Cauchemez S. A new framework and software to Authors’ Contributions estimate time-varying reproduction numbers during epidemics. Am J Epidemiol 2013 Nov; – Kamalich Muniz-Rodriguez, DrPH, and Gerardo Chowell, PhD, contributed 178(9):1505 12.. DOI: https://doi.org/10.1093/aje/kwt133, PMID:24043437. fi 12. Gostic KM, McGough L, Baskerville EB, et al. Practical considerations for measuring the equally as rst coauthors. Kamalich Muniz-Rodriguez, DrPH, participated in effective reproductive number, Rt. PLoS Comput Biol 2020 Dec;16:e1008409. DOI: conceptualization, data curation, formal analysis, validation, visualization, writing of https://doi.org/10.1371/journal.pcbi.1008409, PMID:32607522. the original draft, and manuscript review and editing. Gerardo Chowell, PhD, 13. O’Driscoll M, Harry C, Donnelly CA, Cori A, Dorigatti I. A comparative analysis of participated in conceptualization, data curation, funding acquisition, methodology, statistical methods to estimate the reproduction number in emerging epidemics with resources, supervision, validation, visualization, writing of the original draft, and implications for the current COVID-19 pandemic. Clin Infect Dis. DOI: https://doi.org/10. manuscript review and editing. Jessica S Schwind, PhD, participated in data 1093/cid/ciaa1599, PMID:33079987 curation and manuscript review and editing. Randall Ford, DDS, Sylvia K Ofori, 14. Roosa K, Lee Y, Luo R, et al. Short-term forecasts of the COVID-19 epidemic in Guangdong and Zhejiang, China: February 13–23, 2020. J Clin Med 2020 Feb;9(2):596. MPH, Chigozie A Ogwara, BS, Margaret R Davies, BS, Terrence Jacobs, BS, and DOI: https://doi.org/10.3390/jcm9020596, PMID:32098289. Chi-Hin Cheung, MS, participated in data curation. Logan T Cowan, PhD, 15. Leung K, Wu JT, Liu D, Leung GM. First-wave COVID-19 transmissibility and severity in and Andrew R Hansen, DrPH, participated in data curation and manuscript review China outside Hubei after control measures, and second-wave scenario planning: A and editing. Isaac Chun-Hai Fung, PhD, participated in conceptualization, formal modelling impact assessment. Lancet 2020 Apr;395(10233):1382–93. DOI: https://doi. analysis, funding acquisition, methodology, project administration, resources, org/10.1016/S0140-6736(20)30746-7, PMID:32277878. supervision, validation, visualization, writing of the original draft, and manuscript 16. Cowling BJ, Ali ST, Ng TWY, et al. Impact assessment of non-pharmaceutical fl review and editing. interventions against Coronavirus Disease 2019 and in uenza in Hong Kong: An observational study. Lancet Public Health 2020 May;5(5):e279–88. DOI: https://doi.org/ Note from Isaac Chun-Hai Fung, PhD: I have a team of colleagues and students 10.1016/S2468-2667(20)30090-6, PMID:32311320. who collected COVID-19 data manually, before the New York Times made their 17. Najafi F, Izadi N, Hashemi-Nazari SS, Khosravi-Shadmani F, Nikbakht R, Shakiba E. COVID-19 data set publicly available. My team of data curators are all included as Serial interval and time-varying reproduction number estimation for COVID-19 in western coauthors, given their time and efforts contributed to the project that eventually Iran. New Microbes New Infect 2020 Jul;36:100715. DOI: https://doi.org/10.1016/j.nmni. takes shape in its current form. 2020.100715, PMID:32566233. 18. Zhuang Z, Zhao S, Lin Q, et al. Preliminary estimates of the reproduction number of the coronavirus disease (COVID-19) outbreak in Republic of Korea and Italy by 5 March Funding 2020. Int J Infect Dis 2020 Jun;95:308–10.. DOI: https://doi.org/10.1016/j.ijid.2020.04.044, Gerardo Chowell, PhD, received support from a National Science Foundation PMID:32334115. grant (1414374) as part of the joint National Science Foundation–National Institutes of 19. Moirano G, Schmid M, Barone-Adesi F. Short-term effects of mitigation measures Health–US Department of Agriculture Ecology and Evolution of Infectious Diseases for the containment of the COVID-19 outbreak: An experience from northern Italy. Disaster Med Public Health Prep 2020 Aug;14:e3–4. DOI: https://doi.org/10.1017/ program. Isaac Chun-Hai Fung, PhD, received salary support from the Centers for dmp.2020.119, PMID:32327001 – Disease Control and Prevention (19IPA1908208) for the academic year 2019 2020. 20. Adegboye OA, Adekunle AI, Gayawan E. Early transmission dynamics of novel This article is not part of Isaac Chun-Hai Fung’s Centers for Disease Control and coronavirus (COVID-19) in Nigeria. Int J Environ Res Public Health 2020 Apr;17(9):3054. Prevention-sponsored projects. DOI: https://doi.org/10.3390/ijerph17093054, PMID:32353991

110 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.232 ORIGINAL RESEARCH ARTICLE Time-varying Reproduction Numbers of COVID-19 in Georgia, USA, March 2, 2020 to November 20, 2020

21. Scire J, Nadeau S, Vaughan T, et al. Reproductive number of the COVID-19 epidemic in 40. Chattooga County Health Department. Public health director assesses Northwest Georgia Switzerland with a focus on the cantons of Basel-Stadt and Basel-Landschaft. Swiss Med Wkly coronavirus situation; 2020. Accessed April 17, 2020. www.facebook.com/ 2020 May;150:w20271. DOI: https://doi.org/10.4414/smw.2020.20271, PMID:32365217. ChattoogaDPH/posts/828821817586018:0?__tn__=K-R 22. Office of Management and Budget. To the heads of Executive Departments and 41. Cherokee County Board of Commissioners. Declaration of local emergency and local Establishments; 2015. Accessed April 28, 2020. www.bls.gov/bls/omb-bulletin-15-01- emergency order no. 2020-02; 2020. Accessed April 17, 2020. www.cherokeega.com/_ revised-delineations-of-metropolitan-statistical-areas.pdf. focus/corona-virus/Press-Releases/03-25-2020-CCBOC.pdf 23. The New York Times. Coronavirus (Covid-19) data in the United States; 2020. Accessed 42. Clayton County Government. Clayton County government Facebook profile; 2020. June 15, 2020. https://github.com/nytimes/covid-19-data Accessed April 17, 2020. www.facebook.com/ClaytonCountyGeorgia/ 24. Centers for Disease Control and Prevention. COVID-19 pandemic planning scenarios and 43. Cobb County Board of Commissioners. Chairman issues declaration of emergency for Coronavirus Disease 2019 (COVID-19) 2020; 2020. Accessed November 29, 2020. www. COVID-19 crisis: March 24; 2020. Accessed July 1, 2020. www.youtube.com/watch? cdc.gov/coronavirus/2019-ncov/hcp/planning-scenarios.html#table-2 v=5YwWHcmra4M 25. Byambasuren O, Cardona M, Bell K, Clark J, McLaws M-L, Glasziou P. Estimating the 44. Board of Commissioners of Coweta County. An ordinance for the taking of additional extent of asymptomatic COVID-19 and its potential for community transmission: emergency measures and renewed declaration of a local state of emergency related to Systematic review and meta-analysis. Official Journal of the Association of Medical covid-19; and for other purposes; 2020. Accessed April 17, 2020. www.coweta.ga.us/ Microbiology and Infectious Disease Canada 2020;5(4):223–234. DOI: https://doi.org/10. home/showdocument?id=20166 3138/jammi-2020-0030. 45. DeKalb County. DeKalb CEO Thurmond issues state of emergency executive order; 26. Barry E. Days after a funeral in a Georgia town, coronavirus ‘hit like a bomb’ (New York 2020. Accessed April 17, 2020. www.dekalbcountyga.gov/news/dekalb-ceo-thurmond- Times, March 30, 2020); 2020. Accessed June 27, 2020. www.nytimes.com/2020/03/30/ issues-state-emergency-executive-order us/coronavirus-funeral-albany-georgia.html 46. Dougherty County Board of Commissioners. Media advisory shelter in place; 2020. 27. Lau MSY, Grenfell B, Thomas M, Bryan M, Nelson K, Lopman B. Characterizing Accessed April 17, 2020. www.dougherty.ga.us/filestorage/1800/379008/379614/ superspreading events and age-specific infectiousness of SARS-CoV-2 transmission in DOCOadvisory_ShelterInPlace_032020.pdf Georgia, USA. Proc Natl Acad Sci USA 2020 Sep;117(36):22430–5. DOI: https://doi.org/ 47. Douglas County Board of Health. Declaration of local state of emergency; 2020. Accessed 10.1073/pnas.2011802117, PMID:32820074 April 17, 2020. www.celebratedouglascounty.com/news/ 28. Surgo Foundation. The COVID-19 community vulnerability index (CCVI); 2020. 48. Fayette County. Fayette County will extend closure to the public; 2020. Accessed April 17, Accessed April 27, 2020. https://precisionforcovid.org/ccvi?fbclid=IwAR0VsJVvj- 2020. https://fayettecountyga.gov/ RBnwEC62iJb32R1ZTVnWE0yT1H7SfUXeyF3JEBEKn69sY7JRs 49. Forsyth County Board of Commissioners. Forsyth county chair exercise of 29. Centers for Disease Control and Prevention. Considerations for wearing masks: COVID- emergency powers pursuant to a declaration of local emergency; 2020. Accessed 19 (Coronavirus Disease); 2020. Accessed December 10, 2020. www.cdc.gov/ April 17, 2020. www.forsythco.com/News/PostId/2644/senior-services-may- coronavirus/2019-ncov/prevent-getting-sick/cloth-face-cover-guidance.html newsletter-now-available 30. Boehmer TK, DeVies J, Caruso E, et al. Changing age distribution of the COVID-19 50. Fulton County. Coronavirus updates: Updates on Fulton County facility closures – pandemic: United States, May August 2020. MMWR Morb Mortal Wkly Rep 2020 and service changes; 2020. Accessed April 17, 2020. www.fultoncountyga. – Oct;69(39):1404 9. DOI: https://doi.org/10.15585/mmwr.mm6939e1,PMID: gov/news/2020/04/06/updates-on-fulton-county-facility-closures-and-service- 33001872 changes – 31. McMichael TM, Currie DW, Clark S, et al, Public Health Seattle and King County, 51. Hall County Georgia. General county news; 2020. Accessed April 17, 2020. www. EvergreenHealth, and CDC COVID-19 Investigation Team. Epidemiology of hallcounty.org/CivicAlerts.aspx?AID=751 COVID-19 in a long-term care facility in King County, Washington. N Engl J Med 52. Haralson County Boards of Commissioners. Emergency order March 25, 2020 May;382(21):2005–11. DOI: https://doi.org/10.1056/nejmoa2005412,PMID: 2020; 2020. Accessed April 17, 2020. www.carrollcountyga.com/718/COVID-19- 32220208 Update 32. Wallace M, Hagan L, Curran KG, et al. COVID-19 in correctional and detention 53. Heard County Georgia. Heard County Board of Health shelter in place order. County facilities: United States, February–April 2020. MMWR Morb Mortal Wkly Rep 2020 News; 2020. Accessed April 17, 2020. www.heardcountyga.com/news/2020/2020_ May;69(19):587–90. DOI: https://doi.org/10.15585/mmwr.mm6919e1,PMID: ShelterOrder.html 32407300. 54. Henry County Board of Commissioners. Ordinance #20-02; 2020. Accessed April 17, 33. Dyal JW, Grant MP, Broadwater K, et al. COVID-19 among workers in meat and poultry 2020. www.co.henry.ga.us/Residents/COVID-19 processing facilities: 19 States, April 2020. MMWR Morb Mortal Wkly Rep 2020;69(18): 557–61. DOI: https://doi.org/10.15585/mmwr.mm6918e3, PMID:32379731 55. Jasper County Board of Commissioners. Countywide emergency protective order for all of Jasper County; 2002. Accessed April 17, 2020. https://jaspercountyga.org/emergency- 34. Wallinga J, Teunis P. Different epidemic curves for severe acute respiratory syndrome protective-order/ reveal similar impacts of control measures. Am J Epidemiol 2004 Sep;160(6):509–16. DOI: https://doi.org/10.1093/aje/kwh255, PMID:15353409. 56. Board of Commissioners Lamar County. Shelter in place; 2020. Accessed April 17, 2020. 35. Bartow County Commissioner. Emergency administrative order; 2020:5. Accessed April www.lamarcountyga.com 17, 2020. www.bartowga.org/CommissionerOffice/COVID-19-EMERGENCY_ 57. Meriwether County Georgia Board of Commissioners. Press Release 3-24-2020; 2020. DECLARATION_AMENDMENT_3-26-2020.pdf Accessed April 17, 2020. www.meriwethercountyga.us 36. Board of Commissioners of Butts County. Resolution CR202008A amending local state of 58. Newton County. Newton County measures for COVID-19. Civic Alerts; 2020. Accessed emergency declaration; 2020. Accessed April 17, 2020. https://buttscountyga.com/ April 17, 2020. http://ncboc.com/CivicAlerts.aspx?AID=187 cr202008aamendlsed/ 59. Paulding County Board of Commissioners. Ordinance 20-03: A declaration of state 37. Carroll County Facebook Page. Facebook post: Shelter in place 2020. Accessed April 17, emergency arising because of COVID-19; an ordinance taking immediate emergency 2020. www.facebook.com/carrollcountyga/photos/a.125786450829871/ measures; 2020. Accessed March 1, 2021. http://www.paulding.gov/DocumentCenter/ 3539184499490032/?type=3&theater View/9584/Ordinance-20-03-Decleration-of-Emergency-for-Paulding-3262020. 38. Carroll County Board of Health. Supplement local order of state of emergency. 2020;8. 60. Pickens County. An ordinance for the declaration of a local state of emergency Accessed April 17, 2020. www.carrollcountyga.com/DocumentCenter/View/3515/FINAL- related to COVID-19; 2020. Accessed April 17, 2020. https://pickenscountyga.gov Carroll-County-Board-of-Health---Shelter-in-Place-3-24-20 61. Walton County. An ordinance declaring a state of emergency arising because of 39. Catoosa County. Coronavirus COVID-19 information; 2020. Accessed April 17, 2020. COVID-19 and taking immediate emergency measures; 2020. Accessed April 17, 2020. www.catoosa.com/corona-covid-19 www.waltoncountyga.gov

The Permanente Journal·https://doi.org/10.7812/TPP/20.232 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 111 n ORIGINAL RESEARCH ARTICLE Adverse Events of Sodium–Glucose Cotransporter-2 Inhibitors in Chronic Kidney Disease: A Retrospective Chart Review

Hanul Choi, PharmD1; Leigh-Anh Nguyen, PharmD2; Jenny Wan, PharmD, BCPS2; Hooman Milani, PharmD, MBA1; Kristine McGill, PharmD, BCPS1; Jong Park, MD2 Perm J 2021;25:20.242 E-pub: 03/17/2021 https://doi.org/10.7812/TPP/20.242

ABSTRACT INTRODUCTION Background: The renal benefits of sodium–glucose cotransporter-2 Diabetes is one of the major risk factors of chronic inhibitors (SGLT2) are now well established, and these agents are kidney disease (CKD).1 For the past 2 decades, angio- recommended by the American Diabetes Association and Kidney tensin converting enzyme inhibitors and angiotensinogen Disease: Improving Global Outcomes guidelines for patients with II receptor blockers have been the standard treatment type 2 diabetes and chronic kidney disease. However, the safety options to minimize the renal complications of diabetes profile of SGLT2 inhibitors in chronic kidney disease is not as clear. without a main alternative.2,3 We describe the adverse event rates of SGLT2 inhibitors, primarily fl However in 2019, the US Food and Drug Administration empagli ozin, in Kaiser Permanente Southern California members fl – with diabetic kidney disease. approved a new indication for canagli ozin, a sodium glucose Methods: This study was a multicenter retrospective de- cotransporter-2 (SGLT2) inhibitor, to slow the progression 4 scriptive analysis evaluating Kaiser Permanente Southern Cal- of diabetic kidney disease. is indication was based on the ifornia members with type 2 diabetes and chronic kidney disease Canagliflozin and Renal Outcomes in Type 2 Diabetes and 1, 2, or 3 who first filled an SGLT2 inhibitor prescription in 2018, Nephropathy (CREDENCE) trial, which studied cana- with follow-up through 2019. Primary outcomes were event rates gliflozin use in patients with type 2 diabetes and proteinuric of diabetic ketoacidosis, bone fracture, amputation, urinary tract kidney disease. e trial was discontinued early after finding infection, genital mycotic infection, hyperkalemia, and acute significant renal and cardiovascular benefits in patients on kidney injury. Secondary outcomes were mean changes in esti- canagliflozin.5 Other studies such as the Empagliflozin Car- mated glomerular filtration rates, serum creatine levels, urine diovascular Outcome Event Trial in Type 2 Diabetes Mel- albumin-to-creatinine ratios, and hemoglobin A1c percentages fl during the follow-up period. litus Patients (EMPA-REG OUTCOME) and Canagli ozin Results: Of 213 patients, 39 experienced at least 1 adverse Cardiovascular Assessment Study (CANVAS) Program have fi fi event, for a total of 50 adverse events. Urinary tract infection had also demonstrated signi cant cardiovascular and renal bene ts 6,7 the highest incidence (62.1 events/1000 person-years), followed with SGLT2 inhibitor use. In addition to literature that shows by genital mycotic infection (58.0 events/1000 person-years). renoprotective effects, these effects are likely a class benefit based Favorable changes were observed during the follow-up period for on the SGLT2 inhibitor mechanism of action. is justifies our urine albumin-to-creatinine ratios and hemoglobin A1c per- inclusion of various SGLT2 agents in our study. centages, with mean decreases of 81.8 mg/g and 0.7%, respec- erenalbenefits of SGLT2 inhibitors are now well tively. SGLT2 inhibitors were discontinued in 47.4% of patients, established, and these agents are recommended by the with the top reasons including increase in serum creatinine (8%) American Diabetes Association and Kidney Disease: and urinary or genital side effects (5.6%). Improving Global Outcomes for patients with type 2 Conclusion: Although most patients did not experience ad- 3,8 ff fi verse events, urinary tract infections and genital mycotic infec- diabetes and CKD. However, the adverse e ect pro le tions were more common. Our detection of rates and types of of these agents in the context of CKD is not as clear. adverse effects replicated most results reported in clinical trials. Some drawbacks of SGLT2 inhibitors include multiple Discontinuations were largely attributed to reasons other than adverse effects, warnings for Fournier’s gangrene and adverse events. lower limb amputation, as well as the financial cost.6,9 e CREDENCE trial found that adverse event rates were similar in both canagliflozin treatment and placebo groups.5 With findings from CREDENCE and other trials as a starting point, this study aimed to assess the safety profile of SGLT2 inhibitors, primarily empagliflozin, in the context of CKD. Our study describes the adverse Author Affiliations 1Pharmacy Administration, Kaiser Permanente West Los Angeles Medical Center, Los Angeles, CA event rates in clinical practice at Kaiser Permanente 2Department of Nephrology, Kaiser Permanente West Los Angeles Medical Center, Los Angeles, CA Southern California (KP SCAL), an integrated health system, with the secondary purpose of determining whether Corresponding Author the adverse event rates in practice differ significantly from Hanul Choi, PharmD ([email protected]) those reported in clinical trials.5-7 Keywords: adverse events, CKD, diabetes, renal, SGLT2 inhibitors 112 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.242 ORIGINAL RESEARCH ARTICLE Adverse Events of Sodium–Glucose Cotransporter-2 Inhibitors in Chronic Kidney Disease: A Retrospective Chart Review

METHODS Study Design is is a multicenter retrospective analysis that evaluated the safety profiles of KP SCAL members with diagnoses of type 2 diabetes and CKD 1 [glomerular filtration rate (GFR), > 90 mL/min], CKD 2 (GFR, 60–89 mL/min), or CKD 3 (GFR, 30–59 mL/min) who first filled an SGLT2 inhibitor prescription between January 1, 2018, and De- cember 31, 2018. ese patients were followed for at least 12 months through December 31, 2019. Patients who were pregnant, younger than 18 years, or had a history of kidney transplant were excluded from the study (Figure 1). Figure 1. Flow diagram for study population formation. CKD, chronic kidney Data Collection and Statistical Analysis disease; KP SCAL, Kaiser Permanente Southern California; SGLT2, sodium– glucose cotransporter-2. e SGLT2 inhibitor prescription fill data for this study was retrieved from the KP SCAL CKD Registry. e data included the drug name, first SGLT2 inhibitor prescription the follow-up period. Reasons for discontinuation and their fill date, patient demographics, and CKD stage at first rates were also collected. prescription fill. Chart reviews were conducted by research investigators for each unique patient to collect data on RESULTS adverse events and laboratory values for the follow-up Patient Characteristics period. e first laboratory values were extracted from After excluding 2 patients with a history of kidney transplant, the latest blood tests before or on the date of the first the final study cohort included a total of 213 patients. Table 1 prescription fill. If the prescription was discontinued or shows the baseline characteristics of the study population. inactivated during the follow-up period, the last laboratory e mean patient age was 58, 67% of patients were male, value was from the latest blood test before or on the date of and 31% were white. e mean HbA1c percentage was the last fill. If the prescription was still active, we used the 8.9% and the mean GFR was 75 mL/min/1.73 m2. e most recent blood test within the follow-up period. median UACR was 257 mg/g. A total of 86.9% of pa- e main statistical method used for this study was de- tients were on angiotensin converting enzyme inhibitors scriptive. Adverse event rates were measured by the events per or angiotensinogen II receptor blockers, and 98.1% were 1000 person-years to account for the duration of treatment. on empagliflozin, which is on the Kaiser Permanente Event rate per 1000 person-year was determined by multi- formulary. Other SGLT2 inhibitor prescriptions filled plying the number of adverse events by 1000, then dividing included 2 patients on empagliflozin–linagliptin, 1 pa- by the sum of the time each patient was actively on the tient on canagliflozin, and 1 patient on dapagliflozin. treatment. In addition, χ2 tests were used to compare selected data with statistical significance set at P =0.05. Safety Outcomes Statistical methods used in landmark trials differed from Of the 213 patients in the study, 39 patients experienced those in our study.5-7 We refrained from comparing our an adverse event for a total of 50 adverse events (Figure 2). results directly with the literature, given the limitations of ere was no trend of adverse event rate based on CKD our study and differences in study design. stage (P = 0.79) (Figure 3). Figure 4 shows the primary outcome or individual adverse Study Outcomes event rates in this population stratified by CKD stage. e objective of this study was to describe the adverse ere was no observable correlation of any individual event rates in KP SCAL members with type 2 diabetes and adverse event according to CKD stage (P > 0.05). e CKD stage 1, 2, or 3 on SGLT2 inhibitors. e primary adverse event with the highest incidence was UTI (62.1 outcomesofthisstudyweretheeventratesofdiabetic events/1000 person-years), followed by genital mycotic ketoacidosis, bone fracture, amputation, urinary tract infection (58.0 events/1000 person-years). ere was no infection (UTI), genital mycotic infection, hyperkalemia, incidence of diabetic ketoacidosis. and acute kidney injury. e secondary outcomes of this During the study period, patients were on SGLT2 in- study were the mean changes in estimated GFRs, serum hibitors for a median of 14.8 months (range, 0–24.2 months). creatinine (SCr) levels, urine albumin-to-creatinine ratios Table 2 shows the median time to each individual adverse (UACR), and hemoglobin A1c (HbA1c) percentages during event. e adverse event with the longest median time to event

The Permanente Journal·https://doi.org/10.7812/TPP/20.242 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 113 ORIGINAL RESEARCH ARTICLE Adverse Events of Sodium–Glucose Cotransporter-2 Inhibitors in Chronic Kidney Disease: A Retrospective Chart Review

Table 1. Patient characteristics of Kaiser Permanente Southern California sodium–glucose cotransporter-2 inhibitor study population Variable Value, N = 213 Age, y 58.9 ± 11.6 Sex, n (%) Male 142 (66.7) Race, n (%) White 66 (31.0) Hispanic/Latino 64 (30.0) Asian 50 (23.5) Black or African descent 22 (10.3) – Other/unknown 11 (5.2) Figure 3. Overall adverse event (AE) rate during sodium glucose cotransporter-2 inhibitor use according to chronic kidney disease (CKD) stage. HbA1c, %, mean ± SD 8.9 ± 1.6 SCr, mg/dL, mean ± SD 1.1 ± 0.3 eGFR, mL/min/1.73 m2, mean ± SD 75.4 ± 20.9 UACR, mg/g (range)a 257.3 (0-5, 262.2) Concomitant ACE-I or ARB, n (%) 186 (86.9) SGLT2 inhibitor, n (%) Empagliflozin 209 (86.9) Empagliflozin–linagliptin 2 (0.9) Canagliflozin 1 (0.5) Dapagliflozin 1 (0.5) aValue is the median albumin-to-creatinine ratio, calculated with albumin measured in milligrams and creatinine measured in grams. ACE-I = angiotensin converting enzyme inhibitor; ARB = angiotensin II receptor blocker; fi eGFR = estimated glomerular filtration rate; HbA1c = hemoglobin A1c; SCr = serum Figure 4. Individual adverse event rates strati ed by chronic kidney disease (CKD) creatinine; SD = standard deviation; SGLT2 = sodium–glucose cotransporter-2; UACR = stage. AKI, acute kidney injury; UTI, urinary tract infection. urine albumin-to-creatinine ratio.

percentages during the follow-up period from first fill of the medication to the last fill. Favorable changes were observed in UACRs and HbA1c percentages, with mean decreases of 81.8 mg/g and 0.7%, respectively. However, direct improvements in estimated GFRs or SCr levels were not observed. SGLT2 inhibitors were discontinued based on patient or provider preference in nearly half the study population (47.4%), with the top reasons including increase in SCr (8%), urinary or genital side effects (5.6%), other side effects/ intolerances (6.6%), and unknown (6.1%) (Table 4).

Figure 2. Number of patients who experienced 0, 1, 2, 3, and 4 adverse events DISCUSSION (AEs) during the period of sodium–glucose cotransporter-2 (SGLT-2) inhibitor use. As KP SCAL is a large, integrated health system, chart reviews of the patient health data allowed a comprehensive assessment of patients’ experiences after initiation of wasbonefracture(19.1months),whereastheeventwith SGLT2 inhibitors. is study found that most patients with shortest median time to event was genital mycotic infection diabetic kidney disease on SGLT2 inhibitors did not ex- (3.9 months). perience adverse events, with 50 adverse events experienced by 39 patients, or 18.3% of the total study population. is Effectiveness Outcomes rate may have been affected by multiple contributing factors, Table 3 shows the secondary outcomes or mean changes some of which were not accounted for in this observational in estimated GFRs, SCr levels, UACRs, and HbA1c review of clinical practice.

114 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.242 ORIGINAL RESEARCH ARTICLE Adverse Events of Sodium–Glucose Cotransporter-2 Inhibitors in Chronic Kidney Disease: A Retrospective Chart Review

Table 2. Median time to adverse event from first fill of sodium– diabetes increases the risk of UTIs and genital infections as a glucose cotransporter-2 inhibitor result of elevated levels of urinary glucose. In addition, the Mean time to event, current evidence from randomized controlled trials shows that 10 Adverse event mo (range), N = 213 SGLT2 inhibitors increase the risk of genital infections. Bone fracture 19.1 (5.4–22.2) Despite the absence of a placebo group in this study, landmark Acute kidney injury 10.8 (3.4–25.0) trials show that UTIs and genital mycotic infections had high Amputation 10.7 (3.6–14.1) incidences overall in both placebo and treatment groups, al- Urinary tract infection 7.3 (0.3–22.2) though rates in treatment groups were greater. Further in- fi Hyperkalemia 5.5 (0–14.3) vestigation is necessary to determine the signi cance of the ff Genital mycotic infection 3.9 (0.2–11.5) di erence between the reports of adverse events in this study population compared to placebo. e median time to the occurrence of each adverse event Table 3. Mean changes in estimated glomerular filtration rate, was also analyzed. Adverse events experienced earlier include serum creatinine level, urine albumin-to-creatinine ratio, and genital mycotic infection (3.9 months) and hyperkalemia hemoglobin A1c percentage (5.5 months), whereas those experienced later include acute Change kidney injury (10.8 months) and bone fracture (19.1 months) (mean ± standard deviation), (Table 2). Most of the bone fractures occurred secondary to Parameter N = 213 fi – falls, and not all cases were in patients at high risk for fracture. Estimated glomerular ltration rate 5.3 ± 1.1 fi (mL/min/1.73 m2) One patient experienced hyperkalemia on the day of rst fi Serum creatinine (mg/dL) +0.1 ± 0.1 SGLT2 inhibitor prescription ll, although medical docu- Urine albumin-to-creatinine ratio (mg/g) –81.8 ± 23.9 mentation did not attribute the hyperkalemic event to the HbA1c (%) –0.7 ± 0.1 SGLT2 inhibitor. In addition, correlation was observed between SGLT2 inhibitor use and improvement in UACRs and HbA1c Table 4. Discontinuation rate and time to discontinuation of percentages (Table 2). Despite no apparent improvement in sodium–glucose cotransporter-2 inhibitor GFRs or SCr levels, it would be valuable to assess whether Median time to discontinuation, there was a slower rate of decline in renal function compared Reason for discontinuation n (%) mo (range) to a placebo group or data from literature, as this was the Increase in creatinine 17 (8.0) 8.3 (0–17.2) case in the CREDENCE trial.5 Other side effect/intolerance 14 (6.6) 8.5 (0.2–15.6) Nearly half (47.4%) of the study’s patients discontinued Unknown 13 (6.1) 8.5 (0.9–16.8) SGLT2 inhibitors, with 20.2% of these discontinuations Urinary or genital side effects 12 (5.6) 8.5 (0.2–15.6) attributed to an adverse event (Table 4). Several of these Drug coverage/cost 10 (4.7) 8.4 (1.3–18.5) adverse events included self-reported symptoms that were Ineffective therapy 9 (4.2) 8.1 (0.7–16.7) documented by providers and may not have correlated with Patient request 9 (4.2) 8.3 (0.5–16.6) laboratory results or physical exams. In addition, SGLT2 Other reason 9 (4.2) 8.4 (3.4–19.7) inhibitors were often discontinued independently by the Nonadherence 7 (3.3) 8.6 (6.1–21.0) patient because of other reasons, such as reported adverse Allergic reaction 1 (0.5) 2.3 (n/a) reactions, fear of adverse events, and the financial cost of Total 101 (47.4) higher copays compared to other diabetic agents. Although most reported side effects were expected of SGLT2 in- hibitors, others included numbness/tingling, severe muscle In addition, when the adverse event rates were stratified cramps, swelling, and headache. by CKD stage, they did not differ between stages 1, 2, and Table 5 shows our study’s results alongside the adverse event 3. is points to an SGLT2 inhibitor adverse event profile data in EMPA-REG OUTCOME CREDENCE, and in CKD patients that is potentially similar to the rates in CANVAS Program.5-7 Our study population’s renal function published studies that include patients without CKD. Al- and use of empagliflozin were most similar to that of EMPA- though further studies are needed to establish this similarity, the REG. erefore, our study’s adverse event rates were plotted adverse event profile of patients without CKD may possibly be alongside those of EMPA-REG’s treatment and placebo extrapolated with more confidence to those with CKD given groups (Figure 5). e P values reflect the significance the observations in our study. in difference between this study and the EMPA-REG e highest incidence of adverse events was observed treatment group. Comparing the 2 groups shows a sig- with UTIs, followed by genital mycotic infections. Type 2 nificantly higher rate of UTIs (P < 0.001) and a lower rate

The Permanente Journal·https://doi.org/10.7812/TPP/20.242 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 115 ORIGINAL RESEARCH ARTICLE Adverse Events of Sodium–Glucose Cotransporter-2 Inhibitors in Chronic Kidney Disease: A Retrospective Chart Review

Table 5. Comparison of Kaiser Permanente Southern California adverse event rates with landmark trials Patient Characteristics and Adverse Events KP SCAL EMPA-REG Outcome CREDENCE Trial CANVAS Program Drug Empagliflozin Empagliflozin Canagliflozin Canagliflozin CKD 2–3 Renal function of study population CKD 1–3 eGFR ≥ 30 mL/min eGFR ≥ 30 mL/min UACR > 300 mg/g 62.1 48.3 Urinary tract infectiona 19.7% 40.0 7.0% 6.9% 58.0 8.4–12.6 Genital mycotic infectiona 6.4% 68.8 (female) 6.6% 1.9%–2.9% 29.0 29.7 Hyperkalemiaa N/A 6.9 3.3% 6.9% 29.0 16.9 Acute kidney injurya 1.0% 3.0 3.3% 3.9% 16.6 12.3 Amputationa N/A 6.3 1.9% 3.2% 12.4 11.8 Bone fracturea 3.8% 15.4 1.4% 3.0% 0 2.2 Diabetic ketoacidosisa 0.1% 0.6 0% 0.5% a.Row 1 expresses adverse event rate as event rate per 1000 person-years; row 2 expresses adverse event rate as a percentage. CANVAS = Canagliflozin Cardiovascular Assessment Study; CKD = chronic kidney disease; CREDENCE = Canagliflozin on Renal and Cardiovascular Outcomes in Participants With Diabetic Nephropathy; eGFR = estimated glomerular filtration rate; EMPA-REG Outcome = Empagliflozin Cardiovascular Outcome Event Trial; KP SCAL = Kaiser Permanente Southern California; N/A = not applicable; UACR = urine albumin-to-creatinine ratio.

of acute kidney injury (P = 0.001) in EMPA-REG’s treatment group than our clinical practice group. e discontinuation rate in this study was also higher than the rates in landmark trials. However, limitations in this direct comparison include different study designs, baseline characteristics, study duration, and nuances in the way the adverse events were defined by the study protocols.

Limitations Figure 5. Comparison of adverse event rates with those in the Empagliflozin e findings of this study should be viewed in the context Cardiovascular Outcome Event Trial (EMPA-REG OUTCOME), with P values fl fi of several limitations. First, the study design did not include re ecting the signi cance of the difference between adverse event rates of the fl Kaiser Permanente Southern California (KP SCAL) and EMPA-REG OUTCOME a comparator group, and the results were mostly re ective of treatment groups. AKI, acute kidney injury; DKA, diabetic ketoacidosis; PBO, empagliflozin, which is the preferred SGLT2 agent on the placebo; UTI, urinary tract infection. Kaiser Permanente formulary because of cost relative to other drugs. In addition, although rough comparisons to concomitant antidiabetic agents; these factors were not placebo can be drawn from landmark trials such as the accounted for in this study. EMPA-REG Outcome trial, there are limitations in this type of direct comparison resulting from differences in study Future Implications design, baseline characteristics, study duration, and nuances With this SGLT2 inhibitor safety profile data drawn in the way the adverse events were defined by the study directly from clinical practice, health-care providers now protocols. ere were also inconsistencies among patients in have the ability to expand confidently on the prescription of the timing of laboratory results relative to the first or last fill, SGLT2 inhibitors, inform on the benefits of SGLT2 in- and up to 25% of our patients had no repeat labs after their hibitors in diabetic kidney disease, educate accurately on the first prescription fill. Because of limited access to data, risk of adverse events, provide counseling points, and certain time periods and duration of exposures could not recommend continued use of these agents for renal benefits. be extracted in select cases. Last, confounding variables For instance, patients and providers can expect and monitor include other comorbidities or patient factors, as well as for genital infections and UTIs as earlier and more common

116 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.242 ORIGINAL RESEARCH ARTICLE Adverse Events of Sodium–Glucose Cotransporter-2 Inhibitors in Chronic Kidney Disease: A Retrospective Chart Review

adverse events. In doing so, health-care systems can shift Jenny Wan, PharmD, BCPS, Hooman Milani, PharmD, and Kristine McGill, PharmD, BCPS, participated in the study design, analysis of data, and critical prescribing patterns to target patients with diabetic kidney fi fi review of the manuscript. All authors participated in nal approval of the disease who would likely bene t from SGLT2 inhibitor version to be published. therapy, ultimately improving patient outcomes. Funding CONCLUSION No funding was acquired for this study. In conclusion, most of the patients in this study did not experience adverse events, and the patients who did were References 1. McClellan WM, Flanders WD. Risk factors for progressive chronic kidney disease. J Am more likely to have experienced UTIs or genital mycotic Soc Nephrol 2003 Jul;14:S65–70. DOI: https://doi.org/10.1097/01.asn.0000070147. infections.Inaddition,comparedtolandmarkclinical 10399.9e 2. Abraham HM, White CM, White WB. The comparative efficacy and safety of the trials, our data in a real-world setting showed similar angiotensin receptor blockers in the management of hypertension and other rates and types of adverse events. Accordingly, the no- cardiovascular diseases. Drug Saf 2015 Jan;38(1):33–54. DOI: https://doi.org/10.1007/ s40264-014-0239-7, PMID:25416320. table discontinuation rate was attributed largely to 3. Kidney Disease: Improving Global Outcomes (KDIGO) Diabetes Work Group. KDIGO reasons outside of adverse events. e discontinuation 2020 Clinical Practice Guideline for Diabetes Management in Chronic Kidney Disease. Kidney Int. 2020;98(4S):S1–S115. DOI: https://doi.org/10.1016/j.kint.2020.06.019, PMID: rate in this study was greater than those reported in 11904577. landmark trials, but we must exercise caution in drawing 4. U.S. FDA approves INVOKANA® (canagliflozin) to treat Diabetic Kidney Disease (DKD) conclusions around this comparison, given the varying and reduce the risk of hospitalization for heart failure in patients with Type 2 Diabetes (T2D) and DKD. Johnson and Johnson; 2019. study designs, durations, and documentation methods. 5. Perkovic V, Jardine MJ, Neal B, et al. Canagliflozin and renal outcomes in type 2 diabetes Based on our data, SGLT2 inhibitors should be con- and nephropathy. N Engl J Med 2019 Jun;380(24):2295–306. DOI: https://doi.org/10. 1056/NEJMoa1811744, PMID:30990260. sidered for more widespread use in patients with type 2 6. Zinman B, Wanner C, Lachin JM, et al. Empagliflozin, cardiovascular outcomes, and diabetes and CKD. v mortality in type 2 diabetes. N Engl J Med 2015 Nov;373(22):2117–28. DOI: https://doi. org/10.1056/NEJMoa1504720, PMID:26378978. 7. Neal B, Perkovic V, Mahaffey KW, et al. Canagliflozin and cardiovascular and renal events Disclosure Statement in type 2 diabetes. N Engl J Med 2017 Aug;377(7):644–57. DOI: https://doi.org/10.1056/ The authors have no conflicts of interest to disclose. NEJMoa1611925, PMID:28605608. 8. American Diabetes Association. Introduction: Standards of medical care in diabetes— – Authors’ Contributions 2020. Diabetes Care 2020 Jan;43:S1 2. DOI: https://doi.org/10.2337/dc20-Sint 9. Janssen Pharmaceuticals, Inc. Invokana (canagliflozin) [package insert]. Titusville, NJ: Hanul Choi, PharmD, led the study design, acquisition and analysis of data, Janssen Pharmaceuticals, Inc.; 2013. fi drafting, critical review, and submission of the nal manuscript. Leigh Anh 10. Liu J, Li L, Li S, et al. Effects of SGLT2 inhibitors on UTIs and genital infections in type 2 Nguyen, PharmD, and Jong C Park, MD, participated in the study design, diabetes mellitus: A systematic review and meta-analysis. Sci Rep 2017 Jun;7(1):2824. acquisition and analysis of data, and critical review of the manuscript. DOI: https://doi.org/10.1038/s41598-017-02733-w, PMID:28588220.

The Permanente Journal·https://doi.org/10.7812/TPP/20.242 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 117 n ORIGINAL RESEARCH ARTICLE Visit Content Analysis: Doctor-Patient Communication in Patients with Type 2 Diabetes

Dana A Abdelgadir, BA1; Laurie M Rodriguez, BS2; Maruta A Blatchins, BS1; Pranita Mishra, MPP1; Anjali Gopalan, MD, MS1; Richard W Grant, MD, MPH1 Perm J 2021;25:20.208 E-pub: 03/17/2021 https://doi.org/10.7812/TPP/20.208

ABSTRACT morbidity and mortality seen in T2D.2 Because the man- Introduction: The primary care visit is an important oppor- agement of diabetes has become increasingly complex for tunity to discuss and modify diabetes management. Objective: To gain insight into doctor-patient communication both clinicians and patients, it requires well-informed during primary care visits among English and Spanish speaking strategies for treatment.3 patients with type 2 diabetes and suboptimal glycemic control e primary care visit represents a critical opportunity for (HbA1c > 7%). addressing suboptimal diabetes control because it allows for Methods: We conducted a quantitative content analysis of coordination of care4 and contextualization of diabetes with audiotaped primary care visits in 2 patient cohorts. In Study 1 (31 5 English-speaking patients), we examined factors associated with other medical and nonmedical concerns. Unfortunately, management changes, and in Study 2 (20 Spanish-speaking providers often have limited time with patients and com- patients and their Spanish-speaking providers), we examined the monly need to address multiple, competing health con- association of question asking with HbA1c control. This study was cerns. Recent research has shown that the number of conducted between November 2017 and January 2020 across clinical items addressed during visits has outpaced changes 8 primary care practices within Kaiser Permanente Northern in visit duration, increasing the strain on the already California. resource-limited primary care visit.6 Such time limits may fi Results: In Study 1, the only factor signi cantly associated with leave important topics, such as diabetes care plans, de- a diabetes management change was patient identification of ferred to the end of the visit or entirely unaddressed.7,8 diabetes as a priority prior to the visit (91.7% had a management A better understanding of patient-provider communica- change vs 52.6% of patients who did not identify diabetes as a ff priority; p = 0.02). In Study 2, patients with poorer glycemic control tion during the primary care visit o ers the chance to ff (HbA1c ≥ 10.0) asked significantly fewer questions (3.4 ± 1.8 vs identify potential opportunities to promote e ective diabetes 10.7 ± 6.9 questions per 15 minutes; p = 0.004). Overall, despite management. receiving primary care from language-concordant providers, We conducted 2 studies to quantitatively analyze in- Spanish-speaking Study 2 patients asked fewer questions than person visit interactions between primary care providers English-speaking Study 1 patients (4.5 ± 2.9 vs 7.5 ± 3.7 and their patients with suboptimal glycemic control. In the questions per 15 minutes, respectively; p = 0.004). first study, we examined visit characteristics associated with Conclusion: Our results highlight 2 potential strategies (pre- diabetes management changes among patients who were paring patients for their visits through identifying priorities and prompted to identify their visit priorities prior to their learning how to ask more questions during visits) for improving visit. In the second study, we built on prior work among diabetes primary care. Spanish-speaking patients9,10 to examine the relationship between question-asking and glycemic control. Together, INTRODUCTION these 2 studies provide novel insights into doctor-patient e majority of the 34 million people with type 2 di- communication during the primary care visit. abetes (T2D) in the US have suboptimal glycemic control fi 1 ff (de ned as an HbA1C of 7.0% or higher). E ective METHODS diabetes control is essential for reducing the micro- and Setting macrovascular complications that contribute to the higher Visit audiotaping took place between November 2017 and January 2020 with 14 primary care providers across Author Affiliations 8 primary care practices in Kaiser Permanente Northern 1 Division of Research, Kaiser Permanente Northern California, Oakland, CA California (KPNC). KPNC is a nonprofit integrated care 2San Juan Bautista School of Medicine, Caguas, Puerto Rico delivery system providing care for more than 4.4 million Corresponding Author adult members throughout Northern California,11 includ- Dana A Abdelgadir, BA ([email protected]) ing over 360,000 members with diabetes.12 e distribution of member demographic and socioeconomic characteristics Keywords: doctor-patient communication, primary care, type 2 diabetes is diverse and similar to that of the area population.11 Abbreviations: KPNC = Kaiser Permanente Northern California; T2D = type 2 diabetes. 118 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.208 ORIGINAL RESEARCH ARTICLE Visit Content Analysis: Doctor-Patient Communication in Patients with Type 2 Diabetes

Study Design management. In Study 2, we examined the association of We used quantitative analysis of visit communication question-asking with poor glycemic control (defined as patterns obtained via professionally transcribed visit audio HbA1c ≥ 10.0%). Visit communication patterns were also recordings to conduct a detailed examination of real-time compared between the 2 study populations. In both an- primary care visit communication. We applied this meth- alyses, differences in the outcomes by levels of the exposure odology to 2 separate patient-provider cohorts. variables were compared using χ2 or Fisher’sexacttest.For Study 1, independent predictors of diabetes management Eligibility and Recruitment change were additionally investigated using a multivariate Patient eligibility criteria for both studies included at least logistic regression model that adjusted for visit length, 1 year of membership at KPNC, T2D with a last measured HbA1c, and selection of T2D as a visit priority. Analyses suboptimal HbA1c (HbA1c > 7.0%), and an upcoming were conducted using SAS version 9.4. encounter with their primary care provider. Visits were restricted to language concordant (Spanish or English) RESULTS patient-provider dyads. Informed consent was obtained We Analyzed 51 Unique Patient-Provider Visits (Table 1). from patients and providers to audio-record eligible visits. Study 1: Factors Associated with Change in Management Study 1 was a visit audio recording substudy of a larger In Study 1 (n = 31), the average patient age was 62.9 ± clinical trial (Pre-Visit Prioritization or Complex Patients 9.7 years, 58.1% (18/31) were women, 57.7% were non- With Diabetes, ClinicalTrials.gov NCT02375932)13 in White (15/26, including 4 Black, 9 Asian, and 2 other; which eligible English-speaking patients were sent a secure 5 were missing race/ethnicity information), and the mean electronic message prior to their primary care visit asking HbA1c was 9.0% ± 1.2%. Visits averaged 23.4 ± 7.7 minutes them to select their top 1 or 2 visit priorities from a list of and included 5.6 discussion topics (range, 2-9), 137 ± 42.7 5 options (important changes in your life, new/important exchanges between patients and providers, and 42.4 ± 17.4 health issues, medication concerns, diabetes-related con- questions asked per 15 minutes (34.9 ± 15.4 by provider and cerns, and mood/motivation).14 Study 2 was conducted 7.5 ± 3.7 by patient). independently using the same analytic methods but fo- Two-thirds of patients (21/31, 67%) selected diabetes cusing instead on Spanish-speaking patients with Spanish- as either a first or second visit priority. Identification of speaking providers. Because they were not part of the larger diabetes as a visit priority by the patient prior to the visit clinical trial, these patients did not receive a previsit survey. was the only factor significantly associated with changes Audio recordings were professionally transcribed for all in diabetes management during the visit (Table 2). e visits. e Kaiser Permanente Institutional Review Board results remained unchanged when limited to either first approved this study. or second priority. Nearly all patients (11/12, 91.7%) who had changes in diabetes management during the Statistical Analysis visit had selected diabetes as a visit priority. In contrast, e visits transcripts were reviewed by members of the among patients with no changes in care arising from study team (DA and RG for English language visits, and the visit, only 52.6% (10/19) had selected diabetes as a LR [a native Spanish speaker] for the Spanish language priority (p = 0.02). Selection of diabetes as a visit pri- transcripts). For each transcript reviewed, the reviewers ority by the patient remained significantly associated abstracted the following visit characteristics: visit length, with diabetes management change after adjusting for number of patient-provider exchanges, number of questions visit length and HbA1c (adjusted odds ratio = 10.1; 95% asked by providers and by patients, and the timing of confidence interval = 1.06-95.5; p = 0.045). Among the diabetes-related discussions after standardization of visit patients who did not select diabetes as a visit priority, length. “Exchanges” were determined by the number of the topic of diabetes was more likely to be discussed times each individual spoke, and “timing of diabetes-related toward the latter half of the visit (31.6% of visits vs discussion” was defined by whether initial mention of di- 8.3% of visits for which patients prioritized diabetes), abetes occurred in the first or second half of the visit. although this difference did not reach statistical significance “Question-asking” was defined as a care-related inquiry that (p = 0.2). required a direct response either from patient or provider. In Study 2: Question-Asking by Latinos Study 1, the 2 reviewers also identified intravisit diabetes In Study 2 (n = 20), all patients were Latino. Mean age management changes, defined as either diabetes-related was 55.0 ± 9.3 years, 50% (10/20) were women, and the lifestyle counseling or medication intensification. We then mean HbA1c was 9.4% ± 1.2%. Asking fewer questions examined the associations between visit characteristics and was significantly associated with higher HbA1c. Patients patient-selected visit priorities with these changes in diabetes with poor glycemic control (HbA1c ≥ 10.0%) asked fewer

The Permanente Journal·https://doi.org/10.7812/TPP/20.208 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 119 ORIGINAL RESEARCH ARTICLE Visit Content Analysis: Doctor-Patient Communication in Patients with Type 2 Diabetes

Table 1. Summary of patient and visit characteristics, studies 1 and 2 Variable Study 1 (n = 31) Study 2 (n = 20) p value Patient characteristics Age, y (SD) 62.9 (9.7) 55.1 (9.3) 0.01 Women, n (%) 18 (58.1) 10 (50.0) 0.57 A1C, % (SD) 9.0 (1.2) 9.4 (1.2) 0.24 Race/ethnicitya, n (%) White 11 (42.3) - - Black 4 (15.4) - - Asian 9 (34.6) - - Other 2 (7.7) - - Visit characteristics Visit length, min (SD) 23.4 (7.7) 28.5 (9.4) 0.04 Word count per 15 min, n (SD) Total 2001.4 (344) 1501.1 (437) 0.64 Patient 841.8 (255) 524.1 (204) 0.63 Physician 1159.6 (309) 977.0 (349) 0.06 Questions per 15 min, n (SD) Physician 34.9 (15.4) 27.7 (10.6) 0.07 Patient 7.5 (3.7) 4.5 (2.9) 0.004 Exchanges per 15 min, n (SD) 137.0 (42.8) 94.3 (49.5) 0.002 a.Race/ethnicity information was missing for 5 participants. SD = standard deviation.

questions compared with patients with HbA1c < 10% study had elevated HbA1c measured prior to the analyzed (3.4 ± 1.8 vs 10.7 ± 6.9 questions per 15 minutes; p = 0.004). primary care visit, evidence-based guidelines would have Compared with the English-speaking patients in Study recommended that these patients would have a change in 1, visit lengths in Study 2 were longer (28.5 ± 9.4 minutes) their diabetes management (eg, lifestyle modification ad- with fewer exchanges between patients and providers vice, referral to health education or other services, medi- (94.3 ± 49.5 exchanges). After standardizing visit length cation initiation or dose adjustment) as a consequence of per 15 minutes, Study 2 patients also asked significantly seeing their primary care provider. In Study 1 we found that fewer questions than Study 1 patients (4.5 ± 2.9 vs 7.5 ± 3.7 factors such as visit length, number of topics discussed, per 15 minutes; p = 0.004) (Table 1). questions asked, and current HbA1c level were not asso- ciated with a management change. Rather, patients who DISCUSSION selected diabetes as a priority for their visit were more likely Quantitative content analyses of primary care visit dis- to have a management change. In contrast, patients with cussions provide a unique opportunity to gain insight into suboptimal glycemic control who did not prioritize diabetes doctor-patient interactions and their association with were more likely to discuss the topic during the latter half of diabetes-related health outcomes. In our analyses of patients the visit and less likely to have a regimen change. ese with suboptimally controlled T2D, we found that patients findings suggest that structured visit preparation may help who prioritized diabetes as a topic of discussion prior to the facilitate more productive doctor-patient interactions re- visit were more likely to have a change made to their lated to diabetes management. is strategy of planned diabetes management during the visit. Among Spanish- patient preparedness may help patients bring their diabetes- speaking patients and their Spanish-speaking providers, related needs to the forefront of the visit so that the phy- we found that even in the absence of language barriers sician may be able to better address them. It may also help patients who asked fewer questions had poorer glycemic mitigate the “hand on the doorknob” phenomenon, where control. important issues are raised at the end of the visit7 and Previously described barriers to optimizing the primary thereby allow for more time to discuss and implement care visit include short visit lengths and the need for more changes in diabetes management. Structured visit prepa- collaborative communication.15 Because all patients in our ration may also act as a reminder for physicians to check in

120 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.208 ORIGINAL RESEARCH ARTICLE Visit Content Analysis: Doctor-Patient Communication in Patients with Type 2 Diabetes

Table 2. Patient and visit characteristics by change in diabetes management (study 1) Variable No change (n = 19) Lifestyle and/or medication change (n = 12) p value Patient characteristics Age, y (SD) 62.2 (10.2) 64.0 (9.1) 0.61 Women, n (%) 11.0 (57.9) 7.0 (58.3) 0.98 A1C, % (SD) 9.1 (1.4) 9.0 (0.9) 0.83 Visit characteristics Visit length, min (SD) 22.8 (8.0) 24.2 (7.5) 0.63 Distinct discussion topics, n (SD) Total 5.5 (2.0) 5.7 (2.0) 0.79 Non-diabetes related 3.8 (1.9) 3.5 (1.5) 0.66 Diabetes related 1.7 (0.9) 2.2 (0.9) 0.16 Word count per 15 min, n (SD) Total 1929.9 (379) 2114.6 (254) 0.15 Patient 827.4 (263) 864.7 (250) 0.70 Physician 1102.5 (301) 1250.0 (312) 0.20 Questions asked per 15 min, n (SD) Physician 31.1 (11.7) 41.0 (18.8) 0.08 Patient 7.1 (3.6) 8.1 (4.0) 0.47 Verbal exchanges per 15 min, n (SD) 92.3 (26.7) 90.9 (22.3) 0.89 Patient-defined visit priorities, n (%) Diabetes is any priority 10.0 (52.6) 11.0 (91.7) 0.02 Diabetes priority 0.02 First 5.0 (26.3) 9.0 (75.0) Second 5.0 (26.3) 2.0 (16.7) All priorities are addressed 16.0 (84.2) 11.0 (91.7) 1.00 Diabetes discussion Diabetes topic not mentioned until second half of visit, n (%) 6.0 (31.6) 1.0 (8.3) 0.20 SD = standard deviation.

on diabetes management, given its importance, regardless of language-concordant providers and suggest that language whether patients perceive it as a priority. discordance can be a barrier to effective care.10,18 Removing e Study 1 results confirm prior qualitative work showing the language barrier in our study allowed us to take a closer that having patients identify priorities can help organize visits look at the relationship between question-asking and and enable more productive doctor-patient interactions.15 HbA1c, previously identified as an important issue among Previous research has also shown that patients who re- Latino patients.9 We found a significant association be- ceived previsit prioritization surveys were more likely to tween less question-asking and poor glycemic control in our prepare a list of questions for their physicians and were study. is finding suggests that improving patients’ con- given more choices about their treatment.13 e elicitation fidence with and skills for question-asking may represent of the patient perspective allows clinicians and patients to a potentially powerful strategy in diabetes management. engage in meaningful conversations and contributes to Prior research has underscored that individuals with di- patient-centered care.16 Moreover, research has demon- abetes have one of the highest desires for information strated that patient participation in care has made patients among individuals with various chronic conditions19,20 and more effective in garnering information from physicians and that having more information improved self-management has improved diabetes management.17 Our results support and disease-related patient engagement in the decision- these findings by suggesting that having diabetes as a previsit making process.19,21 Because questions are a key indicator priority may promote interactions that facilitate a change in of patient engagement, patients who ask more questions diabetes management. are likely to receive more detailed explanations and have Prior studies of Spanish-speaking patients with T2D a greater role in decision-making.22 efrequencywith have shown improvements in glycemic control with which patients ask questions is also related to the degree to

The Permanente Journal·https://doi.org/10.7812/TPP/20.208 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 121 ORIGINAL RESEARCH ARTICLE Visit Content Analysis: Doctor-Patient Communication in Patients with Type 2 Diabetes

which doctors provide diagnostic and treatment information.23 Our research was approved by the Kaiser Permanente Institutional Review Prior findings show that an intervention helping patients Board, and all procedures followed were in accordance with the ethical standards of the Institutional Review Board and the Helsinki Declaration of 1975, as revised develop a list of questions for their visit with a physician in 2000. Informed, written consent was obtained from all patients included in resulted in increased patient activation levels.24 is fur- the study. ther emphasizes the significance of question-asking as it relates to doctor-patient communication. Authors’ Contributions e comparison of Study 1 and Study 2 demonstrated All contributing individuals met requirements for authorship of this manuscript. that, even with language barriers removed, Spanish speakers asked fewer questions than their English-speaking coun- Funding NIDDK R01DK099108 (PI: RW Grant), T32DK116684 (PI: RW Grant), and terparts. A common construct in Latino culture is the K24DK109114 (PI: RW Grant). deference to physician authority,9,25 which may lead to a lack of visit preparation and question-asking given the cultural References expectation that it is the responsibility of the provider to 1. Centers for Disease Control and Prevention. National diabetes statistics report, 2020. Centers for Disease Control and Prevention. Accessed February 14, 2020. https:// 26 set the agenda and know what to ask. Our study further www.cdc.gov/diabetes/library/features/diabetes-stat-report.html. builds on prior research that emphasizes the development of 2. Papatheodorou K, Banach M, Bekiari E, Rizzo M, Edmonds M. Complications of diabetes 2017. J Diabetes Res 2018 Mar;2018:3086167. DOI: https://doi.org/10.1155/2018/ question-asking skills to promote patient engagement spe- 3086167. cifically for Latinos.9 Furthermore, patient-facing commu- 3. Reusch JE, Manson JE. Management of type 2 diabetes in 2017: Getting to goal. J Am Med Assoc 2017 Mar;317(10):1015-6. DOI: https://doi.org/10.1001/jama.2017.0241, nication interventions, such as modeling of question-asking PMID:28249081. behaviors, have shown success in helping patients obtain 4. Berkowitz SA, Eisenstat SA, Barnard LS, Wexler DJ. Multidisciplinary coordinated care for type 2 diabetes: A qualitative analysis of patient perspectives. Prim Care more information from physicians and exhibit greater in- Diabetes 2018 Jun;12(3):218-23. DOI: https://doi.org/10.1016/j.pcd.2018.01.005, volvement during the visit.27 Future research should inves- PMID:29397351. tigate question-asking behaviors of other cultural groups and 5. American Diabetes Association. 1. Strategies for improving care. Diabetes Care 2015 Jan;38(Suppl):S5-7. DOI: https://doi.org/10.2337/dc15-S004, PMID:25537709. comparing men vs women in larger sample sizes. 6. Abbo ED, Zhang Q, Zelder M, Huang ES. The increasing number of clinical Some limitations of our study should be considered. e items addressed during the time of adult primary care visits. J Gen Intern Med 2008 Dec;23(12):2058-65. DOI: https://doi.org/10.1007/s11606-008-0805-8, relatively small number of transcribed visits limits our power PMID:18830762. to show smaller differences between groups. However, this 7. Kowalski CP, McQuillan DB, Chawla N, et al. ‘The hand on the doorknob’ Visit agenda setting by complex patients and their primary care physicians. J Am Board Fam Med 2018 is balanced by a greater depth of detail in the visits included Jan-Feb;31(1):29-37. DOI: https://doi.org/10.3122/jabfm.2018.01.170167, PMID: in our study and the relatively large effect sizes of our 29330237. fi 8. Barry CA, Bradley CP, Britten N, Stevenson FA, Barber N. Patients’ unvoiced agendas in signi cant predictors. Further research with a larger study general practice consultations: Qualitative study BMJ 2000 May;320(7244):1246-50. DOI: sample might reveal more differences between groups. In https://doi.org/10.1136/bmj.320.7244.1246. addition, though we were able to show several meaningful 9. Torres DX, Lu WY, Uratsu CS, Sterling SA, Grant RW. Knowing how to ask good questions: Comparing Latinos and non-Latino Whites enrolled in a cardiovascular disease associations, our observational study cannot be used to prevention study. Perm J 2019;23:18-258. DOI: https://doi.org/10.7812/TPP/18-258, determine causality. Future interventional work is needed PMID:30939290. 10. Parker MM, Fernandez´ A, Moffet HH, Grant RW, Torreblanca A, Karter AJ. to confirm the value of our findings. Finally, although Association of patient-physician language concordance and glycemic control for there may be some concern that recording primary care limited-english proficiency Latinos with type 2 diabetes. JAMA Intern Med 2017 Mar; 177(3):380-7. DOI: https://doi.org/10.1001/jamainternmed.2016.8648,PMID: visits may have caused participants to shift their behaviors 28114680. duetotheHawthorneeffect, other investigators have 11. Iturralde E, Sterling SA, Uratsu CS, Mishra P, Ross TB, Grant RW. Changing results- engage and activate to enhance wellness: A randomized clinical trial to improve previously shown that visit recordings do not change visit cardiovascular risk management. J Am Heart Assoc 2019 Dec;8(23):e014021. DOI: behaviors.28 https://doi.org/10.1161/JAHA.119.014021, PMID:31787053. 12. Kaiser Permanente Northern California, Division of Research. 2018 DOR summary analytics report. Diabetes registry; 2019. http://insidedor.kaiser.org/sites/ CONCLUSION DiabetesRegistry/AnalyticsReports/diabetes_registry_report_2018_registry_dor_ version_2019.pdf>. Quantitative visit content analysis can help shed light on 13. Vo MT, Uratsu CS, Estacio KR, et al. Prompting patients with poorly controlled diabetes to the content of patient-provider discussions during primary identify visit priorities before primary care visits: A pragmatic cluster randomized trial. fi J Gen Intern Med 2019 Jun;34(6):831-8. DOI: https://doi.org/10.1007/s11606-018-4756-4, care visits. Our analyses identi ed patient previsit topic PMID:30746642. prioritization and increased question-asking as 2 potential 14. Grant RW, Uratsu CS, Estacio KR, et al. Pre-Visit Prioritization for complex patients with  diabetes: Randomized trial design and implementation within an integrated health care domains for further intervention. ese results suggest that system. Contemp Clin Trials 2016 Mar;47:196-201. DOI: https://doi.org/10.1016/j.cct. new interventions to improve patient visit preparedness and 2016.01.012, PMID:26820612. confidence with asking questions represent a promising 15. Grant RW, Altschuler A, Uratsu CS, et al. Primary care visit preparation and communication for patients with poorly controlled diabetes: A qualitative study of patients strategy for improving diabetes primary care. v and physicians. Prim Care Diabetes 2017 Apr;11(2):148-53. DOI: https://doi.org/10.1016/ j.pcd.2016.11.003, PMID:27916628. 16. Singh Ospina N, Phillips KA, Rodriguez-Gutierrez R, et al. Eliciting the patient’s agenda- Disclosure Statement secondary analysis of recorded clinical encounters. J Gen Intern Med 2019 Jan;34(1): The author(s) have no conflicts of interest to disclose. 36-40. DOI: https://doi.org/10.1007/s11606-018-4540-5, PMID:29968051.

122 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.208 ORIGINAL RESEARCH ARTICLE Visit Content Analysis: Doctor-Patient Communication in Patients with Type 2 Diabetes

17. Greenfield S, Kaplan SH, Ware JE Jr, Yano EM, Frank HJ. Patients’ participation in medical 23. Street RL Jr. Information-giving in medical consultations: The influence of patients’ care: Effects on blood sugar control and quality of life in diabetes. J Gen Intern Med 1988 communicative styles and personal characteristics. Soc Sci Med 1991;32(5):541-8. DOI: Sep-Oct;3(5):448-57. DOI: https://doi.org/10.1007/BF02595921, PMID:3049968. https://doi.org/10.1016/0277-9536(91)90288-n, PMID:2017721. 18. Diamond L, Izquierdo K, Canfield D, Matsoukas K, Gany F. A systematic review of the 24. Deen D, Lu WH, Rothstein D, Santana L, Gold MR. Asking questions: The effect of a brief impact of patient-physician non-english language concordance on quality of care and intervention in community health centers on patient activation. Patient Educ Couns 2011 outcomes. J Gen Intern Med 2019 Aug;34(8):1591-606. DOI: https://doi.org/10.1007/ Aug;84(2):257-60. DOI: https://doi.org/10.1016/j.pec.2010.07.026, PMID:20800414. s11606-019-04847-5, PMID:31147980. 25. Diaz VA Jr. Cultural factors in preventive care: Latinos. Prim Care 2002 Sep;29(3):503-17, 19. Chernyak N, Stephan A, Bachle¨ C, Genz J, Julich¨ F, Icks A. Assessment of information viii. DOI: https://doi.org/10.1016/s0095-4543(02)00010-6, PMID:12529894. needs in diabetes: Development and evaluation of a questionnaire. Prim Care Diabetes 26. Zamudio CD, Sanchez G, Altschuler A, Grant RW. Influence of language and culture in 2016 Aug;10(4):287-92. DOI: https://doi.org/10.1016/j.pcd.2015.11.007, PMID:26777538. the primary care of Spanish-speaking Latino adults with poorly controlled diabetes: A 20. Duggan C, Bates I. Medicine information needs of patients: The relationships between qualitative study. Ethn Dis 2017 Autumn;27(4):379-86. DOI: https://doi.org/10.18865/ed. information needs, diagnosis and disease. Qual Saf Health Care 2008 Apr;17(2):85-9. 27.4.379, PMID:29225438. DOI: https://doi.org/10.1136/qshc.2005.017590, PMID:18385399. 27. Rao JK, Anderson LA, Inui TS, Frankel RM. Communication interventions make a 21. Say R, Murtagh M, Thomson R. Patients’ preference for involvement in medical decision difference in conversations between physicians and patients: A systematic review of the making: A narrative review. Patient Educ Couns 2006 Feb;60(2):102-14. DOI: https://doi. evidence. Med Care 2007 Apr;45(4):340-9. DOI: https://doi.org/10.1097/01.mlr. org/10.1016/j.pec.2005.02.003, PMID:16442453. 0000254516.04961.d5, PMID:17496718. 22. Roter DL. Patient participation in the patient-provider interaction: The effects of patient question 28. Schillinger D, Piette J, Grumbach K, et al. Closing the loop: Physician communication with asking on the quality of interaction, satisfaction and compliance. Health Educ Monogr 1977 diabetic patients who have low health literacy. Arch Intern Med 2003 Jan;163(1):83-90. Winter;5(4):281-315. DOI: https://doi.org/10.1177/109019817700500402, PMID:346537. DOI: https://doi.org/10.1001/archinte.163.1.83, PMID:12523921.

The Permanente Journal·https://doi.org/10.7812/TPP/20.208 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 123 n ORIGINAL RESEARCH ARTICLE Regionalization of Acute Myeloid Leukemia Treatment in a Community-Based Population: Implementation and Early Results

Lisa Y Law, MD1; Stephen P Uong, MS2; Hyma T Vempaty, MD3; Vu H Nguyen, MD4; David Baer, MD4; Vincent X Liu, MD2; Lisa J Herrinton, PhD2 Perm J 2021;25:20.271 E-pub: 03/17/2021 https://doi.org/10.7812/TPP/20.271

ABSTRACT and have substantial comorbidities.1 Intensive induction fol- Introduction: Regionalization of care for acute myeloid leu- lowed by consolidation with further chemotherapy or bone kemia (AML) has not been described for community-based set- marrow transplantation is the standard of care for patients who tings. In 2015, we shifted AML induction from 21 local centers to 3 can tolerate its toxicities,2 which many of the oldest patients regional centers. cannot.3 us, a key challenge in the management of AML is fi Methods: Using time-speci c inception cohorts, we assessed patient assessment and appropriate selection of therapy to whether regionalization was associated with the frequency of balance life expectancy, complications, and quality of life.4 use of induction therapy, receipt of bone marrow transplantation, 60-day mortality (treatment toxicity), and 180-day mortality In 2015, Kaiser Permanente Northern California began (treatment effectiveness). Information for all adult patients di- establishing a regional care pathway to improve AML man- agnosed with AML from 2013 to 2017 was obtained from the electronic health record. Multivariable methods were used to agement. Before regionalization, AML patients were managed estimate the adjusted associations of induction, bone marrow by community-based hematologic oncologists at 21 local cen- transplantation, and death in relation to year of diagnosis before fi and after regionalization. ters. After regionalization, new AML patients were rst triaged Results: Of 661 patients diagnosed during 2013 to 2017, 53% by 1 of the 15 leukemia subspecialists at 1 of the 3 regional ’ fi were ≥ 70 years, 22% were ≥ 80 years, and 10% died within the centers to assess each patient s tness to tolerate induction. week following diagnosis. Comparing 2017 with 2013, the pro- Patients who were fitandwhoagreedtoinductionwerethen portion of patients who received induction therapy increased 2.88 treated at a regional center; others received palliative care locally. times (95% confidence interval [CI] = 1.55-5.35), and the We conducted a longitudinal cohort study using incep- proportion of non-acute promyelocytic leukemia patients tion cohorts to assess the association of year of diagnosis receiving bone marrow transplantation increased 2.00 times with the frequency of use of induction therapy, bone (95% CI = 0.89-4.50). Regionalization was associated with lower marrow transplantation, 60-day mortality (a measure of 180-day mortality (hazard ratio [HR] = 0.64; 95% CI = 0.44-0.92), treatment toxicity), and 180-day mortality (a measure of whereas change in 60-day mortality was not statistically sig- treatment effectiveness). To avoid selection bias, we did not nificant (HR = 0.67; 95%CI = 0.43-1.04). Conclusion: In this community-based population, many pa- directly compare regionally treated patients with locally tients were of advanced age yet benefitted from AML induction treated patients because regionally treated patients were fi therapy delivered at a regionally specialized center. These selected based on their tness to tolerate treatment and early results suggest the benefit of regionalizing subspecialty would have a lower risk of death at baseline that could not be leukemia care. adequately controlled in the statistical analysis. Instead, we compared changes over time, reasoning that on average, inception cohorts of AML patients diagnosed each calendar INTRODUCTION year were similar and did not differ systematically. us, the e incidence of acute myeloid leukemia (AML) rises study examined time trends in treatment and outcomes sharply with age, and most patients with AML are elderly during 2013 to 2017, using the premise that treatment trends stemmed from regionalization while recognizing that Author Affiliations global changes in evidence and adoption could result in trends 1Department of Oncology, Roseville Medical Center, Kaiser Permanente Northern California, Roseville, CA as well. We hypothesized that later year of diagnosis would be 2Division of Research, Kaiser Permanente Northern California, Oakland, CA associated with increased use of induction and receipt of bone 3 Department of Oncology, Santa Clara Medical Center, Kaiser Permanente Northern California, Santa Clara, CA marrow transplantation and lower mortality. 4Department of Oncology, Oakland Medical Center, Kaiser Permanente Northern California, Oakland, CA

Corresponding Author Lisa J Herrinton, PhD ([email protected]) MATERIALS AND METHODS Keywords: acute myeloid leukemia, adverse events, care delivery, community-based studies, health services is project was determined by the National Compliance research, induction chemotherapy Officer of the Kaiser Foundation Research Institute as not fi Abbreviations: AML = acute myeloid leukemia; APL = acute promyelocytic leukemia; CI = confidence interval; meeting the regulatory de nition of human subjects research. HR = hazard ratio; OR = odds ratio. 124 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.271 ORIGINAL RESEARCH ARTICLE Regionalization of Acute Myeloid Leukemia Treatment in a Community-Based Population: Implementation and Early Results

Setting therapy, started on low-intensity induction, and then e longitudinal cohort study was set at Kaiser Permanente transferred back to the local center for subsequent treat- Northern California, a comprehensive, integrated healthcare ment. Patients who are not fit for induction or transfer or delivery system that serves one-third of the population, or who decline (< 5%) are managed locally toward the goal of 4.4 million members, at its 21 medical centers. e palliation. us, the regional centers disproportionately and membership has been well characterized for research and by design manage patients who are younger and have fewer represents the underlying population but underrepresents comorbidities, have better performance status, and are more the very poor and the very wealthy.5 Between 2013 and likely to survive to 60 days. 2017, the adult membership shifted to become slightly younger (aged 18-59, 75.6% in 2013 vs 79.9% in 2017) Study Population and more racially diverse (non-White, 53.6% in 2013 vs e study population included adults newly diagnosed 55.5% in 2017). with AML (ICD-O-3 histology codes 9805-9809, 9840- 9861, 9865-9874, and 9891-9931) between January 2013 Regional Care Pathway and December 2017.7 Cases were identified from the Kaiser Implementation of regional care began in 2015 following Permanente Northern California Cancer Registry.8 To agreements made among the medical group’s 21 oncology ensure adequate information about baseline health status, departments. Because most AML patients present with we restricted the study to patients with at least 1 year of critical disease, referral to 1 of 3 regional leukemia centers is enrollment before their AML diagnosis date but made no made through an urgent phone call. During the call, the other restrictions. local hematologist and 1 of 15 regional leukemia subspe- cialists review the patient’s medical record; discuss clinical Data Collection status, performance status and comorbidities; and assess Information was obtained from the health plan’s elec- the patient’ ability to tolerate low- or high-intensity in- tronic health record. Regional care was defined as care by duction chemotherapy toward the goal of extending sur- 1 of the 15 leukemia subspecialists identified with the re- vival. If the leukemia subspecialist judges the patient as able gional program. Induction therapy starting within 30 days to tolerate intensive induction, then the patient is imme- of diagnosis was identified from medication files. Induction diately transferred. A common triage tool used is the on-line was classified separately for acute promyelocytic leukemia AML score,6 which estimates treatment-related mortality (APL) and non-APL. For APL, we identified all-trans (ie, risk of death by 60 days based on various characteristics retinoic acid with arsenic or anthracycline. For non-APL, and measurements). Once referred, the patient is formally we separately assessed high-intensity and low-intensity evaluated by a leukemia subspecialist to assess eligibility for therapies,9 and receipt of bone marrow transplantation. induction therapy. Patients deemed fit are admitted and e latter was identified from procedure codes recorded cared for by the leukemia service hematologist-oncologist, within 180 days after the diagnosis and from transplant hospitalist, nurse, pharmacist, social worker, patient care history codes recorded within 365 days after diagnosis (see coordinator, and clinical educator who have training Supplemental Materiala). We identified deaths from health and experience caring for acutely ill AML patients. e plan mortality data and Social Security Administration data, patients also receive care from experts in transfusion which were complete through December 31, 2017. We also medicine, infectious diseases, and critical care. A clinical assessed nonfatal complications during the 60 days after trial coordinator evaluates each patient’s eligibility for diagnosis using ICD-10 diagnostic codes recorded in participation in national clinical trials. e15leukemia hospitalization data (see Supplemental Materiala). It is im- subspecialists conference monthly to discuss care pro- portant to keep in mind that the year 2015, when re- tocols and pathways. ey also meet weekly to review gionalization was initiated, was the same year that cases and patient status. A multidisciplinary round takes diagnostic and procedural coding was switched from the place daily, during which patients are evaluated by a team 9th to the 10th edition of the International Classification of including a clinical educator, a patient care coordinator, Diseases (ICD),10 with the ICD-10 containing many andamedicalsocialworker.Alleligiblepatientsare more codes than the ICD-9. referred for bone marrow transplant; the majority of bone Demographic information was obtained from mem- marrow transplants are performedbyStanfordUniversity bership data. Neighborhood deprivation index was cal- Medical Center. culated using geocoded addresses and American Community Patients deemed ineligible for high-intensity induction Survey data.11 e COmorbidity Point Score (COPS) usually are not transferred to the regional center. However, comorbidity score was developed at Kaiser Permanente some patients are transferred, found to be unfit for intensive Northern California and is computed from 70 diagnostic

The Permanente Journal·https://doi.org/10.7812/TPP/20.271 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 125 ORIGINAL RESEARCH ARTICLE Regionalization of Acute Myeloid Leukemia Treatment in a Community-Based Population: Implementation and Early Results

groups recorded in the year before AML diagnosis and (18-59, 60-69, 70-79, ≥ 80 years). To understand selection defined by the Centers for Medicare and Medicaid Services of patients for induction, we also compared patients diag- as Hierarchical Condition Categories and include sepsis, nosed in 2016 and 2017 who were selected for regional versus pneumonia, infection, acute myocardial infarction, cardiac local treatment (see Supplemental Materiala). Analyses were arrest, acute cardiovascular disease, other cardiac conditions, performed using SAS 9.4 and R 3.5.2. renal failure, cancer, liver and pancreatic diseases, cancer, trauma, and other conditions associated with death.12-15 e RESULTS COPS score is calibrated for inpatient mortality, with a Study Population score of 50 being associated with mortality risk of < 1%, a We identified 755 adults who were newly diagnosed with score of 100 with 5% mortality, and a score of 145 with 10% AML during 2013 to 2017. Of these, 94 did not have 1 year mortality.13 e Hierarchical Condition Categories are used of baseline enrollment before diagnosis and were excluded, for Medicare risk adjustment,16 and the COPS score has leaving 661 (88%) who were eligible for the study. During been validated.14 Infection at diagnosis was determined 2016 to 2017, 114 patients received local care, and 135 from hospital admission diagnoses of sepsis, pneumonia, or received regional care. other infection during the 15-day period before the date of AML diagnosis, inclusive. Baseline white blood cell count Baseline Characteristics at AML diagnosis or in the preceding 30 days was obtained Patients diagnosed in 2013 and 2014 were similar to from laboratory data. Baseline estimated glomerular fil- those diagnosed in 2016 and 2017 with respect to age, sex, tration rate was calculated using the CKD-EPI equation17 race/ethnicity, baseline white blood cell count, infection at and the most recent measurement of serum creatinine and diagnosis, and estimated glomerular filtration rate (Table 1). was dichotomized as ≤ 29.9 vs ≥ 30 mL/min/1.73 m2.We Patients diagnosed in 2016 and 2017 were more likely to be also identified patients with APL, therapy-related myeloid married or partnered (p < 0.05). ey also had greater neoplasm, mixed phenotype acute leukemia, unspecified COPS scores (p < 0.05), although this likely stemmed from myeloid leukemia, antecedent myelodysplastic syndrome, or changes in ICD coding. chronic myeloid leukemia in blast crisis using information from the cancer registry. Treatment and Outcomes Among the 42 patients with APL, we observed no change Statistical Analysis over time in the likelihood of initiating induction che- In bivariate analyses, we compared population propor- motherapy within 30 days (2013-2014, 82.6%; 2016-2017, tions of baseline characteristics, induction therapy, bone 78.9%; p = 0.73) (Table 1). However, among the 485 marrow transplantation, and mortality. ese analyses patients with non-APL, use of induction chemotherapy combined cases diagnosed in 2013 and 2014, before the increased (2013-2014, 49.0%; 2016-2017, 65.2%; p < start of regionalization, and combined those diagnosed in 0.001), with FLAG ± anthracycline and azacitidine ac- 2016 and 2017, when regionalization was well underway. counting for most of this increase. In multivariable analysis, We used the χ2 test to assess statistical significance. To diagnosis in 2017 compared with 2013 was associated with calculate the actuarial incidence of death per 100 patient- an increased odds of initiating induction therapy within months, we began follow-up on the patient’sdiagnosis 30 days (odds ratio [OR] = 2.88; 95% CI = 1.55-5.35) date and ended follow-up on the earliest of the date of (Table 2 and Supplemental Table 1a) and was associated death, disenrollment, 60 or 180 days, or the end of the with an increased odds of bone marrow transplantation mortality data on December 31, 2017. In multivariate (OR = 2.00; 95% CI = 0.89-4.50) (Table 2), although the logistic regression analysis, we estimated the odds ratio latter may have resulted from chance. (OR) and 95% confidence interval (CI) for the association Of the entire cohort of 661 patients, 10% died within a of year of diagnosis with receipt of induction therapy in all week of diagnosis. roughout 2013 to 2017, the pro- AML patients and bone marrow transplantation in non- portion of patients who died by 60 days was 34.4% for all APL patients after adjusting for age at diagnosis, sex, race/ ages and 45.1% among patients aged ≥ 66 years. e ethnicity, neighborhood deprivation index, the COPS proportion who died by 180 days was 50.1% for all ages and comorbidity score, elevated white blood cell count, and 63.9% among patients aged ≥ 66 years. About one-third of infection at diagnosis. In multivariate proportional hazards patients died within 60 days, and one-half of patients died analysis, we estimated the adjusted hazard ratio (HR) and within 180 days. In bivariate analyses, these proportions 95% CI for the association of year of diagnosis with the did not differ between 2014 to 2015 and 2016 to 2017 mortality rate at 60 and 180 days. Subgroup analyses were (Table 1). We also conducted multivariable analyses that performed to assess bivariate associations in relation to age accounted for loss to follow-up and for confounding by AML

126 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.271 ORIGINAL RESEARCH ARTICLE Regionalization of Acute Myeloid Leukemia Treatment in a Community-Based Population: Implementation and Early Results

Table 1. Baseline characteristics, treatment, and mortality of Table 1. Baseline characteristics, treatment, and mortality of patients by year, % patients by year, % (continued) 2013-2014 2016-2017 2013-2014 2016-2017 Characteristica (n = 278) (n = 249) Characteristica (n = 278) (n = 249) Year of diagnosis Infection at diagnosise 2013 135 (48.6%) - Yes 78 (28.1%) 63 (25.3%) 2014 143 (51.4%) - eGFR at diagnosis 2015 - - ≤ 29.9 mL/min/1.73 m2 24 (8.3%) 17 (6.8%) 2016 - 119 (47.8%) Treatmentf 2017 - 130 (52.2%) APL induction therapy, 0-30 d n = 23 n = 19 Leukemia subtype ATRA + arsenic 9 (39.1%) 9 (47.4%) De novo AML (none of the below) 215 (77.3%) 199 (79.9%) ATRA + anthracycline 7 (30.4%) 2 (10.5%) Antecedent MDS 25 (9.0%) 21 (8.4%) Other regimen 3 (13.0%) 4 (21.1%) Secondary AML 6 (2.2%) 1 (0.4%) No induction 4 (17.4%) 4 (21.1%) APL 23 (8.3%) 19 (7.6%) Non-APL induction therapy, 0-30 d n = 255 n = 230 CML in blast crisis 5 (1.8%) 4 (1.6%) High intensity Mixed phenotype acute leukemia 1 (0.4%) 1 (0.4%) 7+3 77 (30.2%) 68 (29.6%) Unspecified myeloid leukemia 5 (1.8%) 4 (1.6%) FLAG 2 (0.8%) 22 (9.6%) Age at diagnosis, y MEC 1 (0.4%) - 18-59 72 (25.9%) 57 (22.9%) Low intensity 60-69 56 (20.1%) 60 (24.1%) Azacitidine 38 (14.9%) 51 (22.2%) 70-79 82 (29.5%) 86 (34.5%) Decitabine 5 (2.0%) 6 (2.6%) ≥ 80 68 (24.5%) 46 (18.5%) Other regimen 2 (0.8%) 3 (1.3%) Sex No induction 130 (51.0%) 80 (34.8%) Female 124 (44.6%) 120 (48.2%) Bone marrow transplant 43 (16.9%) 51 (22.2%) Male 154 (55.4%) 129 (51.8%) Proportion who died, 0-60 dg Race/ethnicity All patients 94 (33.8%) 78 (33.9%) Asian 30 (10.8%) 40 (16.1%) APL only 4 (17.4%) 6 (31.6%) Black 17 (6.1%) 21 (8.4%) Non-APL only 90 (35.3%) 72 (34.1%) Hispanic, any race 31 (11.1%) 28 (11.2%) Proportion who died, 0-180 dg White 188 (67.6%) 145 (58.2%) All patients 141 (50.7%) 92 (47.7%) Other/unknown 12 (4.3%) 15 (6.0%) APL only 4 (17.4%) 7 (43.8%) Partner status Non-APL only 137 (53.7%) 85 (48.0%) Married, partnered 151 (54.3%) 160 (64.3%) aNine patients were missing a white blood cell count and 3 were missing an eGFR Single, divorced, widowed 97 (34.9%) 75 (30.1%) measurement. bIncludes history of solid cancer (other than nonmelanoma skin cancer) or myelodysplastic Other/unknown 30 (10.8%) 14 (5.6%) syndrome (9980, 9982-9987, 9989, 9991-9992) using information from the beginning of Neighborhood deprivation indexc the cancer registry in 1973, mixed phenotype acute leukemia (9805-9809), and therapy- related myeloid neoplasm (9920). Quartile 1, least deprived 68 (24.5%) 64 (25.7%) cValues that are more positive represent greater deprivation. Quartile ranges: Q1: −1.60 to Quartile 2 71 (25.5%) 61 (24.5%) < −0.85, Q2: −0.85 to < −0.50, Q3: −0.50 to < 0.07, Q4: 0.07-3.00. dAs regionalization was initiated in 2015, the same year as the ICD-9 to ICD-10 code Quartile 3 74 (26.6%) 58 (23.3%) transition, the elevated COPS score may be related to changes in ICD coding. Quartile 4, most deprived 65 (23.4%) 66 (26.5%) eRecorded during the 15 d before the diagnosis of AML using ICD-9 diagnosis codes COPS scored 001.X-139.X excluding 038 and ICD-10 diagnosis codes A00-B99 excluding A40 and A41. Quartile 1 (0-35) 82 (29.5%) 46 (18.5%) fOnly the first cycle of therapy is shown. 7+3 includes cytarabine and anthracycline ± fl Quartile 2 (36-69) 65 (23.4%) 69 (27.7%) midostaurin; FLAG includes cytarabine, and udarabine ± anthracycline; MEC includes cytarabine, mitoxantrone, and etoposide. Quartile 3 (70-104) 68 (24.1%) 64 (25.7%) gFor this table, the analysis of 60-d mortality included patients diagnosed before November Quartile 4 (105-284) 64 (23.0%) 70 (28.1%) 2017, and the analysis of 180-d mortality included patients diagnosed before July 2017. AML = acute myeloid leukemia; APL = acute promyelocytic leukemia; ATRA = all-trans White blood cell count retinoic acid; CML = chronic myeloid leukemia; EGFR = estimated glomerular filtration rate; > 100,000 32 (11.7%) 27 (11.1%) MDS = myelodysplastic syndrome. (continued in next column)

The Permanente Journal·https://doi.org/10.7812/TPP/20.271 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 127 ORIGINAL RESEARCH ARTICLE Regionalization of Acute Myeloid Leukemia Treatment in a Community-Based Population: Implementation and Early Results

Table 2. Adjusted associationsa of patient and disease characteristics with initiation of induction therapy, referral for bone marrow transplantation, and mortality at 60 and 180 d, 2013-2017 (N = 661) Induction therapy within 30 d Bone marrow transplantationb 60-d mortality 180-d mortality Year of diagnosisa % OR (95% CI) OR (95% CI) HR (95% CI) HR (95% CI) 2013 20 1.00 (Ref.) 1.00 (Ref.) 1.00 (Ref.) 1.00 (Ref.) 2014 22 1.11 (0.61-2.00) 0.88 (0.39-1.99) 0.91 (0.60-1.38) 0.87 (0.62-1.22) 2015 20 1.94 (1.05-3.57) 0.99 (0.44-2.25) 0.99 (0.65-1.52) 0.88 (0.62-1.24) 2016 18 1.83 (0.98-3.41) 0.97 (0.42-2.29) 1.06 (0.68-1.65) 0.91 (0.64-1.32) 2017 20 2.88 (1.55-5.35) 2.00 (0.89-4.50) 0.67 (0.43-1.04) 0.64 (0.44-0.92) aThe models were adjusted for age at diagnosis (18-59, 60-69, 70-79, ≥ 80 y), acute myeloid leukemia type (estimated glomerular filtration rate [APL], non-APL), sex (female, male), race/ ethnicity (white, other), neighborhood deprivation index (continuous), COPS score (quartiles), estimated glomerular filtration rate (eGFR) ≤ 29.9 mL/min/1.73 m2 (no, yes), infection at diagnosis (no, yes), and white blood cell count > 100,000/µL (no, yes). One patient was missing information on the neighborhood deprivation index, 12 on white blood cell count, and 6 on eGFR. bRestricted to non-APL patients only. Patients aged ≥ 80 y were excluded from the model because they were not eligible to receive a referral for bone marrow transplantation. cNeighborhood deprivation index had median −0.5 with interquartile range 0.9. See Supplemental Materiala for full model results. CI = confidence interval; HR = hazard ratio; OR = odds ratio.

complications per 100 person-months were as follows: sepsis, 30 (95% CI = 23-36); pneumonia, 23 (95% CI = 17- 28); other infection, 30 (95% CI = 23-36); major bleeding, 9 (95% CI = 6-12); respiratory failure, 8 (95% CI = 5-11); and embolism and thrombosis, 7 (95% CI = 4-10).

DISCUSSION In 2015, Kaiser Permanente Northern California began implementing a regional care pathway such that patients who were eligible for AML induction therapy were referred to 3 regional centers staffed by 15 leukemia subspecialists. We show that over time, more patients, including the el- derly, received induction therapy, particularly the reduced- intensity regimen azacitidine, among non-APL patients. Reduced-intensity induction regimens are consistent with contemporary evidence-based medicine, offering a thera- Figure 1. Kaplan-Meier plot of survival after AML diagnosis in relation to year of diagnosis, Kaiser Permanente Northern California, 2013-17*. *Log-rank test p = 0.41. peutic option for elderly patients with the goal of prolonging survival and maintaining good quality of life.18 Coincident with these changes, we observed reductions in 60- and type (APL, non-APL), age at diagnosis, sex, race/ethnicity, 180-day mortality, although the former CI was somewhat neighborhood deprivation index, the COPS comorbidity wide. Sixty-day mortality is a measure of treatment tox- score, elevated white blood cell count, and infection at di- icity, whereas 180-day mortality also captures the benefit agnosis. In these analyses, the risks of death by 60 and of treatment. 180 days were lower in 2017 compared with 2013 (60-day: e timing of regionalization of care for AML patients HR = 0.67; 95% CI = 0.43-1.04; 180-day: HR = 0.64; 95% took place around the same time that adoption of reduced- CI = 0.44-0.92) (Table 2), although the 60-day association intensity induction regimens was increasing among community- may have resulted from chance. based oncologists. It is difficult to assess whether these Subgroup analyses stratified by age suggested that in- regimens would have been adopted consistently without the duction increased most in the oldest patients, bone marrow extra effort of regionalization. If so, the effect of region- transplantation increased in younger patients, and mortality alization may be lower than we have estimated. However, declined in the oldest age group (Figure 2; Supplemental the magnitude of change in mortality over the course of the Figure 1a), although the number of patients available for study was larger than would be expected from the choice of subgroup analysis was limited, and none of these differences regimen alone, and other changes resulting from region- was statistically significant. alization likely were beneficial as well. During 2016 to 2017 when the ICD-10 was used Comparing the present study with past population-based to recorded diagnoses, the overall incidence rates of reports is challenging. e oldest patients are often ineligible

128 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.271 ORIGINAL RESEARCH ARTICLE Regionalization of Acute Myeloid Leukemia Treatment in a Community-Based Population: Implementation and Early Results

Figure 2. Time trends in (A) induction, (B) bone marrow transplantation, (C) 60-day mortality, and (D) 180-day mortality, by age group. The plots above show percentages of patients who (A) received induction therapy within 30 days of AML diagnosis, (B) received bone marrow transplant (patients aged 80 or older were not eligible for bone marrow transplant), (C) died within 60 days of AML diagnosis, or (D) died within 180 days of AML diagnosis.

for intensive treatments yet have a high risk of death.19 Most treated with chemotherapy within 30 days of diagnosis studies lumped these patients into groups that concealed at National Cancer Institute-designated cancer centers differences in eligibility for intensive treatment. Ho et al20 or community settings during 1999 to 2014. Patients compared 60-day mortality among 7007 non-APL patients treated in the community were older (≥ 66 years: 40% vs

The Permanente Journal·https://doi.org/10.7812/TPP/20.271 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 129 ORIGINAL RESEARCH ARTICLE Regionalization of Acute Myeloid Leukemia Treatment in a Community-Based Population: Implementation and Early Results

26%; p < 0.0001) and were more likely to have ≥ 3 2016 and 2017, then the study may have under- or over- comorbidities (41% vs 30%; p < 0.0001) than those estimated any benefits resulting from regional care. Indeed, treated at cancer centers. Mortality at 60 days among all we believe that more patients received molecular testing and ages was 24% in those treated in the community and 12% targeted therapy in recent years, and this may have improved in those treated at cancer centers (p < 0.001).21 Asimilar the prognosis, although we doubt this could explain the study by the same authors of 30-day mortality among all entire benefit we observed. APL patients treated in California found that 27% died Other limitations should be considered as well. e within 30 days.22 e study we report used different changes in ICD coding from ICD-9 and ICD-10 likely exclusion criteria than the statewide study resulting in a caused an artefactual increase in comorbidity score, an much older study population (age ≥ 66 years: statewide effect we are seeing across our research studies, because study, 36%; present study, 63%), and among those aged ≥ ICD-10 requires more coding than ICD-9. Similarly, 66 years seen in any setting, 60-day mortality was 36% in patients with more healthcare contacts have greater op- the statewide study and 45% in the present study. portunities for coding comorbidities and complications, ompson et al23 studied the 2013 Medicare fee-for- whereas those who enter hospice or palliative care likely service population (aged ≥ 65 years) of 7568 AML patients. have less documentation. At 30 days, risk-adjusted mortality was 32% in low-volume Regionalization of surgical oncology services has been hospitals (2 cases per year) and 28% in very-high-volume described in Canada and the Veterans’ Administration, hospitals (25 cases per year). A strength of the study was its with the former noting benefits particularly to older patients use of the Medicare cohort, which accurately represented with lung cancer and the latter noting increased use of the oldest-old population, who constitute the majority of therapy overall in the hepatopancreaticobiliary system.25,26 AML patients and who are underrepresented in many In our setting, an additional benefit of regionalization was studies. High-volume centers may have better outcomes due the greater opportunity for patients to participate in national to greater resources and staff experience that improve pa- clinical trials because of the logistical support available in the tient assessment, tailored chemotherapy, and management regional care setting. of side effects, particularly in the oldest-old population.23,24 In conclusion, we observed an association of regionali- Our findings of increased use of induction chemotherapy zation with increased utilization of induction therapy and and bone marrow transplant among older patients com- bone marrow transplantation and decreased 180-day bined with a seeming reduction of treatment-related and mortality and possibly 60-day mortality. e treatment of longer-term mortality are encouraging, but these results are AML has evolved during the past 3 years, with at least 8 early and suggestive. Overall survival data to 180 days are new drugs being FDA approved and with each drug having not adequate for drawing conclusions about long-term unique adverse-effect profiles. As AML treatment becomes survival, and further study is needed. lengthier and more complicated, increased specialization Moving forward, we plan additional steps to improve will be required to deliver appropriate care tailored to the care delivery and outcomes. Ongoing initiatives include patient’s indications and tolerance. We believe that re- developing specific criteria for recommending specific gionalization will benefit AML patients during their initial treatment regimen (ie, high-intensity vs low-intensity in- treatment phase, but because treatment can last for months, duction vs targeted therapy), better documenting systemic ongoing collaboration between the regional and local levels triage, ensuring high-quality and high-yield physician con- ultimately may provide the best care in the future. Col- ferences, improving coordination of care and follow-up of laborative models should be tested to assure excellent patients treated at local centers, and better evaluating patient treatment that is accessible to the patient. v preferences and satisfaction. We are also discussing imple- ’ mentation of telemedicine for patients long-term follow-up Disclosure Statement after they return to their local centers to ensure continuity and The investigators are partners and staff of The Permanente Medical Group and survivorship care. report no other conflicts of interest. Observational studies generally provide poor evidence for ’ treatment effectiveness because patients treated regionally Authors Contributions Lisa Law, MD, assisted in study design, data analysis, and manuscript and locally are not comparable. It was for this reason that we preparation (drafting and critical review); Stephen Uong assisted in data collection, analyzed time-specific inception cohorts, reasoning that, on data management, data analysis, and critical review; Hyma Vempaty assisted in average, the underlying indications for treatment and ability data analysis and drafting of the final manuscript; David Baer assisted in analysis of data and drafting of the final manuscript; Vincent Liu assisted in critical review and to tolerate treatment did not change over the short period of drafting of the final manuscript; Lisa Herrinton assisted in study design, data the study. However, if patients diagnosed with AML analysis, drafting, and critical review of the manuscript. All authors have given final during 2013 and 2014 differed from those diagnosed during approval to the manuscript.

130 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.271 ORIGINAL RESEARCH ARTICLE Regionalization of Acute Myeloid Leukemia Treatment in a Community-Based Population: Implementation and Early Results

13. Escobar GJ, Greene JD, Scheirer P, Gardner MN, Draper D, Kipnis P. Risk-adjusting Funding hospital inpatient mortality using automated inpatient, outpatient, and laboratory This project was supported by The Permanente Medical Group Delivery databases. Med Care 2008 Mar; 46(3):232-9. DOI: https://doi.org/10.1097/MLR. Science and Applied Research initiative. Dr Vincent X Liu was further supported by 0b013e3181589bb6, PMID:18388836. the National Institute of General Medical Sciences under grant number NIH 14. van Walraven C, Escobar GJ, Greene JD, Forster AJ. The Kaiser Permanente inpatient R35GM128672. risk adjustment methodology was valid in an external patient population. J Clin Epidemiol 2010 Jul;63(7):798-803. DOI: https://doi.org/10.1016/j.jclinepi.2009.08.020, PMID: 20004550. Past Presentations 15. Escobar GJ, Gardner MN, Greene JD, Draper D, Kipnis P. Risk-adjusting hospital None. mortality using a comprehensive electronic record in an integrated health care delivery system. Med. Care 2013 May;51(5):446–53. DOI: https://doi.org/10.1097/MLR. 0b013e3182881c8e, PMID:23579354 Supplemental Material 16. Pope GC, Ellis RP, Ash AS, et al. Diagnostic cost group hierarchical condition category aSupplemental Material is available at: www.thepermanentejournal.org/files/ models for Medicare risk adjustment, December 21; 2000. Accessed December 9, 2020. 2021/20.271supp.pdf. https://www.cms.gov/Research-Statistics-Data-and-Systems/Statistics-Trends-and- Reports/Reports/downloads/pope_2000_2.pdf. 17. Levey AS, Stevens LA, Schmid CH, et al. A new equation to estimate glomerular filtration References rate. Ann Intern Med 2009 May;150(9):604-12. DOI: https://doi.org/10.7326/0003-4819- 1. Appelbaum FR. Acute leukemias in adults. Published on clinicalgate.com; 2015. 150-9-200905050-00006, PMID:19414839. Accessed September 25, 2018. https://clinicalgate.com/acute-leukemias-adults-2/ 18. National Comprehensive Cancer Network. Acute myeloid leukemia, version 3.2019; 2. American Cancer Society. Typical treatment of acute myeloid leukemia (Except APL). 2019. Accessed July 29, 2019. https://www.nccn.org/professionals/physician_gls/pdf/ Accessed September 26, 2018. https://www.cancer.org/cancer/acute-myeloid-leukemia/ aml.pdf treating/typical-treatment-of-aml.html. 19. Dinmohamed AG, Visser O, van Norden Y, et al. Treatment, trial participation 3. Juliusson G, Lazarevic V, Horstedt¨ A-S, Hagberg O, Hoglund,¨ M; Acute Leukemia and survival in adult acute myeloid leukemia: A population-based study in the Registry Group. Acute myeloid leukemia in the real world: Why population-based Netherlands, 1989-2012. Leukemia 2016 Jan;30(1):24-31. DOI: https://doi.org/ registries are needed. Blood 2012 Apr;119(17):3890-9. DOI: https://doi.org/10.1182/ 10.1038/leu.2015.188. blood-2011-12-379008. 20. Ho G, Wun T, Muffly L, et al. Decreased early mortality associated with the treatment of 4. Klepin HD. Geriatric perspective: How to assess fitness for chemotherapy in acute acute myeloid leukemia at National Cancer Institute-designated cancer centers in myeloid leukemia. Hematology Am Soc Hematol Educ Program 2014 Dec;2014(1):8-13. California. Cancer 2018 May;124(9):1938-45. DOI: https://doi.org/10.1002/cncr.31296, DOI: https://doi.org/10.1182/asheducation-2014.1.8, PMID:25696829. PMID:29451695. 5. Gordon NP. Similarity of the adult Kaiser Permanente membership in Northern California 21. Ho G, Jonas BA, Li Q, Brunson A, Wun T, Keegan THM. Early mortality and complications to the insured and general population in Northern California: Statistics from the 2011 in hospitalized adult Californians with acute myeloid leukaemia. Br J Haematol 2017 Jun; California Health Interview Survey; 2015. Accessed July 26, 2019. https:// 177(5):791-9. DOI: https://doi.org/10.1111/bjh.14631, PMID:28419422. divisionofresearch.kaiserpermanente.org/projects/memberhealthsurvey/ 22. Ho G, Li Q, Brunson A, Jonas BA, Wun T, Keegan THM. Complications and SiteCollectionDocuments/chis_non_kp_2011.pdf. early mortality in patients with acute promyelocytic leukemia treated in California. 6. Study alliance leukemia: AMLCG. Accessed February 5, 2020. https://www.aml-score.org/ Am J Hematol 2018 Nov;93(11):E370-2. DOI: https://doi.org/10.1002/ajh.25252, 7. World Health Organization. ICD-10: International statistical classification of PMID:30105792. diseases and related health problems: Tenth revision, 2nd ed. Geneva: World 23. Thompson MP, Waters TM, Kaplan EK, McKillop CN, Martin MG. Hospital volume and Health Organization; 2014. acute myeloid leukemia mortality in Medicare beneficiaries aged 65 years and older. 8. National Cancer Institute. Surveillance epidemiology, and end results program. Accessed Blood 2016 Aug;128(6):872-4. DOI: https://doi.org/10.1182/blood-2016-05-716662, May 5, 2019. https://seer.cancer.gov/registrars/guidelines.html PMID:27357700. 9. Cheung E, Perissinotti AJ, Bixby DL, et al. The leukemia strikes back: A review of 24. Hahn AW, Jamy O, Nunnery S, et al. How center volumes affect early outcomes in acute pathogenesis and treatment of secondary AML. Ann Hematol 2019 Mar;98(3):541-59. myeloid leukemia. Clin Lymphoma Myeloma Leuk 2015 Nov;15(11):646-54. DOI: https:// DOI: https://doi.org/10.1007/s00277-019-03606-0, PMID:30666431. doi.org/10.1016/j.clml.2015.07.646, PMID:26386907. 10. World Health Organization. International classification of diseases for oncology (ICD-O-3), 25. Bendzsak AM, Baxter NN, Darling GE, Austin PC, Urbach DR. Regionalization 3rd ed. Geneva: World Health Organization; 2013. and outcomes of lung cancer surgery in Ontario, Canada. J Clin Oncol 2017 Aug; 11. United States Census. American community survey (ACS). Accessed May 15, 2019. 35(24):2772-80. DOI: https://doi.org/10.1200/JCO.2016.69.8076,PMID: https://www.census.gov/programs-surveys/acs 28682689. 12. Kipnis P, Turk BJ, Wulf DA, et al. Development and validation of an electronic 26. Lau K, Salami A, Barden G, et al. The effect of a regional hepatopancreaticobiliary medical record-based alert score for detection of inpatient deterioration outside the surgical program on clinical volume, quality of cancer care, and outcomes in the Veterans ICU. J Biomed Inform 2016 Dec;64:10-9. DOI: https://doi.org/10.1016/j.jbi.2016.09.013, Affairs System. JAMA Surg 2014 Nov;149(11):1153-61. DOI: https://doi.org/10.1001/ PMID:27658885. jamasurg.2014.1711.

The Permanente Journal·https://doi.org/10.7812/TPP/20.271 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 131 n ORIGINAL RESEARCH ARTICLE Unhealthy Drinking Behavior and the ATTAIN Solution: Web-based Automated Alcohol Misuse Interventions

Jennifer Chevinsky, MD, MPH1; Emma Fredua, MPH, CHES2; Ebonie M Vazquez, MD2; Mohamed H Ismail, MD, DrPH2 Perm J 2021;25:20.141 E-pub: 03/17/2021 https://doi.org/10.7812/TPP/20.141

ABSTRACT Approximately 88,000 peoplediefromalcohol-related Background: Up to 30% of American adults may have un- causes annually, making alcohol the third leading pre- healthy drinking behavior, but only 17% get screened. There is ventablecauseofdeathintheUS.In2010,itwases- promise in improving screening via technology, but there is a lack timated that unhealthy drinking behavior (also known as ff of published evidence supporting these e orts. We describe alcohol misuse) cost the US $249 billion. In a system the development of Automated Alcohol Misuse Interventions attempting to avoid preventable morbidity and mortality (ATTAIN), an automated, web-based process to screen for and while controlling costs, addressing alcohol use is an im- manage adults with unhealthy drinking behavior with minimal  involvement of health-care personnel. portant initiative. us, the Centers for Medicare and Method: After creating a strategic business plan, ATTAIN was Medicaid Services and the National Committee for developed for the Southern California Permanente Medical Group Quality Assurance (NCQA) have recommended that using its integrated model of care, electronic medical records, and health-care organizations incorporate alcohol screening as a patient portal. ATTAIN is based on an automated branching quality measure (Healthcare Effectiveness Data and In- questionnaire that screens for unhealthy drinking behavior and, formation Set, or HEDIS).2 when applicable, alcohol use disorders, and incorporates ques- e most current accepted nomenclature for describing tions about readiness to change and interest in medications/ problematic alcohol drinking patterns can be found in the counseling to assist with alcohol consumption reduction. Health fifth version of the Diagnostic and Statistical Manual of fi plan members would be invited via email to ll out the screening Mental Disorders (DSM-V), which uses the terminology questionnaire using the patient portal. Based on their responses, “alcohol misuse (or unhealthy drinking behavior)” and they would receive appropriate automated feedback and a link to “ 3 ” a counseling video about the spectrum of alcohol use. Patients’ alcohol use disorder (mild, moderate, severe) . According 4 responses would be captured in their medical record and sent to a to the US Preventive Services Task Force, alcohol misuse “ designated provider for further help as needed. The process is a spectrum of behaviors, including risky or hazardous would be refined through successive quality improvement pilots. alcohol use . . . . Risky or hazardous alcohol use means We project that ATTAIN will lead to reduced costs for the Southern drinking more than the recommended daily, weekly, or per- California Permanente Medical Group . occasion amounts resulting in increased risk for health Conclusion: This effort has paved the way for using ATTAIN to consequences.” According to the National Institute on improve patient care and to reduce the costs associated with Alcohol Abuse and Alcoholism, “problem drinking that managing unhealthy drinking, and potentially leads to similar becomes severe is given the medical diagnosis of “alcohol processes for other medical conditions and health-related use disorder.” Alcohol use disorder is a chronic, relapsing behaviors. brain disease characterized by compulsive alcohol use, loss of control over alcohol intake, and a negative emotional INTRODUCTION state when not using.5 As of 2013, the DSM-V no longer Excessive use of alcohol contributes to negative health, uses the terminology of alcohol abuse or dependence. ese social, and financialoutcomesforallagegroups.Ina terms now align most closely with the range of alcohol 2015 National Survey on Drug Use and Health,1 it was use disorders. Other terminology in the literature includes estimated that 15.1 million US adults had drinking excessive alcohol use, hazardous alcohol use, unhealthy patterns that would qualify as an alcohol use disorder. alcohol use, or problem drinking, all of which align with the spectrum of alcohol misuse and use disorders. We based our terminology on this current DSM-V nomenclature. ere is a long history of many different kinds of face-to-face Author Affiliations strategies that have been implemented to address the spectrum 1 Loma Linda University Medical Center, Loma Linda, CA of alcohol misuse and use disorders, including screening, brief 2Southern California Permanente Medical Group, Pasadena, CA intervention, and referrals. However they have proved costly Corresponding Author and time-consuming. Far fewer studies, comparatively, have Mohamed H Ismail, MD, DrPH ([email protected]) implemented digital or computer-based interventions to ad- dress unhealthy drinking behaviors. ese initial studies have Keywords: alcohol misuse, automated screening, brief intervention for alcohol use, HEDIS measures, unhealthy drinking behavior 132 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.141 ORIGINAL RESEARCH ARTICLE Unhealthy Drinking Behavior and the ATTAIN Solution: Web-based Automated Alcohol Misuse Interventions

shown promise using short-term, multistage computer- generated personalized interventions for milder disorders or misuse. However, the majority of studies have focused on hospitals, community-based settings, and smaller pop- ulation subsets (such as US veterans or college students).6-30 ere is a gap in research covering computer-based programs that address unhealthy drinking behavior screen- ing and intervention in integrated, managed-care orga- nizations. erefore, our study objective was to develop an automated, computer-based tool that screens people for unhealthy drinking behaviors and alcohol use disorders, and offers appropriate web-based counseling with minimal involvement of health-care personnel. is tool would be Figure 1. Stakeholder map for the health plan implementing Automated Alcohol integrated into the electronic medical record (EMR)– Misuse Interventions. CMS, Centers for Medicare Services; NCQA, National Committee for Quality Assurance (an independent accrediting agency). patient portal interface of an integrated, managed-care setting. We named this tool Automated Alcohol Misuse Interventions (ATTAIN; previously named AAMIDRX, friendly, and nonjudgmental fashion; 3) to partner with Automation of the Alcohol Misuse and Identification & addiction medicine; 4) to launch pilots, get feedback Treatment, but we found the new name to be better for from stakeholders, and refine ATTAIN based on this branding). feedback; and 5) to secure resources needed for long- term success. e critical success factor is the outcome of METHODS the program (the number of people who get screened and Integration Site counseled). Successful outcomes should help us keep the is project was developed for the Southern California program funded and give us the evidence to expand it. Permanente Medical Group (SCPMG), which serves more Budget Assumptions than 4 million Kaiser Permanente health plan members. Of Expenses including the initial program development and those members, approximately 2.68 million are adults, and maintenance costs (salaries, ATTAIN running costs, etc) thus are candidates for alcohol use screening.31 Based on the are covered by the SCPMG. ATTAIN does not generate National Institute on Alcohol Abuse and Alcoholism es- direct revenue and will be offered to the target population at timates that up to 30% of the eligible population may meet no cost. Screening people via ATTAIN is expected to be the screening criteria for unhealthy drinking behavior, it is less expensive than doing so via a doctor’soffice visit. estimated that up to 800,000 of the eligible members e cost of screening and brief intervention (using solely served by the SCPMG may meet this criterion.1,31 For the ATTAIN) is estimated at 10 cents/person (US dollars). e HEDIS 2018 measurement year, health plan-reported cost of screening and brief intervention in the traditional data showed that 43% of eligible members had been primary care office can be estimated based on the following screened, 5% screened positive for unhealthy drinking assumptions and calculations:[(Cost per person assuming behaviors, and 27% of those who screened positive received 1 minute of nurse time to screen and $60/h cost per nurse) documented follow-up.32 (No. of members screened in the office in 2019)] + [(Cost per person assuming 2 minutes of physician time and Strategic Business Plan $180/h cost per clinician) (No. of members who screened Stakeholder Analysis positive in 2019 and would require further screening by the e key external stakeholders were identified as the in- physician)].33 dependent accrediting body (NCQA), the patients who If ATTAIN is successful as intended, it may help reduce would be screened, primary care providers, and addiction the deleterious health consequences of alcohol use disorders medicine providers. ey are listed in Figure 1 and their by helping at-risk people reduce or stop their alcohol intake strategic importance is highlighted in Table 1. before they develop consequences. is may lead to further Internal Analysis and Critical Success Factors cost savings to the SCPMG. Competitive advantages included the SCPMG’sin- Positioning tegrated model of care, EMRs, and patient portal, which e most important target audience of ATTAIN was the together made ATTAIN possible. e necessary plan- SCPMG, followed by the NCQA. Cost is a critical marketing ning steps were 1) to make sure ATTAIN meets regu- factor because ATTAIN is expected to save money for the latory needs; 2) to design the patient interface in a clear, SCPMG compared to its alternative. Positioning messages

The Permanente Journal·https://doi.org/10.7812/TPP/20.141 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 133 ORIGINAL RESEARCH ARTICLE Unhealthy Drinking Behavior and the ATTAIN Solution: Web-based Automated Alcohol Misuse Interventions

included cost savings, measurable, convenient, and easy to implement. ATTAIN as a concept was presented to and

liates receivedfavorablybytheSCPMG.Tofacilitatethe fi SCPMG’s success with ATTAIN and to disseminate beneficial knowledge, the SCPMG entered into a 3-year, nation-wide collaborative effort with the NCQA, with the aim of improving unhealthy drinking behavior screening Cost-effective ways to meet quality measures Demonstration of support by informing regulators May implement this planworks if at it their HMO plan Offer services of similar quality as other regions and follow-up. is has created a broader audience for ATTAIN as it proceeds through its design, pilot, imple- mentation, and possible dissemination. Risks and Mitigation e top 4 risks were identified as outcome failure (in- ffi

providers Health plan af feriority to o ce-based screening rates), failure to address critical patient responses (eg, patients found to have alcohol Addiction medicine use disorder during ATTAIN screening not getting the Reduction in unnecessary work, enhanced effectiveness Readiness to treat thoseneed who their care their schedule use disorders proper follow-up), regulatory failure (failure to get HEDIS

tee for Quality Assurance (independent accrediting body). credit for patients screened via ATTAIN), and patient complaints (about the ATTAIN process). Mitigation s patients Care for patients with alcohol ’ strategies for all risks were put in place, mainly via close collaboration with the key stakeholders. ATTAIN was presented to patient representative focus groups that gave helpfulfeedbacktoaddresspotentialconcernsofpatient privacy. A patient representative was also added as an Reduction in unnecessary work, enhanced effectiveness Awareness and support ofprogram the if asked about it May be spared extra work May have extra work added to Care for the HMO integral part of our team to the NCQA collaborative.

ATTAIN Development After receiving SCPMG executive leadership approval, our team worked with our EMR/information technology development team in concert with SCPMG leaders from addiction medicine and health education, as well as patient representatives. A branching questionnaire was developed health plans thanks to mandates their measures If they approve of ATTAIN, they may need to approveothers it for Give health plan aaccreditation score based and on quality following DSM-V criteria and using screening methods accepted by the National Institute on Alcohol Abuse and Alcoholism and US Preventive Services Task Force. Eli- gible members (18 years or older) would be asked the following two questions to screen for unhealthy drinking behavior (a diagram showing what qualifies as a “drink” is included as a reference for members): “ Better health for their workers Improved health in participating Will appreciate more services being offered to their workers their workers 1. In the past year, how often have you had 4 or more [all women and men 65 years or older] or 5 or more [men < 65 years old] drinks in a 24-hour period?” 2. “In the past year, how many drinks do you have in a typical week?” A response of “never” to the first question and a response t from treatment or fi “ ” “ ” Public sector Private sector External agencies Primary care providers of 7orless (all women and men > 64 years old) or 14 or less (men < 65 years old) to the second question is a negative screen “ ” Very highHigh Medium Low Very high High Medium Medium Medium Medium Low Low price Compliance with the screening Continued use of this plan Approval of ATTAIN to meet may be annoyed byscreening the May bene and ends the questionnaire. A response of once or more to the first question and a response of “8 or more” (all women and men > 64 years old) or “15 or more” (men < 65 years old) to the second question is considered a positive screen. Positive screens automatically populate 12 additional Overall strategic importance Necessity of involvement What they want Altruistic care at an affordable Potential contribution Table 1. Stakeholder analysis for Automated Alcohol Misuse Interventions ATTAIN = Automated Alcohol Misuse Interventions; CMS = Centers for Medicare Services; HMO = Health Maintenance Organization; NCQA = National Commit Who they are HealthRole plan membersHow they will be affected Employer groups Patients receive the screening Contract with health plan for NCQA and CMS Contracted providers Contracted providersquestions Other regions of the health to screen for alcohol use disorders, as well as

134 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.141 ORIGINAL RESEARCH ARTICLE Unhealthy Drinking Behavior and the ATTAIN Solution: Web-based Automated Alcohol Misuse Interventions

Table 2. Screening questions for alcohol use disorders, readiness to change, and interest in further help During the past year how often . . . 1. Have you drunk more than you planned to drink? 2. Have you tried cutting down on your drinking but found it difficult to do so? 3. Have you found yourself spending a lot of time drinking or recovering from hangovers? 4. Had cravings or urges to drink? 5. Has your drinking affected your ability to work, attend school, or take care of your responsibilities at home? 6. Has drinking caused any conflicts between you and your family members or friends? 7. Have you noticed that you are not participating in activities that you previously enjoyed because of your drinking? 8. Have you driven any vehicle or boat, operated heavy machinery, or piloted a plane while “buzzed?” 9. Have you or someone else been injured because of your drinking? 10. Have you found that you keep drinking even though it is harming your physical health and/or mental well-being? 11. Have you found you need to drink greater amounts of alcohol to get the same effect as before? 12. Have you felt sick, shaky, or confused after you stopped drinking? A. How interested are you in cutting back on alcohol consumption? (Choose one of the three following options): “I am interested to know about what you can offer me to help me cut back” (questions b and c populate) “I am not currently interested in cutting back” (questionnaire ends) “I am working on cutting back and would like to do this completely on my own” (questionnaire ends) B. Would you be interested to learn more about medications that can help reduce your desire to drink alcohol? (Yes or No) C. Would you be interested in meeting with a specialist to discuss how you can reduce your alcohol use? (Yes or No) Scoring for alcohol use disorders (range, 0–12) based on questions 1 through 12 0–1: No use disorder 2–3: Mild use disorder 4–5: Moderate use disorder 6 or greater: Severe use disorder All participants who screen positive for unhealthy drinking are asked questions 1 through 12 and question A. Questions B and C depend on the response to question A. Automated Alcohol Misuse Interventions provides a score automatically for questions 1 through 12. For questions 1 through 12, answers are “Once or more” (1 point) or “Never” (0 point).

questions assessing readiness to change, interest in medi- account. Upon successful login, they see the message (and cation, and interest in meeting with a specialist to assist in ashortvideo)describingthe purpose of ATTAIN and alcohol consumption reduction. ATTAIN scores patient asking them to complete the questionnaire. Patient re- responses based on DSM-V criteria for mild, moderate, sponses become part of their EMR and are then sent to a and severe use disorders (Table 2). All responses would go designated provider’s in-basket (a unique EMR inbox for to designated providers for further action as needed. each provider). e implementation of the system was supplemented Based on their responses, patients receive appro- through partnership with key stakeholders—the Primary priatemessagesaswellasaninvitationtoviewavideo Care Department, Addiction Medicine Department, and aboutthespectrumofalcoholuse.Systemanalyticsre- HEDIS-related accreditors—to ensure that the system cord how many individuals click the initial questionnaire meets appropriate standards and that affected physicians link, complete the screening questionnaire, and open and would be aware and engaged in these changes. watch the video. A repeat screening questionnaire can be sent later at a designated time interval to assess for a Patient Portal–EMR Interface change in drinking patterns. If the individuals screen ATTAIN works as follows: eligible health plan members negative for unhealthy drinking behaviors, a reaffirming are identified via the health plan based on age and an active message is sent. ose who screen positive for alcohol use account with the health plan’s internet-based, EMR patient disorders get direct follow-up from a designated provider portal (www.kp.org). An invitation email with the attached team (Figure 2). ATTAIN questionnaire link is sent to these member ac- Safety nets will be built into the system to attempt to counts. Members initially receive an email alert from the catch individuals who do not progress through the algo- health plan, informing them of a care-related message on rithm. Reminders will be sent at set intervals, and there will their account. ey then follow the link in this initial email, be an appointed designated provider to oversee the process which takes them to their portal, where they log in to their and monitor for gaps in the protocol.

The Permanente Journal·https://doi.org/10.7812/TPP/20.141 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 135 ORIGINAL RESEARCH ARTICLE Unhealthy Drinking Behavior and the ATTAIN Solution: Web-based Automated Alcohol Misuse Interventions

when needed throughout the process, while reducing provider burden overall. In an era when physician burnout is on the rise, this in- tervention has the possibility of offloading unhealthy drinking behavior screening from primary care providers who are tasked with a growing list of quality measures to meet at each health- care visit. e system may also see a decrease in costs through freeing up provider time, and through the appropriate screening and follow-up for unhealthy drinking behaviors and alcohol use disorders. Our work has the potential to guide momentum toward computerized screening and follow-up methods for other medical conditions and health-related behaviors.

Limitations Because ATTAIN was designed as a QI tool to be implemented in a real-life setting, there is no plan for randomization of members during testing. e QI team will be able to compare before-and-after results of AT- TAIN on the broader member population. In addition, it is important to note that ATAIN currently excludes the pediatric population (younger than 18 years) as they do not have direct, unsupervised access to a patient portal. ere may also be varied results based on the time of year, resulting from seasonal variation in alcohol use.34 Other policy changes and initiatives, such as promotion from “Mothers Against Drunk Driving,” may influence the results of this proposed intervention.35 Although the Figure 2. Automated Alcohol Misuse Interventions workflow. majority of the SCPMG’smemberpopulationprefers English, 10% prefer Spanish. ere may be barriers for Spanish-speaking members to participate in the screening Pilot Testing and follow-up, even if screening questions and brief inter- ATTAIN was tested and refined by a series of quality vention materials are offered in Spanish. As of this writing, improvement (QI) pilots, starting with a small number of the video describing ATTAIN, and the counseling video on members, with a plan to expand eventually to a region-wide the spectrum of alcohol use, do have Spanish versions. e application. As of this writing, the ATTAIN build and the ATTAIN branching questionnaire has not been translated first 3 QI pilots have been completed. eir results have into Spanish as of yet, but should be by late 2020. ose who been published in a separate article.36 are neither English nor Spanish speaking may not be able to participate, given that additional language options have yet to DISCUSSION be offered. In addition, individuals may choose not to click Past studies have not explored the implementation of a the link for screening, and—even if they do—may opt not to fully automated system for unhealthy drinking behavior fill out the questionnaire. If individuals complete the ques- screening and follow-up (brief intervention) in an inte- tionnaire and receive brief intervention materials (eg, the link grated, managed-care organization. is lack has paved the to the video), they may not fully engage with the material, way for implementing ATTAIN, an automated tool for not even if they do open it. Furthermore, this may be a sensitive only identifying, screening, and counseling individuals for topic to which individuals may not want to respond via a unhealthy drinking behaviors using brief intervention, but computer-based questionnaire, although previous studies also for identifying and screening for alcohol use disorders. have shown promise. It is also not known whether ATTAIN We aim to navigate ATTAIN through the real world of would work just as well if sent to members at random time health care, rather than a controlled research environ- points vs prior to upcoming appointments. ment. Anticipated challenges include patient accep- In the future, there may be options for better integrating tance, completion of both screening and follow-up when the screening into the EMR through the patient portal—for required, and ensuring enough provider intervention example, by including the questions in a social history tab

136 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.141 ORIGINAL RESEARCH ARTICLE Unhealthy Drinking Behavior and the ATTAIN Solution: Web-based Automated Alcohol Misuse Interventions

that automatically inputs into patients’ charts. Another con- 14. Bewick BM, Trusler K, Barkham M, Hill AJ, Cahill J, Mulhern B. The effectiveness of web- based interventions designed to decrease alcohol consumption: A systematic review. sideration is to send the alcohol screening questions packaged Prev Med 2008 Jul;47(1):17–26. DOI: https://doi.org/10.1016/j.ypmed.2008.01.005, in with other questionnaires (eg, drugs, domestic violence, and PMID:18302970. v 15. Rooke S, Thorsteinsson E, Karpin A, Copeland J, Allsop D. Computer-delivered so on), instead of as a stand-alone questionnaire. interventions for alcohol and tobacco use: A meta-analysis. Addiction 2010 Aug;105(8): 1381–90. DOI: https://doi.org/10.1111/j.1360-0443.2010.02975.x, PMID:20528806. Disclaimer 16. Kaner EF, Beyer FR, Garnett C, et al. Personalised digital interventions for reducing This clinical review was not submitted for institutional review board review and hazardous and harmful alcohol consumption in community-dwelling populations. does not involve human subjects. Cochrane Database Syst Rev 2017 Sep;9(9):CD011479. DOI: https://doi.org/10.1002/ 14651858.cd011479.pub2 Disclosure Statement 17. Guillemont J, Cogordan C, Nalpas B, Nguyen-Thanh V, Richard JB, Arwidson P. fl Effectiveness of a web-based intervention to reduce alcohol consumption among French The authors have no con icts of interest to disclose. hazardous drinkers: A randomized controlled trial. Health Educ Res 2017 Aug;32(4): 332–42. DOI: https://doi.org/10.1093/her/cyx052, PMID:28854571. Authors’ Contributions 18. Brendryen H, Johansen A, Duckert F, Nesvag˚ S. A pilot randomized controlled trial of an Mohamed H Ismail, MD, DrPH, participated in the intervention design, critical internet-based alcohol intervention in a workplace setting. Int J Behav Med 2017 Oct; review, drafting of the final manuscript, and submission of the final manuscript. 24(5):768–77. DOI: https://doi.org/10.1007/s12529-017-9665-0, PMID:28755326. Jennifer Chevinsky, MD, MPH, participated in the intervention design, and drafting 19. Baumann S, Gaertner B, Haberecht K, Bischof G, John U, Freyer-Adam J. Who benefits of and critical review of the final manuscript. Emma Fredua, MPH, CHES, and from computer-based brief alcohol intervention? Day-to-day drinking patterns as a moderator of intervention efficacy. Drug Alcohol Depend 2017 Jun;175:119–26. DOI: Ebonie M Vazquez, MD, participated in the intervention design and critical review of https://doi.org/10.1016/j.drugalcdep.2017.01.040, PMID:28412302. fi fi the nal manuscript. All authors have given nal approval to the manuscript. 20. Pedersen ER, Parast L, Marshall GN, Schell TL, Neighbors C. A randomized controlled trial of a web-based, personalized normative feedback alcohol intervention for young-adult Funding veterans. J Consult Clin Psychol 2017 May;85(5):459–70. DOI: https://doi.org/10.1037/ The authors did not receive funding for this study. ccp0000187, PMID:28287799. 21. Johansson M, Sinadinovic K, Hammarberg A, et al. Web-based self-help for problematic – Related Article alcohol use: A large naturalistic study. Int J Behav Med 2017 Oct;24(5):749 59. DOI: https://doi.org/10.1007/s12529-016-9618-z The ATTAIN Solution Tested: Initial Pilot Results of an Automated, Web-based 22. Tebb KP, Erenrich RK, Jasik CB, Berna MS, Lester JC, Ozer EM. Use of theory in Screening Tool for Unhealthy Drinking Behaviors computer-based interventions to reduce alcohol use among adolescents and young adults: A systematic review. BMC Publ Health 2016 Jun;16(1):517. DOI: https://doi.org/10. References 1186/s12889-016-3183-x, PMID:27317330. 1. National Institute on Alcohol Abuse and Alcoholism. Alcohol facts and statistics. National 23. Pedersen ER, Marshall GN, Schell TL. Study protocol for a web-based personalized Institute on Alcohol Abuse and Alcoholism. updated March 2021. Accessed February 6, normative feedback alcohol intervention for young adult veterans. Addict Sci Clin Pract 2018. www.niaaa.nih.gov 2016 Dec;11(1):6. DOI: https://doi.org/10.1186/s13722-016-0055-8 2. National Committee for Quality Assurance. HEDIS® measure: Unhealthy alcohol use 24. Steers ML, Coffman AD, Wickham RE, Bryan JL, Caraway L, Neighbors C. Evaluation of screening and follow-up. National Committee for Quality Assurance. Accessed March 9, alcohol-related personalized normative feedback with and without an injunctive message. 2021. www.ncqa.org J Stud Alcohol Drugs 2016 Mar;77(2):337–42. DOI: https://doi.org/10.15288/jsad.2016. 3. American Psychiatric Association. (2013). Diagnostic and statistical manual of mental 77.337, PMID:26997192. disorders (5th ed.). https://doi.org/10.1176/appi.books.9780890425596 25. Mullen J, Ryan SR, Mathias CW, Dougherty DM. Feasibility of a computer-assisted 4. Moyer VA. Preventive Services Task Force. Screening and behavioral counseling alcohol screening, brief intervention and referral to treatment program for DWI offenders. interventions in primary care to reduce alcohol misuse: U.S. preventive services task force Addict Sci Clin Pract 2015 Dec;10(1):25. DOI: https://doi.org/10.1186/s13722-015-0046-1 recommendation statement. Ann Intern Med 2013 Aug 6;159(3):210–8. page 210. DOI: 26. Merrill JE, Wardell JD, Read JP. Is readiness to change drinking related to reductions in https://doi.org/10.7326/0003-4819-159-3-201308060-00652, PMID:23698791. alcohol use and consequences? A week-to-week analysis. J Stud Alcohol Drugs 2015 Sep; – 5. National Institute on Alcohol Abuse and Alcoholism. Alcohol use disorder. National 76(5):790 8. DOI: https://doi.org/10.15288/jsad.2015.76.790, PMID:26402360. Institute on Alcohol Abuse and Alcoholism. Updated December 2020. Accessed March 9, 27. Lettow BV, Vries HD, Burdorf A, Boon B, Empelen PV. Drinker prototype alteration and 2021. www.niaaa.nih.gov cue reminders as strategies in a tailored web-based intervention reducing adults’ alcohol 6. Freyer-Adam J, Baumann S, Haberecht K, et al. In-person alcohol counseling versus consumption: Randomized controlled trial. J Med Internet Res 2015 Feb;17(2):e35. DOI: computer-generated feedback: Results from a randomized controlled trial. Health Psychol https://doi.org/10.2196/jmir.3551 2018 Jan;37(1):70–80. DOI: https://doi.org/10.1037/hea0000556, PMID:28967769. 28. Sinadinovic K, Wennberg P, Johansson M, Berman AH. Targeting individuals with – 7. Baumann S, Gaertner B, Haberecht K, Bischof G, John U, Freyer-Adam J, How alcohol problematic alcohol use via web-based cognitive behavioral self-help modules, personalized use problem severity affects the outcome of brief intervention delivered in-person versus screening feedback or assessment only: A randomized controlled trial. Eur Addict Res 2014 – through computer-generated feedback letters. Drug Alcohol Depend 2018 Feb;183:82–8. Oct;20(6):305 18. DOI: https://doi.org/10.1159/000362406, PMID:25300885. DOI: https://doi.org/10.1016/j.drugalcdep.2017.10.032, PMID:29241105. 29. Johnson NA, Kypri K, Attia J. Development of an electronic alcohol screening and brief 8. Cadigan JM, Haeny AM, Martens MP, Weaver CC, Takamatsu SK, Arterberry BJ. intervention program for hospital outpatients with unhealthy alcohol use. JMIR Res Protoc Personalized drinking feedback: A meta-analysis of in-person versus computer-delivered 2013 Sep;2(2):e36. DOI: https://doi.org/10.2196/resprot.2697, PMID:24055787. interventions. J Consult Clin Psychol 2015 Apr;83(2):430–7. DOI: https://doi.org/10.1037/ 30. Schulz DN, Candel MJ, Kremers SP, Reinwand DA, Jander A, de Vries H. Effects of a a0038394, PMID:25486373. web-based tailored intervention to reduce alcohol consumption in adults: Randomized 9. Carey KB, Scott-Sheldon LA, Elliott JC, Garey L, Carey MP. Face-to-face versus controlled trial. J Med Internet Res 2013 Sep;15(9):e206. DOI: https://doi.org/10.2196/ computer-delivered alcohol interventions for college drinkers: A meta-analytic review, jmir.2568, PMID:24045005. 1998 to 2010. Clin Psychol Rev 2012 Dec;32(8):690–703. DOI: https://doi.org/10.1016/j. 31. Department of Research & Evaluation. About us. Fast Facts. Kaiser Permanente Southern cpr.2012.08.001, PMID:23022767. California. Accessed on March 9, 2021 from https://www.kp-scalresearch.org/aboutus/fast-facts/. 10. Freyer-Adam J, Baumann S, Haberecht K, et al. In-person and computer-based alcohol 32. Data Consulting and Report Production Unit. ASF HEDIS® 2018-2019 Final Rates (2020) interventions at general hospitals: Reach and retention. Eur J Public Health 2016 Oct; SCPMG Clinical Analysis. Internal Report, unpublished 26(5):844–9. DOI: https://doi.org/10.1093/eurpub/ckv238, PMID:26748101. 33. Bray JW, Zarkin GA, Hinde JM, Mills MJ. Costs of alcohol screening and brief intervention 11. Khadjesari Z, Murray E, Hewitt C, Hartley S, Godfrey C. Can stand-alone computer-based in medical settings: A review of the literature. J Stud Alcohol Drugs 2012 Nov;73(6):911–9. interventions reduce alcohol consumption? A systematic review. Addiction 2011 Feb; DOI: https://doi.org/10.15288/jsad.2012.73.911, PMID:23036208. 106(2):267–82. DOI: https://doi.org/10.1111/j.1360-0443.2010.03214.x, PMID:21083832. 34. Uitenbroek DG. Seasonal variation in alcohol use. J Stud Alcohol 1996 Jan;57(1):47–52. 12. Nair NK Newton NC, Shakeshaft A Wallace P, Teesson M. A systematic review of digital DOI: https://doi.org/10.15288/jsa.1996.57.47, PMID:8747501. and computer-based alcohol intervention programs in primary care. Curr Drug Abuse Rev 2015 35. Fell JC, Voas RB. Mothers Against Drunk Driving (MADD): The first 25 years. Traffic Inj Prev Sep;8(2):111–8. DOI: https://doi.org/10.2174/1874473708666150916113538, PMID:26373848. 2006 Sep;7(3):195–212. DOI: https://doi.org/10.1080/15389580600727705, PMID:16990233. 13. Sundstrom¨ C, Blankers M, Khadjesari Z. Computer-based interventions for problematic 36. Yoon J, Fredua E, Davari SB, Ismail MH. The ATTAIN Solution Tested: Initial Pilot Results alcohol use: A review of systematic reviews. Int J Behav Med 2017 Oct;24(5):646–58. of an Automated, Web-based Screening Tool for Unhealthy Drinking Behaviors. Perm J DOI: https://doi.org/10.1007/s12529-016-9601-8 2021;25:20.143. https://doi.org/10.7812/TPP/20.143.

The Permanente Journal·https://doi.org/10.7812/TPP/20.141 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 137 n REVIEW ARTICLE The Role of Cannabidiol in Neurological Disorders

Sirichai Chayasirisobhon, MD, FAAN1 Perm J 2021;25:20.156 E-pub: 03/01/2021 https://doi.org/10.7812/TPP/20.156

ABSTRACT including N-arachidonoyldopamine, N-arachidonoylglycer- Cannabis has been used for both recreational and medicinal olether, and O-arachidonoylethanolamine were identified purposes for more than 4 millennia. Cannabis has been studied in later on.9 various medical disorders including neurological disorders. There Endogenous ligands for CB1 and cannabinoid receptor are well-known risks of long-term use of cannabis, including low type 2 receptors in humans were named endocannabinoids.10 motivation, lowered cognitive capabilities, and diminished IQ CB1 receptors are found primarily in the brain and peripheral and brain mass. Cannabinoids are compounds in cannabis that nervous system.11 CB1 receptors in the brain are particularly are known to have therapeutic potential. The most abundant chemicals in cannabinoids are delta-9-tetrahydrocannabinol and concentrated in anatomic regions associated with anxiety, cannabidiol. Delta-9-tetrahydrocannabinol has psychotropic ef- cognition, endocrine function, memory, motor coordination, 12,13 fects that limits its use as a pharmacotherapeutic agent. Can- pain sensory perception, and reward. Cannabinoid re- nabidiol is a nonpsychotropic chemical and therefore has become ceptor type 2 receptors, which are mainly found in im- a compound of interest for clinical researchers to study its mune and hematopoietic cells,14 may be playing a role in therapeutic potential. This article reviews the efficacy and safety the immune-suppressive actions of cannabinoids.14 Endo- of cannabidiol in various neurological disorders in humans. cannabinoids are therefore potential therapeutic targets for various pathological conditions, particularly neurological INTRODUCTION disorders.15  Cannabis has been used for both recreational and me- ere are well-known risks of long-term use of cannabis, dicinal purposes for more than 4 millennia.1,2 Cannabis is including low motivation, lowered cognitive capabilities, made up of more than 100 compounds called cannabinoids, and diminished IQ and brain mass. THC can cause psy- ff the most abundant chemicals of which are delta-9-tetra- chotropic e ects, including red eyes, poor muscle coor- hydrocannabinol (THC) and cannabidiol (CBD). e dination, delayed reaction time, increased appetite, and  discovery of THC and CBD led to the identification of sudden mood changes. erefore, its therapeutic po- the cannabinoid receptors. e evidence that THC was tentials are limited. CBD does not have any psychotropic 16 interacting with a specific mammalian target was uncovered properties. in murine neuroblastoma cells that expressed upregulated Cannabis and cannabinoids have been shown to have adenylate cyclase in response to exposure to THC or its a positive impact on a variety of neurological disorders 17-19 20,21 synthetic analogues. is finding led the way for the iso- in humans, including neuropathic pain, migraine, 22 ’ 23,24 ’ lation and cloning of a G protein-coupled receptor that multiple sclerosis, Parkinson s disease, Huntington s 25 26 subsequently was named cannabinoid receptor type 1 (CB1).3 disease, and motor neuron disease.  ffi Later, cannabinoid receptor type 2 was isolated from human is review article discusses the safety and e cacy of fi leukemia cells.4 eidentification of these receptors led to the highly puri ed CBD and CBD-enriched cannabis (CBD: hypothesis that an endocannabinoid system may exist in THC ratio of 20:1). Extensive literature search on fi the mammalian body. e first endogenous cannabinoid PubMed and other scienti c internet platforms was done ligand was isolated from pig brain and was named to prepare this article. N-arachidonoylethanolamide, or anandamide.5 esecond endogenous ligand, which was named 2-arachidonoylglycerol, PHARMACOLOGY AND THERAPEUTIC ASPECTS IN HUMAN was isolated from intestinal tissue.6,7 Both these ligands are Epilepsy arachidonic acid derivatives produced from phospholipid Both animal and human studies have proved the anti- 27-29 precursors through activity-dependent activation of specific convulsant properties of CBD. Several studies have phospholipase enzymes.8 Several other endogenous ligands, been conducted recently using CBD as an adjunctive therapy to assess its safety and efficacy in patients with both focal epilepsy and generalized epilepsy.30-33 A study in 2018 evaluated the efficacy of CBD-enriched Author Affiliation 1Department of Neurology, Kaiser Permanente Medical Center, Orange County, CA cannabis oil extract (CBD:THC ratio of 20:1) for the treatment of drug-resistant epilepsy. Fifty-seven patients Corresponding Author (age, 1–20 years) with drug-resistant epilepsy of various Sirichai Chayasirisobhon, MD, FAAN ([email protected]) etiologies were treated with CBD-enriched cannabis oil Keywords: cannabidiol, cannabinoids, cannabis, CBD, delta-9-tetrahydrocannabinol, THC 138 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.156 REVIEW ARTICLE The Role of Cannabidiol in Neurological Disorders

Table 1. Long-term efficacy and safety of cannabidiol in patients with drug-resistant epilepsy Mean age Median duration Mean dose Efficacya Disease/syndrome (y) (mo) (mg/kg/d) (%) Common adverse events Seizures associated with various etiologies 9.6 (1–20) 18 11b 56 Somnolence, aggression, loss of appetite (57 patients)30 Seizures associated with Dravet syndrome, Lennox 13 (0.4–62) 24 25c 52 Diarrhea, somnolence Gastaut syndrome, tuberous sclerosis complex, and others (607 patients)35 Seizures associated with tuberous sclerosis 13 (1–57) 8.9 27c 61 Diarrhea, seizure, loss of appetite complex (199 patients)36 Seizure associated with mostly genetic epilepsies 9(1–17) 36 25c 26.9 Loss of appetite, diarrhea, weight loss (26 patients)37 aEfficacy, ≥ 50% reduction. bCannabidiol:delta-9-tetrahydrocannabinol, 20:1. cHighly purified cannabidiol.

extract for at least 3 months, with median follow-up of A total of 146 patients (24%) withdrew; the most common 18 months. Forty-six patients were included in the efficacy reasons being lack of efficacy in 89 (15%) and AEs in 32 analysis. Average CBD dose was 11.4 mg/kg/day. Twenty- (5%). e most common AEs were diarrhea (29%) and six patients (56%) had ≥ 50% reduction in mean monthly somnolence (22%). e median number of concomitant seizure frequency. ere was no statistically significant antiepileptic drugs was 3 (range, 0–10), the median dose difference in response rate among various epilepsy etiologies was 25 mg/kg/day, and the median treatment duration was and cannabis strain used. Younger age at treatment onset 48 weeks. Add-on highly purified CBD oral solution re- (< 10 years) and higher CBD dose (> 11 mg/kg/day) were duced the median monthly convulsive seizures by 51% and associated with a better treatment response. Adverse re- total seizures by 48% at 12 weeks, and reductions were actions were reported in 28 patients (46%) and were the similar through 96 weeks. e proportion of patients with ≥ main reason for treatment cessation. e common adverse 50%, ≥ 75%, and 100% reductions in convulsive seizures events (AEs) were somnolence (14%), aggression (9%), loss were 52%, 31%, and 11%, respectively, at 12 weeks, with of appetite (9%), and vomiting (9%). e results suggest similar rates through 96 weeks. e retention rate of 76% that adding CBD-enriched cannabis oil extract to the reflects maintenance of long-term efficacy, generally mild treatment regimen of patients with drug-resistant epilepsy AEs, and improvements in quality of life with CBD.35 may result in a significant reduction in seizure frequency.30 Long-term safety and efficacy of highly purified CBD as GW Pharmaceutical/Greenwich Biosciences conducted an adjunctive treatment of seizures in patients with tuberous randomized, double-blind, placebo-controlled studies to sclerosis complex was evaluated in an open-label extension evaluate the efficacy of highly purified CBD oral solution trial study. A total of 199 patients participated, with a mean add-on therapy in Dravet syndrome and Lennox-Gastaut age of 13 years (range, 1–57 years). e study period was syndrome that showed that it was efficacious for seizures 267 days (range, 18–910 days). irty-nine patients (20%) associated with both these syndromes.31-33 On June 25, withdrew. e mean CBD dose was 27 mg/kg/day. AE 2018, the US Food and Drug Administration approved incidence was 93%, with a serious AE incidence of 15%, Epidiolex, a highly purified CBD oral solution, for the with 6% discontinuing CBD because of AEs. e most treatment of seizures associated with Lennox-Gastaut common AEs (≥ 20%) were diarrhea (42%), seizure syndrome and Dravet syndrome in patients 2 years of (22%), and decreased appetite (20%). Elevated alanine age and older.34 is led to a long-term, expanded-access aminotransferase/aspartate aminotransferase levels were safety and efficacy study of highly purified CBD oral so- reported in 17 (8.5%) patients. Median percentage re- lution in children and adults with drug-resistant epilepsies ductions in seizure frequency (12-week windows over (Table 1).35-37 at study enrolled 607 patients with drug- 48 weeks) were 54% to 68%. Seizure responder rates resistant epilepsies from 25 US-based sites and included (≥ 50%, ≥ 75%, and 100% reduction) ranging from 53% to patients with Lennox-Gastaut syndrome, Dravet syn- 61%, 29% to 45%, and 6% to 11%, respectively, across drome, tuberous sclerosis complex, Aicardi syndrome, 12- week visit windows were observed. Improvement in the CDKL5 deficiency disorder, Doose syndrome, and others. subject/caregiver global impression of change was reported Mean age was 13 years (range, 0.4–62). Patients received by 87% of patients/caregivers at week 26, with 53% highly purified CBD oral solution starting at 2 to 10 mg/kg/ reporting much/very much improvement. It was conclu- day, titrated to a maximum dose of 25 to 50 mg/kg/day. ded that long-term adjunctive CBD treatment was well

The Permanente Journal·https://doi.org/10.7812/TPP/20.156 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 139 REVIEW ARTICLE The Role of Cannabidiol in Neurological Disorders

tolerated in patients with tuberous sclerosis complex, with A recent study conducted a systemic chart review of an AE profile similar to that observed previously. Reduc- pediatric patients who started highly purified CBD from tions in seizures were maintained through 48 weeks, with a January to August 2019. Among 87 patients, 9 (10%) high proportion of patients reporting global improvement.36 developed thrombocytopenia. All patients who developed On June 31, 2020, the US Food and Drug Administration thrombocytopenia were taking valproic acid. No children on approved Epidiolex (highly purified CBD) oral solution for highly purified CBD without valproic acid (0 of 57) de- treatment of seizures associated with tuberous sclerosis veloped thrombocytopenia (P < 0.0001).40 Another recent complex in patients 1 year of age and older. study evaluated food effects on the pharmacokinetics of Another study assessed the long-term safety, tolerability, purified CBD oral capsules in adult patients with drug- and efficacy of highly purified CBD oral solution in children resistant epilepsy. A moderate dose of purified CBD ad- with drug-resistant epilepsy. Highly purified CBD oral ministered with a fatty meal resulted in a 4-fold increase in solution was administered in addition to other antiepileptic Cmax and a 14-fold increase in AUC0-∞. A steady-state treatments in 26 patients, with a starting dose of 5 mg/kg/ concentration from a 300-mg dose of purified CBD, on day and weekly increment increases of 5 mg/kg/day up to a average, was 21.3 µg/mL. e authors concluded that the  fi maximum dose of 25 mg/kg/day. e mean age was 9 years fat content of a meal can lead to signi cant increases in Cmax –  (range, 1 17 years). e duration of therapy ranged from 4 and AUC0-∞, and can account for variability and overall to 53 months (mean, 21 months). e frequency of seizures drug exposure within patients. erefore, patients should be and AEs was monitored during the study period. Fifteen of advised to take CBD with balanced meals to minimize 26 patients discontinued treatment, 2 patients because of fluctuations resulting from food effects.41 serious AEs, and 13 patients because of lack of efficacy. In conclusion. highly purified CBD oral solution is well Eleven patients completed the study. A reduction in the tolerated even at high doses and is another promising drug frequency of seizures of > 50% was noted in 38.4% of patients as an add-on therapy in the treatment of drug-resistant after 3 months of treatment, in 56.7% after 6 months, in epilepsy. 42.3% after 9 months, in 38.4% after 12 months, in 42.3% e overall limitation of the treatment of epilepsy with after 18 months, and in 34.6% after 24 months. ere- CBD may be that the number of patients exposed to long- sponder rates subsequently declined to 26.9% by 36 months term CBD is still low and the rate of AEs over a long period and remained stable through the last follow-up (48 months), of time is not known. In addition, there is no standardi- including 3 patients (11.5%) who remained seizure free. Of zation in the dose and purity of non-Food and Drug 26 patients, 21 (80.8%) reported AEs, among which the Administration-approved CBD. most frequent were reduced appetite (n = 10), diarrhea (n = 9), weight loss (n = 9), status epilepticus (n = 3), catatonia Parkinson’s Disease (n = 2), and hypoalbuminemia (n = 1). elimitationofthis Limited studies suggest that CBD may be used in the study was that the subject dropout rate after a few months treatment of Parkinson’s disease. An open-label pilot study into the study was high, the number of patients exposed to evaluated the efficacy, tolerability, and safety of CBD in highly purified CBD for a long time was low, and therefore patients with Parkinson’s disease with psychotic symptoms the rate of AEs over time may have been underestimated.37 for at least 3 months. Six patients (4 men and 2 women) Because the use of highly purified CBD oral solution for received CBD tablets (approximately 99.9% pure) in a the treatment of drug-resistant epilepsies is increasing, flexible dose, starting with an oral dose of 150 mg/day for a recent study monitored drug interaction between CBD 4 weeks, in addition to their usual therapy. e patients and other antiepileptic drugs. CBD significantly raised were assessed by the Brief Psychiatric Rating Scale, the serum levels of desmethylclobazam (active metabolite of Parkinson Psychosis Questionnaire, and the Unified Par- clobazam), (p = <0.001), eslicarbazepine (p = 0.039), kinson’s Disease Rating Scale (UPDRS). e psychotic rufinamide (p = 0.004), topiramate (p = <0.001), and symptoms evaluated by the Brief Psychiatric Rating Scale zonisamide (p = 0.017). CBD had no significant inter- and the Parkinson Psychosis Questionnaire showed a action with lacosamide, levetiracetam, perampanel, and significant decrease with CBD treatment. CBD did not valproate. Alanine aminotransferase/aspartate aminotrans- worsen motor function and it decreased the total UPDRS ferase levels were significantly greater in participants taking scores. No AEs were observed during the treatment.42 concomitant valproate.38 Another study showed an inter- In another study, CBD was studied in an exploratory action between CBD and brivaracetam, resulting in an double-blind trial in 21 patients with Parkinson’s disease increasing level of brivaracetam.39 One possible mechanism without dementia or comorbid psychiatric conditions. CBD contributing to the increasing drug level is the inhibition of was provided in powdered form (purity, 99%) and was the cytochrome P450 system by CBD. placed in gelatin capsules containing either 75 mg or

140 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.156 REVIEW ARTICLE The Role of Cannabidiol in Neurological Disorders

300 mg. Participants were assigned to 3 groups of 7 Autism Spectrum Disorder subjects each who were treated with placebo, CBD 75 A retrospective study assessed tolerability and efficacy of mg/day, or CBD 300 mg/day. One week before the trial CBD-enriched cannabis in 60 children with autism spec- and during the last week of treatment, participants were trum disorder and severe behavioral problems (age, 11.8 ± assessed with respect to 1) motor and general symptoms 3.5 years; range, 5.0–17.5 years; 77% low functioning; 83% or UPDRS score, 2) well-being and quality of life boys). ese children were treated with an oral preparation (Parkinson’s Disease Questionnaire-39), and 3) possible of CBD and THC at a ratio of 20:1. e dose was the dose neuroprotective effects (brain-derived neurotrophic factor was titrated up to effect (maximal CBD dose, 10 mg/kg/ and proton magnetic resonance spectroscopy). ere were day). After the cannabis treatment, behavioral outbursts no statistically significant differences in UPDRS scores, were much improved or very much improved in 61% of plasma BDNF levels, or proton magnetic resonance patients, according to the Caregiver Global Impression of spectroscopic measures. However, the groups treated with Change. anxiety and communication problems were much or placebo and CBD 300 mg/day had significantly different very much improved in 39% and 47% of patients, respectively. mean total scores on the Parkinson’s Disease Question- Disruptive behaviors improved by 29% from 4.74 ± 1.82, as naire-39 (P =0.05).e findings point to a possible effect recorded at baseline on the Home Situations Questionnaire– of CBD in improving quality-of-life measures in patients Autism Spectrum Disorder, to 3.36 ± 1.56 after CBD with Parkinson’s disease.43 However, it is not conclusive treatment. Parents reported less stress as reflected in Autism whether CBD is effective in the treatment of neurological Parenting Stress Index scores, changing by 33%, from 2.04 ± symptoms of Parkinson’s disease because the number of 0.77 to 1.37 ± 0.59. AEs included sleep disturbances (14%), patients enrolled in the studies was too small. More irritability (9%), and loss of appetite (9%). is study supports clinical trials with larger patient populations are needed the feasibility of CBD and THC therapy in a ratio of 20:1 as a to assess the efficacy and safety of CBD in Parkinson’s promising treatment option for refractory behavioral prob- disease. lems in children with autism spectrum disorder.45

Huntington’s Disease Complex Motor Disorders Very few human clinical trials have investigated CBD in CBD was studied in children with complex motor dis- Huntington’s disease. A study assessed the symptomatic orders. Twenty-five children were enrolled and divided into efficacy and safety of CBD for Huntington’s disease. Fifteen 2 groups. Participants received a CBD:THC formulation patients were evaluated in the baseline–placebo–washout– at 20:1 or 6:1. Both groups showed improvement on the CBD posttreatment order, and 6 patients were evaluated in Cerebral Palsy Child Questionnaire for quality of life at the the baseline–CBD–washout–placebo posttreatment order. end of 5 months. ey also showed significant improvement e total daily dose of CBD (10 mg/kg) was divided into 4 with regard to spasticity and dystonia, sleep difficulties, and capsules, and patients took 2 capsules twice a day. e pain severity. Additional research studies with randomized, patients were evaluated weekly for 15 consecutive weeks. controlled trials are needed to assess more comprehensively the e outcomes of the efficacy were measured by the Marsden efficacy of CBD in children with complex motor disorders.46 and Quinn chorea severity evaluation scale, the Shoulson Currently there are no human studies that have inves- and Fahn functional disability scale for Huntington’s dis- tigated the effects of highly purified CBD or CBD-enriched ease, and 10 other variables. Safety was measured by a cannabis in neuropathic pain, migraine, multiple sclerosis, cannabis side effect inventory. e major therapeutic re- motor neuron disease, and Alzheimer’s disease. sponse variable was chorea severity as measured by the Marsden and Quinn chorea severity evaluation scale, which CONCLUSION showed a small, nonsignificant (P = 0.71) response. e In conclusion, studies on cannabis and cannabinoid effects of CBD and placebo treatments on the Shoulson and compounds have shown benefits in a variety of neurological Fahn disability score and 10 other variables also yielded no disorders in humans, but the therapeutic potentials were significant differences (all P values were > 0.05). e major limited because of the psychotropic effect of THC and the safety response variables measured by the clinical laboratory long-term potential AEs of cannabinoid compounds. CBD tests and the cannabis side effect inventory also showed no has been found to be a promising compound that appears to significant difference (P = 0.98; Mann–Whitney test).44 be safe and efficacious. A highly purified CBD oral solution e limitation of this clinical trial study was the small has shown efficacy and safety in Dravet syndrome, Lennox number of patients. No further clinical trials to assess the Gastaut syndrome, tuberous sclerosis complex, and other efficacy and safety of Huntington’s disease have been drug-resistant epilepsies. However, the clinical trials with published since. highly purified CBD and CBD-enriched cannabis in other

The Permanente Journal·https://doi.org/10.7812/TPP/20.156 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 141 REVIEW ARTICLE The Role of Cannabidiol in Neurological Disorders

neurological disorders are few and inconclusive because of 21. Russo EB. Clinical endocannabinoid deficiency reconsidered: Current research supports the theory in migraine, fibromyalgia, irritable bowel, and other treatment-resistant low participating patient populations. Research in the future syndromes. Cannabis Cannabinoid Res 2016 Jul;1(1):154–65. DOI: https://doi.org/10. should involve larger populations and greater doses of CBD. 1089/can.2016.000, PMID:28861491. 22. Torres- Moreno MC, Papseit E., Torren M, et al. Assessment of efficacy and tolerability of More studies are still needed to understand the full potential medicinal cannabinoids in patients with multiple sclerosis: A systematic review and meta- and long-term effects of CBD. v analysis. JAMA Netw Open 2018 Oct;1(6):e183485. DOI: https://doi.org/10.1001/ jamanetworkopen.2018.3485 23. Balash Y, Bar-Lev Schleider L, KorcZyn AD, et al. Medical cannabis in Parkinson disease: Acknowledgments Real-life patients’ experience. Clin Neuropharmacol 2017 Nov/Dec;40(6):268–72. DOI: The author thanks Suresh Gurbani, MD, PhD, for his assistance. https://doi.org/10.1097/WNF.0000000000000246, PMID:29059132. 24. Lotan I, Treves TA, Roditi Y, Djaldetti R. Cannabis (medical marijuana) treatment for motor Disclosure Statement and non-motor symptoms of Parkinson disease: An open-label observational study. Clin Neuropharmacol 2014 Mar–Apr;37(2):41–4. DOI: https://doi.org/10.1097/WNF. The author has no targeted funding reported. 0000000000000016, PMID:24614667. 25. Saft C, von Hein SM, Lucke¨ T, et al. Cannabinoids for treatment of dystonia in References Huntington’s disease. J Huntingtons Dis 2018 Jun;7(2):167–73. DOI: https://doi.org/10. 1. Russo EB. Cannabis and epilepsy: An ancient treatment returns to the fore. Epilepsy 3233/jhd-170283 Behav 2017 May;70(Pt B):292–7. DOI: https://doi.org/10.1016/j.yebeh.2016.09.040, 26. Riva N, Mora G, SoraruG,etal.Safetyandef` ficacy of nabiximols on spasticity PMID:27989385. symptoms in patients with motor neuron disease (CANALS): A multicentre, double- 2. Friedman D, Sirven JI. Historical perspective on the medical use of cannabis for epilepsy: blind, randomised, placebo-controlled, phase 2 trial. Lancet Neurol 2019 Feb;18(2): Ancient times to the 1980s. Epilepsy Behav 2017 May;70(Pt B):298–301. DOI: https://doi. 155–64. DOI: https://doi.org/10.1016/S1474-4422(18)30406-X, PMID:30554828. org/10.1016/j.yebeh.2016.11.033, PMID:28089286. 27. Devinsky O, Cilio MR, Cross H, et al. Cannabidiol: Pharmacology and potential 3. Matsuda LA, Lolait SJ, Brownstein MJ, Young AC, Bonner TI. Structure of a cannabinoid therapeutic role in epilepsy and other neuropsychiatric disorders. Epilepsia 2014 Jun; receptor and functional expression of the cloned cDNA. Nature 1990 Aug;346(6284): 55(6):791–802. DOI: https://doi.org/10.1111/epi.12631, PMID:24854329. 561–4. DOI: https://doi.org/10.1038/346561a0, PMID:2165569. 28. Naziroglu M. TRPV1 channel: A potential drug target for treating epilepsy. Curr – 4. Munro S, Thomas KL, Abu-Shaar M. Molecular characterization of a peripheral receptor Neuropharmacol 2015 May;12:239 47. DOI: https://doi.org/10.2174/ for cannabinoids. Nature 1993 Sep;365(6441):61–5. DOI: https://doi.org/10.1038/ 1570159x13666150216222543 ´ 365061a0, PMID:7689702. 29. Vilela LR, Lima IV, Kunsch EB, et al. Anticonvulsant effect of cannabidiol in the pentylenetetrazole model: Pharmacological mechanisms, electroencephalographic 5. Devane WA, Hanus L, Breuer A, et al. Isolation and structure of a brain constituent that fi – binds to the cannabinoid receptor. Science 1992 Dec;258(5090):1946–9. DOI: https://doi. pro le, and brain cytokine levels. Epilepsy Behav 2017 Oct;75:29 35. DOI: https://doi.org/ org/10.1126/science.1470919, PMID:1470919. 10.1016/j.yebeh.2017.07.014, PMID:28821005. fi 6. Stella N, Schweitzer P. Piomelli D. A second endogenous cannabinoid that modulates 30. Hausman-Kedem M, Menascu S, Kramer U, et al. Ef cacy of CBD-enriched medical long-term potentiation. Nature 1997 Aug;388(6644):773–8. DOI: https://doi.org/10.1038/ cannabis for treatment of refractory epilepsy in children and adolescents: An – 42015, PMID:9285589. observational, longitudinal study. Brain Dev 2018 Aug;40(7):544 51. DOI: https://doi.org/ 10.1016/j.braindev.2018.03.013, PMID:29674131. 7. Sugiura T, Kishimoto S, Oka S, Gokoh M. Biochemistry, pharmacology and physiology of 31. Devinsky O, Patel AD, Cross JH. et al. Effect of cannabidiol on drop seizures in the Lennox- 2-arachidonoylglycerol, an endogenous cannabinoid receptor ligand. Prog Lipid Res 2006 Gastaut Syndrome. N Eng J Med 2018 May;378:1888–97. DOI: https://doi.org/10.1056/ Sep;45(5):405–46. DOI: https://doi.org/10.1016/j.plipres.2006.03.003, PMID:16678907. NEJMoa1714631 8. Piomelli D. The molecular logic of endocannabinoid signalling. Nat Rev Neurosci 2003 32. Thiele EA, March ED, French JA, et al. Cannabidiol in patients with seizures associated Nov;4(11):873–84. DOI: https://doi.org/10.1038/nrn1247, PMID:14595399. with Lennox-Gastaut syndrome (GWPCARE4): A randomized, double-blind, placebo- 9. De Petrocellis L, Di Marzo V. An introduction to the endocannabinoid system: From the controlled phase 3 trial. Lancet 2018 Mar;391:1086–96. DOI: https://doi.org/10.1016/ – early to the latest concepts. Best Pract Res Clin Endocrinol Metabol 2009 Feb;23(1):1 15. s0140-6736(18)30136-3 DOI: https://doi.org/10.1016/j.beem.2008.10.013, PMID:19285257. 33. Devinsky O, Cross JH, Laux L, et al. Trial of cannabidiol for drug-resistant seizures in the 10. Di Marzo V, Fontana A. Anandamide, an endogenous cannabinomimetic Dravet syndrome. N Engl J Med 2017 May;376(21):2011–20.. DOI: https://doi.org/10. ‘ ’ eicosanoid: Killing two birds with one stone . Prostaglandins Leukot Essent Fatty 1056/NEJMoa1611618, PMID:28538134. – Acids 1995 Jul;53(1):1 11. DOI: https://doi.org/10.1016/0952-3278(95)90077-2, 34. Devinsky O, Nabbout R, Miller I, et al. Long-term cannabidiol treatment in patients with PMID:7675818. Dravet syndrome: An open-label extension trial. Epilepsia 2019 Feb;60(2):294–302. DOI: – 11. Iversen L Cannabis and the brain. Brain 2003 Jun;126(Pt 6):1252 70. DOI: https://doi.org/ https://doi.org/10.1111/ep. 14628 10.1093/brain/awg143, PMID:12764049. 35. Szaflarski JP, Bebin EM, Comi AM, et al. Long-term safety and treatment effects of 12. Adams IB, Martin BR. Cannabis: Pharmacology and toxicology in animals and humans, cannabidiol in children and adults with treatment-resistant epilepsies: Expanded access Addiction 1996 Nov;91(11):1585–614. DOI: https://doi.org/10.1046/j.1360-0443.1996. program results. Epilepsia 2018 Aug;59(8):1540–8. DOI: https://doi.org/10.1111/epi. 911115852.x, PMID:8972919. 14477, PMID:29998598. 13. Herkenham M, Lynn AB, Little MD, et al. Cannabinoid receptor localization in brain. 36. Thiele E, Bebin EM, Filloux F, et al. Long-term safety and efficacy of cannabidiol (CBD) for Proc Natl Acad Sci USA 1990 Mar;87(5):1932–6. DOI: https://doi.org/10.1073/pnas.87.5.1932 the treatment of seizures in patients with tuberous sclerosis complex (TSC) in an open-label 14. Gardner EL, Lowinson JH. Marijuana’s interaction with brain reward systems: Update extension (OLE) trial (GWPCARE6) (677). Neurology 2020 Apr;94(15 Suppl):677. DOI: 1991. Pharmacol Biochem Behav 1991 Nov;40(3):571–80. DOI: https://doi.org/10.1016/ https://n.neurology.org/content/94/15_Supplement/677.abstract 0091-3057(91)90365-9, PMID:1806947. 37. Sands TT, Rahdari S, Oldham MS, Caminha Nunes E, Tilton N, Cilio MR. Long-term 15. Fragnas-Sanchez AI, Torres-Suarez AI. Medical use of cannabinoids. Drugs 2018 Nov; safety, tolerability, and efficacy of cannabidiol in children with refractory epilepsy: Results 78:1665–703. DOI: https://doi.org/10.1007/s40265-018-0996-1 from an expanded access program in the US. CNS Drugs 2019 Jan;33(1):47–60. DOI: 16. Joy JE, Watson SJ, Benson JA Jr. Marijuana and medicine assessing the science base. https://doi.org/10.1007/s40263-018-0589-2, PMID:30460546 Washington, DC: National Academics Press; 1999. 38. Gaston TE, Bebin EM, Cutter GR, Liu Y, Szaflarski JP. Interactions between cannabidiol 17. Wilsey B, Marcotte T, Deutsch R, Gouaux B, Sakai S, Donaghe H. Low-dose vaporized and commonly used antiepileptic drugs. Epilepsia 2017 Sep;58(9):1586–92. DOI: https:// cannabis significantly improves neuropathic pain. J Pain 2013 Feb;14(2):136–48. DOI: doi.org/10.1111/epi.13852, PMID:28782097. https://doi.org/10.1016/j.jpain.2012.10.009, PMID:23237736. 39. Klotz KA, Hirsch M, Heers M, Schulze-Bonhage A, Jacobs J. Effects of cannabidiol on 18. Abrama DI, Jay CA, Shade SB, et al. Cannabis in painful HIV-associated sensory brivaracetam plasma levels. Epilepsia 2019 Jul;60(7):e74–7.DOI: https://doi.org/10.1111/ neuropathy: A randomized placebo-controlled trial. Neurology 2007 Feb;68:515–21. DOI: epi.16071, PMID:31211851. https://doi.org/10.1212/01.wnl.0000253187.66183.9c 40. McNamara NA, Dang LT, Sturza J, et al. Thrombocytopenia in pediatric patients on 19. Johnson JR, Burnell-Nugent M, Lossignol D, Ganae-Motan ED, Potts R, Fallon MT. concurrent cannabidiol and valproic acid. Epilepsia 2020 Aug;61(8):e85–9. DOI: https:// Multicenter, double-blind, randomized, placebo-controlled, parallel-group study of the doi.org/10.1111/epi.16596, PMID:32614070. efficacy, safety, and tolerability of THC:CBD extract and THC extract in patients with 41. Birnhaum AK, Karanam A, Marino SE, et al. Food effect on pharmacokinetics of intractable cancer-related pain. J Pain Symptom Manage 2010 Feb;39(2):167–79. DOI: cannabidiol oral capsules in adult patients with refractory epilepsy. Epilepsia 2019 Aug;60: https://doi.org/10.1016/j.jpainsymman.2009.06.008, PMID:19896326. 1586–92. DOI: https://doi.org/10.1111/epi.16093 20. Cuttler C, Spradlin A, Cleveland M, et al. Short- and long-term effects of cannabis on 42. Zuardi AW, Crippa JA, Hallak JE, et al. Cannabidiol for the treatment of psychosis in headache and migraine. J Pain 2020 May–Jun;21(5–6):720–30. DOI: https://doi.org/10. Parkinson’s disease. J Psychopharmacol 2009 Nov;23(8):979–83. DOI: https://doi.org/10. 1016/j.jpain.2019.11.001 1177/0269881108096519, PMID:18801821.

142 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.156 REVIEW ARTICLE The Role of Cannabidiol in Neurological Disorders

43. Chagas MH, Zuardi AW, Tumas V, et al. Effects of cannabidiol in the treatment 45. Aran A, Cassuto H, Lubotzky A, Wattad N, Hazan E. Brief report: Cannabidiol-rich of patients with Parkinson’s disease: An exploratory double-blind trial. cannabis in children with autism spectrum disorder and severe behavioral problems: A J Psychopharmacol 2014 Nov;28(11):1088–98. DOI: https://doi.org/10.1177/ retrospective feasibility study. J Autism Dev Disord 2019 Mar;49(3):1284–8. DOI: https:// 0269881114550355, PMID:25237116. doi.org/10.1007/s10803-018-3808-2, PMID:30382443. 44. Consroe P, Laguna J, Allender J, et al. Controlled clinical trial of cannabidiol in 46. Libzon S, Schleider LB, Saban N, et al. Medical cannabis for pediatric moderate to severe Huntington’s disease. Pharmacol Biochem Behav 1991 Nov;40(3):701–8. DOI: https://doi. complex motor disorders. J Child Neurol 2018 Aug;33(9):565–71. DOI: https://doi.org/10. org/10.1016/0091-3057(91)90386-g, PMID:1839644. 1177/0883073818773028, PMID:29766748.

The Permanente Journal·https://doi.org/10.7812/TPP/20.156 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 143 n REVIEW ARTICLE What Is Genitourinary Syndrome of Menopause and Why Should We Care?

Kelly Jo Peters, DO1 Perm J 2021;25:20.248 E-pub: 3/10/2021 https://doi.org/10.7812/TPP/20.248

ABSTRACT GSM is caused by decreased estrogen. Estrogen helps the Genitourinary syndrome of menopause (GSM; previously tissue stay lubricated and elastic. As women age and enter known as vulvovaginal atrophy or atrophic vaginitis) involves menopause, they have decreased estrogen and the vaginal symptoms of vaginal dryness, burning, and itching as well as and vulvar tissue starts to thin and weaken and has less dyspareunia, dysuria, urinary urgency, and recurrent urinary tract elasticity and lubrication. Even the length of the vagina can infections. It is estimated that nearly 60% of women in meno- shorten and the entrance to the vagina (introitus) narrows, pause experience GSM but the majority of these women do not often causing pain or difficulty with intercourse. is can bring up this concern with their health care provider. Studies also show that only 7% of health care providers ask women about this also happen prior to natural menopause due to other hypo- condition. This may be due to embarrassment or thinking this is a estrogenic conditions (eg, when a woman is breastfeeding or ff normal part of aging, both by patients and health care providers. has had surgery, radiation, or chemotherapy a ecting her This condition is progressive and may affect many aspects of a ovaries) or as a result of certain medical conditions (eg, woman’s physical, emotional, and sexual health. This article is primary ovarian insufficiency, hypothalamic amenorrhea) or intended to address the signs, symptoms, and significant impact medications (eg, tamoxifen, leuprolide, danazol, medroxy- this condition can have for women and help health care providers progesterone acetate, aromatase inhibitors). be more comfortable knowing how to ask about GSM, diagnosis It may sound like GSM just causes vaginal dryness and it, and review the various treatment options that are available. discomfort, but it can actually affect many aspects of a woman’s health, not only physically but psychologically and INTRODUCTION sexually. In my gynecologic practice, I frequently see women It has been called vulvovaginal atrophy or atrophic vagi- who have entered menopause and may have made it past the fl nitis. e newer term, genitourinary syndrome of menopause hot ashes and night sweats but are now noticing more (GSM), was introduced by the International Society for the dryness and pain with intercourse. When we talk about it, Study of Women’s Sexual Health and the North American some of my patients admit to avoiding any intimate contact Menopause Society (NAMS) in 2014. GSM is defined as “a with their partner because they worry that this may lead to collection of signs and symptoms associated with estrogen sex. Eventually, their partner often starts to feel rejected and ff deficiency that can involve changes to the labia, introitus, the relationship itself su ers. vagina, clitoris, bladder and urethra.”1 Unfortunately, even medical providers with training in women’s health and gynecology often do not get much WHAT IS GSM AND WHY SHOULD WE CARE? education on the vulva and usually even less on sexual What GSM means clinically is that the vaginal and vulvar health. Although we may ask women as they age about hot flashes or night sweats, we are often guilty of not asking tissue becomes thin and dry, which often leads to a burning “ sensation, itching, and pain and dryness during sex. Some- important questions such as Are you noticing any vaginal dryness or trouble lubricating during sex?” or inquiring if times these symptoms are so bad that women are unable to ff ’ have sex (penile/vaginal intercourse), which of course can these changes are a ecting a woman s sexual relationship. contribute to low sex drive. As I discuss with my patients, One study of more than 3,000 women with symptoms of GSM showed that only 7% of providers asked about these most women do not look forward to sex if it hurts! (Note: 2 Although this article is primarily addressed to GSM in changes. heterosexual women, this condition can affect women re- It is estimated that although nearly 60% of women in gardless of sexual preference or practices.) menopause experience GSM, the majority of these women do not discuss this concern with their health care provider.3 is may be due to embarrassment, cultural reasons, or even thinking that this is a normal part of aging and nothing can Author Affiliation be done. We, as providers, may even overlook these changes 1Colorado Permanente Medical Group, Denver, CO or possibly also think they are a normal part of aging. In addition to pain with sex that in turn affects a woman’s Corresponding Author sex drive, GSM can cause discomfort to the point where a Kelly Jo Peters, DO ([email protected]) woman may stop being as physically active, affecting her Keywords: atrophic vaginitis, genitourinary syndrome of menopause, vulvovaginal atrophy 144 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.248 REVIEW ARTICLE What Is Genitourinary Syndrome of Menopause and Why Should We Care?

physical and emotional health. GSM can contribute to Table 1. Vulvar hygiene more frequent vaginal and urinary tract infections due to an Things to avoid increase in vaginal pH and changes in the vaginal micro- Tight-fitting clothing flora. e underlying connective tissue also thins and is Synthetic underwear more susceptible to inflammation or infection. Pelvic organ Scented soaps, body wash, and bubble bath prolapse with urinary retention and/or urinary incontinence Scented detergents may also occur.4 Laundry softener, dryer sheets A diagnosis of GSM should include obtaining a thorough Baby wipes, flushable wipes patient history. Ask the patient about onset, duration, prior Feminine hygiene sprays, douches, and wipes treatment, potential vulvar irritants (Table 1), other medical conditions or medications, and previous surgery, including Dyed/colored toilet paper prior cancer or cancer treatments. Vaginal infections should Constant use of pantiliners be ruled out and sexually transmitted infections should also Washcloths, scrubbies, and loofahs be considered. Try instead fi Clinically, the external genitalia may appear pale and thin. Loose- tting clothing With inflammation, sometimes the tissue can be ery- Cotton underwear in daytime thematous with excoriations. In severe cases, the labia No underwear at night minora may be essentially nonexistent, having fused to the Fragrance-free pH-neutral soaps/detergents labia majora. e introitus may narrow and a urethral Tub bath without additives and at comfortable temperature caruncle is often seen. ere is loss of vaginal rugae and Use fingertips for gentle vulvar washing ideally with water only decreased elasticity of the vagina, which can make dis- Try a sport water bottle, perineal bottle, or bidet tention of the vagina with a speculum very painful for many Gently pat vulva dry women. e cervix may sometimes be difficult to visualize not only due to pain with opening of the speculum, but it Many women find that using lubricants with intercourse also may become flush with the vaginal wall and the cervical helps make sex more comfortable. e brands commonly os itself may become stenotic. At times, an increased yellow found over the counter are usually water-based lubricants. or brown, sometimes malodorous, discharge is present. Many menopausal and perimenopausal women find that Nitrazine paper applied to the introitus can help confirm these absorb fairly quickly and do not provide enough a diagnosis of vaginal atrophy. A normal well-estrogenized comfort. Silicone-based lubricants tend to last longer and vagina will have a pH ranging from 3.5 to 5.0. In the provide more lubrication. Coconut, vitamin E, avocado, or absence of infection (eg, bacterial vaginosis) or semen from olive oils work well but any oil-based lubricant should not recent intercourse, a pH of 5.5 or higher is seen with vaginal be used with condoms, as they weaken latex and some atrophy.5 women find they may be more prone to vaginal infections Other vulvar disease may include some of the same patient with these lubricants. complaints such as external irritation, burning, or itching and ere are also nonhormonal products called vaginal possibly pain with intercourse. For example, lichen sclerosis moisturizers. As women age, most of us notice that our skin usually appears as white skin changes that tend to affect the gets thinner and dryer and needs more moisture. Such labia minora and/or majora and often the perineum or changes also happen to the vagina and vulva. Just like women perianal region (Figure 1). Other disorders could include may use a daily moisturizer on other areas of the body, they lichen planus, lichen simplex chronicus, dermatitis, vitiligo, or can also use a vaginal moisturizer either daily or 2 to 3 times mucous membrane pemphigoid; thus, biopsies are recom- per week. Many of these products contain ingredients like mended to confirm diagnosis of any suspected vulvar disorder those found in facial products such as hyaluronic acid, which or any lesion of the vulva that does not respond to treatment. helps tissue retain moisture and stay lubricated.6 ere are a variety of different treatment options for ere are a variety of prescription treatments available for GSM. e choice of treatment may depend on the severity GSM (Table 3). Low-dose vaginal estrogen is the gold of symptoms and should include a discussion of risks, standard treatment for GSM but women (and their health benefits, and effectiveness, address a patient’s preference, care providers) can still be reluctant to consider this option. and review any concerns she may have regarding hormonal It does not help that the product information has to list treatment. I begin by discussing nonhormonal options possible risks including cardiovascular disease, breast can- (Table 2), such as vaginal lubricants and moisturizers, with cer, and dementia. NAMS has suggested that this “black my patients, but I also give them information regarding box warning” be removed from vaginal estrogen products, as vaginal estrogen and other prescription therapies. the amount of hormone absorbed into the body when used

The Permanente Journal·https://doi.org/10.7812/TPP/20.248 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 145 REVIEW ARTICLE What Is Genitourinary Syndrome of Menopause and Why Should We Care?

Table 2. Nonhormonal options for genitourinary syndrome of menopause Lubricants Water based KY Astroglide Good Clean Love Sylk Pjur YES Silicone based Uberlube Eros Pink ID Millennium Wet Platinum Pjurmed Premium Glide Oil based Elegance Women’s Lubricants Oils (olive, coconut, avocado, vitamin E, Crisco) Moisturizers Replens Figure 1. Loss of pigmentation to bilateral labia majora, minora, and perineum. RepHresh Thinned texture and characteristic “cigarette paper” wrinkling. Small erosions at left Luvena perineum. Lubrigyn Sylk natural intimate moisturizer vaginally is very low and does not have the same potential Yes vaginal moisturizer health risks as systemic estrogen.7 Canestima Even women taking systemic estrogen for vasomotor Femallay Moisturizing Suppositories symptoms of menopause may still experience GSM and benefit from local treatment. If a woman’s primary concern bioidentical estradiol cream (Estrace) but find that the is GSM, local rather than systemic estrogen is recom- amount recommended by the manufacturer is usually too mended, as it has been shown to be more effective for GSM much (2-4 g) and may have higher systemic absorption. I and to have lower risks.8 Vaginal estrogen has also been start with 0.5 to 1 g for most of my patients; as in systemic shown to be more effective in the treatment of recurrent hormone therapy, the lowest effective dose is recommended. urinary tract infections9 and improvement of incontinence, Some women may need to start with slightly higher doses, while systemic estrogen may actually worsen incontinence.10 then decrease the amount as the vulvovaginal tissue health Vaginal estrogen is available in various prescription improves or transition to the vaginal inserts or ring. (Some- forms, including creams, intravaginal tablets or inserts, and what confusingly, there is also a vaginal ring called Femring a vaginal ring. In general, it is recommended to treat daily that provides systemic estrogen to help treat hot flashes for the first 2 weeks, then decrease administration to twice and also works locally to treat GSM, but the ring used for weekly. It can often take 4 to 6 weeks, and sometimes GSM [Estring] has very minimal absorption.) longer, to notice an improvement. Some women may be uncomfortable or physically unable Although the vaginal estrogen inserts or ring have been to insert the cream, tablets, or suppositories. In this case, the shown to have the least systemic absorption,11 I find Estring may be used and changed by the provider every (Figures 2 and 3) that these forms of vaginal estrogen may 3 months in the office. initially not be adequate for my patients with significant Systemic estrogen can increase the thickness of a woman’s atrophy. I often start their treatment course with vaginal uterine lining (the endometrium) and potentially lead to estrogen cream. Even though the systemic absorption may uterine cancer or a precancerous thickening (hyperplasia). be a bit higher, it is still quite low and the cream may be If a woman is taking systemic estrogen (and still has her applied to the vulva as well. I prefer using the more uterus), she also needs to take progesterone to help prevent

146 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.248 REVIEW ARTICLE What Is Genitourinary Syndrome of Menopause and Why Should We Care?

Table 3. Prescription treatments for genitourinary syndrome of menopause Product Vaginal estrogen ring Estring (7.5 µg of estradiol released once a day): inserted by patient or clinician every 90 d Vaginal estrogen insert Vagifem/Yuvafem (10 µg of estradiol): insert 1 tablet vaginally every night for 2 wk, then twice weekly Imvexxy (4 µg and 10 µg of estradiol): insert vaginally every night for 2 wk, then twice weekly Vaginal estrogen cream Estrace (100 µg of estradiol/g): insert 0.5-1 g vaginally every night for 2 wk, then twice a week Premarin (0.625 mg of conjugated estrogen/g): insert 0.5-1 g vaginally every night for 2 wk, then twice a week Other Ospemifene (Osphena; 60 mg daily oral tablet), selective estrogen receptor modulator Prasterone (Intrarosa; 6.5 mg nightly intravaginal suppository), Figure 2. Note the smooth, pale inner labia minora, the prominent erythematous dehydroepiandrosterone urethral caruncle, and narrowed introitus.

the uterine lining from getting too thick. When a woman is not show an increased risk of cancer recurrence.12 Although using only low-dose vaginal estrogen, she does not also need some women are understandably still worried about the to take progesterone because the amount of estrogen potential risk, other women may feel that this is a quality- absorbed into the body is extremely low and does not appear of-life issue and are relieved to know that this is a treatment to increase the risk of uterine cancer (although endometrial option. If vaginal estrogen is considered, it is often advised safety has not been studied beyond 12 months of use). to consider the vaginal estradiol inserts or the low-dose NAMS has suggested removing the boxed warning on estradiol ring due to the fixed amount of medication and the vaginal estrogen but cautions that women are still advised to lack of significant systemic absorption. e Imvexxy vaginal call their provider if they do have any bleeding, as this can inserts come in a very low-dose form of 4 µg, which may be potentially be a warning sign of uterine cancer or hyperplasia. preferable if there is concern about systemic absorption. Side effects of local estrogen may include vaginal dis- It is usually advised to discuss this first with a woman’s charge, vulvovaginal candidiasis, breast tenderness, and oncologist or primary care provider. vaginal bleeding, although these appear to be dose related Women taking aromatase inhibitors for breast cancer and may vary with the formulation. As above, any vaginal treatment are often advised not to use vaginal estrogen. bleeding needs to be investigated and a woman with undi- However, a 2019 meta-analysis of 8 studies showed no agnosed vaginal/uterine bleeding should not be started on increase in serum estradiol levels after 8 weeks of local vaginal estrogen until a thorough evaluation has been per- hormone treatment in women taking aromatase inhibitors, formed. Some patients may also feel that the cream is too which appears reassuring.13 messy, while others may like the lubricating affectitmayhave e risk of venous thromboembolism was not increased and the option to apply a small amount externally to the vulva. with local estrogen therapy based on observational studies.14 Even women with a history of breast cancer or other A newer product called prasterone (Intrarosa) was ap- potential contraindications to systemic estrogen may con- proved by the US Food and Drug Administration (FDA) in sider using low-dose vaginal estrogen. It is still recom- 2016. Prasterone is a nightly vaginal insert that is plant mended to try nonhormonal options first, but for many derived and appears very effective in the treatment of vaginal women this is just not enough. In 2016, the American dryness and painful intercourse. rough intracellular ste- College of Obstetricians and Gynecologists issued an roidogenesis, prasterone is converted into estrogen and opinion stating that vaginal estrogen could be considered for testosterone in the cells of the vagina. It appears to be very women currently undergoing breast cancer treatment or for low risk in terms of low to no absorption of hormones into women with a personal history of breast cancer who are the bloodstream and may eventually be found to be a good unresponsive to nonhormonal treatment and the data do option for women who have contraindications to systemic

The Permanente Journal·https://doi.org/10.7812/TPP/20.248 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 147 REVIEW ARTICLE What Is Genitourinary Syndrome of Menopause and Why Should We Care?

Table 4. Dilator resources Resource Website Vaginismus.com (www.vaginismus.com) Soul Source (www.soulsource.com) MiddlesexMD (https://middlesexmd.com) Cooper Surgical (www.coopersurgical.com) CMT Medical (www.cmtmedical.com) Other dilator options Milli expanding dilator (www.millimedical.com) FeMani vibrating massage wand (https://femaniwellness.com)

costly, and usually require more than 1 treatment with a possible need for retreatment in the future. In addition to dyspareunia, vaginal and bladder infections, prolapse, and incontinence, women with GSM often notice significant pain with pelvic examinations and pap smears. Insertion of the speculum can be quite painful, especially if the vaginal introitus has narrowed significantly. eactual opening of the speculum may be even more painful, especially if a woman is no longer sexually active. e vaginal tissue is thin and loses elasticity. It is important to understand these changes and do our best to help a woman be more com- Figure 3. Labia minora atrophied and no longer existent. Fusion of superior labia fortable so she does not avoid coming to see us. obscuring clitoris. Helpful techniques include using a lubricated narrow Pedersen or a pediatric speculum if necessary. I sometimes estrogen; however, the FDA currently still requires the warning will apply topical lidocaine jelly to the introitus and/or that it has not been studied in women with breast cancer and, as speculum first. I also may use only one gloved, lubricated with vaginal estrogen preparations, should not be used in finger for the pelvic examination. If a woman is very anxious women with undiagnosed abnormal genital bleeding.15 or visibly contracting her pelvic floor muscles, I will ask her Ospemifene (Osphena) is an oral, nonhormonal method to squeeze as hard as she can around my gloved finger, then to treat moderate to severe dyspareunia associated with ask her to breathe and relax as I gently insert my finger a bit vulvovaginal atrophy. It is a selective estrogen receptor more. I always tell my patient to let me know if I am causing modulator and is taken daily. Ospemifene is not an actual her pain and that I will stop at any time if she tells me to. hormone; rather, it is an estrogen agonist/antagonist, acting is may help her relax, allowing a more thorough ex- on estrogen receptors in the vagina to treat vaginal dryness amination and giving her some control over an often un- and subsequent pain with intercourse. It may be helpful in comfortable and intimidating procedure. women who are either unwilling or unable to use vaginal In a woman with very severe atrophy who is unable to estrogen. Side effects include hot flashes/night sweats tolerate any examination or any attempt at penile insertion, and ospemifene may increase the risk for thromboembolic it is often helpful to try treating with vaginal estrogen for 4 complications. Ospemifene may theoretically increase the to 6 weeks as well as have her work with a vaginal dilator to risk of uterine cancer as well; however, as with the vaginal assist in gently stretching the tissue. ere are many estrogen studies, it appears to be safe in studies up to 1 year of companies that sell graduated vaginal dilators. ere is use. It appears to have antiestrogenic effects on the breast but also the Milli, which is a patient-controlled dilator that is not approved for women with breast cancer.16,17 expands 1 mm at a time (Table 4). It can be helpful to refer Other nonhormonal options for GSM now include apatienttoagoodpelvicfloor physical therapist if she physical procedures such as laser therapy, which are reported would like additional instruction and assistance with to help vaginal dryness by causing microabrasions in the dilator use and relaxation. vaginal tissue to stimulate neovascularization and promote My motivation for writing this article was to help increased collagen production. While these procedures may women and their health care providers become more be promising, they are currently not FDA approved, are familiar with GSM and the treatment options available.

148 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.248 REVIEW ARTICLE What Is Genitourinary Syndrome of Menopause and Why Should We Care?

Table 5. Retail websites References 1. North American Menopause Society. Menopause practice: A clinician’s guide, 6th ed. Website Pepper Pike, OH: North American Menopause Society; 2019. MiddlesexMD (www.middlesex.md) 2. Kingsberg SA, Wysocki S, Magnus L, Krychman, ML. Vulvar and vaginal atrophy in postmenopausal women: Findings from the REVIVE (REal Women’s VIews of Treatment Good Vibrations (www.goodvibes.com) Options for Menopausal Vaginal ChangEs) survey. J Sex Med 2013 Jul;10(7):1790-9. Eve’s Garden (www.evesgarden.com) DOI: https://doi.org/10.1111/jsm.12190, PMID:23679050 Adam and Eve (www.adamandeve.com) 3. Levine KB, Williams RE, Hartmann KE. Vulvovaginal atrophy is strongly associated with female sexual dysfunction among sexually active postmenopausal women. Menopause 2008 Jul-Aug; Babeland (www.babeland.com) 15(4 Pt 1):661-6. DOI: https://doi.org/10.1097/gme.0b013e31815a5168, PMID:18698279 fi Target (www.target.com; sexual health) 4. Dessole S, Rubattu G, Ambrosini G, et al.. Ef cacy of low-dose intravaginal estriol on urogenital aging in postmenopausal women. Menopause 2004 Jan-Feb;11(1):49-56. DOI: Walgreens (www.walgreens.com; sexual lubricants) https://doi.org/10.1097/01.GME.0000077620.13164.62, PMID:14716182 CVS (www.cvs.com; sexual health) 5. Nilsson K, Risberg B, Heimer G. The vaginal epithelium in the postmenopause--Cytology, histology and pH as methods of assessment. Maturitas 1995 Jan;21(1):51-56. DOI: Sexuality Resource Center (www.sexualityresources.com) https://doi.org/10.1016/0378-5122(94)00863-3, PMID:7731384 6. Krychman ML, Dweck A, Kingsberg S, Larkin L. The role of moisturizers and lubricants in genitourinary syndrome of menopause and beyond. OBG Management 2017 April:SS1-SS10.  7. Pinkerton J, Liu J, Santoro NF, et al.. Workshop on normal reference ranges for estradiol is is a progressive condition and unfortunately is not in postmenopausal women; commentary from the North American Menopause Society on often addressed by us or our patients until it becomes low-dose vaginal estrogen therapy labeling. Menopause 2020 Jun;27(6):611-3. DOI: https://doi.org/10.1097/gme.0000000000001576 severe. We, as medical providers, need to be aware that 8. Long CY, Liu CM, Hsu SC, Wu CH, Wang CL, Tsai EM. A randomized comparative study of this is an issue for many women and understand the the effects of oral and topical estrogen therapy on the vaginal vascularization and sexual ’ function in hysterectomized postmenopausal women. Menopause 2006 Sep-Oct;13(5): impact it may have on a woman s overall health as well as 737-43. DOI: https://doi.org/10.1097/01.gme.0000227401.98933.0b, PMID:16946685 her sexual health and quality of life. We need to look for it, 9. Perrotta C, Aznar M, Mejia R, Albert X, Ng CW. Oestrogens for preventing recurrent urinary tract infection in postmenopausal women. Cochrane Database Syst Rev 2008 Apr 16;(2): ask about it, and be familiar with options that can help. It CD005131 DOI: https://doi.org/10.1002/14651858.CD005131.pub2, PMID:18425910 can greatly relieve a woman to know that although these 10. Grady D, Brown JS, Vittinghoff E, Applegate W, Varner E, Snyder T; HERS Research can be normal changes associated with aging, she does not Group. Postmenopausal hormones and incontinence: The Heart and Estrogen/Progestin Replacement Study. Obstet Gynecol 2001 Jan;97(1):116-20. DOI: https://doi.org/10.1016/ have to live with them or be embarrassed to ask for treatment s0029-7844(00)01115-7, PMID:11152919 options (Table 5 includes a list of retail websites where 11. Lethaby A, Ayeleke RO, Roberts H. Local oestrogen for vaginal atrophy in postmenopausal women. Cochrane Database Syst Rev 2016 Aug;(8);CD001500. DOI: women may purchase lubricants, moisturizers, or vibrators). I https://doi.org/110.1002/14651858.CD001500.pub3, PMID:27577677 hope this article will help us, as providers, realize the extent 12. Farrell R; American College of Obstetricians and Gynecologists’ Committee on ff ’ Gynecologic Practice. ACOG committee opinion no. 659: The use of vaginal estrogen in that this condition can a ect a woman s life and empower us women with a history of estrogen-dependent breast cancer. Obstet Gynecol 2016 Mar; to know we can offer her some relief. v 127(3):e93-6. DOI: https://doi.org/10.1097/AOG.0000000000001351 13. Pavlovic RT, Jankovic SM, Milovanovic JR, et al. The safety of local hormonal treatment for vulvovaginal atrophy in women with estrogen receptor-positive breast cancer who are Disclosure Statement on adjuvant aromatase inhibitor therapy: Meta-analysis. Clin Breast Canc 2019 Dec;19: The author(s) have no conflicts of interest to disclose. e731-40. DOI: https://doi.org/10.1016/j.clbc.2019.07.007 14. North American Menopause Society (NAMS). The 2020 genitourinary syndrome of menopause position statement of the North American Menopause Society. Menopause Authors’ Contributions 2020 Sep;27(9):976-92. DOI: https://doi.org/10.1097/GME.0000000000001609 Kelly Jo Peters, DO, conceived of the presented idea, developed the tables, 15. Millicent Pharma. Prasterone (Intrarosa) product website. Accessed February 24, 2021. and wrote the final manuscript. The author has given final approval to the https://intrarosahcp.com/ manuscript. 16. Duchesnay. Ospemifene (Osphena) product website. Accessed February 24, 2021. https:// hcp.osphena.com/ 17. Soe LH, Wurz GT, Kao CJ, Degregorio MW. Ospemifene for the treatment of dyspareunia Financial Support associated with vulvar and vaginal atrophy: Potential benefits in bone and breast. Int J Womens No funding was supplied by outside sources. Health 2013 Sep;5:605-11. DOI: https://doi.org/10.2147/IJWH.S39146, PMID:24109197

The Permanente Journal·https://doi.org/10.7812/TPP/20.248 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 149 n CASE REPORT Bringing Down the Door-to-needle Time: Patient Thrombolysed in 6 Minutes—A Case Report

Ankur Verma, MBBS, MEM1; Sanjay Jaiswal, MBBS, MEM1 Perm J 2021;25:20.239 E-pub: 02/24/2021 https://doi.org/10.7812/TPP/20.239

ABSTRACT mmHg, a respiratory rate of 20/minute, a serum random Introduction: It has been clearly established that thrombolysis sugar level of 172 mg/dL, and he was saturating at 100% on using recombinant tissue plasminogen activator is strongly room air. On examination, the patient was aphasic, had left- beneficial for acute stroke patients. The sensitivity of brain tissue sided gaze preference, and had right-sided dense hemiplegia ff to ischemia causes this time dependence on the e ectiveness of with facial involvement, with complete sensory loss of the recombinant tissue plasminogen activator. Early recognition of right half of his body. He had an initial National Institutes stroke and activation of a stroke alert/code are imperative to treat of Health Stroke score of 26. A stroke code was announced acute stroke patients effectively and to realize positive outcomes. Case Presentation: A 68-year-old man with right-sided weak- and the patient was immediately sent for computed to- ness arrived at our emergency room and was thrombolysed in 6 mography (CT) of the brain. During the CT scan, the ’ minutes from time of arrival, after ruling out all contraindications. family was counseled regarding the patient s condition and Conclusion: The case and our rapid thrombolysis protocol that diagnosis, and the probable need for thrombolysis. Consent helped in achieving the 6-minute door-to-needle time are de- for the same was acquired immediately. e patient had no scribed. A structured protocol is recommended to reduce door-to- contraindications for thrombolysis. Because the CT scan was needle times for thrombolysis in acute ischemic stroke. not suggestive of any intracranial hemorrhage, thrombolysis with rTPA was started while the patient was in the CT INTRODUCTION scanner: a 7-mg intravenous bolus followed by 63 mg over  Approximately 11% of deaths around the world are the 1hour. is was achieved in 6 minutes from the time of result of stroke, making it the second most common cause of arrival (36 minutes from time of onset). With the ongoing death.1 In India alone, 1.44 to 1.64 million new cases of acute thrombolytic infusion, the patient underwent CT angiog- 2 raphy of the brain, which revealed a thrombosed distal M1 stroke are reported every year, and the 30-day case fatality  ranges from 18% to 41%.3 Recently, there has been a lot of segment of the left middle cerebral artery (Figure 1). e patient was moved back to the ED, and admission to the emphasis on stroke prevention and early management proto-  cols, and this has led to a decrease in mortality rates worldwide.4 stroke Intensive Care Unit was initiated. e patient achieved One of the major spokes of stroke care includes early remarkable neurological recovery in 2 hours (National In- initiation of thrombolytic therapy. It has been clearly stitutes of Health Stroke Scale score of 5). For secondary established that thrombolysis using recombinant tissue prevention, the patient was kept on conservative manage- plasminogen activator (rTPA) is strongly beneficial for ment with 150 mg acetylsalicylic acid and 80 mg atorvastatin. acute stroke patients.5 Studies have shown that for every Repeat CT angiography of the brain on day 5 showed good flow in the bilateral middle cerebral arteries, including the 15-minute reduction in door-to-needle time, there is a 5%  lower odds of risk-adjusted inhospital mortality.6 We de- M1 segment of the left middle cerebral artery (Figure 2). e scribe a case with a 6-minute door-to-needle time and patient was discharged on day 6 with normal speech and mild provide an overview of our Rapid rombolysis Protocol. hemiparesis, and a National Institutes of Health Stroke Scale score of 1. e patient was advised neurological follow-up CASE REPORT and outpatient physiotherapy. A 68-year-old hypertensive man (being treated for hy- pertension) was brought to our emergency department Rapid Thrombolysis Protocol (ED) with a sudden onset of right-sided weakness and Our inhospital protocol (Table 2) was created with the inability to speak 30 minutes before arrival. On arrival, the aim of reducing our door-to-needle times in acute ischemic patient had a pulse of 79/minute, a blood pressure of 150/90 stroke patients. Patients arriving at the triage area with any symptoms of stroke are brought to the attention of the Author Affiliation senior ED physician by a triage nurse. If we have prehospital 1Department of Emergency Medicine, Max Super Specialty Hospital, New Delhi, India information regarding a stroke patient arriving, a green corridor is established from triage to CT/magnetic reso- Corresponding Author nance imaging. Counseling of the family is done regard- Ankur Verma, MBBS, MEM ([email protected]) ing the condition and diagnosis, and probable need for Keywords: ischemia, stroke, thrombolysis 150 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.239 CASE REPORT Bringing Down the Door-to-needle Time: Patient Thrombolysed in 6 Minutes—A Case Report

Figure 1. Computed tomographic angiography of the brain revealed a thrombosed Figure 2. Repeat computed tomographic angiography of the brain on day 5 shows fl distal M1 segment of the left middle cerebral artery with no ow (arrow). good flow in the bilateral middle cerebral arteries, including the M1 segment of the left middle cerebral artery (arrow).

thrombolysis by the doctor/emergency medical technician of ischemic stroke is established, the patient is given a bolus in the ambulance, and live locations are shared with the dose followed by infusion over 1 hour. All stroke blood ED team. e vital signs of the patient are recorded by samples are sent before giving the bolus drug. e neu- a trained triage nurse while physicians quickly evaluate rologist is then consulted regarding the need for additional the patient in the triage room itself. On confirmation of CT angiography. If required, the infusion is continued in clinical stroke, a stroke code is announced by dialing the the CT scanner. e patient is then sent back to the ED, emergency code. where an ED physician and nurse monitor his or her he- e calls are received by the neurologist on call, radiology, modynamic values and neurological recovery/worsening. the laboratory, and medical administration. e patient is e neurology team meets to determine whether there is sent immediately to radiology for brain CT or magnetic a need for mechanical thrombectomy. e patient is then resonance imaging (diffusion weighted and fluid attenua- sent to the stroke Intensive Care Unit. tion inversion recovery). Magnetic resonance image is done for patients who have wakeup stroke, when the exact time of DISCUSSION onset is not known, or when symptoms and signs are vague. Until recently, the recommended door-to-needle time for If magnetic resonance imaging is contraindicated and a acute ischemic stroke is 60 minutes or less.7 We suggest that radiological diagnosis is required, then a CT perfusion study a structured protocol can greatly reduce door-to-needle times. is completed. While the patient is undergoing CT, the It is well established that early thrombolysis can achieve much family is counseled regarding the patient’s clinical diagnosis better outcomes for ischemic stroke patients.8 e sensitivity of and condition, and probable need for thrombolysis; consent brain tissue to ischemia causes this time dependence on the for the same is obtained. All contraindications are ruled out effectiveness of rTPA.9 us, it is imperative that institutions during the scan. rTPA is brought to the radiology de- have a streamlined, robust stroke protocol. partment by an ED nurse to save time in case thrombolysis Recognition of stroke begins in the prehospital setup. is required. If the CT scan is normal and clinical diagnosis Paramedics who are transporting patients via ambulance

The Permanente Journal·https://doi.org/10.7812/TPP/20.239 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 151 CASE REPORT Bringing Down the Door-to-needle Time: Patient Thrombolysed in 6 Minutes—A Case Report

Table 1. Patient timeline for relevant past medical history and interventions, including relevant personal, family, and psychosocial history; important past interventions; outcomes; and follow-up.a Date Summaries from initial and follow-up visits Diagnostic testing (including dates) Interventions 10/1/2019 Patient presented with right-sided weakness and Random blood sugar test, EKG, complete blood Injected recombinant tissue plasminogen aphasia 30 minutes before arrival. Patient was count, renal function test, liver function test, activator bolus dose followed by infusion over 1 hypertensive and on medication (amlodipine). prothrombin time, INR, and brain CT with brain and hour. Started 6 minutes from the time of arrival neck CTA were performed. CTA revealed a after brain CT. Tablet aspirin, amlodipine, and thrombus in the distal M1 segment of the left atorvastatin were started. Patient recovered middle cerebral artery. All investigations done on neurologically, with National Institutes of Health 10/1/2019. All bloodwork was normal. Stroke scores of 26 and 5 within 2 hours. 10/6/2019 Patient recovered full function neurologically. Repeat brain CTA performed on 10/6/2019 Patient continued on amlodipine, aspirin, and revealed good flow in the bilateral middle cerebral atorvastatin, and planned for discharge the arteries. following day. 10/7/2019 Patient had a full recovery. No investigations performed. Patient was discharged in stable condition on amlodipine, aspirin, and atorvastatin, with follow- up advised after 1 month. 11/8/2019 Patient followed up in the Neurology OPD and was No investigations ordered. Patient advised to continue amlodipine, aspirin, perfectly stable with no residual neurological and atorvastatin. deficits. No adverse events such as recurrent stroke or bleeding occurred. 4/10/2020 At the 6-month follow-up, the patient was leading a No investigations ordered. Patient advised to continue oral medications as normal life with no neurological deficit. No adverse prescribed and to schedule follow-ups every 6 events occurred. months. Relevant past medical history and interventions Known case of hypertension and was taking Antihypertensive medication has been continued. amlodipine tablets. a Include genetic information if available. CT = computed tomography; CTA = computed tomographic angiography; EKG = electrocardiogram; INR = international normalized ratio; OPD = out patient department.

should be trained to recognize stroke and to transfer patients such as a delay in receiving family consent, the family wanting to the appropriate center.10 Triage nurses play an equally a second opinion, a determination of hypertension (which important role in the early recognition of stroke when needs to be controlled prior to thrombolysis), incomplete patients arrive at the ED. Nurse training is paramount for a drug history, and so on. successful stroke protocol. Early recognition by nurses leads Our case does not highlight the target door-to-needle to early diagnosis and shorter door-to-needle times.11 time because it is not possible for every patient. We do e use of a stroke code alert system has been shown to improve time to diagnosis and treatment, and to reduce 12 Table 2. Rapid Thrombolysis Protocol checklist. CT = computed intravenous rTPA door-to-needle times. Worldwide studies tomography; ED = emergency department; EMT = emergency have shown there are multiple inhospital delays when deliv- medical technician; MRI = magnetic resonance imaging. ering rTPA to stroke patients.13 ere is a lot of scope for reducing door-to-needle times to improve outcomes. rough regular audits and protocol checks, many of the delays that hamper timely thrombolysis may be identified and rectified. A robust stroke program requires a significant volume of stroke patients arriving at the ED, trained paramedics and triage nurses, and trained emergency medicine physicians and inhouse neurologists available 24/7. In addition, the radiology department, laboratory, and catheter suite for thrombectomies must be available, along with a stroke In- tensive Care Unit for postthrombolytic care. Last, there should be regular data collection and discussions regarding all cases to recognize delays and revise any protocols if required. Although our report highlights the feasibility of achieving such short door-to-needle times, it may not always be possible because there may be factors that delay thrombolysis,

152 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.239 CASE REPORT Bringing Down the Door-to-needle Time: Patient Thrombolysed in 6 Minutes—A Case Report

recommend that a structured protocol, such as our Rapid 4. Jauch EC, Saver JL, Adams HP Jr, et al. Guidelines for the early management of patients with acute ischemic stroke: A guideline for healthcare professionals from rombolysis Protocol, can reduce mean and median door- the American Heart Association/American Stroke Association. Stroke 2013 Mar; to-needle times substantially (to much less than 60 minutes) 44(3):870–947. DOI: https://doi.org/10.1161/STR.0b013e318284056a,PMID: v 23370205 for acute ischemic stroke. 5. Roth JM. Recombinant tissue plasminogen activator for the treatment of acute ischemic stroke. SAVE Proc 2011 Jul;24(3):257–9. DOI: https://doi.org/10.1080/08998280.2011. 11928729, PMID:21738304 Disclosure Statement 6. Fonarow GC, Smith EE, Saver JL, et al. Timeliness of tissue-type plasminogen activator The authors have no conflicts of interest to disclose. therapy in acute ischemic stroke: Patient characteristics, hospital factors, and outcomes associated with door-to-needle times within 60 minutes. Circulation 2011 Feb;123(7):750–8. DOI: https://doi.org/10.1161/CIRCULATIONAHA.110.974675, Authors’ Contributions PMID:21311083 Ankur Verma, MBBS, MEM, is the principal author and contributed to the 7. Fonarow GC, Smith EE, Saver JL, et al. Improving door-to-needle times in acute ischemic conception, literature review, and design of the manuscript. Sanjay Jaiswal, MBBS, stroke: The design and rationale for the American Heart Association/American Stroke ’ – MEM, contributed to drafting, discussion, and critical revision of the manuscript. Association s target: Stroke initiative. Stroke 2011 Oct;42(10):2983 9. DOI: https://doi.org/ 10.1161/STROKEAHA.111.621342, PMID:21885841 The authors did not have any third-party contributions to design, data collection, 8. Lees KR, Bluhmki E, von Kummer R, et al. Time to treatment with intravenous alteplase data analysis, or manuscript preparation. and outcome in stroke: An updated pooled analysis of ECASS, ATLANTIS, NINDS, and EPITHET trials. Lancet 2010 May;375(9727):1695–703. DOI: https://doi.org/10.1016/ Disclaimer S0140-6736(10)60491-6, PMID:20472172 The views expressed in the article are the authors’ own and not an official 9. Marler JR, Tilley BC, Lu M, et al. Early stroke treatment associated with better outcome: The NINDS rt-PA stroke study. Neurology 2000 Dec;55(11):1649–55. DOI: https://doi.org/ position of the institution. 10.1212/wnl.55.11.1649, PMID:11113218 10. Brice JH, Evenson KR, Lellis JC, et al. Emergency medical services education, Funding community outreach, and protocols for stroke and chest pain in North Carolina. Prehosp – – No funding was received for this case report. Emerg Care 2008 Jul Sep;12(3):366 71. DOI: https://doi.org/10.1080/ 10903120802100100, PMID:18584506 11. Middleton S, Grimley R, Alexandrov AW. Triage, treatment, and transfer: Evidence-based References clinical practice recommendations and models of nursing care for the first 72 hours of 1. Towfighi A, Saver JL. Stroke declines from third to fourth leading cause of death in the admission to hospital for acute stroke. Stroke 2015 Feb;46(2):18–25. DOI: https://doi.org/ United States: Historical perspective and challenges ahead. Stroke 2011 Aug;42(8): 10.1161/strokeaha.114.006139. 2351–5. DOI: https://doi.org/10.1161/STROKEAHA.111.621904, PMID:21778445 12. Meretoja A, Strbian D, Mustanoja S, et al. Reducing in-hospital delay to 20 minutes in 2. Murthy J. Thrombolysis for stroke in India: Miles to go ....Neurol India 2007 Jan–Mar; stroke thrombolysis. Neurology 2012 Jul;79(4):306–13. DOI: https://doi.org/10.1212/ 55(1):3–5. DOI: https://doi.org/10.4103/0028-3886.30415. WNL.0b013e31825d6011, PMID:22622858 3. Dalal PM, Malik S, Bhattacharjee M, et al. Population-based stroke survey in Mumbai, 13. Klingner CM, Brodoehl S, Hohenstein C, et al. A case with 7 min door-to-needle-time and India: Incidence and 28-day case fatality. Neuroepidemiology 2008 Oct;31(4):254–61. an outline of ultrarapid stroke management. Brain Disord Ther 2014 Nov;4(1):153–6. DOI: DOI: https://doi.org/10.1159/000165364, PMID:18931521 https://doi.org/10.4172/2168-975X.1000153

The Permanente Journal·https://doi.org/10.7812/TPP/20.239 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 153 n CASE REPORT Utilization of Nivolumab in Adenoid Cystic Carcinoma After Progression on Platinum-Based Chemotherapy

Diana V Maslov, MD, MS1; Katharine Thomas, MD, MS2; Marc Matrana, MD, MS, FACP3 Perm J 2021;25:20.229 E-pub: 03/01/2021 https://doi.org/10.7812/TPP/20.229

ABSTRACT Clinical findings associated with ACC of the head and Introduction: Adenoid cystic carcinoma (ACC) is a rare neckincludealumponthepalate,underthetongue, malignant neoplasm within the secretory glands of the head or in the bottom of the mouth. ere can be numbness and neck. Clinical findings include a lump on the palate, of the upper jaw, palate, face, or tongue. One can have tongue, or bottom of the mouth. Because symptoms can be difficulty swallowing, hoarseness, and paralysis of the mild, patients go for long periods of time without investi- facial nerve.3 Because many of these symptoms can be gation. ACC is diagnosed using histology. Treatment is by mild, many patients will go for long periods of time surgical resection because there is no effective chemother- apy. Radiation can be effective adjuvant therapy, and proton without investigation.   therapy and stereotactic irradiation can be used for those who e diagnosis of ACC is made using histology. ere ff are ineligible for surgery. Immunotherapy has clinical activity are 3 di erent forms: cribriform, tubular, and solid; 4 for those with metastatic head and neck cancers who progress on solid is the most aggressive. Imaging is used to mea- proton therapy. This case reviews the use of immunotherapy in a sure the tumor as well as to identify recurrence. is can patient with ACC. be done using computed tomography (CT), magnetic Case Presentation: A man in his 20s presented with a 6-month resonance imaging (MRI), or positron emission tomogra- history of nasal congestion, epistaxis, and sinus tenderness. phy (PET). Noncontrast computed tomography of the sinuses revealed a Optimal treatment of ACC is surgical resection, and mass of the lateral wall of the nasal cavity, lateral wall of the radiation may be effective as adjuvant therapy. Unfortu- maxillary sinus, and pterygoid plates. Positron emission to- nately, the mortality rate in those with ACC is high due to mography confirmed metastatic disease in the right iliac crest and right cervical lymph node; biopsy of the nasopharynx local recurrences and late distant metastases. Treatment of confirmed ACC. The patient received proton therapy and advanced disease is considered palliative because there is ff intensity-modulated radiotherapy and completed 2 Phase 1 no e ective chemotherapy; however, proton beam therapy trails but continued to have progressive disease. The patient and stereotactic irradiation can be used for those who are started nivolumab and died 12 weeks later. ineligible for surgery.3 Conclusion: The patient recently received proton therapy, Immunotherapy has been used in the treatment of intensity-modulated radiotherapy, and completed 2 Phase 1 metastatic head and neck cancers. Immunotherapy in- trials but continued to have progressive disease. hibits the PD-L1 pathway, which prevents activation of cytotoxic T cells in lymph nodes and deactivates cyto- INTRODUCTION toxic T cells in dendritic cells. PD-L1 also helps cancer cells adjust to the body’s environment and continue to Adenoid cystic carcinoma (ACC) is a malignant neo- 4 plasm within the secretory glands of the head and neck. It proliferate. Inhibition of the PD-L1 pathway allows  tumor-invading cells to be recognized and destroyed by represents 10% of all neoplasms of the salivary glands. is 4,5 can occur in the breast, bone, lung, and liver but is most cytotoxic T cells. PD-1/PD-L1 inhibitor therapy includes often a metastasis from the head and neck. It very rarely nivolumab, pembrolizumab, atezolizumab, avelumab, and infiltrates the lymphatic system.1 ecauseofACCisun- durvalumab. Immunotherapy has been shown to have sig- nificant clinical activity for those who have progressed on known, but it develops from noninherited genetic changes over 6 one’slife.2 Most patients diagnosed are in their 40s-60s, with a proton therapy. Although the relationship between im- female:male ratio of 3:2.2 munotherapy and ACC has not been well investigated, it has been shown that an estimated 11% of ACCs express PD-L1 on their cell membranes. Also, 70% of tumor- fi 7 Author Affiliations in ltrating monocytes are PD-L1 positive, giving im- 1Department of Internal Medicine, Ochsner Health System, New Orleans, LA munotherapy a target for ACC cancer cells. is case study 2Department of Hematology/Oncology,Louisiana State University, New Orleans, LA reviews the use of immunotherapy, commonly used in 3Ochsner Cancer Institute, New Orleans, LA squamous head and neck cancers, implemented in a patient Corresponding Author with ACC. Diana V Maslov, MD, MS ([email protected])

Keywords: cancer, head and neck, immunology 154 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.229 CASE REPORT Utilization of Nivolumab in Adenoid Cystic Carcinoma After Progression on Platinum-Based Chemotherapy

Date Summaries from initial and follow-up visits Diagnostic testing (including dates) Interventions 9/28/2014 Patient presented to otolaryngology clinic with CT sinuses 10/1/2014, CT soft tissue neck 10/14/ worsening nasal congestion, epistaxis, and sinus 2014 revealed mass of lateral wall of nasal cavity, tenderness. lateral wall of maxillary sinuses, and pterygoid plates. 10/16/2014 . Nasal endoscopy and biopsy confirmed adenoid cell carcinoma. 10/20/2014 Followed up with 0tolaryngology clinic and PET scan revealed metastatic disease in the right discussed treatment options including iliac crest and right cervical lymph node. radiotherapy. 11/7/2014 Had an appointment with radiation oncology to prepare for radiation treatment Had first appointment with medical oncology in which both radiation and surgery options were On 11/18/2014, hip bone biopsy confirmed 11/11/2014 discussed. metastatic disease. Planned for biopsy of iliac crest to confirm metastatic disease. 4/9/2015 Patient follows up with medical oncology at The patient went to MD Anderson between 11/ Ochsner. 2014 and 4/2015 in which he was treated with cisplatin and concurrent proton beam therapy. 8/15/2015 Continued to follow up with Ochsner Oncology and Restaging PET showed new bone lesion in iliac Patient was Treated with external beam well as MD Anderson. crest. radiation. 4/5/2017 Patient continues to be treated both at MD Genetic profiling per patient performed at MD Patient completed 8 cycles on a Phase 1 Trial: Anderson and at Ochsner Oncology. Anderson showed only targetable mutation is Protocol 2014-0066 with PT-112 at MD MDM2. Anderson. Patient followed up with Ochsner Oncology. He has completed Phase 1 Trial: Protocol 2014- 0066 with PT-112 at MD Anderson. 6/27/2017 He had restaging scans showing progression of Patient began nivolumab on 6/30/2017. disease and then went onto a trial of an MDM2 inhibitor at MDACC, but continued to progress. Plan to receive immunotherapy with nivolumab. 9/18/2017 Patient stopped breathing at home. The family called EMS, and he was transferred to the emergency department at a nearby hospital. He died from cardiac arrest.

CASE PRESENTATION bony erosion extending to the lateral wall, pterygoid A white man in his 20s with a past medical history of plates with extension into the lateral left sphenoid sinus, attention deficit hyperactive disorder and multiple treat- and erosion of that left lateral sphenoid body (Figure 1). ments for serous otitis media and sinusitis presented with a Nuclear medicine positron emission tomographyCT skull 6-month history of nasal congestion, jaw pain, epistaxis, and to mid-thigh confirmed the mass in the left nasopharynx as rhinorrhea. e patient was a former smoker of 0.5 pack-years. well as a metastatic right cervical lymph node and hy- His mother had hypertension and paternal grandmother permetabolic activity in the posterior aspect of the right had breast cancer. Vitals revealed high blood pressure of iliac bone (Figure 2). 141/86 and pulse of 99. On physical examination, the patient Biopsies of the last nasal mass and the right iliac bone had normal ears, normal oral cavity, and no lymphadenop- were consistent with cribriform ACC and metastatic ACC athy or thyroid nodules but had sinus tenderness. On flexible (Figure 3). Due to the location of the tumor, he was deemed nasal endoscopy, the nasal cavity contained pooled mucous in to not be a good surgical candidate. e patient began the choana on the left side and a polyp on the posterior aspect concurrent chemoradiation with weekly cisplatin using of the left inferior turbinate. intensity-modulated proton therapy to the primary lesion Magnetic resonance imaging of the temporoman- and later to the iliac bone. Four months later, a new dibular joint showed degenerative changes bilaterally. metastasis appeared to the right ischium. Stereotactic ir- Noncontrast CT of the sinuses revealed an abnormal radiation was initiated. e patient underwent genetic soft tissue lesion within the left maxillary antra with profiling, and it was found that he had a MDM2 mutation;

The Permanente Journal·https://doi.org/10.7812/TPP/20.229 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 155 CASE REPORT Utilization of Nivolumab in Adenoid Cystic Carcinoma After Progression on Platinum-Based Chemotherapy

Figure 1. (A) Abnormal lobular soft tissue lesion within the left maxillary antra with underlying bony erosion extending to erode the lateral wall of the nasal cavity, pterygoid plates with likely extension into the lateral aspect of the left sphenoid sinus with additional underlying erosion of the left lateral sphenoid body. (B) There is fullness in the left pterygopalatine fossa and widening of the left sphenopalatine foramen.

Figure 2. Large hypermetabolic uptake of 8.8 associated with left nasopharynx/maxillary sinus lesion with bone destruction as described, representing the patient’s primary malignancy. Scintigraphic evidence of metastatic lymph node in the right cervical chain with SUV max of 2.7. Focus of hypermetabolic activity in the posterior aspect of the right iliac bone, worrisome for metastatic bone lesion.

he began a phase I trial against this mutation. epatient nivolumab. However, the patient continued to have progressive continued to progress, however, and metastases were found disease and died 12 weeks after nivolumab initiation. in the lung. He began a second trial using milademetan but continued to progress. Immunotherapy is commonly used DISCUSSION AND CONCLUSIONS after proton therapy in head and neck squamous cell cancers, Immunotherapy is a new method of treatment that en- so the patient began the immune check-point inhibitor hances one’s own immune system to attack malignant cells.

156 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.229 CASE REPORT Utilization of Nivolumab in Adenoid Cystic Carcinoma After Progression on Platinum-Based Chemotherapy

Figure 3. (A and B) Both biopsies consistent with cribriform ACC and metastatic ACC.

ere are 2 major types of commercially available immu- nivolumab in this patient with ACC of the head and neck notherapy: PD-1/PD-L1 inhibitors and CTLA-4 immune given the limited options and progression on 2 early-phase checkpoint inhibitors. ese work to promote activation of trials. In addition, a small, single-institution study ex- antineoplastic T-cells, to reduce tumor size, and to increase amined the use of nivolumab in 4 patients with ACC who progression-free survival as well as overall survival.8 had progressed on platinum-based chemotherapy.13 e In 2016, the United States Food and Drug Administration results showed a promising progression-free survival in (FDA) approved of the PD-1 immune checkpoint inhibitors 2 of the patients (8 and 12 months) and an overall survival nivolumab and pembrolizumab in the treatment of patients of 24 months in 1 patient.13 Another study implemented with squamous cell carcinoma (SCC) of the head and neck a phase 2 trial of using single-agent pembrolizumab and in those refractory to platinum-based regimens. e (another PD-1/PD-L1 inhibitor) as salvage therapy in European Commission followed with approval of nivolumab ACC. Five of their 14 patients were alive and progression and pembrolizumab in those with an expression PD-L1 > free at 27 weeks. Of the 14 patients who were medically 50% and progressed on platinum therapy. In 2019, the FDA stable for imaging, response to the treatment was measured granted these treatments as first line in those with metastatic using RECIST criteria. ere was an objective response or unresectable, recurrent head and neck SCC as single rate of 14%.14 Unfortunately, our patient’scoursewasnot agents or as a combination with platinum and fouroruracil.9 as impressive as these aforementioned studies. He lived Commonly, head and neck cancers are SCCs; how- 3 months following immunotherapy treatments. ever, rare cancers of the head and neck, such as ACCs, Immunotherapy has been used in the treatment of met- may occur. ese rare tumors have not shown to be astatic and recurrent head and neck cancers and has shown particularly susceptible to systemic therapies targeted at clinically significant activity in those who have progressed on head and neck SCC.10 Treatment paradigms have been platinum-based chemotherapy. To date, no study has eval- confined to small study designs owing to the rarity of this uated the efficacy of immunotherapy in ACC specifically. tumor.11,12 Our patient demonstrated a survival of 3 months following In this case, our patient with a rare cancer of the head the administration of nivolumab after progressing on 2 early- and neck progressed after using platinum-based therapy. phase trials. ere are many phase II clinical trials studying Because immunotherapy is approved to be used in SCC drug efficacy in those with ACC including but not limited of head and neck, data from these studies were extrapo- to axitinib and avelumab, lenvatinib and pembrolizumab, lated, and it was decided by the treatment team to initiate MYB DNA vaccine and tislelizumab, pembrolizumab and

The Permanente Journal·https://doi.org/10.7812/TPP/20.229 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 157 CASE REPORT Utilization of Nivolumab in Adenoid Cystic Carcinoma After Progression on Platinum-Based Chemotherapy

docetaxel, and recently, nivolumab with ipilimumab and 4. Michot JM, Bigenwald C, Champiat S, et al. Immune-related adverse events with immune 15 checkpoint blockade: A comprehensive review. Eur J Canc 2016 Feb;54:139-48. DOI: radiation therapy. However, limited data exist regarding https://doi.org/10.1016/j.ejca.2015.11.016. the optimal treatment regimens for patients with ACC. 5. Hellmann MD, Ciuleanu TE, Pluzanski A, et al. Nivolumab plus ipilimumab in lung cancer with a high tumor mutational burden. N Engl J Med 2018 May; 378(22):2093-104. DOI: Given the general poor response to cytotoxic chemotherapy https://doi.org/10.1056/NEJMoa1801946, PMID:29658845 used in HNSCC and the favorable toxicity profile of im- 6. Trope M, Triantafillou V, Kohanski MA, et al. Adenoid cystic carcinoma of the sinonasal tract: A review of the national cancer database. Int Forum Allergy Rhinol 2019 Apr;9(4): munotherapy, this therapy may play a role in ACC treat- 427-34. DOI: https://doi.org/10.1002/alr.22255, PMID:30645040. ment. Further research is needed to elucidate the role that 7. Fay AP, Signoretti S, Callea M, et al. Programmed death ligand-1 expression immunotherapy plays in the treatment of this rare cancer. v in adrenocortical carcinoma: An exploratory biomarker study. J Immunother Cancer 2015 Feb;3:3. DOI: https://doi.org/10.1186/s40425-015-0047-3,PMID: 25767716 Informed Consent 8. van den Bulk J, Verdegaal EM, de Miranda NF. Cancer immunotherapy: Broadening Because the patient expired, informed consent was not able to be obtained. An the scope of targetable tumours. Open Biol 2018 Jun;8(6):180037. DOI: https://doi.org/ 10.1098/rsob.180037 effort has been made to anonymize patient information. 9. Cohen EEW, Bell RB, Bifulco CB, et al. The Society for Immunotherapy of Cancer consensus statement on immunotherapy for the treatment of squamous cell carcinoma Disclosure Statement of the head and neck (HNSCC). J Immunother Cancer 2019 Jul;7(1):184. DOI: https:// The author(s) have no conflicts of interest to disclose. doi.org/10.1186/s40425-019-0662-5, PMID:31307547. 10. Roh JL, Lee JI, Choi SH, et al. Prognostic factors and oncologic outcomes of 56 salivary duct carcinoma patients in a single institution: High rate of systemic failure warrants Authors’ Contributions targeted therapy. Oral Oncol 2014 Nov;50(11):e64. DOI: https://doi.org/10.1016/ Diana V Maslov, MD, MS, and Katharine Thomas, MD, MS, wrote this j.oraloncology.2014.08.010, PMID:25218000. manuscript. Marc Matrana, MD, MS, FACP, edited this manuscript. 11. Ross PJ, Teoh EM, A’hern RP, et al. Epirubicin, cisplatin and protracted venous infusion 5-Fluorouracil chemotherapy for advanced salivary adenoid cystic carcinoma. Clin Oncol 2009 May;21(4):311-4. DOI: https://doi.org/10.1016/j.clon.2008.12.009, PMID: Funding Statement 19201585 There was no funding in the preparation or writing of this manuscript. 12. Gilbert J, Li Y, Pinto HA, et al. Phase II trial of taxol in salivary gland malignancies (E1394): A trial of the Eastern Cooperative Oncology Group. Head Neck 2006 Mar;28(3):197-204. DOI: https://doi.org/10.1002/hed.20327, PMID:16470745 References 13. Kokkali S, Ntokou A, Drizou M, et al. Nivolumab in patients with rare head and 1. Adenoid Cystic Carcinoma Organization International. Frequently asked questions about neck carcinomas: A single center’s experience. Clin Cancer Res 2019 Sep;25(17): adenoid cystic carcinoma. Access date 8/21/2020. http://www.accoi.org/faq/#i_was_told_it_is. 5221-30. DOI: https://doi.org/10.1186/s40425-019-0722-x 2. National Center for Advancing Translational Sciences, Genetic and Rare Diseases 14. Habra MA, Stephen B, Campbell M, et al. Phase II clinical trial of pembrolizumab efficacy Information Center (GARD). Adenoid cystic carcinoma. National Institutes of Health; and safety in advanced adrenocortical carcinoma. J Immunother Cancer 2019 Sep;7(1): 2017. Access date 8/20/2020. Rarediseases.info.nih.gov. 253. DOI: https://doi.org/0.1186/s40425-019-0722-x.2, PMID:31533818 3. Net Editorial Board. Adenoid cystic carcinoma: symptoms and signs. Net Editorial Board; 15. Adenoid Cystic Carcinoma Research Foundation. Current Studies. Accessed 8/22/2020. 2017. Cancer.net. https://www.accrf.org/treatment-options/clinical-trials/current-studies/

158 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.229 n CASE REPORT Deadly Neuroinvasive Mosquito-Borne Virus: A Case of Eastern Equine Encephalitis

Natalie Millet, DO1; Saif Faiek, MD1; Daniel Gurrieri, DO1; Karanvir Kals, DO2; William Adams, DO1,3; Edward Hamaty, DO1,3; Manish Trivedi, MD1,4; David Zeidwerg, DO1,5 Perm J 2021;25:20.288 E-pub: 03/01/2021 https://doi.org/10.7812/TPP/20.288

ABSTRACT episode, he was noted to have an altered level of consciousness Eastern equine encephalitis (EEE) is a rare and potentially fatal along with right upper and lower extremity weakness; these neuroinvasive disease with a high mortality rate of > 30%. It is an symptoms were initially thought to be a result of his postictal uncommon vector-borne illness, with an average of 8 cases re- state. Intravenous cancomycin, ceftriaxone, acyclovir, and ported in the United States annually. Alarmingly, in 2019 alone, dexamethasone were initiated, given our high index of fi the Centers for Disease Control and Prevention (CDC) con rmed suspicion for encephalitis versus meningitis. A lumbar 38 cases of EEE virus in the United States, reported from 10 states. puncture was performed with a mildly elevated opening In this report, we describe a 42-year-old man who worked fl primarily in wooded areas and presented to a hospital in southern pressure of 24 cm of water; cerebrospinal uid (CSF) New Jersey with an intractable headache and global facial par- analysis revealed an elevated CSF polymorphonuclear cells aesthesia. He reported multiple tick bites in the weeks prior to his of 11 and normal CSF lymphocytes, and monocytes were 47 presentation. Based on high clinical suspicion, cerebrospinal and 42. e patient had a normal glucose level of 74 mg/dL samples were sent to the CDC, which confirmed the diagnosis of (reference range: 40-70 mg/dL), an elevated protein level of EEE. The patient was treated with supportive care, and, after 104.7 mg/dL (reference range: 15-45 mg/dL), and an elevated spending 9 days on mechanical ventilation in the intensive care serum sodium of 150 (reference range: 135-145 mEq/L). unit, he was extubated and subsequently had some improve- Magnetic resonance imaging of the brain with and without ment of his symptoms with the implementation of an extensive gadolinium contrast demonstrated an area of edema involving physical therapy program. the medial aspect of the left temporal lobe with diffuse ab- We hope this report will contribute to increasing awareness normal T2 signal within the basal ganglia extending into among the public health and medical communities regarding the increasing number of EEE cases and the importance of following the midbrain concerning for encephalitis (Figure 1). prevention measures, especially in areas with high prevalence and early recognition of the disease for treatment. Therapeutic Intervention and Treatment Over the next 16 hours, the patient remained febrile, with temperatures ranging from 102 to 105°F despite antipy- CASE PRESENTATION retics and passive cooling techniques. At approximately 10:00 pm on day 3 of admission, targeted temperature Presenting Concerns management to achieve normothermia was initiated. e On August 28, 2019, at 10:30 am, a 42-year-old male decision was made to insert a prophylactic external ventricular with no significant past medical history who worked pri- drain (EVD) given the increased opening pressure during marily in wooded areas in southern New Jersey presented to lumbar puncture, cerebral edema on imaging, and deteri- our hospital’s emergency department complaining of an oration of the patient’s mental status. Prior to the EVD intractable headache described as the “worst headache of his procedure, the patient’s respiratory condition worsened life.” His headache began the morning of his presentation with paradoxical breathing, accessory muscle use, and a and was associated with global facial paresthesia, nausea, respiratory rate in the fifties. He was intubated at 11:30 pm and generalized malaise. e patient reported multiple tick bites in the weeks preceding his presentation. He was started on intravenous doxycycline on admission for pre- ffi sumed tick-borne illness. Despite reporting improvement in Author A liations 1Department of Medicine, AtlantiCare Regional Medical Center, Atlantic City, NJ his symptoms with supportive care on day 1 of admission, 2Rowan University School of Osteopathic Medicine, Stanford, NJ on day 2 of admission, his headaches recurred and worsened 3Department of Critical Care, AtlantiCare Regional Medical Center, Atlantic City, NJ in severity in the afternoon. He also became febrile, with 4Division of Infectious Disease, AtlantiCare Regional Medical Center, Atlantic City, NJ 5 body temperature as high as 102°F. At 3:28 am on day 3 of Division of Neurology Medicine, AtlantiCare Regional Medical Center, Atlantic City, NJ admission, a rapid response was called because the patient Corresponding Author was actively seizing. He received a total of 8 mg intravenous Saif Faiek, MD ([email protected]) lorazepam with a termination of his seizure. He also started Keywords: arbovirus, eastern equine encephalitis, mosquito-borne virus, Centers for Disease Control and on 1 g of levetiracetam every 12 hours. After the seizure Prevention, CSF, cerebrospinal fluid, EEE, eastern equine encephalitis, EEEV, eastern equine encephalitis virus, EVD, external ventricular drain The Permanente Journal·https://doi.org/10.7812/TPP/20.288 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 159 CASE REPORT Deadly Neuroinvasive Mosquito-Borne Virus: A Case of Eastern Equine Encephalitis

Aedes, Coquillettidia, and Culex species are responsible for transmission to humans to create a bridge between virus- infected birds (primary host) and humans (incidental host). e Culiseta melanura is the vector that maintains the EEEV primary transmission cycle in birds. Cross transmission from its usual reservoirs to other hosts such as humans, horses, swine, and exotic birds occurs at unpredictable intervals. Factors that increase the risk of these transmissions are thought to include complex interaction among human be- haviors, weather, habitat destruction, bird migration, and other variables.2 Affected humans and horses are con- sidered dead-end hosts because they do not develop sufficient viremia levels to infect other susceptible hosts.1 Although infections can occur throughout the year, the peak incidence is in August and September (as happened in the case report), mostly along the Atlantic and Gulf coasts. EEE-infected patients usually present with nonspecific signs and symptoms, including fever, malaise, severe headache, muscle aches, nausea, and vomiting after a 7-10-day in- cubation period.2 When neurological symptoms related to Figure 1. T2 FLAIR MRI of the brain. T1 MI of the brain. encephalitis develop, the clinical condition usually worsens rapidly, with 90% of patients progressing to comatose or stuporous. One-half of the patients develop seizures or focal on day 3 of admission. CSF samples obtained from the EVD neurologic signs. EEE neuroinvasive disease is estimated to were sent to the Centers for Disease Control and Prevention have a case-fatality rate of 30% or higher, with approxi- (CDC), which returned positive for IgM and IgG EEE virus mately 50% of survivors left with debilitating neurological (EEEV) antibodies, confirming the EEEV diagnosis. Blood sequelae.3 In the absence of a human vaccine against EEEV and CSF cultures demonstrated no growth. e patient was and no available antiviral therapies, treatment is primarily treated with supportive care and was successfully extubated supportive.4,5 Between 2003 and 2018, an average of 8 EEE after 9 days of mechanical ventilation. cases were reported annually in the United States, with a range of 4-21 cases per year.3 However, as of December 17, Follow-Up and Outcomes 2019, CDC has received reports of 38 cases of EEE disease Postextubation, he exhibited significant neurological in 2019 alone. Cases were reported from 10 states, including deficits, including moderate aphasia, dysphagia, and global Alabama, Connecticut, Georgia, Indiana, Massachusetts, weakness. He was transferred to a long-term rehabilitation Michigan, New Jersey, North Carolina (1), Rhode Island, center where he underwent aggressive physical and occu- and Tennessee.6 pational therapy with some improvement in his neurologic Healthcare providers should consider EEE infection in and clinical status. e patient followed up with the out- the differential diagnosis of cases concerning for meningitis patient neurology clinic. He had an improvement in his and encephalitis, especially in swamp areas where EEEV weakness and dysphagia but continued to have a moderate mosquito vectors are found. Suspicion for EEE should expressive aphasia as a sequelae of the disease. prompt an urgent workup with the collection of CSF specimens and appropriate imaging. Polymerase chain DISCUSSION reaction analysis from blood and spinal fluid and testing EEEV is a mosquito-borne arbovirus that is considered for EEEV-specific IgM are usually used to confirm the one of the most severe and potentially fatal arboviral en- diagnosis. Imaging can support the diagnosis while de- cephalitides in North America. It consists of 2 subtypes: finitive testing is pending. Magnetic resonance imaging 1) EEEV subtype I, which is found in North America typically demonstrates the involvement of the basal and the Caribbean, and 2) Madariaga virus (EEEV subtype ganglia and thalami, similar to our patient.2 II-IV), which is found in South and Central America. It is transmitted to humans primarily from Aedes or Coquillettidia CONCLUSION mosquitoes. EEEV is maintained in a transmission cycle Providers are encouraged to report suspected EEE between mosquitoes and birds in freshwater swamps.1 infections to their state or local health department to

160 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.288 CASE REPORT Deadly Neuroinvasive Mosquito-Borne Virus: A Case of Eastern Equine Encephalitis

Table 1. Timeline table Relevant past medical history and interventions A 42-year-old male with no significant past medical history who worked primarily in wooded areas in southern New Jersey presented to our hospital’s emergency department complaining of an intractable headache associated with global facial paresthesia, nausea, and generalized malaise. The patient reported multiple tick bites in the weeks preceding his presentation. Date Summaries from initial and follow-up visits Diagnostic testing (including dates) Interventions August 28, 2019 Patient presented with the above-mentioned No relevant testing He was Started on intravenous doxycycline on complaints and symptoms. admission for presumed tickborne illness. August 29, 2019 His headaches reoccurred and worsened in No relevant testing He received antipyretics. severity with, high temperature 102°F. August 30, 2019 A rapid response was called because the patient No relevant testing He received a total of 8 mg intravenous was actively seizing. lorazepam with a termination of his seizure. He was also started on 1 g of levetiracetam every 12 h. August 30, 2019 After the seizure episode, he was noted to have A lumbar puncture was performed with a mildly Intravenous vancomycin, ceftriaxone, an altered level of consciousness along with elevated opening pressure of 24 cm of water; acyclovir, and dexamethasone were initiated, right upper and lower extremity weakness; these CSF analysis revealed a normal glucose level of given our high index of suspicion for symptoms were initially thought to be a result of 74 mg/dL (reference rage: 40-70 mg/dL) and an encephalitis versus meningitis. his postictal state. elevated protein level of 104.7 mg/dL (reference range: 15-45 mg/dL). Magnetic resonance imaging of the brain with and without gadolinium contrast demonstrated an area of edema involving the medial aspect of the left temporal lobe with diffuse abnormal T2 signal within the basal ganglia extending into the midbrain concerning for encephalitis. August 30, 2019 The patient remained febrile with temperatures CSF samples obtained from the EVD were sent The decision was made to insert a prophylactic ranging from 102 to 105°F despite antipyretics to the CDC, which later returned positive for IgM EVD given the increased opening pressure and passive cooling techniques. At and IgG EEEV antibodies confirming the EEEV during lumbar puncture, cerebral edema on approximately 10:00 pm on day 3 of admission, diagnosis. Blood and CSF cultures imaging, and deterioration of the patient’s targeted temperature management to achieve demonstrated no growth. mental status. Prior to the EVD procedure, the normothermia was initiated. patient’s respiratory condition worsened with paradoxical breathing, accessory muscle use, and a respiratory rate in the fifties. He was subsequently intubated at 11:30 pm on day 3 of admission. September 9, The patient was treated with supportive care 2019 and was successfully extubated after 9 days of mechanical ventilation. Follow-up Postextubation, he exhibited significant neurological deficits, including moderate aphasia, dysphagia, and global weakness. He was transferred to a long-term rehabilitation center where he underwent aggressive physical and occupational therapy with progressive improvement in his neurologic and clinical status. CDC = Centers for Disease Control and Prevention; CSF = cerebral spinal fluid; EEEV = eastern equine encephalitis virus; EVD = external ventricular drain.

facilitate diagnosis. Prevention of EEE depends on the Authors’ Contributions community to reduce mosquito populations and pro- All of the authors participated in evaluating the patient and writing the case report. tective measures to decrease exposure to mosquitoes. Increased public awareness and implementation of vec- Funding Statement tor control to mitigate the risk for further transmission The author(s) have no funding source to disclose. will be essential in reducing the risk of new EEEV References outbreaks. v 1. Lindsey NP, Staples JE, Fischer M. Eastern equine encephalitis virus in the United States, 2003- 2016. Am J Trop Med Hyg 2018 May;98(5):1472-7. DOI: https://doi.org/10.4269/ajtmh.17-0927 2. Morens DM, Folkers GK, Fauci AS. Eastern equine encephalitis virus - another emergent Disclosure Statement arbovirus in the United States. N Engl J Med 2019 Nov;381(21):1989-92. DOI: https://doi. The author(s) have no conflicts of interest to disclose. org/10.1056/NEJMp1914328

The Permanente Journal·https://doi.org/10.7812/TPP/20.288 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 161 CASE REPORT Deadly Neuroinvasive Mosquito-Borne Virus: A Case of Eastern Equine Encephalitis

3. Garlick J, Lee TJ, Shepherd P, et al. Locally acquired eastern equine encephalitis virus 5. Jonsson CB, Cao X, Lee J, et al. Efficacy of a ML336 derivative against Venezuelan and disease, Arkansas, USA. Emerg Infect Dis 2016 Dec;22(12):2216–7. DOI: https://doi.org/ eastern equine encephalitis viruses. Antivir Res 2019 Jul;167:25–34. DOI: https://doi.org/ 10.3201/eid2212.160844 10.1016/j.antiviral.2019.04.004 4. Lindsey NP, Martin SW, Staples JE, Fischer M. Notes from the field: Multistate outbreak of 6. CDC. Eastern equine encephalitis virus: US Department of Health and eastern equine encephalitis virus - United States, 2019. MMWR Morb Mortal Wkly Rep Human Services, CDC; 2019. https://www.cdc.gov/easternequineencephalitis/ 2020 Jan;69(2):50–1. DOI: https://doi.org/10.15585/mmwr.mm6902a4 index.html

162 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.288 n CASE REPORT Jejunal Gastrointestinal Stromal Tumor as a Source of Small Bowel Bleeding: A Case Report

Jacob Burch, DO1; Iftiker Ahmad, MD2 Perm J 2021;25:20.257 E-pub: 03/17/2021 https://doi.org/10.7812/TPP/20.257

ABSTRACT CASE PRESENTATION Introduction: In a minority of patients with gastrointestinal Our patient is a 76-year-old man with a past medical bleeding, the offending lesion is not able to be identified using history of hypertension, coronary artery disease, myocardial colonoscopy or esophagogastroduodenoscopy (EGD). For these infarction, gastroesophageal reflux disease, and cataracts. patients, video capsule endoscopy has become a cornerstone for He had previously undergone single-vessel coronary artery the diagnosis of gastrointestinal bleeding in the territory not bypass grafting on 2 occasions and had subsequently un- accessible by colonoscopy or EGD. One uncommon cause of dergone coronary artery stenting 5 times. bleeding from the small bowel is a gastrointestinal stromal tumor.  Case Presentation: We present the case of a 76-year-old man e patient presented to the outpatient gastroenterology who presented with 2 weeks of melena that began after starting clinic with complaints of 2 weeks of melena that had re- dual antiplatelet therapy with aspirin and clopidogrel after un- solved 1 week before being seen (Table 1). His melena dergoing coronary artery stenting. After EGD and colonoscopy started following a heart catheterization with stenting and failed to identify the culprit, the patient underwent video capsule initiation of dual antiplatelet therapy. His complete blood endoscopy, which identified a suspicious area concerning for count on the day of presentation showed a hemoglobin of intussusception. Computed tomography enterography was then 7.9 g/dL (normal: 12.6-16.5 g/dL). e patient underwent performed and showed a short segment of bowel wall thickening. EGD 4 days later, which showed no abnormalities. Repeat The patient underwent laparoscopic small bowel resection and complete blood count on the day of his EGD revealed was found to have a gastrointestinal stromal tumor. hemoglobin of 6.7 g/dL. Following the completion of his EGD, the patient was admitted to the hospital for trans- INTRODUCTION fusion of 2 units of packed red blood cells. He underwent Obscure gastrointestinal bleeding is defined by the colonoscopy the following day, which also found no source American Gastroenterological Association as bleeding for his melena. Capsule endoscopy was performed 1 week from the gastrointestinal tract that persists or recurs later and revealed a questionable segment of intussusception without an obvious etiology after esophagogastroduodeno- of the terminal ileum (Figure 1). Computed tomography scopy (EGD), colonoscopy, and radiologic evaluation of the (CT) enterography with contrast was performed 1 month small bowel.1 later and revealed a short segment of hyperenhancement In patients presenting with gastrointestinal bleeding, and eccentric, mural bowel wall thickening measuring up to EGD and colonoscopy fail to identify the offending le- 8 mm in the left lower quadrant, likely corresponding to the 2 distal jejunum (Figure 2). sion(s) in 5% of cases. Video capsule endoscopy (VCE) and  double-balloon enteroscopy, however, have proven capable e patient underwent laparoscopic segmental small of finding the source of bleeding in approximately 75% of bowel resection 2 months after his initial presentation. these cases.1,3 e most common causes of small bowel Duringthecourseofsurgery,hewasfoundtohavea bleeding include inflammatory bowel disease, angiodys- freely moveable 3.3 × 3.0 × 2.7 cm mass in the mid- plasia, Dieulafoy lesions, neoplasms, ulcerations, and Meckel jejunum, which was resected. Histology demonstrated a diverticulum.3 Among this group, the most common mixed-type GIST with predominantly spindle cell pro- 4,5 liferation that stained positive for CD117 and discovered etiology is angiodysplasia. While neoplasms represent  fi the second most common cause of small bowel bleeding, on GIST 1 (DOG1) (Figure 3). e nal tumor staging they still comprise only 5% to 10% of cases.4 Even more rare among this group is the gastrointestinal stromal tumor (GIST), which accounts for only 7.1% of small bowel malignancies.6 We present the case of a 76-year- Author Affiliations 1Michigan State University Internal Medicine Residency, Sparrow Hospital, East Lansing, MI old man who was found to have a jejunal GIST as the 2Sparrow Hospital/Michigan State University College of Human Medicine Gastroenterology Fellowship, source of gastrointestinal bleeding. is case highlights Michigan Gastroenterology Institute, East Lansing, MI the critical role that VCE can play in the diagnosis of obscure gastrointestinal bleeding. is case was prepared Corresponding Author Jacob Burch, DO ([email protected]) following CARE guidelines.7 Keywords: case report, gastrointestinal bleeding, gastrointestinal stromal tumor, video capsule endoscopy The Permanente Journal·https://doi.org/10.7812/TPP/20.257 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 163 CASE REPORT Jejunal Gastrointestinal Stromal Tumor as a Source of Small Bowel Bleeding: A Case Report

Table 1. Case report timeline Date Summaries from initial and follow-up visitsa Diagnostic testing (including dates) Interventions June 17, 2019 Patient presented 1 wk following the resolution of CBC showed hemoglobin of 7.9 g/dL (normal: Scheduled for diagnostic EGD 2 wk of melena that began after starting aspirin 12.6-16.5 g/dL) and clopidogrel following coronary artery stenting June 21, 2019 EGD performed; no abnormalities seen CBC repeated, with hemoglobin of 6.7 g/dL Patient hospitalized for transfusion of 2 units of PRBCs June 22, 2019 Colonoscopy performed during hospitalization; no CBC repeated, with hemoglobin of 8.6 g/dL Patient discharged following normal abnormalities seen colonoscopy and stabilization of hemoglobin June 28, 2019 Capsule endoscopy performed: questionable Patient scheduled for CTE to further evaluate segment of intussusception seen at terminal ileum possible segment of intussusception July 19, 2019 CTE hyperenhancement and bowel wall Referred to general surgery for excision of thickening in the left lower quadrant, likely concerning segment of bowel corresponding to the distal jejunum August 21, 2019 3.3 × 3.0 × 2.7 cm resected from jejunum Histology significant for spindle cell GIST with CD117 and DOG1 staining aWith regard to the patient’s relevant past medical history and interventions, he had hypertension, coronary artery disease, myocardial infarction, gastroesophageal reflux disease, and cataracts. The patient had undergone single-vessel coronary artery bypass graft 2 times and subsequent coronary artery stenting 5 times and was taking aspirin and clopidogrel. CBC = complete blood count; CTE = computed tomography enterography; DOG1 = discovered on GIST 1; EGD = endoscopic gastroduodenoscopy; GIST = gastrointestinal stromal tumor; PRBC = packed red blood cell.

was T2N0M0 and the patient did not require adjuvant receptors encoded by these genes.9 eactivationofthese therapy. tyrosine kinase receptors subsequently leads to neoplastic growth. DISCUSSION edefinitive diagnosis of GISTs is based on both the GISTs are typically benign, mesenchymal tumors found morphologic and immunohistochemical findings.10,16 e3 throughout the entirety of the gastrointestinal tract. Given morphologic patterns of GISTs are spindle cell (70%), that just 30% are malignant, GISTs represent only 0.1% to epithelioid cell (20%), or mixed (10%).10,12,16 Spindle cell 3% of gastrointestinal malignancies despite making up 80% GISTs are composed of short fascicles or whorls of cells. of gastrointestinal mesenchymal tumors.8,9 e stomach Epithelioid-type GISTs are composed of diffuse or nested represents the most common primary location of GISTs, epithelioid cells. Mixed-type GISTs demonstrate a accounting for approximately 60% of cases, while the small bowel is the second most common, accounting for 20% to 30% of GISTs.10-13 Despite only accounting for 20% to 30% of GISTs, an analysis of 2015 US cancer statistics found that the incidence of localized and small bowel GISTs has been increasing since the turn of the 21st century.13 Approximately 70% of patients with GISTs have symptoms related to their disease, 20% of patients remain asymptomatic, and 10% of GISTs are found on autopsy.14 e most common presentations associated with GISTs are gastrointestinal bleeding with associated anemia, weakness, and abdominal pain.15,16 GISTs are known to arise from the same lineage as the interstitial cells of Cajal; however, the specific cell type from which they arise remains unknown.9,17 While GISTs are most commonly sporadic, they can also be associated with inherited tumor syndromes such as neurofibromatosis type 1, Carney-Stratakis syndrome, and Carney triad.9,15,17,18 Approximately 85% of sporadic GISTs are caused by mutations of either KIT (CD117) or platelet-derived growth factor receptor alpha. ese mutually exclusive mutations subsequently lead to activation of the tyrosine kinase Figure 1. Capsule endoscopy showing a questionable segment of intussusception.

164 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.257 CASE REPORT Jejunal Gastrointestinal Stromal Tumor as a Source of Small Bowel Bleeding: A Case Report

Figure 3. mixed-type GIST with predominantly spindle cell proliferation seen on 40x magnification with hematoxylin and eosin stain.

by these other modalities. For patients presenting with ab- dominal pain or obstructive symptoms, CT imaging may be the initial modality employed to evaluate the symptoms and may identify a mass, but it does not offer a definitive di- agnosis. Contrast-enhanced CT of the abdomen and pelvis Figure 2. CT enterography with contrast showing a short segment of also serves as the recommended imaging modality for hyperenhancement and eccentric, mural bowel wall thickening measuring up to evaluating and staging patients with known GIST, per- 8 mm in the left lower quadrant. forming surveillance after surgical resection, and monitoring treatment response in patients receiving adjuvant or neo- combination of spindle and epithelioid cell architectural adjuvant therapy.12,20 Positron emission tomography-CT patterns.12 e diagnosis of GISTs is confirmed with may be useful in evaluating for early response to tyrosine immunohistochemical markers. KIT (CD117) is present kinase inhibitors, but it does not provide a significant ad- in approximately 95% of GISTs and serves as a confir- vantage over contrast-enhanced CT of the abdomen and mation of this diagnosis. In the absence of KIT, DOG1, pelvis in the routine evaluation of patients with GISTs and also known as anoctamin, or CD34 can be used to may actually lead to more poorly defined images; therefore, confirm this diagnosis.12,16,19 In rare cases in which GIST positron emission tomography-CT does not represent the is strongly suspected but testing for KIT, DOG1, and preferred imaging for patients with GISTs.12,20 CD34 is negative, mutational analysis of known KIT or Despite significant advancements in medical therapies platelet-derived growth factor receptor alpha mutations for GISTs, surgical resection remains the cornerstone of can be used to confirm the diagnosis.19 therapy for resectable GISTs. In easily resectable tumors e diagnostic workup of GISTs is often determined by greater than 2 cm, the therapeutic goal is complete resection size, location, and presenting symptoms. For patients who obtaining negative margins without rupture of the tumor’s present with the most common symptom of GISTs, gas- pseudocapsule.12,18-20 Due to the low propensity for GISTs to trointestinal bleeding (and sequalae), the diagnostic workup spread lymphatically, lymph node resection is not required in often begins with endoscopy. On endoscopy, GISTs will all cases, and it is recommended to pursue this only in patients present as a subepithelial lesion.16 For patients with a with clinical suspicion of lymph node involvement.16,19,20 For suspicious subepithelial lesion seen on endoscopy, endo- patients without metastatic disease but in whom surgical re- scopic ultrasound is the best method of obtaining a biopsy section cannot be easily obtained, neoadjuvant therapy can be for pathologic diagnosis prior to surgical resection. For pa- used to make the tumor more amenable to resection.18,21 tients with gastrointestinal bleeding in whom no offending e discovery of the role that KIT mutations play in the lesions can be seen on endoscopic evaluation with EGD and development of GISTs has been pivotal not only in the colonoscopy, VCE and double-balloon enteroscopy are diagnosis of GISTs but also in their treatment. Tyrosine additional methods that can be used to try to identify the kinase inhibitors have come to comprise the backbone of culprit in the region of the gastrointestinal tract not accessible medical therapy for GISTs. e tyrosine kinase inhibitor

The Permanente Journal·https://doi.org/10.7812/TPP/20.257 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 165 CASE REPORT Jejunal Gastrointestinal Stromal Tumor as a Source of Small Bowel Bleeding: A Case Report

imatinib serves as first-line medical therapy for neoadjuvant, 4. Gunjan D, Sharma V, Rana SS, Bhasin DK. Small bowel bleeding: A comprehensive review. Gastroenterol Rep (Oxf) 2014 Nov;2(4):262-75. DOI: https://doi.org/10.1093/ adjuvant, and advanced GISTs. Adjuvant therapy following gastro/gou025, PMID:24874805 successful surgical resection should be reserved for those 5. Sakai E, Ohata K, Nakajima A, Matsuhashi N. Diagnosis and therapeutic strategies for 12,18,19 small bowel vascular lesions. World J Gastroenterol 2019 Jun;25(22):2720-33. DOI: with high-risk disease. For patients who do not re- https://doi.org/10.3748/wjg.v25.i22.2720, PMID:31235995 spond to initial medical therapy with imatinib, the dose of 6. Hatzaras I, Palesty JA, Abir F, et al. Small-bowel tumors: Epidemiologic and clinical characteristics of 1260 cases from the Connecticut tumor registry. Arch Surg 2007 Mar; imatinib can be increased or the patient can be transitioned 142(3):229-35. DOI: https://doi.org/10.1001/archsurg.142.3.229, PMID:17372046 to second-line therapy. e second-line agent for the 7. Riley DS, Barber MS, Kienle GS, et al. CARE guidelines for case reports: Explanation and medical therapy of GISTs is also a tyrosine kinase inhibitor, elaboration document. J Clin Epidemiol 2017 Sep;89:218-35. DOI: https://doi.org/10. 1016/jclinepi.2017.04.026. 12,18-20 sunitinib. 8. El-Menyar A, Mekkodathil A, Al-Thani H. Diagnosis and management of gastrointestinal stromal tumors: An up-to-date literature review. J Cancer Res Ther 2017 Oct-Dec;13(6): 889-900. DOI: https://doi.org/10.4103/0973-1482.177499, PMID:29237949 CONCLUSION 9. Schaefer IM, Mariño-Enríquez A, Fletcher JA. What is new in gastrointestinal stromal Overlapping timelines in the approval of VCE as well as the tumor? Adv Anat Pathol 2017 Sep;24(5):259-67. DOI: https://doi.org/10.1097/PAP. fi 0000000000000158, PMID:28632504 classi cation and treatment of GIST have led to paralleled 10. Nishida T, Blay J-Y, Hirota S, Kitagawa Y, Kang Y-K. The standard diagnosis, treatment, growth in each of these fields. We feel that these paralleled and follow-up of gastrointestinal stromal tumors based on guidelines. Gastric Cancer growths likely contribute to the previously described increases 2016 Jan;19(1):3-14. DOI: https://doi.org/10.1007/s10120-015-0526-8 11. Miettinen M, Makhlouf H, Sobin LH, Lasota J. Gastrointestinal stromal tumors of the in the incidence of localized and small intestinal GISTs. It is jejunum and ileum: A clinicopathologic, immunohistochemical, and molecular genetic our hope that improved recognition of GISTs as a source of study of 906 cases before imatinib with long-term follow-up. Am J Surg Pathol 2006 Apr; 30(4):477-89. DOI: https://doi.org/10.1097/00000478-200604000-00008 gastrointestinal bleeding and the role VCE can play in this 12. Demetri GD, von Mehren M, Antonescu CR, et al. NCCN Task Force report: Update on diagnosis will lead to more timely diagnosis, treatment, and the management of patients with gastrointestinal stromal tumors. J Natl Compr Canc Netw v 2010 Apr;8:S1-41; quiz S42-4. DOI: https://doi.org/10.6004/jnccn.2010.0116 ultimately the survival of small bowel GISTs. 13. Patel N, Benipal B. Incidence of gastrointestinal stromal tumors in the United States from 2001-2015: A United States cancer statistics analysis of 50 states. Cureus 2019 Feb; Disclosure Statement 11(2):e4120. DOI: https://doi.org/10.7759/cureus.4120 The author(s) have no conflicts of interest to disclose. 14. Gold JS, Dematteo RP. Combined surgical and molecular therapy: The gastrointestinal stromal tumor model. Ann Surg 2006 Aug;244(2):176-84. DOI: https://doi.org/10.1097/01. sla.0000218080.94145.cf, PMID:16858179 Authors’ Contributions 15. Joensuu H, Hohenberger P, Corless CL. Gastrointestinal stromal tumour. Lancet 2013 Jacob Burch, DO, participated in drafting and submitting the manuscript. Iftiker Sep;382(9896):973-83. DOI: https://doi.org/10.1016/S0140-6736(13)60106-3, PMID: Ahmad, MD, reviewed and edited the manuscript. All authors have given final 23623056 approval to the manuscript. 16. Akahoshi K, Oya M, Koga T, Shiratsuchi Y. Current clinical management of gastrointestinal stromal tumor. World J Gastroenterol 2018 Jul;24(26):2806-17. DOI: https://doi.org/10.3748/wjg.v24.i26.2806, PMID:30018476 Financial Support 17. Miettinen M, Lasota J. Histopathology of gastrointestinal stromal tumor. J Surg Oncol 2011 No financial support was received for this work. Dec;104(8):865-73. DOI: https://doi.org/10.1002/jso.21945, PMID:22069171 18. Rutkowski P, Gronchi A, Hohenberger P, et al. Neoadjuvant imatinib in locally advanced References gastrointestinal stromal tumors (GIST): The EORTC STBSG experience. Ann Surg Oncol 1. Raju GS, Gerson L, Das A, Lewis B; American Gastroenterological Association. American 2013 Sep;20(9):2937-43. DOI: https://doi.org/10.1245/s10434-013-3013-7, PMID:23760587 Gastroenterological Association (AGA) Institute medical position statement on obscure 19. Koo DH, Ryu MH, Kim KM, et al. Asian consensus guidelines for the diagnosis and gastrointestinal bleeding. Gastroenterology 2007 Nov;133(5):1694-6. DOI: https://doi.org/ management of gastrointestinal stromal tumor. Cancer Res Treat 2016 Oct;48(4): 10.1053/j.gastro.2007.06.008, PMID:17983811 1155-66. DOI: https://doi.org/10.4143/crt.2016.187, PMID:27384163 2. Szold A, Katz LB, Lewis BS. Surgical approach to occult gastrointestinal bleeding. Am J 20. Casali PG, Abecassis N, Aro HT, et al. Gastrointestinal stromal tumours: ESMO- Surg 1992 Jan;163(1):90-3. DOI: https://doi.org/10.1016/0002-9610(92)90258-s, PMID: EURACAN clinical practice guidelines for diagnosis, treatment and follow-up. Ann Oncol 1733379 2018 Oct;29(Suppl 4):iv68-78. DOI: https://doi.org/10.1093/annonc/mdy095 3. Gerson LB, Fidler JL, Cave DR, Leighton JA. ACG clinical guideline: Diagnosis and 21. Iwatsuki M, Harada K, Iwagami S, et al. Neoadjuvant and adjuvant therapy for management of small bowel bleeding. Am J Gastroenterol 2015 Sep;110(9):1265-88. gastrointestinal stromal tumors. Ann Gastroenterol Surg 2019 Jan;3(1):43-9. PMID: DOI: https://doi.org/10.1038/ajg.2015.246, PMID:26303132 30697609. DOI: https://doi.org/10.1002/ags3.12211

166 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.257 n CASE REPORT Rapid Response of a BRCA2/TP53/PTEN-Deleted Metastatic Uterine Leiomyosarcoma to Olaparib: A Case Report

Minggui Pan, MD, PhD1,2; Kristen Ganjoo, MD3; Amer Karam, MD4 Perm J 2021;25:20.251 E-pub: 3/10/2021 https://doi.org/10.7812/TPP/20.251

ABSTRACT single-gene BRCA2 analysis was performed, which revealed Patients with metastatic uterine leiomyosarcoma (uLMS) have no pathologic BRCA2 mutation but revealed a variant poor prognosis due to limited treatment options, especially when of unknown significance in the BRCA2 gene (specifically disease progresses on doxorubicin and gemcitabine-docetaxel p.P3292L). ere was no significant family history of regimens. Here we report a patient whose metastatic uLMS cancer. e patient was evaluated by both gynecologic BRCA2 TP53 PTEN contains a deep deletion as well as and deep oncology and medical oncology, and the decision was made deletion. The patient responded rapidly to olaparib, a poly (ADP- to initiate neoadjuvant chemotherapy followed by surgical ribose) polymerase inhibitor, after progressing on gemcitabine- docetaxel, doxorubicin, and temozolomide regimens. This case resection. However, after 2 cycles of chemotherapy with report shall be helpful to the treatment of other patients with gemcitabine and docetaxel, the disease progressed, and the  metastatic uLMS that harbors a BRCA2 mutation or deletion. patient experienced worsening pain. e patient had complete resection of the mass, with total hysterectomy and bilateral salpingo-oophorectomies as well as resection of the INTRODUCTION lytic bony lesions, in August 2019. e patient received 4 Uterine leiomyosarcoma (uLMS) is the most common cycles of doxorubicin single agent for adjuvant therapy after histologic subtype of uterine sarcoma, accounting for more the surgical resection. However, in April 2020, 3 months  than 60% of cases. e majority of cases are high grade with after she discontinued doxorubicin, she developed rapid  an aggressive course. e risk of relapse is high after surgical progression with numerous pulmonary metastasis. She was resection of early-stage disease. Treatment options are limited given 2 cycles of oral temozolomide with disease progres- after disease progression on doxorubicin and gemcitabine/ sion. In June 2020, she started olaparib 300 milligram, twice docetaxel regimens. Other agents, such as trabectedin, a day. PET scan performed 6 weeks after the initiation of temozolomide, and pazopanib, have response rates below olaparib showed major partial response (Figure 2A and B). 1-3 10%. We report a case of metastatic uLMS with somatic She continues to tolerate olaparib well without significant deep deletion of BRCA2 that responds rapidly to the poly side effects. Her cough and shortness of breath improved (ADP-ribose) polymerase (PARP) inhibitor olaparib after dramatically a few days after she started olaparib. Eight progressing on doxorubicin, gemcitabine, and docetaxel and months after she initiated olaparib, she continues to oral temozolomide. experience quality of life improvement, walking 5-7 miles a day without shortness of breath. e timeline CASE PRESENTATION of her diagnostic and therapeutic events is summarized in A 54-year-old Black woman presented with pelvic pain in Table 1. May 2019 and was found to have a solid heterogeneous pelvic mass with areas of fluid and hemorrhagic density. e mass measured approximately 20.9 × 22.1 × 12.4 cm on PATIENT PERSCPECTIVE ff ff a computer-aided topography. A positron emission topog- Olaparib was o ered to me after three di erent chemotherapy regimens had failed. My biggest symptoms were shortness of raphy (PET) scan showed the mass to be hypermetabolic breath and pain in the areas of my body where tumors were and arising from the uterus with regions of hypoattenuation growing. The PET scan confirmed that my symptoms were related and photopenia indicating cystic/necrotic changes as well as to the aggressive growth of the tumors. fluorodeoxyglucose-avid lytic expansile right anterior ace- tabular and superior ramus lesion (Figure 1). A biopsy of the mass revealed uterine leiomyosarcoma. StrataNGS (Ann fi Author Affiliations Arbor, Michigan) molecular pro ling of the tumor showed 1  Department of Oncology and Hematology, Kaiser Permanente, Santa Clara, CA deep deletion of BRCA2, PTEN, and TP53 genes. e 2Division of Research, Kaiser Permanente, Oakland, CA StrataNGS result also showed microsatellite stable, tumor 3Division of Oncology, Stanford University School of Medicine, Stanford, CA mutation burden of 8, low programmed death-ligand 1 4Division of Gynecologic Oncology, Stanford University School of Medicine, Stanford, CA expression, and low immune signature profile. She was Corresponding Author evaluated by a genetic counselor, and germline testing of Minggui Pan ([email protected])

Keywords: BRCA2, Olaparib, PARP inhibitor, uterine leiomyosarcoma The Permanente Journal·https://doi.org/10.7812/TPP/20.251 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 167 CASE REPORT Rapid Response of a BRCA2/TP53/PTEN-Deleted Metastatic Uterine Leiomyosarcoma to Olaparib: A Case Report

Table 1. Timeline of the major diagnostic and therapeutic events Timeline Events May 2019 Patient presents with symptoms July 2019 PET scan showed a large uterus mass and bony metastasis July 2019 Biopsy revealed uLMS July 2019 Gemcitabine-docetaxel initiated August 2019 Complete resection of the uLMS and bony metastasis September 2019 Doxorubicin initiated April 2020 PET scan showed progression April 2020 Temozolomide initiated June 2020 Olaparib initiated July 2020 PET scan showed major response

DISCUSSION BRCA1 and BRCA2 play key roles in double-strand Figure 1. Metastatic uLMS at the diagnosis. Blue arrows point to the very large DNA repair by homologous recombination, whereas fluorodeoxyglucose-avid large tumor mass in the abdomen/pelvis with PARP is important in single-strand DNA repair by base tumor necrosis and the bony metastasis to the acetabulum and superior ramus. excision. Inhibition of PARP in cancer cells that are de- ficient in BRCA1 or BRCA2 leads to synthetic lethality as I started taking Olaparib and the side effects I experienced were cancer cells lose the ability to repair both single-strand and fatigue, dizziness, tingling, metallic taste in my mouth, trouble double-strand DNA breaks.4-6 Several PARP inhibitors, with memory and attention as well as aching joints. I found that including olaparib, have shown significant efficacy in the tingling and aching joints got better within weeks, as did the metastatic ovarian, breast, prostate, and pancreatic cancers fatigue. I still experience some dizziness and some trouble with with germline or somatic BRCA1/2 mutation.7-9 BRCA1/2 focus and distraction. The biggest change was that the shortness mutation is uncommon in soft tissue sarcomas (< 1%). of breath started to go away and I went from not being able to However, Seligson et al10 reported a BRCA1/2 mutation walk more than 1/4 mile to averaging between 3-7 miles most in6outof61casesofuLMSthroughreviewingthe days of the week, including some uphill climbs. Notably, from being pushed in a wheelchair a year ago to being able to walk up Cancer Genome Atlas and data from Ohio State Uni-  four floors to the oncology department of Kaiser was a huge versity. ree of 6 cases with BRCA2 mutation treated accomplishment for me. with olaparib had stable disease, and 1 had a partial response.

Figure 2. Representative image of metastatic uLMS to the lungs (A) prior to and (B) after treatment with olaparib. Note the fluorodeoxyglucose-avid lesions in the bilateral lungs are dramatically reduced after olaparib treatment.

168 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.251 CASE REPORT Rapid Response of a BRCA2/TP53/PTEN-Deleted Metastatic Uterine Leiomyosarcoma to Olaparib: A Case Report

Our patient experienced rapid symptomatic improve- References ment and imaging response after initiating olaparib. In- 1. Hensley ML, Blessing JA, Mannel R, Rose PG. Fixed-dose rate gemcitabine plus docetaxel as first-line therapy for metastatic uterine leiomyosarcoma: A Gynecologic terestingly, in addition to the deep deletion of BRCA2,her Oncology Group phase II trial. Gynecol Oncol 2008 Jun;109(3):329-34. DOI: https://doi. tumor harbors deep deletion of 2 other important tumor org/10.1016/j.ygyno.2008.03.010, PMID:18534250 suppressor genes: TP53 and PTEN. ere is in vitro ev- 2. Tap WD. GeDDiS: Insight into frontline therapy in soft tissue sarcoma. Lancet Oncol 2017 fi Oct;18(10):1297-9. DOI: https://doi.org/10.1016/S1470-2045(17)30672-1,PMID:28882535 idence that TP53 as well as PTEN de ciency increases the 3.DemetriGD,vonMehrenM,JonesRL,etal.Efficacy and safety of trabectedin or sensitivity of cancer cell lines to PARP inhibition.11,12 For dacarbazine for metastatic liposarcoma or leiomyosarcoma after failure of fi conventional chemotherapy: Results of a phase III randomized multicenter clinical example, mantle cell lymphoma cells that are de cient in trial. J Clin Oncol 2016 Mar;34(8):786-93. DOI: https://doi.org/10.1200/JCO.2015. both ATM and p53 are more sensitive to olaparib than 62.4734, PMID:26371143 fi 12 4. Farmer H, McCabe N, Lord CJ, et al. Targeting the DNA repair defect in BRCA mutant cells that are de cient in ATM alone, and endometrioid cells as a therapeutic strategy. Nature 2005 Apr;434(7035):917-21. DOI: https://doi.org/ endometrial cancer cells that are deficient of PTEN are 10.1038/nature03445, PMID:15829967 more sensitive to PARP inhibition due to the inability of 5. Lord CJ, Ashworth A. The DNA damage response and cancer therapy. Nature 2012 Jan; 481(7381):287-94. DOI: https://doi.org/10.1038/nature10760, PMID:22258607 the cancer cells to mount a homologous recombination 6. Dedes KJ, Wilkerson PM, Wetterskog D, Weigelt B, Ashworth A, Reis-Filho JS. Synthetic repair response.11,13 Whether or not the deep deletion of lethality of PARP inhibition in cancers lacking BRCA1 and BRCA2 mutations. Cell ’ Cycle 2011 Apr;10(8):1192-9. DOI: https://doi.org/10.4161/cc.10.8.15273,PMID: TP53 and/or PTEN sensitized this patient s metastatic 21487248 uLMS to olaparib is not certain. Further studies in this 7. de Bono J, Mateo J, Fizazi K, et al. Olaparib for metastatic castration-resistant prostate area shall result in additional insight. Our case report adds cancer. N Engl J Med 2020 May;382(22):2091-102. DOI: https://doi.org/10.1056/ fi NEJMoa1911440, PMID:32343890 additional evidence that PARP inhibition may be of bene t 8. Alsop K, Fereday S, Meldrum C, et al. BRCA mutation frequency and patterns of for patients with metastatic uLMS with a somatic BRCA2 treatment response in BRCA mutation-positive women with ovarian cancer: A report from the Australian Ovarian Cancer Study Group. J Clin Oncol 2012 Jul;30(21):2654-63. DOI: deletion and highlights the importance of obtaining NGS in https://doi.org/10.1200/JCO.2011.39.8545, PMID:22711857 the treatment of patients with metastatic cancer. v 9. Moore K, Colombo N, Scambia G, et al. Maintenance olaparib in patients with newly diagnosed advanced ovarian cancer. N Engl J Med 2018 Dec;379(26):2495-505. DOI: https://doi.org/10.1056/NEJMoa1810858, PMID:30345884 Disclosure Statement 10. Seligson ND, Kautto EA, Passen EN, et al. BRCA1/2 functional loss defines a targetable The author(s) have no conflicts of interest to disclose. subset in leiomyosarcoma. Oncologist 2019 Jul;24(7):973-9. DOI: https://doi.org/10.1634/ theoncologist.2018-0448, PMID:30541756 11. Dedes KJ, Wetterskog D, Mendes-Pereira AM, et al. PTEN deficiency in endometrioid Authors’ Contributions endometrial adenocarcinomas predicts sensitivity to PARP inhibitors. Sci Transl Med 2010 Minggui Pan, Kristen Ganjoo, and Amer Karam participated in conceiving the Oct;2(53):53ra75. DOI: https://doi.org/10.1126/scitranslmed.3001538, PMID:20944090 study and writing the manuscript. 12. Williamson CT, Kubota E, Hamill JD, et al. Enhanced cytotoxicity of PARP inhibition in mantle cell lymphoma harbouring mutations in both ATM and p53. EMBO Mol Med 2012 Jun;4(6): 515-27. DOI: https://doi.org/10.1002/emmm.201200229, PMID:22416035 Abbreviations 13. Forster MD, Dedes KJ, Sandhu S, et al. Treatment with olaparib in a patient with PTEN- NGS = next-generation sequencing; PARP = poly (ADP-ribose) polymerase; deficient endometrioid endometrial cancer. Nat Rev Clin Oncol 2011 May;8(5):302-6. DOI: uLMS = uterine leiomyosarcoma https://doi.org/10.1038/nrclinonc.2011.42, PMID:21468130

The Permanente Journal·https://doi.org/10.7812/TPP/20.251 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 169 n CASE REPORT A Case Report of Leptomeningeal Carcinomatosis Secondary to Recurrent Merkel Cell Carcinoma after Avelumab

Pedro Mendoza1; Kathy Lin Chuang, MD2 Perm J 2021;25:20.235 E-pub: 03/25/2021 https://doi.org/10.7812/TPP/20.235

ABSTRACT increasing in occurrence, has a high mortality rate, and lacks Introduction: Merkel cell carcinoma (MCC) is a rare and ag- a response to traditional chemotherapy.8,9 Fortunately, gressive neuroendocrine cancer with a high mortality rate of 33% avelumab—a programmed cell death protein 1/programmed to 46%. Merkel cell is a type of epidermis cell receptor responsible cell death–ligand 1 blocking antibody—obtained accelerated for contact sensitivity and is known to have neuroendocrine approval by the US Food and Drug Administration in 2017 properties. Treatment of Merkel cell carcinoma with avelumab has for treatment of MCC as a result of its success in study been prominsing, but its rarity and poor prognosis necessitates trials. Unlike traditional chemotherapy, however, there are close follow up. Case Presentation: A 71-year-old woman presented with a left currently no known studies on avelumab and the prevention 9 forearm mass that was initially suspected to be a sebaceous cyst. or treatment of metastatic MCC to the leptomeninges. As a After surgical excision and biopsy, she was diagnosed with Merkel novel monotherapy treatment agent for MCC, currently the cell carcinoma. The patient underwent avelumab treatment for standard of care, there is limited knowledge of this med- 2 years, with remission of cancer for 24 months. A positron ication. us, additional studies are needed, especially in emission tomographic scan at 24 months of treatment noted evaluating the rare occurrence of metastatic leptomeningeal uptake in the left axilla and portocaval regions. Despite receiving carcinomatosis secondary to MCC after avelumab treat- different combinations of immunotherapy, chemotherapy, and ment. For Case Presentation outline, please refer to Table 1. radiation, the patient’s cancer metastasized to the leptomeninges. She was transitioned to hospice and passed away 3 months after CASE PRESENTATION diagnosis of leptomeningeal carcinoma. A 71-year-old Hispanic woman with a history of di- Conclusion: This case highlights the efficacy of avelumab in keeping patients in remission, which can offer increased quality of abetes and high blood pressure presented to her primary life. However, it also highlights the aggressive nature of Merkel physician with a painless left proximal forearm nodule cell carcinoma and the importance of surveillance for early de- that had increased slowly in size for the past 6 months. It tection of recurrence. was diagnosed initially as a sebaceous cyst. epatient opted for localized excision of the presumed sebaceous cyst. epathologyresultswerepositiveforMCC.e INTRODUCTION patient then underwent a wide excision of the left proximal Merkel cell is a type of epidermis cell receptor responsible forearm, with pathology showing clear margins (T1N0M0). for contact sensitivity and is known to have neuroendocrine Afterward, the patient completed localized radiation in the properties. MCC is a rare and aggressive neuroendocrine left proximal forearm as recommended. Six months after cancer that, if untreated, can be fatal.2 Although extremely  localized radiation, the patient noted a left distal humerus rare, it can metastasize to the leptomeninges. is metas- mass (Figure 1). A computed tomographic scan of left distal tasis occurs by seeding of tumor cells into the cerebrospinal humerus without contrast showed a 2.6 × 2.5 × 1.9-cm fluid (CSF) and the leptomeninges as a terminal, late-stage 3 subcutaneous soft tissue mass compatible with metastatic complication of various solid tumors. MCC usually disease. A positron emission tomographic (PET) scan presents as a painless lump, which can delay early detection showed left distal humerus lymph node uptake, and left and treatment. Risk factors for MCC include the following: axillary and left subpectoral lymph node uptake. The pa- age (> 50 years old), excessive ultraviolet light exposure, tient underwent left distal humerus mass excision and left smoking, immunocompromised status, and history of in- fi 4-7 axillary lymph node dissection. Pathology con rmed 7 of fection with Merkel cell polyomavirus. MCC has been the 14 left axillary lymph nodes were consistent with metastatic MCC. A follow-up computed tomographic scan was taken of the chest after the surgical resection of the left Author Affiliations subpectoral and left axillary lymph nodes, and it showed a 1Department of Behavioral Neuroscience, University of San Diego, San Diego, CA retained 1.2 × 1-cm left subpectoral cancerous lymph node. 2Kaiser Permanente Riverside Medical Center, Riverside, CA e patient then completed avelumab treatment biweekly Corresponding Author for 24 months (with diphenhydramine and acetamino- Pedro Mendoza ([email protected]) phen prior to each treatment) for T4N1M1 disease.

Keywords: cancer, case report, dermatology, immunology, Merkel cell, oncology 170 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.235 aeRpr fLpoeiga acnmtssScnayt eurn eklCl acnm fe Avelumab after Carcinoma Cell Merkel Recurrent to Secondary Carcinomatosis Leptomeningeal of Report Case A h emnneJournal Permanente The

Table 1. Timeline table; relevant past medical history and interventions include diabetes mellitus type 2 (metformin), hypertension (amlodipine), and overweight (daily water exercise and walking) · tp:/o.r/071/P/025TePraet Journal Permanente The https://doi.org/10.7812/TPP/20.235 Summaries from initial and follow-up visits (patient’s Interventions (pharmaceuticals and dietary primary concerns as well as clinical-initiated diagnostic Diagnostic testing, including dates of relevant testing, supplements, dietary and lifestyle recommendations, Date assessments and diagnoses) such as laboratory, imaging, or surveys procedures) 10/27/2016 Patient had left forearm nodule discomfort and was referred Pathology showed neuroendocrine tumor consistent with to general surgery by the primary doctor for excision. Merkel cell cancer. 11/10/2016 Patient was referred to imaging for cancer evaluation. CT scan of abdomen and pelvis showed no masses. 11/14/2016 Patient was referred to imaging for cancer evaluation. CT scan of the chest showed no masses. 12/1/2016 General surgery completed a wide-excision biopsy of the left forearm with antecubital lymph node biopsy. 12/2016–01/2017 Received localized radiation to the left forearm, weekly, for 10 Physical exam was conducted and was within normal limits wk. during treatment. 9/1/2017 At radiation follow-up visit, the patient presented with 6 wk of Imaging was ordered for left elbow mass and metastatic a left nontender elbow mass that was increasing in size. cancer evaluation. 9/8/2017 CT scan of left elbow mass identified on 9/1/2017 radiation A follow-up PET scan was ordered for metastatic cancer follow-up visit showed mass and enlarged axillary lymph node. evaluation. 9/19/2017 PET scan showed a positive uptake at the medial left elbow, left axilla, and left subpectoral lymph nodes. · o esnlueol.N te sswtotpriso.Cprgt©22 h emnnePes l ihsreserved. rights All Press. Permanente The 2021 © Copyright permission. without uses other No only. use personal For 9/27/2017 Patient underwent general surgery for biopsies of recurrent Biopsies were performed on the left elbow and axillary. mass. Pathology confirmed the presence of Merkel cell carcinoma. 10/2017 Patient began radiation to the left elbow area concurrently Avelumab was administered for 24 mo without any side with avelumab. effects and with negative PET scans. 10/2019 The PET scan obtained at the 24-mo avelumab treatment Patient was placed on ipilimumab and nivolumab, and began Patient was started on proton pump inhibitors, liquid was positive for left subclavian and porta hepatis adenopathy. localized radiotherapy to the left upper chest and abdomen antacids, and antiemetics as needed as a result of gastritis, area. which was most likely induced by the radiation. 1/28/2020–2/3/2020 Patient was admitted for bilateral leg and hand weakness. Patient was diagnosed with GBS and was treated with Patient’s strength recovered. intravenous immunoglobulin. Ipilimumab and nivolumab were discontinued at this point The lumbar puncture for GBS syndrome incidentally was because of a concern of spread of cancer into meninges. positive for Merkel cells. At this point, the patient was pending a new treatment plan. MRI of lumbar spine showed subtle enhancement of anterior thoracolumbar spinal cord T12–L1. Results were unable to confirm GBS or leptomeningeal disease. 2/17/2020–2/27/2020 Patient was admitted for headache, nausea, vomiting, and While the patient was being evaluated for a headache, it was Hypertension was treated with losartan, amlodipine, and back pain. noted the patient was hypertensive and had a cranial nerve lisinopril during hospitalization, and the patient was later 12 palsy. These findings were concerning for leptomeningeal discharged with these medications. Leptomeningeal disease. involvement was treated with brain radiation, carboplatin, While the patient was being evaluated for nausea and and 1 dose of etoposide, followed by cisplatin after vomiting, an endoscopy/sigmoidoscopy was performed and discharge. showed gastric ulcers and immune-mediated colitis During hospitalization, the patient was started on While the patient was being evaluated for back pain, MRI of omeprazole and sucralfate (as needed) for gastritis, the lumbar spine showed leptomeningeal involvement. Senokot and Miralax (as needed) for constipation, and a

Incidentally, during hospitalization, the patient was found to bland diet. The patient was discharged with these REPORT CASE have urinary retention that was attributed to leptomeningeal recommendations. involvement. Patient received hydrocodone/acetaminophen (as needed (continued on following page) 171 CASE REPORT A Case Report of Leptomeningeal Carcinomatosis Secondary to Recurrent Merkel Cell Carcinoma after Avelumab

Follow-up PET scans during avelumab treatment were negative for uptake, including at the retained lymph node. A PET scan at 24 months of treatment with avelumab showed a recurrence of MCC to a left subpectoral lymph node (Figure 2) and the left portacaval region (Figure 3). e patient completed localized radiation to the left subpectoral and portocaval regions, and was started on nivolumab and ipilimumab. A follow-up PET scan after completion of radiation and while taking nivolumab and procedures) ipilimumab showed a positive treatment response resulting from decreased size and intensity of fluorodeoxyglucose

), and overweight (daily water exercise uptake in both locations. e patient was hospitalized for ascending bilateral leg Interventions (pharmaceuticals and dietary and arm weakness, and diagnosed with Guillain Barré supplements, dietary and lifestyle recommendations, for back pain) andreceived naloxone thoracic (as and needed). lumbar Patient spinestarted also radiation. physical Patient therapy then rehabilitation. Patient was referred to urology. Patient received radiation therapy to cervical spine. syndrome 27 months after diagnosis of MCC. is was most likely secondary to nivolumab and ipilimumab, and 6 – the patient’s symptoms improved with intravenous immu- noglobin, hydrocortisone treatment, and discontinuation of nivolumab and ipilimumab. CSF cytology was positive for MCC, and magnetic resonance imaging (MRI) of the lumbar spine revealed leptomeningeal carcinomatosis. e patient then developed hypertension, headache, cranial nerve 12 palsy, and low-back and right-leg pain 14 days after Guillain Barré syndrome hospitalization. e patient’s hypertension, headache, cranial nerve 12 palsy, and back and leg pain were attributed to her leptomeningeal c signal intensities of cervical vertebrae 5 fi disease. e patient’s nausea, vomiting, and abdominal pain were secondary to gastric ulcer seen via endoscopy (most such as laboratory, imaging, or surveys likely radiation induced), immune checkpoint inhibitor- related colitis (based on sigmoidoscopy biopsy), and uri- Diagnostic testing, including dates of relevant testing, MRI of the cervicalnoted spine nonspeci was limited by motion artifact but and thoracic vertebrae 4. Patient was transitioned to hospice care nary retention secondary to neurogenic bladder. e patient received only 1 treatment of carboplatin and etoposide s ’ during hospitalization, and completed whole-brain and lower thoracic–lumbar radiation for leptomeningeal disease. e patient then improved and was discharged with a medication regimen for blood pressure control and symp- tomatic management. Shortly after hospital discharge, the patient developed severe right-arm incoordination secondary to leptomeningeal disease and underwent radiation to the cervical spine, after which symptoms improved. e patient then began weekly cisplatin chemotherapy, but she continued to decline

assessments and diagnoses) neurologically. e patient again developed bilateral lower leg weakness with inability to bear weight, became bed bound, had transient episodes of diplopia, developed left Summaries from initial and follow-up visits (patient and weakness, with uncontrollableMRI right-arm of movements. the cervical spine was ordered for further evaluation. primary concerns as well as clinical-initiated diagnostic and decreased food intake. cranial nerve 6 palsy, and had 1 episode of seizure-like activity, and eventually transitioned to hospice.

DISCUSSION is appears to be the first published case of leptomeningeal carcinomatosis secondary to recurrent MCC treated with avelumab. Extensive PubMed research did not yield any 3/17/2020 Patient had developed bilateral arm pain, hand numbness Table 1.. Timeline table; relevant past medical history and interventions include diabetes mellitus type 2 (metformin), hypertension (amlodipine Date CT = computed tomography; GBS = Guillain-Barré syndrome; MRI = magnetic resonance imaging; PET = positron emission tomography. and walking) (continued) 4/6/20205/2/2020 Patient had seizure-like episode at home, deconditioning, Patient passed away additional case reports of a similar occurrence. is patient’s

172 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.235 CASE REPORT A Case Report of Leptomeningeal Carcinomatosis Secondary to Recurrent Merkel Cell Carcinoma after Avelumab

Figure 2. Positron emission tomographic scan shows positive fluorodeoxyglucose uptake in the left subpectoral region.

cauda equina syndrome.3 e diagnosis of leptomeningeal Figure 1. Left distal humerus mass (outlined in black). carcinomatosis is made by CSF analysis and MRI.3 Ma- lignant cells, pleocytosis, high protein levels, or low glucose risk factors for MCC include age and ultraviolet exposure. are often seen in the CSF. MRI with gadolinium should Initial surgery and radiation for stage 1 MCC include wide include the brain and spine because the disease can affect the excision surgery and/or localized radiation.10 However, entire neuraxis, versus computed tomography, which is less 8 months later, despite surgery and radiation, the patient had sensitive in detecting leptomeningeal cancer.3 e intro- recurrence of MCC on left elbow, with distal lymph node duction of avelumab was seen as a promising agent to fight involvement. Treatment for recurrence of MCC includes MCC (local or distant) and has increased the survival rate as immunotherapy.10 e patient completed biweekly avelumab well as the quality of life when compared to chemotherapy.13 treatments for 24 months with no adverse events; however, a Our patient achieved both quality and quantity of life with follow-up PET scan at 24 months showed metastatic MCC avelumab. However, despite taking avelumab, she devel- to the left chest and portocaval regions. e patient completed oped metastatic leptomeningeal cancer. Treatment for radiation to the chest and portocaval regions, and started nivolumab and ipilimumab treatment, with a positive re- sponse seen on follow-up imaging. e patient subsequently developed GBS, leading to an incidental finding of malignant cells in the CSF, with MRI confirmation of leptomeningeal carcinomatosis. Ipilimumab and nivolumab were dis- continued, and the patient was then switched to cisplatin. Despite eventually completing whole-brain, cervical, thoracic, and lumbar spine radiation for her leptomeningeal disease symptoms, the patient continued to decline neurologically. e last known reported case of leptomeningeal MCC was documented in 2011.11 It is a rare and aggressive form of cancer, with an estimated incidence of about 800 cases per year in the US, and 2- and 5-year overall survival rates of 53.9% and 32.8%, respectively.12 e incidence of lep- tomeningeal carcinomatosis secondary to MCC is extremely rare, with a short life expectancy of weeks to months. Leptomeningeal disease can present as cranial nerve deficits, radicular pain, headache, back pain, visual disturbance, Figure 3. Positron emission tomographic scan showing positive diplopia, hearing loss, psychiatric disorders, seizures, or fluorodeoxyglucose uptake in the portacaval region.

The Permanente Journal·https://doi.org/10.7812/TPP/20.235 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 173 CASE REPORT A Case Report of Leptomeningeal Carcinomatosis Secondary to Recurrent Merkel Cell Carcinoma after Avelumab

leptomeningeal cancer may include radiation, intrathecal Authors’ Contributions 3 chemotherapy, systemic chemotherapy, or surgery. e Pedro Mendoza collected the information and wrote the manuscript with the patient received whole-brain radiotherapy, focal radio- assistance of Kathy Chuang, MD. Both researched, analyzed, and interpreted the therapy in different areas of the spine for palliative relief, information for publication. and systemic chemotherapy because she experienced Funding various symptoms associated with leptomeningeal cancer. No funding was received for this work. She survived her leptomeningeal disease for 3 months despite these treatment efforts. is case highlights the References importance of surveillance while taking avelumab and its 1. Schadendorf D, Lebbé C, Hausen AZ, et al. Merkel cell carcinoma: Epidemiology, prognosis, therapy and unmet medical needs. Eur J Cancer 2017 Jan;71:53–69, DOI: limitations with regard to blood–brain barrier perme- https://doi.org/10.1016/j.ejca.2016.10.022 ability because, although rare, MCC can disseminate into 2. Merkel cell carcinoma treatment (PDQ®): Patient version; n.d. https://www.cancer.gov/ types/skin/patient/merkel-cell-treatment-pdq#_1. Accessed January 9, 2020 the leptomeninges. 3. Nayar G, EjikemeT, ChongsathidkietP, et al. Leptomeningeal disease: Current diagnostic is case also highlights the efficacy of avelumab in keeping and therapeutic strategies. Oncotarget 2017 Aug;8:73312–28, DOI: https://doi.org/10. ff 18632/oncotarget.20272 patients in remission, which can o er increased quality of life. 4. Merkel cell carcinoma risk factors; n.d.. https://www.cancer.org/cancer/merkel-cell-skin- ecasealsounderscorestheimportanceofsurveillancefor cancer/causes-risks-prevention/risk-factrs.html. Accessed January 9, 2020 relapse and early detection of recurrent MCC. With the 5. Merkel cell carcinoma; 2019. https://www.mayoclinic.org/diseases-conditions/merkel-cell- carcinoma/symptoms-causes/syc-20351030. Accessed January 9, 2020 constant advancement of research, this case may highlight 6. Pietropaolo V, Prezioso C, Moens U, et al. Merkel cell polyomavirus and Merkel cell ways that avelumab can be used and patients can be monitored carcinoma. Cancers (Basel) 2020 Jul;12(7):1774. DOI: https://doi.org/10.3390/ cancers12071774 to maintain remission status and increase life expectancy. 7. Spurgeon ME, Lambert PF. Merkel cell polyomavirus: A newly discovered human virus with oncogenic potential. Virology 2013 Jan;435(1):118–30. DOI: https://doi.org/10.1016/j. CONCLUSION virol.2012.09.029 8. Villani A, Fabbrocini G, Costa C, Annunziata MC, Scalvenzi M. Merkel cell carcinoma: is case highlights 3 key points: 1) the importance of Therapeutic update and emerging therapies. Dermatol Ther (Heidelb) 2019 Jun;9: close follow-up visits, especially with localized cancer, for 209–22. DOI: https://doi.org/10.1007/s13555-019-0288-z 9. D’Angelo SP, RussellJ, LebbéC, et al. Efficacy and safety of first-line avelumab treatment the prevention and early detection of metastatic disease; 2) a in patients with stage IV metastatic Merkel cell carcinoma: A preplanned interim analysis rare side effect of immune checkpoint inhibitor causing of a clinical trial. JAMA Oncol 2018 Sep;4(9):e180077. DOI: https://doi.org/10.1001/ jamaoncol.2018.0077 GBS; and 3) despite the use of avelumab (recently approved 10. National Comprehensive Cancer Network; n.d. https://www.nccn.org/professionals/ by US Food and Drug Administration) for MCC, lep- physician_gls/default.aspx. Accessed January 9, 2020 11. Abul-Kasim K, Soderstr¨ om¨ K, Hallsten L. Extensive central nervous system involvement tomeningeal involvement, although rare, can occur and in Merkel cell carcinoma: A case report and review of the literature. J Med Case Rep 2011 should be considered in those patients who present with Jan;5:35. DOI: https://doi.org/10.1186/1752-1947-5-35 12. Liang E, Brower JV, Rice SR, Buehler DG, Saha S, Kimple RJ. Merkel cell carcinoma symptoms of meningeal involvement such as headache, analysis of outcomes: A 30-year experience. PLoS One 2015 Jun;10(6):e0129476. DOI: back pain, or neurologic symptoms. v https://doi.org/10.1371/journal.pone.0129476 13. Kaufman HL, RussellJS, HamidO, et al. Updated efficacy of avelumab in patients with previously treated metastatic Merkel cell carcinoma after ≥1 year of follow-up: JAVELIN Disclosure Statement Merkel 200, a phase 2 clinical trial. J Immunother Cancer 2018 Jan;6(1):7. DOI: https:// The authors have no conflicts of interest to disclose. doi.org/10.1186/s40425-017-0310-x

174 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.235 n COMMENTARY Unconscious Bias Is a Human Condition

Mihal Emberton, MD, MPH, MS1 Perm J 2021;25:20.199 E-pub: 3/17/2021 https://doi.org/10.7812/TPP/20.199

ABSTRACT patterns to figure out how to successfully navigate the The unconscious bias that plagues our police departments is social hierarchy. the same unconscious bias that plagues business in the form of Frank Abagnale, a consultant for the Federal Bureau employee oppression and burnout, that plagues academia in the of Investigation, a faculty member at e Department of fi form of social promotion and xed mindset, that plagues law in Justice’s National Advocacy Center, and an expert on fraud the form of poetic injustice and that plagues politics in the form of prevention,3 recognized at an early age that people defer to disenfranchisement and voter suppression. Each industry has also patterns of authority. Mr Abagnale discovered that when attempted to correct the negative effects of unconscious bias with leadership development in business, Montessori method he presented himself as a professional with a certain level and active learning in academia, restorative justice in law, and of social status and authority, such as an airline pilot or voter participation and town halls in politics. The reason that each physician, the people around him would defer to that position of these efforts is still in its infancy is that our understanding of the of power. I recently asked him if he ever found that despite rules of unconscious bias and the behaviors that stem from that wearing a uniform or nice suit, there were some people who unconscious bias have been incompletely understood until now. were not inclined to assume that he was a professional. He Good people like yourself and like me have unconscious bias; responded, “No, absolutely not… When I had the uniform on, having unconscious bias is a human condition. But those of us no one questioned me, not even other airline personnel” (FW who learn to recognize and overcome our unconscious bias Abagnale, personal communication, 2020 June). become more impactful and powerful stewards of society. Recognizing the patterns of social status and authority help us determine how to successfully relate to other people INTRODUCTION in our social hierarchy. In addition to generally deferring to is manuscript is the second in a 3-part series that those whom we interpret as having authority and/or social begins with “Learning in Humans versus Hierarchies” and status, we may also recognize patterns that we interpret will conclude with “Etiology of Belief-Behavior Systems as placing us above another person in the social hierarchy. and Hierarchies.” Encountering someone appearing homeless or encountering someone younger than ourselves may trigger this unconscious PATTERN RECOGNITION AND THE SOCIAL HIERARCHY social calculation of presumed authority over others. Human beings love to find patterns and identify the rules When we derive the correct rule for an observed pattern, 1,2 for those patterns to solve problems and puzzles.1,2 You can it not only brings joy but also allows us to successfully easily test this theory by observing the people around you. relate to other people in our social hierarchy. However, My child, for example, lights up when she recognizes the sometimes we get the rule for the pattern wrong. A clear patterns and rules for defeating an opponent in games of example of this occurred in June 2020 when a San Francisco strategy, and many physicians, for example, chose to study white woman approached a person of color in her neigh- “ ” medicine because discovering the patterns and rules related borhood who was writing Black Lives Matter on a 4  to health and disease elicits excitement and meaning. retaining wall. is woman likely recognized a pattern that We use our ability to recognize patterns and to find rules generally the people who live in her neighborhood are not for those patterns to also understand our social hierarchies people of color. However, she got the rule for the pattern to successfully interact with other people. For example, we wrong and mistakenly assumed that people of color never often view a person wearing a metal band on their left ring live in her neighborhood. Based on this miscalculated — finger as someone who is married because, based on our rule an assumption that this person of color must be an — experience of this pattern, a ring being worn on this finger outsider , she accused him of defacing private property 4 signifies marriage. In addition to our own experiences, we and called the police to report him as a criminal. It was her also learn about patterns from others. Many families, for miscalculated rule, her unconscious bias, that caused her to example, teach their children to respect and defer to the insights and experiences of elders because, based on that Author Affiliation family’s experience, people who are older generally have 1Permanente Medical Group, Adult and Family Medicine, San Francisco, CA important knowledge, insights, and experience to impart. erefore, we use our experiences of social patterns in Corresponding Author Mihal Emberton, MD, MPH, MS, ([email protected]) combination with education from others about social Keywords: burnout, engagement, organizational culture change, overcoming unconscious bias, social justice The Permanente Journal·https://doi.org/10.7812/TPP/20.199 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 175 COMMENTARY Unconscious Bias Is a Human Condition

mistakenly assume a position of authority over her neighbor, Einstein also recognized this pattern of uncon- which resulted in a deleterious social interaction. scious bias, of supervisors unconsciously disregarding is type of unconscious bias is a human condition, the knowledge, insights, and experiences of subordi- not bounded by culture or place or time. Frank Abagnale nates, when he advocated, “It is important for the highlights the universality of our unconscious bias when he common good to foster individuality [subordinate states that, “roughout my career, I’ve found that the knowledge]: for only the individual can produce the new power of image, whether in personal appearance, marketing ideas which the community [hierarchy] needs for its materials, company correspondence, etc. has a tremendous continuous improvement and requirements – indeed, to influence and bias in the way people treat you, not only in this avoid sterility and petrification.” 5 country but all over the world” (FW Abagnale, personal Because autocratic belief-behavior systems are a human communication, 2020 June). e truth is, unfortunately, that condition, this pattern of unconscious bias can encumber we all carry the burden of possibly miscalculating the rules for any structured hierarchy.1 Carol Dweck, PhD, Stanford the patterns we observe, the burden of unconscious bias. psychology professor, describes this pattern of unconscious bias in education in her description of fixed mindsets.2 GAPS IN KNOWLEDGE AND EXPERIENCE LEAD TO BIAS Douglas McGregor, PhD, Massachusetts Institute of Tech- While we seek out patterns to help identify rules1,2 that nology management professor, described this pattern of un- help us successfully interact within our social hierarchy, we conscious bias in business in his description of eory-X.6,7 all too often end up misidentifying a rule because we do not Albert Eglash, psychologist, described this pattern of un- recognize that we have gaps in our knowledge about others. conscious bias in law and outlined how to overcome this Other people are generally the experts of their own expe- unconscious bias in law in his description of restorative rience, and they will generally have more insight about their justice.8,9 is pattern of unconscious bias within our po- experiences than we will ever have about them. e San litical system, which has been present since the inception of Francisco woman made a conclusion about the “Black Lives our American Democracy in the 1700s, is often described as Matter” neighbor without recognizing that she did not disenfranchisement and voter suppression. 10 know for a fact his neighborhood status. In Mr Abagnale’s e reason that our attempts to overcome this uncon- experience, people always made a conclusion about his status scious bias are still in their infancy is that we are only now and authority without recognizing that they did not really understanding the rules for the patterns that cause and know for a fact if he was a pilot or physician (FW Abagnale, derive from our unconscious bias. e rules are that 1) we all personal communication, 2020 June). can make conclusions despite having knowledge gaps, 2) we Making a decision or conclusion when a knowledge gap generally do not recognize when we have such knowledge is present is the definition of bias, which I also call an gaps, and 3) we then often blame others for the discomfort autocratic belief-behavior system. Generally, when we are of having such a knowledge gap. Generally, structured in our autocratic mindset, we are not aware that we are hierarchies allow autocratic supervisors to maintain au- making decisions or conclusions despite having gaps in tocratic belief-behavior systems because it can be nearly knowledge or experience; generally, by definition, our bias is impossible for subordinates to teach autocratic supervi- unconscious.1 sors that they are demonstrating unconscious bias. Ad- ditionally, subordinates are often punished for making STRUCTURED HIERARCHIES INCUBATE UNCONSCIOUS BIAS autocratic supervisors feel uncomfortable. In order to Unlike making a social calculation about a stranger and overcomeunconsciousbias,supervisorsthemselvesmust how that stranger relates to us in the social hierarchy, discover their knowledge gaps by genuinely valuing and structured hierarchies such as academic institutions, busi- seeking out the knowledge, insights, and experiences of nesses, and political systems, unfortunately incubate our others, especially from those who are lower than them in the unconscious bias. is occurs because the power differential social or organizational hierarchy.1 in our structured hierarchies insulates and protects the An example of how unconscious bias can be protected in a unconscious bias of supervisors such that when a supervisor hierarchy occurs very visibly in our political system. It is is in their autocratic mindset, they often misinterpret generally understood that our legislators spend 4-6 hours of the clues and evidence of such as insurgency on the part their 9-10-hour day fundraising, leaving them very little of the subordinate. Autocratic supervisors will then tend to time to seek out the knowledge, insights, and experiences of punish the subordinate for making the autocratic supervisor their constituents to better understand the social problems uncomfortable rather than recognize that it was the uncon- that the disenfranchised are experiencing.11 Ezra Klein, scious bias of the autocratic supervisor, driven by a knowledge journalist, political commentator, and co-founder and gap, that created the discomfort in the first place.1 editor-at-large for Vox, emphasizes that “congress could, if

176 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.199 COMMENTARY Unconscious Bias Is a Human Condition

it wanted, move to a system of real public financing for their journey, I was able to overcome my bias by reaching elections [to give legislators more time to collaborate with out to my colleague to fill in my knowledge gaps about constituents/disenfranchised]. ey don’t, and the reason is their experiences and hopefully to help fill in their knowledge simple: e money chase makes life miserable for incum- gaps about racism. bents, but it also makes it likelier that they remain in- Since structured hierarchies tend to incubate our un- cumbents. Even as it makes you worse at your job, it makes conscious bias, it is even harder to seek out our knowledge you better at keeping it.”11 In essence, our legislators have and experience gaps when we become supervisors. Many the power to change the system to allow them more time to organizations, for example, attempt 360 evaluations, where seek out and fill in their knowledge gaps in order to create subordinates may “anonymously” provide feedback to their more just social policy, but instead legislators wield their supervisor as a mechanism to identify the supervisor’sknowl- power to maintain the status quo, which ends up insulating edge and performance gaps. How that supervisor uses and their knowledge gaps, perpetuating imperfect social poli- interprets that feedback, however, determines if they are cies, and keeping the disenfranchised oppressed. willing to find and learn from their knowledge and per- formance gaps or not. If the supervisor is in their democratic OVERCOMING UNCONSCIOUS BIAS REQUIRES TENACITY mindset, they will seek out and embrace their knowledge and While it is generally easier to spot the unconscious bias in performance gaps in order to rectify them, but if the su- others, it is often very difficult to recognize and spot it pervisor is in their autocratic mindset, they will instead look within ourselves. For example, a colleague recently shared for evidence that they are already doing an adequate job, their knowledge and experience that “systemic racism does subsequently writing off minority “constructive” feedback as not exist,” and, while I could easily recognize that this outliers. It is not the tool itself but rather how a supervisor colleague had a gap in knowledge and experience, dem- uses that tool that determines if the supervisor is running a onstrating unconscious bias, it took me a longer amount of conformity process or a learning process.1 time with much more effort to recognize the unconscious For example, an autocratic supervisor might ask that all bias that I displayed toward this colleague in response. subordinates complete a 360 evaluation of the supervisor in When I first encountered my colleague’s words, I initially order to avoid getting only “super positive” and “super felt vulnerable and threatened, thinking “this colleague has negative” reviews. Such an autocratic supervisor thinks that no idea what they are talking about and they are definitely they are valuing all their subordinates’ knowledge and ex- not worth my time.” It took me a while to recognize that my perience by asking for all their opinions. However, such an initial feelings and thoughts were clues that I was in my autocratic supervisor is really seeking a majority consensus autocratic mindset, demonstrating unconscious bias toward my that there are no gaps in knowledge or performance, a majority colleague.1 consensus that the status quo is acceptable. In essence, it is In my effort to unravel my unconscious bias, I had to more comfortable to believe that we are functioning ideally, recognize 1) the pattern that I was interpreting as well as without knowledge gaps or performance gaps. 2) the knowledge gap that was causing me to come up with ere are a few key insights about how hierarchies work the wrong rule for the pattern. e pattern that I was that an autocratic supervisor often overlooks. First of all, just recognizing was that I had more knowledge and experience because there are only 1 or 2 people who take the time to than my colleague about racism. However, I was mis- comment on a knowledge gap or performance gap does not interpreting my colleague’s proud display of their knowl- mean that the knowledge gap or performance gap does not edge gap about racism, as evidence that my colleague did not exist or is not valid. Additionally, subordinates will often offer me any other knowledge, insights, or experiences to defer pointing out knowledge gaps or performance gaps in learn from. is is, however, where I had a gap in knowledge; their supervisor because 1) subordinates may feel that I assumed that my colleague did not offer any intellectual speaking out will not change the current situation and/or currency without really knowing for a fact that my colleague 2) it is so much easier and safer for subordinates to just let a had nothing to teach me. e moment a person believes that supervisor have a knowledge gap or performance gap.1 It they have nothing to learn from another is the moment they is for exactly these reasons that finding those knowledge have closed themselves off to discovery.1 and performance gaps can be so difficult and elusive for In order to truly collaborate with my colleague, I had to supervisors. embrace my knowledge gap, remember that this colleague Like many supervisors, I also receive annual 360 feedback has something valuable to teach me, and use this insight to evaluations from my subordinates, and, in order to overcome genuinely and respectfully engage with my colleague. Once my unconscious bias, I intentionally embrace and apply all I recognized that I had unconsciously devalued my colleague the “super negative” insights to my practice improvement. and unconsciously shut myself off to learning more about More specifically, I do not view the suggestions for growth

The Permanente Journal·https://doi.org/10.7812/TPP/20.199 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 177 COMMENTARY Unconscious Bias Is a Human Condition

“ ” as super negative but rather view the suggestions for Abbreviations growth as the very necessary keys to finding and addressing FBI, Federal Bureau of Investigation; NAC, National Advocacy Center. my knowledge and performance gaps. Additionally, I share my “super negative” feedback with my subordinates and References 1. Emberton M. Learning in humans versus hierarchies. Perm J 2020;20:19.083. DOI: then collaborate with my subordinates on how I can suc- https://doi.org/10.7812/TPP/19.083 cessfully address my knowledge and performance gaps. 2. Brown PC, Roediger HL 3rd, McDaniel MA. Make it stick: The science of successful fi learning. Cambridge, MA: The Belknap Press of Harvard University Press; 2014. DOI: Consistently nding and addressing our knowledge gaps is https://doi.org/10.4159/9780674419377 the only way to overcome our unconscious biases, refine our 3. Abagnale FW. About Frank Abagnale. Frank W. Abagnale official website. Accessed leadership skills, and become more successful problem- September 16, 2020. https://www.abagnale.com/aboutfrank.htm 4. DiFeliciantonio C. Feud over Black Lives Matter stencil erupts in SF’sPacific solvers for the hierarchies we serve. Heights. San Francisco Chronicle 2020 Jun 14; 2020. Accessed September 16, 2020. Good people, like yourself and like me, have unconscious https://www.sfchronicle.com/bayarea/article/Feud-over-Black-Lives-Matter-stencil-erupts- in-15338715.php bias; having unconscious bias is a human condition. But 5. Einstein A. The ultimate quotable Einstein. Princeton, NJ: Princeton University Press; 2010. those of us who learn to recognize and overcome our un- 6. Prottas DJ, Nummelin MR. Theory X/Y in the health care setting: Employee perceptions, attitudes, and behaviors. Health Care Manag 2018 Apr/Jun;37(2):109-17. DOI: https://doi. conscious bias become more impactful and powerful org/10.1097/HCM.0000000000000210, PMID:29521893 1 stewards of society. v 7. Morse JJ, Lorsch JW. Beyond theory Y. Harvard Business Review May; 1970; 1970. Accessed September 20, 2020. https://hbr.org/1970/05/beyond-theory-y Author contributions 8. Heath-Thornton D. Restorative justice. Encyclopædia Britannica, Inc. Accessed 26 Aug Emberton conceived and designed the analysis, collected and analyzed the 2018. https://www.britannica.com/topic/restorative-justice data, copyrighted the belief-behavior systems archetype, and wrote the manuscript. 9. Maruna S. The role of wounded healing in restorative justice: An appreciation of Albert Eglash. Restorative Justice 2014 Apr;2(1):9-23. DOI: https://doi.org/10.5235/20504721.2.1.9 10. Carnegie Corporation of New York. Voting rights: A short history. New York: Carnegie Disclosure Statement Corporation of New York; 2019. Accessed September 19, 2020. https://www.carnegie.org/topics/ The author(s) have no conflicts of interest to disclose. topic-articles/voting-rights/voting-rights-timeline/?gclid=CjwKCAjw2Jb7BRBHEiwAXTR4jd- v0lCY-_zczp5BSbfNSJFBiVqY7OSeW_sdhjhX8j7iqqqbC90tZRoC-d0QAvD_BwE 11. Klein E. The most depressing graphic for members of Congress. The Washington Post Funding Sponsors or Financial Contributors 2013 Jan 14; 2013. Accessed September 22, 2020. https://www.washingtonpost.com/ None. news/wonk/wp/2013/01/14/the-most-depressing-graphic-for-members-of-congress/

178 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.199 n COMMENTARY The Ally Book Club: A Tool for Challenging Racism

Jeffrey B Ritterman, MD1; Miranda Ritterman Weintraub, PhD, MPH2 Perm J 2021;25:20.253 E-pub: 03/01/2021 https://doi.org/10.7812/TPP/20.253

e Black Lives Matter movement has encouraged many of How to be an Antiracist is listed by the New York Times us to challenge our personal racism and to uproot the racism Book Review as 1 of 100 Notable Books for 2019. DiAngelo’s within our institutions. is is a national project, long overdue. White Fragility is a New York Times #1 Bestseller. e While we have watched with interest the toppling of American Journal of Public Health wrote that Metzl’s Confederate statues, we were surprised to learn that we, in Dying of Whiteness “has the potential to transform thinking medicine, have statues of our own that need removal. e so thoroughly that if offers to change a paradigm.” marble statue of Dr James Marion Sims that once graced In addition to these books, we also found a journal article New York City’s Central Park no longer stands. For years, it helpful in understanding how racism has penetrated American proudly faced the New York Academy of Medicine. Other medicine. e article, in the prestigious journal e Proceedings statues of Dr Sims can be found at South Carolina’s of the National Academy of Sciences,6 showed that half of the Statehouse and on the Alabama State Capitol grounds. medical students and medical residents tested had the Dr Sims himself once stood at the top of the American misbelief that African Americans are less sensitive to pain medical hierarchy. In 1875 he was elected President of the than white patients and would have undertreated them in American Medical Association. He has been venerated as a clinical situation. the “Father of Gynecology” and as a selfless benefactor to e books and the article changed us. We were shocked women. But there was a malevolent side to Dr Sims. He and horrified by some of what we learned. We also felt a not only was a slaveholder himself, but he practiced and responsibility to share the information widely with everyone perfected his gynecological surgery on slaves. Despite the involved in medical care. Our mission is to educate ourselves availability of ether at the time, he chose to do this without and our colleagues and then to actively uproot racism in anesthesia. e slave patients would cry out as they were medicine and in healthcare. forcibly held down while Sims sliced and sutured their We have started a Book and Journal Club within Kaiser genitalia. Our work in uprooting racism in medicine will Permanente to begin to address racism in medicine. e not be finished while statues of Sims still stand. But there leader of the Kaiser East Bay Club, Dr Noriko Yoshikawa, is much more that we also need to do. named it the “Becoming an Ally Book Club” and explained One of us is a white doctor, the other a white epide- its purpose this way: miologist. We both had the privilege of attending excellent is book club is geared towards those who are wondering colleges and postgraduate schools. We have participated in how to combat racism. e recent murders of George Floyd, continuing education throughout our careers. But we did Ahmaud Arbery and Breonna Taylor have been, for many of not learn about Dr Sims and his mistreatment of slaves in us, the most recent wake up call, that there is something very college, in medical or graduate school, nor during our many wrong in our society. And yet, we find ourselves not knowing continuing education courses and trainings. We learned exactly what we can do to help move things in the right about Dr Sims from reading “Medical Apartheid”1 by direction. In this book club, we will be exploring books to help Harriet Washington and “Medical Bondage”2 by Deirdre guide us, in better understanding racism in our society, and Cooper Owens. Since we did not learn about racism in what part we are playing in racism. medicine during our formal education, we began a self- But what does it mean to combat racism? Ibram Kendi education project. We also read White Fragility: Why it’s asserts that the only way to combat racism is to “become so Hard for White People to Talk about Racism3 by Robin an antiracist.” He explains: DiAngelo, How To be an Antiracist4 byIbramX.Kendi, But there is no neutrality in the racism struggle. e opposite and Dying of Whiteness: How the Politics of Racial Resentment of “racist” isn’t “not racist.” It is antiracist…One endorses either is Killing America’s Heartland5 by Jonathan M. Metzl. the idea of a racial hierarchy as a racist, or racial equity as an e level of scholarship in these books is quite extraor- antiracist…ere is no in-between safe space of “not racist.”7 dinary. Washington has won many awards for Medical Apartheid, including the National Book Critics Circle Award fi ’ Author Affiliations for Non ction. Cooper Owens Medical Bondage breaks new 1Department of Cardiology, Kaiser Permanente Richmond Medical Center, Richmond, CA ground by telling the stories of the exploited slaves who 2Department of Graduate Medical Education, Kaiser Permanente Oakland Medical Center, Oakland, CA endured experimental surgeries that were the basis for advances in gynecology. Kendi won the National Book Corresponding Author Miranda Ritterman Weintraub, PhD, MPH, ([email protected]) Award for his earlier book, Stamped From the Beginning. Keywords: antiracist, Black, racism, race, White The Permanente Journal·https://doi.org/10.7812/TPP/20.253 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 179 COMMENTARY The Ally Book Club: A Tool for Challenging Racism

Kendi teaches us that, “the only way to undo racism is to happily offer our help to anyone who would like to begin consistently identify and describe it—and then dismantle it.”7 an Ally Book Club. In his remarkable book, Kendi shareswithushissuccessful While reading books and articles is a great place to start, struggle against advanced colon cancer. He goes on to describe we must be mindful that we also need to act. In the “Book racism as a cancer and asks, “What if we treated racism in the Club” discussion groups, we can begin, together, to address way we treat cancer?” For a medical audience there could larger issues. What hiring practices should Kaiser Perma- be no more apt analogy. Kendi’s prescription: nente adopt in order to diversify our workforce? What can Saturate the body politic with the chemotherapy or im- Kaiser do to build the educational pipeline to support munotherapy of antiracist policies that shrink the tumors of African American success from elementary school through racial inequities, that kill undetectable cancer cells. Remove college and medical/nursing/physician assistant school to a any remaining racist policies, the way surgeons remove the successful career at Kaiser? What trainings do we need to tumors. Ensure there are clear margins, meaning no cancer develop to ensure that our workforce is antiracist? What cells of inequity left in the body politic, only the healthy cells of should the Kaiser Medical School curriculum include to equity…Detect and treat a recurrence early, before it can educate future doctors to become antiracist providers? What grow and threaten the body politic.”8 special efforts are needed to address racial disparities in One of us (J.B.R.) joined Kaiser Richmond in 1981, health outcomes? What research should Kaiser engage in to while the other (M.R.W.) was just a child. Since that time lessen racial health disparities? Kaiser Permanente has made enormous strides in providing Together, we can begin to address these questions, and excellent health care and has become a leader in American more, as we move toward a healthier and more just society. medicine. What will it take for Kaiser to become the leader It’s what the work of doctoring is all about. v in uprooting racism from medicine and healthcare? We owe our excellence in providing healthcare to an Disclosure Statement fl integrated team of primary care providers, specialists, The author(s) have no con icts of interest to disclose. subspecialists, nurses, technicians, engineers, pharmacists, Authors’ Contributions housekeepers, managers, planners, researchers, epidemiol- JBR development of manuscript. MRW revision of manuscript. ogists, phlebotomists, medical assistants, and more. We also benefited from visionary leaders. To become a leader in Funding uprooting racism, we need everyone on our team to un- None. learn their own personal racism and to learn how to be- References come an antiracist. We need our leadership to once again 1. Washington HA. Medical apartheid. New York, NY: Anchor Books; 2008. provide us with a vision that propels us into a positive and 2. Owens DC. Medical bondage. Athens, GA: University of Georgia Press; 2017. 3. Diangelo R. White fragility. Boston, MA: Beacon Press; 2018. just future. 4. Kendi IX. How to be an antiracist. New York, NY: Random House; 2019. Many of us in medicine have attended journal clubs to 5. Metzl JM. Dying of whiteness. New York, NY: Basic Books; 2019. help keep us up to date. It will take a lot of education to 6. Hoffman KM, Trawalter S, Axt JR, Oliver MN. Racial bias in pain assessment and treatment recommendations, and false beliefs about biological differences between blacks and getusuptodateonuprootingracism.WhatwillKaiser whites. Proc Natl Acad Sci USA 2016 April;113:4296-301. DOI: https://doi.org/10.1073/ be like a year from now if all of the white staff,and pnas 15160471131073 Accessed August 20, 2020. 7. Kendi IX. How to be an antiracist. New York: Penguin Random House LLC; 2019; p 9. everyone else who wants to, join Ally Book Clubs? While 8. Kendi IX. How to be an antiracist. New York: Penguin Random House LLC; 2019; we have just begun and our experience is limited, we p 237-8.

180 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.253 n COMMENTARY Etiology of Belief–behavior Systems and Hierarchies

Mihal Emberton, MD, MPH, MS1 Perm J 2021;25:20.269 E-pub: 3/3/2021 https://doi.org/10.7812/TPP/20.269

ABSTRACT: maintains a democratic belief–behavior system that allows for In order to understand the well documented patterns of enhanced problem solving and values each person’sintel- mental health, human learning, human behavior, and the me- lectual currency as a resource for learning. is also describes chanics of hierarchies such as academic institutions, political a true democracy and social justice.5 systems, and business organizations, one must discover the rules Although many evolutionary biologists believe the second and pathways that cause those patterns. My Belief-Behavior goal of life is to pass on one’s genes through reproduction, Systems archetype is the first of its kind to reconcile the theories also known as highest fitness,6 there have been well-recognized and insights from social sciences, political science, psychiatry, and evolutionary biology into a unifying paradigm which explains limitations to this theory. Martin Nowak, PhD, Professor how socialization and human interactions evolved into the pat- of Mathematics and Biology at Harvard University and terns we recognize today. More importantly is that this new Director of the Program for Evolutionary Dynamics, contribution to our understanding of human behavior within recognized that the evolutionary theory of survival for the hierarchies provides the key insights to guide the restoration and purpose of reproduction cannot be explained by the evolution repair of our dysfunctional hierarchies which, unfortunately, all of social cooperation because cooperation often leads to too often oppress, manipulate and exploit our humanity. decreased reproductive fitness.7 In addition, Hudson Kern Reeve, Professor of Neurobiology and Behavior at Cornell University, and Paul Sherman, PhD, Professor Emeritus INTRODUCTION of Neurobiology and Behavior at Cornell University, also is manuscript is the third in a 3-part series that began recognized the incompletenatureofthetheoryofsurvival fl with “Learning in Humans Versus Hierarchies,” followed in order to reproduce because this theory con icts with “ by “Unconscious Bias Is a Human Condition, published in behaviors such as recreational sex, incest avoidance, cigarette ”8 e Permanente Journal.” and alcohol use, adoption, and abortion. Although some scientists might argue that behaviors such EVOLUTION OF COOPERATION FOR IMPROVED SURVIVAL as cooperation or recreational sex are merely outliers, my AND QUALITY OF LIFE framework for belief–behavior systems,5 on the other hand, It is generally understood that, once born, the first goal of addresses these known limitations of the theory that we life is survival. To survive, individuals must secure suste- survive in order to pass on our genes, by providing inclusive fi nance (food and water), protection from the elements understanding about the evolutionary bene t of coopera- (shelter and clothing), and safety (protection from threats to tion, recreational sex, adoption, and abortion as behaviors life). And to solve the problem of survival, individuals must that improve individual survival and quality of life (physical fi allocate their resources for learning, time, assets, labor, and comfort, intellectual growth, and emotional ful llment).  — knowledge in such a way as to solve the problem of survival. e same resources used to solve the problem of survival — Societies form to combine the resources of time, assets, time, assets, labor, and knowledge are also needed and used labor, and knowledge to secure survival more easily.1-3 e to secure quality of life. And hierarchies also have the ca- combination of these resources leads to a reorganization of pacity to help us improve our quality of life in addition to those resources, also known as division of labor,which improving survival. allows for the development of expertise to solve focused In summary, the evolution of cooperation improves social survival problems more effectively. With the division survival by better solving the problems of securing sus- of labor also comes the development of hierarchies,creating tenance, acquiring protection from the elements and supervisors to solve the new problems created by the safety, as well as improving quality of life by improving fi combination of resources and the division of labor. physical comforts, emotional ful llment, and intellectual An ideal or true hierarchy forms when the people doing the work (laborers or subordinates) select a supervisor to oversee Author Affiliations the division of labor and allocation of resources. Evolu- 1Department of Adult and Family Medicine, The Permanente Medical Group, Oakland, CA tionary biologists call this process prestige strategy and note that the selection of an individual to act as the supervisor results Corresponding Author from admiration and voluntary deference.4 More specifically, the Mihal Emberton, MD, MPH, MS ([email protected]) group tends to elevate a person to a position of power who Keywords: burnout, engagement, learning process, organizational culture change, overcoming unconscious bias, successful leadership, teaching collaboration The Permanente Journal·https://doi.org/10.7812/TPP/20.269 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 181 COMMENTARY Etiology of Belief–behavior Systems and Hierarchies

development. erefore, there is an evolutionary benefit for other alternative is forsubordinatestousearesistance cooperation, the development of hierarchies, and thus belief–behavior system, challenging supervisors on their development of belief–behavior systems to navigate coop- knowledge or performance gap in an attempt to drive eration and hierarchies. learning for the benefit of the hierarchy, and thus benefitting the subordinates who rely on the hierarchy. e resistance EVOLUTION OF COMPETITION BETWEEN INDIVIDUAL strategy, however, is risky, because often autocratic supervi- AND HIERARCHY sors view confrontation as insurgency and put more energy Although there are benefits of forming a hierarchy, such into coercing subordinates to submit to their imperfect as improved ability to solve survival and quality-of-life solution.10 problems, the creation of a hierarchy at the same time creates a power differential in relation to the allocation PATHOLOGY OF AUTOCRATIC HIERARCHIES and control of the resources for learning. In a hierarchy, When subordinates suffer coercion and conformity from the supervisor generally gains control over the collective autocratic supervisors, the subordinates, justifiably, feel resources of time, assets, and labor to solve both the focused oppressed and replaceable,5 and show signs of depression social survival/quality-of-life problems as well as the prob- and anxiety. e fact that 7.1% of adults—17.3 million lems created by the division of labor (hierarchy), whereas the people—in the US reported symptoms of severe depression subordinates only generally retain control over their own in 2017,11 and 19.1% of adults—an estimated 46.5 million knowledge and expertise.5 people—reported symptoms of anxiety,12 does not mean When the supervisors in a hierarchy are functioning in that 1 in 4 US adults has dysfunctional brain chemistry, but their democratic belief–behavior systems, they not only value rather that the hierarchies in which we work and live are the knowledge and expertise of subordinates as intellectual often themselves dysfunctional. eWorldHealthOr- currency to solve organizational problems, but also they ganization emphasizes this fact in their recognition and allocate the resources for learning in such a way as to drive definition of “burnout” as an occupational phenomenon.13 organizational innovation and growth. However, the cre- In addition, our understanding about adverse childhood ation of a hierarchy, a power differential, also comes with a experiences14 and trauma-informed care15 highlight our very fundamental flaw: it incubates the unconscious bias, the burgeoning realization that dysfunctional hierarchies have knowledge or experience gap, of the supervisor.9 When a negative effect on subordinates. supervisors toggle into their autocratic belief–behavior system, A very public incidence of the negative effects of an they unconsciously overlook the fact that the subordinates’ autocratic hierarchy occurred earlier this year when US knowledge, insights, and experience are a resource for Navy Captain Brett Crozier was forced to engage his re- learning, the intellectual currency needed to solve organi- sistance belief–behavior system to highlight the autocracy, the zational problems, and thus misallocate resources to drive knowledge gap of his hierarchy, to save his crew from a conformity and status quo. e moment supervisors un- coronavirus outbreak on his ship.16 If Captain Crozier’s consciously believe they have nothing to learn from their supervisor truly valued Crozier’s knowledge, insights, and subordinates is the moment they close themselves off to experience, the supervisorwouldhaveusedCrozier’s discovery.5 intellectual currency to help solve the problem of the Even though an autocratic supervisor’s intention is to coronavirus outbreak on the ship. Instead, Crozier had to protect the common good of the hierarchy, he or she be- work outside the chain of command to challenge his haves in such a way as to protect the hierarchy at the expense hierarchy to validate and act upon his insights. And as a of subordinate engagement in problem solving, suppressing result, the autocratic hierarchy felt vulnerable and subordinates’ intellect. When an idea challenges autocratic threatened; it attacked and blamed Crozier as an insur- supervisors’ unconscious knowledge gap, they feel vulner- gent, and engaged in coercive behaviors to end Crozier’s able and threated by the new idea, attack and blame sub- career.17 Alternatively, Crozier could have chosen to ordinates for making them feel bad, and engage in coercive safeguard his career by using his disengagement belief– interactions.5 is then forces subordinates to respond in 1 behavior system, letting his hierarchy keep its knowledge of 2 ways (Figure 1). gap regarding the severity of the coronavirus outbreak on If subordinates want to protect their place in the hierarchy his ship and maintaining the organizational status quo, but for individual survival/quality of life, they will use a dis- this would have placed the health and safety of his crew at engagement belief–behavior system, aligning with autocratic risk for the benefit of his individual survival in the supervisors to make the autocratic supervisors feel good about hierarchy. their knowledge or experience gaps, while acknowledging Autocratic dysfunction is not only observed in the mil- this will also maintain the organizational status quo. e itary, but also it is observed in our other hierarchies.

182 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.269 COMMENTARY Etiology of Belief–behavior Systems and Hierarchies

Figure 1. Subordinate survival and quality-of-life belief–behavior systems in autocratic environment: disengagement vs resistance. Copyright 2020 by Mihal Emberton, MD, MPH, MS

The Permanente Journal·https://doi.org/10.7812/TPP/20.269 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 183 COMMENTARY Etiology of Belief–behavior Systems and Hierarchies

Historically, as many American businesses began to procure of suffocation and status quo for the subordinates, which profits at the expense of subordinates, a national movement impacts subordinates’ survival and quality of life negatively, to advocate for and protect America’s subordinate workers creating subordinate disenfranchisement. Once a hierarchy from the antisocial behaviors of their hierarchies began with has created a significant level of disenfranchisement, it the establishment of the American Federation of Labor in becomes susceptible to manipulative–exploitive belief–behavior 1886,18 followed by the establishment of the US Depart- systems (Figure 2). ment of Labor in 1913,19 and continued with the enactment When a disengagement belief–behavior system (collabora- of additional policies to protect the survival and quality of tive interactions with conformity process) is no longer used life of subordinate workers.20 ese movements and to survive supervisors’ autocratic belief–behavior system, but policies, however, have not been enough to correct the rather is used to harness the power from the disenfranchised autocratic dysfunction of many of our hierarchies. e for individual survival/quality of life at the expense of the current Teachers’ Union contract in San Francisco, for hierarchy, it becomes a manipulative belief–behavior system example, highlights that the knowledge, insights, and that describes, for example, cults and fascism. A dema- experiences of teachers must not be interpreted as in- gogue, recognizing when an autocratic political system has surgency and must rather be valued and protected, when it created disenfranchised constituents, aligns with the dis- states the following: enfranchised around that truth, telling them that if they give “e District and the Union agree that academic freedom the demagogue power, the demagogue will right the policy [the right and responsibility to study, investigate, present, wrongs that oppress them. UK Conservative Party Advisor interpret, and discuss all the relevant facts and ideas in the Dominic Cumming’s “Take back control” slogan around field of his or her professional competence] is essential to the UK’s 2016 Brexit referendum,22 and Donald Trump’s the fulfillment of the purposes of the San Francisco Unified “Drain the swamp,23” are classic and recent examples of this School District, and they acknowledge that fundamental pattern of alignment. e demagogue, however, cannot need to protect teachers from unreasonable censorship or right the policy wrongs, because it was the existence of the restraint which might interfere with their obligation to pursue disenfranchised in the first place that allowed the dema- truth in the performance of their jobs with the District.”21 gogue to gain power, and thus the demagogue must In addition to oppressing subordinates and driving or- maintain the status quo to maintain disenfranchisement to ganizational conformity and status quo, autocratic super- maintain his or her power. visors tend to promote further autocracy within a hierarchy. is is also where the development of false hierarchies and When supervisors are in their autocratic mind-set, subor- manipulative collaboration comes into play. In order for the dinates who make them feel good about their knowledge demagogue to try to relieve the sense of oppression that the gaps are subordinates who use their disengagement belief– disenfranchised are experiencing, the demagogue invents a behavior systems, and thus those subordinates are more false hierarchy, telling the disenfranchised that it is okay for likely to be promoted than subordinates who try to drive them to oppress another group because this “other group” is change through their resistance belief–behavior system. truly at the bottom of the social hierarchy and is the reason Subordinates who use their disengagement belief–behavior for their disenfranchisement (eg, sexism, racism, agism, systems to survive within the hierarchy do not gain much xenophobia, etc) in the first place. experience in how to facilitate or participate in learning Such a demagogue, who harnesses the power of the processes,sothatwhentheyarepromotedtosupervisor, disenfranchised for individual gain at the expense of the they often do not have the understanding or skills to hierarchy, may also toggle toward the resistance belief–behavior toggle into their democratic belief–behavior system. side of the axis—coercive interactions to drive a learning Without the knowledge, experience, and skills for fa- process—that becomes an exploitive belief–behavior system cilitating learning processes, disengaged subordinates- when it is used to gain resources and power at the expense turned-supervisors tend to mirror the autocracy that got of the disenfranchised, and, thus, at the sociopolitical level, them promoted in the first place, creating a hierarchy of describes slavery, imprisonment, and war. autocratically minded supervisors. Unfortunately, autoc- Although the Diagnostic and Statistical Manual of Mental racy breeds autocracy. Disorders is an extensive collection of the patterns of mental health signs and symptoms,24 it is far from describing the AUTOCRACY LEADS TO VULNERABILITY AND FURTHER causes for those patterns. e Diagnostic and Statistical DYSFUNCTION IN HIERARCHIES Manual of Mental Disorders, for example, describes the Autocracy, the fundamental flaw of hierarchies, is also what patterns of personality disorders such as histrionic, anti- makes hierarchies vulnerable to additional dysfunction. social, narcissistic, and borderline as maladaptive, creating Once a hierarchy toggles into autocracy, it harbors a culture social disfunction,25 without explaining that it is the drive

184 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.269 COMMENTARY Etiology of Belief–behavior Systems and Hierarchies

Figure 2. Harnessing the disenfranchised to improve individual survival and quality of life at the expense of the hierarchy: manipulative vs exploitive. Copyright 2019 by Mihal Emberton, MD, MPH, MS

The Permanente Journal·https://doi.org/10.7812/TPP/20.269 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 185 COMMENTARY Etiology of Belief–behavior Systems and Hierarchies

for individual survival and improved quality of life at the emotional fulfillment and intellectual growth to the parent as physical or intellectual expense of others, at the expense of the well. And aren’t we generally content if our children do disenfranchised and the hierarchy, that creates the patho- not have children of their own if that is the quality of life logical social dysfunction. they want and choose? In addition, abortion and rec- Although the development of belief–behavior systems reational sex, for example, also generally improve survival evolved to help us collaborate around improved survival and and quality of life and are thus evolutionary adaptations, quality of life, these same belief–behavior systems can also rather than outliers to the theory that our goal in life is to become a social liability. When the behaviors for individual reproduce our genes. survival and quality of life are used at the expense of others’ Although evolutionary biologists Reeve and Sherman try survival and quality of life, those belief–behavior systems to redefine adaptation as “a phenotypic variant that results in become pathological or antisocial. Now that the rules and the highest fitness among a specified set of variants in a pathways for belief–behavior systems are clear, it is self- given environment”8 my work—defining the dichotomies, evident that a true or ideal hierarchy—made up of demo- continua, and evolution of human behavior—reinforces the cratically minded supervisors—is what protects and harnesses interpretation of evolutionary theory by Kampourakis and the evolutionary advantage of cooperation, avoiding the Offer that humans can use a variety of belief–behavior systems creation of disenfranchised subordinates and preventing the (phenotypes and adaptations) to survive and improve their dysfunction or corruption from autocratic–manipulative– quality of life based on their environment (place in the exploitive belief–behavior systems. hierarchy and type of hierarchy). Einstein noted, “Atheory is the more impressive the greater the simplicity of it EVOLUTIONARY LEGACY: GENES OR KNOWLEDGE? premises, the more different kinds of things it relates and Some evolutionary biologists interpret Darwin’s and the more extended its area of applicability,”29 which Spencer’s theories of evolution as processes that describe the describes my belief–behavior systems archetype accu- evolution of the mind, rather than the reproduction of rately. e next step is to use this archetype to refine our genes. Kostas Kampourakis (Biology Faculty at the Uni- educational, business, political, and social hierarchies to versity of Geneva, and editor of a book series on science) become true democracies, not only to improve their and colleagues discuss Darwin’stheoryofnatural selection function, but to improve the health and well-being of as a “creative process” that favors the accumulation of those who live and work within those hierarchies—in adaptations.26 John Offer, Professor of Social eory and essence, to restore and rebuild our humanity. v Policy at the Institute for Research in Social Sciences and School of Criminology, Politics, and Social Policy at the Disclosure Statement University of Ulster, UK, describes Spencer’stheoryof The author has no conflicts of interest to disclose. evolution of individual life and social life as a process of ’ Authors’ Contributions individual adaptations to one s environment such that one The author conceived and designed the analysis, collected and analyzed the 27 canpassontheiracquired characteristics to others. One data, copyrighted the belief–behavior systems archetype, and wrote the does not “acquire” genes during one’s lifetime, but one does manuscript. acquire knowledge, insight,andexperience,whichwedo pass on to others—our children, subordinates, peers, and Funding The author did not receive any funding. leaders—to improve our social survival and quality of life. We can think of our evolutionary legacy in our role as References parents. As a parent, what is the one thing that you hope or 1. Wilson EO. One giant leap: How insects achieved altruism and colonial life. Bioscience wish for your child or children? Do you wish they can have 2008 Jan;58(1):17–25. DOI: https://doi.org/10.1641/b580106 2. Clutton-Brock T. Cooperation between non-kin in animal societies. Nature 2009 Nov; as many children as possible, starting as early as possible, to 462(7269):51–7. DOI: https://doi.org/10.1038/nature08366, PMID:19890322. pass on their genes in a robust fashion, understanding that 3. Nowak MA, Tarnita CE, Wilson EO. The evolution of eusociality. Nature 2010 28 Aug;466(7310):1057–62. DOI: https://doi.org/10.1038/nature09205,PMID: this strategy may decrease their individual survival and 20740005. leave them fewer resources to improve their individual quality 4. Jimenez AV, Mesoudi A. Prestige-biased social learning: Current evidence and ff outstanding questions. Nature 2019 Feb;5(20):1–12. DOI: https://doi.org/10.1057.s41599- of life? Or,doyouwishforthemtobee ective thinkers and 019-0228-710.1057/s41599-019-0228-7 problem solvers so they can improve their individual survival 5. Emberton M. Learning in humans versus hierarchies. Perm J 2020;20:19.083. DOI: and quality of life? eideathatwesurvivetoimproveour https://doi.org/10.7812/TPP/19.083 6. Orr HA. Fitness and its role in evolutionary genetics. Nat Rev Genet 2009 Aug;10(8): quality of life (physical comforts, emotional fulfillment, and 531–9. DOI: https://doi.org/10.1038/nrg2603, PMID:19546856. intellectual development) also explains why we can and often 7. Nowak MA. Five rules for the evolution of cooperation. Science 2006 Dec;314(5805): ff 1560–3. DOI: https://doi.org/10.1126/science.1133755, PMID:17158317. do invest in raising children who are not our genetic o spring, 8. Reeve HK, Sherman PW. Adaptation and the goals of evolutionary research. Q Rev Biol because nurturing the development of children often provides 1993 Mar;68(1):1–32. DOI: https://doi.org/10.1086/417909

186 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. The Permanente Journal·https://doi.org/10.7812/TPP/20.269 COMMENTARY Etiology of Belief–behavior Systems and Hierarchies

9. Emberton M. Unconscious bias is a human condition. Perm J 2021;25:20.199. https://doi. 20. U.S. Department of Labor. Summary of the major laws of the Department of Labor. org/10.7812/TPP/20.199. Accessed December 8, 2020. www.dol.gov/general/aboutdol/majorlaws 10. Emberton M. Autocratic–democratic and manipulative–exploitive belief–behavior 21. United Educators of San Francisco. Contract between San Francisco Unified School District and systems; 2019. United Educators of San Francisco, covering certificated personnel July 1, 2017–June 30, 2020. 11. National Institute of Mental Health. Major depression: Prevalence of major depressive Accessed December 10, 2020. https://archive.sfusd.edu/en/assets/sfusd-staff/contract%20and% episode among adults. 2019. Accessed October 1, 2020. www.nimh.nih.gov/health/ 20salary%20schedules/Certificated%20Collective%20Bargaining%20Agreement%207-1-17% statistics/major-depression.shtml 20thru%206-30-20%20(pre-final).pdf 12. National Institute of Mental Health. Any anxiety disorder: Prevalence of any anxiety 22. Adam K. ‘Get Brexit done’: Boris Johnson’s effective but misleading slogan in the British disorder among adults. 2017. Accessed October 1, 2020. www.nimh.nih.gov/health/ election. The Washington Post; 2019. Accessed December 8, 2020. www.washingtonpost. statistics/any-anxiety-disorder.shtml com/world/europe/get-brexit-done-boris-johnsons-effective-but-misleading-slogan-in-the-uk- 13. World Health Organization. Burn-out an “occupational phenomenon”: International election/2019/12/12/ec926baa-1c62-11ea-977a-15a6710ed6da_story.html classification of diseases. 2019. Accessed October 1, 2020. www.who.int/mental_health/ 23. Editorial Board. Trump shattered his promise to ‘drain the swamp’: The self-dealing would evidence/burn-out/en be epic in a second term. The Washington Post; 2020. Accessed October 9, 2020. www. 14. Walsh D, McCartney G, Smith M, Armour G. Relationship between childhood washingtonpost.com/opinions/2020/09/16/trump-promised-to-drain-swamp-self-dealing/? socioeconomic position and adverse childhood experiences (ACEs): A systematic review. arc404=true J Epidemiol Community Health 2019 Dec;73(12):1087–93. DOI: https://doi.org/10.1136/ 24. American Psychiatric Association. Diagnostic and statistical manual of mental disorders jech-2019-212738, PMID:31563897. (DSM-5). Accessed September 29, 2020. www.psychiatry.org/psychiatrists/practice/ 15. Butler LD, Critelli FM, Rinfrette ES. Trauma-informed care and mental health. Dir dsm Psychiatr 2011;31:197–210. 25. Skodol, A. Overview of personality disorders. Merck manual 2019. Accessed October 6, 16. Gafni M, Garofoli J. A captain’s choice. San Francisco Chronicle; 2020. Accessed 2020. www.merckmanuals.com/professional/psychiatric-disorders/personality-disorders/ October 1, 2020. www.sfchronicle.com/nation/article/Capt-Crozier-The-man-who-risked- overview-of-personality-disorders his-career-to-15179363.php 26. Kampourakis K, Nehm R, Wong ASL, El-Hani CN. Peer review and Darwinian 17. Miller TC, Rose M. The firing of the captain of the USS Theodore Roosevelt is the selection. Sci Educ 2015 Nov;24(9–10):1055–7.DOI: https://doi.org/10.1007/ latest in a troubling Navy pattern. Business Insider; 2020. Accessed October 3, s11191-015-9789-z 2020. www.businessinsider.com/firing-of-uss-theodore-roosevelt-captain-fits-navy- 27. Offer J. From ‘natural selection’ to ‘survival of the fittest’: On the significance of Spencer’s discipline-pattern-2020-4 refashioning of Darwin in the 1860s. J Classical Sociol 2014 May;14(2):156–77. DOI: 18. The early labor movement. PBS history detectives: Special investigations. Accessed https://doi.org/10.1177/1468795X13491646 December 8, 2020. www.pbs.org/opb/historydetectives/feature/the-early-labor- 28. World Health Organization. Maternal mortality. 2019. Accessed December 8, 2020. www. movement/ who.int/news-room/fact-sheets/detail/maternal-mortality 19. MacLaury J. A brief history: The U.S. Department of Labor. U.S. Department of 29. Einstein A. The ultimate quotable Einstein. Princeton, NJ: Princeton University Press; Labor. Accessed December 8, 2020. www.dol.gov/general/aboutdol/history/dolhistoxford 2010.

The Permanente Journal·https://doi.org/10.7812/TPP/20.269 The Permanente Journal·For personal use only. No other uses without permission. Copyright © 2021 The Permanente Press. All rights reserved. 187