IBM Health in Oncology

Scientific Evidence 2020 Contents

03 Foreword

06 Key studies 06 Clinical decision support 14 Clinical trials 18 Genomics 24 Real world data

28 Bibliography Foreword

Nathan Levitan, MD, MBA publications, precision oncology, and clinical We are tackling the early application of AI Chief Medical Officer trial availability. in care by working closely with our IBM Watson Health Oncology & Genomics users to understand their needs and evolve IBM Watson Health has been collaborating our products. As an oncologist, I have seen first-hand with physicians across the globe to the very human toll of cancer. With the understand their needs and to leverage Throughout this journey, we recognize the rising incidence and mortality of cancer -based technologies strengths of our clients and partners, viewing globally, physicians and patients around the to help them facilitate the delivery of high our progress as a continuous evolution world face unprecedented complexity in quality, cost effective care. Many of these that must be rooted in scientific data that navigating treatment while health systems solutions apply to all types of healthcare, provides insight into how our products are are strained managing care for growing including but not limited to oncology. impacting care teams today in the real world. populations of patients. Examples include secure hybrid data To-date, more than 100 studies have been platforms, large scale claims-based analytics, conducted including the evaluation of our As demand for oncology care rises, the natural language processing to ingest data products’ efficacy. Here we provide a sample global oncology workforce has not kept pace, from structured and unstructured sources, of these publications for your consideration. leading to a shortage of expertise.1 As a image analysis, drug discovery, genomics, result, physicians face formidable productivity longitudinal care management, and benefits and workflow challenges. In the US, over 40% administration. We also provide timely and of physicians experience burnout according to comprehensive resources for healthcare 01. Wilson BE et al. Lancet Oncol. 2019;20(6):769- a 2018 survey. While this is an improvement professionals pertaining to drug information 780. from previous years studied, physicians and broad-based medical practice. 02. Shanafelt TD et al. Mayo Clin Proc. remain at an increased risk for burnout.2 EMR 2019;94(9):1681-1694. Foreword documentation and payer-preauthorization Examples of capabilities that are focused requirements are noteworthy. In some specifically on cancer care include decision Key studies: developing countries resource constraints support for treatment selection, literature Clinical decision support are significant. Globally, many physicians curation, clinical trials matching, and Clinical trials are challenged by the need to stay current interpretation of next generation sequencing Genomics on the management of multiple neoplastic results. Our solutions have supported the Real world data diseases, the proliferation of research delivery of cancer care in over 15 countries. Bibliography 3 Gretchen Purcell Jackson, cancer care. Our natural language processing handful of studies have shown that use of MD, PhD, FACS, FACMI, FAMIA has been found to accurately identify relevant oncology decision support helped physicians 2 Vice President and Chief Science Officer publications from bibliographic databases. identify treatment options they did not IBM Watson Health Automated clinical trial screening technology previously consider that lead to changes has been shown to reliably exclude ineligible in therapy in 5 to 13.6% of cases12,13 and As a practicing pediatric surgeon and health patients3,4,5 and accurately determine in one study led to improved adherence informatician, I know well the important trial eligibility for breast and to guidelines.13 At the , use of role of scientific evidence in supporting patients.4,5 In the area of cancer genome automated clinical trial screening technology clinical decisions and informing technology- variant data annotation and categorization, supported increasing patient enrollment in investment decisions. IBM Watson Health is automated tools have been shown to have breast cancer trials by 84%14 and reduced focused on combining data, analytics, and high concordance with expert molecular screening time by 50% in lung cancer artificial intelligence (AI) to create solutions tumor boards in identifying actionable trials.15 AI-enabled technology not only that can empower decision makers with mutations from next generation tumor identified actionable mutations of genes with actionable insights, with the goal of helping sequencing data.6,7 FDA-approved targeted therapies, but also them to improve health and healthcare provided additional therapeutic insights in a delivery. Watson Health is building on IBM’s Once technical performance has been third of cases surveyed.6,16 long history of leading with science, and demonstrated, the next step in evaluating we are proud to present a growing body a clinical AI tools involves conducting Finally, this compilation wouldn’t be of scientific evidence demonstrating the studies of usability and workflow in the complete without highlighting the value of performance and impact of Watson Health clinical setting. Our scientific portfolio real-world data to inform practice alongside solutions in the area of cancer care. includes studies providing evidence of user traditional prospective research. The rich and patient satisfaction8,9 and improved data assets offered by IBM Watson Health The process of validating these tools requires patient engagement with oncology decision coupled with the experience of our scientists a systematic progression of studies to support.10 A community oncology practice have delivered important insights for current demonstrate the performance, applicability, found a 78% reduction in time to screen practice and identified disparities and gaps and value. The first step in evaluating any patients for clinical trials using automated in care.17,18,19,20 health is maintaining clinical trial screening technology compared the output so that it is technically accurate to manual screening,3,4 and AI-enabled We are grateful to our pioneering or correct. With oncology decision support, technology has completed annotation and collaborators who have provided Foreword a systematic review of evaluation studies categorization of sequencing data in a fraction evidence for the performance, usability, and has demonstrated that therapeutic options of the time required for manual curation.6,11 impact of IBM Watson Health oncology and Key studies: have strong agreement with decisions made genomics solutions. Clinical decision support by expert multidisciplinary tumor boards — Ultimately, IBM Watson Health strives to Clinical trials greater than with individual clinicians.1 These create tools to help clinicians as they work Genomics findings support clinical applicability and help to improve oncology care. When used in Real world data illustrate the need for decision support in multidisciplinary tumor board settings, a Bibliography 4 01. Arriga Y, Hekmat R, Darulis K, Wang S, Felix 04. Beck JT, Rammage M, Jackson GP, Preininger 09. Wang Z, Yu Z, Zhang X. Artificial intelligence-based 15. Levantakos K, Helgeson J, Mansfield AS, W, Dankwa-Mullan I, Rhee K, Jackson GP. A AM, Dankwa-Mullan I, Roebuck MC, Torres A, clinical decision-support system improves cancer Deering E, Schwecke A, Adjei A, Molina J, systematic review of concordance studies using holtzen H, Coverdill SE, Williamson MP, Cahu Q, treatment and patient satisfaction. J Clin Oncol 37, Hocum C, Halfdanarson T, Marks R, Parikh K, Watson for Oncology (WfO) to support breast Rhee K, Vinegra M. Artificial intelligence tool 2019 (suppl; abstr e18303). Pomerleau K, Coverdill S, Rammage M, Haddad cancer treatment decisions: a four-year global for optimizing eligibility screening for clinical 10. Fang J, Zhu Z, Wang H, Hu F, Liu Z, Guo X, Chen J, Li T. Implementation of artificial intelligence experience. In: Proceedings of the 2019 San trials in a large community cancer center. JCO C, Shen Y, Xu Q. The establishment of a new medical (AI) for lung cancer clinical trial matching in a Antonio Breast Cancer Symposium; 2019 Dec Clin Cancer Inform. 2020 Jan; 4:50-59. doi: model for tumor treatment combined with Watson for tertiary cancer center. Annals of Oncology. 2019; 10-14; San Antonio, TX. Philadelphia (PA): AACR; 10.1200/CCI.19.00079 Oncology, MDT and patient involvement. J Clin Oncol. 30(Supplement_2): //doi.org/10.1093/annonc/ Cancer Res 2020;80(4 Suppl):Abstract nr P4- 05. Alexander M, Solomon B, Ball DL, Sheerin M, 2018;36(suppl; abstr e18504). mdz065. 14-05. Dankwa-Mullan I, Preininger AM, Jackson GP, 11. Wrzeszczynski K, Frank M, Koyama T, Rhrissorrakrai 16. Kim M, Snowdon J, Weeararatne SD, Felix W, Lim L, 02. Suarez Saiz FJ, Sanders C, Stevens RJ, Nielson Herath DM. Evaluation of an artificial intelligence K, Robine N, Utro F, Emde A, Chen B, Arora K, Shah Dankwa-Mullan I, Lee YK, Lee E, Jeon Km Lee JS, R, Britt MW, Preininger A, Jackson G. Use of clinical trial matching system in Australian lung M, Vacic V, Norel R, Bilal E, Bergmann E, Vogel J, Zang DY, Kim HJ, Kim HY, Han B. Clinical insights to identify relevant research cancer patients. JAMIA Open, ooaa002, https:// Bruce J, Lassman A, Canoll P, Grommes C, Harvey S, for hematological malignancies from an artificial publications in clinical oncology. J Clin Oncol 37, doi.org/10.1093/jamiaopen/ooaa002 Parida L, Michelini V, Zody M, Jobanputra V, Royyuru intelligence decision-support tool. J Clin Oncol 37, 2019 (suppl; abstr 6558. 06. Patel N, Michelini V, Snell J , Balu S, Hoyle A, A, Darnell R. Comparing sequencing assays and 2019 (suppl; abstr e13023). 03. Beck J, Vinegra M, Dankwa-Mullan I, Torres Parker J, Hayward M, Eberhard D, Salazar A, human-machine analyses in actionable genomics 17. George J, Tkacz JP, Roebuck C, Reyes F, Arriaga Y, A, Simmons C, Holtzen H, Urman A, Roper McNeillie P, Xu J, Huettner C, Koyama T, Utro for glioblastoma [published online July 11, 2017]. Jackson GP, Dankwa-Mullan I. Real world evidence N, Norden A, Rammage M, Hancock S, Lim F, Rhrissorrakrai K, Norel R, Bilal E, Royyuru Neurol Genet.2017;3(4):e164. doi: 10.1212/ study of factors associated with breast conserving K, Rao P, Coverdill S, Roberts L, Williamson A, Parida L, Earp H, Grilley-Olson J, Hayes NXG.0000000000000164 surgery for females diagnosed with early stage P, Howell M, Chau Q, Culver K, Sweetman R. D, Harvey S, Sharpless N, Kim W. Enhancing 12. Somashekhar SP, Sepúlveda MJ, Shortliffe EH, Kumar cancer. JNCCN. 2020 March 20; 18(3.5):HSR20- Cognitive technology addressing optimal cancer next-generation sequencing-guided cancer RC, Rauthan A, Patil P, Yethadka R. A prospective 085. doi: doi.org/10.6004/jnccn.2019.7466 clinical trial matching and protocol feasibility care through cognitive computing [published blinded study of 1000 cases analyzing the role of 18. Arriaga YE, Rosario B, Scheufele L, Rajmane A, South in a community cancer practice. J Clin Oncol. online November 20, 2017]. Oncologist. artificial intelligence: Watson for oncology and change B, Kefayati S, George J, Bullock T, Jackson GP, Rhee 2017;35 (suppl; abstr 6501). doi: 10.1200/ 2018;23(2):179-185. doi: 10.1634/ in decision making of a Multidisciplinary Tumor Board K. Complete human papillomavirus vaccination JCO.2017.35.15_suppl.6501 theoncologist.2017-0170 (MDT) from a tertiary care cancer center. J Clin Oncol coverage over a 13 year period in a large population 07. Kim M, Snowdon J, Weeararatne SD, Felix W, 37, 2019 (suppl; abstr 6533). of privately insured US patients. ASCO20 Virtual Lim L, Dankwa-Mullan I, Lee YK, Lee E, Jeon Km 13. Jiang Z, Xu F, Sepúlveda MJ, Li J, Wang H, Liu Z, Yin Scientific Program. May 29 – 31, 2020. Lee JS, Zang DY, Kim HJ, Kim HY, Han B. Clinical Y, Yan M, Song Y, Guo J, Roebuck M, Geng C, Tang, 19. Dankwa-Mullan I, Roebuck C, Tkacz J, Ren Y, insights for hematological malignancies from an J. Concordance, decision impact and guidelines Fayanju OM, Ren Y, Jackson G, Ariaga Y. Disparities artificial intelligence decision-support tool. J Clin adherence using artificial intelligence in high-risk in receipt of and time to adjuvant therapy after Oncol 37, 2019 (suppl; abstr e13023). breast cancer. J Clin Oncol. 2018;36 (suppl; abstr lumpectomy. ASCO20 Virtual Scientific Program. 08. Rocha HAL, Dankwa-Mullan I, Juacaba SF, e18566). May 29 – 31, 2020. Foreword Preininger A, Felix W, Thompson JV, Bright T, 14. Haddad T, Helgeson J, Pomerleau K, Makey M, 20. Wang S, Huang H, Arriaga Y, Tkacz J, Preininger Jackson GP, Meneleu P. An evaluation of artificial Lombardo , Coverdill S, Urman A, Rammage M, Goetz AM, Solomon M, Jackson GP, Dankwa-Mullan Key studies: intelligence-based clinical decision supports M, LaRusso N. Impact of a cognitive computing clinical I. Biomarker testing patterns and trends among Clinical decision support use in Brazil. J Clin Oncol 37, 2019 (suppl; abstr trial matching system in an ambulatory oncology patients with metastatic lung cancer. ASCO20 Clinical trials e18081). practice. J Clin Oncol. 2018;36 (suppl; abstr 6550). Virtual Scientific Program. May 29 – 31, 2020. Genomics Real world data

Bibliography 5 Clinical decision support

Foreword

Key studies: Clinical decision support Clinical trials Genomics Real world data

Bibliography 6 A prospective blinded study The“ study suggest[s] that MDT evaluated 1,000 breast, lung, and colorectal cancer cases of 1000 cases analyzing the cognitive computing decision role of artificial intelligence: support system[s] holds MDT was presented with Watson Watson for Oncology in substantial promise for Oncology’s treatment options change of decision making to reduce cognitive burden of a multidisciplinary tumor on oncologist[s] by providing MDT reviewed and finalized their decision board (MDT) from a tertiary expert, updated, recent care cancer centre* evidence-based [evidence- The MDT changed their decision

Somashekhar SP et al. J Clin Oncol 37, 2019 informed] insights for in 13.6% of the cases. (suppl; abstr 6533). treatment-related decisions *no contributing IBM author making. Link to study → Excerpt from abstract

Reason for decision change:

Evidence for newer 55% 30% treatments(s)

More personalized treatment alternatives

Foreword New genotypic, phenotypic and clinical insights Key studies: 15% Clinical decision support Clinical trials Genomics Real world data

Bibliography 7 Concordance, decision When“ treatment decisions 1,997 breast cancer cases from impact and guidelines were altered, the newly CSCO database adherence using artificial selected therapies showed intelligence in high-risk greater adherence to breast cancer* professional treatment

Jiang Z et al. J Clin Oncol. 2018;36 guidelines. Disclosure of Watson for 106 cases(5%) (suppl; abstr e18566). Oncology options resulted in *no contributing IBM author Excerpt from abstract prescriber treatment changes in 106 or 5% of cases Link to study →

The guideline adherence rate 97% improved in the 106 cases where decision changes were made from 89 to 97%

Foreword

Key studies: Clinical decision support Clinical trials Genomics Real world data

Bibliography 8 The establishment of a new The“ new model combined Doctor and patient survey results indicated: Standardization and personalization of treatment recommendations medical model for tumor with human brain, artificial Greater patient engagement in treatment combined with intelligence (AI) and decision making Watson for Oncology, MDT cancer patients enriches and patient involvement* the traditional MDT

Fang J et al. J Clin Oncol. 2018;36(suppl; abstr [multidisciplinary team] e18504). model. It is a new kind of *no contributing IBM author medical model which is Link to study → more effective.

Excerpt from abstract Multidisciplinary team (MDT)

Watson Patient for Oncology

Foreword

Key studies: Clinical decision support Clinical trials Genomics Real world data

Bibliography 9 Artificial intelligence-based […]“ patients build stronger Enhanced patient knowledge around The new 7-step model assisted by disease and treatment options can Watson for Oncology was compared clinical decision-support confidence with their health increase confidence in achieving positive to non-CDS system method (n = 70; outcomes. A new model of cancer care new = 50; traditional = 20) system improves cancer care team and are willing to consultation assisted by Watson for treatment and patient believe they will benefit from Oncology was evaluated. satisfaction the treatment plans.

Wang Z et al. J Clin Oncol. 2019; 37(suppl): abstract Excerpt from abstract e18303. The 7-step model: Patients in 7-step process Link to study → Introduce WfO to patients indicated higher satisfaction in treatment options, confidence

Patients express desires in health care workers, and willingness to follow

Oncologist presents medical condition treatment regimen.

Discussion with team

Input patients info WfO and review options

Foreword Discuss and finalize options with patients Key studies: Clinical decision support Clinical trials Genomics Patient feedback Real world data

Bibliography 10 Use of machine learning The“ use of machine A model was trained, using abstracts and NCCN titles from PubMed, to identify relevant to identify relevant learning to identify relevant clinical papers based on articles cited by 3 NCI-PDQ research publications publications may reduce expert oncology sources: Hemonc.org in clinical oncology the time clinicians spend

Suarez Saiz F et al. J Clin Oncol 37, 2019 finding pertinent evidence (suppl; abstr 6558). for a patient.

Link to study → Excerpt from abstract Balanced training data: 988 papers were classified with: On-topic set: cited in at least two expert sources 0.93 accuracy Off-topic set: published in (95% CI, 0.9–0.96; p < 0.0001) lower-ranked journals 0.95 sensitivity 0.91 specificity

Foreword

Key studies: Clinical decision support Clinical trials Genomics Real world data

Bibliography 11 A systematic review of Overall concordance between The artificial intelligence-based clinical decision-support system for Watson for studies of concordance WfO therapeutic options Oncology has been deployed in several institutions across the world. This study is a with expert opinion for and treatment decisions of systematic review and meta-analysis of global a globally implemented MTBs and ICs was high, but studies evaluating concordance between WfO therapeutic options and treatment decisions oncology clinical decision- significantly higher with MTBs. of multidisciplinary tumor boards (MTBs) or individual clinicians (ICs). support system Concordance varied for cancer

Arriaga Y et al. Proceedings of the 2020 AMIA types and countries indicating Informatics Summit. 2020. 7 7.5 % Reviewed 27 unique study publications a need for localization for of 9,302 patients from 6 countries: China, Link to study → regional difference. India, South Korea, Brazil, Thailand and the . WfO and MTBs N = 4,020 patients 15 studies Mean concordance for China, India and South Korea WfO and MTBs significantly Mean concordance was 77.5% (SD 17.2%) higher than WfO and ICs (P < 0.0001)

6 7.4 %

Foreword WfO and ICs N = 5,282 patients 12 studies Key studies: Clinical decision support Brazil, China, South Korea, Thailand, US Clinical trials Genomics Mean concordance was 67.4% Real world data (SD 13.7%)

Bibliography 12 Initial experience with Feedback on the utility of IBM The IBM Cancer Guidelines Navigator (CGN) was implemented hospital-wide at the Cancer Guidelines Cancer Guidlines Navigator Ocean Road Cancer Institute (ORCI) to help clinicians reduce treatment variability by Navigator, a tool to for easy and efficient use increasing adherence to standard evidence- standardize and improve of this digital reference based care. the quality of cancer care system designed to support CGN presents corresponding treatment options in the National Comprehensive in Sub-Saharan Africa, easy and efficient access Cancer Network (NCCN) Harmonized at Ocean Road Cancer to regionalized cancer- Guidelines™ for Sub-Saharan Africa after clinicians enter a cancer patient case. Institute in Tanzania treatment guidelines is

Coma LY et al. J Clin Onc. 2020;38 promising for future Tanzania (suppl; abstr e14106). expansion plans. 31 ORCI clinical and IT staff underwent CGN training; 12 answered a survey about their Link to study → experiences and reported:

Benefits of the tool: Areas for improvement:

75% 42% Quick access to guidelines and Expanding cancer coverage evidence 25% 58% Better integration into the workflow Ease of use Foreword 25% Key studies: Offline access Clinical decision support Clinical trials Genomics Real world data

Bibliography 13 Clinical trials

Foreword

Key studies: Clinical decision support Clinical trials Genomics Real world data

Bibliography 14 A pilot study to implement Implementation“ of Watson Clinical trials are critical to expanding understanding of disease treatment; how- an artificial intelligence (AI) for CTM system with a CRC ever, screening for clinical trial enrollment is complex and time-consuming, leading to system for gastrointestinal team may enable high volume low rates of enrollment for newly diagnosed cancer Clinical Trial Matching* patient screening for a large cancer patients.

Jin Z et al. Ann Oncol. 2019;30 (suppl 5) v582. number of clinical trials in an 35 patients with newly diagnosed efficient manner and promote 35 Link to study → patients gastrointestinal cancer screened for awareness of clinical trial 50 clinical trials by clinical research 50 coordinators with Watson for Clinical Trial opportunities within the GI clinical trials Matching (CTM) and manual methods oncology practice.

Excerpt from manuscript

Average Time to Screen Average trials found (minutes per patient) (per patient)

p<0.0001 p<0.0001

40 10 30.5 30 8 6 7.66 20 4 10.1 10 2 0 0 1.97 Foreword

Key studies: CTM Manual CTM Manual Clinical decision support Clinical trials *Mayo Clinic has a business collaboration with IBM Genomics Watson Health. This activity is not undertaken to allow Real world data IBM to indicate Mayo Clinic endorsement of any IBM Bibliography product or service. 15 Impact of a cognitive Cognitive“ technology In July 2016, Mayo Clinic* implemented IBM Watson for Clinical Trial Matching computing clinical supports increased with a team of screening clinical research coordinators in its ambulatory practice trial matching system enrollment in clinical for patients with breast cancer at the in an ambulatory trials for breast cancer. Rochester campus. oncology practice In the 18 months after implementation, 84% there was on average an 84 percent increase Haddad T et al. J Clin Oncol. 2018;36 in enrollment to Mayo’s systemic therapy (suppl; abstr 6550). clinical trials for breast cancer. The time to screen an individual patient for clinical Link to study → trial matches also fell when compared with traditional manual methods.

Average monthly patient enrollment Ambulatory breast cancer practice

10

5 6.4 3.5 0

Pre CTM With CTM This was further increased to 8.5 patients/month when 84% increase including accruals to breast cancer cohorts of multi- in monthly disease, phase I trials within the experimental cancer Foreword enrollment therapeutics program.

Key studies: Clinical decision support Clinical trials *Mayo Clinic has a business collaboration with IBM Genomics Watson Health. This activity is not undertaken to allow Real world data IBM to indicate Mayo Clinic endorsement of any IBM Bibliography product or service. 16 Artificial intelligence tool This AI based clinical trial 997 Less than 5% of cancer WCTM system processed data for 997 unique patients enroll in clinical for optimizing eligibility matching system reliably patients across a set of 4 clinical trials. screening for clinical trials identified eligible patients trials. Community cancer in a large community for most trials in less time centers play a vital role in Percentage of agreement between IBM clinical trial recruitment. cancer center. compared to manual WCTM and manual review for 239 randomly selected cases: review. This indicates This study evaluated the Beck JT et al. JCO Clin Cancer Inform. potential for decreasing 64.3-94.0% ® 2020;4:50-59. practitioner workload leading Attribute extraction performance of IBM Watson™ for Clinical Trial to increased efficiency of trial 81-96% Link to study → enrollment in busy practices. Trial eligibility determination Matching (WCTM) as compared to manual review for a large number of patients Eligibility screening time for the same 90 patients for 3 trials: with breast cancer at Highlands Oncology Group. 120 90 110 60 30 24 0 Foreword Manual Watson for Key studies: clinical trial Clinical Trial Clinical decision support screening Matching Clinical trials Genomics Real world data

Bibliography 17 Genomics

Foreword

Key studies: Clinical decision support Clinical trials Genomics Real world data

Bibliography 18 Clinical insights for WfG“ variant interpretation 54 10 South Korean patient cases with cases were randomly selected for hematological malignancies correlated well with manually hematological malignancies were analyzed manual interpretation analysis from an artificial intelligence curated expert opinion and by Watson for Genomics (WfG) decision-support tool identified clinically actionable

Kim M et al. J Clin Oncol. 2019;37 insights missed by manual (suppl; abstr e13023). interpretation… WfG has

Link to study → obviated the need for 71% 90% of cases had at least one clinically of the manually interpreted cases labor-intensive manual actionable therapeutic alteration were concordant with WfG analysis curation of clinical trials 33% WfG identified 9 more and therapy, enabling our of cases had genes that were targeted (33%) clinically actionable center to exponentially scale by a US FDA approved therapy variants not found in our NGS operations. 20% manual assessment. Excerpt from abstract of cases without therapeutic alterations, WfG identified additional diagnostic or prognostic insights

Foreword

Key studies: Clinical decision support Clinical trials Genomics Real world data

Bibliography 19 Enhancing NGS-guided Molecular“ tumor 1,018 Watson for Genomics analyzed 1,018 patient cancer center care through boards empowered by cases previously sequenced and analyzed cognitive computing cognitive computing can

Patel N et al. The Oncologist. 2018;23(2):179-185 significantly improve patient care by providing Link to study → a fast, cost-effective, and Providing current, accurate information on comprehensive approach for newly approved therapeutic options and open clinical trials requires considerable data analysis in the delivery manual curation performed mainly by members of molecular tumor boards (MTBs). of precision medicine.

Excerpt from abstract Watson for Genomics’ automated analysis of genomic data took under 3 minutes per under 3 min patient case.

In 99% of cases, Watson for Genomics 99% identified variants previously defined as actionable by the human-only molecular tumor board.

Foreword In 32% of the patient cases, Watson for 32% Key studies: Genomics found additional potentially Clinical decision support clinically actionable variants that a Clinical trials molecular tumor board had not identified. Genomics Real world data

Bibliography 20 Genomic analysis of Insights gained from Next 31 Insights gained from NGS 31 South Korean patients with MPN Myeloproliferative Generation Sequencing (NGS) underwent NGS. Results underwent can inform cancer care Neoplasm (MPN) Patients and variant annotation are variant interpretation and annotation by for hematologic , IBM Watson for Genomics. Results were from a single institution in useful for risk stratification compared to a cohort of 151 MPN patients especially BCR-ABL previously published in the New England South Korea reveal novel and understanding of disease Journal of Medicine. negative neoplasms (MPN). pathogenic mutations and development of MPN. In Mutational variants of perturbed pathways this study researchers, Two novel pathogenic mutations in Korean patients with MPN CALR were identified: c. 1162delG and Weeraratne D et al. J Clin Onc. 2020;38 identified a different genetic c.1100_1145del were studied to identify (suppl; abstr 19533). variant profile for Korean mutational profile variations

Link to study → patients with MPN than the NOTCH1 pathogenic mutations were specific to demographics. exclusive comparison cohort.

TP53 mutations were significantly enriched

MPL mutations were not detected

Foreword

Key studies: Clinical decision support Clinical trials Genomics Real world data

Bibliography 21 Comprehensive analysis Thyroid and cutaneous For patients with advanced stage or Five categories of biomarker evidence in refractory cancers, next generation 366 genes were validated against 2847 of advanced stage solid melanoma cancers have the sequencing with variant categorization TCGA samples of advanced tumors. and annotation may provide insights to Watson™ for Genomics was used for variant tumors from TCGA reveal most level 1 (FDA approved physicians into potential precision targets. categorization and annotation. widespread variation of drugs). Colorectal cancers This study examines the strength of clinical genomics evidence levels have the most R1 (resistant) evidence of various advanced stage tumor samples from The Cancer Genome Atlas across cancer types variants. Kidney and prostate (TCGA).

Weeraratne D et al. J Clin Onc. 2020;38 cancers had no Level 1 (suppl; abstr 13547). evidence and the greatest

Link to study → proportion of unactionable tumors. Level 3 and 4 Strength of biomarker/drug response associations was used for annotation with level 1/R1 strongest and level 4 weakest variants are promising as from clinical literature and FDA drug guidelines potential treatment targets. Cancer type N Level 1 % Level R1 % Level 2A % Level 2B % Level 3A % Level 3B % Level 4% Unaction-able Breast 514 1.2 0 0 0 35.2 23.5 2.7 37.4

Esophageal 495 3.0 0 0 0 72.1 6.7 5.7 12.5

Kidney 476 0 0 0 0.9 21.8 9.9 14.9 52.5

Colorectal 346 2.9 56.1 0 0 4.6 23.7 1.7 11.0

Gastric 241 6.2 0 0 0 23.2 46.5 5.8 18.3

Cutaneous 236 43.2 0 3.4 0 36.9 2.1 6.8 7.6 melanoma Foreword Thyroid 208 76.4 0 0 0 5.8 1.4 4.3 12.0 Key studies: Clinical decision support Lung 176 13.1 0.6 1.7 0 27.8 44.9 2.3 9.7 Clinical trials Prostate 155 0 0 0 1.9 7.7 15.5 2.6 72.3 Genomics Real world data

Bibliography 22 Association of mutational Tumor mutational profile Methodology profile and human findings complimented Immunochemical staining of p16 to papillomavirus status in previous studies and provide detect presence HNSCC tumor of HPV Evaluation of patients with head and neck potential therapeutic target samples from 128 tumor mutational squamous carcinoma areas for future research. patients in the profile and Veterans Health association with Next Generation (HNSCC) Administration tumor HPV status Sequencing through the Veteran Affair’s Doerstling S et al. AMP 2019. National Precision Oncology Program Link to study →

Findings The increasing incidence of

Mutations within TP53 and the p16/CDK/ head and neck squamous cell Rb pathway were more common in p16- carcinoma (HNSCC) is thought negative tumors (non-HPV). to be associated with increased RAS pathway mutations occurred exclusively rates of human papillomavirus in p16-positive HPV tumors. (HPV) infection. In addition, FBXW7 mutations were observed only in individuals with HPV+ tumors p16-positive HPV tumors, with borderline statistical significance. have different recovery trajectories. Study sought to Foreword further explore these associations Key studies: Clinical decision support by examining differences in Clinical trials mutational profiles between Genomics Real world data HPV+ and HPV- tumors. Bibliography 23 Real world data

Foreword

Key studies: Clinical decision support Clinical trials Genomics Real world data

Bibliography 24 Disparities in receipt of and Disparities for receipt of Patients from communities with a high proportion of these races significantly less time to adjuvant therapy adjuvant therapy following 36,270 likely to receive combination of ART and AET (P < 0.001) after lumpectomy breast conserving therapy Analysis used IBM® MarketScan® claims data for 36,270 privately insured patients Black Asian Hispanic with breast cancer who had not received were observed across (RRR 0.14) (RRR 0.13) (RRR 0.45) neoadjuvant therapy. Dankwa-Mullan I et al. J Clin Onc. 2020;38 multiple demographic (suppl; abstr 534). variables presenting Associations with longer median time to Associations with longer median time to Link to study → opportunities for treatment for ART treatment for ACT improvements in HIV/AIDS Cerebrovascular disease timely care. + 11 days + 6.0 days (P = 0.01) (P < 0.001)

HIV/AIDS and residing in a highly Moderate to severe liver disease concentrated Black community + 8.5 days + 8.5 days (P < 0.001) (P = 0.01) High-density Asian community HIV/AIDS and residing in a high-density + 18.0 days Asian community (P < 0.001) + 12.2 days (P = 0.04)

In women with breast cancer, initiation of adjuvant treatment following Foreword breast conserving therapy (BCS) lacks consistency. After controlling for Key studies: sociodemographic covariates, factors associated with time to treatment Clinical decision support Clinical trials (TTT) and relative risk ratio (RR) for post-BSC adjuvant therapy were Genomics evaluated for adjuvant radiation therapy (ART), adjuvant cytotoxic Real world data chemotherapy (ACT), adjuvant endocrine therapy (AET). Bibliography 25 Complete human Although it increased over 21,384,851 Despite the Healthy People Retrospectively reviewed IBM MarketScan papillomavirus vaccination time, complete vaccination claims for 21,384,851 commercially 2020 goal of 80% vaccination coverage over a 13 year coverage for human insured patients coverage for HPV, vaccination period in a large population papillomavirus (HPV) did rates are low in the US, of privately insured not meet the Healthy People Vaccination Coverage by Age and Year (Female) especially in adolescents. Variability in defining measures US patients 2020 goal in the population 50% studied. This analysis 40% of vaccination in published 30% Arriaga Y et al. J Clin Onc. 2020;38 identified gaps in vaccination studies may contribute. (suppl; abstr 1511). coverage that varied by 20% 10% Link to study → health plan type and region, 0% This analysis aimed to examine which may have policy and 11-12Y 13-15Y 16-17Y 18-26Y complete vaccination in a practice implications. privately insured US population Vaccination Coverage by Age and Year (Male) over a 13 year period. 50% 40% 30% 20% 10% 0% Foreword 11-12Y 13-15Y 16-17Y 18-26Y Key studies: Clinical decision support Clinical trials 2010 2014 2018 Genomics * Note zero vaccination coverage for females in 2006 and Real world data for males in 2006 and 2010; aligns with FDA approval Bibliography 26 Biomarker testing patterns The likelihood of biomarker 12% of the 8,977 patients with metastatic 12% lung cancer had claims for biomarker testing and trends among patients metastatic lung testing between 1/1/2013 – 12/31/2018 with metastatic lung cancer varied with several factors such as age, insurance plan 20.6% of patients were tested in 2018, 20.6% a significant increase from 8.4% in 2013 (P < 0.0001) Wang S et al. J Clin Onc. 2020;38 type, sex and comorbidities (suppl; abstr 13667). highlighting opportunities to inform outreach policy for Link to study → underserved populations. Factors associated with a lower likelihood Biomarker testing in patients of testing included: with metastatic lung cancer → Increasing age can aid oncologists in → Enrollment in a preferred provider making targeted treatment health plan decisions. This analysis → Pre-existing diabetes or congestive heart failure assessed sociodemographic factors related to testing Factors associated with a higher likelihood in a commercially insured of testing included: population in the IBM® → Age under 55 MarketScan® database. Foreword → Females Key studies: → Residence in Northeastern US Clinical decision support Clinical trials Genomics Real world data

Bibliography 27 04. Arriaga Y, Hekmat R, Draulis K, et al. A 09. Beck JT, Rammage M, Jackson GP, et al. Bibliography systematic review of studies of concordance Artificial intelligence tool for optimizing eligibility with expert opinion for a globally implemented screening for clinical trials in a large community oncology clinical decision system. Proceedings cancer center. JCO Clin Cancer Inform. 2020 For insight into the scientific evidence of of the 2020 AMIA Informatics Summit; 2020 Jan; 4:50-59. doi: 10.1200/CCI.19.00079 Watson Health Oncology, please reference March. 10. Chen CY, Hung HC, Chiu HY, et al. Enhancing these additional publications. 05. Arriaga Y, Hekmat R, Darulis K, et al. A evidence-based medicine skills in oncology systematic review of concordance studies using training with cognitive technology. J Clin Oncol. Watson for Oncology (WfO) to support breast 2019;37 (suppl): abstract 10532. cancer treatment decisions: a four-year global 11. Chen P, Sun T, Li T, et al. Can AI technology experience. In: Proceedings of the 2019 San augment tumor board treatment decisions for Antonio Breast Cancer Symposium; 2019 Dec high risk stage II colon cancer care? J Clin Oncol. 10-14; San Antonio, TX. Philadelphia (PA): AACR; 2018;36 (suppl; abstr e18582). 01. Alexander M, Solomon B, Ball DL, et al. An Cancer Res 2020;80(4 Suppl):Abstract nr P4- 12. Chen S, Chen Z, Xiao H, Peng Z, Peng S, Kuang evaluation of an artificial intelligence clinical 14-05. M. Can artificial intelligence support the clinical trial matching system in Australian lung cancer 06. Bach P, Zauderer M, Gucalp A, et al. Beyond decision making for hepatocellular carcinoma? patients. JAMIA Open, ooaa002, doi:10.1093/ Jeopardy!: harnessing IBM’s Watson to improve Gut. 2019; 68:A134. doi: dx.doi.org/10.1136/ jamiaopen/ooaa002 oncology decision making. J Clin Oncol. gutjnl-2019-IDDFAbstracts.263 02. Armstrong J, Hintze B, Vashistha V, et al. 2013;31:6508. DOI: 10.1200/jco.2013.31.15_ 13. Chen ZB, Chen SL, Liang RM, et al. Can artificial Prescribing practices based on recommendations suppl.6508 intelligence support the clinical decision making of the Veterans Health Administration’s precision 07. Baek J, Ahn S, Urman A, et al. Use of a cognitive for Barcelona clinic liver cancer stage 0/a oncology program. Presented at the Association computing system for treatment of colon and hepatocellular carcinoma in China? J Clin Oncol. of VA Hematology/Oncology 2019 Annual gastric cancer in South Korea. J Clin Oncol. 2019;37 (suppl): abstr e15364. Meeting. September 20-22, 2019; Minneapolis, 2017;35 (suppl; abstr e18204). doi: 10.1200/ 14. Choi YI, Chung JW, Kim KO, et al. Concordance Foreword MN. JCO.2017.35.15_suppl.e18204 rate between clinicians and Watson for Oncology 03. Arriaga Y, Hekmat R, Draulis K, et al. A review of 08. Beck J, Vinegra M, Dankwa-Mullan I, et al. among patients with advanced gastric cancer: Key studies: gynecological cancers studies of concordance Cognitive technology addressing optimal cancer early, real-world experience in Korea. Can J Clinical decision support with individual clinicians or multidisciplinary clinical trial matching and protocol feasibility Gastroenterol Hepatol. 2019; 2019: 8072928. Clinical trials tumor boards for an artificial intelligence-based in a community cancer practice. J Clin Oncol. 10.1155/2019/8072928. Genomics clinical decision-support system. J Clin Oncol. 2017;35 (suppl; abstr 6501). doi: 10.1200/ 15. Coma LY, Ndumbalo J, Mwaiselage J, et al Initial Real world data 2020;38 (suppl; abstr e14070). JCO.2017.35.15_suppl.6501 experience with Cancer Guidelines Navigator,

Bibliography 28 a tool to standardize and improve the quality 21. Fang J, Guo X, Zhu Z, et al. Watson for Oncology https://doi.org/10.1161/str.50.suppl_1.WP400 38. Kelley MJ, Duffy J, Hintze BJ, Williams CD, of cancer care in Sub-Saharan Africa, at Ocean applied to teaching and remote consulting model. 29. Helgeson J, Rammage M, Urman A, et al. Clinical Spector NL. Implementation of precision Road Cancer Institute in Tanzania. J Clin Oncol. J Clin Oncol. 2019; 37 (suppl): abstract 6545. performance pilot using cognitive computing for oncology in the Veterans Health Administration 2020;38 (suppl; abstr e14070). 22. Fang J, Zhu Z, Wang H, et al. The establishment clinical trial matching at Mayo Clinic. J Clin Oncol. (VHA). J Clin Oncol. 2017;35 (suppl; abstr 6507). 16. Curioni-Fontecedro A. A new era of oncology of a new medical model for tumor treatment 2018;36 (suppl; abstr e18588). 39. Kim D, Kim YY, Lee JH, et al. A comparative study through artificial intelligence. ESMO Open. combined with Watson for Oncology, MDT and 30. Herath D, Wilson-Ing D, Ramos E, Morstyn G. of Watson for Oncology and tumor boards in 2017 May 15;2(2):e000198. doi: 10.1136/ patient involvement. J Clin Oncol. 2018;36 (suppl; Assessing the natural language processing breast cancer treatment. KJCO. 2019; 15:3-6. esmoopen-2017-000198. eCollection 2017. abstr e18504). capabilities of IBM Watson for Oncology using 40. Kim EJ, Woo HS, Cho JH, et al. Early experience 17. Dankwa-Mullan I, Arriaga Y, Mayor C, et al. 23. Frank MO, Koyama T, Rhrissorrakrai K, et al. real Australian lung cancer cases. J Clin Oncol. with Watson for oncology in Korean patients Experiences and lessons learned from global Sequencing and curation strategies for identifying 2016;34, abstr e18229. with colorectal cancer. PLoS One. 2019;14(3): implementation of clinical decision support candidate glioblastoma treatments. BMC Medical 31. Itahashi K, Kondo S, Kubo T, et al. Evaluating e0213640. doi:10.1371/journal.pone.0213640 for oncology. Podium presentation: At the Genomics. 2019; 12(1): 56. doi: 10.1186/ clinical genome sequence analysis by Watson for 41. Kim M, Kim BH, Kim JM, et al. Concordance International Congress on Cancer and Clinical s12920-019-0500-0. Genomics. Front Med. 2018;5:305. doi: 10.3389/ in postsurgical radioactive iodine therapy Oncology; December 5, 2019; Singapore. 24. Fu J, Gucalp A, Zauderer M, et al. Steps in fmed.2018.00305 recommendations between Watson for 18. Doerstling S, Winski D, Hintze B, et al. developing Watson for Oncology, a decision 32. Jiang Z, Xu F, Sepúlveda MJ, et al. Concordance, Oncology and clinical practice in patients with Association of mutational profile and human support system to assist physicians choosing decision impact and guidelines adherence using differentiated thyroid carcinoma. Cancer 2019. papillomavirus status in patients with head first-line metastatic breast cancer (MBC) artificial intelligence in high-risk breast cancer. J doi: doi.org/10.1002/cncr.32166 and neck squamous cell carcinoma. Poster therapies: Improved performance with machine Clin Oncol. 2018;36 (suppl; abstr e18566). 42. Kim M, Park HY, Kho BG, Park CK, Oh IJ, Kim YC. presented at: AMP; November 7-9, 2019; learning. J Clin Oncol. 2015:566. DOI:10.1200/ 33. Jin Z, Haddad T, Hubbard J, et al. A pilot study to Concordance between artificial intelligence and Baltimore, MD. jco.2015.33.15_suppl.566 implement an artificial intelligence (AI) system for multidisciplinary tumor board for lung cancer. 19. Napolitano e Ferreira EN, Neto MM, Huang H, et 25. Garcia C, Uzbelger G. Artificial intelligence to gastrointestinal cancer clinical trial matching. Ann Am J Respir Crit Care Med. 2019; 199(A2610). al. Comprehensive analysis of advanced stage help the practitioner chose the right treatment: Oncol. 2019;30(Supplement 5):1434P. 43. Kim M, Snowdon J, Weeraratne SD, et al. Clinical solid tumors from TCGA reveal widespread Watson for Oncology. In: Nordlinger B, Villani C, 34. Katsoulakis E, Duffy J, Hintze B, Spector N, Kelley insights for hematological malignancies from an variation of genomics evidence levels across Rus D (eds) Healthcare and Artificial Intelligence. M. Comparison of annotation services for the artificial intelligence decision-support tool. J Clin cancer types. J Clin Oncol. 2020;38 (suppl; abstr Springer, Cham. doi: doi.org/10.1007/978-3- VA Precision Oncology Program. J Clin Oncol. Oncol. 2019;37 (suppl; abstract e13023). e14070. 030-32161-1_11 2018;36 (suppl; abstr 12109). 44. Kim M, Weeararatne SD, Snowdon J, et al. 20. Epstein A, Zauderer M, Gucalp A, et al. Next 26. Guo X, Gao S, Yang L, et al. Analysis of Chinese 35. Katsoulakis E, Duffy JE, Hintze B, Spector NL, Comparative analysis of prognostic molecular steps for IBM Watson Oncology: scalability acral and mucosal melanoma patient genomic Kelley MJ. Comparison of annotation services signatures in Asian and Caucasian AML to additional malignancies. J Clin Oncol. and neoantigen profiles in cancer vaccine for next-generation sequencing in a large-scale populations. J Clin Oncol. 2019;37 (suppl; 2014;32:6618. DOI: 10.1200/jco.2014.32.15_ development: a pilot study. J Clin Oncol. 2019;37 precision oncology program. JCO Precision abstract e13025.) suppl.6618 (suppl): abstract e14300. Oncology. 2020 Mar; no. 4; 212-221. DOI: 45. Kim YY, Oh SJ, Chun YS, Lee WK, Park HK. Gene 27. Haddad T, Helgeson J, Pomerleau K, et al. Impact 10.1200/PO.19.00118 expression assay and Watson for Oncology of a cognitive computing clinical trial matching 36. Kaufman J, Duffy J, Xu J, et al. Association of for optimization of treatment in ER-positive, system in an ambulatory oncology practice. J Clin NGS mutational pattern with immune checkpoint HER2-negative breast cancer. PLoS One. Foreword Oncol. 2018;36 (suppl; abstr 6550). inhibitor clinical benefit in solid tumors. J Clin 2018;13(7):e0200100. doi: 10.1371/journal. Key studies: 28. Hamilton JG, Garzon MG, Westerman JS, et al. “A Oncol. 2018;36 (suppl; abstr e24274). pone.0200100 Tool, Not a Crutch”: Patient Perspectives About 37. Keikes L, Medlock S, van de Berg D, et al. Content 46. Kris M, Gucalp A, Epstein A, et al. Assessing the Clinical decision support IBM Watson for Oncology Trained by Memorial evaluation of Watson for Oncology: a feasibility performance of Watson for Oncology, a decision Clinical trials Sloan Kettering [published online January 29, study. J Clin Oncol. 2018;36 (suppl 30; abstr support system, using actual contemporary Genomics 2019]. J Oncol Pract. 2019;Jop1800417. doi: 235). clinical cases. J Clin Oncol. 2015;33:8023. Real world data

Bibliography 29 47. Lee KA, Felix W, Jackson G. Concordance, 53. Liang J, Li T, Zhang SS, et al. Reasons for Translational Cancer Research. 2019;8(2). 69. Sahajpal N, Mondal A, Ahluwalia M, et al. Clinical decision impact, and satisfaction for a discordance in treatment approaches between 61. Patel N, Michelini V, Snell J , et al. Enhancing utility of comprehensive genomic pathway and computerized clinical decision support system oncology practice and clinical decision support in next-generation sequencing-guided cancer integrated network analyses in personalized in treatment of lung cancer patients. European China. J Clin Oncol. 2019; 37(suppl): abstract care through cognitive computing [published oncology. ASCO20 Virtual Scientific Program; Lung Cancer Congress. April 11, 2019; Geneva, 6555. online November 20, 2017]. Oncologist. May 29 – 31, 2020. J Clin Oncol. 2020;38 (suppl; Switzerland. 54. Liang J, Tang C, Wang X, et al. Impact of 2018;23(2):179-185. doi: 10.1634/ abstr ). 48. Lee K, Lee SH, Preininger A, Shim J, Jackson decision-support system and guideline treatment theoncologist.2017-0170 70. Saiz F, Urman A, Sanders C, Britt M, Nielsen R, GP. Patient satisfaction with oncology clinical concordance on response rate in advanced lung 62. Pérez L, Cantalapiedra D, Valero-Hervas D, et al. Stevens R. First experiences with an AI-assisted decision support in South Korea. J Clin Oncol. cancer. J Clin Oncol. 2019;37 (suppl; abstract The application of cognitive computing technology clinical evidence system to evaluate clinical 37(suppl): abstract e18329. e20006). in genomics in precision oncological medicine: consensus among clinical trial publications. J 49. Lee KW, Ahn SM, Chung JW, et al. Assessing 55. McDougall RJ. Computer knows best? The need the Sistemas Genomicos experience. J Clin Oncol. Clin Oncol. 2018;36 (suppl; abstr e18583). concordance with Watson for Oncology, a for value-flexibility in medical AI [published 2018;36 (suppl; abstr e18544). 71. Saiz F, Urman A, Sanders C, Britt M, Nielsen R, cognitive computing decision support system online November 22, 2018]. J Med Ethics. 2018; 63. Poonen P, Duffy J, Hintze BJ, et al. Genomic Stevens R. IBM Watson Evidence Service (WES): for colon cancer treatment in Korea. JCO pii: medethics-2018-105118. doi: 10.1136/ analysis of metastatic solid tumors in veterans: a system for retrieval, summation and insight Clin Cancer Inform. 2018 Dec;2(2):1-8. doi: medethics-2018-105118 findings from the VHA national Precision Oncology generation of relevant clinical evidence for 10.1200/CCI.17.00109 56. McNabb J, Wan S, Bright T, Unertl K, Weeraratne D, Program. J Clin Oncol. 2019; 37(suppl): abstract personalized oncology. J Clin Oncol. 2018;36 50. Levantakos K, Helgeson J, Mansfield AS, et al. McCoy A. Implementation of artificial intelligence- 3074. (suppl;abstr e18588). Implementation of artificial intelligence (AI) for enabled clinical decision support: a systematic 64. Prasannakum SS, Ramya Y, Rohit CK, Ashwin K, 72. Sakai KT, M. Shimizu, S. Takahama, T. et al. lung cancer clinical trial matching in a tertiary review on clinical outcomes. Proceedings of the Patil P, Rauthan A. Triple blinded prospective A comparative study of curated contents by cancer center. Ann Oncol. 2019;30(Suppl_2). 2020 AMIA Informatics Summit; 2020 March study assessing the impact of genomics & artificial knowledge-based curation system in cancer doi: 10.1093/annonc/mdz065 23-26. intelligence Watson for oncology (WFO) on clinical sequencing. Sci Rep. 2019;9(1):11340. 51. Li TL, Chen C, Zhang SS, et al. Deployment and 57. McNamara D, Goldberg SL, Latts L, et al. MDT’s decision of adjuvant systemic therapy for doi: 10.1038/s41598-019-47673-9 integration of a cognitive technology in China: Differential impact of cognitive computing hormone receptor positive early breast carcinoma. 73. Sarre-Lazcano C, Armengol Alonso A, Huitzil experiences and lessons learned. J Clin Oncol. augmented by real world evidence on novice and Ann Oncol. 2019;30(Supplement 5). Melendez F, et al. Cognitive computing in 2019;37(suppl; abstract 6538). expert oncologists [published online September 65. Rhrissorrakrai, K. Koyama T, Parida L. Watson oncology: a qualitative assessment of IBM 52. Liang J, Li TL, Dankwa-Mullan I, et al. Employing 11, 2019]. Cancer. 2019;8(15):6578-6584. doi: for Genomics: moving Watson for Oncology in Mexico. J Clin Oncol. an oncology decision-support system to 10.1002/cam4.2548 forward [published online July 21, 2016]. Trends 2017;35(suppl; abstr e18166). doi: 10.1200/ quantify treatment variation. J Clin Oncol. 2019; 58. Mitne Neto M, Fornari AR, Moreira LP, et al. in Cancer. 2016;2(8):392-395. doi: 10.1016/j. JCO.2017.35.15_suppl.e18166 37(suppl); abstract E18067. Performance and validation of a tumor mutation trecan.2016.06.008 74. Seidman A, Pilewskie M, Robson M, et al. profiling, based on artificial intelligence 66. Rocha HAL, Dankwa-Mullan I, Juacaba SF, et al. Integration of multi-modality treatment planning annotation, to assist oncology decision making. J Shared-decision making in prostate cancer with for early stage breast cancer (BC) into Watson Clin Oncol. 2019; 37(suppl): abstract e13148. clinical decision-support. J Clin Oncol. 2019;37 for Oncology, a decision support system: 59. Miyano S. Artificial intelligence for cancer genomic (suppl; abstract 16576). seeing the forest and the trees. J Clin Oncol. medicine: understanding cancer is beyond 67. Rocha HAL, Dankwa-Mullan I, Juacaba SF, et 2015;33:e12042. Foreword human ability [published online Januarly 11, al. An evaluation of artificial intelligence-based 75. Somashekhar S, Kumarc R, Rauthan A, et al. Key studies: 2019]. Brain Nerve. 2019;71(1):25-32. epub clinical decision supports use in Brazil. J Clin Double blinded validation study to assess 2019/01/11. [article in Japanese] doi: 10.11477/ Oncol. 2019;37(suppl; abstract e18081). performance of IBM artificial intelligence Clinical decision support mf.1416201213 68. Rocha HAL, Emani S, Arruda CAM, et al. Non-user platform, Watson for Oncology in comparison Clinical trials 60. Pan H, Tao J, Qian M, et al. Concordance physician perspectives about an oncology clinical with Manipal multidisciplinary tumour board – Genomics assessment of Watson for Oncology in breast decision-support system: a qualitative study. J Real world data cancer chemotherapy: first China experience. Clin Oncol. 2020;38 (suppl; abstr e14070). Bibliography 30 first study of 638 breast cancer cases [abstract]. 79. Somachekhar SP, Yethadka R, Rohit Kumar C, system and treatments selected by clinicians as a (MPN) patients from a single institution in South In: Proceedings of the 2016 San Antonio Breast Rajgopal AK, Rauthan A, Patil P. Triple blinded function of cancer type or stage. Journal of Global Korea reveal novel pathogenic mutations and Cancer Symposium; 2016 Dec 6-10; San prospective study assessing the impact of Oncology. 2019; 5(Supplement):95. perturbed pathways. J Clin Oncol. 2020;38 Antonio, TX. Philadelphia (PA): AACR; Cancer Res genomics: EndoPredict and artificial intelligence 87. Suwanvecho S, Suwanrsume H, Jirakulaporn T, et (suppl; abstr e19533). 2017;77(4 Suppl): abstract S6-07. Watson for Oncology (WfO) on MDT’s decision of al. A blinded comparison of patient treatments 95. Wrzeszczynski K, Frank M, Koyama T, et al. 76. Somashekhar SP, Sepulveda MJ, Shortliffe EH, adjuvant systemic therapy for hormone receptor to therapeutic options presented by an artificial Comparing sequencing assays and human- et al. A prospective blinded study of 1000 cases positive early breast carcinoma. J Clin Oncol. intelligence-based clinical decision-support machine analyses in actionable genomics for analyzing the role of artificial intelligence: Watson 2019;37 (suppl): abstract e18013. system. Ann Oncol. 2019; 30(Supplement 5). glioblastoma [published online July 11, 2017]. for Oncology and change in decision making 80. Soochit A, Zhang C, Li T, Dankwa-Mullan I, Liu 88. Suwanvecho S, Suwanrusme H, Sangtian M, Neurol Genet. 2017;3(4):e164. doi: 10.1212/ of a multidisciplinary tumor board (MDT) from J. Concordance assessment of an artificial et al. Concordance assessment of a cognitive NXG.0000000000000164 a tertiary care center. J Clin Oncol. 2019;37 intelligence decision support tool for primary and computing system in Thailand. J Clin Oncol. 96. Wrzeszczynski KO, Rahman S, Frank OM, et al. (suppl):abstract 6533. recurrent cervical cancer at an academic cancer 2017;35 (suppl; abstr 6589). doi: 10.1200/ Identification of targetable BRAF ∆N486_P490 77. Somashekhar S, Sepúlveda M, Norden A et al. center. Presented at: International Gynecologic JCO.2017.35.15_suppl.6589 variant by whole genome sequencing leading to3 Early experience with IBM Watson for Oncology Cancer Society (IGCS 2018); September 14 -16, 89. Tian Y, Liu X, Wang Z, et al. Concordance between dabrafenib-induced remission of a BRAF-mutant (WFO) cognitive computing system for lung 2018; Kyoto, Japan. Watson for Oncology and a multidisciplinary team pancreatic adenocarcinoma [published online and colorectal cancer treatment. J Clin Oncol. 81. Stevens R, Siaz F, Futreal A, Chin L. A cognitive clinical decision-making team for gastric cancer September 13, 2019]. Molecular Case Studies. 2017;35 (suppl;abstr 8527). doi: 10.1200/ approach to improve patient screening for clinical and the prognostic implications: retrospective doi:10.1101/mcs.a004424 JCO.2017.35.15_suppl.8527 trials. European Molecular Biology Laboratory study. JMIR. 2020;22(2):e14122. doi: 97. Wu A, Chen P, Li T, Dankwa-Mullan I, Sun T, Rhee 78. Somashekhar S, Sepulveda M, Puglielli S, et al. Conference on Cancer Genomics (EMBL 2015); 10.2196/14122 K. Real world survival outcomes in patients with Watson for Oncology and breast cancer treatment November 1 – 4, 2015; Heidelberg, Germany. 90. Tojo A. Clinical sequencing in leukemia with high risk stage II colon cancer at a Beijing Cancer recommendations: agreement with an expert 82. Suarez Saiz F, Sanders C, Stevens RJ, et al. Use the assistance of artificial intelligence. Rinsho Hospital. J Clin Oncol. 2018;36 (suppl; abstr multidisciplinary tumor board [published online of machine learning to identify relevant research Ketsueki. 2017;58(10):1913-1917. [article in e15670). January 9, 2018]. Ann Oncol. 2018;29(2):418- publications in clinical oncology. J Clin Oncol. Japanese] doi: 10.11406/rinketsu.58.1913 98. Xu F, Sepulveda MJ, Jiang Z, et al. Artificial 423. doi: 10.1093/annonc/mdx781 2019;37 (suppl; abstr 6558). 91. Vashistha V, Armstrong J, Winski D, et al. Barriers intelligence treatment decision support for 83. Suwanrusme H, Issarachai S, Umsawasdi T, et al. to prescribing targeted therapies for NSCLC complex breast cancer among oncologists with Concordance assessment of a cognitive clinical patients with highly-actionable gene variants in varying expertise. JCO Clin Cancer Inform. decision support tool in patients with solid tumors. the VA National Precision Oncology Program. J 2019;3:1-15. doi: 10.1200/CCI.18.00159 J Clin Oncol. 2018;36 (suppl; abstr e18584). Clin Oncol. 2020;38 (suppl; abstr 2005). 99. Xu J, Sun T, Hua S. Concordance assessment of 84. Suwanvecho S, Shortlifee E, Suwanrusme H, et al. 92. Vashishta V, Poonnen PJ, Patel VL, et al. Medical IBM Watson for Oncology with MDT in patients A blinded evaluation of a clinical decision-support oncologist perspectives of the Veterans Affairs with breast cancer. In: Proceedings of the 2018 system at a regional cancer care center. J Clin National Precision Oncology Program. J Natl San Antonio Breast Cancer Symposium; 2018 Oncol. 2019;37 (suppl; abstr 6553). Compr Canc Netw. 2019;17:3.5. doi: 10.6004/ Dec 4-8; San Antonio, TX. Philadelphia (PA). 85. Suwanvecho S, Suwanrusme H, Issarachai S, et jnccn.2018.7238 Cancer Research. 2019;79(4 Suppl:Abstract al. A blinded comparison of patient treatments 93. Wang Z, Zhonghe Y, Zhang X. Artificial nr P3-14-06. doi: 10.1158/1538-7445. Foreword to therapeutic options presented by an artificial intelligence-based clinical decision-support SABCS18-P3-14-06 Key studies: intelligence-based clinical decision-support system improves cancer treatment and 100. Xu J, Yang P, Xue S, et al. Translating cancer system. Presented at: ESMO Congress 2019; patient satisfaction. J Clin Oncol. 2019;37 genomics into precision medicine with artificial Clinical decision support September 30, 2019; Barcelona, Spain. (suppl):abstract e18303. intelligence: applications, challenges and future Clinical trials 86. Suwanvecho S, Suwanrusme H, Issarachia S, et al. 94. Weeraratne D, Huang H, Brotman D, et al. perspectives. Human Genetics. 2019; 138:109- Genomics Concordance between a clinical decision-support Genomic analysis of myeloproliferative neoplasm 124. Doi: 10.1007/s00439-019-01970-5 Real world data

Bibliography 31 101. Yao S, Wang R, Qian K, Zhang Y. Real world cancer. J Clin Oncol. 2018;36 (suppl; abstr study for the concordance between IBM Watson e18569). for Oncology and clinical practice in advanced 106. Zauderer M, Gucalp A, Epstein A, et al. Piloting non-small cell lung cancer patients at a lung IBM Watson Oncology within Memorial Sloan cancer center in China. Thorac Cancer. 2020. Kettering’s regional network. J Clin Oncol. 2020/03/20. 10.1111/1759-7714.13391 2014;32:e17653. DOI: 10.1200/jco.2014.32.15_ 102. Yu SH, Kim MS, Kim TH, et al. Early experience suppl.e17653 with Watson for Oncology, a clinical decision- 107. Zhang C, Soochit A, Dankwa-Mullan I, Li T, Liu support system for prostate cancer treatment J. Evaluation of personalized treatment with recommendations. J Urol. 2019; 201(4S): e268. artificial intelligence decision-support tools doi: 10.1097/01.JU.0000555489.90194.e3 for patients with ovarian cancer. Presented at: 103. You HS, Gao CX, Wang HB, et al. Concordance International Gynecologic Cancer Society (IGCS of treatment recommendations for metastatic 2018); September 14 -16, 2018; Kyoto, Japan. non-small-cell lung cancer between Watson https://igcs2018.com/scientific-information/ for Oncology system and medical team. Cancer interactive-program#.W7ZG9HtKjiw. Accessed Manag Res. 2020 Mar 16;12:1947-1958. October 4, 2018. eCollection 2020. doi: 10.2147/CMAR.S244932 108. Zhang W, Qi S, Zhuo J, Wen S, Fang C. 104. Yu Z, Wang Z, Ren X, et al. Practical exploration Concordance study in hepatectomy and research of Watson for Oncology clinical recommendations between Watson for decision support system in real-world and Oncology and clinical practice for patients with localized practice. J Clin Oncol. 2019;37 (suppl; hepatocellular carcinoma in China. World J Surg. abstr e18304). 2020;44(6):1945-1953. doi: 10.1007/s00268- 105. Yuwen D, Zhang W, Wu J, et al. Concordance 020-05401-9 evaluation of an artificial intelligence technology 109. Zhang X, Chen Y, Yang Y, et al. Comparative with a multidisciplinary tumor board in gastric analysis of target gene exon sequencing by cognitive technology using next generation sequencing platform in Chinese patients with lung cancer. J Clin Oncol. 2018;36 (suppl; abstr e24254). 110. Zhou N, Zhang CT, Lv HY, et al. Concordance study between IBM Watson for Oncology and clinical practice for patients with cancer in China [published online September 4, 2018]. Oncologist. 2018. pii: theoncologist.2018-0255. Foreword doi: 10.1634/theoncologist.2018-0255 Key studies: Clinical decision support Clinical trials Genomics Real world data

Bibliography 32 © Copyright IBM Corporation 2020

IBM, the IBM logo, .com, and Watson Health are trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the web at “Copyright and trademark information” at www.ibm.com/legal/copytrade.

Learn more at ibm.com/watson- health/oncology-and-genomics