Artificial Intelligence and Digital Pathology: What Pathologists Need to Know Toby C

Artificial Intelligence and Digital Pathology: What Pathologists Need to Know Toby C. Cornish, MD, PhD Associate Professor of Pathology Medical Director of Informatics Department of Pathology University of Colorado DISCLOSURE

In the past 12 months, I have had a significant financial interest or other relationship with the manufacturer(s) of the following product(s) or provider(s) of the following service(s) that will be discussed in my presentation.

• Leica Biosystems, Inc. Advanced Staining and Imaging Advisory Board: Consulting Fees

The IBM Watson Legacy (A cautionary tale?) Background: Electronic decision support is increasingly prevalent in clinical practice. Traditional tools map guidelines into an interactive platform. An alternative method builds on experience-based learning.

Methods: Memorial Sloan-Kettering (MSK), IBM and WellPoint teamed to develop IBM Watson – a cognitive computing system leveraging natural language processing (NLP), machine learning (ML) and massive parallel processing – to help inform clinical decision making. We made a prototype for lung cancers using manufactured and anonymized patient cases. We configured this tool to read medical language and extract specific attributes from each case to identify appropriate treatment options benchmarked against MSK expertise, anonymized patient cases and published evidence. Treatment options reflect consensus guidelines and MSK best practices where guidelines are not granular enough to match treatments to unique patients. Analysis and building accuracy is ongoing and iterative.

Journal of Clinical Oncology 31, no. 15_suppl (May 2013) 6508-6508.

Aug. 11, 2018 https://www.statnews.com/wp-content/uploads/2018/09/IBMs-Watson-recommended-unsafe-and-incorrect-cancer-treatments-STAT.pdf https://www.statnews.com/wp-content/uploads/2018/09/IBMs-Watson-recommended-unsafe-and-incorrect-cancer-treatments-STAT.pdf What is Artificial Intelligence? Have you used Artificial Intelligence in your clinical practice? Have you used AI in your practice? The “AI Effect”

"A lot of cutting-edge AI has filtered into general applications, often without being called AI because once something becomes useful enough and common enough it's not labelled AI anymore.“

Nick Bostrum Oxford University Professor & Founding Director of the Future of Humanity Institute

https://www.cnn.com/2006/TECH/science/07/24/ai.bostrom/ Artificial intelligence

• Production of intelligent-seeming behavior by machines

• Concept dates back to the early 1950s

• Comprises a broad range of approaches “Levels” of AI “Levels” of AI

• Artificial Narrow Intelligence (a.k.a Weak AI):

• Artificial General Intelligence (a.k.a Strong AI):

• Artificial Super Intelligence “Levels” of AI

General Intelligence Narrow Intelligence Super Intelligence >>> (Strong AI) >>> (Weak AI) ≈

Human Intelligence Artificial Narrow Intelligence

● Also known as “Weak AI” ● Highly specialized algorithms designed to answer specific, useful questions in narrowly defined problem domains ● e.g. identify breast cancer metastasis in a lymph node Artificial General Intelligence

● Also known as “Strong AI” ● Capable of general reasoning and human-like intelligence ● May equal or even surpass human capabilities in some ways ● e.g. perform “universal” histopathologic diagnoses Artificial Super Intelligence • AI that far exceeds human intelligence, surpassing it in all aspects — from creativity, to general wisdom, to problem-

solving singularity - is - https://steemit.com/ai/@ghasemkiani/what Approaches to AI Artificial Intelligence Machine Learning

Deep Learning

Adapted from Goodfellow et al., Deep Learning. 2016 Artificial Intelligence

Machine Learning

Deep Learning

Adapted from Goodfellow et al., Deep Learning. 2016 Machine learning

Artificial Intelligence • Term coined in 1959 by Arthur Samuel at IBM Machine Learning

Deep Learning • Attempts to have a computer perform a task without being explicitly programmed

• Three major types of ML: 1. Unsupervised learning 2. Supervised learning 3. Reinforcement learning Unsupervised Learning ● Goal is to determine unknown data patterns/groupings ○ Takes a set of unlabeled test data ○ Discovers commonalities in data

● Examples ○ Hierarchical clustering ○ k-means clustering ○ Principal component analysis ○ Gaussian mixture models ○ Hidden markov models

Image from: https://i.pinimg.com/736x/0d/43/e4/0d43e45d7ad2e78e45cd8f276a9a5b5e--types-of-machines-machine-learning.jpg Supervised Learning ● Trains a model to predict outputs from inputs ○ Requires labeled training data set ○ Once trained, the model can predict outputs for novel inputs

● Examples ○ Bayesian classifiers ○ Decision trees ○ Support vector machines ○ Linear regression ○ Random Decision Forests Image from: https://i.pinimg.com/736x/0d/43/e4/0d43e45d7ad2e78e45cd8f276a9a5b5e--types-of-machines-machine-learning.jpg Supervised Machine Learning, cont.

Artificial • Model that predicts outputs from sample inputs Intelligence Machine Learning – Iterative refinement of model’s performance using a Deep feedback loop Learning

• Classical machine learning relies heavily on extracting or selecting salient features from raw data (“feature engineering”) Labels v. Features

• Label – The “ground truth” in the training data – Therefore it is also what a model predicts • y variable in a simple linear regression (dependent variable or criterion variable) – Level of label = level of prediction – Key to supervised learning – Examples: diagnosis, classification, grade, presence or absence of a histologic feature, patient outcome Labels v. Features

• Features – The input variable(s) of the model • x variable in a simple linear regression (independent variable or predictor variable) – Can range from 1 variable to millions of variables – Feature engineering • the process of using domain knowledge of the data to select and / or derive meaningful features from raw data – Examples: • Simple measurements (e.g. tumor size), lab values, age • descriptors of color, texture, size, shape Supervised Machine learning workflow

Hyper-parameter tuning

Weight adjustment Features Raw Training set Historical & … Training … Build Model Data Labels 70% Results … Extract … Validation set Features … 20% … Validation … … … Results …

Label 10% s Test set … Test … Model Results

Expert

Adapted from https://www.capgemini.com/2016/05/machine-learning-has-transformed-many-aspects-of-our-everyday-life/ Supervised Machine learning workflow

Hyper-parameter tuning

Weight adjustment Features Raw Training set Historical & … Training … Build Model Data Labels 70% Results … Extract … Validation set Features … 20% … Validation … … … Results …

Label 10% s Test set … Test … Model Results

Expert

Adapted from https://www.capgemini.com/2016/05/machine-learning-has-transformed-many-aspects-of-our-everyday-life/ Training data in machine learning

• Large amounts of data and expert labeling are the keys to successful training of machine learning algorithms

• The amount of data needed is typically proportional to the difficulty of the task Training data in machine learning

https://hackernoon.com/%EF%B8%8F-big-challenge-in-deep-learning-training-data-31a88b97b282 Artificial Intelligence Machine Learning

Deep Learning

Adapted from Goodfellow et al., Deep Learning. 2016 Deep learning

Artificial Intelligence Machine Learning

Deep • The concept dates to the 1940s Learning • “Artificial neural networks” (1980s-90s) • “Deep learning” (c. 2006) • Multiple hidden layers in an artificial neural network • Computationally intensive • Until recently not feasible with large datasets Deep learning, cont.

Artificial Intelligence Machine Learning

• Avoids the need to define specific features in Deep the data as inputs Learning

• Discovers the features from the raw data provided during training

• Hidden layers in the artificial neural network represent increasingly more complex features in the data Machine learning v. deep learning

Tumor Not Tumor

Adapted from https://towardsdatascience.com/why-deep-learning- is-needed-over-traditional-machine-learning-1b6a99177063 A deep learning model example

Output (object identity)

3rd hidden layer (object parts)

2nd hidden layer (corners and contours)

1st hidden layer (edges)

Visible layer (input pixels)

Goodfellow et al., Deep Learning. 2016 AI on the Rise (again) Google Search Trends 100%

90%

80%

70%

60%

50%

40%

30%

20%

10%

Artificial intelligence Machine learning Deep learning Google Search Trends 100%

90%

80%

70%

60%

50% Watson wins 40% Jeopardy

30%

20%

10%

Artificial intelligence Machine learning Deep learning Gartner Hype Cycle

Peak of Inflated Expectations

Plateau of Productivity

Slope of Enlightenment Expectations

Trough of Disillusionment Technology Trigger Time Hype Cycle for Artificial Intelligence (2019)

https://www.gartner.com/smarterwithgartner/top-trends-on-the-gartner-hype-cycle-for-artificial-intelligence-2019/ Expectations Early approaches 1974 1980 Expert Systems 1987 Time 1993 Deep Learning Expectations Early approaches 1974 1st “AI Winter”1st “AI 1980 Expert Systems 1987 2nd “AI Winter” 2nd “AI Time 1993 Machine Learning What is an “AI Winter”?

• Extended period of time in which enthusiasm for, and investment in, AI “dried up”

• Generally attributed to: – Over-promising – Under-delivering – Creation of an economic bubble – Infighting – Inadequate computational hardware

• Two major “winters” are recognized along with many smaller events Expectations Early approaches 1974 1st “AI Winter”1st “AI 1980 Expert Systems 1987 2nd “AI Winter” 2nd “AI Time 1993 Deep Learning ? 3rd “AI Winter”? 3rd “AI ? ?

Artificial Intelligence in Pathology https://www.healthcareitnews.com/news/ai-set-transform-digital-pathology-hospitals-says-frost- sullivan 800 786

711 700

600

500

443

400

309 297 300 PubmedArticles 250

201 200 155 155 141 137 128 124 119

100 88 52 26 26 31 11 14 12 18 19 17 17 14 3 5 6 8 5 7 9 9 9 1 1 1 1 2 7 8 8 0 1986198719881989199019911992199319941995199619971998199920002001200220032004200520062007200820092010201120122013201420152016201720182019

Pathology and ("artificial intelligence" or "machine learning" or "deep learning") "Computational Pathology"

Bartels PH et al. Anal Quant Cytol Histol. 1988 Aug;10(4):299-306 800 786

711 700

600

500

443

400

309 297 300 PubmedArticles 250

201 200 155 155 141 137 128 124 119

Pathology and ("artificial intelligence" or "machine learning" or "deep learning") "Computational Pathology" Fuchs TJ, et al. Med Image Comput Comput Assist Interv. 2008;11(Pt 2):1-8. How will AI affect the Practice of Pathology? “If you work as a radiologist, you’re like the coyote that’s already over the edge of the cliff but hasn’t yet looked down so [you don’t] realize that there’s no ground underneath. People should stop training radiologists now; its just completely obvious that within five years, deep learning is going to do better than radiologists.”

Geoffrey Hinton The “Godfather of Deep Learning” and 2018 Turing Award Recipient

Machine Learning and the Market for Intelligence 2016 October 27, 2016 Toronto The future of AI in pathology

• Variety of opinions – Practicing pathologists – AI / Digital Pathology enthusiasts – Data scientists – Informaticists – Vendors The future of AI in pathology

• Depends on the time scale under consideration – 1 year? – 5 years? – 10 years? – 20 years? – 30 years? – 100 years? The future of AI in pathology

• Depends on what the underlying technology is – Deep learning? – A new paradigm in Artificial Intelligence? Strong (General)

Potential of AI

Weak (Narrow) Negative Positive Impact of AI Strong (General) Utopians

Potential of AI

Weak (Narrow) Negative Positive Impact of AI Strong (General) Dystopians Utopians

Potential of AI

Weak (Narrow) Negative Positive Impact of AI Strong (General) Dystopians Utopians

Potential of AI

Skeptics Weak (Narrow) Negative Positive Impact of AI Strong (General) Dystopians Utopians

Potential of AI

Skeptics Pragmatists Weak (Narrow) Negative Positive Impact of AI The pragmatist’s view of AI in Pathology A pragmatist’s view of AI in Pathology

• Super or General AI that could replace a pathologist is very far off, but not impossible

• Narrow AI tools will reach the early market within 3 - 5 years and the mainstream market within 5 - 8 years A pragmatist’s view of AI in Pathology

• Use AI as an adjunct to other traditional tests and methods • Derive diagnostic, prognostic and predictive information not possible before • Increase visibility and importance of pathologists in the healthcare enterprise • Create new billable tests (where applicable) • “Automate” the boring stuff • “Automate” the time-consuming stuff • “Automate” the hard stuff A pragmatist’s view of AI in Pathology

• AI will become just another tool we adopt and use alongside our traditional tools

• Consider previous scenarios: – External examination v. dissection – Macroscopic examination v. microscopic examination – Electron microscopy v. light microscopy – IHC v. H&E – Molecular pathology v. histopathology The fact that immunocytochemistry of human tissues is now a rich and complex field (in terms both of the different techniques available and the large number of different molecular entities detectable in human tissues) has tended to cause a polarisation among pathologists. There are the "monoclonal antibody revolutionaries" and an opposing group of "contras". This polarisation is not, as far as we know, described in the literature but is nevertheless to be found in pathology departments throughout the world. Mason DY & Gatter KC. J Clin Pathol. 1987, 40(9):1042-54 Skeptic?

Mason DY & Gatter KC. J Clin Pathol. 1987, 40(9):1042-54 Utopian?

Dystopian?

Mason DY & Gatter KC. J Clin Pathol. 1987, 40(9):1042-54 Utopian?

Dystopian?

Mason DY & Gatter KC. J Clin Pathol. 1987, 40(9):1042-54 Barriers to adoption Barriers to AI adoption

• Algorithm development

• Skepticism and dystopianism

• Operationalization Challenges in operationalizing AI

• Lack of “digital substrate”, i.e. digital pathology

• No standard model for AI workflow No DP = No AI Samples 3 Labels

4 1 2 Test Query 6 Orders Orders Orders 7 Test Orders Results Results Results Results 8 11 EHR/CPOE 10 LIS 9 Middleware Provider Instrument Figure 2: Architecture of a computational pathology app and platform. All work is done by the platform, but the app contains all the ‘intelligence’; it defines the algorithm, on which slides to run it, and how to present the results to the pathologist. This upfront programming consists of two steps: 1) select applicable slide types (stain, organ, etc.), and 2) program the app’s tasks on the analytics engine. These tasks are performed automatically in step 3) when a relevant slide is scanned. Finally when a pathologist reviews a relevant case, step 4) displays and allows interaction with the pre-calculated results. Figure 2: Architecture of a computational pathology app and platform. All work is done by the platform, but the app contains all the ‘intelligence’; it defines the algorithm, on which slides to run it, and how to present the results to the pathologist. This upfront programming consists of two steps: 1) select applicable slide types (stain, organ, etc.), and 2) program the app’s tasks on the analytics engine. These tasks are performed automatically in step 3) when a relevant slide is scanned. Finally when a pathologist reviews a relevant case, step 4) displays and allows interaction with the pre-calculated results. Thoughts from another domain… Kasparov v. Deep Blue

• May 11, 1997 - 29.jpg • Match result: Deep Blue (W) - Kasparov: 3½ - 2½ https://commons.wikimedia.org/wiki/File:Kasparov https://www.flickr.com/photos/amitrajit/ What have they been up to since? Deep Blue com/photos/amitrajit/ . flickr . //www : Computer History Museum (CHM), Mountain View,

California, US https Garry Kasparov

• In June 1998, Kasparov played the first public game of human- computer collaborative chess against Veselin Topalov • Each used a regular computer with off-the-shelf chess software • Kasparov called this “advanced chess,” and it has later been called “centaur chess” • Retired in 2005 “Freestyle” chess

• 2005: the first “freestyle” chess tournament • Teams could consist of any number of humans or computers • Some teams consisted of chess grand masters • The most powerful chess computer at the time was also entered • The winning team consisted of young, amateur players, Steven Cramton and Zackary Stephen and their computers “weak human + machine + better process was superior to a strong computer alone and, more remarkable, superior to a strong human + machine + inferior process.” - Garry Kasparov "The Chess Master and the Computer“ The New York Review of Books, 2/11/2010 http://www.nybooks.com/articles/2010/02/1 1/the-chess-master-and-the-computer/ Summary

• AI / Computational pathology is coming with near term applications of narrow (weak) AI to clinical problems

• AI is just another technology to incorporate into our practice

• Significant development and deployment challenges persist

• AI will only endanger jobs for people that fail to embrace it Questions?

Toby C. Cornish, M.D., Ph.D Associate Professor of Pathology University of Colorado School of Medicine [email protected] @TobyCornish