Artificial Intelligence and Digital Pathology: What Pathologists Need to Know Toby C. Cornish, MD, PhD Associate Professor of Pathology Medical Director of Informatics Department of Pathology University of Colorado DISCLOSURE In the past 12 months, I have had a significant financial interest or other relationship with the manufacturer(s) of the following product(s) or provider(s) of the following service(s) that will be discussed in my presentation. • Leica Biosystems, Inc. Advanced Staining and Imaging Advisory Board: Consulting Fees The IBM Watson Legacy (A cautionary tale?) Background: Electronic decision support is increasingly prevalent in clinical practice. Traditional tools map guidelines into an interactive platform. An alternative method builds on experience-based learning. Methods: Memorial Sloan-Kettering (MSK), IBM and WellPoint teamed to develop IBM Watson – a cognitive computing system leveraging natural language processing (NLP), machine learning (ML) and massive parallel processing – to help inform clinical decision making. We made a prototype for lung cancers using manufactured and anonymized patient cases. We configured this tool to read medical language and extract specific attributes from each case to identify appropriate treatment options benchmarked against MSK expertise, anonymized patient cases and published evidence. Treatment options reflect consensus guidelines and MSK best practices where guidelines are not granular enough to match treatments to unique patients. Analysis and building accuracy is ongoing and iterative. Journal of Clinical Oncology 31, no. 15_suppl (May 2013) 6508-6508. Aug. 11, 2018 https://www.statnews.com/wp-content/uploads/2018/09/IBMs-Watson-recommended-unsafe-and-incorrect-cancer-treatments-STAT.pdf https://www.statnews.com/wp-content/uploads/2018/09/IBMs-Watson-recommended-unsafe-and-incorrect-cancer-treatments-STAT.pdf What is Artificial Intelligence? Have you used Artificial Intelligence in your clinical practice? Have you used AI in your practice? The “AI Effect” "A lot of cutting-edge AI has filtered into general applications, often without being called AI because once something becomes useful enough and common enough it's not labelled AI anymore.“ Nick Bostrum Oxford University Professor & Founding Director of the Future of Humanity Institute https://www.cnn.com/2006/TECH/science/07/24/ai.bostrom/ Artificial intelligence • Production of intelligent-seeming behavior by machines • Concept dates back to the early 1950s • Comprises a broad range of approaches “Levels” of AI “Levels” of AI • Artificial Narrow Intelligence (a.k.a Weak AI): • Artificial General Intelligence (a.k.a Strong AI): • Artificial Super Intelligence “Levels” of AI General Intelligence Narrow Intelligence Super Intelligence >>> (Strong AI) >>> (Weak AI) ≈ Human Intelligence Artificial Narrow Intelligence ● Also known as “Weak AI” ● Highly specialized algorithms designed to answer specific, useful questions in narrowly defined problem domains ● e.g. identify breast cancer metastasis in a lymph node Artificial General Intelligence ● Also known as “Strong AI” ● Capable of general reasoning and human-like intelligence ● May equal or even surpass human capabilities in some ways ● e.g. perform “universal” histopathologic diagnoses Artificial Super Intelligence • AI that far exceeds human intelligence, surpassing it in all aspects — from creativity, to general wisdom, to problem- solving singularity - is - https://steemit.com/ai/@ghasemkiani/what Approaches to AI Artificial Intelligence Machine Learning Deep Learning Adapted from Goodfellow et al., Deep Learning. 2016 Artificial Intelligence Machine Learning Deep Learning Adapted from Goodfellow et al., Deep Learning. 2016 Machine learning Artificial Intelligence • Term coined in 1959 by Arthur Samuel at IBM Machine Learning Deep Learning • Attempts to have a computer perform a task without being explicitly programmed • Three major types of ML: 1. Unsupervised learning 2. Supervised learning 3. Reinforcement learning Unsupervised Learning ● Goal is to determine unknown data patterns/groupings ○ Takes a set of unlabeled test data ○ Discovers commonalities in data ● Examples ○ Hierarchical clustering ○ k-means clustering ○ Principal component analysis ○ Gaussian mixture models ○ Hidden markov models Image from: https://i.pinimg.com/736x/0d/43/e4/0d43e45d7ad2e78e45cd8f276a9a5b5e--types-of-machines-machine-learning.jpg Supervised Learning ● Trains a model to predict outputs from inputs ○ Requires labeled training data set ○ Once trained, the model can predict outputs for novel inputs ● Examples ○ Bayesian classifiers ○ Decision trees ○ Support vector machines ○ Linear regression ○ Random Decision Forests Image from: https://i.pinimg.com/736x/0d/43/e4/0d43e45d7ad2e78e45cd8f276a9a5b5e--types-of-machines-machine-learning.jpg Supervised Machine Learning, cont. Artificial • Model that predicts outputs from sample inputs Intelligence Machine Learning – Iterative refinement of model’s performance using a Deep feedback loop Learning • Classical machine learning relies heavily on extracting or selecting salient features from raw data (“feature engineering”) Labels v. Features • Label – The “ground truth” in the training data – Therefore it is also what a model predicts • y variable in a simple linear regression (dependent variable or criterion variable) – Level of label = level of prediction – Key to supervised learning – Examples: diagnosis, classification, grade, presence or absence of a histologic feature, patient outcome Labels v. Features • Features – The input variable(s) of the model • x variable in a simple linear regression (independent variable or predictor variable) – Can range from 1 variable to millions of variables – Feature engineering • the process of using domain knowledge of the data to select and / or derive meaningful features from raw data – Examples: • Simple measurements (e.g. tumor size), lab values, age • descriptors of color, texture, size, shape Supervised Machine learning workflow Hyper-parameter tuning Weight adjustment Features Raw Training set Historical & … Training … Build Model Data Labels 70% Results … Extract … Validation set Features … 20% … Validation … … … Results … Label 10% s Test set … Test … Model Results Expert Adapted from https://www.capgemini.com/2016/05/machine-learning-has-transformed-many-aspects-of-our-everyday-life/ Supervised Machine learning workflow Hyper-parameter tuning Weight adjustment Features Raw Training set Historical & … Training … Build Model Data Labels 70% Results … Extract … Validation set Features … 20% … Validation … … … Results … Label 10% s Test set … Test … Model Results Expert Adapted from https://www.capgemini.com/2016/05/machine-learning-has-transformed-many-aspects-of-our-everyday-life/ Training data in machine learning • Large amounts of data and expert labeling are the keys to successful training of machine learning algorithms • The amount of data needed is typically proportional to the difficulty of the task Training data in machine learning https://hackernoon.com/%EF%B8%8F-big-challenge-in-deep-learning-training-data-31a88b97b282 Artificial Intelligence Machine Learning Deep Learning Adapted from Goodfellow et al., Deep Learning. 2016 Deep learning Artificial Intelligence Machine Learning Deep • The concept dates to the 1940s Learning • “Artificial neural networks” (1980s-90s) • “Deep learning” (c. 2006) • Multiple hidden layers in an artificial neural network • Computationally intensive • Until recently not feasible with large datasets Deep learning, cont. Artificial Intelligence Machine Learning • Avoids the need to define specific features in Deep the data as inputs Learning • Discovers the features from the raw data provided during training • Hidden layers in the artificial neural network represent increasingly more complex features in the data Machine learning v. deep learning Tumor Not Tumor Tumor Not Tumor Adapted from https://towardsdatascience.com/why-deep-learning- is-needed-over-traditional-machine-learning-1b6a99177063 A deep learning model example Output (object identity) 3rd hidden layer (object parts) 2nd hidden layer (corners and contours) 1st hidden layer (edges) Visible layer (input pixels) Goodfellow et al., Deep Learning. 2016 AI on the Rise (again) Google Search Trends 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Artificial intelligence Machine learning Deep learning Google Search Trends 100% 90% 80% 70% 60% 50% Watson wins 40% Jeopardy 30% 20% 10% 0% Artificial intelligence Machine learning Deep learning Gartner Hype Cycle Peak of Inflated Expectations Plateau of Productivity Slope of Enlightenment Expectations Trough of Disillusionment Technology Trigger Time Hype Cycle for Artificial Intelligence (2019) https://www.gartner.com/smarterwithgartner/top-trends-on-the-gartner-hype-cycle-for-artificial-intelligence-2019/ Early approaches Expert Systems Deep Learning Expectations 1974 1980 1987 1993 Time Early approaches Expert Systems Machine Learning Expectations 1st “AI Winter” 2nd “AI Winter” 1974 1980 1987 1993 Time What is an “AI Winter”? • Extended period of time in which enthusiasm for, and investment in, AI “dried up” • Generally attributed to: – Over-promising – Under-delivering – Creation of an economic bubble – Infighting – Inadequate computational hardware • Two major “winters” are recognized along with many smaller events Early approaches Expert Systems Deep Learning ? Expectations 1st “AI Winter” 2nd “AI Winter” 3rd “AI Winter”? 1974 1980 1987 1993 ? ? Time