ADVANCEMENTS AND TRENDS IN MEDICAL IMAGE ANALYSIS USING DEEP LEARNING
A presentation by Shekoofeh Azizi University of British Columbia, Canada, Ph.D. 2014-2018 Electrical and Computer Engineering
Isfahan University of Technology, Iran, M.Sc. 2011-2013 Computer Engineering / Hardware Design
Isfahan University of Technology, Iran, B.Sc. 2007-2011 Computer Engineering / Hardware Engineering
Philips Research North America, 2015-now National Institutes of Health (NIH), 2015-now
MICCAI Student Board Officer, 2016-now Women in MICCAI, 2017-now
2 UBC Queen’s University SFU Univ. of Western Ontario VGH Robarts Research
Technical Univ. of Munich ETH Zurich
NVidia Sejong Univ., Korea Philips IBM Univ. of Colorado NIH
Size is related to the number of collaborators
Academic Collaborators
Industrial/Clinical Collaborators
3 OUTLINE
Deep Learning Medical Imaging Challenges and Vision Opportunities
4 DEEP LEARNING
5 ARTIFICIAL INTELLIGENCE VS. DATA SCIENCE
AI is a technique which enables machines to mimic the “human behaviour”
It’s 17°C Hey Google, and sunny in Weather in Victoria Victoria! Voice Services
Voice to Command Brain/Model Voice Output
User Google Home Google Home
6 ARTIFICIAL INTELLIGENCE VS. DATA SCIENCE
Data Science is about processes and systems to extract knowledge or insights from data in Artificial Intelligence various forms. (AI)
Machine Learning is the connection Machine Learning between data science and artificial (ML) intelligence since machine learning is the process of learning from data over time.
Data Science Deep Learning (DL)
7 MACHINE LEARNING (ML)
Machine learning is the process of learning from data over time.
Reinforcement Learning
Andrew Ng, Machine Learning Course, Coursera. 9 DEEP LEARNING (DL)
Inspired by the functionality of our brains Number of sides: 4? Closed form shape?
Perpendicular sides? Square ? Equal sides?
10 DEEP LEARNING (DL)
Cat Feature Learning Dog
11 DEEP LEARNING (DL)
12 WHY DEEP LEARNING?
Consistent improvement over the state-of-the-art across a large variety of domains.
Over 14 million images and 20 thousand categories. 13 WHY DEEP LEARNING?
Tensorflow Object Detection API, 2015. Drive.ai’s self-driving car handle California city streets on a rainy night.
14 HEALTHCARE AND MEDICAL IMAGE ANALYSIS
15 THE ROLE OF IMAGING IN HEALTHCARE
Diagnosis Quantification Planning Monitoring Intervention
Slide Credit: NVidia 16 OPPORTUNITIES FOR AI IN RADIOLOGY
Machine learning software will serve as a very experienced clinical assistant, augmenting the doctor and making workflow more efficient and accurate.
Reconstruction Analysis Big Data Medical Report
AI for Image Reconstruction ML/DL for Medical Image Pattern Recognition Natural Language from Sensors Analysis Processing
17 APPLICATION OF AI IN MEDICAL IMAGE ANALYSIS
Segmentation:
• Assign each pixel of the image to a class
• In computer vision: need to generalize to different scenes, lightning, pose, corner-cases.
• In medical imaging: need to be precise at pixel level, account for variations in scan quality, artifacts, anatomy.
• Images vs. Volumes Brain MRI Segmentation 3D Ultrasound Volumetric Segmentation Project Clara, NVidia
Ker, Justin, et al. "Deep learning applications in medical image analysis." IEEE Access 6 (2018): 9375-9389. Razzak, Muhammad Imran, "Deep Learning for Medical Image Processing: Overview, Challenges and the 18 Future." In Classification in BioApps, pp. 323-350. Springer, Cham, 2018. APPLICATION OF AI IN MEDICAL IMAGE ANALYSIS
Detection/Classification:
• Predicting Lung Cancer Using CT Scan
• Red is showing Cancer Region
• Accuracy 0.86
• Kaggle Competition (1 million)
19 © http://blog.kaggle.com/2017/06/29/2017-data-science-bowl-predicting-lung-cancer-2nd-place-solution-write-up-daniel-hammack-and-julian-de-wit/ CHALLENGES
• Requires extensive inter-organization collaboration
• Data annotation: Medical Doctors – Noise and sparse labeling – Tedious and expensive Medical Physicists – Rare disease • Data variability Computer Scientist • Interpretability of the decision making model and acceptance by health profession 20 Example of Medical Image Analysis Using Deep Learning:
Prostate Cancer Diagnosis Using TEMPORAL ENHANCED ULTRASOUND
21 TEMPORAL ENHANCED ULTRASOUN (TeUS)
[Moradi’07, Moradi’09, Imani’15, Khojaste’15, Ghavidel’16 ] S. Azizi, et al., “Detection and grading of prostate cancer using temporal enhanced ultrasound: combining deep neural networks 22 and tissue mimicking simulations,” Journal of Computer Assisted Radiology and Surgery (IJCARS): MICCAI’16 special issues, 2017. TEMPORAL ENHANCED ULTRASOUN (TeUS)
Feature Learning Cancer
Classification
Benign
[Moradi’07, Moradi’09, Imani’15, Khojaste’15, Ghavidel’16 ] 23 PROSTATE CANCER GRADING USING TeUS CHALLENGES
Clinically significant
...
Benign GS 3+3 GS 3+4 GS 4+3 GS 4+4
Clinically less significant
© Correas et al. 2013, Iczkowski et al. 2011. S. Azizi, et al., “Detection and grading of prostate cancer using temporal enhanced ultrasound: combining deep neural networks 24 and tissue mimicking simulations,” Journal of Computer Assisted Radiology and Surgery (IJCARS): MICCAI’16 special issues, 2017. PROSTATE CANCER GRADING USING TeUS CHALLENGES Benign or other tissue types?
? ?
Exact location of the Exact location of the ROIs with unknown cancer: unknown. Gleason patterns: unknown. pathology: other tissue types.
Clinically significant
...
Benign GS 3+3 GS 3+4 GS 4+3 GS 4+4
Clinically less significant
© Correas et al. 2013, Iczkowski et al. 2011. S. Azizi, et al., “Detection and grading of prostate cancer using temporal enhanced ultrasound: combining deep neural networks 25 and tissue mimicking simulations,” Journal of Computer Assisted Radiology and Surgery (IJCARS): MICCAI’16 special issues, 2017. FEATURE LEARNING + DISTRIBUTION LEARNING
Benign GS 4+4 GS 3+3 GS 3+4 GS 4+3
Other Tissue Type Benign Cluster
Feature 2 2 2 2 2 2 Feature Feature Feature Feature Feature Feature
Cluster of Gleason Cluster of Gleason Pattern 3 Pattern 4 Feature 1 Feature Space S. Azizi, et al., “Detection and grading of prostate cancer using temporal enhanced ultrasound: combining deep neural networks 26 and tissue mimicking simulations,” Journal of Computer Assisted Radiology and Surgery (IJCARS): MICCAI’16 special issues, 2017. FEATURE LEARNING + DISTRIBUTION LEARNING
Deep Belief Network (DBN)
20 ROIs
Feature Space Distribution Clustering (F1,F2) Learning Model Target
Training Dataset Visible Layer Hidden Layers Benign Cluster
Test Data ? ? GS3 GS4
? ? ? ? ? ? ? ? Trained Clustering ? ? ? ? Deep Network Model
? ? 2 Feature ? ? ? ?
Cluster of Gleason Cluster of Gleason Pattern 3 Feature 1 Pattern 4
S. Azizi, et al., “Detection and grading of prostate cancer using temporal enhanced ultrasound: combining deep neural networks 27 and tissue mimicking simulations,” Journal of Computer Assisted Radiology and Surgery (IJCARS): MICCAI’16 special issues, 2017. DATA VARIABILITY: RF VS. B-MODE
Radio Frequency (RF) : Beamformer − Richer source of information than B-mode. Beamformed − Not accessible on commercial scanners. RF Data
Back-end Signal Processing
Scan Conversion
B
-
mode mode Image
S. Azizi, et al., “Transfer learning from RF to B-mode temporal enhanced ultrasound features for prostate cancer detection,” 28 Journal of Computer Assisted Radiology and Surgery: IPCAI’17 special issues, 2017. TRANSFER LEARNING: RF VS. B-MODE TeUS
Unlabeled B-mode TeUS Data RF TeUS Data
Preprocessing and Feature Extraction Objective function: Reconstruction error + KL divergence + Loss function
RF TeUS B-mode TeUS
. . .
퐃퐊퐋 퐑퐅, 퐁퐦퐨퐝퐞 = − 퐑퐅 퐢 퐥퐨퐠 퐁퐦퐨퐝퐞(퐢)
퐢
......
RF-mimicking TeUS . . . or 퐁퐦퐨퐝퐞 TeUS
S. Azizi, et al., “Investigating deep recurrent neural networks for prostate cancer detection: analysis of temporal enhanced 29 ultrasound,” IEEE Transaction on Medical Imaging (TMI), 2018.
TRANSFER LEARNING: RF VS. B-MODE TeUS
Unlabeled B-mode TeUS Data RF TeUS Data Labeled B-mode TeUS Data
Preprocessing and Feature Extraction Preprocessing and Feature Extraction
RF TeUS Transfer learning network B-mode TeUS
. . . RF-mimicking TeUS
......
. . . . . Joint Classification Network
RF-mimicking TeUS Benign vs. Cancer
S. Azizi, et al., “Investigating deep recurrent neural networks for prostate cancer detection: analysis of temporal enhanced 30 ultrasound,” IEEE Transaction on Medical Imaging (TMI), 2018. TRANSFER LEARNING: RF VS. B-MODE TeUS DECISION MAKING MODEL
Performance: Area under ROC Curve = 0.96 Run-time = 1.66 ± 0.32 second for 100 frames
S. Azizi, et al., “Transfer learning from RF to B-mode temporal enhanced ultrasound features for prostate cancer detection,” Journal of Computer Assisted Radiology and Surgery: IPCAI’17 special issues, 2017. S. Azizi, et al., “Investigating deep recurrent neural networks for prostate cancer detection: analysis of temporal enhanced 31 ultrasound,” IEEE Transaction on Medical Imaging (TMI), 2018.
MODEL INTERPRETATION
Benign Gleason Pattern 3 Gleason Pattern 4 Layer 3:
6 hidden neurons
Layer 2: 50 hidden neurons
Layer 1: Back Propagation 100 hidden neurons DBN Trained Visible Layer 50 spectral features
Absolute Difference
Low-frequency components
32 MODEL INTERPRETATION
Computer Scientist
Medical Doctors Medical Physicists
33 MODEL INTERPRETATION Cancer Benign Cell Nuclei (Scatterers) Speckle
Cell Nuclei Speckle
Tissue response = f (Acoustic signal, Tissue microstructure,…)
Hunt, J. W., et. al. “The subtleties of ultrasound images of an ensemble of cells: simulation from regular and more random distributions 34 of scatterers.” Ultrasound in medicine & biology, 21(3), 329-341, 1995. MODEL INTERPRETATION
Digital Pathology
Finite Ultrasound Temporal Ultrasound Element Simulations Generation Simulations (Field II)
Feature Extraction
Cancer Normal
Time
Nuclei Location Extraction
- K. Iczkowski, et al., "Digital quantification of five high-grade PCa patterns, including the cribri-form pattern, and their association with adverse outcome", American Journal of Clinical Pathology (2011). (University of Colorado) - S. Bayat, et al., “Tissue mimicking simulations for temporal enhanced US-based tissue typing”, SPIE 2017. 35 HOW WILL AI IMPACT THE HEALTHCARE LANDSCAPE?
36 DEEP LEARNING MOMENTUM BUILDING
Medical Imaging Papers Using DL AI Across Healthcare Academic Pubs.
Slide Credit: NVidia, 37 DL for Health Informatics - Daniele Ravi, et. al., IEEE Journal of Biomedical and Health Informatics, Vol. 21, No. 1, January 2017 CAMBRIAN EXPLOSION
Slide Credit: NVidia 38 UNCERTAINTY VS. ACCURACY
• Uncertainty • Is it hard to say I don’t know? • Human level accuracy • Noise
Gal, Yarin, and Zoubin Ghahramani. "Dropout as a Bayesian approximation: Representing model uncertainty in deep learning." international conference on machine learning. 2016. 39 Who Said What: Modeling Individual Labelers Improves Classification, Guan et al., AAAI (Google Brain) UNCERTAINTY VS. ACCURACY
• Uncertainty • Is it hard to say I don’t know? • Human level accuracy • Noise
Gal, Yarin, and Zoubin Ghahramani. "Dropout as a Bayesian approximation: Representing model uncertainty in deep learning." international conference on machine learning. 2016. 40 Who Said What: Modeling Individual Labelers Improves Classification, Guan et al., AAAI (Google Brain) UNCERTAINTY VS. ACCURACY
• Uncertainty • Is it hard to say I don’t know? • Human level accuracy • Noise
Video of the first self-driving car crash that killed a pedestrian in the US shows how the autonomous Uber failed to slow down before it hit a 49-year-old woman walking her bike across the street. It has raised fresh questions about why the vehicle did not stop when a human entered its path.
Ker, Justin, et al. "Deep learning applications in medical image analysis." IEEE Access 6 (2018): 9375-9389. Greenspan, Hayit,, et al, "Guest editorial deep learning in medical imaging: Overview and future promise 41 of an exciting new technique." IEEE Transactions on Medical Imaging 35.5 (2016): 1153-1159. INTERPRETABILITY
• How well can we get along with machines that are unpredictable?
• A patient who is being told that he/she must undergo chemotherapy is unlikely to accept the answer, “The machine learning algorithm said so, based on previous case data and your current condition.”
42 RESEARCH VISION
• Effective measurement of uncertainty, CIHR CHRP: Artificial discovering the source of it and integrating Intelligence, Health and Society proper solutions in deep learning-based decision making models.
CIFAR for AI ? uncertainty Interpretability
Collaborative Research and Development (CRD) Grants
43 CONCLUSION What I learned from AI in Medical Image Analysis:
Accuracy DEEP LEARN NG Between Hopes and Fears Interpretability Uncertainty
44 THANK YOU! QUESTIONS? 46