ADVANCEMENTS AND TRENDS IN MEDICAL IMAGE ANALYSIS USING DEEP LEARNING A presentation by Shekoofeh Azizi University of British Columbia, Canada, Ph.D. 2014-2018 Electrical and Computer Engineering Isfahan University of Technology, Iran, M.Sc. 2011-2013 Computer Engineering / Hardware Design Isfahan University of Technology, Iran, B.Sc. 2007-2011 Computer Engineering / Hardware Engineering Philips Research North America, 2015-now National Institutes of Health (NIH), 2015-now MICCAI Student Board Officer, 2016-now Women in MICCAI, 2017-now 2 UBC Queen’s University SFU Univ. of Western Ontario VGH Robarts Research Technical Univ. of Munich ETH Zurich NVidia Sejong Univ., Korea Philips IBM Univ. of Colorado NIH Size is related to the number of collaborators Academic Collaborators Industrial/Clinical Collaborators 3 OUTLINE Deep Learning Medical Imaging Challenges and Vision Opportunities 4 DEEP LEARNING 5 ARTIFICIAL INTELLIGENCE VS. DATA SCIENCE AI is a technique which enables machines to mimic the “human behaviour” It’s 17°C Hey Google, and sunny in Weather in Victoria Victoria! Voice Services Voice to Command Brain/Model Voice Output User Google Home Google Home 6 ARTIFICIAL INTELLIGENCE VS. DATA SCIENCE Data Science is about processes and systems to extract knowledge or insights from data in Artificial Intelligence various forms. (AI) Machine Learning is the connection Machine Learning between data science and artificial (ML) intelligence since machine learning is the process of learning from data over time. Data Science Deep Learning (DL) 7 MACHINE LEARNING (ML) Machine learning is the process of learning from data over time. Reinforcement Learning Andrew Ng, Machine Learning Course, Coursera. 9 DEEP LEARNING (DL) Inspired by the functionality of our brains Number of sides: 4? Closed form shape? Perpendicular sides? Square ? Equal sides? 10 DEEP LEARNING (DL) Cat Feature Learning Dog 11 DEEP LEARNING (DL) 12 WHY DEEP LEARNING? Consistent improvement over the state-of-the-art across a large variety of domains. Over 14 million images and 20 thousand categories. 13 WHY DEEP LEARNING? Tensorflow Object Detection API, 2015. Drive.ai’s self-driving car handle California city streets on a rainy night. 14 HEALTHCARE AND MEDICAL IMAGE ANALYSIS 15 THE ROLE OF IMAGING IN HEALTHCARE Diagnosis Quantification Planning Monitoring Intervention Slide Credit: NVidia 16 OPPORTUNITIES FOR AI IN RADIOLOGY Machine learning software will serve as a very experienced clinical assistant, augmenting the doctor and making workflow more efficient and accurate. Reconstruction Analysis Big Data Medical Report AI for Image Reconstruction ML/DL for Medical Image Pattern Recognition Natural Language from Sensors Analysis Processing 17 APPLICATION OF AI IN MEDICAL IMAGE ANALYSIS Segmentation: • Assign each pixel of the image to a class • In computer vision: need to generalize to different scenes, lightning, pose, corner-cases. • In medical imaging: need to be precise at pixel level, account for variations in scan quality, artifacts, anatomy. • Images vs. Volumes Brain MRI Segmentation 3D Ultrasound Volumetric Segmentation Project Clara, NVidia Ker, Justin, et al. "Deep learning applications in medical image analysis." IEEE Access 6 (2018): 9375-9389. Razzak, Muhammad Imran, "Deep Learning for Medical Image Processing: Overview, Challenges and the 18 Future." In Classification in BioApps, pp. 323-350. Springer, Cham, 2018. APPLICATION OF AI IN MEDICAL IMAGE ANALYSIS Detection/Classification: • Predicting Lung Cancer Using CT Scan • Red is showing Cancer Region • Accuracy 0.86 • Kaggle Competition (1 million) 19 © http://blog.kaggle.com/2017/06/29/2017-data-science-bowl-predicting-lung-cancer-2nd-place-solution-write-up-daniel-hammack-and-julian-de-wit/ CHALLENGES • Requires extensive inter-organization collaboration • Data annotation: Medical Doctors – Noise and sparse labeling – Tedious and expensive Medical Physicists – Rare disease • Data variability Computer Scientist • Interpretability of the decision making model and acceptance by health profession 20 Example of Medical Image Analysis Using Deep Learning: Prostate Cancer Diagnosis Using TEMPORAL ENHANCED ULTRASOUND 21 TEMPORAL ENHANCED ULTRASOUN (TeUS) [Moradi’07, Moradi’09, Imani’15, Khojaste’15, Ghavidel’16 ] S. Azizi, et al., “Detection and grading of prostate cancer using temporal enhanced ultrasound: combining deep neural networks 22 and tissue mimicking simulations,” Journal of Computer Assisted Radiology and Surgery (IJCARS): MICCAI’16 special issues, 2017. TEMPORAL ENHANCED ULTRASOUN (TeUS) Feature Learning Cancer Classification Benign [Moradi’07, Moradi’09, Imani’15, Khojaste’15, Ghavidel’16 ] 23 PROSTATE CANCER GRADING USING TeUS CHALLENGES Clinically significant ... Benign GS 3+3 GS 3+4 GS 4+3 GS 4+4 Clinically less significant © Correas et al. 2013, Iczkowski et al. 2011. S. Azizi, et al., “Detection and grading of prostate cancer using temporal enhanced ultrasound: combining deep neural networks 24 and tissue mimicking simulations,” Journal of Computer Assisted Radiology and Surgery (IJCARS): MICCAI’16 special issues, 2017. PROSTATE CANCER GRADING USING TeUS CHALLENGES Benign or other tissue types? ? ? Exact location of the Exact location of the ROIs with unknown cancer: unknown. Gleason patterns: unknown. pathology: other tissue types. Clinically significant ... Benign GS 3+3 GS 3+4 GS 4+3 GS 4+4 Clinically less significant © Correas et al. 2013, Iczkowski et al. 2011. S. Azizi, et al., “Detection and grading of prostate cancer using temporal enhanced ultrasound: combining deep neural networks 25 and tissue mimicking simulations,” Journal of Computer Assisted Radiology and Surgery (IJCARS): MICCAI’16 special issues, 2017. FEATURE LEARNING + DISTRIBUTION LEARNING Benign GS 4+4 GS 3+3 GS 3+4 GS 4+3 Other Tissue Type Benign Cluster Feature 2 2 2 2 2 2 Feature Feature Feature Feature Feature Feature Cluster of Gleason Cluster of Gleason Pattern 3 Pattern 4 Feature 1 Feature Space S. Azizi, et al., “Detection and grading of prostate cancer using temporal enhanced ultrasound: combining deep neural networks 26 and tissue mimicking simulations,” Journal of Computer Assisted Radiology and Surgery (IJCARS): MICCAI’16 special issues, 2017. FEATURE LEARNING + DISTRIBUTION LEARNING Deep Belief Network (DBN) 20 ROIs Feature Space Distribution Clustering (F1,F2) Learning Model Target Training Dataset Visible Layer Hidden Layers Benign Cluster Test Data ? ? GS3 GS4 ? ? ? ? ? ? ? ? Trained Clustering ? ? ? ? Deep Network Model ? ? 2 Feature ? ? ? ? Cluster of Gleason Cluster of Gleason Pattern 3 Feature 1 Pattern 4 S. Azizi, et al., “Detection and grading of prostate cancer using temporal enhanced ultrasound: combining deep neural networks 27 and tissue mimicking simulations,” Journal of Computer Assisted Radiology and Surgery (IJCARS): MICCAI’16 special issues, 2017. DATA VARIABILITY: RF VS. B-MODE Radio Frequency (RF) : Beamformer − Richer source of information than B-mode. Beamformed − Not accessible on commercial scanners. Data RF Back-end Signal Processing Scan Conversion B - mode Image mode S. Azizi, et al., “Transfer learning from RF to B-mode temporal enhanced ultrasound features for prostate cancer detection,” 28 Journal of Computer Assisted Radiology and Surgery: IPCAI’17 special issues, 2017. TRANSFER LEARNING: RF VS. B-MODE TeUS Unlabeled B-mode TeUS Data RF TeUS Data Preprocessing and Feature Extraction Objective function: Reconstruction error + KL divergence + Loss function RF TeUS B-mode TeUS . 퐃퐊퐋 퐑퐅, 퐁퐦퐨퐝퐞 = − 퐑퐅 퐢 퐥퐨퐠 퐁퐦퐨퐝퐞(퐢) 퐢 . RF-mimicking TeUS . or 퐁퐦퐨퐝퐞 TeUS S. Azizi, et al., “Investigating deep recurrent neural networks for prostate cancer detection: analysis of temporal enhanced 29 ultrasound,” IEEE Transaction on Medical Imaging (TMI), 2018. RF ultrasound,” IEEE Transaction on Medical Imaging (TMI), 2018(TMI), Imaging on Medical Transaction IEEE ultrasound,” enhanced temporal of analysis detection: cancer prostate for networks neural recurrent deep“Investigating Azizi, et al., S. - mimicking TeUS TRANSFER LEARNING: RF TRANSFER LEARNING: B - mode TeUS Unlabeled B Preprocessing and Feature Extraction and Feature Preprocessing . - . Data mode TeUS . RF TeUS Data RF TeUS . RF TeUS VS. VS. Preprocessing and Feature Extraction and Feature Preprocessing Joint Network Joint Classification Labeled B Transfer learning network Transfer B RF Benign Benign Cancer vs. - - MODE mimicking TeUS - mode TeUS Data mode TeUS T e US 30 TRANSFER LEARNING: RF VS. B-MODE TeUS DECISION MAKING MODEL Performance: Area under ROC Curve = 0.96 Run-time = 1.66 ± 0.32 second for 100 frames S. Azizi, et al., “Transfer learning from RF to B-mode temporal enhanced ultrasound features for prostate cancer detection,” Journal of Computer Assisted Radiology and Surgery: IPCAI’17 special issues, 2017. S. Azizi, et al., “Investigating deep recurrent neural networks for prostate cancer detection: analysis of temporal enhanced 31 ultrasound,” IEEE Transaction on Medical Imaging (TMI), 2018. MODEL INTERPRETATION Benign Gleason Pattern 3 Gleason Pattern 4 Layer 3: 6 hidden neurons Layer 2: 50 hidden neurons Layer 1: Back Propagation 100 hidden neurons Trained DBN Visible Layer 50 spectral features Absolute Difference Low-frequency components 32 MODEL INTERPRETATION Computer Scientist Medical Doctors Medical Physicists 33 MODEL INTERPRETATION Cancer Benign Cell Nuclei (Scatterers) Speckle Cell Nuclei Speckle Tissue response = f (Acoustic signal, Tissue microstructure,…) Hunt, J. W., et. al. “The subtleties of ultrasound