9/7/17
Visual Computing in Medicine
Hans-Christian Hege
Int. Summer School 2017 on Deep Learning and Visual Data Analysis, Ostrava, 07.Sept.2017
Acknowledgements
Hans Lamecker Stefan Zachow Dagmar Kainmüller
Heiko Ramm Britta Weber Daniel Baum
’
1 9/7/17
Visual Computing
Image-Related Disciplines
Data Processing
Non-Visual Data
Image/Video Analysis Data Acquisition Computer Graphics Computer Vision Computer Animation Imaging VR, AR Data Visualization Visual Data
Image/Video Processing
2 9/7/17
Visual Computing source: Wikipedia
Visual computing = all computer science disciplines handling images and 3D models, i.e. computer graphics, image processing, visualization, computer vision, virtual and augmented reality, video processing, but also includes aspects of pattern recognition, human computer interaction, machine learning and digital libraries.
Core challenges are the acquisition, processing, analysis and rendering of visual information (mainly images and video).
Application areas include industrial quality control, medical image processing and visualization, surveying, robotics, multimedia systems, virtual heritage, special effects in movies and television, and computer games.
Images (mathematically)
Image:
• Domain : compact; 2D, 3D; or 2D+t, 3D+t ⇒ video often: • Range : grey values, color values, “hyperspectral” values often: • Practical computing: domain and range are discretized • Domain “sampled” (pixels, voxels) • Range “quantized”, e.g., • Function piecewise constant or smooth interpolant
3 9/7/17
Image Examples (I)
Grey value images
Cone beam image in dentistry Windowing 0 256 0 4095 window entire grey value range medium-width window small-width window depicted
⇒ poor image contrast ⇒ good overall contrast, ⇒ high contrast for covering soft-tissue bone and teeth and bone
Image Examples (II) 2D, 3D, …
X-ray projection Electron Tomography (resolution: 1,5 nm)
4 9/7/17
Image Examples (III)
static dynamic
MRI of a head (resolution: < 1mm) Real-time MRI of a human heart (resolution 2mm / 50 ms)
Image Examples (IV)
Color images
RGB image layers („color channels“) color image
Color space: 3 dimensional Pixel values = coordinates in color space
Light microscopy (in anatomic pathology)
5 9/7/17
Image Examples (IV)
Vector images
Flow mapping in cardiology
Images (IV)
Tensor images
Diffusion tensors (2D slice in 3D) visualized by ellipsoids Fiber tracks
6 9/7/17
Visual Computing in Medicine 3DExcite - Living Heart – Powerwall ( Dassault Systemes )
Visual Computing in Medicine
Acquisition, processing, analysis and rendering of all visual information (images/videos, 3D models) that arises during diagnosis, treatment and prevention.
Requires techniques from
• Image/video processing, pattern recognition, computer vision, machine learning • Computer graphics, visualization, computer vision, virtual and augmented reality, human computer interaction
7 9/7/17
Data in Medicine
Medicine: science & practice of the diagnosis, treatment and prevention of disease, whether physical or mental.
A medical treatment involves several processes, where the patient's health status is always the center of attention.
Health Status • Medical history ⇒ Lots of Data! • Directly accessible parameters • Data from imaging E.g. in cardiology nowadays per standard examination: about 2 GB • Data from laboratory
Medicine: Overall Process, Data Processing
(Computer Aided) (Computer Aided) (Computer Aided) (Computer Aided) Diagnosis Therapy Planning Treatment Healing Prediction
Initial State Changing State Changing State Predicted State
Anamnesis Measurement Monitoring Monitoring Simulation Measurement Simulation Simulation Visualization Visualizaion Simulation Visualization Visualization Visualization
8 9/7/17
Simulation
Personalized Simulations in Medicine
In diagnosis: reveal information that cannot be measured, e.g., compute the loading of a knee joint for different types of movement, given the individual anatomy and the body weight.
In therapy: plan and optimize treatments / surgeries e.g., enable surgeon to try different surgery techniques pre-operatively, given the current anatomical and physiological state
In prevention: make long-term predictions of health state, e.g., depending on different life styles, given the current health state
9 9/7/17
Personalized Simulations in Medicine
In chemical space mainly ordinary differential equations (ODEs) (systems biology) and stochastic differential equations (SDEs)
In space & time mainly partial differential equations (physical) ⇒ Finite Element methods ⇒ Anatomical models required
10 9/7/17
Anatomy Reconstruction - an exemplary task of visual computing
A Vision
11 9/7/17
Let these guys inspire our imagination… *)
Dr. „Bones“ McCoy Spock Physician on the star fleet spaceship Enterprise
*) Richard A. Robb, Biomedical Imaging, Visualization, and Analysis, Wiley-Liss, 2000
Dr. McCoy‘s Ultimate Healing Device
A very compact handheld device:
1. Point it to the body of the patient; then the complete anatomic, physiological, biochemical and metabolic status is instantaneously determined and displayed. ⇒ “tricorder”
2. Place it on the diseased or injured region; then complete cure is effected.
12 9/7/17
Our Less Ambitious Aim
We are satisfied with creation of a reconstructed 3D patient model suitable for planning, optimization and control of a therapy
è
Patient Models
13 9/7/17
Patient Models
population average patient-specific patient-specific postmortem postmortem antemortem (!) real model virtual model virtual model
anatomystuff.co.uk Teran, Sifakis, Blemker, … & Fedkiw Zachow, Muigg, Hildebrandt, Doleisch & Hege Creating and simulating skeletal muscle Visual exploration of nasal airflow. from the visible human data set. IEEE Trans Visual Comput Graph, 2009. IEEE Trans Visual Comput Graph, 2005.
Patient-Specific Models
Model
anatomical è geometrical (this talk)
functional è physical / mathematical / numerical (biomechanical, physiological, …)
14 9/7/17
Example Applications: Surgical Reconstruction How must bones be shaped? Sobotta ©
Lamecker, Zachow, Hege, Zöckler, Haberl: Zachow, Lamecker, Elsholtz & Stiller: Surgical Treatment of Craniosynostosis based on a Statistical Reconstruction of mandibular dysplasia using a statistical 3D Krebs 3D-Shape Model, CARS 2006 shape model. International Congress Series (Vol. 1281, pp. 1238-1243). Elsevier, 2006.
Example Applications: Implant Design / Fitting / Individual Manufacturing
fractures joint replacement Mccullochlaw.net
dentures
15 9/7/17
Example Applications: Functional Simulation
Tetrahedral heart model Beating heart
Zhang Y, Bajaj. C. Finite element meshing for cardiac analysis. ICES Technical Report 04-26, University of Texas, Austin, 2004.
Anatomical & Functional Models
Functional Models
systems of ordinary differential equations
systems of partial differential equations
⇒ Requirements for Anatomical Models
good geometric approximation of shapes
good numerical approximation of functions
16 9/7/17
Anatomy Reconstruction: On All Length Scales
Whole body MRI
Blood vessels in brain, MRI angiography (7T)
Trabecular structure of the radius bone micro-CT
Pyramidal neuron from the hippocampus, CFM
Golgi apparatus in cell, ET
Reconstruction Pipeline
17 9/7/17
Images ⇒ Models
Images • Anatomical information ⇒ anatomical models • Functional information ⇒ functional models
Anatomy reconstruction • Identification and segmentation of anatomical units • Creation of (discrete) geometrical shape representations
Anatomy Reconstruction Pipeline
Image Data
Image Filtering
Image Registration
Image Segmentation
Surface Reconstruction
Surface Improvement
Volumetric Grid Generation
18 9/7/17
Denoising
Improve results of later processing steps (edge detection) • Median filter • Anisotropic diffusion • Nonlocal means • Statistical methods • Machine learning
Image Denoising with ML
Use sparse coding combined with deep networks J. Xie, L. Xu, and E. Chen Image Denoising and Inpainting with pre-trained with a denoising auto-encoder Deep Neural Networks. In Advances in Neural Information Processing Systems, 341–349 (2012).
BLS-GSM = Bayes Least Squares with a Gaussian Scale-Mixture KSVD = Dictionary Learning via SVD SSDA = Stacked Sparse Denoising Auto-encoder
Feed-forward convolutional neural network K. Zhang, W. Zuo, Y. Chen, D. Meng, to separate the noise from the noisy image and L. Zhang. Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising. IEEE Trans Imag Proc, 26:7 (2017), 3142 - 3155
Not yet applied to medical image data!
19 9/7/17
Image Registration
Necessary when • Multimodal imaging is used • Acquisitions made at different times • Imaged organs are moving
Overlay of 2 images in checkerboard pattern
unregistered registered
Image Registration
Ingredients: • Set of allowed spatial transformations: rigid, affine, free, … • Similarity measure, based on corresponding features • Optimization procedure
20 9/7/17
Image Registration Literature:
J Modersitzki: Numerical Methods for Image Registration, Oxford University Press, 2004 S Klein, M Staring, K Murphy, MA Viergever, JPW Pluim, Elastix: a toolbox for intensity- S Henn, K Witsch: Iterative Multigrid based medical image registrations, IEEE Trans Regularization Techniques For Image Matching, Med Imag 29:1, (2010) 196-205 SIAM J. Sci. Comput., 23:4, (2001), 1077-1093
Software: • Elastix toolbox, http://elastix.isi.uu.nl • FAIR, www.mic.uni-luebeck.de/people/jan-modersitzki • ITK - Segmentation & Registration Toolkit, https://itk.org
Image Registration using ML S. Wang, M. Kim, G. Wu, D. Shen • Unsupervised Deep Learning Scalable High Performance Image Registration Framework by • Convolutional Stacked Auto-Encoder (CSAE) Unsupervised Deep Feature • Training: 3D image patches (∼ 104; 21x21x21) sampled Representations Learning from ∼ 107 voxels In: SK Zhou, H Greenspan, D Shen, Deep Learning for Medical Image • Multilayer encoder network transfers high-dim 3D patches Analysis, Elsevier, 2017, pp. 245-269 to low-dim feature representations • Decoder network recovers 3D image patches from the learned representations by acting as feedback to refine inferences in the encoder network • Learned feature representations steer the correspondence • Many further studies are detection in a general (sophisticated) image registration necessary (other image framework modalities, anatomical regions, …) Applied to 7T-MRI brain images: • Strategies to deal with Consistently better results than state-of-the-art methods! (hyper-)parameters
21 9/7/17
Image Segmentation
Hard segmentation
● Classification of pixels (voxels) ● Unique assignments
Soft segmentation
● Non-unique assignments ● Probabilistic class memberships
Problem of Image Segmentation (1)
In order to determine pixel/voxel labels correctly, often
• various non-obvious image properties are necessary, including non-local ones
• image information does not suffice; additionally previous knowledge is necessary (⇒ Bayesian methods)
Additionally, object definitions are often pure conventions and sometimes they depend on the task.
22 9/7/17
Problem of Image Segmentation (2)
Problem of image segmentation solved excellently by the evolution of biological visual systems. ⇒ develop computer-based vision inspired by biological vision
Approaches (two extremes): • Construct algorithms, which operate directly on image features • Develop learnable algorithms (e.g., artificial neural networks), which implement principles of biological intelligenc
Image Segmentation Methods
• Thresholding: global, adaptive (e.g. Otsu‘s method)
• Clustering methods: partition image into k clusters (k-means, histogram- based clustering) • Edge detection methods: find edges & connect edge segments
• Region growing methods: start with a seed and grow according to some similarity criterion
• Watershed methods: image gradient magnitude = topographic surface; water placed at any pixels flows downhill; pixels draining to a common minimum form a segment
23 9/7/17
Image Segmentation Methods
• Graph partitioning methods: pixels or groups of pixels are associated with nodes of a graph; edge weights define the (dis)similarity between the neighborhood pixels; partitioning of the graph according to various criteria: normalized cuts, random walker, ... • PDE-based methods: e.g. level set method: contour = 0-level of scalar function; start with a seed contour and propagate it until it reaches the object boundary • Variational segmentation: energy functionals are minimized, e.g. of Mumford-Shah type • Model-based segmentation: assumption that structures of interest/organs have a repetitive shape; probabilistic model explaining the variation of the shape; use this model as prior when segmenting
Interactive Image Segmentation
• Fast, interactive algorithms (intelligent scissors, graph cuts, …)
• Utilize human visual system
Segmentation Editor in Amira
24 9/7/17
Automatic Image Segmentation
• Utilization of a-priori information (shape of organ) • Training data sets (sufficiently large number) • Statistical 3D Shape Models (SSM) • Model-based segmentation
Automatic Image Segmentation • Roughly place shape template into the image data - using Generalized Hough transform • Iteratively adapt the shape model to the image data - guided by grey value profiles
25 9/7/17
Automatic Image Segmentation
• ‘Intensity profiles’ guide the deformation process: • At each vertex of surface mesh, commonly along surface normals, intensities are sampled along line segments • On each profile, a cost function is derived from image data for a number of equidistant sampling points • Minimum cost determines (locally optimal) new position for the respective vertex
• Patient-specific characteristics not contained in SSM limit accuracy of segmentations
• Accuracy can be increased by subsequent free form deformations
Seim H, Kainmueller D, Heller M, Lamecker H, Zachow S, Hege H-C: Automatic Segmentation of the Pelvic Bones from CT Data Based on a Statistical Shape Model. Proc VCBM, pp. 93-100, 2007
Automatic Image Segmentation: Accurate Joint Segmentation
Problem:
Solution: simultaneously segment multiple adjacent objects & incorporate knowledge about their spatial relationship
26 9/7/17
Automatic Image Segmentation: Accurate Joint Segmentation
D. Kainmueller, H. Lamecker, S. Zachow, H-C. Hege. Coupling Deformable Models for Multi-object Segmentation. ISBMS, LNCS vol. 5104, pp. 69-78. distal femur proximal tibia Springer, 2008.
acetabulum proximal femur
Model initialization with Graph cuts optimization statistical shape models
Automatic Image Segmentation Using CNNs
• Huge progress during past years; often superior to previous state-of-the-art techniques • Big advantage: Methods require no feature engineering; adapt flexibly to the problem Part 3: Medical Image Segmentation, In: they are trained for Zhou SK, Greenspan H and Shen D (eds.) Deep Learning for Medical Image Analysis, • Field much too large, to be presented here Academic Press, 2017, pp. 177-242 • Current methods applicable to medical images • Depend very strongly on image modality & anatomical region • Require parameters to be tuned • More generic methods are needed
27 9/7/17
Grid Generation
E.g. for FE-simulation: surface and volume mesh
• Unstructured • Multi-material (⇒ generalized MC) • Locally adaptive resolution • Control on element quality
Rinside / Routside Quality = (Rinside / Routside )ideal
Improvement of Surface Meshes Details, Complexity and Mesh Quality
high res
simplified
optimized
28 9/7/17
Improvement of Surface Meshes • Consistent remeshing of Zilske M, Lamecker H, Zachow S: • Non-manifold triangle meshes Remeshing of non-manifold surfaces Eurographics 2008, pp. 211-214 • With user-defined feature lines • Result mesh • With high regularity and triangle quality • Preserved geometry & topology of the • input mesh • feature skeleton • Based on local operations only
Improvement of Surface Meshes
29 9/7/17
Improvement of Surface Meshes
Improvement of Surface Meshes
Remeshing of non-manifold triangulations
Rinside / Routside Quality = (Rinside / Routside )ideal
30 9/7/17
Generation of Volumetric Meshes
Automatic control of element size using a “sizing field”
Lamecker H, Mansi T, Relan J, Billet F, Sermesant M, Ayache N, Delingette H: Adaptive Tetrahedral Meshing for Personalized Cardiac Simulations Proc. MICCAI Workshop on Cardiovascular Interventional Imaging and Biophysical Modelling (CI2BM), pp. 149-158, 2009.
Generation of Volumetric Meshes Mesh generation by advancing front method and with consideration of the element size
Consistent handling of heterogeneous inputs, including CAD data:
Kahnt M, Ramm H, Lamecker H, Zachow S: Feature-Preserving, Multi-Material Mesh Generation using Hierarchical Oracles. MICCAI Workshop on Mesh Processing in Medical Image Analysis, LNCS 7599, pp. 101-111, 2012.
31 9/7/17
Generation of Volumetric Meshes Sinuses (for simulation of nasal airflow):
Zachow, S ; P. Muigg ; Th. Hildebrandt ; H. Doleisch ; H.-C. Hege: Visual Analysis of Nasal Airflow. IEEE TVCG, 15:8, pp. 1407-1414 (2009)
Further Applications
32 9/7/17
Surgery Planning Orthopedic surgery planning: decision support based on individual biomechanics
in cooperation with
Clinical Research with Virtual Patients Premise: validated numerical model • Knee replacement • Study with 328 automatically cement-less implanted tibiae • Result: inter-patient variability of bone strain at the implant-bone interface (over a full gait cycle)
Galloway F, Seim H, Kahnt M, Nair P, Worsley P, Taylor M: A Large Scale Finite Element Study of an Osseointegrated Cementless Tibial Tray; J Bone Joint Surg Br, 2012 in cooperation with
33 9/7/17
Biomechanical Simulation
Conclusions
34 9/7/17
Visual Computing in Medicine
• We are on the way to the digital patient
• Problem: Utilize the available data intelligently along the whole medical procedure
• Extract essential information automatically
• Visualize essential information
• Chance: Personalized medicine
• Requires patient-specific simulation
• Requires patient-specific anatomical models
Creation of Patient-Specific Anatomical Models
• Two major problem areas in practice: • Image segmentation • Meshing • Within past 5 years: more progress than probably most experts expected, especially in the field of image segmentation, due to machine learning • Current state: • “Everything” can be segmented • But typically this requires • Design of a specific algorithm • At least: adaptation of (possibly many) free parameters • Large training data sets
35 9/7/17
Research Topics • Learnable segmentation algorithms: Find systematic ways for construction of neural networks, particularly for segmentation & registration
• Track uncertainties explicitely; deliver error bounds
• Extend anatomical models to functional models (⇒ biophysical quantitative imaging)
• Improve numerical simulation methods
• Develop simulation-based decision support systems
• Simplify, simplify, simplify…
Medical Tricorder: Science Fiction becomes Reality
The Qualcomm Tricorder X PRIZE: A $10 million competition to bring healthcare to the palm of your hand. 300 teams participated, 2 winners Award ceremony on April 12, 2017
See https://tricorder.xprize.org
36 9/7/17
Textbooks
B. Preim, C. Botha I.N. Bankman (ed.) Visual Computing in Medicine Handbook of Medical Image Processing 2nd ed., Morgan Kaufman, 2014 Academic Press, 2009 812 pp 984 pp
37 9/7/17
Y.J. Zhang Geometric Modeling and Mesh Generation from Scanned Images Academic Press, 2017 458 pp
S.K. Zhou, H. Greenspan, D. Shen (eds.) S.K. Zhou (ed.) Deep Learning for Medical Image Medical Image Recognition, Segmentation Analysis and Parsing Academic Press, 2017 Academic Press 2015 458 pp 542 pp
38 9/7/17
• www.zib.de/visual
• www.zib.de/hege
39