INRIA's Research Centers

Statistical computing on manifolds and data assimilation: from medical images to anatomical and physiological models

Centrale OMA/MA33 - MVA 2016-2017

X. Pennec Introduction to medical image acquisition and processing

Asclepios team 2004, route des Lucioles B.P. 93 06902 Sophia Antipolis Cedex

http://www-sop.inria.fr/asclepios 1

•4 INRIA’s Research Centers

INRIA Lille Nord Europe

INRIA Paris Rocquencourt •INRIA Nancy •Grand Est INRIA •INRIA Rennes Saclay •Bretagne Atlantique Île-de-France

2,800 scientists in INRIA Grenoble computer science Rhône-Alpes INRIA Bordeaux 1300 researchers and teaching- Sud-Ouest researchers •INRIA Sophia Antipolis 1000 Ph.D. students •Méditerranée 500 postdoctoral fellows

Statistical computing on manifolds and data assimilation: from medical images to anatomical and physiological models Medical Imaging – Module 2 - MVA 2016-2017

Fri Jan 6: Introduction to Medical Image Analysis and Processing [XP]

Fri Jan 13: Statistics on Riemannian manifolds and Lie groups [XP]

Fri Jan 20: Biomechanical Modelling and Simulation [HD]

Fri Jan 27: Diffeomorphic transformations for image registration [XP]

Fri Feb 17: Statistics on deformations for computational anatomy [XP]

Fri Feb 24: Manifold-valued image processing: the tensor example [XP]

Fri Mar 3: Data assimilation methods for physiological modeling [HD]

Fri Mar 10: Surgery Simulation [HD]

Fri Mar 17: Exam [Hervé Delingette, Xavier Pennec]

1 Course overview

The data of Computational Anatomy & Physiology

Image acquisition

Medical Image Analysis processing

1895

Roentgen

First Nobel prize in Physics in 1901

Todays’s Medical Imaging modalities

CT Scan MRI PET Ultrasound

•Source :T. Peters

2 Dynamic Images (4-D)

CT Scan MRI

Bio-signals

ECG Pressure Sensor

Temperature

Medical Imaging in clinical practice

TEACHING

Image DataBase

PREVENTION DIAGNOSIS THERAPY Pre-operative Patient Patient Patient

Post-operative robot

•CT •US •X-Ray •MRI •X-Ray •US •US •Video •NM •iMRI

Per-operative

3 Course overview

The data of Computational Anatomy & Physiology

Image acquisition

 Introduction

 Tomography

 Nuclear medicine

 MRI

Medical Image Analysis processing

Today MRI X-Scan

Density and X-ray structure of absorption protons density

PET / SPECT Ultrasounds

Density of Variations of Radioactive acoustic isotopes impedance

Different imaging modalities

 X-ray

 Magnetic resonance imaging

 anatomic, functional, angiographic, diffusion, spectroscopic, tagged

 Transmission Tomography (X Scan)

 Nuclear Medicine :

 Positron emission tomography (PET)

 Single photon emission tomography (SPECT)

 Ultrasonography

 Histological Imaging, confocal in-vivo microscopy, molecular imaging,…

4 Characteristics of medical images

Intensity values are related to physical tissue characteristics which in turn may relate to a physiological phenomenon

Anatomy

Physics

Physiology

Volumetric medical images

Very often medical images are volumetric Voxel Representation

I (x,y,z)

Example of volumetric images : CT-scan (Scanner)

Size: 512 x 512 x 128 Resolution: 0.5 x 0.5 x 1 mm

5 Volumetric images

T2 MRI

Course overview

The data of Computational Anatomy & Physiology

Image acquisition

 Introduction

 Tomography

 Nuclear medicine

 MRI

Medical Image Analysis processing

Principle of CT Imaging (1)

X-Ray X-Ray

6 Principle of CT Imaging (2)

Input X-Ray intensity : Ni

Measured Output X-Ray intensity : No

Ni N0

Exponential attenuation : Objective :

 measure (x) = absorption coefficient of X-ray

Computed Tomography

Principle : • Reconstruct n dimensional function (image) from projected data of (n–1) dimension

Radon Transform (1917) • “Two dimension and three dimension object can be reconstructed from the infinite set of projection data”.

Fourier Slice Theorem 1D Fourier Transform of Projected slice of 2D field (x,y)

is equal to 1D slice in the same direction of the 2D Fourier Transform of (x,y)

7 Basic principle of CT -Reconstruction of 2 dimensional image-

Projection Data curvilinear integral of absorption coefficient regarding Y

y y

object x x

X-ray tube Data Acquisition field Reconstruction field Simple Backprojection

Reconstruction Principle Backprojection based on inverse Radon Transform

1 , . 2

cos , sin

In practice use filtered back-projection to remove blur

Model Image Simple Filtered Backprojection Backprojection 23

Spiral (3D) CT X-ray tube and detectors rotate 360 deg Patient table is continuously moving Produce an helix of image projections 3D reconstruction

8 CT Scan Imaging Measure absorption coefficient of X-ray related to density of tissue

Invasive image modality (ionizing rays)

Absolute Hounsfield Unit water 1024 water

 HU(water)=0, HU(air)= -1024, HU(Bone)=175 to 3000

 Coded on signed 12 bits

Tomotensitometrie (Scanner X)

Course overview

 Introduction

 Image acquisition

 Tomography

 Nuclear medicine

 MRI

 Image processing

9 Principle of nuclear imaging

 Introduction into the patient body of a couple (radio-isotope / vector molecule)

Vector Molecule  Targets organ (drug, protein, blood cells…) Radio-isotope  Detection of the molecule

 Emission imaging : the targeted organ emits radioactivity  Reflect the metabolic function of the organ  Metabolic or functional imaging  Local relative concentration (relative)  Concentration evolution during time

Nuclear Medicine / radioactivity

 Nucleus (Rutherford) A= nucleon number Isobars A = constant Z = proton number Isotopes Z = constant N = neutron number Isotones N = constant

 Radioactivity (Curie)

Alpha: Helium nucleus Beta: 1/ electron - 2/ positron +  2 photons  (511 kev) Gamma: Photon

Nuclear Medicine

Density of radioactive tracers

10 Single photon gamma imaging

 Radio-isotopes Single photon emitters Technetium Tc 99m 6 h 140 kev Portative generator Iodine I 131 8 j 360 kev Reacteur (fission) Iodine I 123 13 h 159 kev Cyclotron (industry) Thallium Tl 201 73 h 80 kev Cyclotron (industry) Krypton (Kr 81 m), Gallium (Ga 67), Indium (In 111), Xenon (Xe 133, gaz)

Single photon gamma imaging

Heart (myocardium perfusion)  Stress/rest exam perfusion / perfusion  Healthy area perfusion / (hypo/non-)perfusion  Zone at risk (ischemia) (hypo/non-)perfusion / (hypo/non-)perfusion  Infarcted Area

Positron emission tomography (PET)

 Radio-isotopes Emission : positron (+)  Annihilation  2 photons of 511 kev at 180º Positron emitters Carbon 11C 20 mn cyclotron (medical) Nitrogen 13N 10 mn cyclotron (medical) Oxygen 15O 2 mn cyclotron (medical) Fluor 18F 112 mn cyclotron (medical)

 Physiological molecules 15 water  H2O glucose  fluoro-deoxyglucose (F18DG)

11 Positron emission tomography (PET)

PET Scan

https://www.youtube.com/watch?v=GHLBcCv4rqk

Course overview

 Introduction

 Image acquisition

 Tomography

 Nuclear medicine

 MRI

 Image processing

12 Magnetic resonance imaging

Density and structure of protons

Magnetic resonance imaging

Sagittal Coronal or Frontal Axial or Transverse

dimension: 256 x 256 x 128 résolution: 1x1x1.5 mm

MRI: a few dates

• 1946: MR phenomenon - Bloch et Purcell • 1952: Nobel prize - Bloch et Purcell • 1950-1970: development but no imaging • 1980: MRI feasibility • 1986 - …: real development

13 MRI: One modality with multiple sequences

• Anatomic MRI: T1, T2, DP weighted images • Angiographic MR • Functional MR: cognitive studies • Diffusion MR: brain connectivity • MR Spectroscopy

No absolute quantification

Magnetism at the molecular level

Electric charges in motion • magnetic momentum • Precession motion in a magnetic field

Bloch’s Equations

 Link between spin and magnetic momentum

μ Nuclear magnetic momentum  Gyromagnetic ratio

 Fundamental motion equation

S: Angular speed (spin) m: Momentum (mechanic)

 In a magnetic field ∧

14 Bloch’s Equations d Thus ∧    0  t cos (Lt )         sin ( t ) B0   0  t L       B0   z0 

Larmor’s frequency L   B0

Magnetism at the macroscopic level

P+1/2=0.5000049

B0=1.5 T P-1/2=0.4999951 49

Magnetism at the macroscopic level

1023 spins

15 Resonance

Magnetic Resonance / excitation

• Electro-magnetic field at Larmor’s frequency

L   B0

• Hydrogen protons enter into resonance

Flip of the macroscopic momentum M

Magnetic Resonance / excitation

16 Magnetic Resonance / relaxation

• Return to equilibrium / B0: time constant T1

dM z M z   M  B z  dt T1

• Spin dephasing: Time constant T2

dM x, y M x, y   M  B x, y  dt T2

Magnetic Resonance / relaxation T1 (ms) TISSUE T2(ms) 0.5 T 1.5 T Muscle 550 870 45 Heart 580 865 55 Liver 325 490 50 Kidney 495 650 60 Spleen 495 650 58 Fat 215 262 85 Brain, grey matter 655 920 100 Brain, white matter 540 785 90

540

MRI basics in video

https://www.youtube.com/watch?v=djAxjtN_7VE

17 MRI / frequency selection

 X encoding by frequency

 Y encoding by phase

 Several measures are necessary

Anatomical MRI

Proton density T1 T2

Angiographic MRI

•INFLOW • Phases

Maximum intensity projection (MIP)

18 Angiographic MRI

X-Scan / radiology Selective injection of a contrast agent in one artery

Tagged MRI

19 Avantages of MRI • Non-Invasive: MRI does not depend on potentially harmful ionizing radiation, as do standard x-ray and CT scans.

• MRI scans are not obstructed by bone, gas, or body waste, which can hinder other imaging techniques

• Can see through bone (the skull) and deliver high quality pictures of the brain's delicate soft tissue structures

• Images of organs and soft tissues

Drawbacks of MRI • Pacemakers not allowed

• Not suitable for claustrophobic persons

• Tremendous amount of noise during a scan

• MRI scans require patients to hold very still for extended periods of time. MRI exams can range in length from 20 minutes to 90 minutes or more.

• Orthopedic hardware (screws, plates, artificial joints) in the area of a scan can cause severe artifacts

• High cost

Course overview

 Introduction

 Image acquisition

 Tomography

 Nuclear medicine

 MRI

 Other types of images

 Image processing

20 Echography

Local variation of acoustic impedance

Echography

Gall Blader

Summary of Main Medical Imaging Modalities MRI CT-Scanner Measure Measure

Density and Density of structure of X-Ray Protons absorption

Nuclear Imaging Ultrasound Measure Measure

Density of Variations of injected Acoustic isotopes Impedance

21 New images 200 microns

Optical Coherent Tomography (OCT) Colon crypts Elastometry (MRI, US, etc.) Spectroscopic Imaging Terahertz Imaging Cardiac fibers Fibered Confocal Imagery in vivo etc.

microvessels Source : Mauna Kea Technologies

Medical Imaging Classification (1)

Dimensionality

X Ray IRM Gated-SPECT 4D (3D+T) 2D 3D

Medical Imaging Classification (2)

Anatomical vs functional Imagery

CT with MRI PET scan contrast agent

22 Multiparametric Images

MRI T1, T2 Angio MRI DTI fMRI

Course overview

The data of Computational Anatomy & Physiology

Image acquisition

Medical Image Analysis processing

 Medical Imaging

 Registration

 Segmentation

Medical Image Processing vs Computer Vision

Computer Vision Medical Image Processing

•Projective Geometry •Complex Image Formation •Occluding Objects •Large Datasets •Intensity depends on lighting •Patient Images

•Cartesian Geometry •Easy to acquire •Statistics Information

•Low dimensionality •Intensity links to physics •Patient Images

23 Why Medical Image Analysis?

Imprecise Precise Gets tired Never gets tired Inconsistent Consistent Expensive Cheap Capable to learn Stupid thing! Lots Knowledge Few Knowledge ......

General Trends in Medical Imaging

. More image modalities,

. Improved image qualities

. Global Aging : More patients, less doctors

. Digital Patient Record Infrastructure

A new concept

The Digital Patient

Medical in silico Geometry Computational Images Physics Models and Physiology & in vivo Bio‐signals Statistics Cognition Tools

Personnalisation

24 Types of Computational Models Geometrical / Anatomical

Reconstruction of Brain, skull and Ventricles from MRI

Types of Computational Models Geometrical / Anatomical Iconic / Appearance

T1w T1wgad T2w FA ADC

Different appearance of brain tumors in MR

Types of Computational Models Geometrical / Anatomical Iconic / Appearance Physical Models

Liver Biomechanics

25 Types of Computational Models Geometrical / Anatomical Iconic / Appearance Physical Models Physiological Models

Growth of a Glioblastoma in brain tissue

Types of Computational Models Geometrical / Anatomical Iconic / Appearance Physical Models Physiological Models Cognitive Models

Brain Activation In fMRI

Generic, Patient Specific, Statistical Models Statistical = Geometrical/Anatomical Variability across a Iconic / Appearance population Physical Models Physiological Models + Cognitive Models

Personalized = Generic = Specific to a given patient Represent whole population

26 Applications of Digital Patient

Image Computer-Aided Fusion Diagnosis

Therapy Therapy Simulation Planning

Therapy Guidance

Applications of Digital Patient

Therapy TEACHING

Simulation Image DataBase Therapy PREVENTION DIAGNOSIS THERAPY Therapy Planning Pre-operative Guidance Patient Patient Patient

Post-operative •robot Computer-Aided Computer-Aided Diagnosis Diagnosis Image •CT Fusion •US •X-Ray •MRI Image •X-Ray Image •Video •US •US Fusion •NM Fusion •iMRI

Per-operative

Classification of 3D image processing problems

 Segmentation (organs, lesions, activations,…)

 Registration (comparison, fusions)

 Motion analysis (cardiac imaging)

 Deformable models (Surgery simulation)

 Medical Robotics (image guided surgery, telesurgery…)

27 Course overview

The data of Computational Anatomy & Physiology

Image acquisition

Medical Image Analysis processing

 Medical Imaging

 Registration

 Segmentation

Medical Image Registration A dual problem

 Find the point y of image J which is corresponding (homologous) to each points x of image I.

 Determine the best transformation T that superimposes homologous points

yk xk

J I

Principal Applications

 Fusion of multimodal images

 Temporal evolution of a pathology

 Inter-subject comparisons

 Superposition of an atlas

 Augmented reality

28 Fusion of Multimodal Images

MRI PET X-scan

Histology US Visible Man

Temporal Evolution

coronal coronal

Time 1 Time 2

sagittal axial sagittal axial

Inter-Subject comparison

29 Registration to an Atlas

Voxel Man

Augmented reality Brigham & Women ’s Hospital

E. Grimson

Intuitive Example How to register these two images?

30 Feature-based/Image-based approach

Feature detection (here, points of high curvature) 2 Measure: for instance S(T)   T(xk )  y k k

Feature-based/Image-based approach

Feature detection (here, points of high curvature) 2 Measure: for instance S(T)   T(xk )  y k k

100

Feature-based/Image-based approach

No segmentation!  2 S(T )  (i  j ) Interpolation: j  J(T(x )) Measure: e.g.  k k k k k

 jk ik xk T(xk )

101

31 Feature-based/Image-based approach

No segmentation!  2 Measure: e.g. S(T )  (ik  jk ) k

T1 =Id

102

Feature-based/Image-based approach

No segmentation!  2 Partial overlap Measure: e.g. S(T )  (ik  jk ) k

103

Classes of Transformations T

 Rigid (displacement)

 Similarities

 Affine (projective for 2D / 3D)

 Polynomials

 Splines

 Free-form deformations

104

32 Classification of registration problems

Type of transformation

 Parametric Rigid (displacement), similarity, affine, projective

 Deformables Polynomial, spline, free-form deformations Type of acquisition

 Monomodal

 Multimodal Homology of observed objects

 Intra-subject (generally a well posed problem)

 Inter-subject (one-to-one correspondences, regularization ?)

107

Mathematical Formulation of registration (Brown, 1992)

Registration: Given two datasets (images) I and J, find the geometric transformation T that « best » aligns the physically homologous points (voxels)

Tˆ  arg max S(I, J,T ) TΤ

Optimization algorithm Similarity measure

Transformation space (rigid, affine, elastic,…)

108 108

Geometric methods Extract geometric features

 Invariant by the chosen transformations

 Points

 Segments

 Frames

Given two sets of features, registration consists in:

 Feature identification (similarity): Match homologous features

 Localization: Estimate the transformation T

Algorithms

 Interpretation trees

 Alignement

 Geometric Hashing

 ICP

109

33 Liver puncture guidance using augmented reality S. Nicolau, IRCAD / INRIA 3D (CT) / 2D (Video) registration

 2D-3D EM-ICP on fiducial markers

 Certified accuracy in real time Validation

 Bronze standard (no gold-standard)

 Phantom in the operating room (2 mm)

 10 Patient (passive mode): < 5mm (apnea)

[ S. Nicolau, PhD’04 MICCAI05, ECCV04,, IS4TM03, Comp. Anim. & Virtual World 2005 ]

110

Intensity-based methods

No geometric feature extraction

Advantages:

 Noisy images and/or low resolution

 Multimodal images

Drawbacks:

 All voxels must be taken into account

111

Classification of existing measures Assumed relationship

Affine Intensityof image I

Intensity of image J Adapted measures

Correlation coefficient

1  IJ (T )  (ik  I )( jk  J ) n I J k

112

34 Classification of existing measures Assumed relationship

Functional Intensityof image I

Intensity of image J Adapted measures Woods’ criterion (1993) Woods’ variants (Ardekani, 95; Alpert, 96; Nikou, 97) Correlation ratio (Roche, 98) Var[E(I | J (T ))]  2  Var(I)

113

Classification of existing measures Assumed relationship

Statistical Intensityof image I

Intensity of image J Adapted measures Joint Entropy (Hill, 95; Collignon, 95) Mutual Information (Collignon, 95; Viola, 95) Normalized Mutual Information (Studholme, 98) P(i, j) MI (I, J )  H (I)  H (J )  H (I, J )  P(i, j)log ij P(i)P( j)

114

Roboscope

MR coordinate system US coordinate system

TMR/US TMR0 / MR1

Robot / patient coordinate system MR 1 US 1 over time over MR 0 with surgical plan

US 2 Virtual MR 2 deformation

• Rigid matching tools for MR Brain • MR / US registration • US tracking of brain US n deformations Virtual MR n

115

35 MR-US Images

Pre - Operative MR Image Per - Operative US Image axial axial

coronal sagittal coronal sagittal Acquisition of images : L. & D. Auer, M. Rudolf

118

Ultrasound image / MRI registration

Elementary principles of US imagery

Irf (x)  Z.u (x)   (x) x Z Logarithmic u compression

I(x)  Alog Z.u (x)  B  (x)

US MRI Grad MRI 119

Ultrasound image / MRI registration

Assumption: acoustic impedance is a function of the MR signal (denote by J)

Z(x)  g(J (x))  Z(x)  g(J (x))J (x)

Relation between US and MR signals

I(x)  f J (x), J.u (x)   (x)

In practice, the influence of I(x)  f J(x), J(x)  (x) orientation is neglected

120

36 Bivariate Correlation Ratio

I function of 2 variables

I  f (J,| J |)

2 iterated stages Dependency Hypothesis  Robust polynomial approx. of f

 Estimation of T:

ˆ ˆ 2 T  arg min ( I(xk )  f (J (T (xk ), J (T (xk ) ) ) T  k A. Roche, X. Pennec, G. Malandain, and N. A. Rigid Registration of 3D Ultrasound with MR Images: a New Approach Combining Intensity and Gradient Information. IEEE Transactions on Medical Imaging, 20(10):1038--1049, October 2001.

121

Typical Registration Result with Bivariate Correlation Ratio

Pre - Operative MR Image Per - Operative US Image axial axial

Registered

coronal sagittal coronal sagittal Acquisition of images : L. & D. Auer, M. Rudolf

122

US Intensity MR Intensity and Gradient

123

37 Course overview

The data of Computational Anatomy & Physiology

Image acquisition

Medical Image Analysis processing

 Medical Imaging

 Registration

 Segmentation

124

Segmentation Task Huge number of available algorithms

Possible classifications :

 Generic vs task-oriented

 Bottom-up vs Top-down approaches

 Boundary vs Region approaches

 Supervised or non supervised

Image Segmentation Approaches

•Thresholding •Intensity Only

•Contrast Agent •Intensity- •in CT only •Lesions in CT / MR •Classification •gray matter •white matter •csf

•No Typical Shape •Typical Shape

- 126 •05/01/2017 •126

38 Image Segmentation Approaches

•Mathematical Morphology •Intensity and

•MRF (graph •connexity cuts, RW, watershed) •between regions •Vessels / tumors / bones /lesions •Machine •Grey / White matter Learning (RF, SVM, Boost,ML) in MR

•No Typical Shape •Typical Shape ••CamilleEzequiel Couprie, Geremia Leo, Olivier Grady, Clatz Laurent, Bjoern Najman H. Menze and, HuguesEnder Konukoglu Talbot , "Power, Antonio watershed: Criminisi, andA UnifyingNicholas Graph-BasedAyache. Spatial Optimization Decision Forests Framework" for MS ,Lesion IEEE Transactions Segmentation on Patternin Multi-Chann Analysisel and Magnetic Machine ResonanceIntelligence, Images vol. 33 .(7),NeuroImage pp 1384-1399, 57(2):378-90, (2011) July 2011 - 127

Image Segmentation Approaches

•Atlas based Segmentation

•Intensity and shape

•Heart

•Liver •Bones

•No Typical Shape •Typical Shape

- 128

Image Segmentation Approaches

•Atlas based Segmentation

•Intensity •Deformable Templates and shape

•Heart

•Liver •Bones

•No Typical Shape •Typical Shape

- 129