Medical Image Registration Computational Anatomy & Physiology

Home , Image registration

Computational Anatomy & Physiology M2 BioComp 2013-2014 http://www-sop.inria.fr/teams/asclepios/cours/CBB-2013-2014/

X. Pennec

Medical Image registration

Asclepios team 2004, route des Lucioles B.P. 93 06902 Sophia Antipolis Cedex

http://www-sop.inria.fr/asclepios 1

Goals of Registration A dual problem

 Find the point y of image J which is corresponding (homologous) to each points x of image I.

 Determine the best transformation T that superimposes homologous points

yk xk

J I

Principal Applications

 Fusion of multimodal images

 Temporal evolution of a pathology

 Inter-subject comparisons

 Superposition of an atlas

 Augmented reality

1 Fusion of Multimodal Images

MRI PET X-scan

Histology US Visible Man

Temporal Evolution

coronal coronal

Time 1 Time 2

sagittal axial sagittal axial

Inter-Subject comparison

2 Registration to an Atlas

Voxel Man

Augmented reality Brigham & Women ’s Hospital

E. Grimson

Classes of problems vs. applications

Temporal evolution Intra Subject – Monomodal

Multimodal image fusion Intra Subject - Multimodal

Inter-subject comparison Inter Subject - Monomodal

Superposition on an atlas Inter Subject - Multimodal

Intra Subject: Rigid or deformable

Inter Subject: deformable

3 Intuitive Example How to register these two images?

Feature-based/Image-based approach

Feature detection (here, points of high curvature) 2 Measure: for instance S(T)   T (xk )  y k k

Feature-based/Image-based approach

Feature detection (here, points of high curvature) 2 Measure: for instance S(T)   T (xk )  y k k

4 Feature-based/Image-based approach

No segmentation!  2 S(T )  (i  j ) Interpolation: j  J(T(x )) Measure: e.g.  k k k k k

 jk ik xk T(xk )

Feature-based/Image-based approach

No segmentation!  2 Measure: e.g. S(T )  (ik  jk ) k

T1 =Id

Feature-based/Image-based approach

No segmentation!  2 Partial overlap Measure: e.g. S(T )  (ik  jk ) k

5 Classes of Transformations T

 Rigid (displacement)

 Similarities

 Affine (projective for 2D / 3D)

 Polynomials

 Splines

 Free-form deformations

Classes of Transformations T Rigid: T (x)  Rx  t

 Rotation and translation

 6 parameters: (R: 3; t: 3)

 invariants: distances (isometry), frame orientation, curvatures, angles, straight lines

Similarities: T (x)  s.Rx  t

 Add a global scale factor

 7 parameters

 invariants: ratio of distances, orientation, angles, straight lines

Classes of Transformations T

Affine: T (x)  Bx  t  3x3 matrix B

 12 parameters: (B: 9; t: 3)

 invariants: straight lines, parallelism

Quadratic: T (x)  xt Ax  Bx  t  Add a symmetric 3x3 matrix A

 18 parameters (A: 6; B: 9; t: 3)

 invariants: do not preserve straight lines any more

6 Classes of Transformations T Splines:

 Local polynomials of degree d, with a global continuity of degree C(d-1).

 number of parameters: depend on the number of control points (knots)

 locally affine: simplified version

Free form transformations: T (x)  x  u(x)

 a vector u(x) is attached to each point x

 parameters: at most 3 times the number of voxels

 regularization: constrain to homeomorphisms (diffeomorphisms)

Classification of registration problems

Type of transformation

 Parametric Rigid (displacement), similarity, affine, projective

 Deformables Polynomial, spline, free-form deformations Type of acquisition

 Monomodal

 Multimodal Homology of observed objects

 Intra-subject (generally a well posed problem)

 Inter-subject (one-to-one correspondences, regularization ?)

Mathematical Formulation of registration (Brown, 1992)

Registration: Given two datasets (images) I and J, find the geometric transformation T that « best » aligns the physically homologous points (voxels)

Tˆ  arg max S(I, J,T ) TΤ

Optimization algorithm Similarity measure

Transformation space (rigid, affine, elastic,…)

21 21

7 Geometric methods Extract geometric features

 Invariant by the chosen transformations

 Points

 Segments

 Frames

Given two sets of features, registration consists in:

 Feature identification (similarity): Match homologous features

 Localization: Estimate the transformation T

Algorithms

 Interpretation trees

 Alignement

 Geometric Hashing

 ICP

Artificial markers

Stereotactic frame

 Invasive

 External markers

 Motion  Short time use

Anatomical markers

Find geometric invariants to characterize a small number of singular points on anatomical surfaces

Generalization of edges and corner points to differentiable surfaces

8 Iterative closest point (ICP)

One global criterion, C(A,T)  dist(T(P), A(P )) 2  i i i alternatively minimized over

* 2  Step 1: matches A  Argmin dist(T(Pi ), A(Pi )) A i * 2  Step 2: transformation T  Argmin dist(T(Pi ), A(Pi )) T i Positive and decreasing criterion: convergence Robustness w.r.t. outliers: robust distances

2 (x) || x || (standard mean ) 2 2 2 dist(x, y)  (|| x  y || ) (x)  min || x || ,  (saturated mean ) (x) || x || (median)

1. Optimization for matches

Qj Pi

2. Optimization for T

Qj Pi

9 1-bis Optimization for matches

Qj Pi

2-bis Optimization for T

Qj Pi

Least squares

Problem:

 We have n matched points ( x i , y i ) and we want to minimize: 2 C(A,t)   yi  A xi t C i  2(yi  A xi  t)  0 t i

topt  y  A x

 In barycentric coordinates ~x  x  x i i ~ ~ 2 ~ C(A,topt )  C(A)   yi  Axi yi  yi  y i

10 Least squares affine transformation

1 ~ ~ 2 t t C(A)   yi  Axi  Tr( yy ) Tr(A xx A )  2Tr(A. yx ) N i

1 ~ ~ t 1 ~ ~ t t 1 ~ ~ t  xx   xi xi ,  yy   yi yi ,  yx   xy   xi yi N i N i N i 1 Optimum: V C(A)  lim C(A  V )  C(A)  0 V  0  t t V C(A)  2Tr(A. xx .V ) Tr( yx .V )  0 =Tr(A.Bt) Froebenius dot product on the space of matrices:  B,V  0 V  B  0

(1) V C(A)  0  A.xx  K  A   yx .xx

Augmented reality guided radio-frequency tumor ablation Current operative setup at IRCAD (Strasbourg, France)

 Per-operative CT “guidance”

 Respiratory gating Marker based 3D/2D rigid registration

S. Nicolau, X.Pennec, A. Garcia,L. Soler, N. Ayache

Increase 3D/2D registration accuracy: A new Extended Projective Point Criterion

M N 2 Pl(T*M ) ml Standard criterion:   i  i l1 i1

 image space minimization (ISPPC)

 noise only on 2D data

Complete statistical assumptions + ML estimation

 Gaussian noise on 2D and 3D data

 Hidden variables Mi (exact 3D positions) 2 ~ 2 M N l ~ l N P (T * M i )  mi M i  M i   2   2 l1 i1 2 2D i1 2 3D

11 Registration for Augmented reality

Whole loop accuracy evaluation Stereoscopic HD video acquisition Left monitor with AR Right monitor with AR & needle tracking & needle tracking

Real scene: phantom + needle

Whole loop accuracy evaluation

Left monitor with AR Right monitor with AR & needle tracking & needle tracking

Endoscopic control

Real scene: phantom + needle

12 Whole loop accuracy evaluation

Liver puncture guidance using augmented reality S. Nicolau, IRCAD / INRIA 3D (CT) / 2D (Video) registration

 2D-3D EM-ICP on fiducial markers

 Certified accuracy in real time Validation

 Bronze standard (no gold-standard)

 Phantom in the operating room (2 mm)

 10 Patient (passive mode): < 5mm (apnea)

[ S. Nicolau, PhD’04 MICCAI05, ECCV04,, IS4TM03, Comp. Anim. & Virtual World 2005 ]

Geometric methods

Advantages

 automatic (no initialization, except ICP)

 fast (after the feature segmentation)

 robust to outliers

Drawbacks

 Requires a high resolution and a low noise

 Invariant features: Rigid, monomodal, same subject

 Possible exception: skull images

13 Intensity-based methods

No geometric feature extraction

Advantages:

 Noisy images and/or low resolution

 Multimodal images

Drawbacks:

 All voxels must be taken into account

Compare multi-modal images?

Which similarity criterion ?

 Many available criteria:

 SSD, Correlation, Mutual Information...?

 Variable costs and performances

 Where is the optimum ?

Maintz & Viergever, Survey of Registration Methods, Medical Image Analysis 1997

Course overview

Introduction

Multimodal Intensity-based Registration

 Classification of existing measures

 A general framework

Deformable intensity-based Registration

Some extensions

Conclusion

14 Classification of existing measures Assumed relationship

Intensity conservation Intensity o ’image I

Intensity of image J Adapted measures

Sum of Square Differences  2 S(T )  (ik  jk ) Sum of Absolute value of Differences k Measures of intensity differences (Buzug, 97)

 Interpolation: jk  J(T(xk ))

Classification of existing measures Assumed relationship

Affine Intensity of image I

Intensity of image J Adapted measures

Correlation coefficient

1  IJ (T )   (ik  I )( jk  J ) n I J k

Classification of existing measures Assumed relationship

Functional Intensity of image I

Intensity of image J Adapted measures Woods’ criterion (1993) Woods’ variants (Ardekani, 95; Alpert, 96; Nikou, 97) Correlation ratio (Roche, 98) Var[E(I | J (T ))]  2  Var(I)

15 Classification of existing measures Assumed relationship

Statistical Intensity of image I

Intensity of image J Adapted measures Joint Entropy (Hill, 95; Collignon, 95) Mutual Information (Collignon, 95; Viola, 95) Normalized Mutual Information (Studholme, 98) P(i, j) MI (I, J )  H (I)  H (J )  H (I, J )  P(i, j)log ij P(i)P( j)

Course overview

Introduction

Multimodal Intensity-based Registration

 Classification of existing measures

 A general framework

Deformable intensity-based Registration

Some extensions

Conclusion

A general framework

 A. Roche proposed a unifying maximum likelihood framework

 Physical and statistical modeling of the image acquisition process

 Create a hierarchy of criteria

A. Roche, G. Malandain and N.A. : Unifying maximum likelihood approaches in medical image registration. International Journal of Imaging Systems and Technology : Special Issue on 3D Imaging 11(1), 71-80, 2000.

• Based on pioneer works of (Costa et al, 1993), (Viola, 1995), (Leventon & Grimson, 1998), (Bansal et al, 1998)

16 Principle of the method [A. Roche]

Generative model of images

– probabilistic – parametric T  I Spatial Acquisition Image I  S transfo. model Anatomical ScèneScene S prior  J Acquisition Image J model Maximum likelihood Inference

max P(I, J |T, I , J , S ) T , I , J ,S Auxiliary variables: q

Generic model

Scene: discrete random field (segmentation)

Assumptions: – Spatial independence – Stationarity

 P(S)   (sl ) l

Generic model

Images: noisy measures of the scene

i  f (s )   s  s(T (x ))  k k k k k  jl  g(sl ) l

Assumptions on  and : – white (spatial indep.) – Stationary  P(I, J | S)  G (i  f (s ))   ε k k – Gaussian k  G ( j  g(s )) – Additive   η l l – Independent of each other l

17 Likelihood function

Likelihood = joint law of images P(I, J )   P(I, J | S)P(S)dS

 L(T, ) with   ( , f , g, ε , η )

Log likelihood

P (i , j  ) log L(T, )  log  k k  logP (i )  logP ( j )     k   l kA P (ik )P ( jk ) k l (T, ) ( ) K with P (i, j)   G (i  f )G ( j  g )   p  ε p  η p p1

Example: X-Scan / MRI rigid registration

The estimation of  allows the a posteriori estimation of the scene

Limitations

 In general, the maximization w.r.t.  is costly (EM segmentation algorithm)

 With additional assumptions on , one can approach the solution analytically

18 Particular case 1 Assumptions

 f  g    ik  jk   k | |  |  | T  Spatial Gaussian Image J transfo noise Image I

Maximum Likelihood

 c  2  max L(T, )  nlog (ik  jk )   Ecte  2   n k  SSD

Particular case 2 Assumptions

g injective    ik  f ( jk )   k , f constant by part | |  |  | T f  Spatial Intensity Gaussian Image J Transfo. function noise Image I

Maximum Likelihood

 c  2  max L(T, )  nlog min (ik  f ( jk ))   Ecte  2 f   n k  Squared fit error

Particular case 2

Limit case: correlation ratio (CR)

Squared fit error

 2 min (ik  f ( jk )) f   2 (T)  k I|J n Var(I) Different normalization of the likelihood criteria

Equivalence ML / CR

19 Generalization of the Correlation Ratio

Robust metric

min  (I  f (J T ))  gene (T)  f  : M-estimator of scale I|J min (I  ) (Rousseeuw, 87)  cte

Space in which one seeks the function f

polynomials, splines, ... Intensity in in Intensity I Intensity in J

Particular case 3

Assumptions

Known 

Spatial Gaussian transfo. noise Image I Discrete field Scene S

Gaussian Image J noise Unknown number of classes Known 

Particular case 3

Maximum Likelihood

1 Pˆ(i, j)  G (i  i )G ( j  j ) (Parzen)   ε k  η k n k

Pˆ(i , j ) max L(T, P)  log k k  corrective terms P  ˆ ˆ  k P(ik )P( jk ) n  Mutual Informatio n

20 Choice of a criterion

 Choosing a criterion imposes to deeply understand the physical image acquisition process

 Whenever several models are known, choosing the one with the smallest number of degrees of freedom increase the robustness of the approach.

Roboscope

MR coordinate system US coordinate system

TMR/US TMR0 / MR1

Robot / patient coordinate system MR 1 US 1 over time over MR 0 with surgical plan

US 2 Virtual MR 2 deformation

• Rigid matching tools for MR Brain • MR / US registration • US tracking of brain US n deformations Virtual MR n

Manipulator

Steady Hand Motion Compensation Active Motion Constraints

QuickTime™ and a Intel Indeo® Video 5.0 decompressor are needed to see this picture.

Courtesy B. Davies & S. Starkie

21 Manipulator

Steady Hand Motion Compensation Active Motion Constraints

QuickTime™ and a Intel Indeo® Video 5.0 decompressor are needed to see this picture.

Courtesy B. Davies & S. Starkie

MR-US Images

Pre - Operative MR Image Per - Operative US Image axial axial

coronal sagittal coronal sagittal Acquisition of images : L. & D. Auer, M. Rudolf

Ultrasound image / MRI registration

Elementary principles of US imagery

Irf (x)  Z.u (x)  (x) x Z Logarithmic u compression

I(x)  Alog Z.u (x)  B  (x)

US MRI Grad MRI 71

22 Ultrasound image / MRI registration

Assumption: acoustic impedance is a function of the MR signal (denote by J)

Z(x)  g(J (x))  Z(x)  g(J (x))J (x)

Relation between US and MR signals

I(x)  f J (x), J.u (x)  (x)

In practice, the influence of I(x)  f J(x), J (x)  (x) orientation is neglected

Bivariate Correlation Ratio

I function of 2 variables

I  f (J,| J |)

2 iterated stages Dependency Hypothesis  Robust polynomial approx. of f

 Estimation of T:

ˆ ˆ 2 T  arg min ( I(xk )  f (J (T (xk ), J (T (xk ) ) ) T  k A. Roche, X. Pennec, G. Malandain, and N. A. Rigid Registration of 3D Ultrasound with MR Images: a New Approach Combining Intensity and Gradient Information. IEEE Transactions on Medical Imaging, 20(10):1038--1049, October 2001.

Typical Registration Result with Bivariate Correlation Ratio

Pre - Operative MR Image Per - Operative US Image axial axial

Registered

coronal sagittal coronal sagittal Acquisition of images : L. & D. Auer, M. Rudolf

23 US Intensity MR Intensity and Gradient