Computational Anatomy & Physiology M2 BioComp 2013-2014 http://www-sop.inria.fr/teams/asclepios/cours/CBB-2013-2014/ X. Pennec Medical Image registration Asclepios team 2004, route des Lucioles B.P. 93 06902 Sophia Antipolis Cedex http://www-sop.inria.fr/asclepios 1 Goals of Registration A dual problem Find the point y of image J which is corresponding (homologous) to each points x of image I. Determine the best transformation T that superimposes homologous points T yk xk J I 2 Principal Applications Fusion of multimodal images Temporal evolution of a pathology Inter-subject comparisons Superposition of an atlas Augmented reality 3 1 Fusion of Multimodal Images MRI PET X-scan Histology US Visible Man 4 Temporal Evolution coronal coronal Time 1 Time 2 sagittal axial sagittal axial 5 Inter-Subject comparison 6 2 Registration to an Atlas Voxel Man 7 Augmented reality Brigham & Women ’s Hospital E. Grimson 8 Classes of problems vs. applications Temporal evolution Intra Subject – Monomodal Multimodal image fusion Intra Subject - Multimodal Inter-subject comparison Inter Subject - Monomodal Superposition on an atlas Inter Subject - Multimodal Intra Subject: Rigid or deformable Inter Subject: deformable 9 3 Intuitive Example How to register these two images? 10 Feature-based/Image-based approach Feature detection (here, points of high curvature) 2 Measure: for instance S(T) T (xk ) y k k 11 Feature-based/Image-based approach Feature detection (here, points of high curvature) 2 Measure: for instance S(T) T (xk ) y k k 12 4 Feature-based/Image-based approach No segmentation! 2 S(T ) (i j ) Interpolation: j J(T(x )) Measure: e.g. k k k k k T jk ik xk T(xk ) 13 Feature-based/Image-based approach No segmentation! 2 Measure: e.g. S(T ) (ik jk ) k T1 =Id 14 Feature-based/Image-based approach No segmentation! 2 Partial overlap Measure: e.g. S(T ) (ik jk ) k T2 15 5 Classes of Transformations T Rigid (displacement) Similarities Affine (projective for 2D / 3D) Polynomials Splines Free-form deformations 16 Classes of Transformations T Rigid: T (x) Rx t Rotation and translation 6 parameters: (R: 3; t: 3) invariants: distances (isometry), frame orientation, curvatures, angles, straight lines Similarities: T (x) s.Rx t Add a global scale factor 7 parameters invariants: ratio of distances, orientation, angles, straight lines 17 Classes of Transformations T Affine: T (x) Bx t 3x3 matrix B 12 parameters: (B: 9; t: 3) invariants: straight lines, parallelism Quadratic: T (x) xt Ax Bx t Add a symmetric 3x3 matrix A 18 parameters (A: 6; B: 9; t: 3) invariants: do not preserve straight lines any more 18 6 Classes of Transformations T Splines: Local polynomials of degree d, with a global continuity of degree C(d-1). number of parameters: depend on the number of control points (knots) locally affine: simplified version Free form transformations: T (x) x u(x) a vector u(x) is attached to each point x parameters: at most 3 times the number of voxels regularization: constrain to homeomorphisms (diffeomorphisms) 19 Classification of registration problems Type of transformation Parametric Rigid (displacement), similarity, affine, projective Deformables Polynomial, spline, free-form deformations Type of acquisition Monomodal Multimodal Homology of observed objects Intra-subject (generally a well posed problem) Inter-subject (one-to-one correspondences, regularization ?) 20 Mathematical Formulation of registration (Brown, 1992) Registration: Given two datasets (images) I and J, find the geometric transformation T that « best » aligns the physically homologous points (voxels) Tˆ arg max S(I, J,T ) TΤ Optimization algorithm Similarity measure Transformation space (rigid, affine, elastic,…) 21 21 7 Geometric methods Extract geometric features Invariant by the chosen transformations Points Segments Frames Given two sets of features, registration consists in: Feature identification (similarity): Match homologous features Localization: Estimate the transformation T Algorithms Interpretation trees Alignement Geometric Hashing ICP 22 Artificial markers Stereotactic frame Invasive External markers Motion Short time use 23 Anatomical markers Find geometric invariants to characterize a small number of singular points on anatomical surfaces Generalization of edges and corner points to differentiable surfaces 24 8 Iterative closest point (ICP) One global criterion, C(A,T) dist(T(P), A(P )) 2 i i i alternatively minimized over * 2 Step 1: matches A Argmin dist(T(Pi ), A(Pi )) A i * 2 Step 2: transformation T Argmin dist(T(Pi ), A(Pi )) T i Positive and decreasing criterion: convergence Robustness w.r.t. outliers: robust distances 2 (x) || x || (standard mean ) 2 2 2 dist(x, y) (|| x y || ) (x) min || x || , (saturated mean ) (x) || x || (median) 25 1. Optimization for matches Qj Pi 26 2. Optimization for T Qj Pi 27 9 1-bis Optimization for matches Qj Pi 28 2-bis Optimization for T Qj Pi 29 Least squares Problem: We have n matched points ( x i , y i ) and we want to minimize: 2 C(A,t) yi A xi t C i 2(yi A xi t) 0 t i topt y A x In barycentric coordinates ~x x x i i ~ ~ 2 ~ C(A,topt ) C(A) yi Axi yi yi y i 30 10 Least squares affine transformation 1 ~ ~ 2 t t C(A) yi Axi Tr( yy ) Tr(A xx A ) 2Tr(A. yx ) N i 1 ~ ~ t 1 ~ ~ t t 1 ~ ~ t xx xi xi , yy yi yi , yx xy xi yi N i N i N i 1 Optimum: V C(A) lim C(A V ) C(A) 0 V 0 t t V C(A) 2Tr(A. xx .V ) Tr( yx .V ) 0 <A,B>=Tr(A.Bt) Froebenius dot product on the space of matrices: B,V 0 V B 0 (1) V C(A) 0 A.xx K A yx .xx 31 Augmented reality guided radio-frequency tumor ablation Current operative setup at IRCAD (Strasbourg, France) Per-operative CT “guidance” Respiratory gating Marker based 3D/2D rigid registration S. Nicolau, X.Pennec, A. Garcia,L. Soler, N. Ayache 32 Increase 3D/2D registration accuracy: A new Extended Projective Point Criterion M N 2 Pl(T*M ) ml Standard criterion: i i l1 i1 image space minimization (ISPPC) noise only on 2D data Complete statistical assumptions + ML estimation Gaussian noise on 2D and 3D data Hidden variables Mi (exact 3D positions) 2 ~ 2 M N l ~ l N P (T * M i ) mi M i M i 2 2 l1 i1 2 2D i1 2 3D 33 11 Registration for Augmented reality 34 Whole loop accuracy evaluation Stereoscopic HD video acquisition Left monitor with AR Right monitor with AR & needle tracking & needle tracking Real scene: phantom + needle 35 Whole loop accuracy evaluation Left monitor with AR Right monitor with AR & needle tracking & needle tracking Endoscopic control Real scene: phantom + needle 36 12 Whole loop accuracy evaluation 37 Liver puncture guidance using augmented reality S. Nicolau, IRCAD / INRIA 3D (CT) / 2D (Video) registration 2D-3D EM-ICP on fiducial markers Certified accuracy in real time Validation Bronze standard (no gold-standard) Phantom in the operating room (2 mm) 10 Patient (passive mode): < 5mm (apnea) [ S. Nicolau, PhD’04 MICCAI05, ECCV04,, IS4TM03, Comp. Anim. & Virtual World 2005 ] 39 Geometric methods Advantages automatic (no initialization, except ICP) fast (after the feature segmentation) robust to outliers Drawbacks Requires a high resolution and a low noise Invariant features: Rigid, monomodal, same subject Possible exception: skull images 41 13 Intensity-based methods No geometric feature extraction Advantages: Noisy images and/or low resolution Multimodal images Drawbacks: All voxels must be taken into account 42 Compare multi-modal images? Which similarity criterion ? Many available criteria: SSD, Correlation, Mutual Information...? Variable costs and performances Where is the optimum ? Maintz & Viergever, Survey of Registration Methods, Medical Image Analysis 1997 44 Course overview Introduction Multimodal Intensity-based Registration Classification of existing measures A general framework Deformable intensity-based Registration Some extensions Conclusion 45 14 Classification of existing measures Assumed relationship Intensity conservation Intensity o ’image I Intensity of image J Adapted measures Sum of Square Differences 2 S(T ) (ik jk ) Sum of Absolute value of Differences k Measures of intensity differences (Buzug, 97) Interpolation: jk J(T(xk )) 46 Classification of existing measures Assumed relationship Affine Intensity of image I Intensity of image J Adapted measures Correlation coefficient 1 IJ (T ) (ik I )( jk J ) n I J k 47 Classification of existing measures Assumed relationship Functional Intensity of image I Intensity of image J Adapted measures Woods’ criterion (1993) Woods’ variants (Ardekani, 95; Alpert, 96; Nikou, 97) Correlation ratio (Roche, 98) Var[E(I | J (T ))] 2 Var(I) 48 15 Classification of existing measures Assumed relationship Statistical Intensity of image I Intensity of image J Adapted measures Joint Entropy (Hill, 95; Collignon, 95) Mutual Information (Collignon, 95; Viola, 95) Normalized Mutual Information (Studholme, 98) P(i, j) MI (I, J ) H (I) H (J ) H (I, J ) P(i, j)log ij P(i)P( j) 49 Course overview Introduction Multimodal Intensity-based Registration Classification of existing measures A general framework Deformable intensity-based Registration Some extensions Conclusion 51 A general framework A. Roche proposed a unifying maximum likelihood framework Physical and statistical modeling of the image acquisition process Create a hierarchy of criteria A. Roche, G. Malandain and N.A. : Unifying maximum likelihood approaches in medical image registration. International Journal of Imaging Systems and Technology : Special Issue on 3D Imaging 11(1), 71-80, 2000. • Based on pioneer works of (Costa et al, 1993), (Viola, 1995), (Leventon & Grimson, 1998), (Bansal et al, 1998) 52 16 Principle of the method [A. Roche] Generative model of images – probabilistic – parametric T I Spatial Acquisition Image I S transfo.