Metric Learning for Diffeomorphic Image Registration

Home , Image (mathematics)

Metric learning for diﬀeomorphic image registration.

Fran¸cois-Xavier Vialard

Metric learning for diﬀeomorphic image registration.

Fran¸cois-XavierVialard

Universit´eParis-Est Marne-la-Vall´ee joint work with M. Niethammer and R. Kwitt.

IHP, March 2019. Metric learning for diﬀeomorphic image Outline registration.

Fran¸cois-Xavier Vialard

1 Introduction to diﬀeomorphisms group and Riemannian tools

2 Choice of the metric

3 Spatially dependent metrics

4 Metric learning

5 SVF metric learning Metric learning for diﬀeomorphic image Example of problems of interest registration.

Fran¸cois-Xavier Vialard

Introduction to diﬀeomorphisms group and Riemannian tools

Choice of the metric

Spatially dependent metrics

Metric learning

SVF metric learning Given two shapes, find a diffeomorphism of R3 that maps one shape onto the other Metric learning for diffeomorphic image Example of problems of interest registration.

3 Fran¸cois-Xavier Given two shapes, ﬁnd a diﬀeomorphism of R that maps one Vialard

shape onto the other Introduction to diffeomorphisms group and Riemannian tools Different data types and different way of representing them. Choice of the metric

Spatially dependent metrics

Metric learning

SVF metric learning

Figure– Two slices of 3D brain images of the same subject at different ages Metric learning for diffeomorphic image Example of problems of interest registration. 3 Given two shapes, find a diffeomorphism of R that maps one Fran¸cois-Xavier shape onto the other Vialard Introduction to diffeomorphisms group and Riemannian tools

Choice of the metric

Spatially dependent metrics

Metric learning

SVF metric learning

Deformation by a diﬀeomorphism

Figure– Diﬀeomorphic deformation of the image Metric learning for diﬀeomorphic image Variety of shapes registration.

Fran¸cois-Xavier Vialard

Introduction to diﬀeomorphisms group and Riemannian tools

Choice of the metric

Spatially dependent metrics

Metric learning

SVF metric learning

Figure– Diﬀerent anatomical structures extracted from MRI data Metric learning for diﬀeomorphic image Variety of shapes registration.

Fran¸cois-Xavier Vialard

Introduction to diﬀeomorphisms group and Riemannian tools

Choice of the metric

Spatially dependent metrics

Metric learning

SVF metric learning

Figure– Different anatomical structures extracted from MRI data Metric learning for diffeomorphic image A Riemannian approach to diffeomorphic registration.

Fran¸cois-Xavier registration Vialard

Introduction to diﬀeomorphisms group and Riemannian tools

Choice of the metric

Spatially dependent Several diffeomorphic registration methods are available: metrics Metric learning • Free-form deformations B-spline-based diffeomorphisms by D. SVF metric learning Rueckert • Log-demons (X.Pennec et al.) • Large Deformations by Diffeomorphisms (M. Miller,A. Trouvé,L. Younes) • ANTS Only the two last ones provide a Riemannian framework. Metric learning for diffeomorphic image A Riemannian approach to diffeomorphic registration.

Fran¸cois-Xavier registration Vialard

Introduction to diffeomorphisms group and Riemannian tools n Choice of the metric • vt ∈ V a time dependent vector field on R . Spatially dependent • ϕt ∈ Diff , the flow defined by metrics Metric learning

SVF metric learning ∂t ϕt = vt (ϕt ) . (1)

Action of the group of diﬀeomorphism G0 (ﬂow at time 1):

Π: G0 × C → C , . Π(ϕ, X ) = ϕ.X

2 1 R 1 2 Right-invariant metric on G0: d(ϕ0,1, Id) = 2 0 |vt |V dt. −→ Strong metric needed on V (Mumford and Michor: Vanishing Geodesic Distance on...) Right invariant distance on G0 Z 1 2 2 d(Id, ϕ) = inf |vt | dt , 2 V v∈L ([0,1],V ) 0

−→ geodesics on G0.

V is a Reproducing kernel Hilbert Space (RKHS): (pointwise evaluation continuous) =⇒ Existence of a matrix function kV (kernel) deﬁned on U × U such that: hv(x), ai = hkV (., x)a, viV .

Metric learning for diﬀeomorphic image Matching problems in a diﬀeomorphic framework registration.

Fran¸cois-Xavier Vialard

1 n Introduction to U a domain in R diﬀeomorphisms group 1 and Riemannian tools 2 V a Hilbert space of C vector ﬁelds such that: Choice of the metric

Spatially dependent kvk1,∞ 6 C|v|V . metrics Metric learning

SVF metric learning Right invariant distance on G0 Z 1 2 2 d(Id, ϕ) = inf |vt | dt , 2 V v∈L ([0,1],V ) 0

−→ geodesics on G0.

Metric learning for diﬀeomorphic image Matching problems in a diﬀeomorphic framework registration.

Fran¸cois-Xavier Vialard

1 n Introduction to U a domain in R diﬀeomorphisms group 1 and Riemannian tools 2 V a Hilbert space of C vector ﬁelds such that: Choice of the metric

Spatially dependent kvk1,∞ 6 C|v|V . metrics Metric learning V is a Reproducing kernel Hilbert Space (RKHS): (pointwise SVF metric learning evaluation continuous) =⇒ Existence of a matrix function kV (kernel) defined on U × U such that: hv(x), ai = hkV (., x)a, viV . Metric learning for diffeomorphic image Matching problems in a diffeomorphic framework registration.

Fran¸cois-Xavier Vialard

1 n Introduction to U a domain in R diﬀeomorphisms group 1 and Riemannian tools 2 V a Hilbert space of C vector ﬁelds such that: Choice of the metric

Right invariant distance on G0 Z 1 2 2 d(Id, ϕ) = inf |vt | dt , 2 V v∈L ([0,1],V ) 0

−→ geodesics on G0. Metric learning for diﬀeomorphic image Variational formulation registration.

Fran¸cois-Xavier Vialard

Introduction to Find the best deformation, minimize diﬀeomorphisms group and Riemannian tools J (ϕ) = inf d(ϕ.A, B)2 (2) Choice of the metric Spatially dependent ϕ∈GV | {z } metrics similarity measure Metric learning

SVF metric learning Metric learning for diﬀeomorphic image Variational formulation registration.

Fran¸cois-Xavier Find the best deformation, minimize Vialard Introduction to diﬀeomorphisms group 2 J (ϕ) = inf d(ϕ.A, B) (2) and Riemannian tools ϕ∈G V | {z } Choice of the metric similarity measure Spatially dependent metrics

Tychonov regularization: Metric learning

SVF metric learning 1 2 J (ϕ) = R(ϕ) + 2 d(ϕ.A, B) . (3) | {z } 2σ Regularization | {z } similarity measure

Riemannian metric on GV :

Z 1 1 2 R(ϕ) = |vt |V dt (4) 2 0

is a right-invariant metric on GV . Main issues for practical applications: • choice of the metric (prior), • choice of the similarity measure.

Metric learning for diﬀeomorphic image Optimization problem registration.

Fran¸cois-Xavier Minimizing Vialard Introduction to 1 diﬀeomorphisms group 1 Z 1 and Riemannian tools J (v) = |v |2 dt + d(ϕ .A, B)2 . t V 2 0,1 Choice of the metric 2 0 2σ Spatially dependent In the case of landmarks: metrics Metric learning k SVF metric learning 1 Z 1 1 X J (ϕ) = |v |2 dt + kϕ(x ) − y k2 , 2 t V 2σ2 i i 0 i=1 In the case of images: Z 2 2 d(ϕ0,1.I0, Itarget ) = |I0 ◦ ϕ1,0 − Itarget | dx . U Metric learning for diﬀeomorphic image Optimization problem registration.

Fran¸cois-Xavier Minimizing Vialard Introduction to 1 diﬀeomorphisms group 1 Z 1 and Riemannian tools J (v) = |v |2 dt + d(ϕ .A, B)2 . t V 2 0,1 Choice of the metric 2 0 2σ Spatially dependent In the case of landmarks: metrics Metric learning k SVF metric learning 1 Z 1 1 X J (ϕ) = |v |2 dt + kϕ(x ) − y k2 , 2 t V 2σ2 i i 0 i=1 In the case of images: Z 2 2 d(ϕ0,1.I0, Itarget ) = |I0 ◦ ϕ1,0 − Itarget | dx . U Main issues for practical applications: • choice of the metric (prior), • choice of the similarity measure. Metric learning for diﬀeomorphic image Why does the Riemannian framework matter? registration.

Fran¸cois-Xavier Vialard Generalizations of statistical tools in Euclidean space: Introduction to diffeomorphisms group • Distance often given by a Riemannian metric. and Riemannian tools • Straight lines → geodesic defined by Choice of the metric Spatially dependent Z 1 metrics 2 Metric learning Variational definition: arg min kc˙kc(t) dt = 0 , c(t) 0 SVF metric learning

Equivalent (local) deﬁnition: ∇c˙ c˙ =c ¨ + Γ(c)(c ˙, c˙) = 0 .

• Average → Fr´echet/Karcher mean.

Variational deﬁnition: arg min{x → E[d 2(x, y)]dµ(y)} 2 Critical point deﬁnition: E[∇x d (x, y)]dµ(y)] = 0 .

• PCA → Tangent PCA or PGA. • Geodesic regression, cubic regression...(variational or algebraic) Metric learning for diﬀeomorphic image Karcher mean on 3D images registration.

Fran¸cois-Xavier Vialard

Introduction to diﬀeomorphisms group Init. guesses and Riemannian tools Choice of the metric

Spatially dependent metrics

Metric learning 1 iteration SVF metric learning

2 iterations

3 iterations 1 2 3 4 Ai Ai Ai Ai m Figure– Average image estimates Ai , m ∈ {1, ··· , 4} after i =0, 1, 2 and 3 iterations. Metric learning for diﬀeomorphic image Interpolation, Extrapolation registration.

Fran¸cois-Xavier Vialard

Introduction to diﬀeomorphisms group and Riemannian tools

Choice of the metric

Spatially dependent metrics

Metric learning

SVF metric learning

Figure– Geodesic regression (MICCAI 2011) Metric learning for diﬀeomorphic image Interpolation, Extrapolation registration.

Fran¸cois-Xavier Vialard

Introduction to diﬀeomorphisms group and Riemannian tools

Choice of the metric

Spatially dependent metrics

Metric learning

SVF metric learning

Figure– Extrapolation of happiness Metric learning for diﬀeomorphic image registration.

Fran¸cois-Xavier Vialard

Introduction to diﬀeomorphisms group and Riemannian tools

Choice of the metric

Spatially dependent metrics

Metric learning

What metric to choose? SVF metric learning The norm on V is simply Z Z 2 1/2 2 kvkV = hv(x), (Lv)(x)i dx = (L v) (x) dx . Ω Ω Scale parameter important!

2 − kx−yk n kσ(x, y) = e σ2 kernel/operator (Id −σ∆) (5)

• σ small: good matching but non regular deformations and more local minima. • σ large: poor matching but regular deformations and more global minima.

Metric learning for diffeomorphic image Choosing the right-invariant metric registration. Right-invariant metric: Eulerian fluid dynamic viewpoint on Fran¸cois-Xavier Vialard regularization. Introduction to Space V of vector fields is defined equivalently by diffeomorphisms group and Riemannian tools • its kernel K such as Gaussian kernel, Choice of the metric n • its differential operator, for instance (Id −σ∆) for Sobolev Spatially dependent metrics spaces. Metric learning

SVF metric learning Scale parameter important!

2 − kx−yk n kσ(x, y) = e σ2 kernel/operator (Id −σ∆) (5)

• σ small: good matching but non regular deformations and more local minima. • σ large: poor matching but regular deformations and more global minima.

The norm on V is simply SVF metric learning Z Z 2 1/2 2 kvkV = hv(x), (Lv)(x)i dx = (L v) (x) dx . Ω Ω Metric learning for diffeomorphic image Choosing the right-invariant metric registration. Right-invariant metric: Eulerian fluid dynamic viewpoint on Fran¸cois-Xavier Vialard regularization. Introduction to Space V of vector fields is defined equivalently by diffeomorphisms group and Riemannian tools • its kernel K such as Gaussian kernel, Choice of the metric n • its differential operator, for instance (Id −σ∆) for Sobolev Spatially dependent metrics spaces. Metric learning

The norm on V is simply SVF metric learning Z Z 2 1/2 2 kvkV = hv(x), (Lv)(x)i dx = (L v) (x) dx . Ω Ω Scale parameter important!

2 − kx−yk n kσ(x, y) = e σ2 kernel/operator (Id −σ∆) (5)

• σ small: good matching but non regular deformations and more local minima. • σ large: poor matching but regular deformations and more global minima. Figure– Left to right: Small scale, large scale and multi scale.

Metric learning for diﬀeomorphic image Sum of kernels and multiscale registration.

Choice of mixture of Gaussian kernels: (Risser, Vialard et al. 2011) Fran¸cois-Xavier Vialard n 2 − kx−yk X σ2 i Introduction to K(x, y) = αi e (6) diﬀeomorphisms group i=1 and Riemannian tools Choice of the metric

Spatially dependent metrics

Metric learning

SVF metric learning Metric learning for diﬀeomorphic image Sum of kernels and multiscale registration.

Spatially dependent metrics

Metric learning

SVF metric learning

Figure– Left to right: Small scale, large scale and multi scale. Metric learning for diﬀeomorphic image Decomposition over diﬀerent scales registration.

Fran¸cois-Xavier Vialard

Introduction to diﬀeomorphisms group and Riemannian tools

Try to disentangle contributions at each scale: (Bruveris, Risser, Choice of the metric

Vialard, 2012, Siam MMS) using semi-direct product of groups. Spatially dependent metrics Consider Gσ1 , Gσ2 two diﬀeomorphism groups at diﬀerent scales Metric learning associated with Vσ1 and Vσ2 . SVF metric learning

Semi-direct product of groups Gσ1 n Gσ2 . Non-linear extension of the inﬁmal convolution of norms: n o kvk2 = min kv k2 + kv k2 v = v + v . (7) 1 V1 2 V2 1 2 (v1,v2)∈V1×V2

Non-linear extension −→ semi-direct product of groups. How to introduce spatially varying metric?

Using kernels: χi being a partition of unity of the domain.

n X K = χi Ki χi ,. (8) i=1 This kernel is associated to the following variational interpretation:

( n n ) 2 X 2 X kvk = min kvi kV χi vi = v . (9) (v ,...,v )∈V ×...×V i 1 n 1 n i=1 i=1 −→ possibility to introduce soft-symmetries...

Metric learning for diﬀeomorphic image From Eulerian to Lagrangian viewpoints registration.

Fran¸cois-Xavier Vialard

Spatial correlation of the deformation: need for local deformability Introduction to diﬀeomorphisms group on the tissues. and Riemannian tools

Toward a more Lagrangian point of view. Choice of the metric

Spatially dependent metrics

Metric learning

SVF metric learning Metric learning for diﬀeomorphic image From Eulerian to Lagrangian viewpoints registration.

Fran¸cois-Xavier Vialard

Spatial correlation of the deformation: need for local deformability Introduction to diﬀeomorphisms group on the tissues. and Riemannian tools

Toward a more Lagrangian point of view. Choice of the metric

Spatially dependent How to introduce spatially varying metric? metrics Metric learning

Using kernels: χi being a partition of unity of the domain. SVF metric learning

n X K = χi Ki χi ,. (8) i=1 This kernel is associated to the following variational interpretation:

( n n ) 2 X 2 X kvk = min kvi kV χi vi = v . (9) (v ,...,v )∈V ×...×V i 1 n 1 n i=1 i=1 −→ possibility to introduce soft-symmetries... • More natural interpretation of spatially varying metrics. • Left action + left invariant metric =⇒ no induced Riemannian metric.

Metric learning for diﬀeomorphic image Left-invariant metrics registration.

Fran¸cois-Xavier Vialard

Miccai 2013, Marsden’s Fields volume, Schmah, Risser, Vialard Introduction to diﬀeomorphisms group Change of point of view: choose body-coordinates and convective and Riemannian tools velocity: Choice of the metric Spatially dependent metrics Z 1 1 2 Metric learning J (ϕ) = kv(t)kV dt + E(ϕ(1) · I , J), (10) 2 0 SVF metric learning under the convective velocity constraint:

∂t ϕ(t) = dϕ(t) · v(t) , (11)

where dϕ(t) is the tangent map of ϕ(t). Metric learning for diﬀeomorphic image Left-invariant metrics registration.

Fran¸cois-Xavier Vialard

∂t ϕ(t) = dϕ(t) · v(t) , (11)

where dϕ(t) is the tangent map of ϕ(t).

• More natural interpretation of spatially varying metrics. • Left action + left invariant metric =⇒ no induced Riemannian metric. Metric learning for diﬀeomorphic image Diﬀerence with LDDMM registration.

The path look diﬀerent: Fran¸cois-Xavier Vialard

Introduction to diﬀeomorphisms group and Riemannian tools

Choice of the metric

Spatially dependent metrics

Metric learning

Figure– The green curves Right-LDM geodesic path, The blue curves SVF metric learning Left-LDM geodesic path.

LDDMM LIDM Metric learning for diﬀeomorphic image What’s next registration.

Fran¸cois-Xavier Vialard

Introduction to diﬀeomorphisms group and Riemannian tools

Choice of the metric

Spatially dependent Left-invariance is more Lagrangian but the metric is ﬁxed as in the metrics

Eulerian situation! Metric learning • On a template, learning the metric. SVF metric learning

Motivation: • Better matching results: i.e better regularization or matching. • Better matching quality for organs with (segmented) tumors. Metric learning for diﬀeomorphic image Metric learning registration.

Fran¸cois-Xavier Vialard

Introduction to diﬀeomorphisms group and Riemannian tools

Choice of the metric

Spatially dependent metrics

Metric learning

SVF metric learning

Figure– Given a collection of shape and a template, learn the metric. Registering the template T onto the image In consists in minimizing:

Z 1 1 2 −1 JIn,K (v) = kv(t)kV dt + d(T ◦ ϕ(1) , In) , (12) 2 0 where ∂t ϕ(t) = v(t) ◦ ϕ(t) . Equivalent to minimize

Z 1 1 −1 JIn,K (v) = hP(t)∇I (t), K?(P(t)∇I (t)i dt+d(T ◦ϕ(1) , In) , 2 0 (13)

Metric learning for diﬀeomorphic image Metric learning: High-dimensional inverse registration.

Fran¸cois-Xavier problem Vialard

Introduction to (Miccai 2014: Vialard, Risser) diﬀeomorphisms group and Riemannian tools

• (In)n=1,...,N be a population of N images. Choice of the metric

Spatially dependent • T be a template (Karcher mean for instance). metrics

Metric learning

SVF metric learning Equivalent to minimize

Z 1 1 −1 JIn,K (v) = hP(t)∇I (t), K?(P(t)∇I (t)i dt+d(T ◦ϕ(1) , In) , 2 0 (13)

Metric learning for diﬀeomorphic image Metric learning: High-dimensional inverse registration.

Fran¸cois-Xavier problem Vialard

Introduction to (Miccai 2014: Vialard, Risser) diﬀeomorphisms group and Riemannian tools

• (In)n=1,...,N be a population of N images. Choice of the metric

Spatially dependent • T be a template (Karcher mean for instance). metrics Registering the template T onto the image In consists in Metric learning minimizing: SVF metric learning

Z 1 1 2 −1 JIn,K (v) = kv(t)kV dt + d(T ◦ ϕ(1) , In) , (12) 2 0 where ∂t ϕ(t) = v(t) ◦ ϕ(t) . Metric learning for diﬀeomorphic image Metric learning: High-dimensional inverse registration.

Fran¸cois-Xavier problem Vialard

Introduction to (Miccai 2014: Vialard, Risser) diﬀeomorphisms group and Riemannian tools

• (In)n=1,...,N be a population of N images. Choice of the metric

Spatially dependent • T be a template (Karcher mean for instance). metrics Registering the template T onto the image In consists in Metric learning minimizing: SVF metric learning

Z 1 1 2 −1 JIn,K (v) = kv(t)kV dt + d(T ◦ ϕ(1) , In) , (12) 2 0 where ∂t ϕ(t) = v(t) ◦ ϕ(t) . Equivalent to minimize

Z 1 1 −1 JIn,K (v) = hP(t)∇I (t), K?(P(t)∇I (t)i dt+d(T ◦ϕ(1) , In) , 2 0 (13) • Regularization on M. Prior for M close to Id. Minimize

N β 2 1 X F(M) = dS++ (M, Id) + min JIn (v, M) , (15) 2 N v n=1

where d 2 can be chosen as • Aﬃne invariant metric (Pennec et al.) −1 −1 g1 = Tr(M (δM)M (δM)). • (Modiﬁed) Wasserstein metric.

Problem Problem: matrix M is huge: (dn)2 where d = 2, 3 dimension and n number of voxels. Computing the logarithm is costly.

Metric learning for diﬀeomorphic image Optimize over K? Ill posed! registration.

• Incorporate the smoothness constraint by deﬁning Fran¸cois-Xavier Vialard ˆ ˆ 2 d d K = {KMK | M SDP operator on L (R , R )} , (14) Introduction to diﬀeomorphisms group and Riemannian tools

M symmetric positive deﬁnite matrix. Choice of the metric

Spatially dependent metrics

Metric learning

SVF metric learning Metric learning for diﬀeomorphic image Optimize over K? Ill posed! registration.

M symmetric positive deﬁnite matrix. Choice of the metric • Regularization on M. Prior for M close to Id. Minimize Spatially dependent metrics N Metric learning β 2 1 X SVF metric learning F(M) = dS++ (M, Id) + min JIn (v, M) , (15) 2 N v n=1

where d 2 can be chosen as • Aﬃne invariant metric (Pennec et al.) −1 −1 g1 = Tr(M (δM)M (δM)). • (Modiﬁed) Wasserstein metric.

Problem Problem: matrix M is huge: (dn)2 where d = 2, 3 dimension and n number of voxels. Computing the logarithm is costly. Encode the symmetric matrix as NNT and perform the 1 2 optimization on N. The regularization reads 2 kN − Idk The gradient is

∇L2 F(N) = β(N − Id)− (16) N 1 X Z 1 (Kˆ ? Pn(t)) ⊗ (NKˆ ? Pn(t)) + (NKˆ ? Pn(t)) ⊗ (Kˆ ? Pn(t))dt , 2N n=1 0 (17)

Metric learning for diﬀeomorphic image Wasserstein metric registration.

Fran¸cois-Xavier Vialard

Pros: Easy to compute Introduction to diﬀeomorphisms group Cons: Non complete metric. and Riemannian tools

Choice of the metric

Trick Spatially dependent metrics T The map N 7→ NN is a Riemannian submersion from Mn(R) Metric learning equipped with the Frobenius norm to the space of SDP matrices SVF metric learning equipped with the Wasserstein metric Metric learning for diﬀeomorphic image Wasserstein metric registration.

Fran¸cois-Xavier Vialard

Pros: Easy to compute Introduction to diﬀeomorphisms group Cons: Non complete metric. and Riemannian tools

Choice of the metric

Encode the symmetric matrix as NNT and perform the 1 2 optimization on N. The regularization reads 2 kN − Idk The gradient is

∇L2 F(N) = β(N − Id)− (16) N 1 X Z 1 (Kˆ ? Pn(t)) ⊗ (NKˆ ? Pn(t)) + (NKˆ ? Pn(t)) ⊗ (Kˆ ? Pn(t))dt , 2N n=1 0 (17) Metric learning for diﬀeomorphic image Reducing the problem dimension registration.

Fran¸cois-Xavier Vialard

Introduction to diﬀeomorphisms group and Riemannian tools

Choice of the metric

Spatially dependent Idea metrics Learn at large scales and not at ﬁne scale. Metric learning SVF metric learning Introduce:

2 d d K = {KMˆ ΠKˆ + Kˆ(Id − Π)Kˆ | M SDP operator on L (R , R )} . (18) with Π an orthogonal projection on a ﬁnite dimensional parametrization of vector ﬁelds: use of splines. For a given quality of overlap, better smoothness of the deformations.

Metric learning for diﬀeomorphic image Experiments registration.

Fran¸cois-Xavier • 40 subjects of the LONI Probabilistic Brain Atlas (LPBA40). Vialard • All 3D images were affinely aligned to subject 5 using ANTS. Introduction to diffeomorphisms group and Riemannian tools Table– Reference results Choice of the metric Spatially dependent No Reg SyN Kfine Kref metrics TO 0.665 0.750 0.732 0.712 Metric learning SVF metric learning DetJMax 1 3.17 4.65 1.66 DetJMin 1 0.047 0.46 0.67 DetJStd 0 0.17 0.11 0.063

Table– Average results.

DiagM GridM1 GridM2 K20 K30 TO 0.711 0.710 0.704 0.710 0.704

DetJMax 1.66 1.61 1.41 1.62 1.50 DetJMin 0.68 0.70 0.67 0.73 0.66 DetJStd 0.062 0.059 0.049 0.056 0.063 Metric learning for diﬀeomorphic image Experiments registration.

Table– Average results.

DiagM GridM1 GridM2 K20 K30 TO 0.711 0.710 0.704 0.710 0.704

DetJMax 1.66 1.61 1.41 1.62 1.50 DetJMin 0.68 0.70 0.67 0.73 0.66 DetJStd 0.062 0.059 0.049 0.056 0.063 For a given quality of overlap, better smoothness of the deformations. Metric learning for diﬀeomorphic image Main issues registration.

Fran¸cois-Xavier Vialard

Introduction to diﬀeomorphisms group and Riemannian tools

Choice of the metric

Spatially dependent metrics • The metric is ﬁxed in Eulerian coordinates. Metric learning • The metric is template based. SVF metric learning

Make the method adaptive to any pairs of images on a simpler model. Equivalent momentum formulation: Find m ∈ L2(Ω, Rd ) ⊂ V ∗ such that 1 hm, K ? mi + Sim(I ◦ ϕ−1, J) (21) 2 1 s.t. −1 −1 ∂t ϕ (t, x) + Dϕ (t, x)(K ? m(t, x)) = 0 . (22) Numerical discretization: Central diﬀerences in space and 20 timesteps in time of RK4.

Metric learning for diﬀeomorphic image SVF model: A simple model registration.

Fran¸cois-Xavier Vialard

Based on Metric learning for image registration, CVPR 2019, Introduction to diﬀeomorphisms group Niethammer, Kwitt, Vialard. and Riemannian tools

Let v(x) be a vector ﬁeld. Find a v minimizer of Choice of the metric

Spatially dependent 1 metrics kvk2 + Sim(I ◦ ϕ−1, J) (19) 2 V 1 Metric learning SVF metric learning

∂t ϕt = v(ϕt ) . (20) Metric learning for diﬀeomorphic image SVF model: A simple model registration.

Fran¸cois-Xavier Vialard

Based on Metric learning for image registration, CVPR 2019, Introduction to diﬀeomorphisms group Niethammer, Kwitt, Vialard. and Riemannian tools

Let v(x) be a vector ﬁeld. Find a v minimizer of Choice of the metric

Spatially dependent 1 metrics kvk2 + Sim(I ◦ ϕ−1, J) (19) 2 V 1 Metric learning SVF metric learning

∂t ϕt = v(ϕt ) . (20) Equivalent momentum formulation: Find m ∈ L2(Ω, Rd ) ⊂ V ∗ such that 1 hm, K ? mi + Sim(I ◦ ϕ−1, J) (21) 2 1 s.t. −1 −1 ∂t ϕ (t, x) + Dϕ (t, x)(K ? m(t, x)) = 0 . (22) Numerical discretization: Central differences in space and 20 timesteps in time of RK4. Metric learning for diffeomorphic image Parametrization of the metric registration. Fix a collection of scales σ0 < . . . < σN−1 and set Fran¸cois-Xavier −|x|2/σ2 Vialard Gi (x) = e i . Introduction to def. diffeomorphisms group v0(x) = (K(w) ? m0)(x) and Riemannian tools Choice of the metric N−1 Z def. X p p Spatially dependent = wi (x) Gi (|x − y|) wi (y)m0(y) dy , (23) metrics i=0 y Metric learning SVF metric learning Problem: wi should be sufficiently smooth to guarantee diffeomorphisms. Introduce pre-weights ωi (x) and fix Kσ, with σ small:

Kσ ? ωi = wi and X wi (x) = 1 . i Learning the metric is still ill-posed: −r N−1 r σN−1 X σN−1 OMT([ w) = log wi log (24) σ0 σi i=0

Is 0 for (wi ) = (0,..., 0, 1). Metric learning for diﬀeomorphic image Objective function registration.

Fran¸cois-Xavier Optimize momentum + metric. Vialard

Introduction to diﬀeomorphisms group −1 and Riemannian tools Obj0,1(m, ω) = argmin λhm0, v0i + Sim[I0 ◦ Φ (1), I1] + m0 Choice of the metric

Z Spatially dependent λOMT OMT([ w(x)) dx + metrics Metric learning v uN−1 2 SVF metric learning uX Z λTVt γ(k∇I0(x)k)k∇ωl (x)k2 dx , (25) l=0

where γ(x) ∈ R+ is an edge indicator function γ(k∇I k) = (1 + αk∇I k)−1, α > 0 .

Then, minimize X i Obj0,1(mi,j , ω ) , (26) i,j

i where ω = fθ(Ii ). Metric learning for diﬀeomorphic image Parametrize and learn the pre-weights ωi registration.

Fran¸cois-Xavier Vialard

Introduction to The pre-weights are parametrized by a 2-layers net: diﬀeomorphisms group and Riemannian tools

Choice of the metric (ωi )i=1,...,N = ShallowNet(I ) . Spatially dependent metrics Input: current image, Output: pre-weights. Metric learning SVF metric learning

ShallowNet = conv(d, n1) → BatchNorm → lReLU → conv(n1, N) → BatchNorm → weighted-linear-softmax

clamp (wj + zj − z) σ (z) = 0,1 , (27) w j PN−1 i=0 clamp0,1(wi + zi − z)

2 PN−1 2 The weights wj are reasonably initialized: wi = σi /( j=0 σj ) Metric learning for diﬀeomorphic image Optimization registration.

Fran¸cois-Xavier Vialard

Introduction to diﬀeomorphisms group and Riemannian tools

Choice of the metric

Spatially dependent Shared parameters: ShallowNetwork parameters, metrics Individual parameters: Momentum for each pair. Metric learning SVF metric learning 1 (1) initialize with reasonable weights and optimize over momentums, 2 (2) Jointly optimize on the shared and individual parameters: use SGD with (Nesterov) momentum, different batch size in 2d/3d, 50 epochs in 2d, less in 3D. Metric learning for diffeomorphic image Experiments on synthetic data registration. 1) Generate concentric circular regions with random radii and Fran¸cois-Xavier Vialard associate different multi-Gaussian weights to these regions. Introduction to We associate a fixed multi-Gaussian weight to the diffeomorphisms group background. and Riemannian tools 2) Randomly create vector momenta at the borders of the Choice of the metric Spatially dependent concentric circles. metrics 3) Add noise (for texture) and compute forward model, to Metric learning obtain source image, similar for target image. SVF metric learning

1 disp error (est-GT) [pixel] 0

2.0

1.5

1.0

0.5 disp error (est-GT) [pixel] 0.0 global t=0.01;o=5_l t=0.10;o=5_l t=0.25;o=5_l t=0.01;o=15_l t=0.01;o=25_l t=0.01;o=50_l t=0.01;o=75_l t=0.10;o=15_l t=0.10;o=25_l t=0.10;o=50_l t=0.10;o=75_l t=0.25;o=15_l t=0.25;o=25_l t=0.25;o=50_l t=0.25;o=75_l t=0.01;o=100_l t=0.10;o=100_l t=0.25;o=100_l

Figure– Displacement error (in pixel) with respect to the ground truth for various values of the total variation penalty, λTV (t) and the OMT penalty, λOMT (o). Results for the interior (top) and the exterior (bottom) rings show subpixel registration accuracy for all local metric optimization results (* l). Overall, local metric optimization substantially improves registrations over the results obtained via initial global multi-Gaussian regularization (global). Metric learning for diﬀeomorphic image Experiments on synthetic data registration.

Source image Target image Fran¸cois-Xavier Vialard

Introduction to diﬀeomorphisms group and Riemannian tools

Choice of the metric

Spatially dependent metrics Warped source Deformation grid Std.dev. Metric learning 0.195 0.190 SVF metric learning 0.185

0.180

0.175

0.170

0.165 λOMT = 15

0.195

0.190

0.185

0.180

0.175

0.170 λOMT = 50

0.195

0.190

0.185

0.180

0.175

0.170

λOMT = 100 0.165

Figure– λTV = 0.1. Overall variance is similar but the true weights are not recovered: weights on the outer ring [0.05, 0.55, 0.3, 0.1] Metric learning for diﬀeomorphic image Experiments registration.

w0(σ = 0.01) w1(σ = 0.05) w2(σ = 0.10) w3(σ = 0.20) Fran¸cois-Xavier Vialard λOMT = 15 Introduction to diﬀeomorphisms group and Riemannian tools

Choice of the metric

Spatially dependent metrics

Metric learning

λOMT = 50 SVF metric learning

λOMT = 100

Figure Metric learning for diﬀeomorphic image On 2D real data: LPBA40 registration.

Fran¸cois-Xavier Source image Target image Vialard

Introduction to diﬀeomorphisms group and Riemannian tools

Choice of the metric

Spatially dependent metrics Warped source Deformation grid Std. dev. Metric learning 0.197

0.196 SVF metric learning

0.195

0.194

0.193

0.192

λOMT = 15 0.191

0.198

0.197

0.196

0.195

0.194

0.193

λOMT = 50 0.192

0.199

0.198

0.197

0.196

0.195

0.194 λOMT = 100 Metric learning for diﬀeomorphic image Performance on 3D data: CUMC12 registration.

Fran¸cois-Xavier Training on diﬀerent dataset: Vialard

Trained on different dataset, test on CUMC12 Introduction to diffeomorphisms group Training 132 image pairs on CUMC12, 90 image pairs on and Riemannian tools MGH10, 150 image pairs on IBSR18. Choice of the metric Spatially dependent metrics Method mean std 1% 5% 50% 95% 99% p MW-stat sig? Metric learning FLIRT 0.394 0.031 0.334 0.345 0.396 0.442 0.463 <1e−10 17394.0 AIR 0.423 0.030 0.362 0.377 0.421 0.483 0.492 <1e−10 17091.0 SVF metric learning ANIMAL 0.426 0.037 0.328 0.367 0.425 0.483 0.498 <1e−10 16925.0 ART 0.503 0.031 0.446 0.452 0.506 0.556 0.563 <1e−4 11177.0 Demons 0.462 0.029 0.407 0.421 0.461 0.510 0.531 <1e−10 15518.0 FNIRT 0.463 0.036 0.381 0.410 0.463 0.519 0.537 <1e−10 15149.0 Fluid 0.462 0.031 0.401 0.410 0.462 0.516 0.532 <1e−10 15503.0 SICLE 0.419 0.044 0.300 0.330 0.424 0.475 0.504 <1e−10 17022.0 SyN 0.514 0.033 0.454 0.460 0.515 0.565 0.578 0.072 9677.0 SPM5N8 0.365 0.045 0.257 0.293 0.370 0.426 0.455 <1e−10 17418.0 SPM5N 0.420 0.031 0.361 0.376 0.418 0.471 0.494 <1e−10 17160.0 SPM5U 0.438 0.029 0.373 0.394 0.437 0.489 0.502 <1e−10 16773.0 SPM5D 0.512 0.056 0.262 0.445 0.523 0.570 0.579 0.315 9043.0 m/c global 0.480 0.031 0.421 0.430 0.482 0.530 0.543 <1e−10 13864.0 m/c local 0.517 0.034 0.454 0.461 0.521 0.568 0.578 0.263 9163.0 c/c global 0.480 0.031 0.421 0.430 0.482 0.530 0.543 <1e−10 13864.0 c/c local 0.520 0.034 0.455 0.463 0.524 0.572 0.581 - - - i/c global 0.480 0.031 0.421 0.430 0.482 0.530 0.543 <1e−10 13863.0 i/c local 0.518 0.035 0.454 0.460 0.522 0.571 0.581 0.338 8972.0 Table– Statistics for mean target overlap ratios for CUMC12 for different methods. Metric learning for diffeomorphic image Conclusion registration.

Fran¸cois-Xavier Vialard

Introduction to diffeomorphisms group Summary and Riemannian tools Adaptive metric learning in SVF. Choice of the metric Spatially dependent Avoid end to end training for preserving diffeomorphic metrics properties. Metric learning Diffeomorphic guarantees at test time (no guarantee for DL SVF metric learning methods: VoxelMorph). Perspectives Combine it with momentum prediction (QuickSilver like). Use it in LDDMM. Incorporate richer deformations descriptors. Paper to appear: Metric learning for image registration, CVPR 2019, Niethammer, Kwitt, Vialard.