Diffeomorphic Shape Analysis

Laurent Younes Johns Hopkins University January 2014

Partially supported by ONR, NIH, NSF

Geilo, January 2014 Laurent Younes 1 Introduction

• Shape analysis is the study of variation of shapes in relation with other variables. • Once a suitable “shape space” is defined, analyzing shapes involves: – Comparing them (using distances) – Finding properties of datasets in this space – Testing statistical hypotheses

Geilo, January 2014 Laurent Younes 2 Some entry points to the literature

• Landmarks / point sets: Kendall, Bookstein, Small, Dryden… • Active shapes: Taylor, Cootes,… • Registration: Demons, Diffeomorphic Demons (Thirion, Guimond, Ayache, Pennec, Vercauteren,…) • Harmonic analysis: SPHARM (Gerig, Steiner,…) • Medial axis and related, M-REPS (Pizer, Damon,…) • of curves / surfaces – Theory (Mumford, Michor, Yezzi, Mennucci, Srivastava, Mio, Klassen…) • Statistics on manifolds of curves/surfaces (Pennec, Fletcher, Joshi, Marron…)

Geilo, January 2014 Laurent Younes 3 On “Diffeomorphometry”…

• Precursors (fluid registration): Christensen-Miller-Rabbit; Thirion… • Large Deformation Diffeomorphic Metric Mapping – LDDMM – Images: Beg, Miller, Trouvé, Y.; – Landmarks Miller, Joshi; – Measures (Glaunès, Trouvé, Y.), Currents (Glaunès, Vaillant) – Hamiltonian methods (Glaunès, Trouvé, Vialard, Y…) • Metamorphosis (Miller, Trouvé, Holm, Y…)

Geilo, January 2014 Laurent Younes 4 Shape Spaces

• Shape spaces are typically modeled as differential manifolds, but… • Almost all practical methods use a shape representation in a linear space of “local coordinates” • Linear statistical methods can then be employed to analyze the data. • For some of these methods (e.g., PCA), it is also important that linear combinations of the representations make sense, too.

Geilo, January 2014 Laurent Younes 5 Example: Kendall Shape Space

• Consider the family of collections of N distinct points in d ! d N • This forms an open subset of Q = (! ) • Identify collections that are can be deduced from each other by rotation, translation and scaling. • It is a set of subsets of Q (quotient space). • It can be structured as a differential . • The Euclidean metric on Q transforms into a Riemannian metric in the quotient space.

Geilo, January 2014 Laurent Younes 6 Example: Spaces of Curves

2 • The set Q of smooth functions q : [ 0 , 1 ] → ! is a Fréchet space. • Quotient out translation rotations scaling and change of parameter to obtain a shape space. • Sobolev norms on Q trickle down to Riemannian metrics on the shape space. • See papers from Michor, Mumford, Shah…

Geilo, January 2014 Laurent Younes 7 Shapes Spaces and Deformable Templates • Ulf Grenander’s metric pattern theory involves transformation groups that act on shapes. • The transformations have a cost, represented by an effort functional. • Under additional assumptions, this induces a metric in the shape space. • The construction that follows will be based on these principles. • The is the group of . • The shapes are anything that can be deformed.

Geilo, January 2014 Laurent Younes 8 Tangent space representations

• There is a natural way to build linear representations when the the shape space is modeled as a , i.e.: – a topological space that can be mapped locally to a on which differentials can be computed (with consistency relations between local maps). – with a metric which can be used to compute lengths, and shortest paths between two points, which are called .

Geilo, January 2014 Laurent Younes 9 Exponential Charts (I)

• Geodesics can be characterized by a second order differential equation on the manifold. • This equation has (in general) a unique solution given its initial position and first derivative. • Fix the initial position (“template”). Denote it S . • The space of all derivatives of curves in S that start from T S S is the tangent space to S at S (notation S ).

Geilo, January 2014 Laurent Younes 10 Exponential Charts (II)

• The exponential map associates to each vector v T S ∈ S the solution at time t =1 of the equation starting at S in the direction v. • It is a map from T S to S , with notation S

v ∈ T S exp (v) ∈ S S  S • It can (generally) be restricted to a neighborhood of 0 in T S S over which it is one to one, providing so-called normal local coordinates or an exponential chart at S .

Geilo, January 2014 Laurent Younes 11 SHAPES AND DIFFEOMORPHISMS

Geilo, January 2014 Laurent Younes 12 General Approach

• We will define shape spaces via the action of diffeomorphisms on them. • This action will induce a differential structure on the shape space. • It also allows to compare them: two shapes are similar if one can be obtained from the other via a small deformation.

Geilo, January 2014 Laurent Younes 13 Notation

• Diffeomorphisms of  d will be denoted φ, ψ, etc. They are invertible differentiable transformations with a differentiable inverse.

• Vector fields over  d will be denoted v, w, etc. They are differentiable mappings from  d onto itself.

Geilo, January 2014 Laurent Younes 14 Ordinary Differential Equations

• Diffeomorphisms can be built as flows associated to ordinary differential equations (ODEs). • Let ( t , x ) ! v ( t , x ) be a time-dependent vector field. Assume that v is differentiable in x with bounded first derivative over a time interval [ 0 , T ] . • Then there exists a unique solution of the ODE y ! = v ( t , y ) y(0) x with initial condition = 0 defined over the whole interval [ 0,T]

Geilo, January 2014 Laurent Younes 15 Flows

• The flow associated with the ODE y! = v ( t , y ) is the mapping (t,x) ! ϕ(t,x) such that ϕ ( t , x ) is the solution at time t of the ODE initialized with y ( 0 ) = x . • In other terms:

⎧∂tϕ = v(t,ϕ) ⎨ ⎩ϕ(0,x) = x

Geilo, January 2014 Laurent Younes 16 Flows (II)

• Assuming that v ( t , ⋅ ) is continuously differentiable with bounded derivative, uniformly in time, the associated flow is such that x ! ϕ(t,x) is a at all times. • If v has more (space) derivatives, they are inherited by ϕ . • We will build diffeomorphisms as flows associated to vector fields that belong in a specified reproducing kernel .

Geilo, January 2014 Laurent Younes 17 Hilbert Spaces

• A Hilbert space H is an infinite-dimensional vector space with an inner-product, such that the associated norm that makes it a complete . • The inner product between two elements in H is denoted h,h' H • The set of bounded linear functionals µ : H → ! is the topological dual of H, denoted H * . • We will use the notation µ h : = µ ( h ) for µ ∈H *,h ∈H ( )

Geilo, January 2014 Laurent Younes 18 Riesz Theorem

• The Riesz representation theorem states that H and H * are isometric. * • µ ∈ H if and only if there exists h ∈ H such that, for all h ' ∈ H : µh′ = h,h' ( ) H • The correspondence h will be denoted K , so that µ ! H K : H * H and its inverse A : H ! H * H → H • We therefore have

µh = K µ,h and h,h' = A hh′ ( ) H H H ( H )

Geilo, January 2014 Laurent Younes 19 RKHS

d • A Hilbert space H of functions defined over ! , with k values in ! is an RKHS, if H ⊂ L2 (!d ,!k ) ∩ C p (!d ,!k ) 0 ( p ≥ 0 ) with continuous inclusions in both spaces. (Here C p ( d , k ) is the space of p times differentiable 0 ! ! vector fields that converge to 0 (will all derivatives up to order p) at infinity, equipped with the supremum norm. • This means that for some constant C, and for all v ∈H max v , v ≤ C v ( 2 p,∞ ) H

Geilo, January 2014 Laurent Younes 20 RKHS

d k • If H is an RKHS, and x ∈ ! , a ∈ ! the linear form aδ : v ! aT v(x) x belongs to H * . • General notation: if θ is a (scalar) measure and a is a vector field defined on its , a θ is the vector measure

(a | w) a(x)T w(x)d (x) θ = ∫ θ

Geilo, January 2014 Laurent Younes 21 RKHS

• The reproducing kernel of V is a matrix-valued function K V d d defined on ! × ! by

K H (x, y)a = K H (aδ y )(x) d with a,x ∈! K (x, y) • H is a k by k matrix. • We will denote K i the ith column of K and by K i j its i,j H H H entry. One has, in particular

K i (⋅,x), K j (⋅, y) = K ij (x, y) H H H H

Geilo, January 2014 Laurent Younes 22 Usual examples (scalar kernels)

I • k is the d by d identity matrix. • Gaussian kernel: K (x, y) ∝ exp(− x − y 2 /2σ 2 )I H ‖ ‖ k I • Cauchy Kernel: K (x, y) ∝ k H 2 2 1+ x − y σ • Power of Laplacian:

K (x, y) ∝ P ( x − y /σ )exp(− x − y /σ )I H m ‖ ‖ ‖ ‖ k with P = 1 , P = 1 + t and P P tP 0 1 ∂t m − m = − m−1

Geilo, January 2014 Laurent Younes 23 The Interpolation Problem

• Let H be a Hilbert space and µ , , µ ∈ H * . 1 … m • Solve the problem

⎡ h → min ⎢ H ⎢subject to µ h = λ ,k = 1,…,m ⎣ ( k ) k

m • Solution: h = a K µ where a , … , a are identified ∑ k=1 k H k 1 m using the constraints.

Geilo, January 2014 Laurent Younes 24 Application

• H is a scalar RKHS (k = 1). x , ,x ∈ d • 1 … m ! • Solve ⎡ h → min ⎢ H subject to h(x ) = λ ,k = 1,…,m ⎣⎢ k k

m • Solution: h(⋅) = K (⋅,x )a ∑ k=1 H k k

m with: K (x ,x )a = λ , k = 1,…,m ∑ l=1 H k l l k

Geilo, January 2014 Laurent Younes 25 Attainable Diffeomorphisms

• Fix V, an RKHS of vector fields with at least one derivative ( p ≥ 1 ). • We consider ODEs of the form y! = v ( t , y ) with v(t,⋅) ∈V • The flow evolution ( t , ) v ( t , ( t , ) ) can be interpreted ∂tϕ ⋅ = ϕ ⋅ as a control system in which the deformation, φ, is the state driven by the time-dependent vector field v (t,⋅) • The space of attainable diffeomorphisms is

G = ψ : ∃T,∃v(t,⋅), v(t,⋅) bounded,ψ = ϕ v (T,⋅) V { V } v where ϕ is the flow associated to v.

Geilo, January 2014 Laurent Younes 26 Attainable Diffeomorphisms

G d • V is a subgroup of the group of diffeomorphisms of ! • There is no loss of generality in taking T = 1 in the definition. • One defines a distance on G by V 1 v dV (ψ ,ψ ′) = min ‖v(t,)‖V dt :ϕ (1,ψ ) =ψ ′ {∫ 0 } and

1 2 2 v dV (ψ ,ψ ′) = min ‖v(t,)‖V dt :ϕ (1,ψ ) =ψ ′ {∫ 0 }

Geilo, January 2014 Laurent Younes 27 Attainable Diffeomorphisms

• Computing d ( ψ , ψ ′ ) = d ( i d ,ψ ( ψ ′ ) − 1 ) is equivalent to V V ! solving the optimal control problem

1 2 ‖v(t)‖V dt → min ∫0

subject to ϕ(0) = id, ∂ ϕ = v(t,ϕ), ϕ(1) =ψ (ψ ′)−1 t !

Geilo, January 2014 Laurent Younes 28 Riemannian Interpretation

d • V is the distance associated to the Riemannian metric −1 δϕ = δϕ !ϕ ϕ V −1 since, using ϕ ! " ϕ = v ,

⎧ 1 ⎫ d (ψ ,ψ ′) = min ‖ϕ!(t)‖ dt :ϕ(1) =ψ ′,ϕ(0) =ψ V ⎨∫ 0 ϕ ⎬ ⎩ ⎭ 1/2 ⎧ 1 ⎫ = min ‖ϕ!(t)‖2 dt :ϕ(1) =ψ ′,ϕ(0) =ψ ⎨∫ 0 ϕ ⎬ ⎩ ⎭

Geilo, January 2014 Laurent Younes 29 Differentiable Manifold

• Let M be a Hausdorff topological space. An n- dimensional local chart on M is a pair ( U , Φ ) where U is n open in M and Φ : U → V ⊂ ! is a homeomorphism. (U , ),(U , ) p • Two charts 1 Φ 1 2 Φ 2 are C -compatible if either U ∩ U = ∅ or Φ ! Φ − 1 is C p on Φ (U ∩U ) ⊂ n 1 2 2 1 1 1 2 ! • M is an n-dimensional C p manifold if it can be covered with local charts that are all pairwise C p compatible (which form an atlas).

Geilo, January 2014 Laurent Younes 30 Something like this

Geilo, January 2014 Laurent Younes 31 Tangent Vectors on Manifolds Tangent Vectors on a differentiable manifold are velocities of trajectories in the manifold. They represent infinitesimal displacements.

The collection of all tangent vectors at a given point is an n- dimensional vector space, called the tangent space at this point.

Geilo, January 2014 Laurent Younes 32 Riemannian Manifold

• A Riemannian manifold is a differentiable manifold with an inner product on each of its tangent spaces.

p ∈ M, v tangent to M at p → v p

Geilo, January 2014 Laurent Younes 33 Riemannian distance

• Adding the norms of infinitesimal displacements, one can compute the lengths of trajectories on a manifold. • The Riemannian distance is the length of the shortest path between two points: 1 d( p, p') = inf x!(t) dt : x(0) = p,x(1) = p' {∫ 0 x(t) } • Curves that achieve the shortest length are called (minimizing) geodesics. They extend the notion of straight lines to manifolds.

Geilo, January 2014 Laurent Younes 34 Back to Diffeomorphisms

• Manifold: group G of diffeomorphisms. It is an open space of a Banach or Fréchet space and has infinite dimension. • Infinitesimal displacements, δ ϕ , are vector fields. • The Riemannian metric is

δϕ = δϕ !ϕ −1 ϕ V

Geilo, January 2014 Laurent Younes 35 Exponential Charts

• On a Riemannian manifold, one can define radial coordinates or exponential charts. • Example: (Latitude, Longitude) on Earth. – Meridian lines stemming from the North Pole are geodesics. – A point on Earth is measured by specifying which meridian it belongs to (Longitude) and where it is on this meridian (latitude).

Geilo, January 2014 Laurent Younes 36 Exponential Charts

• Geodesics t ! γ ( t ) must satisfy a second-order differential equation. • They are uniquely defined by their initial conditions (γ (0),γ!(0)) • Definition: exp (v) = γ (1) p where γ is the geodesic with initial conditions γ (0) = p,γ!(0) = v is the exponential map on the manifold.

Geilo, January 2014 Laurent Younes 37 Exponential Charts

• Fixing p, the function v ! e xp ( v ) is a local chart p mapping a neighborhood of 0 in the tangent space to p to a neighborhood of p on the manifold. • It is called the exponential chart. • (On the sphere, longitude provides the direction of v and latitude provides the norm.) • The inverse map: p ' v such that p ' = e xp ( v ) is defined → p in a neighborhood of p and provides exponential coordinates (Riemannian logarithm).

Geilo, January 2014 Laurent Younes 38 Application to Diffeomorphisms

• Take M = G, a group of diffeomorphisms, p = i d , the identity map. • The Riemannian logarithm of ψ is obtained by solving the optimal control problem 1 2 ‖v(t)‖V dt → min ∫0 subject to ϕ(0) = id, ∂ ϕ = v ϕ, ϕ(1) =ψ . t !

• The logarithm is then given by v ( 0 ) .

Geilo, January 2014 Laurent Younes 39 EPDiff

• The geodesic equation on diffeomorphisms is called EPDiff. • It expresses the conservation of “momentum” along optimal trajectories.

⎪⎧∂tϕ = v !ϕ ⎨ ∂ A v + ad*A v = 0 ⎩⎪ t V v V where ad : w → Dv w − Dww v (id,v ) • Solving this equation with initial conditions 0 provides a local chart of G around the identity.

Geilo, January 2014 Laurent Younes 40 Shape Spaces

• Assume that the shape space M is an open subset of a Q . • Assume that diffeomorphisms act on shapes, with notation: (ϕ,q)  ϕ ⋅q • Thus: id ⋅q = q ϕ ⋅(ψ ⋅q) = (ϕ !ψ )⋅q

Geilo, January 2014 Laurent Younes 41 Infinitesimal Action

• The infinitesimal action of vector fields: (v,q)  v ⋅q is defined by

v ⋅q := ∂ (ϕ ⋅q) , ϕ = id, ( ) v ε ε =0 0 ∂εϕ ε=0 =

• Let ξ : v  v ⋅ q . q • We assume that, for all q ∈ M , ξ : V → Q is well defined q and bounded.

Geilo, January 2014 Laurent Younes 42 Riemannian Metric on Shapes

• Define a metric on M via “Riemannian submersion”, yielding

δq = inf v :ξ v = δq q { V q } • The associated distance is then given by d (q ,q ) = inf d (id,ϕ) :ϕ ⋅q = q M 0 1 { G 0 1}

1 • Or: 2 ‖v(t)‖V dt → min ∫0 q(0) = q , ∂ q = ξ v, q(1) = q . subject to 0 t q 1

Geilo, January 2014 Laurent Younes 43 Large Deformation Diffeomorphic Metric Mapping • The generic “LDDMM” problem is

1 2 1 v(t,.) +U(q(1)) → min 2 ∫0‖ ‖V

subject to q(0) = q(0) and q!(t) = ξ v(t) q(t) • This is an infinite-dimensional optimal control problem, with v as control, q as state and U an end-point cost assumed to be differentiable. • Typically, U measures the discrepancy between q(1) and the “target” shape q . 1

Geilo, January 2014 Laurent Younes 44 Geodesics equations

• Optimal paths must satisfy * ⎧∂t q = ξqKV ξq p ⎪ ⎪ * ⎨∂t p + (∂q ξqv) p = 0 ⎪ * v = K ξ p ⎩⎪ V q with q(0) = q , p(0) p . 0 = 0 for some p M * . 0 ∈ • The mapping p q ( 1 ) provides a coordinate system 0 → equivalent to the exponential chart.

Geilo, January 2014 Laurent Younes 45 Example: Parametrized sets

• q: continuous embedding of S (Riemannian manifold) in  d with ϕ ⋅q = ϕ  q ξ v = v  q q • If p is a measure on S, ξ * a = q * a and q

* (ξ K ξ p)(x) = K (q(x),q( y))dp( y) q V q ∫S V

Geilo, January 2014 Laurent Younes 46 Special case: Point Sets / Landmarks

q (q , ,q ) d • = 1 … N is a finite set of distinct points in  . q ( (q ), , (q )) ϕ ⋅ = ϕ 1 … ϕ N ξ v = (v(q ),…,v(q )) q 1 N N * • Q = Q * = (  d ) N ξ p = p δ q ∑ k qk k=1

• Then N ( K * p) K (q ,q ) p ξq V ξq k = ∑ V k l l l=1

Geilo, January 2014 Laurent Younes 47 Point Set Matching LDDMM Problem

• Plug into the generic problem:

1 2 1 v(t,.) +U(q(1)) → min 2 ∫0‖ ‖V

subject to q(0) = q(0) and q!(t) = ξ v(t) q(t) the constraint that v ( t , ⋅) = K ξ * p ( t ) for some V q(t) N p(t) = p (t) { k }k=1

N • One then has 2 v(t) = pT K (q ,q ) p V ∑ k V k l l k,l=1

Geilo, January 2014 Laurent Younes 48 Reduction

• The problem becomes finite dimensional: N 1 1 p (t)T K (q (t),q (t)) p (t)dt +U(q(1)) → min 2 ∫0 ∑ k V k l l k,l=1 N subject to q(0) q(0) and q (t) K (q (t),q (t)) p (t) = !k = ∑ V k l l l=1 • This is a classical optimal control problem with state q and control p. • One can use the adjoint method to solve it numerically.

Geilo, January 2014 Laurent Younes 49 Adjoint Method

• Consider the general optimal control problem 1 g(q(t),u(t))dt +U(q(1)) → min ∫0 (0) subject to q(0) = q and q!(t) = f (q(t),u(t)) u • Consider q as a function of u, say q uniquely defined by

u (0) u u q (0) = q and q! (t) = f (q (t),u(t))

• Let 1 F(u) = g(qu (t),u(t))dt +U(qu (1)) → min ∫0

Geilo, January 2014 Laurent Younes 50 Adjoint Method

• The adjoint methods computes ∇F(u) • Introduce the Hamiltonian H ( p,q) pT f (q,u) g(q,u) u = − • Step 1: Given u, solve for the state equation q! = f (q,u) or q! = ∂ H p u

• Step 2: Set p(1) = −∇U(q(1)) • Step 3: Solve p! = − ∂ H (backward in time) q u • Step 4: Let F(u)(t) H( p(t),q(t)) ∇ = − ∂u

Geilo, January 2014 Laurent Younes 51 Data Terms

• Several types of data terms q ! U ( q ) have been developed depending on the interpretation made for q. • If q ( q , , q ) are labeled landmarks, use = 1 … N N 2 U(q) q q(1) = ∑ k − k k=1 • If q { q , , q } are unlabeled, introduce the measure = 1 … N

N µ = δ q ∑ qk k=1

Geilo, January 2014 Laurent Younes 52 Data Terms

2 • Let U ( q ) = ‖ µ − µ (1 ) ‖ with q q 2 µ = K (x, y)dµ(x)dµ( y) ∫∫ H • This is also applicable to weighted sums of point masses. • Example: discretize the line measure dl along a curve by N µq = (dlk )δ q ∑ k k=1 and use this representation for curve comparison. • Same idea for triangulated surfaces.

Geilo, January 2014 Laurent Younes 53 Data Terms

• For oriented curves and surfaces: use vector measures instead of scalar ones (involving normal vector). • Equivalent to current-based surface comparison (Vaillant- Glaunès). • Other variants have been developed (collections of curves, surfaces to sections,…)

Geilo, January 2014 Laurent Younes 54 Application: tracking tagged MRI

Geilo, January 2014 Laurent Younes 55 Matching curves from tagged MRI data

Geilo, January 2014 Laurent Younes 56 Matching Triangulated Surfaces

Geilo, January 2014 Laurent Younes 57