multilinear algebra for visual geometry

Alberto Ruiz http://dis.um.es/~alberto DIS, University of Murcia

MVIGRO workshop Berlin, june 28th 2013 contents

1. linear spaces 2. tensor diagrams 3. Grassmann algebra 4. multiview geometry 5. tensor equations intro diagrams multiview equations linear algebra

the most important computational tool: linear = solvable

nonlinear problems: linearization + iteration

multilinear problems are easy (' linear)

visual geometry is multilinear (' solved) intro diagrams multiview equations linear spaces

I objects are represented by arrays of coordinates (depending on chosen basis, with proper rules of transformation)

I linear transformations are extremely simple (just multiply and add), and invertible in closed form (in least squares sense)

I duality: linear transformations are also a linear space intro diagrams multiview equations linear spaces

objects are linear combinations of other elements:

i vector x has coordinates x in basis B = {ei}

X i x = x ei i

Einstein’s convention: i x = x ei intro diagrams multiview equations change of basis

 1 1   0 0 c1 c2 3 1 0 i [e1, e2] = [e1, e2]C, C = 2 2 = ej = cj ei c1 c2 1 2

x1 x1 x01 x = x1e + x2e = [e , e ] = [e , e ]C C−1 = [e0 , e0 ] 1 2 1 2 x2 1 2 x2 1 2 x02 | 0{z 0 } [e1,e2] | {z } x01 x02 intro diagrams multiview equations covariance and contravariance

if the basis transforms according to C:

0 0 [e1, e2] = [e1, e2]C

vector coordinates transform according to C−1:

x01 x1 = C−1 x02 x2

in the previous example: 7 3 1 2 2  0,4 −0,2 7 = , = 4 1 2 1 1 −0,2 0,6 4 intro diagrams multiview equations covariance and contravariance

I things that transform like the basis are covariant (subindex, ‘row’, ‘function-like’)

I things that transform ‘inversely’ are contravariant (superindex, ‘column’, ‘vector-like’)

vector coordinates are contravariant intro diagrams multiview equations scalar linear functions

n the coordinates of linear functions R → R (‘covectors’) are covariant:

T 1 n k f(x) = a x = a1x + . . . anx = akx

0 a Tx0 = aTC C−1x |{z} | {z } a0T x0

0 r ak = ckar intro diagrams multiview equations linear transformations

j in bases {ei} and {gi} the transformation A has coordinates {ai }

j j i y = ai x in bases {e0 = Ce} and {g0 = Dg}: A0 = D−1AC

0t j i t t r r as = csajqi , qi dt = δi

the ‘input’ is covariant and the ‘output’ is contravariant intro diagrams multiview equations bilinear functions

bilinear functions have two covariant (function-like) inputs.

C(u, v) = uTCv

conics (quadratic forms) are similar to bilinear functions:

C(x) = xTCx = 0

line (dual) conics have two contravariant (vector-like) inputs,

and so on. . . intro diagrams multiview equations multilinear functions

several vector and covector arguments, linear in all of them

I application and composition: contraction

I partial application: produces a multilinear function of the remaining arguments

I duality: no fixed input/output role a transformation of vectors transforms covectors in the opposite direction intro diagrams multiview equations matrix notation

too many things are described by raw 2D arrays:

I data sets (e.g. a collection of vectors)

I linear transformations

I bilinear functions, quadratic forms

I homographies, Fundamental Matrix, etc.. . . matrix notation and manipulation tricks are not suitable for multilinear algebra

alternative to algebraic indexed expressions: tensor diagrams intro diagrams multiview equations scalar, invariant

same value s in all bases square edges intro diagrams multiview equations vector

k coordinates x depend on chosen basis {ek}

k x = x ek

contravariant slot: convex, ‘pointed’ edge intro diagrams multiview equations linear function

T 1 n k f(x) = a x = a1x + ... + anx = akx

0 a Tx0 = aTCC−1x

covariant slot: ‘concave’ edge intro diagrams multiview equations function application

T fa(x) = a x

produces an invariant:

index contraction = connecting complementary slots intro diagrams multiview equations transformation

i j i y = FA(x) = Ax = ajx = y

covariant and contravariant slots: intro diagrams multiview equations transformation

works both ways intro diagrams multiview equations dual transformations intro diagrams multiview equations composition intro diagrams multiview equations identity intro diagrams multiview equations bilinear function

e.g. an inner product

given two vectors produces an invariant:

a linear function is the inner product with some vector: intro diagrams multiview equations quadratic forms

exercises:

I conics

I dual conics

I tangent to a conic / polar lines intro diagrams multiview equations self-contraction? intro diagrams multiview equations tensor product

creates a multilinear function from the product of the results of several multilinear functions:

multiply function coordinates before the arguments are applied intro diagrams multiview equations trace

any transformation can be decomposed as (sum of)

there is an invariant: intro diagrams multiview equations contraction

multilinear function application and/or composition tensor product followed by sum of ‘diagonal slice’ of repeated indexes. implemented by ‘matrization’ and normal matrix product intro diagrams multiview equations homogeneous coordinates

projective geometry: geometric objects represented by subspaces

n in P points and hyperplanes are dual points are contravariant, hyperplanes are covariant

2 in P points and lines are easy, e.g. incidence line-point: lT x = 0

3 in P points and planes are easy, lines are awkward we want a uniform representation of points, lines, planes, incidence, etc. for spaces of any dimension intro diagrams multiview equations symmetric tensors

a multilinear function with undistinguished slots

F (..., x,..., y,...) = F (..., y,..., x,...)

commutative function symmetric coordinates intro diagrams multiview equations antisymmetric tensors

F (..., x,..., y,...) = −F (..., y,..., x,...)

F (..., x,..., x,...) = 0

responds only to the linear independent component of the input

F (..., x + y,..., x + z,...) = F (..., y,..., z,...) intro diagrams multiview equations full antisymmetric tensor

Levi-Civita symbol 3 e.g. in R :

when fully applied computes the ‘determinant’ hypervolume of parallelepiped defined by the inputs undistinguished inputs (except sign) extremely useful: universal linear solver intro diagrams multiview equations exterior product

antisymmetrized tensor product

obtains an antisymmetric tensor represents the subspace spanned by the inputs ‘flat’, ‘blade’, ‘r-vector’, ‘multivector’, ‘extensor’ intro diagrams multiview equations exterior product

representation of the subspace, independent of the ‘generators’ (up to scale) intro diagrams multiview equations exterior product

An r-vector can be represented as a dense multilinear array, or as coordinates in suitable basis:

ei ∧ ej = eij

ei ∧ ei = 0

eij = −eji Grassmann Algebra

cf. geometric algebra, Pl¨ucker coordinates, etc. intro diagrams multiview equations example

a 3D line T a = [1, 2, 0, 0] = e1 + 2e2 T b = [0, 3, 0, 4] = 3e2 + 4e3

a ∧ b = 3e12 + 4e14 + 8e24

 0 3 0 4 −3 0 0 8 a ∧ b =    0 0 0 0 −4 −8 0 0 intro diagrams multiview equations the whole space

represented by the full contravariant antisymmetric tensor: intro diagrams multiview equations transformation of r-vectors intro diagrams multiview equations metric tensor

vectors depicted as arrows in space are actually not very meaninful without an inner product, giving distances and angles. intro diagrams multiview equations metric tensor

dual coordinates (raise/lower indexes): intro diagrams multiview equations inner product of r-vectors

induced by the standard inner product for vectors:

what does it mean? intro diagrams multiview equations inner product of r-vectors

obtains the orthogonal complement of the projection

x

a

a ∧ b

(a ∧ b) · x b intro diagrams multiview equations Hodge dual

there is an isomorphism between r-vectors and (n − r)-vectors e.g. dual of a line in 3D space:

inner product with whole space gives ‘complementary’ subspace (x ∧ y is automatically computed by contraction with ) intro diagrams multiview equations cross product

3 in R : intro diagrams multiview equations

[x]×

k [x]× = ijkx = aij

=   a  0 a b b   −a 0 c c −b −c 0 intro diagrams multiview equations dualization

universal tool for geometric constructions:

I ijk... transforms ‘span of points’ into ‘generic intersection of hyperplanes’ ijk... I  intersects (dual) hyperplanes to get ‘span of points’ intro diagrams multiview equations normal vs dual representation

P2

p ∗p l= p1 ∧ p2 *l 3 × 3 3 × 3 3 3

point line P3 π= p1 ∧ p2 ∧ p3 4 × 4 × 4 ∗p p 4 × 4 × 4 l= p1 ∧ p2 ∗l ∗π 4 × 4 4 × 4 4 4

point line plane intro diagrams multiview equations join (∧) and meet (∨)

join meet l l

p π ∨ l

p π ∨ l l l

 is analogous to a NOR logic gate intro diagrams multiview equations cross product trick in P2

p1 p ∗l p 2 1 ∗l2 ∗l

p1 ∗l1 ∗l2 ∗l

p p 2 intro diagrams multiview equations incidence

geometric objects Q = a ∧ b ∧ c ∧ ... and Q = x ∧ y ∧ ... P is in Q:

I Q ∧ P = 0

I ∗Q · P = PQ = 0 example: incidence line (a ∧ b) - point (x)

3 P2 P P4 = 0

ijk 03 03×3 intro diagrams multiview equations rule 1

dual of dual = identity (up to scale) intro diagrams multiview equations rule 2

transformation of whole space = whole space intro diagrams multiview equations inversion intro diagrams multiview equations preimage

point reprojection

line reprojection intro diagrams multiview equations nullspace

the preimage of nothing intro diagrams multiview equations rank deficient

nullspace intro diagrams multiview equations rank deficient

generalization of rule 2 intro diagrams multiview equations triangulation intro diagrams multiview equations triangulation

graph simplification (rule 1) intro diagrams multiview equations homography induced by a plane

(same circuit!)

π

x0 x

changing input/ouput role of slots intro diagrams multiview equations two-view geometry

Fundamental Matrix: gives image of the optical ray

preimage + projection + dual intro diagrams multiview equations two-view geometry

F

l = Fx

i 0j x Fij x = 0 x0TFx = 0 intro diagrams multiview equations algebraic expressions

line joining the epipole and the image of any point in the ray

0 + F = [e ]×NM

needs epipole and pseudoinverse

alternative: multilinear expression based just on contractions

a b pqrs c d Fij = iabMp Mq  Nr Ns cdj intro diagrams multiview equations three-view geometry

epipolar transfer intro diagrams multiview equations three-view geometry

l = T (l0, l00) intro diagrams multiview equations trifocal tensor

image of the intersection of the planes reprojected from two lines: 0 00 I preimages of l and l

I ‘meet’ of those planes

I projection onto 3rd view intro diagrams multiview equations trifocal tensor

ij i j pqrs c d Tk = MpNq  Pr Ps cdk intro diagrams multiview equations trifocal transfer intro diagrams multiview equations trifocal incidence intro diagrams multiview equations trifocal incidence intro diagrams multiview equations four views

3 half-points gives the image in fourth view

x intro diagrams multiview equations quadrifocal tensor

the image of the intersection of three reprojected planes

pqrs i j pqrs c d Q = ApBq  Cr Ds intro diagrams multiview equations diagram manipulation

example: camera extraction from F intro diagrams multiview equations intro diagrams multiview equations intro diagrams multiview equations tensor diagrams

advantages:

I intuitive derivation

I intuitive usage

computation:

I graph reduction rules

I tensor equations intro diagrams multiview equations factorization

most linear problems can be solved by matrix factorization: composition of easily invertible transformations:

I QR

I CHOL

I EIG

I SVD

I ... intro diagrams multiview equations SVD

VT S A = U

I linear systems

I overconstrained (least squares)

I underconstrained (minimum norm)

I nullspace, range, rank, condition number, matrix norms

I optimal projection to rank-r manifold, or SO(n), etc.

I low rank factorization intro diagrams multiview equations many applications

I dimension reduction (PCA)

I orthogonal procrustes

I collaborative filtering (matrix completion)

I latent semantic analysis (document-term, bag of words)

I google’s pageRank (eigenvector of hyperlinks)

I . . . everything intro diagrams multiview equations linear equations

standard linear system (several right-hand sides)

X

A = B

X = A+B = VS+UTB

X = A \ B intro diagrams multiview equations least squares

Ax = USVTx = b

‘rotate’ the linear system into a simple form y = VTx, c = UTb

S y c =

0 r

||Ax − b|| = ||r|| intro diagrams multiview equations tensor equations

multidimensional system AX = B

kl xpq

= aijkl bijpq

can be flattened to a standard system

I ij equations

I kl variables

I pq right-hand sides intro diagrams multiview equations example

given a trifocal tensor, we can linearly extract one camera given the others:

i pqrs c d j ij (Mp Pr Ps cdk)Nq = Tk

= T intro diagrams multiview equations tensor equations

multilinear-multidimensional system AXY Z ... = B

X Y

Z = A B

solved by alternating least squares intro diagrams multiview equations camera extraction from T

=

=

(to force equal solution) intro diagrams multiview equations tensor factorizations

HOSVD: orthogonal core Tucker decomposition: sum of rank-1 components intro diagrams multiview equations homogeneous system

Ax = 0

U S VT

x A = r

||Ax|| = r intro diagrams multiview equations multidimensional homogeneous system

similar to a linear model for a data set

T l xi = 0

we can fit a multilinear model: uvw... I the data set is a multidimensional block Dn of tensor products of the vector variables in each observation

I the model tensor should produce a multidimensional zero for each data item intro diagrams multiview equations multidimensional homogeneous system

linear computation of F

uv Dn Fuv = 0n intro diagrams multiview equations multidimensional homogeneous system

linear computation of T

uvw ij Dn vipwjq Tu = 0npq | u{z } Anipjq intro diagrams multiview equations model structure

naive equivalent of F for any number of views views elements rank 2 9 8 3 27 17 4 81 31 5 243 51 6 729 78 7 2187 113 ...... intro diagrams multiview equations raw transfer

points required to ‘learn’ a ‘transfer’ machine views elements rank points 3 27 22 11 4 81 44 22 5 243 77 38.5 6 729 123 61.5 7 2187 184 92 ...... intro diagrams multiview equations homogeneous equality

projective relations (equality up to scale) reduce to standard homogeneous systems:

Ax ∝ b

 X

A B = 0 intro diagrams multiview equations multiview geometry - SfM intro diagrams multiview equations multiview geometry - SfM

bilinear homogenous system:

v v v p Inc = CcpPn intro diagrams multiview equations practical issues

I noise

I outliers

I missing data

I deformations / several motions

I sparsity / big data

I metric solutions

experimental computational tool: hTensor see also Matlab Tensor toolbox, etc. intro diagrams multiview equations references

I A. Ruiz, P.E. L´opez de Teruel. Diagram Techniques for Multiple View Geometry. ICCV 2009.

I hTensor library (github)

I Dorst, Fontijne, Mann. Geometric Algebra for Computer Science: an OO approach. Morgan Kauffmann 2007