EIGENVECTORS OF TENSORS Bernd Sturmfels University of California at Berkeley

How many eigenvectors does a 3 × 3 × 3-tensor have? How many singular vector triples does a 3×3×3-tensor have? A tensor has rank r if it is the sum of r tensors of rank 1. (not fewer).

Tensor decomposition:

I Express a given tensor as a sum of rank 1 tensors.

I Use as few summands as possible.

Textbook: JM Landsberg: Tensors: Geometry and Applications, 2012.

Tensors and their rank

A tensor is a d-dimensional array of numbers T = (ti1i2···id ). For d = 1 this is a vector, and for d = 2 this is a matrix. n ×n ×···×n The tensor space R 1 2 d has dimension n1n2 ··· nd . T has rank 1 if it is the outer product of d vectors u, v,..., w:

ti1i2···id = ui1 vi2 ··· wid .

The set of tensors of rank 1 is the Segre variety. We often use C. Tensors and their rank

A tensor is a d-dimensional array of numbers T = (ti1i2···id ). For d = 1 this is a vector, and for d = 2 this is a matrix. n ×n ×···×n The tensor space R 1 2 d has dimension n1n2 ··· nd . T has rank 1 if it is the outer product of d vectors u, v,..., w:

ti1i2···id = ui1 vi2 ··· wid .

The set of tensors of rank 1 is the Segre variety. We often use C.

A tensor has rank r if it is the sum of r tensors of rank 1. (not fewer).

Tensor decomposition:

I Express a given tensor as a sum of rank 1 tensors.

I Use as few summands as possible.

Textbook: JM Landsberg: Tensors: Geometry and Applications, 2012. Open Problem [Comon’s Conjecture] Is the rank of every symmetric tensor equal to its rank as a general tensor?

True for d = 2: every rank 1 decomposition of a symmetric matrix t t t T = u1v1 + u2v2 + ··· + ur vr . transforms into a decomposition into rank 1 symmetric matrices: t t t T = v1v1 + v2v2 + ··· + vr vr

Symmetric tensors

An n×n× · · · ×n-tensor T = (ti1i2···id ) is symmetric if it is unchanged under permuting indices. The vector space n n+d−1
Symd (R ) of symmetric tensors has dimension d . T has rank 1 if it is the d-fold outer product of a vector v:

ti1i2···id = vi1 vi2 ··· vid . The set of symmetric tensors of rank 1 is the Veronese variety.

A symmetric tensor has rank r if it is the sum of r such tensors. Symmetric tensors

An n×n× · · · ×n-tensor T = (ti1i2···id ) is symmetric if it is unchanged under permuting indices. The vector space n n+d−1
Symd (R ) of symmetric tensors has dimension d . T has rank 1 if it is the d-fold outer product of a vector v:

ti1i2···id = vi1 vi2 ··· vid . The set of symmetric tensors of rank 1 is the Veronese variety.

A symmetric tensor has rank r if it is the sum of r such tensors.

Open Problem [Comon’s Conjecture] Is the rank of every symmetric tensor equal to its rank as a general tensor?

True for d = 2: every rank 1 decomposition of a symmetric matrix t t t T = u1v1 + u2v2 + ··· + ur vr . transforms into a decomposition into rank 1 symmetric matrices: t t t T = v1v1 + v2v2 + ··· + vr vr The gradient of T defines a polynomial map of degree d − 1:

n n ∇T : R → R .

n A vector v ∈ R is an eigenvector of the tensor T if

(∇T )(v) = λ · v for some λ ∈ R.

Polynomials and their eigenvectors Symmetric tensors correspond to homogeneous polynomials

n X T = ti1i2···id · xi1 xi2 ··· xid i1,...,id =1 The tensor has rank r if T is a sum of r powers of linear forms:

r r X ⊗d X d T = λj vj = λj (v1j x1 + v2j x2 + ··· + vnj xn) . j=1 j=1 Polynomials and their eigenvectors Symmetric tensors correspond to homogeneous polynomials

n X T = ti1i2···id · xi1 xi2 ··· xid i1,...,id =1 The tensor has rank r if T is a sum of r powers of linear forms:

r r X ⊗d X d T = λj vj = λj (v1j x1 + v2j x2 + ··· + vnj xn) . j=1 j=1

The gradient of T defines a polynomial map of degree d − 1:

n n ∇T : R → R .

n A vector v ∈ R is an eigenvector of the tensor T if

(∇T )(v) = λ · v for some λ ∈ R. n−1 n−1 In projective geometry we prefer ∇T : P 99K P . Eigenvectors of T are fixed points( λ 6= 0) or base points( λ = 0).

n−1 Fact: These are nonlinear dynamical systems on P .

[Lim, Ng, Qi: The spectral theory of tensors and its applications, 2013]

What is this good for?

Consider the optimization problem of maximizing a n homogeneous polynomial T over the unit sphere in R . Lagrange multipliers lead to the equations

(∇T )(v) = λ · v for some λ ∈ R.

Fact: The critical points are the eigenvectors of T . What is this good for?

Consider the optimization problem of maximizing a n homogeneous polynomial T over the unit sphere in R . Lagrange multipliers lead to the equations

(∇T )(v) = λ · v for some λ ∈ R.

Fact: The critical points are the eigenvectors of T .

n−1 n−1 In projective geometry we prefer ∇T : P 99K P . Eigenvectors of T are fixed points( λ 6= 0) or base points( λ = 0).

n−1 Fact: These are nonlinear dynamical systems on P .

[Lim, Ng, Qi: The spectral theory of tensors and its applications, 2013] Linear maps

Real symmetric n×n-matrices (tij ) correspond to quadratic forms

n n X X T = tij xi xj i=1 j=1

By the Spectral Theorem, there exists a real decomposition

r X 2 T = λj (v1j x1 + v2j x2 + ··· + vnj xn) . j=1

Here r is the rank and the λj are the eigenvalues of T . The eigenvectors vj = (v1j , v2j ,..., vnj ) are orthonormal. One can compute this decomposition by the Power Method:

n−1 n−1 Iterate the linear map ∇T : P 99K P Fixed points of this dynamical system are eigenvectors of T . Quadratic maps

Symmetric n×n×n-tensors (tijk ) correspond to cubic forms

n n n X X X T = tijk xi xj xk i=1 j=1 k=1

We are interested in low rank decompositions

r X 3 T = λj (v1j x1 + v2j x2 + ··· + vnj xn) . j=1

One idea to find this decomposition is the Tensor Power Method:

n−1 n−1 Iterate the quadratic map ∇T : P 99K P Fixed points of this dynamical system are eigenvectors of T .

Bad News: The eigenvectors are usually not the vectors vi in the low rank decomposition ... unless the tensor is odeco. n+1
The set of odeco tensors is a variety of dimension 2 in n Symd (C ). Its defining polynomial equations are very nice. [Robeva: Orthogonal decomposition of symmetric tensors, 2015]

Odeco tensors A symmetric tensor T is odeco (= orthogonally decomposable) if

n n X ⊗d X d T = λj vj = λj (v1j x1 + ··· + vnj xn) , j=1 j=1

n where {v1, v2,..., vn} is an orthogonal basis of R . The tensor power method works well for odeco tensors: Theorem If λj > 0 then the vi are precisely the robust eigenvectors of T . [Anandkumar, Ge, Hsu, Kakade, Telgarsky: Tensor decompositions for learning latent variable models, J. Machine Learning Research, 2014] [Kolda: Symmetric orthogonal tensor decomposition is trivial, 2015] Odeco tensors A symmetric tensor T is odeco (= orthogonally decomposable) if

n n X ⊗d X d T = λj vj = λj (v1j x1 + ··· + vnj xn) , j=1 j=1

n where {v1, v2,..., vn} is an orthogonal basis of R . The tensor power method works well for odeco tensors: Theorem If λj > 0 then the vi are precisely the robust eigenvectors of T . [Anandkumar, Ge, Hsu, Kakade, Telgarsky: Tensor decompositions for learning latent variable models, J. Machine Learning Research, 2014] [Kolda: Symmetric orthogonal tensor decomposition is trivial, 2015]

n+1
The set of odeco tensors is a variety of dimension 2 in n Symd (C ). Its defining polynomial equations are very nice. [Robeva: Orthogonal decomposition of symmetric tensors, 2015] Fermat: Odeco tensor : T = x3 + y 3 + z3

2 2 2 2 2 ∇T : P 99K P , (x : y : z) 7→ (x : y : z ) (1 : 0 : 0) , (0 : 1 : 0) , (0 : 0 : 1) , (1 : 1 : 0) , (1 : 0 : 1) , (0 : 1 : 1) , (1 : 1 : 1) Answer: Seven.

Our question How many eigenvectors does a symmetric 3 × 3 × 3-tensor have ?

How many critical points does a cubic have on the unit 2-sphere? Our question How many eigenvectors does a symmetric 3 × 3 × 3-tensor have ?

How many critical points does a cubic have on the unit 2-sphere?

Fermat: Odeco tensor : T = x3 + y 3 + z3

2 2 2 2 2 ∇T : P 99K P , (x : y : z) 7→ (x : y : z ) (1 : 0 : 0) , (0 : 1 : 0) , (0 : 0 : 1) , (1 : 1 : 0) , (1 : 0 : 1) , (0 : 1 : 1) , (1 : 1 : 1) Answer: Seven. Open Problem: Can all eigenvectors be real? Yes, if n = 3: All 1 + (d−1) + (d−1)2 fixed points can be real.

Proof: Let T be a product of d linear forms. d
The 2 vertices of the line arrangement are the base points. d
The analytic centers of the 2 + 1 regions are the fixed points.

Let’s count Theorem n Consider a general symmetric tensor T ∈ Symd (R ). n−1 The number of complex eigenvectors in P equals

n−1 (d − 1)n − 1 X = (d − 1)i . d − 2 i=0

[Cartwright, St: The number of eigenvalues of a tensor, 2013] [Fornaess, Sibony: Complex dynamics in higher dimensions, 1994]

Classical: Isologues, Laguerre Nets (Thanks, Igor) Proof: Let T be a product of d linear forms. d
The 2 vertices of the line arrangement are the base points. d
The analytic centers of the 2 + 1 regions are the fixed points.

Let’s count Theorem n Consider a general symmetric tensor T ∈ Symd (R ). n−1 The number of complex eigenvectors in P equals

n−1 (d − 1)n − 1 X = (d − 1)i . d − 2 i=0

[Cartwright, St: The number of eigenvalues of a tensor, 2013] [Fornaess, Sibony: Complex dynamics in higher dimensions, 1994]

Classical: Isologues, Laguerre Nets (Thanks, Igor)

Open Problem: Can all eigenvectors be real? Yes, if n = 3: All 1 + (d−1) + (d−1)2 fixed points can be real. Let’s count Theorem n Consider a general symmetric tensor T ∈ Symd (R ). n−1 The number of complex eigenvectors in P equals

n−1 (d − 1)n − 1 X = (d − 1)i . d − 2 i=0

[Cartwright, St: The number of eigenvalues of a tensor, 2013] [Fornaess, Sibony: Complex dynamics in higher dimensions, 1994]

Classical: Isologues, Laguerre Nets (Thanks, Igor)

Open Problem: Can all eigenvectors be real? Yes, if n = 3: All 1 + (d−1) + (d−1)2 fixed points can be real.

Proof: Let T be a product of d linear forms. d
The 2 vertices of the line arrangement are the base points. d
The analytic centers of the 2 + 1 regions are the fixed points. Discriminant The eigendiscriminant is the irreducible polynomial in the entries

ti1i2···id which vanishes when two eigenvectors come together. Theorem The degree of eigendiscriminant is n(n − 1)(d − 1)n−1.

[Abo, Seigal, St: Eigenconfigurations of tensors, 2015]

Example 1 (d = 2) The discriminant of the characteristic polynomial of an n × n-matrix is an equation of degree n(n − 1).

Example 2 (n = d = 3) The eigendiscriminant of

is an equation of degree 24.

[Breiding, B¨urgisser:Distribution of the eigenvalues ...., 2015] Gradient Dynamics: Matrices correspond to bilinear forms

n n X1 X2 T = tij xi yj i=1 j=1

This defines a rational map

n −1 n −1 n −1 n −1 (∇xT, ∇yT ): P 1 × P 2 99K P 1 × P 2 (u , v) 7→ (T t v , T u )

The fixed points of this map are the singular vector pairs of T .

Singular vectors Given a rectangular matrix T , one seeks to solve the equations

T u = σv and T t v = σu.

The scalar σ is a singular value and (u, v) is a singular vector pair. Singular vectors Given a rectangular matrix T , one seeks to solve the equations

T u = σv and T t v = σu.

The scalar σ is a singular value and (u, v) is a singular vector pair.

Gradient Dynamics: Matrices correspond to bilinear forms

n n X1 X2 T = tij xi yj i=1 j=1

This defines a rational map

n −1 n −1 n −1 n −1 (∇xT, ∇yT ): P 1 × P 2 99K P 1 × P 2 (u , v) 7→ (T t v , T u )

The fixed points of this map are the singular vector pairs of T . Theorem For a general n1×n2× · · · ×nd -tensor T , the number of singular n1−1 nd −1 vector tuples is the coefficient of z1 ··· zd in the polynomial

d ni ni Y (zbi ) − zi where zbi = z1 + ··· + zi−1 + zi+1 + ··· + zd . zi − zi i=1 b

[Friedland, Ottaviani: The number of singular vector tuples...., 2014]

Example: d = 3, n1 = n2 = n3 = 3: 2 2 2 2 2 2 2 2 2 (zb1 +zb1z1+z1 )(zb2 +zb2z2+z2 )(zb3 +zb3z3+z3 ) = ··· + 37z1 z2 z3 + ···

Multilinear forms n ×n ×···×n Tensors T in R 1 2 d correspond to multilinear forms. The singular vector tuples of T are fixed points of the gradient map n −1 n −1 n −1 n −1 ∇T : P 1 × · · · × P d 99K P 1 × · · · × P d . Example: d = 3, n1 = n2 = n3 = 3: 2 2 2 2 2 2 2 2 2 (zb1 +zb1z1+z1 )(zb2 +zb2z2+z2 )(zb3 +zb3z3+z3 ) = ··· + 37z1 z2 z3 + ···

Multilinear forms n ×n ×···×n Tensors T in R 1 2 d correspond to multilinear forms. The singular vector tuples of T are fixed points of the gradient map n −1 n −1 n −1 n −1 ∇T : P 1 × · · · × P d 99K P 1 × · · · × P d . Theorem For a general n1×n2× · · · ×nd -tensor T , the number of singular n1−1 nd −1 vector tuples is the coefficient of z1 ··· zd in the polynomial

d ni ni Y (zbi ) − zi where zbi = z1 + ··· + zi−1 + zi+1 + ··· + zd . zi − zi i=1 b

[Friedland, Ottaviani: The number of singular vector tuples...., 2014] Multilinear forms n ×n ×···×n Tensors T in R 1 2 d correspond to multilinear forms. The singular vector tuples of T are fixed points of the gradient map n −1 n −1 n −1 n −1 ∇T : P 1 × · · · × P d 99K P 1 × · · · × P d . Theorem For a general n1×n2× · · · ×nd -tensor T , the number of singular n1−1 nd −1 vector tuples is the coefficient of z1 ··· zd in the polynomial

d ni ni Y (zbi ) − zi where zbi = z1 + ··· + zi−1 + zi+1 + ··· + zd . zi − zi i=1 b

[Friedland, Ottaviani: The number of singular vector tuples...., 2014]

Example: d = 3, n1 = n2 = n3 = 3: 2 2 2 2 2 2 2 2 2 (zb1 +zb1z1+z1 )(zb2 +zb2z2+z2 )(zb3 +zb3z3+z3 ) = ··· + 37z1 z2 z3 + ··· Conclusion Eigenvectors of square matrices are central to linear algebra.

Eigenvectors of tensors are a natural generalization. Pioneered in numerical multilinear algebra, these now have many applications. [Lek-Heng Lim: Singular values and eigenvalues of tensors...., 2005] [Liqun Qi: Eigenvalues of a real supersymmetric tensor, 2005]

Fact: This lecture serves as an invitation to applied algebraic geometry.

Remember: The word variety is not scary.

The terms Segre variety and Veronese variety refer to tensors of rank 1. Given some data, getting close to these is highly desirable. Answers: 1 + (4 − 1) + (4 − 1)2 = 13 = 6 vertices + 7 regions 4 Y 2 2
2 2 2 2 zbi + zbi zi + zi = ··· + 997 · z1 z2 z3 z4 + ··· i=1

300px-Complete-quads.svg.pngPop 300 quiz×127 pixels 07/11/15 09:59

How many eigenvectors does a 3 × 3 × 3 × 3-tensor have? How many singular vector triples does a 3×3×3×3-tensor have?

https://upload.wikimedia.org/wikipedia/commons/thumb/7/7b/Complete-quads.svg/300px-Complete-quads.svg.png Page 1 of 1 300px-Complete-quads.svg.pngPop 300 quiz×127 pixels 07/11/15 09:59

How many eigenvectors does a 3 × 3 × 3 × 3-tensor have? How many singular vector triples does a 3×3×3×3-tensor have?

Answers: 1 + (4 − 1) + (4 − 1)2 = 13 = 6 vertices + 7 regions 4 Y 2 2
2 2 2 2 zbi + zbi zi + zi = ··· + 997 · z1 z2 z3 z4 + ··· i=1

https://upload.wikimedia.org/wikipedia/commons/thumb/7/7b/Complete-quads.svg/300px-Complete-quads.svg.png Page 1 of 1 Advertisement