Spectrum and Pseudospectrum of a Tensor Lek-Heng Lim University of California, Berkeley July 11, 2008 L.-H. Lim (Berkeley) Spectrum and Pseudospectrum of a Tensor July 11, 2008 1 / 27 Matrix eigenvalues and eigenvectors One of the most important ideas ever invented. I R. Coifman et. al.: \Eigenvector magic: eigenvectors as an extension of Newtonian calculus." Normal/Hermitian A I Invariant subspace: Ax = λx. > > I Rayleigh quotient: x Ax=x x. > 2 I Lagrange multipliers: x Ax − λ(kxk − 1). > I Best rank-1 approximation: minkxk=1kA − λxx k. Nonnormal A −1 −1 I Pseudospectrum: σ"(A) = fλ 2 C j k(A − λI ) k > " g. ∗ I Numerical range: W (A) = fx Ax 2 C j kxk = 1g. ∗ I Irreducible representations of C (A) with natural Borel structure. ∗ I Primitive ideals of C (A) with hull-kernel topology. How can one define these for tensors? L.-H. Lim (Berkeley) Spectrum and Pseudospectrum of a Tensor July 11, 2008 2 / 27 DARPA mathematical challenge eight One of the twenty three mathematical challenges announced at DARPA Tech 2007. Problem Beyond convex optimization: can linear algebra be replaced by algebraic geometry in a systematic way? Algebraic geometry in a slogan: polynomials are to algebraic geometry what matrices are to linear algebra. Polynomial f 2 R[x1;:::; xn] of degree d can be expressed as > > f (x) = a0 + a1 x + x A2x + A3(x; x; x) + ··· + Ad (x;:::; x): n n×n n×n×n n×···×n a0 2 R; a1 2 R ; A2 2 R ; A3 2 R ;:::; Ad 2 R . Numerical linear algebra: d = 2. Numerical multilinear algebra: d > 2. L.-H. Lim (Berkeley) Spectrum and Pseudospectrum of a Tensor July 11, 2008 3 / 27 Hypermatrices Totally ordered finite sets: [n] = f1 < 2 < ··· < ng, n 2 N. Vector or n-tuple f :[n] ! R: > n If f (i) = ai , then f is represented by a = [a1;:::; an] 2 R . Matrix f :[m] × [n] ! R: m;n m×n If f (i; j) = aij , then f is represented by A = [aij ]i;j=1 2 R . Hypermatrix (order 3) f :[l] × [m] × [n] ! R: l;m;n l×m×n If f (i; j; k) = aijk , then f is represented by A = aijk i;j;k=1 2 R . J K X [n] [m]×[n] [l]×[m]×[n] Normally R = ff : X ! Rg. Ought to be R ; R ; R . L.-H. Lim (Berkeley) Spectrum and Pseudospectrum of a Tensor July 11, 2008 4 / 27 Hypermatrices and tensors Up to choice of bases n a 2 R can represent a vector in V (contravariant) or a linear functional in V ∗ (covariant). m×n A 2 R can represent a bilinear form V × W ! R (contravariant), ∗ ∗ a bilinear form V × W ! R (covariant), or a linear operator V ! W (mixed). l×m×n A 2 R can represent trilinear form U × V × W ! R (contravariant), bilinear operators V × W ! U (mixed), etc. A hypermatrix is the same as a tensor if 1 we give it coordinates (represent with respect to some bases); 2 we ignore covariance and contravariance. L.-H. Lim (Berkeley) Spectrum and Pseudospectrum of a Tensor July 11, 2008 5 / 27 Basic operation on a hypermatrix m×n A matrix can be multiplied on the left and right: A 2 R , p×m q×n X 2 R , Y 2 R , > p×q (X ; Y ) · A = XAY = [cαβ] 2 R where Xm;n cαβ = xαi yβj aij : i;j=1 l×m×n A hypermatrix can be multiplied on three sides: A = aijk 2 R , p×l q×m r×n J K X 2 R , Y 2 R , Z 2 R , p×q×r (X ; Y ; Z) ·A = cαβγ 2 R J K where Xl;m;n cαβγ = xαi yβj zγk aijk : i;j;k=1 L.-H. Lim (Berkeley) Spectrum and Pseudospectrum of a Tensor July 11, 2008 6 / 27 Numerical range of a tensor Covariant version: A· (X >; Y >; Z >) := (X ; Y ; Z) ·A: Gives convenient notations for multilinear functionals and multilinear l m n operators. For x 2 R ; y 2 R ; z 2 R , Xl;m;n A(x; y; z) := A· (x; y; z) = aijk xi yj zk ; i;j;k=1 Xm;n A(I ; y; z) := A· (I ; y; z) = aijk yj zk : j;k=1 n×n Numerical range of square matrix A 2 C , ∗ ∗ W (A) = fx Ax 2 C j kxk2 = 1g = fA(x; x ) 2 C j kxk2 = 1g: n×···×n Plausible generalization to cubical hypermatrix A 2 C , ( fA(x; x∗;:::; x) 2 j kxk = 1g odd order; W (A) = C k ∗ ∗ fA(x; x ;:::; x ) 2 C j kxkk = 1g even order: L.-H. Lim (Berkeley) Spectrum and Pseudospectrum of a Tensor July 11, 2008 7 / 27 Symmetric hypermatrices n×n×n Cubical hypermatrix aijk 2 R is symmetric if J K aijk = aikj = ajik = ajki = akij = akji : Invariant under all permutations σ 2 Sk on indices. k n S (R ) denotes set of all order-k symmetric hypermatrices. Example Higher order derivatives of multivariate functions. Example Moments of a random vector x = (X1;:::; Xn): Z Z n n mk (x) = E(xi1 xi2 ··· xik ) = ··· xi1 xi2 ··· xik dµ(xi1 ) ··· dµ(xik ) : i1;:::;ik =1 i1;:::;ik =1 L.-H. Lim (Berkeley) Spectrum and Pseudospectrum of a Tensor July 11, 2008 8 / 27 Symmetric hypermatrices Example Cumulants of a random vector x = (X1;:::; Xn): 2 3n X p−1 Q Q κk (x) = 4 (−1) (p − 1)!E xi ··· E xi 5 : i2A1 i2Ap A1t···tAp =fi1;:::;ik g i1;:::;ik =1 For n = 1, κk (x) for k = 1; 2; 3; 4 are the expectation, variance, skewness, and kurtosis. Important in Independent Component Analysis (ICA). L.-H. Lim (Berkeley) Spectrum and Pseudospectrum of a Tensor July 11, 2008 9 / 27 Multilinear spectral theory n×n×n Let A 2 R (easier if A symmetric). 1 How should one define its eigenvalues and eigenvectors? 2 What is a decomposition that generalizes the eigenvalue decomposition of a matrix? l×m×n Let A 2 R 1 How should one define its singualr values and singular vectors? 2 What is a decomposition that generalizes the singular value decomposition of a matrix? Somewhat surprising: (1) and (2) have different answers. L.-H. Lim (Berkeley) Spectrum and Pseudospectrum of a Tensor July 11, 2008 10 / 27 Tensor ranks (Hitchcock, 1927) m×n Matrix rank. A 2 R . rank(A) = dim(span fA ;:::; A g) (column rank) R •1 •n = dim(span fA ;:::; A g) (row rank) R 1• m• Pr T = minfr j A = i=1ui vi g (outer product rank): Multilinear rank. A 2 l×m×n. rank (A) = (r (A); r (A); r (A)), R 1 2 3 r (A) = dim(span fA ;:::; A g) 1 R 1•• l•• r (A) = dim(span fA ;:::; A g) 2 R •1• •m• r (A) = dim(span fA ;:::; A g) 3 R ••1 ••n l×m×n Outer product rank. A 2 R . Pr rank⊗(A) = minfr j A = i=1ui ⊗ vi ⊗ wi g l;m;n where u ⊗ v ⊗ w : = ui vj wk i;j;k=1. J K L.-H. Lim (Berkeley) Spectrum and Pseudospectrum of a Tensor July 11, 2008 11 / 27 Matrix EVD and SVD Rank revealing decompositions. 2 n Symmetric eigenvalue decomposition of A 2 S (R ), > Xr A = V ΛV = λi vi ⊗ vi ; i=1 where rank(A) = r, V 2 O(n) eigenvectors, Λ eigenvalues. m×n Singular value decomposition of A 2 R , > Xr A = UΣV = σi ui ⊗ vi i=1 where rank(A) = r, U 2 O(m) left singular vectors, V 2 O(n) right singular vectors, Σ singular values. L.-H. Lim (Berkeley) Spectrum and Pseudospectrum of a Tensor July 11, 2008 12 / 27 One plausible EVD and SVD for hypermatrices Rank revealing decompositions associated with the outer product rank. 3 n Symmetric outer product decomposition of A 2 S (R ), Xr A = λi vi ⊗ vi ⊗ vi i=1 where rankS(A) = r, vi unit vector, λi 2 R. l×m×n Outer product decomposition of A 2 R , Xr A = σi ui ⊗ vi ⊗ wi i=1 l m n where rank⊗(A) = r, ui 2 R ; vi 2 R ; wi 2 R unit vectors, σi 2 R. L.-H. Lim (Berkeley) Spectrum and Pseudospectrum of a Tensor July 11, 2008 13 / 27 Another plausible EVD and SVD for hypermatrices Rank revealing decompositions associated with the multilinear rank. l×m×n Singular value decomposition of A 2 R , A = (U; V ; W ) ·C where rank (A) = (r ; r ; r ), U 2 l×r1 , V 2 m×r2 , W 2 n×r3 1 2 3 R R R r ×r ×r have orthonormal columns and C 2 R 1 2 3 . 3 n Symmetric eigenvalue decomposition of A 2 S (R ), A = (U; U; U) ·C where rank (A) = (r; r; r), U 2 n×r has orthonormal columns and R 3 r C 2 S (R ). L.-H. Lim (Berkeley) Spectrum and Pseudospectrum of a Tensor July 11, 2008 14 / 27 Variational approach to eigenvalues/vectors m×n A 2 R symmetric. Eigenvalues and eigenvectors are critical values and critical points of > 2 x Ax=kxk2: Equivalently, critical values/points of x>Ax constrained to unit sphere. Lagrangian: > 2 L(x; λ) = x Ax − λ(kxk2 − 1): n Vanishing of rL at critical (xc ; λc ) 2 R × R yields familiar Axc = λc xc : L.-H. Lim (Berkeley) Spectrum and Pseudospectrum of a Tensor July 11, 2008 15 / 27 Variational approach to singular values/vectors m×n A 2 R .