Eigenvalues of tensors and some very basic spectral hypergraph theory Lek-Heng Lim Matrix Computations and Scientific Computing Seminar April 16, 2008 Lek-Heng Lim (MCSC Seminar) Eigenvalues of tensors April 16, 2008 1 / 26 Hypermatrices Totally ordered finite sets: [n] = f1 < 2 < ··· < ng, n 2 N. Vector or n-tuple f :[n] ! R: > n If f (i) = ai , then f is represented by a = [a1;:::; an] 2 R . Matrix f :[m] × [n] ! R: m;n m×n If f (i; j) = aij , then f is represented by A = [aij ]i;j=1 2 R . Hypermatrix (order 3) f :[l] × [m] × [n] ! R: l;m;n l×m×n If f (i; j; k) = aijk , then f is represented by A = aijk i;j;k=1 2 R . J K X [n] [m]×[n] [l]×[m]×[n] Normally R = ff : X ! Rg. Ought to be R ; R ; R . Lek-Heng Lim (MCSC Seminar) Eigenvalues of tensors April 16, 2008 2 / 26 Hypermatrices and tensors Up to choice of bases n a 2 R can represent a vector in V (contravariant) or a linear functional in V ∗ (covariant). m×n A 2 R can represent a bilinear form V × W ! R (contravariant), ∗ ∗ a bilinear form V × W ! R (covariant), or a linear operator V ! W (mixed). l×m×n A 2 R can represent trilinear form U × V × W ! R (contravariant), bilinear operators V × W ! U (mixed), etc. A hypermatrix is the same as a tensor if 1 we give it coordinates (represent with respect to some bases); 2 we ignore covariance and contravariance. Lek-Heng Lim (MCSC Seminar) Eigenvalues of tensors April 16, 2008 3 / 26 Matrices make useful models A matrix doesn't always come from an operator. Can be a list of column or row vectors: I gene-by-microarray matrix, I movies-by-viewers matrix, I list of codewords. Can be a convenient way to represent graph structures: I adjacency matrix, I graph Laplacian, I webpage-by-webpage matrix. Useful to regard them as matrices and apply matrix operations: > I A gene-by-microarray matrix, A = UΣV gives cellular states (eigengenes), biological phenotype (eigenarrays), k I A adjacency matrix, A counts number of paths of length ≤ k from node i to node j. Ditto for hypermatrices. Lek-Heng Lim (MCSC Seminar) Eigenvalues of tensors April 16, 2008 4 / 26 Basic operation on a hypermatrix m×n A matrix can be multiplied on the left and right: A 2 R , p×m q×n X 2 R , Y 2 R , > p×q (X ; Y ) · A = XAY = [cαβ] 2 R where Xm;n cαβ = xαi yβj aij : i;j=1 l×m×n A hypermatrix can be multiplied on three sides: A = aijk 2 R , p×l q×m r×n J K X 2 R , Y 2 R , Z 2 R , p×q×r (X ; Y ; Z) ·A = cαβγ 2 R J K where Xl;m;n cαβγ = xαi yβj zγk aijk : i;j;k=1 Lek-Heng Lim (MCSC Seminar) Eigenvalues of tensors April 16, 2008 5 / 26 Basic operation on a hypermatrix Covariant version: A· (X >; Y >; Z >) := (X ; Y ; Z) ·A: Gives convenient notations for multilinear functionals and multilinear l m n operators. For x 2 R ; y 2 R ; z 2 R , Xl;m;n A(x; y; z) := A· (x; y; z) = aijk xi yj zk ; i;j;k=1 Xm;n A(I ; y; z) := A· (I ; y; z) = aijk yj zk : j;k=1 Lek-Heng Lim (MCSC Seminar) Eigenvalues of tensors April 16, 2008 6 / 26 Inner products and norms 2 n > Pn ` ([n]): a; b 2 R , ha; bi = a b = i=1 ai bi . 2 m×n > Pm;n ` ([m] × [n]): A; B 2 R , hA; Bi = tr(A B) = i;j=1 aij bij . 2 l×m×n Pl;m;n ` ([l] × [m] × [n]): A; B 2 R , hA; Bi = i;j;k=1 aijk bijk . In general, `2([m] × [n]) = `2([m]) ⊗ `2([n]); `2([l] × [m] × [n]) = `2([l]) ⊗ `2([m]) ⊗ `2([n]): Frobenius norm Xl;m;n kAk2 = a2 : F i;j;k=1 ijk Lek-Heng Lim (MCSC Seminar) Eigenvalues of tensors April 16, 2008 7 / 26 Symmetric hypermatrices n×n×n Cubical hypermatrix aijk 2 R is symmetric if J K aijk = aikj = ajik = ajki = akij = akji : Invariant under all permutations σ 2 Sk on indices. k n S (R ) denotes set of all order-k symmetric hypermatrices. Example Higher order derivatives of multivariate functions. Example Moments of a random vector x = (X1;:::; Xn): Z Z n n mk (x) = E(xi1 xi2 ··· xik ) = ··· xi1 xi2 ··· xik dµ(xi1 ) ··· dµ(xik ) : i1;:::;ik =1 i1;:::;ik =1 Lek-Heng Lim (MCSC Seminar) Eigenvalues of tensors April 16, 2008 8 / 26 Symmetric hypermatrices Example Cumulants of a random vector x = (X1;:::; Xn): 2 3n X p−1 Q Q κk (x) = 4 (−1) (p − 1)!E xi ··· E xi 5 : i2A1 i2Ap A1t···tAp =fi1;:::;ik g i1;:::;ik =1 For n = 1, κk (x) for k = 1; 2; 3; 4 are the expectation, variance, skewness, and kurtosis. Important in Independent Component Analysis (ICA). Lek-Heng Lim (MCSC Seminar) Eigenvalues of tensors April 16, 2008 9 / 26 Numerical multilinear algebra Bold claim: every topic discussed in Golub-Van Loan has a multilinear generalization. Numerical tensor rank (GV Chapter 2) Conditioning of multilinear systems (GV Chapter 3) Unsymmetric eigenvalue problem for hypermatrices (GV Chapter 7) Symmetric eigenvalue problem for hypermatrices (GV Chapter 8) Tensor approximation problems (GV Chapter 12) Lek-Heng Lim (MCSC Seminar) Eigenvalues of tensors April 16, 2008 10 / 26 DARPA mathematical challenge eight One of the twenty three mathematical challenges announced at DARPA Tech 2007. Problem Beyond convex optimization: can linear algebra be replaced by algebraic geometry in a systematic way? Algebraic geometry in a slogan: polynomials are to algebraic geometry what matrices are to linear algebra. Polynomial f 2 R[x1;:::; xn] of degree d can be expressed as > > f (x) = a0 + a1 x + x A2x + A3(x; x; x) + ··· + Ad (x;:::; x): n n×n n×n×n n×···×n a0 2 R; a1 2 R ; A2 2 R ; A3 2 R ;:::; Ad 2 R . Numerical linear algebra: d = 2. Numerical multilinear algebra: d > 2. Lek-Heng Lim (MCSC Seminar) Eigenvalues of tensors April 16, 2008 11 / 26 Multilinear spectral theory n×n×n Let A 2 R (easier if A symmetric). 1 How should one define its eigenvalues and eigenvectors? 2 What is a decomposition that generalizes the eigenvalue decomposition of a matrix? l×m×n Let A 2 R 1 How should one define its singualr values and singular vectors? 2 What is a decomposition that generalizes the singular value decomposition of a matrix? Somewhat surprising: (1) and (2) have different answers. Lek-Heng Lim (MCSC Seminar) Eigenvalues of tensors April 16, 2008 12 / 26 Tensor ranks (Hitchcock, 1927) m×n Matrix rank. A 2 R . rank(A) = dim(span fA ;:::; A g) (column rank) R •1 •n = dim(span fA ;:::; A g) (row rank) R 1• m• Pr T = minfr j A = i=1ui vi g (outer product rank): Multilinear rank. A 2 l×m×n. rank (A) = (r (A); r (A); r (A)), R 1 2 3 r (A) = dim(span fA ;:::; A g) 1 R 1•• l•• r (A) = dim(span fA ;:::; A g) 2 R •1• •m• r (A) = dim(span fA ;:::; A g) 3 R ••1 ••n l×m×n Outer product rank. A 2 R . Pr rank⊗(A) = minfr j A = i=1ui ⊗ vi ⊗ wi g l;m;n where u ⊗ v ⊗ w : = ui vj wk i;j;k=1. J K Lek-Heng Lim (MCSC Seminar) Eigenvalues of tensors April 16, 2008 13 / 26 Eigenvalue and singular value decompositions Rank revealing decompositions associated with outer product rank. 3 n Symmetric eigenvalue decomposition of A 2 S (R ), Xr A = λi vi ⊗ vi ⊗ vi (1) i=1 Pr where rankS(A) = min r A = i=1 λi vi ⊗ vi ⊗ vi = r. I P. Comon, G. Golub, L, B. Mourrain, \Symmetric tensor and symmetric tensor rank," SIAM J. Matrix Anal. Appl. l×m×n Singular value decomposition of A 2 R , Xr A = ui ⊗ vi ⊗ wi (2) i=1 where rank⊗(A) = r. I V. de Silva, L, \Tensor rank and the ill-posedness of the best low-rank approximation problem," SIAM J. Matrix Anal. Appl. (1) used in applications of ICA to signal processing; (2) used in applications of the parafac model to analytical chemistry. Lek-Heng Lim (MCSC Seminar) Eigenvalues of tensors April 16, 2008 14 / 26 Eigenvalue and singular value decompositions Rank revealing decompositions associated with the multilinear rank. 3 n Symmetric eigenvalue decomposition of A 2 S (R ), A = (U; U; U) ·C (3) where rank (A) = (r; r; r), U 2 n×r has orthonormal columns and R 3 r C 2 S (R ). l×m×n Singular value decomposition of A 2 R , A = (U; V ; W ) ·C (4) where rank (A) = (r ; r ; r ), U 2 l×r1 , V 2 m×r2 , W 2 n×r3 1 2 3 R R R r ×r ×r have orthonormal columns and C 2 R 1 2 3 . I L. De Lathauwer, B. De Moor, J. Vandewalle \A multilinear singular value decomposition," SIAM J. Matrix Anal. Appl., 21 (2000), no. 4. I B. Savas, L, \Best multilinear rank approximation with quasi-Newton method on Grassmannians," preprint. Lek-Heng Lim (MCSC Seminar) Eigenvalues of tensors April 16, 2008 15 / 26 Eigenvalues Normal A I Invariant subspace: Ax = λx. > > I Rayleigh quotient: x Ax=x x.