Tensors I: Basic Operations and Representations 1 Overview Tensors: Vectors, matrices and so on … Definition Operators PARAFAC/Candecomp, polyadic, CP Tucker, HOSVD 2 Different Matrix Products Kronecker product A = (a1 an ), B = (b1 bm ) Matrix case: a B a B a B 11 12 1n a B a B a B A⊗ B = 21 22 2n = am1B am2 B amn B = (a1 ⊗b1 a1 ⊗b2 a1 ⊗b3 an ⊗bm−1 an ⊗bm ) 5 7 15 21 1 3 5 7 6 8 18 24 A = , B = ,⇒ A⊗ B = 2 4 6 8 10 14 20 28 12 16 24 32 3 Different Matrix Products Vector case (row or column form): a ⊗b = (a1 an )⊗ (b1 bm ) = = (a1b an b) = (a1b1 ⋅⋅ a1bm anb1 ⋅⋅ anbm ) b 1 a ⊗b = (a1 an )⊗ = bm a b a b 1 1 n 1 = (a1b an b) = a1bm anbm 4 Different Matrix Products Khatri-Rao product: A = (a1 an ), B = (b1 bn ); A• B = (a1 ⊗b1 a2 ⊗b2 a3 ⊗b3 an−1 ⊗bn−1 an ⊗bn ) = matching columnwise Kronecker product only for matrices with the same number of columns! 5 21 1 3 5 7 6 24 A = , B = ,⇒ A• B = 2 4 6 8 10 28 12 32 5 Different Matrix Products Hadamard product: a a b b 11 1n 11 1n A = , B = , am1 amn bm1 bmn a b a b a b 11 11 12 12 1n 1n a b a b a b A∗ B = 21 21 22 22 2n 2n am1bm1 am2bm2 amnbnm only for matrices of equal size! 6 Definition Tensor as multi-indexed object: x 1 x = (x )n = (x )n = = ( ) One index: vector: i i=1 i1 i =1 or x x1 xn 1 xn a a 1,1 1,m A = (A )n, m = (A )n1 , n2 = Two indices: matrix: i, j = = i1 ,i2 = = i 1, j 1 i1 1,i2 1 an,1 an,m a1,1,2 n, m l n , n n 1 2 3 a1,1,1 a1,2,1 Three indices: cube: A = (Ai, j,k ) = (Ai ,i ,i ) i=1, j=1,k=1 1 2 3 i1 =1,i2 =1,i3 =1 a2,1,1 n , n , ,n Multi-index: x = (x ) 1 2 N i1i2 ...iN i =1,i =1,.....,i =1 1 2 N 7 Motivation: Why tensors? PDE for two-dimensional problems: (aux )x + (bu y ) y = f (x, y) aui−1, j + aui+1, j − (a + b)ui, j + bui, j−1 + bui, j+1 Discretization in 2D: = f h2 i, j ui,j can be seen as a vector or as a 2-way tensor=matrix. Linear system Au=f with block matrix A: Aij,kmukm = fij So matrix Aij,km can be also seen as a 4-way tensor 8 Motivation: Why tensors? PDE with a number of additional parameters, high-dimensional problems: + + = auxx bu yy cuzz f for discrete sets of parameters aijk Leads to linear system Amn for each i,j,k Amn,ijk Classical matrix/vector problems but for huge problems: Represent vector/matrix by tensor with efficient representation. x = x i i1...iN 9 Graphical Notation i Vector (1 leg): (xi )i ↔ i Matrix (2 legs): a ↔ ( ij )i, j j i Cube (3 legs): x ↔ ( ijk )i, j,k k j i1 i2 … iN General tensor with N legs … (x ) ↔ i1...iN i ,...,i 1 N 10 Graphical Notation Matrix-vector product – contraction over index i: a ⋅(x ) = y ↔ i ↔ ( ij )i, j i i ( j )j j j ∑ aij xi = aij xi = y j i Einstein notation, shared indices are contracted via summation. No distinction between covariant and contravariant! 11 Basic Operations y x y i1 Contraction ∑ i1 i1 gives scalar z i1 x x i Tensor product x i y i gives 2-tensor zi i 1 1 2 1 2 = z y i1 i2 i2 x y ′ ′ ′ ′ = z ′ ′ ′ ′ More general: ∑ i1 in iN i1 in−1inin+1 iM i1 in−1in+1 iN i1 in−1in+1 iM in i‘1..i‘n-1 i‘n+1..i‘M y z in = x i1i‘1…in-1i‘n-1in+1i‘n+1…iNi‘M i1..in-1 in+1..iN 12 Tensor as data hive of different form T kron(x, y) = x ⊗ y = (x1 y xn y) = (xi yi ) seen as a column vector 1 2 i1 ,i2 x y x y 1 1 1 m xyT = = (x y ) seen as a matrix i1 i2 i ,i 1 2 xn y1 xn ym = kron(yT , x) = yT ⊗ x y x y x 1 1 1 n yxT = = (x y ) seen as a matrix i1 i2 i ,i 1 2 ym x1 ym xn = kron(xT , y) = xT ⊗ y x y = (x y ) i1 i2 i ,i seen as a two-leg tensor 1 2 13 Matrix i1 A A Matrix: i1i2 i2 x A x = z i1 z Operations: Contractions ∑ i1i2 i1 i2 A i2 i1 i2 i 1 i1 A y = z A z ∑ i1i2 i2 i1 i 2 i2 y x i Ai i xi yi = z 1 ∑ 1 2 1 2 A z i1i2 i2 y i3 i B 3 A B = C i2 C ∑ i1i2 i2i3 i1i3 i1 i2 A i1 A x = C Tensor product: i1i2 i3 i1i2i3 14 Three Leg as Standardexample i1 i3 A i1i2i3 i2 Operations: Contractions in i1 , i2 , i3 or combinations gives tensor with less legs. Tensor product gives tensor with more legs. (ai ) See tensor as - collection of vectors fiber 1 i2i3 (A ) - collection of matrices slices i i 1 2 i3 A = A - large matrix, unfolding {i1i2 }i3 j1i3 Operations between tensors are defined by contracted indices. 15 Fibers 13 16 19 22 1 4 7 10 A: 3 x 4 x 2 – tensor 14 17 20 23 2 5 8 11 15 18 21 24 3 6 9 12 1 4 7 10 13 16 19 22 Mode-1 fibers, X:,j,k: 2 5 8 11 14 17 20 23 3 6 9 12 15 18 21 24 1 2 3 13 14 15 4 5 6 16 17 18 Mode-2 fibers, X : j,:,k 7 8 9 19 20 21 10 11 12 22 23 24 1 2 3 4 5 6 7 8 9 10 11 12 Mode-3 fibers, Xj,k,:: 13 14 15 16 17 18 19 20 21 22 23 24 16 Slices 13 16 19 22 1 4 7 10 A: 3 x 4 x 2 – tensor 14 17 20 23 2 5 8 11 15 18 21 24 3 6 9 12 1 4 7 10 13 16 19 22 Frontal slices, 1,2: X:,:,k 2 5 8 11 14 17 20 23 3 6 9 12 15 18 21 24 1 13 4 16 7 19 10 22 Lateral slices, 1,3: X:,k,: 2 14 5 17 8 20 11 23 3 15 6 18 9 21 12 24 13 16 19 22 14 17 20 23 15 18 21 24 Horizontal sl. 2,3: Xk,:,: 1 4 7 10 2 5 8 11 3 6 9 12 17 Matricification 13 16 19 22 1 4 7 10 A: 3 x 4 x 2 – tensor 14 17 20 23 2 5 8 11 15 18 21 24 3 6 9 12 1 4 7 10 13 16 19 22 A = A i1{i2i3} i1 j1 Mode-1 unfolding: A(1) = 2 5 8 11 14 17 20 23 3 6 9 12 15 18 21 24 j1=i2+n2(i3-1) 1 2 3 13 14 15 4 5 6 16 17 18 Mode-2 unfolding A(2) = Ai {i i } 7 8 9 19 20 21 2 1 3 10 11 12 22 23 24 1 2 3 4 5 6 7 8 9 10 11 12 = Mode-3 unfolding A(3) 13 14 15 16 17 18 19 20 21 22 23 24 T Vectorization: vec(A) = (1 2 23 24) 18 General Matricification Tensor A → A = A Matrix i1...inin+1...iN {i1...in }{in+1...iN } ij i = i1 + n2 (i2 −1) + n2n3 (i3 −1) +...+ n2 nn (in −1), j = in+1 + nn+2 (in+2 −1) + nn+2nn+3 (in+3 −1) +...+ nn+2 nN (iN −1). or with any partitioning of the indices in two groups (rows/columns) General remark on notation: many properties/operations with tensors are formulated using totally different notations! ►,◄,ʘ, ⊗ ,•,,×, 19 Basis Transformation I J K Tensor (1) (2) (3) A = ∑ ∑ ∑ Aijk ei ⊗ e j ⊗ ek i=1 j=1 k =1 (l) (l) (l) Change of basis ei = Q e'i = (1) (1) ⊗ (2) (2) ⊗ (3) (3) = A' pqr ∑∑∑ AijkQ e'i Q e' j Q e'k i j k pqr I J K (1) (2) (3) = ∑ ∑ ∑ Qpi Qqj Qrk Aijk i=1 j=1 k =1 Notation: A'= (Q(1) ,Q(2) ,Q(3) )⋅ A 20 n-Mode Product of Tensor with Matrix Tensor Matrix In A , U : (A× U ) = a ⋅u =: B i1...in ...iN jin n i1...in−1 jin+1...iN ∑ i1...iN jin i1... j...iN in =1 Contraction over in, in replaced by index j:=i‘n j i1 in iN i1…… j …………. iN In the n-mode product each mode-n fiber is multiplied B = U ⋅ A by the matrix U: i1...in−1 ,:,in+1...iN i1...in−1 ,:,in+1...iN Useful relation between n-mode product and mode-n-unfolding: B(n) = U ⋅ A(n) Unfold tensor A to matrix, multiply by U, fold back to tensor B.