From Matrix to Tensor: The Transition to Numerical Multilinear Algebra Lecture 3. Transpositions, Kronecker Products, and Contractions Charles F. Van Loan Cornell University The Gene Golub SIAM Summer School 2010 Selva di Fasano, Brindisi, Italy ⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Where We Are Lecture 1. Introduction to Tensor Computations Lecture 2. Tensor Unfoldings Lecture 3. Transpositions, Kronecker Products, Contractions Lecture 4. Tensor-Related Singular Value Decompositions Lecture 5. The CP Representation and Rank Lecture 6. The Tucker Representation Lecture 7. Other Decompositions and Nearness Problems Lecture 8. Multilinear Rayleigh Quotients Lecture 9. The Curse of Dimensionality Lecture 10. Special Topics ⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions What is this Lecture About? Preview Now we turn our attention to the engagement of these multidimensional data objects in two elementary operations. Transposition. While there is only one way transpose a matrix there are an exponential number of ways to transpose an order-d tensor. This will require (surprise surprise) more facility with multi-indexing. Contraction. The contraction of two tensors can be regarded as a generalization of matrix-matrix multiplication. Analogous to how the serious business of matrix decompositions builds on matrix multiplication, we will find that tensor contractions have a foundational role to play in tensor computations. ⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions What is this Lecture About? More “Bridging the Gap” The Kronecker product helps bridge the gap between matrix computations and tensor computations. For example, the contraction between two tensors can sometimes be “reshaped” into a matrix computation that involves Kronecker products. So in advance of our introduction to tensor contractions, we will get familiar with this all-important matrix operation and some of its nearby “cousins.” ⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Tensor Transposition Example: A(1:4, 1:5, 1:6) → B(1:5, 1:6, 1:4) A = randn([ 4 5 6]); B = zeros([5 6 4]); for i1=1:4 for i2 = 1:5 for i3 = 1:6 B(i2,i3,i1) = A(i1,i2,i3); end end end How to say this: B = A< [ 2 3 1 ] >. ⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Tensor Transposition: The 3rd Order Case Six possibilities... If A ∈ IRn1×n2×n3 , then there are 6 = 3! possible transpositions identified by the notation A< [i j k] > where [i j k] is a permutation of [1 2 3]: A< [1 2 3] > b ijk < [1 3 2] > A bikj < [2 1 3] > A bjik B = =⇒ = aijk A< [2 3 1] > b jki A< [3 1 2] > b kij < [3 2 1] > A bkji for i = 1:n1, j = 1:n2, k = 1:n3. ⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Tensor Transposition: General Case Definition n1×···×nd If A ∈ IR and p = [p1,..., pd ] is a permutation of the <p> n ×···×n index vector 1:d, then A ∈ IR p1 pd denotes the p-transpose of A and is defined by <p> A (jp1 ,..., jpd ) = A(j1,..., jd ) where 1 ≤ jk ≤ nk for k = 1:d. ⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Matlab Tensor Toolbox: Transposition % If A is an order -d tensor and % p is a permutation of 1:d, then % the following assigns the p- transpose % of A to B ... B = permute(A,p) Problem 3.1. Any square matrix can be written as the sum of a symmetric matrix and a skew-symmetric matrix: „ A + AT « „ A − AT « A = + 2 2 Formulate a generalization of this result for order-3 tensors and write a Matlab function that embodies your idea. ⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Tensor Notation: Subscript Vectors (Review) Reference n1×···×nd If A ∈ IR and i = (i1,..., id ) with 1 ≤ ik ≤ nk for k = 1:d, then A(i) ≡ A(i1,..., ik ) We say that i is a subscript vector. Bold font will be used designate subscript vectors. Bounds If L and R are subscript vectors having the same dimension, then L ≤ R means that Lk ≤ Rk for all k. Special Cases A subscript vector of all ones is denoted by 1. (Dimension clear from context.) If N is an integer, then N = N · 1. ⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Tensor Transposition: General Case Definition n1×···×nd If A ∈ IR and p = [p1,..., pd ] is a permutation of the <p> n ×···×n index vector 1:d, then A ∈ IR p1 pd denotes the p-transpose of A and is defined by A<p>(j(p)) = A(j) 1 ≤ j ≤ n. ⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Problem 3.2. Verify that if p and q are both permutations of 1:d, then `A<p>´<q> = A<p(q)> where A ∈ IRn1×···×nd . Problem 3.3. Suppose A ∈ IRn1×···×nd and that B = A[ p ] where p is a permutation of 1:d. Specify a permutation matrix P so that B(k) = A(p(k))P. ⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions What is a Kronecker Product? Definition B ⊗ C is a block matrix whose ij-th block is bij C. An Example... " # " # b11 b12 b11C b12C ⊗ C = b21 b22 b21C b22C Replicated Block Structure ⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions What Is a Kronecker Product? There are three ways to regard A = B ⊗ C... If b11 ··· b1n c11 ··· c1q . .. ⊗ . .. A = . . bm1 ··· bmn cp1 ··· cpq then (i). A is an m-by-n block matrix with p-by-q blocks. (ii). A is an mp-by-nq matrix of scalars. (iii). A is an unfolded 4-th order tensor A ∈ IRp×q×m×n: A(i1, i2, i3, i4) = B(i3, i4)C(i1, i2) ⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions All Possible Entry-Entry Products 4 × 9 = 36 c11 c12 c13 b11 b12 ⊗ c21 c22 c23 b21 b22 c31 c32 c33 = b11c11 b11c12 b11c13 b12c11 b12c12 b12c13 b11c21 b11c22 b11c23 b12c21 b12c22 b12c23 b11c31 b11c32 b11c33 b12c31 b12c32 b12c33 b21c11 b21c12 b21c13 b22c11 b22c12 b22c13 b21c21 b21c22 b21c23 b22c21 b22c22 b22c23 b21c31 b21c32 b21c33 b22c31 b22c32 b22c33 ⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Hierarchical Kronecker Products of Kronecker Products B ⊗ C ⊗ D = c11 c12 c13 c14 d11 d12 d13 b11 b12 c21 c22 c23 c24 ⊗ ⊗ d21 d22 d23 b21 b22 c31 c32 c33 c34 d31 d32 d33 c41 c42 c43 c44 A 2-by-2 block matrix whose entries are 4-by-4 block matrices whose entries are 3-by-3 matrices. ⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Kronecker Product Properties Some Basic Facts... (B ⊗ C)T = BT ⊗ C T (B ⊗ C)−1 = B−1 ⊗ C −1 (B ⊗ C)(D ⊗ F ) = BD ⊗ CF B ⊗ (C ⊗ D) = (B ⊗ C) ⊗ D Note that B ⊗ C 6= C ⊗ B. However, it can be shown that C ⊗ B = PT (B ⊗ C)Q where P and Q are perfect shuffle permutations. More later. ⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Kronecker Products and Structure Structured Factors Produce a Structured Product... B and C B ⊗ C Nonsingular Nonsingular Lower(Upper) triangular Lower(Upper) Triangular Banded Block Banded Symmetric Symmetric Positive Definite Positive Definite Stochastic Stochastic Toeplitz Block Toeplitz Permutation Permutation Orthogonal Orthogonal ⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Kronecker Products and Data Sparsity Example 1. If B, C ∈ IRm×m and A ≈ B ⊗ C 2 then we are using O(m ) numbers ( the bij and cij ) to 4 represent/approximate O(m ) data (the aij .) Example 2. m×m If Bij ∈ IR and r X A ≈ Bk,1 ⊗ Bk,2 ⊗ ··· ⊗ Bk,d k=1 then we are using rdm2 numbers to represent/approximate a matrix with m2d entries. ⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Kronecker Products and Fast Matrix-Vector Multiplication A Basic Reshaping Result... Y = CXBT ⇔ vec(Y ) = (B ⊗ C)vec(X ) In Practice... function y = KronTimesVector(B,C,y) % y = kron(B,C)*x [mb,nb] = size(B); [mc,nc] = size(C); Y = C*reshape(x,nc,nb)*B’; y = reshape(Y,mc*mb,1); If both matrices are n-by-n, then work is O(n3). Ordinarily, an n2-by-n2 matrix-vector product would be O(n4). ⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Problem 3.4. How would you compute the matrix-vector product q×q pqr y = (Ip ⊗ A ⊗ Ir )x where A ∈ IR and x ∈ IR ? Assess the amount of work. Problem 3.5. How would you compute the matrix-vector product n×n n3 y = (A1 ⊗ A2 ⊗ A3)x where Ai ∈ IR and x ∈ IR ?. Hint: (A1 ⊗ A2 ⊗ A3) = (A1 ⊗ In2 )(In ⊗ A2 ⊗ In)(In2 ⊗ A3) Assess the amount of work. Problem 3.6. How would you compute the matrix-vector product n×n n3 y = (A1 ⊗ ··· ⊗ Ad )x where Ai ∈ IR and x ∈ IR ?. Assess the amount of work. ⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Kronecker Products and Matrix Factorizations Just take the Kronecker Product of the Factors... LU: (PB ⊗ PC )(B ⊗ C) = (LB ⊗ LC )(UB ⊗ UC ) T Cholesky: B ⊗ C = (LB ⊗ LC )(LB ⊗ LC ) T Schur: (QB ⊗ QC ) (B ⊗ C)(QB ⊗ QC ) = TB ⊗ TC QR: B ⊗ C = (QB ⊗ QC )(RB ⊗ RC ) T SVD: B ⊗ C = (UB ⊗ UC )(ΣB ⊗ ΣC )(VB ⊗ VC ) There are some annoying details if the matrices are rectangular, e.g., (ΣB ⊗ ΣC ) needs to be permuted to obtain a true diagonal form.