Lecture 3. Transpositions, Kronecker Products, and Contractions

Home , Kronecker product, Multilinear algebra, Tensor, Tensor contraction, Transpose

From Matrix to Tensor: The Transition to Numerical Multilinear Algebra

Lecture 3. Transpositions, Kronecker Products, and Contractions

Charles F. Van Loan

Cornell University

The Gene Golub SIAM Summer School 2010 Selva di Fasano, Brindisi, Italy

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Where We Are

Lecture 1. Introduction to Tensor Computations Lecture 2. Tensor Unfoldings Lecture 3. Transpositions, Kronecker Products, Contractions Lecture 4. Tensor-Related Singular Value Decompositions Lecture 5. The CP Representation and Rank Lecture 6. The Tucker Representation Lecture 7. Other Decompositions and Nearness Problems Lecture 8. Multilinear Rayleigh Quotients Lecture 9. The Curse of Dimensionality Lecture 10. Special Topics

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions What is this Lecture About?

Preview Now we turn our attention to the engagement of these multidimensional data objects in two elementary operations.

Transposition. While there is only one way transpose a matrix there are an exponential number of ways to transpose an order-d tensor. This will require (surprise surprise) more facility with multi-indexing.

Contraction. The contraction of two tensors can be regarded as a generalization of matrix-matrix multiplication. Analogous to how the serious business of matrix decompositions builds on matrix multiplication, we will ﬁnd that tensor contractions have a foundational role to play in tensor computations.

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions What is this Lecture About?

More “Bridging the Gap”

The Kronecker product helps bridge the gap between matrix computations and tensor computations. For example, the contraction between two tensors can sometimes be “reshaped” into a matrix computation that involves Kronecker products.

So in advance of our introduction to tensor contractions, we will get familiar with this all-important matrix operation and some of its nearby “cousins.”

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Tensor Transposition

Example: A(1:4, 1:5, 1:6) → B(1:5, 1:6, 1:4) A = randn([ 4 5 6]); B = zeros([5 6 4]); for i1=1:4 for i2 = 1:5 for i3 = 1:6 B(i2,i3,i1) = A(i1,i2,i3); end end end

How to say this: B = A< [ 2 3 1 ] >.

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Tensor Transposition: The 3rd Order Case

Six possibilities...

If A ∈ IRn1×n2×n3 , then there are 6 = 3! possible transpositions identiﬁed by the notation A< [i j k] > where [i j k] is a permutation of [1 2 3]:     A< [1 2 3] > b    ijk   < [1 3 2] >     A   bikj       < [2 1 3] >     A   bjik  B = =⇒ = aijk A< [2 3 1] > b    jki       A< [3 1 2] >   b     kij   < [3 2 1] >     A   bkji 

for i = 1:n1, j = 1:n2, k = 1:n3.

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Tensor Transposition: General Case

Deﬁnition

n1×···×nd If A ∈ IR and p = [p1,..., pd ] is a permutation of the

n ×···×n index vector 1:d, then A ∈ IR p1 pd denotes the p-transpose of A and is deﬁned by

A (jp1 ,..., jpd ) = A(j1,..., jd )

where 1 ≤ jk ≤ nk for k = 1:d.

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Matlab Tensor Toolbox: Transposition

% If A is an order -d tensor and % p is a permutation of 1:d, then % the following assigns the p- transpose % of A to B ...

B = permute(A,p)

Problem 3.1. Any square matrix can be written as the sum of a symmetric matrix and a skew-symmetric matrix:

„ A + AT « „ A − AT « A = + 2 2 Formulate a generalization of this result for order-3 tensors and write a Matlab function that embodies your idea.

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Tensor Notation: Subscript Vectors (Review)

Reference

n1×···×nd If A ∈ IR and i = (i1,..., id ) with 1 ≤ ik ≤ nk for k = 1:d, then A(i) ≡ A(i1,..., ik ) We say that i is a subscript vector. Bold font will be used designate subscript vectors.

Bounds If L and R are subscript vectors having the same dimension, then L ≤ R means that Lk ≤ Rk for all k.

Special Cases A subscript vector of all ones is denoted by 1. (Dimension clear from context.) If N is an integer, then N = N · 1.

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Tensor Transposition: General Case

Deﬁnition

n1×···×nd If A ∈ IR and p = [p1,..., pd ] is a permutation of the

n ×···×n index vector 1:d, then A ∈ IR p1 pd denotes the p-transpose of A and is deﬁned by

(j(p)) = A(j) 1 ≤ j ≤ n.

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Problem 3.2. Verify that if p and q are both permutations of 1:d, then

´ = A

where A ∈ IRn1×···×nd .

Problem 3.3. Suppose A ∈ IRn1×···×nd and that B = A[ p ] where p is a permutation of 1:d. Specify a permutation matrix P so that B(k) = A(p(k))P.

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions What is a Kronecker Product?

Deﬁnition

B ⊗ C is a block matrix whose ij-th block is bij C.

An Example... " # " # b11 b12 b11C b12C ⊗ C = b21 b22 b21C b22C

Replicated Block Structure

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions What Is a Kronecker Product?

There are three ways to regard A = B ⊗ C...

If     b11 ··· b1n c11 ··· c1q  . .. .  ⊗  . .. .  A =  . . .   . . .  bm1 ··· bmn cp1 ··· cpq then (i). A is an m-by-n block matrix with p-by-q blocks. (ii). A is an mp-by-nq matrix of scalars. (iii). A is an unfolded 4-th order tensor A ∈ IRp×q×m×n:

A(i1, i2, i3, i4) = B(i3, i4)C(i1, i2)

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions All Possible Entry-Entry Products

4 × 9 = 36

  c11 c12 c13 b11 b12 ⊗  c21 c22 c23  b21 b22 c31 c32 c33 =   b11c11 b11c12 b11c13 b12c11 b12c12 b12c13  b11c21 b11c22 b11c23 b12c21 b12c22 b12c23     b11c31 b11c32 b11c33 b12c31 b12c32 b12c33     b21c11 b21c12 b21c13 b22c11 b22c12 b22c13     b21c21 b21c22 b21c23 b22c21 b22c22 b22c23  b21c31 b21c32 b21c33 b22c31 b22c32 b22c33

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Hierarchical

Kronecker Products of Kronecker Products

B ⊗ C ⊗ D =   c11 c12 c13 c14   d11 d12 d13 b11 b12  c21 c22 c23 c24  ⊗   ⊗  d21 d22 d23  b21 b22  c31 c32 c33 c34  d31 d32 d33 c41 c42 c43 c44

A 2-by-2 block matrix whose entries are 4-by-4 block matrices whose entries are 3-by-3 matrices.

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Kronecker Product Properties

Some Basic Facts... (B ⊗ C)T = BT ⊗ C T

(B ⊗ C)−1 = B−1 ⊗ C −1

(B ⊗ C)(D ⊗ F ) = BD ⊗ CF

B ⊗ (C ⊗ D) = (B ⊗ C) ⊗ D

Note that B ⊗ C 6= C ⊗ B. However, it can be shown that

C ⊗ B = PT (B ⊗ C)Q

where P and Q are perfect shuﬄe permutations. More later.

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Kronecker Products and Structure

Structured Factors Produce a Structured Product...

B and C B ⊗ C Nonsingular Nonsingular Lower(Upper) triangular Lower(Upper) Triangular Banded Block Banded Symmetric Symmetric Positive Deﬁnite Positive Deﬁnite Stochastic Stochastic Toeplitz Block Toeplitz Permutation Permutation Orthogonal Orthogonal

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Kronecker Products and Data Sparsity

Example 1. If B, C ∈ IRm×m and A ≈ B ⊗ C 2 then we are using O(m ) numbers ( the bij and cij ) to 4 represent/approximate O(m ) data (the aij .)

Example 2. m×m If Bij ∈ IR and

r X A ≈ Bk,1 ⊗ Bk,2 ⊗ ··· ⊗ Bk,d k=1

then we are using rdm2 numbers to represent/approximate a matrix with m2d entries.

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Kronecker Products and Fast Matrix-Vector Multiplication

A Basic Reshaping Result...

Y = CXBT ⇔ vec(Y ) = (B ⊗ C)vec(X )

In Practice... function y = KronTimesVector(B,C,y) % y = kron(B,C)*x [mb,nb] = size(B); [mc,nc] = size(C); Y = C*reshape(x,nc,nb)*B’; y = reshape(Y,mc*mb,1);

If both matrices are n-by-n, then work is O(n3). Ordinarily, an n2-by-n2 matrix-vector product would be O(n4).

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Problem 3.4. How would you compute the matrix-vector product q×q pqr y = (Ip ⊗ A ⊗ Ir )x where A ∈ IR and x ∈ IR ? Assess the amount of work.

Problem 3.5. How would you compute the matrix-vector product n×n n3 y = (A1 ⊗ A2 ⊗ A3)x where Ai ∈ IR and x ∈ IR ?. Hint:

(A1 ⊗ A2 ⊗ A3) = (A1 ⊗ In2 )(In ⊗ A2 ⊗ In)(In2 ⊗ A3) Assess the amount of work.

Problem 3.6. How would you compute the matrix-vector product n×n n3 y = (A1 ⊗ ··· ⊗ Ad )x where Ai ∈ IR and x ∈ IR ?. Assess the amount of work.

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Kronecker Products and Matrix Factorizations

Just take the Kronecker Product of the Factors...

LU: (PB ⊗ PC )(B ⊗ C) = (LB ⊗ LC )(UB ⊗ UC )

T Cholesky: B ⊗ C = (LB ⊗ LC )(LB ⊗ LC )

T Schur: (QB ⊗ QC ) (B ⊗ C)(QB ⊗ QC ) = TB ⊗ TC

QR: B ⊗ C = (QB ⊗ QC )(RB ⊗ RC )

T SVD: B ⊗ C = (UB ⊗ UC )(ΣB ⊗ ΣC )(VB ⊗ VC )

There are some annoying details if the matrices are rectangular, e.g.,

(ΣB ⊗ ΣC ) needs to be permuted to obtain a true diagonal form.

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions ⊗ ⊗ Problem 3.7. How would you minimize k (A1 ··· Ad )x − b k2 given m×n that each A1,..., Ad ∈ IR has rank ri < n?

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Perfect Shuﬄe Permutations

Transposition is a Permutation This:  T a11 a12 b11 b12 b13 =  a21 a22  b21 b22 b23 a31 a32 Is the same as this:       b11 1 0 0 0 0 0 a11  b21   0 0 0 1 0 0   a21         b12   0 1 0 0 0 0   a31    =      b22   0 0 0 0 1 0   a12         b13   0 0 1 0 0 0   a22  b23 0 0 0 0 0 1 a32

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Perfect Shuﬄe Permutations

Formal Deﬁnition

If n = pr then the mod-p perfect shuﬄe permutation Sp,r is deﬁned by  x(1:r:n)   x(2:r:n)  S x =   p,r  .   .  x(r:r:n) where x ∈ IRn.

Implication for High Performance Computing: Matrix transpose involves non-unit stride access to matrix elements.

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Problem 3.8. (a) Think about the perfect shuﬄing of a 52-card deck and explain why we use the term “perfect shuﬄe” when referring to Sp,r . (b) T Show that Sr,p = Sp,r . (c) Explain why Sp,r is aptly called the “mod-p sort permutation.”

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Perfect Shuﬄe Permutations

The Connection to Matrix Transpose... If X ∈ IRp×r and Y = X T , then

vec(Y ) = Sp,r · vec(X )

The Connection to Kronecker Products...

m1×n1 m2×n2 If A1 ∈ IR and A2 ∈ IR then

⊗ T ⊗ A2 A1 = Sm1,m2 · (A1 A2) · Sn1,n2

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Perfect Shuﬄe Permutations

The Connection to Tensor Transpose

If A ∈ IRn1×n2×n3 , then

 A< [1 2 3] >   I     n1n2n3       A< [1 3 2] >   S ⊗ I     n2,n3 n1       < [2 1 3] >     A   In ⊗ Sn ,n  B = =⇒ vec(B) = 3 1 2 ·vec(A) A< [2 3 1] > S    n1,n2n3       A< [3 1 2] >   S     n1n2,n3       < [3 2 1] >   ⊗  A (Sn1,n2 In3 )Sn1n2,n3

Implication for High Performance Computing: Tensor transpose involves non-unit stride access to matrix elements.

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions n1×···×nd Problem 3.9. Suppose A ∈ IR , N = n1 ··· nd , and that p is a permutation of 1:d that involves swapping a single pair of indices, e.g., [1 4 3 2 5]. Determine a permutation matrix P ∈ IRN×N so that if B = A< p >, then vec(B) = P · vec(A).

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Tensor Contractions

Deﬁnition A contraction between two tensors produces a third tensor through a summation process.

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Linear Algebra Operations in Tensor Contraction Terms

The Contraction of an Order-1 Tensor with an Order-1 Tensor

n X c = a(k)·b(k) (Dot Product) k=1

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Linear Algebra Operations in Tensor Contraction Terms

The Contraction of an Order-2 Tensor and an Order-1 Tensor

n X c(i) = A(i, k)·b(k) (Matrix-Vector Product) k=1

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Linear Algebra Operations in Tensor Contraction Terms

The Contraction of an Order-2 Tensor and an Order-2 Tensor

n X C(i, j) = A(i, k)·B(k, j) (Matrix-Matrix Product) k=1

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Tensor Contractions: Subscript Level

Example 1. (A Single-Index Contraction)

X C(i, j, α3, α4, β3, β4, β5) = A(i, k, α3, α4) ·B(k, j, β3, β4, β5) k ⇑ ⇑ ⇑ 7th 4th 5th Order Order Order

A = A(1:n1, 1:n2, 1:n3, 1:n4)

B = B(1:m1, 1:m2, 1:m3, 1:m4, 1:m5)

⇒ C = C(1:n1, 1:m2, 1:n3, 1:n4, 1:m3, 1:m4, 1:m5)

Order(C) = Order(A) + Order(B) - 2

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Tensor Contractions: Subscript Level

Example 2. (A Double-Index Contraction) X X C(i, j, p, q) = A(i, s, p, k) ∗ B(s, j, k, q) k s ⇑ ⇑ ⇑ 4th 4th 4th Order Order Order

Order(C) = Order(A) + Order(B) - 2 - 2

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Tensor Contractions: Summation Notation

Summation

If L = (L1,..., Lk ) and R = (R1,..., Rk ) then

R R R R X X1 X2 Xk ≡ ···

i=L i1=L1 i2=L2 ik =Lk

Example: Frobenius Norm of A ∈ IRn1×···×nd v u n uX 2 k A kF = t A(i) i=1

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Tensor Contractions: Subscript Level

Example 3. (A 4-Index Contraction)

C(γ1, γ2, γ3, γ4)

= m X A(i1, i2, i3, i4)·W (i1, γ1)·X (i2, γ2)·Y (i3, γ3)·Z(i4, γ4) i=1 ⇑ ⇑ ⇑ ⇑ ⇑ 4th 2nd 2nd 2nd 2nd Order Order Order Order Order

Order(C) = Order(A) + Order(W) + Order(X )+ Order(Y)+ Order(Z) - 2 - 2 - 2 - 2

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Towards a High-Performance Contraction Framework

In Reality...

Tensor computations are matrix computations. A contraction can be regarded as a generalization of matrix-matrix multiplication.

The Challenge...

Expose the underlying matrix multiplications with good notation. Two Reasons: 1. To facilitate tensor-level thinking. 2. To facilitate level-3 BLA exploitation

Let us revisit the ﬁve examples rephrasing them to expose the underlying matrix products.

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Towards a High-Performance Contraction Framework

The Block Vec Unfolding 2 A(: , : , 1, 1, 1) 3 6 A(: , : , 2, 1, 1) 7 6 7 6 7 6 A(: , : , 1, 2, 1) 7 6 7 6 A(: , : , 2, 2, 1) 7 6 7 n1×n2×2×2×2 Blockvec(A) = 6 7 A ∈ IR 6 A(: , : , 1, 1, 2) 7 6 7 6 A(: , : , 2, 1, 2) 7 6 7 6 7 4 A(: , : , 1, 2, 2) 5 A(: , : , 2, 2, 2)

The Block-Vector Product 2 3 F1G1 2 3 6 F1G2 7 » – G1 6 7 F1 6 F1G3 7 ⊗ 4 G2 5 = 6 7 F2 6 F2G1 7 G3 6 7 4 F2G2 5 F2G3

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Tensor Contractions: Matrix Level

Example 1.

X C(i, j, α3, α4, β3, β4, β5) = A(i, k, α3, α4)B(k, j, β3, β4, β5) k

As a collection of matrix-matrix products...

C(: , : , α3, α4, β3, β4, β5) = A(: , : , α3, α4)B(: , : , β3, β4, β5)

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Tensor Contractions: Matrix Level

As a Block Vector Product...

 B(: , : , 1, 1, 1)    C(: , : , 1, 1, 1, 1, 1)  B(: , : , 2, 1, 1)       C(: , : , 1, 1, 2, 1, 1)  A(: , : , 1, 1)  B(: , : , 1, 2, 1)       C(: , : , 1, 1, 1, 2, 1)   A(: , : , 2, 1)   B(: , : , 2, 2, 1)   .  =   ⊗    .   A(: , : , 1, 2)   B(: , : , 1, 1, 2)   .      A(: , : , 2, 2)  B(: , : , 2, 1, 2)   C(: , : , 2, 2, 1, 2, 2)    C(: , : , 2, 2, 2, 2, 2)  B(: , : , 1, 2, 2)  B(: , : , 2, 2, 2)

32 matrix-matrix products

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Problem 3.10. Suppose A ∈ IRn1×···×nd and B ∈ IRm1×···×me with n2 = m1. If

n X2 C(α1, β2, α3, . . . , αd , β3, . . . , βe ) = A(α1, k, α3, . . . , αd ) ·B(k, β2, . . . , βe ) k=1 then we refer to C as the canonical contraction of A and B. Write a Matlab function C = CanonContract(A,B) that computes this tensor given that A and B are tensors.

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Towards a High-Performance Contraction Framework

Block Matrix Unfolding for A = A(1:n1, 1:n2, 1:n3, 1:n4)

  A(: , : , 1, 1) A(: , : , 1, 2) ···A(: , : , 1, n4)    A(: , : , 2, 1) A(: , : , 2, 2) ···A(: , : , 2, n4)  BlockMat(A) =    . . .. .   . . . .    A(: , : , n3, 1) A(: , : , n3, 2) ···A(: , : , n3, n4)

A(i, j, k, `) is the (i, j) entry of block (k, `)

Note that tenmat(A,[1 3],[2 4]) produces the BlockMat unfolding.

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Tensor Contractions: Matrix Level

Example 2. X X C(i, j, p, q) = A(i, s, p, k) ·B(s, j, k, q) k s

As a collection of matrix products and sums... X C(: , : , p, q) = A(: , : , p, k) ·B(: , : , k, q) k

As a product of two block matrices...

BlockMat(C) = BlockMat(A) · BlockMat(B)

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Problem 3.11. Write an eﬃcient Matlab function C = Eg2(A,B) that takes two order-4 tensors A and B and produces the order-4 tensor deﬁned in Example 2.

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Tensor Contractions: Matrix Level

Example 3.

C(m1, m2, m3, m4) = m X A(i1, i2, i3, i4)·W (i1, m1)·X (i2, m2)·Y (i3, m3)·Z(i4, m4) i=1

As a block-matrix product...

BlockMat(C) = (W ⊗ Y ) · BlockMat(A) · (X ⊗ Z)

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Problem 3.12. Write an eﬃcient Matlab function C = Eg5(A,W,X,Y,Z) that takes an order-4 tensor A and four suitably dimensioned matrices W , X , Y , and Z and produces the order-4 tensor C deﬁned in Example 5.

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions High Performance Tensor Computations

There are Locality Issues... If the B(: , : , p, r) and C(: , : , r, q) are stored continguously, then

X X A(i, j, p, q) = B(i, s, p, k) ∗ C(s, j, k, q) k s is nice because the participating “slices” are laid out nicely: X A(: , : , p, q) = B(: , : , p, r) ∗ C(: , : , r, q) r Unfriendly: X X A(i, j, p, q) = B(i, p, s, k) ∗ C(k, s, j, q) k s

LAPACK has it easy: BC, BC T , BT C , BT C T . Tensor BLAS?

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions High Performance Tensor Computations

There are Data Structure/Layout Issues...

SupposeA = A(1:n1,..., 1:n7).

Can regard A as a 4-th order tensor with 3rd order entries. Store the 3rd order subtensors contiguously.

Can regard A as an aggregation of “little” 7th order tensors. (Like a block matrix.)

The what-ﬁts-in-cache inequality:

7 m ≤ Ncache

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions Summary of Lecture 3.

Key Words

A transposition of an order-d tensor is deﬁned by a permutation p of 1:d. Element i is mapped to element p(i).

The Kronecker product of two matrices is a highly structured block matrix.

Perfect shuﬄe permutations are a handy way to describe tensor transposition.

A contraction between two tensors is a summation process that produces a new tensor.

⊗ Transition to Computational Multilinear Algebra ⊗ Lecture 3. Transpositions, Kronecker Products, Contractions