Introduction to the Problem Strassen's Algorithm Intro. to Tensor Rank Bounding ! with Tensor Rank Fast Matrix Multiplication = Calculating Tensor Rank Caltech Math 10 Presentation Chinmay Nirkhe November 15th, 2016 Chinmay Nirkhe Fast Matrix Multiplication Introduction to the Problem Strassen's Algorithm Intro. to Tensor Rank Bounding ! with Tensor Rank Statement of the Problem Question n×n Given two matrices A; B 2 C , what is the algorithmic complexity of calculating C = AB? X Cij = Aik Bkj k Assume that arithmetic operations (i.e. +; ×) of two values in C is constant time (O(1)). Chinmay Nirkhe Fast Matrix Multiplication Introduction to the Problem Strassen's Algorithm Intro. to Tensor Rank Bounding ! with Tensor Rank Big-O Notation Definition (Big-O) For functions f ; g : N ! N, we say f 2 O(g) (or f = O(g)) if 9 c > 0 s.t. f (n) ≤ cg(n) for all but finitely many n. Chinmay Nirkhe Fast Matrix Multiplication Introduction to the Problem Strassen's Algorithm Intro. to Tensor Rank Bounding ! with Tensor Rank A na¨ıve O(n3) algorithm Calculating each Cij can be done in O(n) computations by X Cij = Aik Bkj k 2 2 3 n such values Cij =) n · O(n) = O(n ). Chinmay Nirkhe Fast Matrix Multiplication Remark (Trivial Bounds) 2 ≤ ! ≤ 3. We saw an O(n3) algorithm. C has n2 entries so it takes at least O(n2) computations to just write C. Introduction to the Problem Strassen's Algorithm Intro. to Tensor Rank Bounding ! with Tensor Rank Definition of ! Definition (Coefficient of Matrix Multiplication) We define the coefficient of matrix multiplication ! as n o ! = inf α s.t. 8 > 0; 9 alg. of complexity O(nα+) : Chinmay Nirkhe Fast Matrix Multiplication Introduction to the Problem Strassen's Algorithm Intro. to Tensor Rank Bounding ! with Tensor Rank Definition of ! Definition (Coefficient of Matrix Multiplication) We define the coefficient of matrix multiplication ! as n o ! = inf α s.t. 8 > 0; 9 alg. of complexity O(nα+) : Remark (Trivial Bounds) 2 ≤ ! ≤ 3. We saw an O(n3) algorithm. C has n2 entries so it takes at least O(n2) computations to just write C. Chinmay Nirkhe Fast Matrix Multiplication Introduction to the Problem Strassen's Algorithm Intro. to Tensor Rank Bounding ! with Tensor Rank Goal revisited The goal is to (ideally) show ! = 2. The latest result by Coppersmith-Winnograd is ! ≤ 2:3728639. Chinmay Nirkhe Fast Matrix Multiplication Introduction to the Problem Strassen's Algorithm Intro. to Tensor Rank Bounding ! with Tensor Rank How do we improve !? 2×2 Start by considering A; B 2 C . Then, 8 C = A B + A B > 11 11 11 12 21 > <C12 = A11B12 + A12B22 C = A B + A B > 21 21 11 22 21 > :C22 = A21B12 + A22B22 Clearly there are 8 multiplications terms here. We are going to reduce this to 7 multiplications. Chinmay Nirkhe Fast Matrix Multiplication Introduction to the Problem Strassen's Algorithm Intro. to Tensor Rank Bounding ! with Tensor Rank 7 multiplications M1 = (A11 + A22)(B11 + B22) M2 = (A21 + A22)B11 8 C = M + M − M + M > 11 1 4 5 7 M3 = A11(B12 − B22) > <C12 = M3 + M5 M4 = A22(B21 − B11) C = M + M > 21 2 4 M5 = (A11 + A12)B22 > :C22 = M1 − M2 + M3 + M6 M6 = (A21 − A11)(B11 + B12) M7 = (A12 − A22)(B21 + B22) Chinmay Nirkhe Fast Matrix Multiplication 2m×2m Let f (m) be the complexity of multiplying A; B 2 C . m m f (m) = 7f (m − 1) + c0(2 × 2 ) f (m) = O(7m) Introduction to the Problem Strassen's Algorithm Intro. to Tensor Rank Bounding ! with Tensor Rank A bound on ! due to 7 multiplications and recursion ! 2m×2m A11 A12 Consider matrices A; B 2 C ; m 2 N. Write A = A21 A22 2m−1×2m−1 for Aij 2 C and B similarly. Use the 7 multiplication trick to recursively multiply A and B. Chinmay Nirkhe Fast Matrix Multiplication f (m) = O(7m) Introduction to the Problem Strassen's Algorithm Intro. to Tensor Rank Bounding ! with Tensor Rank A bound on ! due to 7 multiplications and recursion ! 2m×2m A11 A12 Consider matrices A; B 2 C ; m 2 N. Write A = A21 A22 2m−1×2m−1 for Aij 2 C and B similarly. Use the 7 multiplication trick to recursively multiply A and B. 2m×2m Let f (m) be the complexity of multiplying A; B 2 C . m m f (m) = 7f (m − 1) + c0(2 × 2 ) Chinmay Nirkhe Fast Matrix Multiplication Introduction to the Problem Strassen's Algorithm Intro. to Tensor Rank Bounding ! with Tensor Rank A bound on ! due to 7 multiplications and recursion ! 2m×2m A11 A12 Consider matrices A; B 2 C ; m 2 N. Write A = A21 A22 2m−1×2m−1 for Aij 2 C and B similarly. Use the 7 multiplication trick to recursively multiply A and B. 2m×2m Let f (m) be the complexity of multiplying A; B 2 C . m m f (m) = 7f (m − 1) + c0(2 × 2 ) f (m) = O(7m) Chinmay Nirkhe Fast Matrix Multiplication Introduction to the Problem Strassen's Algorithm Intro. to Tensor Rank Bounding ! with Tensor Rank A bound on ! due to 7 multiplications and recursion Let N = 2m, then f (m) = O(Nlog2 7). Theorem (Strassen's) ! ≤ log2 7 ≈ 2:8074 Chinmay Nirkhe Fast Matrix Multiplication No. Can we find a simpler way to multiply a 3 × 3 matrix using k multiplications s.t. log3 k < log2 7? No. Introduction to the Problem Strassen's Algorithm Intro. to Tensor Rank Bounding ! with Tensor Rank Can we multiply C2×2 with < 7 multiplications? Chinmay Nirkhe Fast Matrix Multiplication Can we find a simpler way to multiply a 3 × 3 matrix using k multiplications s.t. log3 k < log2 7? No. Introduction to the Problem Strassen's Algorithm Intro. to Tensor Rank Bounding ! with Tensor Rank Can we multiply C2×2 with < 7 multiplications? No. Chinmay Nirkhe Fast Matrix Multiplication No. Introduction to the Problem Strassen's Algorithm Intro. to Tensor Rank Bounding ! with Tensor Rank Can we multiply C2×2 with < 7 multiplications? No. Can we find a simpler way to multiply a 3 × 3 matrix using k multiplications s.t. log3 k < log2 7? Chinmay Nirkhe Fast Matrix Multiplication Introduction to the Problem Strassen's Algorithm Intro. to Tensor Rank Bounding ! with Tensor Rank Can we multiply C2×2 with < 7 multiplications? No. Can we find a simpler way to multiply a 3 × 3 matrix using k multiplications s.t. log3 k < log2 7? No. Chinmay Nirkhe Fast Matrix Multiplication Introduction to the Problem Strassen's Algorithm Intro. to Tensor Rank Bounding ! with Tensor Rank The Matrix Multiplication Tensor Definition (The Matrix Multiplication Tensor) nm×mp×pn For fixed n, the matrix multiplication tensor T 2 C defined by Tij0;jk0;ki 0 = 1fi=i 0;j=j0;k=k0g for i; i 0; j; j0; k; k0 2 f1;:::; ng. Notation The tensor is often notated as hn; m; pi. Chinmay Nirkhe Fast Matrix Multiplication Introduction to the Problem Strassen's Algorithm Intro. to Tensor Rank Bounding ! with Tensor Rank M.M. Tensor of h2; 2; 2i Chinmay Nirkhe Fast Matrix Multiplication Introduction to the Problem Strassen's Algorithm Intro. to Tensor Rank Bounding ! with Tensor Rank Strassen's equations M1 = (A11 + A22)(B11 + B22) M2 = (A21 + A22)B11 M3 = A11(B12 − B22) M4 = A22(B21 − B11) M5 = (A11 + A12)B22 M6 = (A21 − A11)(B11 + B12) M7 = (A12 − A22)(B21 + B22) Chinmay Nirkhe Fast Matrix Multiplication Definition (Tensor Rank) For an arbitrary tensor T , the rank of T (not. R(T )) is the minimum number of rank 1 tensors summing to T . Introduction to the Problem Strassen's Algorithm Intro. to Tensor Rank Bounding ! with Tensor Rank Rank of a Tensor Definition (Rank 1 Tensor) nm×mp×np A tensor T 2 C is a rank 1 tensor if it can be expressed nm mp np as T = u ⊗ v ⊗ w where u 2 C ; v 2 C ; w 2 C . i.e. Tijk = ui vj wk Chinmay Nirkhe Fast Matrix Multiplication Introduction to the Problem Strassen's Algorithm Intro. to Tensor Rank Bounding ! with Tensor Rank Rank of a Tensor Definition (Rank 1 Tensor) nm×mp×np A tensor T 2 C is a rank 1 tensor if it can be expressed nm mp np as T = u ⊗ v ⊗ w where u 2 C ; v 2 C ; w 2 C . i.e. Tijk = ui vj wk Definition (Tensor Rank) For an arbitrary tensor T , the rank of T (not. R(T )) is the minimum number of rank 1 tensors summing to T . Chinmay Nirkhe Fast Matrix Multiplication Introduction to the Problem Strassen's Algorithm Intro. to Tensor Rank Bounding ! with Tensor Rank R(h2; 2; 2i) ≤ 7. T1 = (1; 0; 0; 1) ⊗ (1; 0; 0; 1) ⊗ (1; 0; 0; 1) T2 = (0; 0; 1; 1) ⊗ (1; 0; 0; 0) ⊗ (0; 0; 1; −1) T3 = (1; 0; 0; 0) ⊗ (0; 1; 0; −1) ⊗ (0; 1; 0; 1) T4 = (0; 0; 0; 1) ⊗ (−1; 0; 1; 0) ⊗ (1; 0; 1; 0) T5 = (1; 1; 0; 0) ⊗ (0; 0; 0; 1) ⊗ (−1; 1; 0; 0) T6 = (−1; 0; 1; 0) ⊗ (1; 1; 0; 0) ⊗ (0; 0; 0; 1) T7 = (0; 1; 0; −1) ⊗ (0; 0; 1; 1) ⊗ (1; 0; 0; 0) Chinmay Nirkhe Fast Matrix Multiplication Introduction to the Problem Strassen's Algorithm Intro. to Tensor Rank Bounding ! with Tensor Rank Strassen's equations M1 = (A11 + A22)(B11 + B22) M2 = (A21 + A22)B11 8 C = M + M − M + M > 11 1 4 5 7 M3 = A11(B12 − B22) > <C12 = M3 + M5 M4 = A22(B21 − B11) C = M + M > 21 2 4 M5 = (A11 + A12)B22 > :C22 = M1 − M2 + M3 + M6 M6 = (A21 − A11)(B11 + B12) M7 = (A12 − A22)(B21 + B22) Chinmay Nirkhe Fast Matrix Multiplication Introduction to the Problem Strassen's Algorithm Intro.