San José State University Math 250: Mathematical Methods for Data Visualization Review of Linear Algebra and Multivariable Calculus Dr. Guangliang Chen Linear Algebra and Multivariable Calculus Review Introduction This course focuses on the following mathematical objects: Vectors: 1-D arrays; • Matrices: 2-D arrays • Tensors: 3-D arrays (or higher) • Dr. Guangliang Chen | Mathematics & Statistics, San José State University2/55 Linear Algebra and Multivariable Calculus Review Notation: vectors Vectors are denoted by boldface lowercase letters (such as a, b, u, v, x, y): 1 They are assumed to be in column form, e.g., a = (1, 2, 3)T = 2 • 3 3 To indicate their dimensions, we use notation like a ∈ R . • The ith element of a vector a is written as ai or a(i). • Dr. Guangliang Chen | Mathematics & Statistics, San José State University3/55 Linear Algebra and Multivariable Calculus Review Some special vectors: The zero vector • T n 0 = (0, 0,..., 0) ∈ R The vector of ones • T n 1 = (1, 1,..., 1) ∈ R The canonical basis vectors of Rn • T n ei = (0,..., 0, 1, 0,..., 0) ∈ R , i = 1, . , n When the dimension is not specified, it is implied by the context, e.g., aT 1, where a = (1, 2, 3)T Dr. Guangliang Chen | Mathematics & Statistics, San José State University4/55 Linear Algebra and Multivariable Calculus Review Notation: matrices Matrices are denoted by boldface UPPERCASE letters (such as A, B, U, V). m×n Similarly, we write A ∈ R to indicate its size (if either m or n is equal to 1, then the matrix reduces to a vector). The (i, j) entry of A is denoted by aij, or A(i, j) as in MATLAB. The ith row of A is denoted by A(i, :) while its jth column is written as A(:, j), as in MATLAB. Dr. Guangliang Chen | Mathematics & Statistics, San José State University5/55 Linear Algebra and Multivariable Calculus Review Special matrices: n×n The zero matrix: On ∈ R • n×n The identity matrix: In ∈ R • n×n The matrix of ones: Jn ∈ R • 0 0 ··· 0 1 0 0 0 1 1 ··· 1 0 0 ··· 0 0 1 0 0 1 1 ··· 1 On = . , In = . , Jn = . .. .. .. 0 0 ··· 0 0 0 0 1 1 1 ··· 1 Similarly, we may drop the subscript n when the size of the matrix is clear based on the context. Dr. Guangliang Chen | Mathematics & Statistics, San José State University6/55 Linear Algebra and Multivariable Calculus Review Description of matrix shape m×n We characterize a matrix A ∈ R based on its shape as follows: Dr. Guangliang Chen | Mathematics & Statistics, San José State University7/55 Linear Algebra and Multivariable Calculus Review n×n A diagonal matrix is a square matrix A ∈ R whose off diagonal entries are all zero (aij = 0 for all i 6= j): a11 . A = .. ann A diagonal matrix is uniquely defined by a vector that contains all the diagonal entries, and denoted as follows: 1 A = diag(1, 2, 3) = diag(a), a = 2 3 Dr. Guangliang Chen | Mathematics & Statistics, San José State University8/55 Linear Algebra and Multivariable Calculus Review Matrix multiplication m×n n×p Let A ∈ R and B ∈ R . Their matrix product is an m × p matrix n X AB = C = (cij), cij = aikbkj = A(i, :) · B(:, j). k=1 j j C= A B i b i m p m n n p × × × Dr. Guangliang Chen | Mathematics & Statistics, San José State University9/55 Linear Algebra and Multivariable Calculus Review It is possible to obtain one full row (or column) of C at a time via matrix-vector multiplication: C(i, :) = A(i, :) · B, C(:, j) = A · B(:, j) b b b C= A B b C= A B b b b b b b b b b b Dr. Guangliang Chen | Mathematics & Statistics, San José State University 10/55 Linear Algebra and Multivariable Calculus Review The full matrix C can be written as a sum of rank-1 matrices: n C = X A(:, k) · B(k, :). k=1 C= A B = + + + b b b b · · · b b Dr. Guangliang Chen | Mathematics & Statistics, San José State University 11/55 Linear Algebra and Multivariable Calculus Review Example 0.1. ! 1 2 3 1 1 0 4 5 6 1 0 1 7 8 9 Dr. Guangliang Chen | Mathematics & Statistics, San José State University 12/55 Linear Algebra and Multivariable Calculus Review Further interpretation of AB when one of the matrices is actually a vector: If A = (a1, . , an) is a row vector (m = 1): • n X AB = aiB(i, :) ←− linear combination of rows of B i=1 T If B = (b1, . , bn) is a column vector (p = 1): • n X AB = bjA(:, j) ←− linear combination of columns of A j=1 Dr. Guangliang Chen | Mathematics & Statistics, San José State University 13/55 Linear Algebra and Multivariable Calculus Review Example 0.2. 1 2 3 1 2 3 −1 −1 0 1 4 5 6 , 4 5 6 0 7 8 9 7 8 9 1 Dr. Guangliang Chen | Mathematics & Statistics, San José State University 14/55 Linear Algebra and Multivariable Calculus Review n Finally, below are some identities involving the vector 1 ∈ R : 1 1 1 ... 1 1 1 1 ... 1 T T 1 1 = n, 11 = . 1 1 ... 1 = . = Jn . .. 1 1 1 ... 1 m×n For any matrix A ∈ R , A1 = X A(:, j), (vector of row sums) j 1T A = X A(i, :), (horizontal vector of column sums) i 1T A1 = X X A(i, j)(total sum of all entries) i j Dr. Guangliang Chen | Mathematics & Statistics, San José State University 15/55 Linear Algebra and Multivariable Calculus Review Graphical illustration Dr. Guangliang Chen | Mathematics & Statistics, San José State University 16/55 Linear Algebra and Multivariable Calculus Review Computational complexity n m×n n×p Let x, y ∈ R and A ∈ R , B ∈ R . Then Summing up the entries of x: O(n) (n − 1 additions); • Computing the dot product xT y: O(n) (2n − 1 operations in total: • n multiplications and n − 1 additions). This implies that computing the norm of x, kxk2 = xT x, also has O(n) complexity. Multiplying a matrix and a vector Ax: O(mn) (m dot product • operations) Multiplying two matrices AB: O(mnp). In particular, computing • 2 n×n 3 C for a square matrix C ∈ R takes O(n ) time. Dr. Guangliang Chen | Mathematics & Statistics, San José State University 17/55 Linear Algebra and Multivariable Calculus Review An application of matrix associative law in computing Matrix multiplications follow the associative and distributive laws (but there is no commutative law). m×n n×p p×1 In a special case where A ∈ R , B ∈ R , x ∈ R , we have (AB)x = A(Bx) Important note: Although mathematically the same, the second way of multiplying them via two matrix-vector multiplications is much faster! First way: O(mnp + mp) complexity • Second way: O(mn + np) complexity • Dr. Guangliang Chen | Mathematics & Statistics, San José State University 18/55 Linear Algebra and Multivariable Calculus Review The Hadamard product m×n Another way to multiply two matrices of the same size, say A, B ∈ R , is through the Hadamard product, also called the entrywise product: m×n C = A ◦ B ∈ R , with cij = aijbij. For example, ! ! ! 0 2 −3 1 0 −3 0 0 9 ◦ = . −1 0 −4 2 1 −1 −2 0 4 Dr. Guangliang Chen | Mathematics & Statistics, San José State University 19/55 Linear Algebra and Multivariable Calculus Review An important application of the entrywise product is in efficiently computing the product of a diagonal matrix and a rectangular matrix by software. a1 B(1, :) a1B(1, :) . . . A B = .. . = . |{z} diagonal an B(n, :) anB(n, :) b1 . AB = [A(:, 1) ... A(:, n)] .. |{z} diagonal bn = [b1A(:, 1) . bnA(:, n)] Dr. Guangliang Chen | Mathematics & Statistics, San José State University 20/55 Linear Algebra and Multivariable Calculus Review We may implement the direct row/column operations using the Hadamard product. T n For example, in the former case, let a = diag(A) = (a1, . , an) ∈ R , which represents the diagonal of A. Then A B = [a ... a] ◦B. |{z} |{z} | {z } n×n n×p p copies The former takes O(n2p) operations, while the latter takes only O(np) operations, which is one magnitude fewer. Dr. Guangliang Chen | Mathematics & Statistics, San José State University 21/55 Linear Algebra and Multivariable Calculus Review B= B ◦ For example, −1 1 2 3 10 −1 −1 −1 −1 1 2 3 10 0 4 5 6 10 = 0 0 0 0 ◦4 5 6 10 1 7 8 9 10 1 1 1 1 7 8 9 10 Dr. Guangliang Chen | Mathematics & Statistics, San José State University 22/55 Linear Algebra and Multivariable Calculus Review Matrix transpose m×n n×m The transpose of a matrix A ∈ R is another matrix B ∈ R with T bij = aji for all i, j. We denote the transpose of A by A . n×n T A square matrix A ∈ R is said to be symmetric if A = A. m×n n×p Let A ∈ R , B ∈ R , and k ∈ R. Then (AT )T = A • (kA)T = kAT • (A + B)T = AT + BT • (AB)T = BT AT • Dr. Guangliang Chen | Mathematics & Statistics, San José State University 23/55 Linear Algebra and Multivariable Calculus Review Matrix inverse n×n A square matrix A ∈ R is said to be invertible if there exists another square matrix of the same size B such that AB = BA = I. In this case, B is called the matrix inverse of A and denoted as B = A−1.