Linear Algebra Review Kaiyu Zheng October 2017 Linear algebra is fundamental for many areas in computer science. This document aims at providing a reference (mostly for myself) when I need to remember some concepts or examples. Instead of a collection of facts as the Matrix Cookbook, this document is more gentle like a tutorial. Most of the content come from my notes while taking the undergraduate linear algebra course (Math 308) at the University of Washington. Contents on more advanced topics are collected from reading different sources on the Internet. Contents 3.8 Exponential and 7 Special Matrices 19 Logarithm...... 11 7.1 Block Matrix.... 19 1 Linear System of Equa- 3.9 Conversion Be- 7.2 Orthogonal..... 20 tions2 tween Matrix Nota- 7.3 Diagonal....... 20 tion and Summation 12 7.4 Diagonalizable... 20 2 Vectors3 7.5 Symmetric...... 21 2.1 Linear independence5 4 Vector Spaces 13 7.6 Positive-Definite.. 21 2.2 Linear dependence.5 4.1 Determinant..... 13 7.7 Singular Value De- 2.3 Linear transforma- 4.2 Kernel........ 15 composition..... 22 tion.........5 4.3 Basis......... 15 7.8 Similar........ 22 7.9 Jordan Normal Form 23 4.4 Change of Basis... 16 3 Matrix Algebra6 7.10 Hermitian...... 23 4.5 Dimension, Row & 7.11 Discrete Fourier 3.1 Addition.......6 Column Space, and Transform...... 24 3.2 Scalar Multiplication6 Rank......... 17 3.3 Matrix Multiplication6 8 Matrix Calculus 24 3.4 Transpose......8 5 Eigen 17 8.1 Differentiation... 24 3.4.1 Conjugate 5.1 Multiplicity of 8.2 Jacobian....... 25 Transpose..8 Eigenvalues..... 18 8.3 The Chain Rule... 25 3.5 Inverse........9 5.2 Eigendecomposition 18 3.6 Trace......... 10 9 Algorithms 25 3.7 Power........ 11 6 The Big Theorem 19 9.1 Gauss-Seidel Method 25 1 Notation We denote vectors using bold lower case letters such as x, matrices using bold upper case letters such as X, and entries of matrices using normal upper case letters such as Xij or Xi;j (The comma is used if the indices are expressed by equations). The vector ei by default means the ith column vector in an indentity matrix with dimension depending on the context. 1 Linear System of Equations Definition 1.1 (Row Echelon Form). Each variable can be the leading variable for at most one equation. For example, x1 + x2 + x3 − x4 = 0 −x2 + 7x4 − x5 = −1 (1) x4 + x5 = 2 Definition 1.2. Linear systems are equivalent if they are related by a sequence of elemen- tary operations: (1) Interchange position of rows (2) Multiply an equal constant (3) Add a multiple of one equation to another Definition 1.3 (Augmented Matrix). The linear system a11x1 + a12x2+ ··· + a1mxm = b1; . (2) an1x1 + an2x2+ ··· + anmxm = bn can be written as an augmented matrix as follows: 2 3 a11 : : : a1m b1 6 . .. 7 4 . 5 (3) an1 : : : anm bn Definition 1.4 (Row Echelon Form). A matrix is in row echelon form if a) Every leading term is in a column to the left of the leading term of the row below it. 2 b) Any zero rows are at the bottom of the matrix For example, the left matrix below is not an echelon form, because \0=7" has no leading variable. It is an inconsistent matrix. The right matrix is a echelon form. 21 2 3 0 03 21 −2 5 2 −13 40 0 1 2 35 40 3 4 5 6 5 0 0 0 0 7 0 0 22 14 4 The leading variable positions in the matrix are called pivot positions. A column in the matrix that contains a pivot position is a pivot column. The process of converting a linear system into echelon form is Gaussian Elimination. Definition 1.5 (Reduced Row Echelon Form). A matrix is said to be in reduced row echelon form if: a) all pivot positions have 1 b) the only nonzero term in each pivot column is the pivot c) it is in row echelon form. Try finding the reduced row echelon form of the following matrix: 20 3 4 5 6 3 41 −2 5 2 −15 (4) 3 0 1 2 5 Definition 1.6 (Homogeneity). A homogeneous linear equation is a1x1 + a2x2 + ··· + anxn = 0 (5) The equation is said to be in homogeneous form. A linear system where all equations are in homogeneous form is a homogenous system. Every homogenous system is consistent, i.e. solvable. 2 Vectors n Definition 2.1 (Norm). The norm, or magnitude of a vector a 2 R is defined as the L2-norm of the vector. v u n uX 2 jaj = t ai (6) i=1 3 n Definition 2.2 (Dot Product). (Algebraic definition) Let a and b be two vectors in R . Then the dot product (or inner product) between a and b is defined as: n T X a · b = a b = aibi (7) i=1 (Geometric definition) The dot product of two Euclidean vectors a and b is defined by a · b = jajjbjcos(θa;b) (8) Also, The dot product w · x = b is a hyperplane, where w is normal to it. n Definition 2.3 (Projection). Let a and b be two vectors in R . The projection of b onto a is defined a · b a a · b proj b = = a (9) a jaj jaj jaj2 n Definition 2.4 (Outer Product). Let a and b be two vectors in R . Then the outer T product (or tensor product) between a and b is defined such that (ab )ij = aibj: 2 3 a1b1 a1b2 ··· a1bn 6a2b1 a2b2 ··· a2bn7 abT = 6 7 (10) 6 . .. 7 4 . 5 anb1 anb2 ··· anbn Definition 2.5 (Linear Combination). If u1; u2; ··· um are vectors and c1; c2 ··· cm are scalars, then c1u1 + c2u2 + ··· + cmum is a linear combination of the vectors. n Definition 2.6 (Span). Let fu1; ··· ; umg be a set of m vectors in R . The span of the set is the set of linear combinations of u1 ··· um. 213 233 For example, suppose u1 = 425 and u2 = 425 , what is the span of fu1; u2g? A vector 3 1 2a3 2a3 21 3 a3 v = 4b5 2 spanfu1; u2g if and only if 9s; t:su1 + tu2 = 4b5. s; t exist if 42 2 b5 has c c 3 1 c 21 3 a 3 a solution. This matrix is reduced to 40 4 2a − b 5, therefore it has a solution when 0 0 a − 2b + c a − ab + c = 0 holds. So the span of fu1; u2g is the plane x − 2y + z = 0. Definition 2.7 (Relation of Span and Augmented Matrix). If a vector v is in the span of vectors fu1; ··· ; umg then the matrix [u1 ··· um v] has at least 1 solution. Theorem 2.1 (Relation of Span and Linearly Independence). If u 2 spanfu1; ··· ; umg then spanfu1; ··· ; umg = spanfu; u1; ··· ; umg 4 2.1 Linear independence n Definition 2.8 (Linear Independence). Let fu1; ··· ; umg be a set of vectors in R . If the only solution to the equation x1u1 + ··· + xmum = 0 is the trivial solution (i.e. all zeros), then u1 ··· um are linearly independent. Fact: If any set of vector contains 0, this set of vectors are not linearly independent. Definition 2.9 (Orthonormal Vectors). Vectors in a set U = fu1; ··· umg are orthonormal if every vector in U is a unit vector and every pair ui; uj 2 U of vectors are orthogonal, T i.e. ui uj = 0. Theorem 2.2. Every set of orthonormal vectors is linearly independent (i.e. the vectors in the set are linearly independent). 2.2 Linear dependence n Theorem 2.3 (Linear Dependence). Let fu1; ··· ; umg be a set of vectors in R . If n < m, the set is linearly dependent. Corollary 2.3.1 (Relation of Span and Linearly Independence). If there is a set of m n n linearly independent vectors in R that spans all of R , then m = n. Theorem 2.4 (Relation of Linear Combination and Linearly Dependence). Let fu1; ··· ; umg n be a set of vectors in R . The vectors in this set are linearly dependent if one vector is a linear combination of others. 2.3 Linear transformation m n Definition 2.10 (Linear Transformation). Function T : R ! R is a linear transforma- m m tion if for all v; u 2 R and for all r 2 R, T (v + u) = T v + T u and T (rv) = rT (v). R n m is the domain, and R is the co-domain. For u 2 R , T (u) is the image of u under T . n Definition 2.11 (Subspace). A subset S of R is a subspace if S satisfies: a) S contains 0. b) if u and v are in S then u + v is also in S.(closure under addition) c) If r is a real number, and u 2 S then, ru 2 S.(closure under multiplication) m n Definition 2.12 (One-to-one and On-to). Let T : R ! R , T (v) = Av thus T is a linear transformation. T is one-to-one (injective) if and only if T (x) = 0 has only the trivial solution (i.e. x = 0), or equivalently, T (a) = T (b) implies a = b. This means the columns n of A are linearly independent. T is on-to (surjective) if and only if columns of A span R . 5 Note, A is a n × m matrix. If m > n, T is not one-to-one.