APPLIED MATRIX THEORY j Lecture Notes for Math 464/514 Presented by DR. MONIKA NITSCHE j Typeset and Editted by ERIC M. BENNER j STUDENTS PRESS December 3, 2013 Copyright © 2013 Contents 1 Introduction to Linear Algebra1 1.1 Lecture 1: August 19, 2013 .......................... 1 About the class, 1. Linear Systems, 1. Example: Application to boundary value problem, 2. Analysis of error, 3. Solution of the discretized equation, 4. 2 Matrix Inversion5 2.1 Lecture 2: August 21, 2013 .......................... 5 Gaussian Elimination, 5. Inner-product based implementation, 7. Office hours and other class notes, 8. Example: Gauss Elimination, 8. 2.2 Lecture 3: August 23, 2013 .......................... 8 Example: Gauss Elimination, cont., 8. Operation Cost of Forward Elimination, 9. Cost of the Order of an Algorithm, 10. Validation of Lower/Upper Triangular Form, 11. Theoretical derivation of Lower/Upper Form, 11. 2.3 HW 1: Due August 30, 2013 ......................... 12 3 Factorization 15 3.1 Lecture 4: August 26, 2013 .......................... 15 Elementary Matrices, 15. Solution of Matrix using the Lower/Upper factorization, 18. Sparse and Banded Matrices, 18. Motivation for Gauss Elimination with Pivoting, 19. 3.2 Lecture 5: August 28, 2013 .......................... 19 Motivation for Gauss Elimination with Pivoting, cont., 19. Discussion of well-posedness, 20. Gaussian elimination with pivoting, 21. 3.3 Lecture 6: August 30, 2013 .......................... 22 Discussion of HW problem 2, 22. PLU factorization, 22. 3.4 Lecture 7: September 4, 2013......................... 24 PLU Factorization, 24. Triangular Matrices, 25. Multiplication of lower triangular ma- trices, 25. Inverse of a lower triangular matrix, 25. Uniqueness of LU factorization, 26. Existence of the LU factorization, 26. 3.5 Lecture 8: September 6, 2013......................... 27 About Homeworks, 27. Discussion of ill-conditioned systems, 27. Inversion of lower triangular matrices, 28. Example of LU decomposition of a lower triangular matrix, 28. Banded matrix example, 29. iii Nitsche and Benner Applied Matrix Theory 3.6 Lecture 9: September 9, 2013......................... 29 Existence of the LU factorization (cont.), 29. Rectangular matrices, 31. 3.7 HW 2: Due September 13, 2013 ....................... 32 4 Rectangular Matrices 35 4.1 Lecture 10: September 11, 2013 ....................... 35 Rectangular matrices (cont.), 35. Example of RREF of a Rectangular Matrix, 37. 4.2 Lecture 11: September 13, 2013 ....................... 38 Solving Ax = b, 38. Example, 38. Linear functions, 39. Example: Transpose operator, 40. Example: trace operator, 40. Matrix multiplication, 41. Proof of transposition property, 42. 4.3 Lecture 12: September 16, 2013 ....................... 42 Inverses, 42. Low rank perturbations of I, 43. The Sherman{Morrison Formula, 44. Finite difference example with periodic boundary conditions, 44. Examples of pertur- bation, 45. Small perturbations of I, 45. 4.4 Lecture 13: September 18, 2013 ....................... 46 Small perturbations of I (cont.), 46. Matrix Norms, 47. Condition Number, 48. 4.5 HW 3: Due September 27, 2013 ....................... 49 5 Vector Spaces 55 5.1 Lecture 14: September 20, 2013 ....................... 55 Topics in Vector Spaces, 55. Field, 55. Vector Space, 56. Examples of function spaces, 57. 5.2 Lecture 15: September 23, 2013 ....................... 58 The four subspaces of Am×n, 58. 5.3 Lecture 16: September 25, 2013 ....................... 61 The Four Subspaces of A, 62. Linear Independence, 63. 5.4 Lecture 17: September 27, 2013 ....................... 64 Linear functions (rev), 64. Review for exam, 64. Previous lecture continued, 65. 5.5 Lecture 18: October 2, 2013.......................... 66 Exams and Points, 66. Continuation of last lecture, 66. 6 Least Squares 69 6.1 Lecture 19: October 4, 2013.......................... 69 Least Squares, 69. 6.2 Lecture 20: October 7, 2013.......................... 70 Properties of Transpose Multiplication, 71. The Normal Equations, 71. Exam 1, 73. 6.3 Lecture 21: October 9, 2013.......................... 74 Exam Review, 74. Least squares and minimization, 74. 6.4 HW 4: Due October 21, 2013......................... 76 iv Nitsche and Benner Applied Matrix Theory 7 Linear Transformations 81 7.1 Lecture 22: October 14, 2013 ......................... 81 Linear Transformations, 83. Examples of Linear Functions, 83. Matrix representation of linear transformations, 83. 7.2 Lecture 23: October 16, 2013 ......................... 84 Basis of a linear transformation, 84. Action of linear transform, 87. Change of Basis, 88. 7.3 Lecture 24: October 21, 2013 ......................... 89 Change of Basis (cont.), 89. 7.4 Lecture 25: October 23, 2013 ......................... 91 Properties of Special Bases, 91. Invariant Subspaces, 93. 7.5 HW 5: Due November 4, 2013 ........................ 94 8 Norms 99 8.1 Lecture 26: October 25, 2013 ......................... 99 Difinition of norms, 99. Vector Norms, 99. The two norm, 99. Matrix Norms, 101. Induced Norms, 102. 8.2 Lecture 27: October 28, 2013 ......................... 102 Matrix norms (review), 102. Frobenius Norm, 102. Induced Matrix Norms, 104. 8.3 Lecture 28: October 30, 2013 ......................... 106 The 2-norm, 106. 9 Orthogonalization with Projection and Rotation 109 9.1 Lecture 28 (cont.) ................................ 109 Inner Product Spaces, 109. 9.2 Lecture 29: November 1, 2013 ........................ 110 Inner Product Spaces, 110. Fourier Expansion, 111. Orthogonalization Process (Gramm-Schmidt), 111. 9.3 Lecture 30: November 4, 2013 ........................ 112 Gramm{Schmidt Orthogonalization, 112. 9.4 Lecture 31: November 6, 2013 ........................ 115 Unitary (orthogonal) matrices, 116. Rotation, 117. Reflection, 118. 9.5 HW 6: Due November 11, 2013 ....................... 118 9.6 Lecture 32: November 8, 2013 ........................ 120 Elementary orthogonal projectors, 120. Elementary reflection, 121. Complimentary Subspaces of V, 121. Projectors, 121. 9.7 Lecture 33: November 11, 2013........................ 122 Projectors, 122. Representation of a projector, 123. 9.8 Lecture 34: November 13, 2013........................ 124 n Projectors, 124. Decompositions of R , 125. Range Nullspace decomposition of An×n, 126. 9.9 HW 7: Due November 22, 2013 ....................... 126 v Nitsche and Benner Applied Matrix Theory 9.10 Lecture 35: November 15, 2013........................ 128 Range Nullspace decomposition of An×n, 128. Corresponding factorization of A, 129. 10 Singular Value Decomposition 131 10.1 Lecture 35 (cont.) ................................ 131 Singular Value Decomposition, 131. 10.2 Lecture 36: November 18, 2013........................ 132 Singular Value Decomposition, 132. Existence of the Singular Value Decomposition, 133. 10.3 Lecture 37: November 20, 2013........................ 136 Review and correction from last time, 136. Singular Value Decomposition, 136. Geometric interpretation, 138. 10.4 Lecture 38: November 22, 2013........................ 139 Review for Exam 2, 139. Norms, 139. More major topics, 140. 10.5 HW 8: Due December 10, 2013 ....................... 142 10.6 Lecture 39: November 27, 2013........................ 144 Singular Value Decomposition, 144. SVD in Matlab, 145. 11 Additional Topics 149 11.1 Lecture 39 (cont.) ................................ 149 The Determinant, 149. 11.2 Lecture 40: December 2, 2013 ........................ 150 Further details for class, 150. Diagonalizable Matrices, 150. Eigenvalues and eigenvec- tors, 150. Index 155 Other Contents 157 vi UNIT 1 Introduction to Linear Algebra 1.1 Lecture 1: August 19, 2013 About the class The textbook for the class will be Matrix Analysis and Applied Linear Algebra by Meyer. Another highly recommended text is Laub's Matrix Analysis for Scientists and Engineers. Linear Systems A linear system may be of the general form Ax = b: (1.1.1) This may be represented in several equivalent ways. 2x1 + x2 − 3x3 = 18; (1.1.2a) −4x1 + 5x3 = −28; (1.1.2b) 6x1 + 13x2 = 37: (1.1.2c) This also may be put in matrix form 0 1 0 1 0 1 2 1 −3 x1 18 @−4 0 5A @x2A = @−28A: (1.1.3) 6 13 0 x3 37 Finally, a the third common form is vector form: 0 21 0 11 0−31 0 181 @−4A x1 + @ 0A x2 + @ 5A x3 = @−28A: (1.1.4) 6 13 0 37 1 Nitsche and Benner Unit 1. Introduction to Linear Algebra y y(t) t t0 t1 t2 t3 ··· tn Figure 1.1. Finite difference approximation of a 1D boundary value problem. Example: Application to boundary value problem We will use finite difference approximations on a rectangular grid to solve the system, − y00(t) = f(t); for t 2 [0; 1]; (1.1.5) with the boundary conditions y(0) = 0; (1.1.6a) y(1) = 0: (1.1.6b) This is a 1D version of the general Laplace equation represented by, − ∆u = f (1.1.7) or in more engineering/science form − r2u = f: (1.1.8) The Laplace operator in cartesian coordinates, r2u = r · (ru); (1.1.9a) = uxx + uyy + uzz: (1.1.9b) Finite Difference Approximation Let tj = j∆t, with j = 0;:::;N. The approximate forms of the solution yj ≈ y(tj). Now we need to approximate the derivatives with discrete values of the variables. The forward difference approximation is 0 yj+1 − yj y (tj) = ; (1.1.10) tj+1 − tj or y − y y0(t ) = j+1 j ; (1.1.11) j ∆t 2 1.1. Lecture 1: August 19, 2013 Applied Matrix Theory The backward difference approximation is y − y y0(t ) = j j−1 : (1.1.12) j ∆t The centered difference approximation is y − y y0(t ) = j+1 j−1 : (1.1.13) j 2∆t Each of these are useful approximations to the first derivative that have