University of Alabama at Birmingham Department of Mathematics Numerical Linear Algebra Lecture Notes for MA 660 (1997{2014) Dr Nikolai Chernov Summer 2014 Contents 0. Review of Linear Algebra 1 0.1 Matrices and vectors . 1 0.2 Product of a matrix and a vector . 1 0.3 Matrix as a linear transformation . 2 0.4 Range and rank of a matrix . 2 0.5 Kernel (nullspace) of a matrix . 2 0.6 Surjective/injective/bijective transformations . 2 0.7 Square matrices and inverse of a matrix . 3 0.8 Upper and lower triangular matrices . 3 0.9 Eigenvalues and eigenvectors . 4 0.10 Eigenvalues of real matrices . 4 0.11 Diagonalizable matrices . 5 0.12 Trace . 5 0.13 Transpose of a matrix . 6 0.14 Conjugate transpose of a matrix . 6 0.15 Convention on the use of conjugate transpose . 6 1. Norms and Inner Products 7 1.1 Norms . 7 1.2 1-norm, 2-norm, and 1-norm . 7 1.3 Unit vectors, normalization . 8 1.4 Unit sphere, unit ball . 8 1.5 Images of unit spheres . 8 1.6 Frobenius norm . 8 1.7 Induced matrix norms . 9 1.8 Computation of kAk1, kAk2, and kAk1 .................... 9 1.9 Inequalities for induced matrix norms . 9 1.10 Inner products . 10 1.11 Standard inner product . 10 1.12 Cauchy-Schwarz inequality . 11 1.13 Induced norms . 12 1.14 Polarization identity . 12 1.15 Orthogonal vectors . 12 1.16 Pythagorean theorem . 12 1.17 Orthogonality and linear independence . 12 1.18 Orthonormal sets of vectors . 13 1.19 Orthonormal basis (ONB) . 13 1.20 Fourier expansion . 13 1.21 Orthogonal projection . 13 1.22 Angle between vectors . 14 1.23 Orthogonal projection onto a subspace . 14 1.24 Degenerate case . 14 1.25 Gram-Schmidt orthogonalization . 15 ii 1.26 Construction of ONB . 15 1.27 Legendre polynomials (optional) . 15 1.28 Orthogonal complement . 16 1.29 Orthogonal direct sum . 16 1.30 Some useful formulas . 16 2. Unitary Matrices 18 2.1 Isometries . 18 2.2 Characterization of isometries - I . 18 2.3 Characterization of isometries - II . 18 2.4 Characterization of isometries - III . 18 2.5 Identification of finite-dimensional inner product spaces . 19 2.6 Unitary and orthogonal matrices . 19 2.7 Lemma . 19 2.8 Matrices of isometries . 20 2.9 Group property . 20 2.10 Orthogonal matrices in 2D . 20 2.11 Characterizations of unitary and orthogonal matrices . 21 2.12 Determinant of unitary matrices . 21 2.13 Eigenvalues of unitary matrices . 21 2.14 Invariance principle for isometries . 22 2.15 Orthogonal decomposition for complex isometries . 22 2.16 Lemma . 23 2.17 Orthogonal decomposition for real isometries . 23 2.18 Unitary and orthogonal equivalence . 24 2.19 Unitary matrices in their simples form . 24 2.20 Orthogonal matrices in their simples form . 24 3. Hermitian Matrices 26 3.1 Adjoint matrices . 26 3.2 Adjoint transformations . 26 3.3 Riesz representation theorem . 27 3.4 Quasi-linearity . 27 3.5 Remark . 27 3.6 Existence and uniqueness of adjoint transformation . 27 3.7 Relation between Ker T ∗ and Range T ..................... 28 3.8 Selfadjoint operators and matrices . 28 3.9 Examples . 28 3.10 Hermitian property under unitary equivalence . 28 3.11 Invariance principle for selfadjoint operators . 29 3.12 Spectral Theorem . 29 3.13 Characterization of Hermitian matrices . 30 3.14 Eigendecomposition for Hermitian matrices . 30 3.15 Inverse of a selfadjoint operator . 30 3.16 Projections . 31 3.17 Projections (alternative definition) . 31 iii 3.18 \Complimentary" projections . 31 3.19 Orthogonal projections . 32 3.20 Characterization of orthogonal projections . 32 4. Positive Definite Matrices 33 4.1 Positive definite matrices . 33 4.2 Lemma . 34 4.3 Sufficient condition for positive definiteness . 34 4.4 Bilinear forms . 34 4.5 Representation of bilinear forms . 35 4.6 Corollary . 35 4.7 Hermitian/symmetric forms . 35 4.8 Quadratic forms . 35 4.9 Theorem . 35 4.10 Positive definite forms and operators . 35 4.11 Theorem . 35 4.12 Properties of Hermitian matrices . 36 4.13 Eigenvalues of positive definite matrices . 37 4.14 Inverse of a positive definite matrix . 37 4.15 Characterization of positive definite matrices . 38 4.16 Characterization of positive semi-definite matrices . 38 4.17 Full rank and rank deficient matrices . 38 4.18 Products A∗A and AA∗ ............................. 38 4.19 Spectral radius . 39 4.20 Spectral radius for Hermitian matrices . 40 4.21 Theorem on the 2-norm of matrices . 40 4.22 Example . 42 4.23 Corollary for the 2-norm of matrices . 42 5. Singular Value Decomposition 44 5.1 Singular value decomposition (SVD) . 44 5.2 Singular values and singular vectors . 46 5.3 Real SVD . 46 5.4 SVD analysis . 47 5.5 Useful relations - I . 47 5.6 Computation of SVD.