APPLIED LINEAR ALGEBRA Giorgio Picci November 24, 2015 1 Contents 1 LINEAR VECTOR SPACES AND LINEAR MAPS 10 1.1 Linear Maps and Matrices . 11 1.2 Inverse of a Linear Map . 12 1.3 Inner products and norms . 13 1.4 Inner products in coordinate spaces (1) . 14 1.5 Inner products in coordinate spaces (2) . 15 1.6 Adjoints. 16 1.7 Subspaces . 18 1.8 Image and kernel of a linear map . 19 1.9 Invariant subspaces in Rn ........................... 22 1.10 Invariant subspaces and block-diagonalization . 23 2 1.11 Eigenvalues and Eigenvectors . 24 2 SYMMETRIC MATRICES 25 2.1 Generalizations: Normal, Hermitian and Unitary matrices . 26 2.2 Change of Basis . 27 2.3 Similarity . 29 2.4 Similarity again . 30 2.5 Problems . 31 2.6 Skew-Hermitian matrices (1) . 32 2.7 Skew-Symmetric matrices (2) . 33 2.8 Square roots of positive semidefinite matrices . 36 2.9 Projections in Rn ............................... 38 2.10 Projections on general inner product spaces . 40 3 2.11 Gramians. 41 2.12 Example: Polynomial vector spaces . 42 3 LINEAR LEAST SQUARES PROBLEMS 43 3.1 Weighted Least Squares . 44 3.2 Solution by the Orthogonality Principle . 46 3.3 Matrix least-Squares Problems . 48 3.4 A problem from subspace identification . 50 3.5 Relation with Left- and Right- Inverses . 51 3.6 The Pseudoinverse . 54 3.7 The Euclidean pseudoinverse . 63 3.8 The Pseudoinverse and Orthogonal Projections . 64 3.9 Linear equations . 66 4 3.10 Unfeasible linear equations and Least Squares . 68 3.11 The Singular value decomposition (SVD) . 70 3.12 Useful Features of the SVD . 72 3.13 Matrix Norms . 73 3.14 Generalization of the SVD . 75 3.15 SVD and the Pseudoinverse . 79 4 NUMERICAL ASPECTS OF L-S PROBLEMS 81 4.1 Numerical Conditioning and the Condition Number . 86 4.2 Conditioning of the Least Squares Problem . 90 4.3 The QR Factorization . 91 4.4 The role of orthogonality . 97 4.5 Fourier series and least squares .................... 100 5 4.6 SVD and least squares . 101 5 INTRODUCTION TO INVERSE PROBLEMS 102 5.1 Ill-posed problems . 103 5.2 From ill-posed to ill-conditioned . 104 5.3 Regularized Least Squares problems ................. 105 6 Vector spaces of second order random variables (1) 106 6.1 Vector spaces of second order random variables (2) . 107 6.2 About \random vectors" . 108 6.3 Sequences of second order random variables . 109 6.4 Principal Components Analysis (PCA) . 111 6.5 Bayesian Least Squares Estimation . 114 6.6 The Orthogonal Projection Lemma . 115 6 6.7 Block-diagonalization of Symmetric Positive Definite matrices . 119 6.8 The Matrix Inversion Lemma (ABCD Lemma) . 122 6.9 Change of basis . 123 6.10 Cholesky Factorization . 124 6.11 Bayesian estimation for a linear model ................ 126 6.12 Use of the Matrix Inversion Lemma ................. 127 6.13 Interpretation as a regularized least squares . 128 6.14 Application to Canonical Correlation Analysis (CCA) . 129 6.15 Computing the CCA in coordinates . 134 7 KRONECKER PRODUCTS 135 7.1 Eigenvalues . 137 7.2 Vectorization . 138 7 7.3 Mixing ordinary and Kronecker products: The mixed-product property . 141 7.4 Lyapunov equations . 145 7.5 Symmetry . 147 7.6 Sylvester equations . 152 7.7 General Stein equations . 154 8 Circulant Matrices 158 8.1 The Symbol of a Circulant ....................... 162 8.2 The finite Fourier Transform ...................... 163 8.3 Back to Circulant matrices . 166 8 Notation A>: transpose of A. A∗: transpose conjugate of (the complex matrix) A . σ(A): the spectrum (set of the eigenvalues) of A. Σ(A): the set singular values of A. A+: pseudoinverse of A. Im (A): image of A. ker(A): kernel of A. A{−1g: inverse image with respect to A. A−R: right-inverse of A (AA−R = I). A−L: left-inverse of A (A−LA = I). 9 1 LINEAR VECTOR SPACES AND LINEAR MAPS A vector space is a mathematical structure formed by a collection of elements called vectors, which may be added together and multiplied by numbers, called scalars. Scalars may be real or complex numbers, or generally elements of any field F . Accordingly the vector space is called real- or complex- or an F − vector space. The operations of vector addition and multiplication by a scalar must satisfy certain natural axioms which we shall not need to report here. The modern definition of vector space was introduced by Giuseppe Peano in 1888. Examples of (real) vector spaces are the arrows in a fixed plane or in the three-dimensional space representing forces or velocity in Physics. Vectors may however be very general objects such as functions or polynomials etc. provided they can be added together and multiplied by scalars to give elements of the same kind. The vector space composed of all the n-tuples of real or complex numbers is known as a coordinate space and is usually denoted by Rn or Cn. 10 1.1 Linear Maps and Matrices The concepts of linear independence, basis, coordinates, etc. are given for granted. Vector spaces admitting a basis consisting of a finite number n of elements are called n−dimensional vector spaces. Example: the complex numbers C are a two-dimensional real vector space, with a two dimensional basis consisting of 1 and the imaginary unit i. A function between two vector spaces f : V ! W is a linear map if for all scalars α; β and all vectors v1; v2 in V f(αv1 + βv2) = αf(v1) + βf(v2) When V and W are finite dimensional, say n- and m- dimensional, a linear map can be represented by a m × n matrix with elements in the field of scalars. The matrix acts by multiplication on the coordinates of the vectors of V , written as n × 1 matrices (which are called column vectors) and provides the coordinates of the image vectors in W . The matrix hence depends on the choice of basis in the two vector spaces. The set of all n × m matrices with elements in R (resp. C) form a real (resp. complex) vector space of dimension mn. These vector spaces are denoted Rn×m or Cn×m respectively. 11 1.2 Inverse of a Linear Map Let V and W be finite dimensional, say n- and m- dimensional. By choosing bases in the two spaces any linear map f : V ! W is represented by a matrix A 2 Cm×n. Proposition 1.1 If f : V ! W is invertible the matrix A must also be invertible and the two vector spaces must have the same dimension (say n). Invertible matrices are also called non-singular. The inverse A−1 can be computed by the so-called Cramer rule 1 A−1 = Adj(A) det A where he \algebraic adjoint " Adj(A) is the transpose of a matrix having in position (i; j) the determinant of the complement to row i and column j (an n − 1 × n − 1 matrix) multiplied by the factor (−1)i+j. This rule is seldom used for actual computations. There is a wealth of algorithms to compute inverses which apply to matrices of specific structure. In fact computing inverses is seldom of interest per se; one may rather have to look for algorithms which compute solutions of a linear system of equations Ax = b. 12 1.3 Inner products and norms An inner product on V is a map h·; ·i : V × V ! C satisfying the following require- ments. • Conjugate symmetry: hx; yi = hy; xi • Linearity in the first argument: hax; yi = ahx; yihx + y; zi = hx; zi + hy; zi • Positive-definiteness: hx; xi ≥ 0 ; hx; xi = 0 ) x = 0 The norm induced by a inner product is kxk = +phx; xi: This is the \length" of the vector x. Directly from the axioms, one can prove the Cauchy- Schwarz inequality: for x; y elements of V jhx; yij ≤ kxk · kyk with equality if and only if x and y are linearly dependent. This is one of the most important inequalities in mathematics. It is also known in the Russian literature as the CauchyBun- yakovskySchwarz inequality. 13 1.4 Inner products in coordinate spaces (1) In the vector space Rn (in Cn you must use conjugation), the bilinear function: n n > h·; ·i : R × R −! R; hu; vi := u v Column vectors has all the prescribed properties to be an inner product. It induces the Euclidean norm on n: R p n > k · k : R −! R+ ; kuk := u u The bilinear form defined on Cn×m × Cn×m by hA; Bi :(A; B) 7! tr (AB¯>) = tr (B¯>A) (1.1) where tr denotes trace and B¯ is the complex conjugate of B, is a bona fide inner product on Cn×m. The matrix norm defined by the inner product (1.1), 1=2 > 1=2 kAkF := hA; Ai = [tr AA¯ ] (1.2) is called the Frobenius, or the weak norm of A. 14 1.5 Inner products in coordinate spaces (2) More general inner products in Cn can be defined as follows. Definition 1.1 A square matrix A 2 Cn×n is Hermitian if A¯> = A and positive semidefinite if x¯>Ax ≥ 0 for all x 2 Cn. The matrix is called positive definite if x¯>Ax can be zero only when x = 0. There are well-known tests of positive definiteness based on checking the signs of the prin- cipal minors which should all be positive. Given an Hermitian positive definite matrix Q we define the weighted inner product h·; ·iQ in the coordinate space Cn by setting > hx; yiQ : = x¯ Qy This clearly satisfies the axioms of inner product.