MSc in Systems and Control Engineering Lab component for Mathematical Techniques module Numerical Linear Algebra: Matrix Decompositions by G.D. Halikias and J. Kiskiras London October 2006 1 Introduction The purpose of this lab is to twofold: The first aim is to familiarize you with the main linear algebraic Matlab functions, related to the MSc module “Mathematical Techniques”, with special emphasis on eigenvalues and eigen- vectors, spectral decomposition of square matrices, the Jordan and Schur form of a matrix, the singular value decomposition and some of its applica- tions (computation of rank and nullity of a matrix, generation of orthogo- nal bases of its range and kernel, solution of linear equations, reduced-rank matrix approximations). This material, along with some basic theory and simple computational exercises is contained in sections 2 and 3. You can go through these exercises at your own pace (during or outside the formal lab sessions); for additional theory background consult your notes and the suggested textbooks. For difficulties with Matlab programming consult the lab demonstrators and use liberally Matlab’s on-line help functions. The exercises included in section 4 are more demanding, both analytically and computationally. Do not attempt these before you have been familiar with the basics of Matlab programming and the exercises in sections 2 and 3. The exercises include: 1. Programming a function that reduces a matrix into row echelon form using a sequence of elementary transformations, and using your pro- gramme to solve linear equations and calculate the numerical rank and row-span of a matrix. 2. Verifying “Sylvester’s law of inertia” via simple numerical experiments. 3. Verifying Perron/Frobenious theorem for non-negative matrices and calculating numerically the dominant eigenvalue and eigenvector via an iterative procedure (“Power method”). 4. Implementing least-squares estimation of linear models by fitting the “best” to a set of points lying on the plane. Note that all exercises form part of the formal assessment of the module. Submit your report by the set deadline; this should include answers to all questions, Matlab code, plots and analytical calculations, whenever needed, to support your arguments. Do not include lengthy pieces of theory in your report! 2 2 Eigenvalues and eigenvectors The term “eigenvalue” is a partial translation of the German “eigenvert”. A complete translation would be something like “own value” or “characteristic value”, but these are rarely used. As an application, eigenvalues can corre- spond to frequencies of vibration, or critical values of stability parameters, or energy levels of atoms. To motivate the definitions of eigenvalue and eigenvector and to give a simple example of one of their applications, we begin by considering the 2£2 matrix · ¸ 3 3 A = 1 5 If we set £ ¤0 x1 = 3 ¡1 and £ ¤0 x2 = 1 1 then it is easy to verify that · ¸ · ¸ · ¸ · ¸ · ¸ · ¸ · ¸ · ¸ 3 3 3 6 3 3 3 1 6 1 Ax = = = 2 and Ax = = = 6 1 1 5 ¡1 ¡2 ¡1 2 1 5 1 6 1 In other words the linear transformation A simply multiplies the vector x1 by a factor of two and the vector x2 by a factor of six. We call the number two an eigenvalue of A corresponding to the eigenvector x1 and the number six an eigenvalue of the matrix A corresponding to the eigenvector x2, respectively. Notice that in this case x1 and x2 are linearly independent and hence they form a basis of R2. Then it follows that every x 2 R2 can be written uniquely in the form x = c1x1 + c2x2, where c1; c2 2 R, so that Ax = A(c1x1 + c2x2) = c1Ax1 + c2Ax2 = 2c1x1 + 6c2x2 and more generally k k k A = 2 c1x1 + 6 c2x2 Thus, it is obvious that the knowledge of the eigenvalues and eigenvectors of a matrix can be used to simplify computations with the matrix. 3 Problem 2.1. The fundamental algebraic eigen-problem is the determination of those values of ¸ for which the set of n homogeneous linear equations in n unknowns Ax = ¸x (1) has a non-trivial solution. Equation (1) may be written in the form (¸I ¡ A)x = 0 and for almost all ¸ this set of equations has the unique solution x = 0. Non-trivial solutions arise if and only if the matrix (¸I ¡A) is singular, that is det(¸I ¡ A) = 0 Definition 2.1. The equation det(¸I ¡ A) = 0 is called the characteristic equation of A, while the polynomial p(¸) := det(¸I ¡ A) is called the charac- teristic polynomial of A. Remark 2.1. Expanding the determinant in the above definition, the char- acteristic polynomial can be written as n¡1 n det(¸I ¡ A) = ®0 + ®1¸ + ¢ ¢ ¢ + ®n¡1¸ + ¸ Yn (2) = (¸ ¡ ¸i) i=1 Since the coefficient of ¸n is not zero, the characteristic equation always has n roots (i.e. the eigenvalues of A), say ¸1; ¸2; : : : ; ¸n, which may be real or complex, of any multiplicities up to n. In general, if the matrix (¸I ¡ A) is of rank less than (n ¡ 1) then there will be more than one independent vectors satisfying (¸I ¡ A)x = 0. It is evident that if x is a solution to (¸I ¡ A)x = 0, then kx is also a solution for every scalar k, so that even when (¸I ¡ A) is of rank n ¡ 1, the eigenvector corresponding to ¸ is arbitrary to the extent of a constant multiplier. This gives rise to the following definition. 4 Definition 2.2. For any eigenvalue ¸ of a n-by-n matrix A, the set ker(¸I ¡ A) = fx 2 Rn :(¸I ¡ A)x = 0g is the nullspace of the matrix ¸I ¡ A, a subspace of Rn, or else called the eigenspace corresponding to the eigenvalue ¸. · ¸ 3 3 Example 2.1. Find the eigenvalues of A = and find all eigenvectors. 1 5 The characteristic equation of A is µ· ¸¶ ¸ ¡ 3 ¡3 det(¸I ¡ A) = 0 ) det = 0 ) (¸ ¡ 3)(¸ ¡ 5) ¡ 3 = 0 ¡1 ¸ ¡ 5 ) ¸2 ¡ 8¸ + 12 = 0 ) (¸ ¡ 2)(¸ ¡ 6) = 0 Hence the eigenvalues of A are ¸1 = 2 and ¸2 = 6, with corresponding eigenvectors · ¸ · ¸ 3 1 x = and x = 1 ¡1 2 1 respectively. The eigenspace corresponding to the eigenvalue 2 is ½ · ¸ ¾ ½ · ¸ ¾ 1 3 3 x 2 R2 : x = 0 = c : c 2 R 1 3 ¡1 The eigenspace corresponding to the eigenvalue 6 is ½ · ¸ ¾ ½ · ¸ ¾ ¡3 3 1 x 2 R2 : x = 0 = c : c 2 R : 1 ¡1 1 According to definition 2.1 and remark 2.1 if ¸i is an eigenvalue of a matrix A, then it is a root of the characteristic polynomial p(¸) := det(¸I ¡ A) = Qn i=1(¸ ¡ ¸i). The multiplicity of this root, i.e. the number of times the factor (¸ ¡ ¸i) appears in equation (2), is called the algebraic multiplicity of the eigenvalue ¸i. If the algebraic multiplicity of an eigenvalue ¸i is equal to 1 then ¸i is said to be simple. Proposition 2.1. Any n £ n matrix of rank n ¡ r (1 · r · n) has a zero eigenvalue (¸i = 0) of multiplicity m ¸ r. 5 Further, there is another notion of multiplicity of an eigenvalue. Definition 2.3. The dimension of the eigenspace ker(¸I ¡ A) is called ge- ometric multiplicity of the eigenvalue ¸. Essentially, the above definition suggests that the geometric multiplicity of an eigenvalue, say ¸i, is equal to the number of linearly independent eigenvectors associated with ¸i. Proposition 2.2. Geometric multiplicity of an eigenvalue cannot exceed its algebraic multiplicity. In other words, if the n £ n matrix A has an eigenvalue ¸i and ¸iI ¡ A has rank n ¡ r, then ¸i has multiplicity m ¸ r. Definition 2.4. If the algebraic multiplicity of an eigenvalue ¸i exceeds its geometric multiplicity, then ¸i is said to be a defective eigenvalue. Further, a matrix with a defective eigenvalue is referred to as a defective matrix. Example 2.2. Consider the matrix 2 3 17 ¡10 ¡5 A = 4 45 ¡28 ¡155 ¡30 20 12 In order to find the eigenvalues of A, we need to find the roots of 02 31 ¸ ¡ 17 10 5 det @4 ¡45 ¸ + 28 15 5A = 0 30 ¡20 ¸ ¡ 12 2 or equivalently, (¸ + 3)(¸ ¡ 2) = 0. The eigenvalues are therefore ¸1 = ¡3 and ¸2 = 2. An eigenvector corresponding to the eigenvalue -3 is a solution of the system (¡3I ¡ A)x = 0, i.e. 2 3 2 3 20 ¡10 ¡5 1 (3I + A)x = 0 ) 4 45 ¡25 ¡155 x = 0 with root x1 = 4 3 5 ¡30 20 15 ¡2 6 An eigenvector corresponding to the eigenvalue 2 is a solution of the system (2I ¡ A)x = 0, i.e. 2 3 2 3 2 3 ¡15 10 5 1 2 4¡45 30 15 5 x = 0 with roots x2 = 405 ; x3 = 435 30 ¡20 ¡10 3 0 Note that the eigenspace corresponding to the eigenvalue -3 is a line passing through the origin, while the eigenspace corresponding to the eigenvalue 2 is a plane through the origin. Note also that the eigenvectors x1; x2; x3 are linearly independent and so form a basis for R3. Example 2.3. Consider the matrix 2 3 2 ¡1 0 A = 41 0 05 0 0 3 In order to find the eigenvalues of A, we need to find the roots of 02 31 ¸ ¡ 2 1 0 det @4 ¡1 ¸ 0 5A = 0 0 0 ¸ ¡ 3 2 in other words, (¸ ¡ 3)(¸ ¡ 1) = 0. The eigenvalues are therefore ¸1 = 3 and ¸2 = 1. An eigenvector corresponding to the eigenvalue 3 is a solution of 2 3 2 3 1 1 0 0 (3I ¡ A)x = 0 ) 4¡1 3 05 x = 0; with root x1 = 405 0 0 0 1 An eigenvector corresponding to the eigenvalue 1 is a solution of 2 3 2 3 ¡1 1 0 1 (I ¡ A)x = 0 ) 4¡1 1 0 5 x = 0; with root x2 = 415 0 0 ¡2 0 Corollary 2.1.