Numerical Linear Algebra (Continued) CS 740 Ajit Rajwade 1 Eigenvectors and Eigenvalues • Given an n x n matrix A, a non-zero vector v is said to be an eigenvector of A with eigenvalue λ if Av = λv. • In other words, when transformed by matrix A, vector v either shrinks (|λ| < 1) or expands (|λ| > 1) (decreases or increases in magnitude) but its direction remains the same or flips over. Of course, when |λ| = 1, the magnitude of v undergoes no change. 2 Eigenvectors and Eigenvalues • The preservation of the direction of the vector v when transformed by A gives rise to the name “eigen” – which is German for “self”. • Eigenvectors and eigenvalues come in useful in several applications in physics, image and signal processing and machine learning. • Here we stick to real matrices, though almost all the theory is applicable to complex matrices as well. 3 Examples 4 Non-uniqueness • Eigenvectors and eigenvalues are not unique. An n x n matrix has n eigenvectors (not necessarily all distinct) and n eigenvalues (not necessarily all distinct). • An eigenvector when scaled by an arbitrary non-zero constant remains an eigenvector with the same eigenvalue. By convention, eigenvectors are normalized to unit magnitude. 5 Characteristic equation • The eigenvalues of A can be obtained by finding the roots of the following equation called the characteristic equation: Av v Iv (A I)v 0 This is a homogeneous equation and it will have a solution if and only if (A I) is a singular matrix. det(A I) 0 • This equation has n (not necessarily distinct or real) roots – the eigenvalues of A. The equation can be written as follows: p() ( 1)( 2 )...( n ) 0 6 Characteristic equation • The characteristic equation of an n x n matrix A has degree n. • It always has n roots – this is as per a result called the fundamental theorem of algebra. • The roots may not be distinct. • The roots may not be real even if A is real. 7 Matrix of eigenvectors and eigenvalues A Rnn Av1 1v1 Av2 2v2 . Avn nvn 0 0 1 2 0 A(v | v |...| v ) ( v | v |...| v ) (v | v |...| v ) . 0 1 2 n 1 1 2 2 n n 1 2 n . 0 0 0 0 0 n AV V 8 http://en.wikipedia.org/wiki/Eigenvalues_and_eigenvectors Physical example 1 In this transformation, the red arrow changes direction but the blue arrow does not. The blue arrow is an eigenvector of this shear mapping, and since its length is unchanged its eigenvalue is 1. The red direction in the second figure is also an eigenvector – it is the direction in which the cloth is stretched and it has an eigenvalue greater than 1. 9 Physical example 1 • This example helps us design transformation matrices that accomplish a certain task. • Let’s say you wanted to stretch by a factor of 2 in direction [1,1] and apply no stretch or shrinking in direction [1,0]. • We treat these directions as eigenvectors with eigenvalues 2 and 1 respectively. • This gives us 1 1 2 0 V , 1 0 0 1 AV V 1 1 1 A VV 10 0 2 Physical example 2 • Consider the following matrix which you will often see in computer graphics: cos sin 0 sin cos 0 0 0 1 • It is a matrix that rotates a 3D point about the Z axis – the Z coordinate remains unchanged, but the X and Y coordinates change by a rotation in the XY plane. • In this case, the Z axis, i.e. vector (0 0 1)t is an eigenvector with eigenvalue 1. • If the axis of rotation changes, the rotation matrix changes. But the axis of rotation will always be an eigenvector of the rotation matrix with eigenvalue 1. 11 Properties of Eigenvalues and Eigenvectors • Trace of a matrix = sum of its eigenvalues • Determinant of a matrix = product of its eigenvalues (hence a singular matrix has at least one 0 eigenvalue). • A matrix is invertible iff (= if and only if) all its eigenvalues are non-zero. • The eigenvalues of A-1 are reciprocals of the eigenvalues of A (can you prove this?). • The eigenvectors of A and A-1 are the same (can you reason why?). • Every eigenvalue of an orthonormal/unitary matrix has absolute value 1 (why?). • Complex eigenvalues always occur in conjugate pairs, i.e. if a+ib is an eigenvalue of real matrix A for eigenvector v, then a-ib will also be an eigenvalue of A for eigenvector v* (can you prove this?). 12 Properties of Eigenvalues and Eigenvectors • Given the eigenvalues and eigenvectors of A, what can you say about the eigenvalues and eigenvectors of A-σI for some scalar σ? • Given the eigenvalues and eigenvectors of A, what can you say about the eigenvalues and eigenvectors of A2 or An for some n > 0? • Note that An =A*A*A*…*A (product with itself n-1 times). This is different from A.^n for which A.^n(i,j) = A(i,j)^n. 13 Properties of Eigenvalues and Eigenvectors • Trace of a matrix = sum of its eigenvalues • Determinant of a matrix = product of its eigenvalues p() ( 1)( 2 )...( n ) det(I A) n n p(0) (1) 12...n det(A) (1) det(A) det(A) 12...n n1 The coefficien t of in p() is (1 2 ... n ). The coefficien t of the term containing n1 in det(I A) can be proved to be the negative sum of the diagonal entries of A, i.e. - trace(A). 14 Hence trace(A) (1 2 ... n ). Eigenvectors and eigenvalues of special matrices • Diagonal matrix: eigenvalues are elements along the diagonal, eigenvectors are the column vectors normalized to unit magnitude. • Symmetric matrix: eigenvalues are real (proof), eigenvectors corresponding to distinct eigenvalues are orthogonal (proof). AV V, A VV T • The converse is also true, i.e. a matrix with orthonormal eigenvectors and real eigenvalues is symmetric. A VV T AT (VV T )T VV T A 15 Computation of Eigenvectors and Eigenvalues • The following MATLAB commands compute eigenvectors and eigenvalues: [V,D] = eig(A) returns two outputs. D is a diagonal matrix containing the eigenvalues. V is a matrix whose columns are the corresponding right eigenvectors. [V,D] = eigs(A) returns a diagonal matrix D of A's six largest magnitude eigenvalues and a matrix V whose columns are the corresponding eigenvectors. [V,D] = eigs(A,k) returns a diagonal matrix D of A's k largest magnitude eigenvalues and a matrix V whose columns are the corresponding eigenvectors. 16 Computation of Eigenvectors and Eigenvalues • Eigenvalues can be computed by computing the roots of the characteristic equation. • This is easy for 2 x 2 matrices, but not for larger matrices, since polynomials in arbitrary degree (greater than 5) usually have no closed form solutions. • How do you compute eigenvectors and eigenvalues? 17 Computing eigenvectors • Assume n x n matrix A has a unique eigenvalue λ1 of largest magnitude. Let the corresponding eigenvector be v1. • Consider vector x0 which is expressible as a linear combination of eigenvectors of A, i.e. n x0 ivi i1 • Repeated multiplication of A with x0 converges to a multiple of v1. Why? 18 n x0 ivi i1 k n n n x A k x A k v k v k v i v k 0 i i i i i 1 1 1 i i i1 i1 i2 1 k 11v1 Each such term tends to 0, as k tends to ∞, because λ1 is the largest magnitude eigenvalue of A. This method of eigenvector computation is called as power iteration. It converges to the eigenvector with the largest eigenvalue. Caution: The possibility that the starting vector x0 has no component in the direction v1 (i.e. if α1 = 0) is very small, and can be eliminated by slightly and randomly perturbing the starting vector. Caution: If there are two or more eigenvalues that are the largest and have the same magnitude, the starting vector will converge to some linear combination of the corresponding eigenvectors. Caution: If the initial vector is real and A is real, it will not converge to a complex eigenvector!19 Computation of Eigenvectors and Eigenvalues • How do you compute the largest eigenvalue given the eigenvector? t t t v1 Av1 Av1 1v1 v1 Av1 v11v1 1 t v1v1 • What if you wanted to compute the eigenvector corresponding to the smallest eigenvalue? Work with A-1 instead of A. • What about other eigenvectors? There exist more advanced algorithms for those, which we skip in this course. But we handle one special case. 20 Computation of Eigenvectors and Eigenvalues • Let A be a symmetric matrix. So its eigenvalues are real and its eigenvectors are orthonormal. AV VΛ n T t A VV ivivi i1 • To compute the second eigenvector, work with the following matrix instead of A (v1 ) t A A 1v1v1 where v1 and λ1 are the estimated first eigenvector and first eigenvalue. • The eigenvector of A(-v1) corresponding to its largest eigenvalue is the eigenvector of A corresponding to its second largest eiegnvalue. 21 Computation of Eigenvectors and Eigenvalues • Several more advanced algorithms exist – we skip the algorithms in this course. • The algorithms have ready implementation in a famous library called LAPACK – written in Fortran 90, with C/C++/MATLAB interfaces. 22 Singular Value Decomposition (SVD) 23 Singular value Decomposition • For any m x n matrix A, the following decomposition always exists: A USVT , A Rmn , T T mm U U UU Im,U R , T T nn Diagonal matrix with non- V V VV In,V R , negative entries on the mn diagonal – called singular S R values.