Introduction to Linear Algebra V Jack Xin (Lecture) and J. Ernie Esser (Lab) ∗ Abstract Eigenvalue, eigenvector, Hermitian matrices, orthogonality, orthonormal basis, singular value decomposition. 1 Eigenvalue and Eigenvector For an n × n matrix A, if A x = λ x; (1.1) has a nonzero solution x for some complex number λ, then x is eigenvector corresponding to eigenvalue λ. Equation (1.1) is same as saying x belongs to the null space of A − λI, or A − λI is singular or the so called characteristic equation holds: det(A − λI) ≡ p(λ) = 0; (1.2) p(λ) is a polynomial of degree n, hence n complex eigenvalues. In Matlab, eigenvalues and eigenvectors are given by [V,D]=eig(A), where columns of V are eigenvectors, D is a diagonal matrix with entries being eigenvalues. Matrix A is diagonalizable (A = V DV −1, D diagonal) if it has n linearly independent eigenvectors. A sufficient condition is that all n eigenvalues are distinct. 2 Hermitian Matrix For any complex valued matrix A, define AH = A¯T , where bar is complex conjugate. A is Hermitian if AH = A, for example: 3 2 − i A = 2 + i 4 ∗Department of Mathematics, UCI, Irvine, CA 92617. 1 A basic fact is that eigenvalues of a Hermitian matrix A are real, and eigenvectors of distinct eigenvalues are orthogonal. Two complex column vectors x and y of the same dimension are orthogonal if xH y = 0. The proof is short and given below. Consider eigenvalue equation: Ax = λx, and let α = xH Ax, then: α¯ = αH = (xH Ax)H = xH Ax = α; so α is real. On the other hand, α = λxH x, so λ is real. Let xi (i=1,2) be eigenvectors corresponding to distinct eigenvalues λi (i=1,2). We have the identities: H H H (Ax1) x1 = x1 Ax2 = λ2x1 x2; H H H H H H (Ax1) x2 = (x2 Ax1) = (λ1x2 x1) = λ1x1 x2; H so λ1 6= λ2 implies x1 x2 = 0. It follows that by choosing orthogonal basis for each eigenspace, Hermitian matrix A has n-orthonormal (orthogonal and of unit length) eigen-vectors, which become an orthogonal basis for Cn. Putting orthonomal eigenvectors as columns yield a matrix U so that U H U = I, which is called unitary matrix. If A is real, unitary matrix becomes orthogonal matrix U T U = I. Clearly a Hermitian matrix can be diagonalized by a unitary matrix (A = UDU H ). The necessary and sufficient condition for unitary diagonalization of a matrix is that it is normal, or satisfying the equation: AAH = AH A: This includes any skew-Hermitian matrix (AH = −A). 3 Orthogonal Basis n T In R , let v1, v2, ..., vn be n orthonormal column vectors, vi vj = δij (=1 if i=j, otherwise 0). Then each vector v has the representation: n X T v = cj vj; cj = v vj: j=1 2 Here cjvj is the projection of v onto vj. Pn If u = i=1 bjvj, then: n T X u v = bjcj; j=1 and n 2 T X 2 kuk2 = length of u squared = u u = bj ; j=1 which is called Parseval formula (generalization of Pythagorean theorem). An example of N-dimensional orthogonal basis is given by the discrete cosine transform: X = DCT ∗ x; where DCT is n × n orthogonal matrix: 1 DCT = w(k) cos π n − (k − 1)=N ; 2 k=1:N;n=1:N p w(1) = 1= N; w(k) = p2=N, if k ≥ 2. In Matlab, X = dct(x), DCT=dct(eye(n)), n ≥ 2. 4 Singular Value Decomposition (SVD) For a general real m × n matrix A, a factorization similar to orthogonal diagonalization of symmetric matrices (AT = A) is SVD. Suppose m ≥ n, then there are m × m orthogonal matrix U and n×n orthogonal matrix V , also non-negative numbers σ1 ≥ σ2 ≥ · · · ≥ σn ≥ 0, such that A = UΣV T ; (4.3) and Σ = [diag([σ1 σ2 ··· σn]); 0]; is m × n matrix, 0 is m-n zero rows of dimension n. The numbers σi's are called singular values. It follows from (4.3) that AT A = V ΣT ΣV T , T 2 Σ Σ = diag([σ1; σ2 ··· σn]); 3 2 T so σj (j=1:n) are real eigenvalues of A A, while columns of V are corresponding orthogonal eigenvectors. From AV = UΣ, we see that each column of U is uj = Avj/σj, j=1, 2, ..., r, where r is the rank of A (or the number of nonzero singular values). Check that uj's are orthonormal. Putting uj's (j=1:r) together gives part of the column vectors of U (the U1 in U = [U1 U2]), the other part U2 is the orthogonal complement. Since uj's (j=1:r) span than ? T the range of A (range(A)), U2 consist of orthonormal column vectors in range(A) = N(A ), the nullspace of AT . In Matlab, [U,S,V]=svd(A) gives the result (S in lieu of Σ). Keeping k < r of the T singular values gives the rank-k approximation of A, or A ≈ USkV , where Sk is obtained from S by zeroing out σj (j=k+1:r), so called low rank approximation, which is useful in image compression among other applications. The approximation is optimal in Frobenius sense (or in the sense of Euclidean, l2, norm of matrices). References [1] S. Leon, Linear Algebra with Applications, Pearson, Prentice-Hall, 2010. 4.