DOCSLIB.ORG

VII.E) the Singular Value Decomposition (SVD

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
VII.E) the Singular Value Decomposition (SVD (VII.E) The Singular Value Decomposition (SVD) In this section we describe a generalization of the Spectral Theo- rem to non-normal operators, and even to transformations between different vector spaces. This is a computationally useful generaliza- tion, with applications to data fitting and data compression in par- ticular. It is used in many branches of science, in statistics, and even machine learning. As we develop the underlying mathematics, we will encounter two closely associated constructions: the polar de- composition and pseudoinverse, in each case presenting the version for operators followed by that for matrices. To begin, let (V, h·, ·i) be an n-dimensional inner-product space, T : V ! V an operator. By Spectral Theorem III, if T is normal then it has a unitary eigenbasis B, with complex eigenvalues s1,..., ss. In particular, T is unitary iff these jsij = 1 (since preserving lengths of a unitary eigenbasis is equivalent to preserving lengths of all vectors), and self-adjoint iff the si are real. So T is both unitary and self-adjoint iff T has only +1 and −1 as eigenvalues, which is to say T is an ? orthogonal reflection, acting on V = W ⊕ W by IdW on W and ? −IdW? on W . But the broader point here is that we should have in mind the following “table of analogies”: operators numbers normal C unitary S1 self-adjoint R ?? R≥0 where S1 ⊂ C is the unit circle. So, how do we complete it? 1 2 (VII.E) THE SINGULAR VALUE DECOMPOSITION (SVD) DEFINITION 1. T is positive (resp. strictly positive) if it is nor- mal, with all eigenvalues in R≥0 (resp. R>0). Since they are unitarily diagonalizable with real eigenvalues, pos- itive operators are self-adjoint. Given a self-adjoint (or normal) op- erator T, any ~v 2 V is a sum of orthogonal eigenvectors w~ j with eigenvalues sj; so 2 h~v, T~vi = ∑hw~ i, Tw~ ji = ∑ sjhw~ i, w~ ji = ∑ sjjjw~ jjj i,j i.j j is in R≥0 for all ~v if and only if T is positive. If V is complex, the condition that h~v, T~vi ≥ 0 (8~v) on its own gives self-adjointness (Ex- ercise VII.D.7), hence positivity, of T. (If V is real, this condition alone isn’t enough.) EXAMPLE 2. Orthogonal projections are positive, since they are self-adjoint with eigenvalues 0 and 1. Polar decomposition. A nonzero complex number can be writ- iq ten (in “polar form”) as a product re , for unique r 2 R>0 and eiq 2 S1. If T is a normal operator, then it has a unitary eigen- iq iqn basis B = fvˆ1,..., vˆng, with [T]B = diagfr1e 1 ,..., rne g (ri 2 R≥0), and defining jTj, U by [jTj]B = diagfr1,..., rng and [U]B = diagfeiq1 ,..., eiqn g, we have T = UjTj with U unitary and jTj pos- itive, and UjTj = jTjU. One might expect such “polar decompo- sitions” of operators to stop there. But there exists an analogue for arbitrary transformations, one that will lead on to the SVD as well! To formulate this general polar decomposition, let T : V ! V be an arbitrary linear transformation. Notice that T†T is self-adjoint1 and, in fact, positive: h~v, T†T~vi = hT~v, T~vi ≥ 0, † since h·, ·i is an inner product. So we have a unitary B with [T T]B = diagfl1,..., lng, and all li 2 R≥0. (Henceforth we impose the con- vention that l1 ≥ l2 ≥ ... ≥ ln.) Taking nonnegative square 1trivially so: (T†T)† = T†T†† = T†T. (VII.E) THE SINGULAR VALUE DECOMPOSITION (SVD) 3 p roots mi = li 2 R≥0, we define a new operator jTj by [jTj]B = 2 † diagfm1,..., mng; clearly jTj = T T, and jTj is positive. THEOREM 3. There exists a unitary operator U such that T = UjTj. This “polar decomposition” is unique exactly when T is invertible. More- over, U and jTj commute exactly when T is normal. 2 PROOF. We have kT~vk = hT~v, T~vi = h~v, T†T~vi = h~v, jTj2~vi = hjTj~v, jTj~vi = kjTj~vk2 since jTj is self-adjoint. Consequently T and jTj have the same kernel, hence (by Rank + Nullity) the same rank; so while their images (as subspaces of V) may differ, they have the same dimension. ? By Spectral Theorem I, V = imjTj ⊕ kerjTj. So dim(kerjTj) = dim((imT)?), and we fix an invertible transformation U0 : kerjTj ! (imT)? sending some choice of unitary basis to another unitary ba- sis. Writing ~v = jTjw~ +~v0 (with jTj~v0 =~0), we now define ? ? U : imjTj ⊕ kerjTj ! imT ⊕ (imT)? | {z } | {z } V V by U~v := Tw~ + U0~v0. Since kTw~ + U0~v0k2 = kTw~ k2 + kU0~v0k2 = kjTjw~ k2 + k~v0k2 = k~vk2, U is unitary; and taking ~v = jTjw~ gives UjTjw~ = Tw~ (8w~ 2 V). −1 Now if T is invertible, then so is jTj (all mi > 0) =) U = TjTj . If it isn’t, then kerjTj 6= f0g and non-uniqueness enters in the choice of U0. Finally, if U and jTj commute, then they simultaneously unitarily diagonalize, and therefore so does T, making T normal. If T is nor- mal, we have constructed U and jTj above, and they commute. 4 (VII.E) THE SINGULAR VALUE DECOMPOSITION (SVD) EXAMPLE 4. Consider V = R4 with the dot product, and 0 2 0 0 0 1 B C B 0 1 0 0 C A = [T]eˆ = B C . @ 0 1 1 0 A 0 0 0 0 Since A has a Jordan block, we know it is not diagonalizable; and so T is not normal. However, 0 4 0 0 0 1 B C † t B 0 2 1 0 C −1 [T T]eˆ = AA = B C = SBDSB @ 0 1 1 0 A 0 0 0 0 with B := feˆ1, eˆ2 +pj−eˆ3, eˆ2 − j+eˆ3, eˆp4g and D := diagf4, h+, h−, 0g ±1+ 5 3± 5 2 (where j± := 2 and h± := 2 = j±). Taking the positive 1 square root D 2 := diagf2, j+, j−, 0g, we get 0 1 2 0 0 0 B 0 p3 p1 0 C 1 −1 B 5 5 C [jTj]eˆ = jAj := SBD 2 S = B C . B B 0 p1 p2 0 C @ 5 5 A 0 0 0 0 On W := spanfeˆ1, eˆ2, eˆ3g = imT = imjTj, we must have UjW = −1 ? TjW (jTj jW ) ; while on spanfeˆ4g = W , U can be taken to act by any scalar of norm 1. Therefore 0 1 1 0 0 0 B 0 p2 p−1 0 C B 5 5 C [U]eˆ = Q := B C , B 0 p1 p2 0 C @ 5 5 A 0 0 0 eiq and A = QjAj. Geometrically, jAj dilates the B-coordinates of a vector, then Q performs a rotation. (VII.E) THE SINGULAR VALUE DECOMPOSITION (SVD) 5 SVD for endomorphisms2. Now let jTj be the (unique) positive † square root of T T as above, and B = fvˆ1,..., vˆng the unitary basis of V under which [jTj]B = diagfm1,..., mng. DEFINITION 5. These fmig are called the singular values of T. † Writing T = UjTj and wˆ i := Uvˆi, U U = IdV =) hwˆ i, wˆ ji = † hU Uvˆi, vˆji = hvˆi, vˆji = dij =)C := fwˆ 1,..., wˆ ng = U(B) is unitary. For ~v 2 V, we have n n T~v = UjTj ∑hvˆi,~vivˆi = ∑hvˆi,~viUjTjvˆi i=1 i=1 n n (1) = ∑ mihvˆi,~viUvˆi = ∑ mihvˆi,~viwˆ i. i=1 i=1 Viewing hvˆi, ·i =: `vˆi : V ! C as a linear functional, and wˆ i : C ! V as a map (sending a 2 C to awˆ i), (1) becomes n = ◦ ` (2) T ∑ miwˆ i vˆi . i=1 How does this look in matrix terms? Let A be an arbitrary3 uni- tary basis of V, and set A := [T]A. Using f1g as a basis of C, (2) yields n n = [ ] [` ] = [ ] [ ]∗ (3) A ∑ mi A wˆ i f1g f1g vˆi A ∑ mi wˆ i A vˆi A. i=1 i=1 ∗ Here [wˆ i]A[vˆi]A is a (n × 1) · (1 × n) matrix product, yielding an n × n matrix of rank one. So (2) (resp. (3)) decompose T (resp. A) into a sum of rank one endomorphisms (resp. matrices) weighted by the singular values. This is a first version of the Singular Value Decom- position. 2The reader may of course replace “unitary” and C by “orthonor- mal”/”orthogonal” and R in what follows. 3What you should actually have in mind here is, in the case where V = Cn with dot product, the standard basis eˆ. 6 (VII.E) THE SINGULAR VALUE DECOMPOSITION (SVD) REMARK 6. If T is positive, we can take U = IdV, so that wˆ i ◦ `vˆi = vˆi ◦ `vˆi =: Prvˆi is the orthogonal projection onto spanfvˆig. Thus (2) merely restates Spectral Theorem I in this case. In fact, if T is nor- † mal, then T, T T, and jTj simultaneously diagonalize. If m˜1,..., m˜n are the eigenvalues of T, then we evidently have mi = jmij; U may then be defined by [U]B = diagfm˜i/mig (with 0/0 interpreted as 1). n Thus (2) reads T = ∑i=1 m˜iPrvˆi , which restates Spectral Theorem III. So in this sense, the SVD generalizes the results in §VII.D. Now since U sends B to C, we have C [T]B = C [U]B[jTj]B = diagfm1,..., mng =: D | {z } In and thus (4) A = [T]A = A[I]CC [T]BB[I]A =: S1DS2, with S1 and S2 unitary matrices (as they change between unitary bases).
Load more

Recommended publications
  • Singular Value Decomposition (SVD)
    San José State University Math 253: Mathematical Methods for Data Visualization Lecture 5: Singular Value Decomposition (SVD) Dr. Guangliang Chen Outline • Matrix SVD Singular Value Decomposition (SVD) Introduction We have seen that symmetric matrices are always (orthogonally) diagonalizable. That is, for any symmetric matrix A ∈ Rn×n, there exist an orthogonal matrix Q = [q1 ... qn] and a diagonal matrix Λ = diag(λ1, . , λn), both real and square, such that A = QΛQT . We have pointed out that λi’s are the eigenvalues of A and qi’s the corresponding eigenvectors (which are orthogonal to each other and have unit norm). Thus, such a factorization is called the eigendecomposition of A, also called the spectral decomposition of A. What about general rectangular matrices? Dr. Guangliang Chen | Mathematics & Statistics, San José State University3/22 Singular Value Decomposition (SVD) Existence of the SVD for general matrices Theorem: For any matrix X ∈ Rn×d, there exist two orthogonal matrices U ∈ Rn×n, V ∈ Rd×d and a nonnegative, “diagonal” matrix Σ ∈ Rn×d (of the same size as X) such that T Xn×d = Un×nΣn×dVd×d. Remark. This is called the Singular Value Decomposition (SVD) of X: • The diagonals of Σ are called the singular values of X (often sorted in decreasing order). • The columns of U are called the left singular vectors of X. • The columns of V are called the right singular vectors of X. Dr. Guangliang Chen | Mathematics & Statistics, San José State University4/22 Singular Value Decomposition (SVD) * * b * b (n>d) b b b * b = * * = b b b * (n<d) * b * * b b Dr.
    [Show full text]
  • Eigen Values and Vectors Matrices and Eigen Vectors
    EIGEN VALUES AND VECTORS MATRICES AND EIGEN VECTORS 2 3 1 11 × = [2 1] [3] [ 5 ] 2 3 3 12 3 × = = 4 × [2 1] [2] [ 8 ] [2] • Scale 3 6 2 × = [2] [4] 2 3 6 24 6 × = = 4 × [2 1] [4] [16] [4] 2 EIGEN VECTOR - PROPERTIES • Eigen vectors can only be found for square matrices • Not every square matrix has eigen vectors. • Given an n x n matrix that does have eigenvectors, there are n of them for example, given a 3 x 3 matrix, there are 3 eigenvectors. • Even if we scale the vector by some amount, we still get the same multiple 3 EIGEN VECTOR - PROPERTIES • Even if we scale the vector by some amount, we still get the same multiple • Because all you’re doing is making it longer, not changing its direction. • All the eigenvectors of a matrix are perpendicular or orthogonal. • This means you can express the data in terms of these perpendicular eigenvectors. • Also, when we find eigenvectors we usually normalize them to length one. 4 EIGEN VALUES - PROPERTIES • Eigenvalues are closely related to eigenvectors. • These scale the eigenvectors • eigenvalues and eigenvectors always come in pairs. 2 3 6 24 6 × = = 4 × [2 1] [4] [16] [4] 5 SPECTRAL THEOREM Theorem: If A ∈ ℝm×n is symmetric matrix (meaning AT = A), then, there exist real numbers (the eigenvalues) λ1, …, λn and orthogonal, non-zero real vectors ϕ1, ϕ2, …, ϕn (the eigenvectors) such that for each i = 1,2,…, n : Aϕi = λiϕi 6 EXAMPLE 30 28 A = [28 30] From spectral theorem: Aϕ = λϕ 7 EXAMPLE 30 28 A = [28 30] From spectral theorem: Aϕ = λϕ ⟹ Aϕ − λIϕ = 0 (A − λI)ϕ = 0 30 − λ 28 = 0 ⟹ λ = 58 and
    [Show full text]
  • Banach J. Math. Anal. 2 (2008), No. 2, 59–67 . OPERATOR-VALUED
    Banach J. Math. Anal. 2 (2008), no. 2, 59–67 Banach Journal of Mathematical Analysis ISSN: 1735-8787 (electronic) http://www.math-analysis.org . OPERATOR-VALUED INNER PRODUCT AND OPERATOR INEQUALITIES JUN ICHI FUJII1 This paper is dedicated to Professor Josip E. Peˇcari´c Submitted by M. S. Moslehian Abstract. The Schwarz inequality and Jensen’s one are fundamental in a Hilbert space. Regarding a sesquilinear map B(X, Y ) = Y ∗X as an operator- valued inner product, we discuss operator versions for the above inequalities and give simple conditions that the equalities hold. 1. Introduction Inequality plays a basic role in analysis and consequently in Mathematics. As surveyed briefly in [6], operator inequalities on a Hilbert space have been discussed particularly since Furuta inequality was established. But it is not easy in general to give a simple condition that the equality in an operator inequality holds. In this note, we observe basic operator inequalities and discuss the equality conditions. To show this, we consider simple linear algebraic structure in operator spaces: For Hilbert spaces H and K, the symbol B(H, K) denotes all the (bounded linear) operators from H to K and B(H) ≡ B(H, H). Then, consider an operator n n matrix A = (Aij) ∈ B(H ), a vector X = (Xj) ∈ B(H, H ) with operator entries Xj ∈ B(H), an operator-valued inner product n ∗ X ∗ Y X = Yj Xj, j=1 Date: Received: 29 March 2008; Accepted 13 May 2008. 2000 Mathematics Subject Classification. Primary 47A63; Secondary 47A75, 47A80. Key words and phrases. Schwarz inequality, Jensen inequality, Operator inequality.
    [Show full text]
  • Singular Value Decomposition (SVD) 2 1.1 Singular Vectors
    Contents 1 Singular Value Decomposition (SVD) 2 1.1 Singular Vectors . .3 1.2 Singular Value Decomposition (SVD) . .7 1.3 Best Rank k Approximations . .8 1.4 Power Method for Computing the Singular Value Decomposition . 11 1.5 Applications of Singular Value Decomposition . 16 1.5.1 Principal Component Analysis . 16 1.5.2 Clustering a Mixture of Spherical Gaussians . 16 1.5.3 An Application of SVD to a Discrete Optimization Problem . 22 1.5.4 SVD as a Compression Algorithm . 24 1.5.5 Spectral Decomposition . 24 1.5.6 Singular Vectors and ranking documents . 25 1.6 Bibliographic Notes . 27 1.7 Exercises . 28 1 1 Singular Value Decomposition (SVD) The singular value decomposition of a matrix A is the factorization of A into the product of three matrices A = UDV T where the columns of U and V are orthonormal and the matrix D is diagonal with positive real entries. The SVD is useful in many tasks. Here we mention some examples. First, in many applications, the data matrix A is close to a matrix of low rank and it is useful to find a low rank matrix which is a good approximation to the data matrix . We will show that from the singular value decomposition of A, we can get the matrix B of rank k which best approximates A; in fact we can do this for every k. Also, singular value decomposition is defined for all matrices (rectangular or square) unlike the more commonly used spectral decomposition in Linear Algebra. The reader familiar with eigenvectors and eigenvalues (we do not assume familiarity here) will also realize that we need conditions on the matrix to ensure orthogonality of eigenvectors.
    [Show full text]
  • A Singularly Valuable Decomposition: the SVD of a Matrix Dan Kalman
    A Singularly Valuable Decomposition: The SVD of a Matrix Dan Kalman Dan Kalman is an assistant professor at American University in Washington, DC. From 1985 to 1993 he worked as an applied mathematician in the aerospace industry. It was during that period that he first learned about the SVD and its applications. He is very happy to be teaching again and is avidly following all the discussions and presentations about the teaching of linear algebra. Every teacher of linear algebra should be familiar with the matrix singular value deco~??positiolz(or SVD). It has interesting and attractive algebraic properties, and conveys important geometrical and theoretical insights about linear transformations. The close connection between the SVD and the well-known theo1-j~of diagonalization for sylnmetric matrices makes the topic immediately accessible to linear algebra teachers and, indeed, a natural extension of what these teachers already know. At the same time, the SVD has fundamental importance in several different applications of linear algebra. Gilbert Strang was aware of these facts when he introduced the SVD in his now classical text [22, p. 1421, obselving that "it is not nearly as famous as it should be." Golub and Van Loan ascribe a central significance to the SVD in their defini- tive explication of numerical matrix methods [8, p, xivl, stating that "perhaps the most recurring theme in the book is the practical and theoretical value" of the SVD. Additional evidence of the SVD's significance is its central role in a number of re- cent papers in :Matlgenzatics ivlagazine and the Atnericalz Mathematical ilironthly; for example, [2, 3, 17, 231.
    [Show full text]
  • Limited Memory Block Krylov Subspace Optimization for Computing Dominant Singular Value Decompositions
    Limited Memory Block Krylov Subspace Optimization for Computing Dominant Singular Value Decompositions Xin Liu∗ Zaiwen Weny Yin Zhangz March 22, 2012 Abstract In many data-intensive applications, the use of principal component analysis (PCA) and other related techniques is ubiquitous for dimension reduction, data mining or other transformational purposes. Such transformations often require efficiently, reliably and accurately computing dominant singular value de- compositions (SVDs) of large unstructured matrices. In this paper, we propose and study a subspace optimization technique to significantly accelerate the classic simultaneous iteration method. We analyze the convergence of the proposed algorithm, and numerically compare it with several state-of-the-art SVD solvers under the MATLAB environment. Extensive computational results show that on a wide range of large unstructured matrices, the proposed algorithm can often provide improved efficiency or robustness over existing algorithms. Keywords. subspace optimization, dominant singular value decomposition, Krylov subspace, eigen- value decomposition 1 Introduction Singular value decomposition (SVD) is a fundamental and enormously useful tool in matrix computations, such as determining the pseudo-inverse, the range or null space, or the rank of a matrix, solving regular or total least squares data fitting problems, or computing low-rank approximations to a matrix, just to mention a few. The need for computing SVDs also frequently arises from diverse fields in statistics, signal processing, data mining or compression, and from various dimension-reduction models of large-scale dynamic systems. Usually, instead of acquiring all the singular values and vectors of a matrix, it suffices to compute a set of dominant (i.e., the largest) singular values and their corresponding singular vectors in order to obtain the most valuable and relevant information about the underlying dataset or system.
    [Show full text]
  • Computing the Polar Decomposition---With Applications
    This 1984 technical report was published as N. J. Higham. Computing the polar decomposition---with applications. SIAM J. Sci. Stat. Comput., 7(4):1160-1174, 1986. The appendix contains an alternative proof omitted in the published paper. Pages 19-20 are missing from the copy that was scanned. The report is mainly of interest for how it was produced. - The cover was prepared in Vizawrite 64 on a Commodore 64 and was printed on an Epson FX-80 dot matrix printer. - The main text was prepared in Vuwriter on an Apricot PC and printed on an unknown dot matrix printer. - The bibliography was prepared in Superbase on a Commodore 64 and printed on an Epson FX-80 dot matrix printer. COMPUTING THE POLAR DECOMPOSITION - WITH APPLICATIONS N.J. Higham • Numerical Analysis Report No. 94 November 1984 This work was carried out with the support of a SERC Research Studentship • Department of Mathematics University of Manchester Manchester M13 9PL ENGLAND Univeriity of Manchester/UMIST Joint Nu•erical Analysis Reports D•p•rt••nt of Math••atic1 Dep•rt•ent of Mathe~atic1 The Victoria Univ•rsity University of Manchester Institute of of Manchester Sci•nce •nd Technology Reque1t1 for individu•l t•chnical reports •ay be addressed to Dr C.T.H. Baker, Depart•ent of Mathe••tics, The University, Mancheiter M13 9PL ABSTRACT .. A quadratically convergent Newton method for computing the polar decomposition of a full-rank matrix is presented and analysed. Acceleration parameters are introduced so as to enhance the initial rate of convergence and it is shown how reliable estimates of the optimal parameters may be computed in practice.
    [Show full text]
  • Geometry of Matrix Decompositions Seen Through Optimal Transport and Information Geometry
    Published in: Journal of Geometric Mechanics doi:10.3934/jgm.2017014 Volume 9, Number 3, September 2017 pp. 335{390 GEOMETRY OF MATRIX DECOMPOSITIONS SEEN THROUGH OPTIMAL TRANSPORT AND INFORMATION GEOMETRY Klas Modin∗ Department of Mathematical Sciences Chalmers University of Technology and University of Gothenburg SE-412 96 Gothenburg, Sweden Abstract. The space of probability densities is an infinite-dimensional Rie- mannian manifold, with Riemannian metrics in two flavors: Wasserstein and Fisher{Rao. The former is pivotal in optimal mass transport (OMT), whereas the latter occurs in information geometry|the differential geometric approach to statistics. The Riemannian structures restrict to the submanifold of multi- variate Gaussian distributions, where they induce Riemannian metrics on the space of covariance matrices. Here we give a systematic description of classical matrix decompositions (or factorizations) in terms of Riemannian geometry and compatible principal bundle structures. Both Wasserstein and Fisher{Rao geometries are discussed. The link to matrices is obtained by considering OMT and information ge- ometry in the category of linear transformations and multivariate Gaussian distributions. This way, OMT is directly related to the polar decomposition of matrices, whereas information geometry is directly related to the QR, Cholesky, spectral, and singular value decompositions. We also give a coherent descrip- tion of gradient flow equations for the various decompositions; most flows are illustrated in numerical examples. The paper is a combination of previously known and original results. As a survey it covers the Riemannian geometry of OMT and polar decomposi- tions (smooth and linear category), entropy gradient flows, and the Fisher{Rao metric and its geodesics on the statistical manifold of multivariate Gaussian distributions.
    [Show full text]
  • Chapter 6 the Singular Value Decomposition Ax=B Version of 11 April 2019
    Matrix Methods for Computational Modeling and Data Analytics Virginia Tech Spring 2019 · Mark Embree [email protected] Chapter 6 The Singular Value Decomposition Ax=b version of 11 April 2019 The singular value decomposition (SVD) is among the most important and widely applicable matrix factorizations. It provides a natural way to untangle a matrix into its four fundamental subspaces, and reveals the relative importance of each direction within those subspaces. Thus the singular value decomposition is a vital tool for analyzing data, and it provides a slick way to understand (and prove) many fundamental results in matrix theory. It is the perfect tool for solving least squares problems, and provides the best way to approximate a matrix with one of lower rank. These notes construct the SVD in various forms, then describe a few of its most compelling applications. 6.1 Eigenvalues and eigenvectors of symmetric matrices To derive the singular value decomposition of a general (rectangu- lar) matrix A IR m n, we shall rely on several special properties of 2 ⇥ the square, symmetric matrix ATA. While this course assumes you are well acquainted with eigenvalues and eigenvectors, we will re- call some fundamental concepts, especially pertaining to symmetric matrices. 6.1.1 A passing nod to complex numbers Recall that even if a matrix has real number entries, it could have eigenvalues that are complex numbers; the corresponding eigenvec- tors will also have complex entries. Consider, for example, the matrix 0 1 S = − . " 10# 73 To find the eigenvalues of S, form the characteristic polynomial l 1 det(lI S)=det = l2 + 1.
    [Show full text]
  • Computing the Polar Decomposition—With Applications
    Computing the Polar Decomposition—with Applications Higham, Nicholas J. 1986 MIMS EPrint: 2007.9 Manchester Institute for Mathematical Sciences School of Mathematics The University of Manchester Reports available from: http://eprints.maths.manchester.ac.uk/ And by contacting: The MIMS Secretary School of Mathematics The University of Manchester Manchester, M13 9PL, UK ISSN 1749-9097 SIAM J. Sci. STAT. COMPUT. (C) 1986 Society for Industrial and Applied Mathematics Vol. 7, No. 4, October 1986 OO7 COMPUTING THE POLAR DECOMPOSITION---WITH APPLICATIONS* NICHOLAS J. HIGHAMf Abstract. A quadratically convergent Newton method for computing the polar decomposition of a full-rank matrix is presented and analysed. Acceleration parameters are introduced so as to enhance the initial rate of convergence and it is shown how reliable estimates of the optimal parameters may be computed in practice. To add to the known best approximation property of the unitary polar factor, the Hermitian polar factor H of a nonsingular Hermitian matrix A is shown to be a good positive definite approximation to A and 1/2(A / H) is shown to be a best Hermitian positive semi-definite approximation to A. Perturbation bounds for the polar factors are derived. Applications of the polar decomposition to factor analysis, aerospace computations and optimisation are outlined; and a new method is derived for computing the square root of a symmetric positive definite matrix. Key words, polar decomposition, singular value decomposition, Newton's method, matrix square root AMS(MOS) subject classifications. 65F25, 65F30, 65F35 1. Introduction. The polar decomposition is a generalisation to matrices of the familiar complex number representation z r e i, r ->_ 0.
    [Show full text]
  • Lecture 15: July 22, 2013 1 Expander Mixing Lemma 2 Singular Value
    REU 2013: Apprentice Program Summer 2013 Lecture 15: July 22, 2013 Madhur Tulsiani Scribe: David Kim 1 Expander Mixing Lemma Let G = (V; E) be d-regular, such that its adjacency matrix A has eigenvalues µ1 = d ≥ µ2 ≥ · · · ≥ µn. Let S; T ⊆ V , not necessarily disjoint, and E(S; T ) = fs 2 S; t 2 T : fs; tg 2 Eg. Question: How close is jE(S; T )j to what is \expected"? Exercise 1.1 Given G as above, if one picks a random pair i; j 2 V (G), what is P[i; j are adjacent]? dn #good cases #edges 2 d Answer: P[fi; jg 2 E] = = n = n = #all cases 2 2 n − 1 Since for S; T ⊆ V , the number of possible pairs of vertices from S and T is jSj · jT j, we \expect" d jSj · jT j · n−1 out of those pairs to have edges. d p Exercise 1.2 * jjE(S; T )j − jSj · jT j · n j ≤ µ jSj · jT j Note that the smaller the value of µ, the closer jE(S; T )j is to what is \expected", hence expander graphs are sometimes called pseudorandom graphs for µ small. We will now show a generalization of the Spectral Theorem, with a generalized notion of eigenvalues to matrices that are not necessarily square. 2 Singular Value Decomposition n×n ∗ ∗ Recall that by the Spectral Theorem, given a normal matrix A 2 C (A A = AA ), we have n ∗ X ∗ A = UDU = λiuiui i=1 where U is unitary, D is diagonal, and we have a decomposition of A into a nice orthonormal basis, m×n consisting of the columns of U.
    [Show full text]
  • Singular Value Decomposition
    Math 312, Spring 2014 Jerry L. Kazdan Singular Value Decomposition Positive Definite Matrices Let C be an n × n positive definite (symmetric) matrix and consider the quadratic polynomial hx;Cxi. How can we understand what this\looks like"? One useful approach is to view the image 2 2 of the unit sphere, that is, the points that satisfy kxk = 1. For instance, if hx;Cxi = 4x1 + 9x2 , 2 2 then the image of the unit disk, x1 + x2 = 1, is an ellipse. For the more general case of hx;Cxi it is fundamental to use coordinates that are adapted to the matrix C . Since C is a symmetric matrix, there are orthonormal vectors v1 , v2 ,. , vn in which C is a diagonal matrix. These vectors vj are eigenvectors of C with corresponding eigenvalues λj : so Cvj = λjvj . In these coordinates, say x = y1v1 + y2v2 + ··· + ynvn: (1) and Cx = λ1y1v1 + λ2y2v2 + ··· + λnynvn: (2) Then, using the orthonormality of the vj , 2 2 2 2 kxk = hx; xi = y1 + y2 + ··· + yn and 2 2 2 hx;Cxi = λ1y1 + λ2y2 + ··· + λnyn: (3) For use in the singular value decomposition below, it is con- ventional to number the eigenvalues in decreasing order: λmax = λ1 ≥ λ2 ≥ · · · ≥ λn = λmin; so on the unit sphere kxk = 1 λmin ≤ hx;Cxi ≤ λmax: As in the figure, the image of the unit sphere is an \ellipsoid" whose axes are in the direction of the eigenvectors and the corresponding eigenvalues give half the length of the axes. There are two closely related approaches to using that a self-adjoint matrix C can be orthogonally diagonalized.
    [Show full text]