DOCSLIB.ORG

Lecture 8: Fast Linear Solvers (Part 6)

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Lecture 8: Fast Linear Solvers (Part 6) Lecture 8: Fast Linear Solvers (Part 6) 1 Nonsymmetric System of Linear Equations • The CG method requires to 퐴 to be an 푛 × 푛 symmetric positive definite matrix to solve 퐴풙 = 풃. • If 퐴 is nonsymmetric: – Convert the system to a symmetric positive definite one – Modify CG to handle general matrices 2 Normal Equation Approach The normal equations corresponding to 퐴풙 = 풃 are 퐴푇퐴풙 = 퐴푇풃 • If 퐴 is nonsingular then 퐴푇퐴 is symmetric positive definite and the CG method can be applied to solve 퐴푇퐴풙 = 퐴푇풃 (CG normal residual -- CGNR). • Alternatively, we can first solve A퐴푇풚 = 풃 for 풚, then 풙 = 퐴푇풚. • Disadvantages: – Each iteration requires 퐴푇퐴 or A퐴푇 – Condition number of 퐴푇퐴 or A퐴푇 is square of that of 퐴. However, CG works well if condition number is small 3 Arnoldi Iteration • The Arnoldi method is an orthogonal projection onto a Krylov subspace Κ푚(퐴, 풓0) for 푛 × 푛 nonsymmetric matrix 퐴. Here 푚 ≪ 푛. • Arnoldi reduces 퐴 to a Hessenberg form. Upper Hessenberg matrix: zero entries below the first subdiagonal. 2 3 4 1 2 5 1 9 0 2 1 2 0 0 3 2 Lower Hessenberg matrix: zero entries above the first superdiagonal. 3 2 0 0 2 5 1 0 1 2 1 2 3 4 3 2 4 Mechanics of Arnoldi Iteration 푛×푛 푛 • For 퐴 ∈ 푅 , a given vector 풓0 ∈ 푅 defines a sequence of Krylov subspaces Κ푚(퐴, 풓0). Matrix 2 푚−1 푛×푚 Κ푚 = [풓0|퐴풓0|퐴 풓0| … |퐴 풓0] ∈ 푅 is the corresponding Krylov matrix. • The Gram-Schmidt procedure for forming an orthonormal basis for Κ푚 is called the Arnoldi process. – Theorem. The Arnoldi procedure generates a reduced QR factorization of Krylov matrix Κ푚 in the form Κ푚 = 푉푚푅푚 with 푛×푚 푉푚 ∈ 푅 and having orthonormal columns and with a 푚×푚 triangular matrix 푅푚 ∈ 푅 . Furthermore, with the 푇 푚 × 푚 − 푢푝푝푒푟 Hessenberg matrix 퐻푚, we have 푉푚 퐴푉푚 = 퐻푚. 5 Let 퐻푚 be a 푚 × 푚 Hessenberg matrix: ℎ11 ℎ12 … ℎ1푚 ℎ ℎ … ℎ 퐻 = 21 22 2푚 푚 0 ⋱ ⋱ ⋮ 0 … ℎ푚,푚−1 ℎ푚푚 Let (푚 + 1) × 푚 퐻 푚 be the extended matrix of 퐻푚: ℎ11 ℎ12 … ℎ1푚 ℎ21 ℎ22 … ℎ2푚 퐻 푚 = 0 ⋱ ⋱ ⋮ 0 … ℎ푚,푚−1 ℎ푚푚 0 … 0 ℎ푚+1,푚 The Arnoldi iteration produces matrices 푉푚, 푉푚+1 and 퐻 푚 for matrix 퐴 satisfying: 푇 퐴푉푚 = 푉푚+1퐻 푚 = 푉푚퐻푚 + 풘푚풆푚 Here 푉푚, 푉푚+1 have orthonormal columns 푉푚 = [풗1 풗2 … |풗푚], 푉푚+1 = [풗1 풗2 … |풗푚|풗푚+1] 6 The 푚th column of the equation: 퐴풗푚 = ℎ1푚풗1 + ℎ2푚풗2 + ⋯ + ℎ푚푚풗푚 + ℎ푚+1,푚풗푚+1 Therefore, ℎ1푚 = 퐴풗푚 ∙ 풗1 ⋮ ℎ푚+1,푚 = ||퐴풗푚 − ℎ1푚풗1 … − ℎ푚푚풗푚|| 풗푚+1 = (퐴풗푚 − ℎ1푚풗1 … − ℎ푚푚풗푚)/ℎ푚+1,푚 7 Arnoldi Algorithm Choose 풓0 and let 풗1 = 풓0/| 풓0 | for 푗 = 1, … , 푚 − 1 푗 푇 풘 = 퐴풗푗 − 푖=1((퐴풗푗) 풗푖)풗푖 풗푗+1 = 풘/||풘||2 endfor Remark: Choose 풗1. Then for 푗 = 1, … , 푚 − 1, first multiply the current Arnoldi vector 풗푗 by A, and orthonormalize 퐴풗푗 against all previous Arnoldi vectors. 8 푇 • 푉푚 푉푚 = 퐼푚×푚. • If Arnoldi process breaks down at 푚푡ℎ step, 풘푚 = ퟎ is still well- defined but not 풗푚+1, and the algorithm stop. • In this case, the last row of 퐻푚 is set to zero, ℎ푚+1,푚 = 0 9 Stable Arnoldi Algorithm Choose 풙0 and let 풗1 = 풙0/| 풙0 | . for 푗 = 1, … , 푚 풘 = 퐴풗푗 for 푖 = 1, … , 푗 ℎ푖푗 =< 풘, 풗푖 > 풘 = 풘 − ℎ푖푗풗푖 endfor ℎ푗+1,푗 = ||풘||2 if ℎ푗+1,푗 = 0, then stop 풗푗+1 = 풘/ℎ푗+1,푗 endfor 10 Generalized Minimum Residual (GMRES) Method Let the Krylov space associated with 퐴풙 = 풃 be Κ푘 퐴, 풓0 = 2 푘−1 푠푝푎푛 풓0, 퐴풓0, 퐴 풓0, … , 퐴 풓0 , where 풓0 = 풃 − 퐴 풙0 for some initial guess 풙0. The 푘푡ℎ (푘 ≥ 1) iteration of GMRES is the solution to the least squares problem: 푚푖푛푖푚푖푧푒풙∈풙0+Κ푘||풃 − 퐴풙||2, i.e. Find 풙푘 ∈ 풙0 + Κ푘 such that ||풃 − 퐴풙푘||2 = 푚푖푛풙∈풙0+Κ푘||풃 − 퐴풙||2 • Remark: the GMRES was proposed in “Y. Saad and M. Schultz, GMRES a generalized minimal residual algorithm for solving nonsymmetric linear systems, SIAM J. Sci. Statist. Comput., 7 (1986), pp. 856–869.” 11 푘−1 푗 If 풙 ∈ 풙0 + Κ푘, then 풙 = 풙0 + 푗=0 훾푗퐴 풓0. 푘−1 푗+1 So 풃 − 퐴풙 = 풃 − 퐴풙0 − 푗=0 훾푗퐴 풓0 = 풓0 − 푘 푗 푗=1 훾푗−1퐴 풓0 . 12 • Theorem (Kelly). Let A be a nonsingular diagonalizable matrix. Assume that A has only k distinct eigenvalues. Then GMRES will terminate in at most k iterations. • Least Square via QR factorization Let 퐴 ∈ 푅푚×푛 (푚 ≥ 푛), and 풃 ∈ 푅푚 be given. Find 풙 ∈ 푅푛 so that the norm of 풓 = 풃 − 퐴풙 is minimized. Algorithm 1. Compute the QR factorization 퐴 = 푄 푅 2. Compute vector 푄 ∗풃 3. Solve the upper triangular system 푅 풙 = 푄 ∗풃 for 풙 Reference: Numerical Linear Algebra, L.N. Trefethen, D. Bau, III 13 GMRES Implementation • The 푘푡ℎ (푘 ≥ 1) iteration of GMRES is the solution to the least squares problem: 푚푖푛푖푚푖푧푒풙∈풙0+Κ푘||풃 − 퐴풙||2 • Suppose we have used Arnoldi process constructed an orthogonal basis 푉푘 for Κ푘 퐴, 풓0 . 푇 – 풓0 = 훽푉푘풆1, where 풆1 = (1,0,0, … ) , 훽 = ||풓0||2 – Any vector 풛 ∈ Κ푘 퐴, 풓0 can be written as 풛 = 푘 푘 푘 푙=1 푦푙풗푙 , where 풗푙 is the 푙푡ℎ column of 푉푘. Denote 푇 푘 풚 = (푦1, 푦2, … , 푦푘) ∈ 푅 . 풛 = 푉푘풚 14 Since 풙 − 풙0 = 푉푘풚 for some coefficient vector 풚 ∈ 푘 푅 , we must have 풙푘 = 풙0 + 푉푘풚 where 풚 minimizes ||풃 − 퐴(풙0 + 푉푘풚)||2 = ||풓0 − 퐴푉푘풚||2. • The 푘푡ℎ (푘 ≥ 1) iteration of GMRES now is equivalent to a least squares problem in 푅푘, i.e. 푚푖푛푖푚푖푧푒풙∈풙0+Κ푘||풃 − 퐴풙||2 = 푚푖푛푖푚푖푧푒푦∈푅푘||풓0 − 퐴푉푘풚||2 – Remark: This is a linear least square problem, which can be solved by QR factorization. However, 퐴푉푘 must be computed at each iteration. 푇 푇 – The associate normal equation is (퐴푉푘) 퐴푉푘풚 = (퐴푉푘) 풓0. – But we will solve it differently. 15 • Let 풙푘 be 푘푡ℎ iterative solution of GMRES. Define: 풓푘 = 풃 − 퐴풙푘 = 풓0 − 퐴 풙푘 − 풙0 = 훽푉푘+1풆1 − 퐴 풙0 + 푉푘풚 − 풙0 = 훽푉푘+1풆1 − 푘 푘 푉푘+1퐻 푘풚 = 푉푘+1(훽풆1 − 퐻 푘풚 ) Using orthonomality of 푉푘+1: 푚푖푛푖푚푖푧푒풙∈풙0+Κ푘||풃 − 퐴풙||2 푘 = 푚푖푛푖푚푖푧푒푦∈푅푘||훽풆1 − 퐻푘풚 ||2 16 “C.T. Kelley, Iterative Methods for Linear and Nonlinear Equations” . 17 푘 푚푖푛푖푚푖푧푒푦∈푅푘||훽풆1 − 퐻푘풚 ||2 Theorem. Let 푛 × 푘 (푘 ≤ 푛) matrix 퐵 be with linearly independent columns (full column rank). Let 퐵 = 푄푅 be a 푄푅 factorization of 퐵. Then for each 풃 ∈ 푅푛, the equation 퐵풖 = 풃 has a unique least-square solution, given by 풖 = 푅−1푄푇풃. Using Householder reflection to do QR factorization (푘+1)×(푘+1) gives 퐻 푘 = 푄푘+1푅 푘 where 푄푘+1 ∈ 푅 is (푘+1)×푘 orthogonal and 푅 푘 ∈ 푅 has the form 푅 푅 = 푘 , where 푅 ∈ 푅푘×푘 is upper triangular. 푘 0 푘 18 • 풗푗 may become nonorthogonal as a result of round off errors. 푘 – ||훽풆1 − 퐻 푘풚 ||2 which depends on orthogonality, will not hold and the residual could be inaccurate. – Replace the loop in Step 2c of Algorithm gmresa with 19 “C.T. Kelley, Iterative Methods for Linear and Nonlinear Equations” . 20 “C.T. Kelley, Iterative Methods for Linear and Nonlinear Equations” . 21 Modified Gram-Schmidt Process with Reorthogonalization Test Reorthogonalization If 퐴푣푘||2 + 훿 푣푘+1||2 = ||퐴푣푘||2 to working precision. −3 훿 = 10 22 Givens Rotations 푘 푚푖푛푖푚푖푧푒푦∈푅푘||훽풆1 − 퐻푘풚 ||2 involves QR factorization. Do QR factorizations of 퐻 푘by Givens Rotations. • A 2 × 2 Givens rotation is a matrix of the form 푐 −푠 퐺 = where 푐 = cos (휃), 푠 = sin (휃) for 푠 푐 휃 ∈ [−휋, 휋]. The orthogonal matrix 퐺 rotates the vector (푐, −푠)푇, which makes an angle of −휃 with the 푥-axis, through an angle 휃 so that it overlaps the 푥-axis. 푐 1 퐺 = −푠 0 23 An 푁 × 푁 Givens rotation 퐺푗(푐, 푠) replaces a 2 × 2 block on the diagonal of the 푁 × 푁 identity matrix with a 2 × 2 Givens rotations. 퐺푗(푐, 푠) is with a 2 × 2 Givens rotations in rows and columns 푗 and 푗 + 1. 24 • Givens rotations can be used in reducing Hessenberg matrices to triangular form. This can be done in 푂 푁 floating-point operations. • Let 퐻 be an 푁 × 푀(푁 ≥ 푀) upper Hessenberg matrix with rank 푀. We reduce 퐻 to triangular form by first multiplying the matrix by a Givens rotations that zeros ℎ21 (values of ℎ11 and subsequent columns are changed) 25 2 2 • Step 1: Define 퐺1(푐1, 푠1) by 푐1 = ℎ11/ ℎ11 + ℎ21 and 2 2 푠1 = −ℎ21/ ℎ11 + ℎ21. Replace 퐻 by 퐺1퐻. 2 2 • Step 2: Define 퐺2(푐2, 푠2) by 푐2 = ℎ22/ ℎ22 + ℎ32 and 2 2 푠2 = −ℎ32/ ℎ22 + ℎ32. Replace 퐻 by 퐺2퐻. Remark: 퐺2 does not affect the first column of 퐻. • … 2 2 • Step j: Define 퐺푗(푐푗, 푠푗) by 푐푗 = ℎ푗푗/ ℎ푗푗 + ℎ푗+1,푗 and 2 2 푠푗 = −ℎ푗+1,푗/ ℎ푗푗 + ℎ푗+1,푗. Replace 퐻 by 퐺푗퐻. Setting 푄 = 퐺푁 … 퐺1. 푅 = 푄퐻 is upper triangular. 26 Let 퐻 푘 = 푄푅 by Givens rotations matrices. 푘 푚푖푛푖푚푖푧푒푦∈푅푘||훽풆1 − 퐻푘풚 ||2 푘 = 푚푖푛푖푚푖푧푒푦∈푅푘||푄(훽풆1 − 퐻푘풚 )||2 푘 = 푚푖푛푖푚푖푧푒푦∈푅푘||훽푄풆1 − 푅풚 ||2 27 28 Preconditioning Basic idea: using GMRES on a modified system such as 푀−1퐴풙 = 푀−1풃.
Load more

Recommended publications
  • A Distributed and Parallel Asynchronous Unite and Conquer Method to Solve Large Scale Non-Hermitian Linear Systems Xinzhe Wu, Serge Petiton
    A Distributed and Parallel Asynchronous Unite and Conquer Method to Solve Large Scale Non-Hermitian Linear Systems Xinzhe Wu, Serge Petiton To cite this version: Xinzhe Wu, Serge Petiton. A Distributed and Parallel Asynchronous Unite and Conquer Method to Solve Large Scale Non-Hermitian Linear Systems. HPC Asia 2018 - International Conference on High Performance Computing in Asia-Pacific Region, Jan 2018, Tokyo, Japan. 10.1145/3149457.3154481. hal-01677110 HAL Id: hal-01677110 https://hal.archives-ouvertes.fr/hal-01677110 Submitted on 8 Jan 2018 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. A Distributed and Parallel Asynchronous Unite and Conquer Method to Solve Large Scale Non-Hermitian Linear Systems Xinzhe WU Serge G. Petiton Maison de la Simulation, USR 3441 Maison de la Simulation, USR 3441 CRIStAL, UMR CNRS 9189 CRIStAL, UMR CNRS 9189 Université de Lille 1, Sciences et Technologies Université de Lille 1, Sciences et Technologies F-91191, Gif-sur-Yvette, France F-91191, Gif-sur-Yvette, France [email protected] [email protected] ABSTRACT the iteration number. These methods are already well implemented Parallel Krylov Subspace Methods are commonly used for solv- in parallel to profit from the great number of computation cores ing large-scale sparse linear systems.
    [Show full text]
  • Implicitly Restarted Arnoldi/Lanczos Methods for Large Scale Eigenvalue Calculations
    NASA Contractor Report 198342 /" ICASE Report No. 96-40 J ICA IMPLICITLY RESTARTED ARNOLDI/LANCZOS METHODS FOR LARGE SCALE EIGENVALUE CALCULATIONS Danny C. Sorensen NASA Contract No. NASI-19480 May 1996 Institute for Computer Applications in Science and Engineering NASA Langley Research Center Hampton, VA 23681-0001 Operated by Universities Space Research Association National Aeronautics and Space Administration Langley Research Center Hampton, Virginia 23681-0001 IMPLICITLY RESTARTED ARNOLDI/LANCZOS METHODS FOR LARGE SCALE EIGENVALUE CALCULATIONS Danny C. Sorensen 1 Department of Computational and Applied Mathematics Rice University Houston, TX 77251 sorensen@rice, edu ABSTRACT Eigenvalues and eigenfunctions of linear operators are important to many areas of ap- plied mathematics. The ability to approximate these quantities numerically is becoming increasingly important in a wide variety of applications. This increasing demand has fu- eled interest in the development of new methods and software for the numerical solution of large-scale algebraic eigenvalue problems. In turn, the existence of these new methods and software, along with the dramatically increased computational capabilities now avail- able, has enabled the solution of problems that would not even have been posed five or ten years ago. Until very recently, software for large-scale nonsymmetric problems was virtually non-existent. Fortunately, the situation is improving rapidly. The purpose of this article is to provide an overview of the numerical solution of large- scale algebraic eigenvalue problems. The focus will be on a class of methods called Krylov subspace projection methods. The well-known Lanczos method is the premier member of this class. The Arnoldi method generalizes the Lanczos method to the nonsymmetric case.
    [Show full text]
  • Krylov Subspaces
    Lab 1 Krylov Subspaces Lab Objective: Discuss simple Krylov Subspace Methods for finding eigenvalues and show some interesting applications. One of the biggest difficulties in computational linear algebra is the amount of memory needed to store a large matrix and the amount of time needed to read its entries. Methods using Krylov subspaces avoid this difficulty by studying how a matrix acts on vectors, making it unnecessary in many cases to create the matrix itself. More specifically, we can construct a Krylov subspace just by knowing how a linear transformation acts on vectors, and with these subspaces we can closely approximate eigenvalues of the transformation and solutions to associated linear systems. The Arnoldi iteration is an algorithm for finding an orthonormal basis of a Krylov subspace. Its outputs can also be used to approximate the eigenvalues of the original matrix. The Arnoldi Iteration The order-N Krylov subspace of A generated by x is 2 n−1 Kn(A; x) = spanfx;Ax;A x;:::;A xg: If the vectors fx;Ax;A2x;:::;An−1xg are linearly independent, then they form a basis for Kn(A; x). However, this basis is usually far from orthogonal, and hence computations using this basis will likely be ill-conditioned. One way to find an orthonormal basis for Kn(A; x) would be to use the modi- fied Gram-Schmidt algorithm from Lab TODO on the set fx;Ax;A2x;:::;An−1xg. More efficiently, the Arnold iteration integrates the creation of fx;Ax;A2x;:::;An−1xg with the modified Gram-Schmidt algorithm, returning an orthonormal basis for Kn(A; x).
    [Show full text]
  • Accelerating the LOBPCG Method on Gpus Using a Blocked Sparse Matrix Vector Product
    Accelerating the LOBPCG method on GPUs using a blocked Sparse Matrix Vector Product Hartwig Anzt and Stanimire Tomov and Jack Dongarra Innovative Computing Lab University of Tennessee Knoxville, USA Email: [email protected], [email protected], [email protected] Abstract— the computing power of today’s supercomputers, often accel- erated by coprocessors like graphics processing units (GPUs), This paper presents a heterogeneous CPU-GPU algorithm design and optimized implementation for an entire sparse iter- becomes challenging. ative eigensolver – the Locally Optimal Block Preconditioned Conjugate Gradient (LOBPCG) – starting from low-level GPU While there exist numerous efforts to adapt iterative lin- data structures and kernels to the higher-level algorithmic choices ear solvers like Krylov subspace methods to coprocessor and overall heterogeneous design. Most notably, the eigensolver technology, sparse eigensolvers have so far remained out- leverages the high-performance of a new GPU kernel developed side the main focus. A possible explanation is that many for the simultaneous multiplication of a sparse matrix and a of those combine sparse and dense linear algebra routines, set of vectors (SpMM). This is a building block that serves which makes porting them to accelerators more difficult. Aside as a backbone for not only block-Krylov, but also for other from the power method, algorithms based on the Krylov methods relying on blocking for acceleration in general. The subspace idea are among the most commonly used general heterogeneous LOBPCG developed here reveals the potential of eigensolvers [1]. When targeting symmetric positive definite this type of eigensolver by highly optimizing all of its components, eigenvalue problems, the recently developed Locally Optimal and can be viewed as a benchmark for other SpMM-dependent applications.
    [Show full text]
  • Iterative Methods for Linear System
    Iterative methods for Linear System JASS 2009 Student: Rishi Patil Advisor: Prof. Thomas Huckle Outline ●Basics: Matrices and their properties Eigenvalues, Condition Number ●Iterative Methods Direct and Indirect Methods ●Krylov Subspace Methods Ritz Galerkin: CG Minimum Residual Approach : GMRES/MINRES Petrov-Gaelerkin Method: BiCG, QMR, CGS 1st April JASS 2009 2 Basics ●Linear system of equations Ax = b ●A Hermitian matrix (or self-adjoint matrix ) is a square matrix with complex entries which is equal to its own conjugate transpose, 3 2 + i that is, the element in the ith row and jth column is equal to the A = complex conjugate of the element in the jth row and ith column, for 2 − i 1 all indices i and j • Symmetric if a ij = a ji • Positive definite if, for every nonzero vector x XTAx > 0 • Quadratic form: 1 f( x) === xTAx −−− bTx +++ c 2 ∂ f( x) • Gradient of Quadratic form: ∂x 1 1 1 f' (x) = ⋮ = ATx + Ax − b ∂ 2 2 f( x) ∂xn 1st April JASS 2009 3 Various quadratic forms Positive-definite Negative-definite matrix matrix Singular Indefinite positive-indefinite matrix matrix 1st April JASS 2009 4 Various quadratic forms 1st April JASS 2009 5 Eigenvalues and Eigenvectors For any n×n matrix A, a scalar λ and a nonzero vector v that satisfy the equation Av =λv are said to be the eigenvalue and the eigenvector of A. ●If the matrix is symmetric , then the following properties hold: (a) the eigenvalues of A are real (b) eigenvectors associated with distinct eigenvalues are orthogonal ●The matrix A is positive definite (or positive semidefinite) if and only if all eigenvalues of A are positive (or nonnegative).
    [Show full text]
  • Implicitly Restarted Arnoldi/Lanczos Methods for Large Scale Eigenvalue Calculations
    https://ntrs.nasa.gov/search.jsp?R=19960048075 2020-06-16T03:31:45+00:00Z NASA Contractor Report 198342 /" ICASE Report No. 96-40 J ICA IMPLICITLY RESTARTED ARNOLDI/LANCZOS METHODS FOR LARGE SCALE EIGENVALUE CALCULATIONS Danny C. Sorensen NASA Contract No. NASI-19480 May 1996 Institute for Computer Applications in Science and Engineering NASA Langley Research Center Hampton, VA 23681-0001 Operated by Universities Space Research Association National Aeronautics and Space Administration Langley Research Center Hampton, Virginia 23681-0001 IMPLICITLY RESTARTED ARNOLDI/LANCZOS METHODS FOR LARGE SCALE EIGENVALUE CALCULATIONS Danny C. Sorensen 1 Department of Computational and Applied Mathematics Rice University Houston, TX 77251 sorensen@rice, edu ABSTRACT Eigenvalues and eigenfunctions of linear operators are important to many areas of ap- plied mathematics. The ability to approximate these quantities numerically is becoming increasingly important in a wide variety of applications. This increasing demand has fu- eled interest in the development of new methods and software for the numerical solution of large-scale algebraic eigenvalue problems. In turn, the existence of these new methods and software, along with the dramatically increased computational capabilities now avail- able, has enabled the solution of problems that would not even have been posed five or ten years ago. Until very recently, software for large-scale nonsymmetric problems was virtually non-existent. Fortunately, the situation is improving rapidly. The purpose of this article is to provide an overview of the numerical solution of large- scale algebraic eigenvalue problems. The focus will be on a class of methods called Krylov subspace projection methods. The well-known Lanczos method is the premier member of this class.
    [Show full text]
  • Limited Memory Block Krylov Subspace Optimization for Computing Dominant Singular Value Decompositions
    Limited Memory Block Krylov Subspace Optimization for Computing Dominant Singular Value Decompositions Xin Liu∗ Zaiwen Weny Yin Zhangz March 22, 2012 Abstract In many data-intensive applications, the use of principal component analysis (PCA) and other related techniques is ubiquitous for dimension reduction, data mining or other transformational purposes. Such transformations often require efficiently, reliably and accurately computing dominant singular value de- compositions (SVDs) of large unstructured matrices. In this paper, we propose and study a subspace optimization technique to significantly accelerate the classic simultaneous iteration method. We analyze the convergence of the proposed algorithm, and numerically compare it with several state-of-the-art SVD solvers under the MATLAB environment. Extensive computational results show that on a wide range of large unstructured matrices, the proposed algorithm can often provide improved efficiency or robustness over existing algorithms. Keywords. subspace optimization, dominant singular value decomposition, Krylov subspace, eigen- value decomposition 1 Introduction Singular value decomposition (SVD) is a fundamental and enormously useful tool in matrix computations, such as determining the pseudo-inverse, the range or null space, or the rank of a matrix, solving regular or total least squares data fitting problems, or computing low-rank approximations to a matrix, just to mention a few. The need for computing SVDs also frequently arises from diverse fields in statistics, signal processing, data mining or compression, and from various dimension-reduction models of large-scale dynamic systems. Usually, instead of acquiring all the singular values and vectors of a matrix, it suffices to compute a set of dominant (i.e., the largest) singular values and their corresponding singular vectors in order to obtain the most valuable and relevant information about the underlying dataset or system.
    [Show full text]
  • The Influence of Orthogonality on the Arnoldi Method
    View metadata, citation and similar papers at core.ac.uk brought to you by CORE provided by Elsevier - Publisher Connector Linear Algebra and its Applications 309 (2000) 307±323 www.elsevier.com/locate/laa The in¯uence of orthogonality on the Arnoldi method T. Braconnier *, P. Langlois, J.C. Rioual IREMIA, Universite de la Reunion, DeÂpartement de Mathematiques et Informatique, BP 7151, 15 Avenue Rene Cassin, 97715 Saint-Denis Messag Cedex 9, La Reunion, France Received 1 February 1999; accepted 22 May 1999 Submitted by J.L. Barlow Abstract Many algorithms for solving eigenproblems need to compute an orthonormal basis. The computation is commonly performed using a QR factorization computed using the classical or the modi®ed Gram±Schmidt algorithm, the Householder algorithm, the Givens algorithm or the Gram±Schmidt algorithm with iterative reorthogonalization. For the eigenproblem, although textbooks warn users about the possible instability of eigensolvers due to loss of orthonormality, few theoretical results exist. In this paper we prove that the loss of orthonormality of the computed basis can aect the reliability of the computed eigenpair when we use the Arnoldi method. We also show that the stopping criterion based on the backward error and the value computed using the Arnoldi method can dier because of the loss of orthonormality of the computed basis of the Krylov subspace. We also give a bound which quanti®es this dierence in terms of the loss of orthonormality. Ó 2000 Elsevier Science Inc. All rights reserved. Keywords: QR factorization; Backward error; Eigenvalues; Arnoldi method 1. Introduction The aim of the most commonly used eigensolvers is to approximate a basis of an invariant subspace associated with some eigenvalues.
    [Show full text]
  • Power Method and Krylov Subspaces
    Power method and Krylov subspaces Introduction When calculating an eigenvalue of a matrix A using the power method, one starts with an initial random vector b and then computes iteratively the sequence Ab, A2b,...,An−1b normalising and storing the result in b on each iteration. The sequence converges to the eigenvector of the largest eigenvalue of A. The set of vectors 2 n−1 Kn = b, Ab, A b,...,A b , (1) where n < rank(A), is called the order-n Krylov matrix, and the subspace spanned by these vectors is called the order-n Krylov subspace. The vectors are not orthogonal but can be made so e.g. by Gram-Schmidt orthogonalisation. For the same reason that An−1b approximates the dominant eigenvector one can expect that the other orthogonalised vectors approximate the eigenvectors of the n largest eigenvalues. Krylov subspaces are the basis of several successful iterative methods in numerical linear algebra, in particular: Arnoldi and Lanczos methods for finding one (or a few) eigenvalues of a matrix; and GMRES (Generalised Minimum RESidual) method for solving systems of linear equations. These methods are particularly suitable for large sparse matrices as they avoid matrix-matrix opera- tions but rather multiply vectors by matrices and work with the resulting vectors and matrices in Krylov subspaces of modest sizes. Arnoldi iteration Arnoldi iteration is an algorithm where the order-n orthogonalised Krylov matrix Qn of a matrix A is built using stabilised Gram-Schmidt process: • start with a set Q = {q1} of one random normalised vector q1 • repeat for k =2 to n : – make a new vector qk = Aqk−1 † – orthogonalise qk to all vectors qi ∈ Q storing qi qk → hi,k−1 – normalise qk storing qk → hk,k−1 – add qk to the set Q By construction the matrix Hn made of the elements hjk is an upper Hessenberg matrix, h1,1 h1,2 h1,3 ··· h1,n h2,1 h2,2 h2,3 ··· h2,n 0 h3,2 h3,3 ··· h3,n Hn = , (2) .
    [Show full text]
  • Section 5.3: Other Krylov Subspace Methods
    Section 5.3: Other Krylov Subspace Methods Jim Lambers March 15, 2021 • So far, we have only seen Krylov subspace methods for symmetric positive definite matrices. • We now generalize these methods to arbitrary square invertible matrices. Minimum Residual Methods • In the derivation of the Conjugate Gradient Method, we made use of the Krylov subspace 2 k−1 K(r1; A; k) = spanfr1;Ar1;A r1;:::;A r1g to obtain a solution of Ax = b, where A was a symmetric positive definite matrix and (0) (0) r1 = b − Ax was the residual associated with the initial guess x . • Specifically, the Conjugate Gradient Method generated a sequence of approximate solutions x(k), k = 1; 2;:::; where each iterate had the form k (k) (0) (0) X x = x + Qkyk = x + qjyjk; k = 1; 2;:::; (1) j=1 and the columns q1; q2;:::; qk of Qk formed an orthonormal basis of the Krylov subspace K(b; A; k). • In this section, we examine whether a similar approach can be used for solving Ax = b even when the matrix A is not symmetric positive definite. (k) • That is, we seek a solution x of the form of equation (??), where again the columns of Qk form an orthonormal basis of the Krylov subspace K(b; A; k). • To generate this basis, we cannot simply generate a sequence of orthogonal polynomials using a three-term recurrence relation, as before, because the bilinear form T hf; gi = r1 f(A)g(A)r1 does not satisfy the requirements for a valid inner product when A is not symmetric positive definite.
    [Show full text]
  • Finding Eigenvalues: Arnoldi Iteration and the QR Algorithm
    Finding Eigenvalues: Arnoldi Iteration and the QR Algorithm Will Wheeler Algorithms Interest Group Jul 13, 2016 Outline Finding the largest eigenvalue I Largest eigenvalue power method I So much work for one eigenvector. What about others? I More eigenvectors orthogonalize Krylov matrix I build orthogonal list Arnoldi iteration Solving the eigenvalue problem I Eigenvalues diagonal of triangular matrix I Triangular matrix QR algorithm I QR algorithm QR decomposition I Better QR decomposition Hessenberg matrix I Hessenberg matrix Arnoldi algorithm Power Method How do we find the largest eigenvalue of an m × m matrix A? I Start with a vector b and make a power sequence: b; Ab; A2b;::: I Higher eigenvalues dominate: n n n n A (v1 + v2) = λ1v1 + λ2v2 ≈ λ1v1 Vector sequence (normalized) converged? I Yes: Eigenvector I No: Iterate some more Krylov Matrix Power method throws away information along the way. I Put the sequence into a matrix K = [b; Ab; A2b;:::; An−1b] (1) = [xn; xn−1; xn−2;:::; x1] (2) I Gram-Schmidt approximates first n eigenvectors v1 = x1 (3) x2 · x1 v2 = x2 − x1 (4) x1 · x1 x3 · x1 x3 · x2 v3 = x3 − x1 − x2 (5) x1 · x1 x2 · x2 ::: (6) I Why not orthogonalize as we build the list? Arnoldi Iteration I Start with a normalized vector q1 xk = Aqk−1 (next power) (7) k−1 X yk = xk − (qj · xk ) qj (Gram-Schmidt) (8) j=1 qk = yk = jyk j (normalize) (9) I Orthonormal vectors fq1;:::; qng span Krylov subspace n−1 (Kn = spanfq1; Aq1;:::; A q1g) I These are not the eigenvectors of A, but make a similarity y (unitary) transformation Hn = QnAQn Arnoldi Iteration I Start with a normalized vector q1 xk = Aqk−1 (next power) (10) k−1 X yk = xk − (qj · xk ) qj (Gram-Schmidt) (11) j=1 qk = yk =jyk j (normalize) (12) I Orthonormal vectors fq1;:::; qng span Krylov subspace n−1 (Kn = spanfq1; Aq1;:::; A q1g) I These are not the eigenvectors of A, but make a similarity y (unitary) transformation Hn = QnAQn Arnoldi Iteration - Construct Hn We can construct the matrix Hn along the way.
    [Show full text]
  • LARGE-SCALE COMPUTATION of PSEUDOSPECTRA USING ARPACK and EIGS∗ 1. Introduction. the Matrices in Many Eigenvalue Problems
    SIAM J. SCI. COMPUT. c 2001 Society for Industrial and Applied Mathematics Vol. 23, No. 2, pp. 591–605 LARGE-SCALE COMPUTATION OF PSEUDOSPECTRA USING ARPACK AND EIGS∗ THOMAS G. WRIGHT† AND LLOYD N. TREFETHEN† Abstract. ARPACK and its Matlab counterpart, eigs, are software packages that calculate some eigenvalues of a large nonsymmetric matrix by Arnoldi iteration with implicit restarts. We show that at a small additional cost, which diminishes relatively as the matrix dimension increases, good estimates of pseudospectra in addition to eigenvalues can be obtained as a by-product. Thus in large- scale eigenvalue calculations it is feasible to obtain routinely not just eigenvalue approximations, but also information as to whether or not the eigenvalues are likely to be physically significant. Examples are presented for matrices with dimension up to 200,000. Key words. Arnoldi, ARPACK, eigenvalues, implicit restarting, pseudospectra AMS subject classifications. 65F15, 65F30, 65F50 PII. S106482750037322X 1. Introduction. The matrices in many eigenvalue problems are too large to allow direct computation of their full spectra, and two of the iterative tools available for computing a part of the spectrum are ARPACK [10, 11]and its Matlab counter- part, eigs.1 For nonsymmetric matrices, the mathematical basis of these packages is the Arnoldi iteration with implicit restarting [11, 23], which works by compressing the matrix to an “interesting” Hessenberg matrix, one which contains information about the eigenvalues and eigenvectors of interest. For general information on large-scale nonsymmetric matrix eigenvalue iterations, see [2, 21, 29, 31]. For some matrices, nonnormality (nonorthogonality of the eigenvectors) may be physically important [30].
    [Show full text]