Locally Optimal Block Preconditioned Conjugate Gradient Method for Hierarchical Matrices
Total Page:16
File Type:pdf, Size:1020Kb
Load more
Recommended publications
-
Overview of Iterative Linear System Solver Packages
Overview of Iterative Linear System Solver Packages Victor Eijkhout July, 1998 Abstract Description and comparison of several packages for the iterative solu- tion of linear systems of equations. 1 1 Intro duction There are several freely available packages for the iterative solution of linear systems of equations, typically derived from partial di erential equation prob- lems. In this rep ort I will give a brief description of a numberofpackages, and giveaninventory of their features and de ning characteristics. The most imp ortant features of the packages are which iterative metho ds and preconditioners supply; the most relevant de ning characteristics are the interface they present to the user's data structures, and their implementation language. 2 2 Discussion Iterative metho ds are sub ject to several design decisions that a ect ease of use of the software and the resulting p erformance. In this section I will give a global discussion of the issues involved, and how certain p oints are addressed in the packages under review. 2.1 Preconditioners A go o d preconditioner is necessary for the convergence of iterative metho ds as the problem to b e solved b ecomes more dicult. Go o d preconditioners are hard to design, and this esp ecially holds true in the case of parallel pro cessing. Here is a short inventory of the various kinds of preconditioners found in the packages reviewed. 2.1.1 Ab out incomplete factorisation preconditioners Incomplete factorisations are among the most successful preconditioners devel- op ed for single-pro cessor computers. Unfortunately, since they are implicit in nature, they cannot immediately b e used on parallel architectures. -
Accelerating the LOBPCG Method on Gpus Using a Blocked Sparse Matrix Vector Product
Accelerating the LOBPCG method on GPUs using a blocked Sparse Matrix Vector Product Hartwig Anzt and Stanimire Tomov and Jack Dongarra Innovative Computing Lab University of Tennessee Knoxville, USA Email: [email protected], [email protected], [email protected] Abstract— the computing power of today’s supercomputers, often accel- erated by coprocessors like graphics processing units (GPUs), This paper presents a heterogeneous CPU-GPU algorithm design and optimized implementation for an entire sparse iter- becomes challenging. ative eigensolver – the Locally Optimal Block Preconditioned Conjugate Gradient (LOBPCG) – starting from low-level GPU While there exist numerous efforts to adapt iterative lin- data structures and kernels to the higher-level algorithmic choices ear solvers like Krylov subspace methods to coprocessor and overall heterogeneous design. Most notably, the eigensolver technology, sparse eigensolvers have so far remained out- leverages the high-performance of a new GPU kernel developed side the main focus. A possible explanation is that many for the simultaneous multiplication of a sparse matrix and a of those combine sparse and dense linear algebra routines, set of vectors (SpMM). This is a building block that serves which makes porting them to accelerators more difficult. Aside as a backbone for not only block-Krylov, but also for other from the power method, algorithms based on the Krylov methods relying on blocking for acceleration in general. The subspace idea are among the most commonly used general heterogeneous LOBPCG developed here reveals the potential of eigensolvers [1]. When targeting symmetric positive definite this type of eigensolver by highly optimizing all of its components, eigenvalue problems, the recently developed Locally Optimal and can be viewed as a benchmark for other SpMM-dependent applications. -
A Geometric Theory for Preconditioned Inverse Iteration. III: a Short and Sharp Convergence Estimate for Generalized Eigenvalue Problems
A geometric theory for preconditioned inverse iteration. III: A short and sharp convergence estimate for generalized eigenvalue problems. Andrew V. Knyazev Department of Mathematics, University of Colorado at Denver, P.O. Box 173364, Campus Box 170, Denver, CO 80217-3364 1 Klaus Neymeyr Mathematisches Institut der Universit¨atT¨ubingen,Auf der Morgenstelle 10, 72076 T¨ubingen,Germany 2 Abstract In two previous papers by Neymeyr: A geometric theory for preconditioned inverse iteration I: Extrema of the Rayleigh quotient, LAA 322: (1-3), 61-85, 2001, and A geometric theory for preconditioned inverse iteration II: Convergence estimates, LAA 322: (1-3), 87-104, 2001, a sharp, but cumbersome, convergence rate estimate was proved for a simple preconditioned eigensolver, which computes the smallest eigenvalue together with the corresponding eigenvector of a symmetric positive def- inite matrix, using a preconditioned gradient minimization of the Rayleigh quotient. In the present paper, we discover and prove a much shorter and more elegant, but still sharp in decisive quantities, convergence rate estimate of the same method that also holds for a generalized symmetric definite eigenvalue problem. The new estimate is simple enough to stimulate a search for a more straightforward proof technique that could be helpful to investigate such practically important method as the locally optimal block preconditioned conjugate gradient eigensolver. We demon- strate practical effectiveness of the latter for a model problem, where it compares favorably with two -
16 Preconditioning
16 Preconditioning The general idea underlying any preconditioning procedure for iterative solvers is to modify the (ill-conditioned) system Ax = b in such a way that we obtain an equivalent system Aˆxˆ = bˆ for which the iterative method converges faster. A standard approach is to use a nonsingular matrix M, and rewrite the system as M −1Ax = M −1b. The preconditioner M needs to be chosen such that the matrix Aˆ = M −1A is better conditioned for the conjugate gradient method, or has better clustered eigenvalues for the GMRES method. 16.1 Preconditioned Conjugate Gradients We mentioned earlier that the number of iterations required for the conjugate gradient algorithm to converge is proportional to pκ(A). Thus, for poorly conditioned matrices, convergence will be very slow. Thus, clearly we will want to choose M such that κ(Aˆ) < κ(A). This should result in faster convergence. How do we find Aˆ, xˆ, and bˆ? In order to ensure symmetry and positive definiteness of Aˆ we let M −1 = LLT (44) with a nonsingular m × m matrix L. Then we can rewrite Ax = b ⇐⇒ M −1Ax = M −1b ⇐⇒ LT Ax = LT b ⇐⇒ LT AL L−1x = LT b . | {z } | {z } |{z} =Aˆ =xˆ =bˆ The symmetric positive definite matrix M is called splitting matrix or preconditioner, and it can easily be verified that Aˆ is symmetric positive definite, also. One could now formally write down the standard CG algorithm with the new “hat- ted” quantities. However, the algorithm is more efficient if the preconditioning is incorporated directly into the iteration. To see what this means we need to examine every single line in the CG algorithm. -
Numerical Linear Algebra Solving Ax = B , an Overview Introduction
Solving Ax = b, an overview Numerical Linear Algebra ∗ A = A no Good precond. yes flex. precond. yes GCR Improving iterative solvers: ⇒ ⇒ ⇒ yes no no ⇓ ⇓ ⇓ preconditioning, deflation, numerical yes GMRES A > 0 CG software and parallelisation ⇒ ⇓ no ⇓ ⇓ ill cond. yes SYMMLQ str indef no large im eig no Bi-CGSTAB ⇒ ⇒ ⇒ no yes yes ⇓ ⇓ ⇓ MINRES IDR no large im eig BiCGstab(ℓ) ⇐ yes Gerard Sleijpen and Martin van Gijzen a good precond itioner is available ⇓ the precond itioner is flex ible IDRstab November 29, 2017 A + A∗ is strongly indefinite A 1 has large imaginary eigenvalues November 29, 2017 2 National Master Course National Master Course Delft University of Technology Introduction Program We already saw that the performance of iterative methods can Preconditioning be improved by applying a preconditioner. Preconditioners (and • Diagonal scaling, Gauss-Seidel, SOR and SSOR deflation techniques) are a key to successful iterative methods. • Incomplete Choleski and Incomplete LU In general they are very problem dependent. • Deflation Today we will discuss some standard preconditioners and we will • Numerical software explain the idea behind deflation. • Parallelisation We will also discuss some efforts to standardise numerical • software. Shared memory versus distributed memory • Domain decomposition Finally we will discuss how to perform scientific computations on • a parallel computer. November 29, 2017 3 November 29, 2017 4 National Master Course National Master Course Preconditioning Why preconditioners? A preconditioned iterative solver solves the system − − M 1Ax = M 1b. 1 The matrix M is called the preconditioner. 0.8 The preconditioner should satisfy certain requirements: 0.6 Convergence should be (much) faster (in time) for the 0.4 • preconditioned system than for the original system. -
Solving Symmetric Semi-Definite (Ill-Conditioned)
Solving Symmetric Semi-definite (ill-conditioned) Generalized Eigenvalue Problems Zhaojun Bai University of California, Davis Berkeley, August 19, 2016 Symmetric definite generalized eigenvalue problem I Symmetric definite generalized eigenvalue problem Ax = λBx where AT = A and BT = B > 0 I Eigen-decomposition AX = BXΛ where Λ = diag(λ1; λ2; : : : ; λn) X = (x1; x2; : : : ; xn) XT BX = I: I Assume λ1 ≤ λ2 ≤ · · · ≤ λn LAPACK solvers I LAPACK routines xSYGV, xSYGVD, xSYGVX are based on the following algorithm (Wilkinson'65): 1. compute the Cholesky factorization B = GGT 2. compute C = G−1AG−T 3. compute symmetric eigen-decomposition QT CQ = Λ 4. set X = G−T Q I xSYGV[D,X] could be numerically unstable if B is ill-conditioned: −1 jλbi − λij . p(n)(kB k2kAk2 + cond(B)jλbij) · and −1 1=2 kB k2kAk2(cond(B)) + cond(B)jλbij θ(xbi; xi) . p(n) · specgapi I User's choice between the inversion of ill-conditioned Cholesky decomposition and the QZ algorithm that destroys symmetry Algorithms to address the ill-conditioning 1. Fix-Heiberger'72 (Parlett'80): explicit reduction 2. Chang-Chung Chang'74: SQZ method (QZ by Moler and Stewart'73) 3. Bunse-Gerstner'84: MDR method 4. Chandrasekaran'00: \proper pivoting scheme" 5. Davies-Higham-Tisseur'01: Cholesky+Jacobi 6. Working notes by Kahan'11 and Moler'14 This talk Three approaches: 1. A LAPACK-style implementation of Fix-Heiberger algorithm 2. An algebraic reformulation 3. Locally accelerated block preconditioned steepest descent (LABPSD) This talk Three approaches: 1. A LAPACK-style implementation of Fix-Heiberger algorithm Status: beta-release 2. -
On the Performance and Energy Efficiency of Sparse Linear Algebra on Gpus
Original Article The International Journal of High Performance Computing Applications 1–16 On the performance and energy efficiency Ó The Author(s) 2016 Reprints and permissions: sagepub.co.uk/journalsPermissions.nav of sparse linear algebra on GPUs DOI: 10.1177/1094342016672081 hpc.sagepub.com Hartwig Anzt1, Stanimire Tomov1 and Jack Dongarra1,2,3 Abstract In this paper we unveil some performance and energy efficiency frontiers for sparse computations on GPU-based super- computers. We compare the resource efficiency of different sparse matrix–vector products (SpMV) taken from libraries such as cuSPARSE and MAGMA for GPU and Intel’s MKL for multicore CPUs, and develop a GPU sparse matrix–matrix product (SpMM) implementation that handles the simultaneous multiplication of a sparse matrix with a set of vectors in block-wise fashion. While a typical sparse computation such as the SpMV reaches only a fraction of the peak of current GPUs, we show that the SpMM succeeds in exceeding the memory-bound limitations of the SpMV. We integrate this kernel into a GPU-accelerated Locally Optimal Block Preconditioned Conjugate Gradient (LOBPCG) eigensolver. LOBPCG is chosen as a benchmark algorithm for this study as it combines an interesting mix of sparse and dense linear algebra operations that is typical for complex simulation applications, and allows for hardware-aware optimizations. In a detailed analysis we compare the performance and energy efficiency against a multi-threaded CPU counterpart. The reported performance and energy efficiency results are indicative of sparse computations on supercomputers. Keywords sparse eigensolver, LOBPCG, GPU supercomputer, energy efficiency, blocked sparse matrix–vector product 1Introduction GPU-based supercomputers. -
Preconditioned Inverse Iteration and Shift-Invert Arnoldi Method
Preconditioned inverse iteration and shift-invert Arnoldi method Melina Freitag Department of Mathematical Sciences University of Bath CSC Seminar Max-Planck-Institute for Dynamics of Complex Technical Systems Magdeburg 3rd May 2011 joint work with Alastair Spence (Bath) 1 Introduction 2 Inexact inverse iteration 3 Inexact Shift-invert Arnoldi method 4 Inexact Shift-invert Arnoldi method with implicit restarts 5 Conclusions Outline 1 Introduction 2 Inexact inverse iteration 3 Inexact Shift-invert Arnoldi method 4 Inexact Shift-invert Arnoldi method with implicit restarts 5 Conclusions Problem and iterative methods Find a small number of eigenvalues and eigenvectors of: Ax = λx, λ ∈ C,x ∈ Cn A is large, sparse, nonsymmetric Problem and iterative methods Find a small number of eigenvalues and eigenvectors of: Ax = λx, λ ∈ C,x ∈ Cn A is large, sparse, nonsymmetric Iterative solves Power method Simultaneous iteration Arnoldi method Jacobi-Davidson method repeated application of the matrix A to a vector Generally convergence to largest/outlying eigenvector Problem and iterative methods Find a small number of eigenvalues and eigenvectors of: Ax = λx, λ ∈ C,x ∈ Cn A is large, sparse, nonsymmetric Iterative solves Power method Simultaneous iteration Arnoldi method Jacobi-Davidson method The first three of these involve repeated application of the matrix A to a vector Generally convergence to largest/outlying eigenvector Shift-invert strategy Wish to find a few eigenvalues close to a shift σ σ λλ λ λ λ 3 1 2 4 n Shift-invert strategy Wish to find a few eigenvalues close to a shift σ σ λλ λ λ λ 3 1 2 4 n Problem becomes − 1 (A − σI) 1x = x λ − σ each step of the iterative method involves repeated application of A =(A − σI)−1 to a vector Inner iterative solve: (A − σI)y = x using Krylov method for linear systems. -
Preconditioning the Coarse Problem of BDDC Methods—Three-Level, Algebraic Multigrid, and Vertex-Based Preconditioners
Electronic Transactions on Numerical Analysis. Volume 51, pp. 432–450, 2019. ETNA Kent State University and Copyright c 2019, Kent State University. Johann Radon Institute (RICAM) ISSN 1068–9613. DOI: 10.1553/etna_vol51s432 PRECONDITIONING THE COARSE PROBLEM OF BDDC METHODS— THREE-LEVEL, ALGEBRAIC MULTIGRID, AND VERTEX-BASED PRECONDITIONERS∗ AXEL KLAWONNyz, MARTIN LANSERyz, OLIVER RHEINBACHx, AND JANINE WEBERy Abstract. A comparison of three Balancing Domain Decomposition by Constraints (BDDC) methods with an approximate coarse space solver using the same software building blocks is attempted for the first time. The comparison is made for a BDDC method with an algebraic multigrid preconditioner for the coarse problem, a three-level BDDC method, and a BDDC method with a vertex-based coarse preconditioner. It is new that all methods are presented and discussed in a common framework. Condition number bounds are provided for all approaches. All methods are implemented in a common highly parallel scalable BDDC software package based on PETSc to allow for a simple and meaningful comparison. Numerical results showing the parallel scalability are presented for the equations of linear elasticity. For the first time, this includes parallel scalability tests for a vertex-based approximate BDDC method. Key words. approximate BDDC, three-level BDDC, multilevel BDDC, vertex-based BDDC AMS subject classifications. 68W10, 65N22, 65N55, 65F08, 65F10, 65Y05 1. Introduction. During the last decade, approximate variants of the BDDC (Balanc- ing Domain Decomposition by Constraints) and FETI-DP (Finite Element Tearing and Interconnecting-Dual-Primal) methods have become popular for the solution of various linear and nonlinear partial differential equations [1,8,9, 12, 14, 15, 17, 19, 21, 24, 25]. -
The Preconditioned Conjugate Gradient Method with Incomplete Factorization Preconditioners
Computers Math. Applic. Vol. 31, No. 4/5, pp. 1-5, 1996 Pergamon Copyright©1996 Elsevier Science Ltd Printed in Great Britain. All rights reserved 0898-1221/96 $15.00 4- 0.00 0898-1221 (95)00210-3 The Preconditioned Conjugate Gradient Method with Incomplete Factorization Preconditioners I. ARANY University of Miskolc, Institute of Mathematics 3515 Miskolc-Egyetemv~iros, Hungary mat arany©gold, uni-miskolc, hu Abstract--We present a modification of the ILUT algorithm due to Y. Sand for preparing in- complete factorization preconditioner for positive definite matrices being neither diagonally dominant nor M-matrices. In all tested cases, the rate of convergence of the relevant preconditioned conjugate gradient method was at a sufficient level, when linear systems arising from static problem for bar structures with 3D beam elements were solved. Keywords--Preconditioned conjugate gradient method, Incomplete factorization. 1. INTRODUCTION For solving large sparse positive definite systems of linear equations, the conjugate gradient method combined with a preconditoning technique seems to be one of the most efficient ways among the simple iterative methods. As known, for positive definite symmetric diagonally dom- inant M-matrices, the incomplete factorization technique may be efficiently used for preparing preconditioners [1]. However, for more general cases, when the matrix is neither diagonally domi- nant nor M-matrix, but it is a positive definite one, some incomplete factorization techniques [2,3] seem to be quite unusable, since usually, during the process, a diagonal element is turning to zero. We present a modification to the ILUT algorithm of Sand [2] for preparing preconditioners for the above mentioned matrices. -
Performance of H-Lu Preconditioning for Sparse Matrices
COMPUTATIONAL METHODS IN APPLIED MATHEMATICS, Vol.8(2008), No.4, pp.336–349 c 2008 Institute of Mathematics of the National Academy of Sciences of Belarus PERFORMANCE OF H-LU PRECONDITIONING FOR SPARSE MATRICES L. GRASEDYCK1 W. HACKBUSCH2, AND R. KRIEMANN2 Abstract — In this paper we review the technique of hierarchical matrices and put it into the context of black-box solvers for large linear systems. Numerical examples for several classes of problems from medium- to large-scale illustrate the applicability and efficiency of this technique. We compare the results with those of several direct solvers (which typically scale quadratically in the matrix size) as well as an iterative solver (algebraic multigrid) which scales linearly (if it converges in O(1) steps). 2000 Mathematics Subject Classification: 65F05, 65F30, 65F50, 65N55. Keywords: hierarchical matrices, black-box clustering, preconditioning, LU, direct solver, iterative solver, performance. 1. Introduction Many methods for the fast solution of large-scale linear systems exploit some hierarchical structure of the system. For example, when the system stems from the finite element (FE) discretization of a partial differential equation (PDE) posed in a domain Ω ⊂ Rd, then one can exploit the knowledge of the underlying geometrical problem. Whereas multigrid meth- ods use a discretization of the pde on different levels of resolution, hierarchical (H-) matrices use the geometry information associated to the degrees of freedom on a single level in order to find a distance measure for degrees of freedom. Once this distance measure is available, an almost black-box type arithmetic can be applied for the inversion or factorization of the system matrix. -
High Efficiency Spectral Analysis and BLAS-3
High Efficiency Spectral Analysis and BLAS-3 Randomized QRCP with Low-Rank Approximations by Jed Alma Duersch A dissertation submitted in partial satisfaction of the requirements for the degree of Doctor of Philosophy in Applied Mathematics in the Graduate Division of the University of California, Berkeley Committee in charge: Professor Ming Gu, Chair Assistant Professor Lin Lin Professor Kameshwar Poolla Fall 2015 High Efficiency Spectral Analysis and BLAS-3 Randomized QRCP with Low-Rank Approximations Copyright 2015 by Jed Alma Duersch i Contents Contents i 1 A Robust and Efficient Implementation of LOBPCG 2 1.1 Contributions and results . 2 1.2 Introduction to LOBPCG . 3 1.3 The basic LOBPCG algorithm . 4 1.4 Numerical stability and basis selection . 6 1.5 Stability improvements . 10 1.6 Efficiency improvements . 13 1.7 Analysis . 22 1.8 Numerical examples . 25 2 Spectral Target Residual Descent 32 2.1 Contributions and results . 32 2.2 Introduction to interior eigenvalue targeting . 33 2.3 Spectral targeting analysis . 34 2.4 Notes on generalized spectral targeting . 42 2.5 Numerical experiments . 48 3 True BLAS-3 Performance QRCP using Random Sampling 53 3.1 Contributions and results . 53 3.2 Introduction to QRCP . 54 3.3 Sample QRCP . 58 3.4 Sample updates . 64 3.5 Full randomized QRCP . 66 3.6 Parallel implementation notes . 71 3.7 Truncated approximate Singular Value Decomposition . 73 3.8 Experimental performance . 77 ii Bibliography 93 iii Acknowledgments I gratefully thank my coauthors, Dr. Meiyue Shao and Dr. Chao Yang, for their time and expertise. I would also like to thank Assistant Professor Lin Lin, Dr.