Iterative Methods Sparsity Sparse Matrices and Iterative Methods K. Cooper1 1Department of Mathematics Washington State University 2018 Cooper Washington State University Introduction Iterative Methods Sparsity Iterative Methods Consider the problem of solving Ax = b, where A is n × n. Why would we use an iterative method? I Avoid direct decomposition (LU, QR, Cholesky) I Replace with iterated matrix multiplication 3 I LU is O(n ) flops. 2 I . matrix-vector multiplication is O(n )... I so if we can get convergence in e.g. log(n), iteration might be faster. Cooper Washington State University Introduction Iterative Methods Sparsity Jacobi, GS, SOR Some old methods: I Jacobi is easily parallelized. I . but converges extremely slowly. I Gauss-Seidel/SOR converge faster. I . but cannot be effectively parallelized. I Only Jacobi really takes advantage of sparsity. Cooper Washington State University Introduction Iterative Methods Sparsity Sparsity When a matrix is sparse (many more zero entries than nonzero), then typically the number of nonzero entries is O(n), so matrix-vector multiplication becomes an O(n) operation. This makes iterative methods very attractive. It does not help direct solves as much because of the problem of fill-in, but we note that there are specialized solvers to minimize fill-in. Cooper Washington State University Introduction Iterative Methods Sparsity Krylov Subspace Methods A class of methods that converge in n iterations (in exact arithmetic). We hope that they arrive at a solution that is “close enough” in fewer iterations. Often these work much better than the classic methods. They are more readily parallelized, and take full advantage of sparsity. Cooper Washington State University Introduction Iterative Methods Sparsity Possibilities Sparse matrices are quite common in computation I Finite differences for PDEs I Finite element for PDEs I Integral equations with localized kernels Cooper Washington State University Introduction Iterative Methods Sparsity Structures Nonzero elements shown in blue. Note: nz denotes number of nonzeros out of ∼17 million entries. Cooper Washington State University Introduction Iterative Methods Sparsity Some formats There are a few ways to store sparse matrices that are obvious. I Diagonals - 1+ entry per nonzero I Coordinates - 3 entries per nonzero I Row- or column-oriented coordinates - 2+ entries per nonzero Cooper Washington State University Introduction Iterative Methods Sparsity Diagonal (DIA) Matrix Sparse Storage 21 0 0 0 2 03 21 3 1 03 63 4 0 0 05 7 64640 7 6 7 6 7 60 6 7 0 0 07 60 8 7 07 6 7 ! 6 7 60 0 8 9 0 07 60 2 9 07 6 7 6 7 41 0 0 2 3 05 40 5 3 25 040056 0 0 65 Offsets: −3 −1 0 3 Cooper Washington State University Introduction Iterative Methods Sparsity Coordinate (COO) Matrix Sparse Storage 21 0 0 0 2 03 63 4 0 0 05 7 6 7 0 0 1 1 1 ::: 5 5 5 rows 60 6 7 0 0 07 6 7 ! 0 4 0 1 5 ::: 1 4 5 cols 60 0 8 9 0 07 6 7 1 2 3 45 ::: 456 vals 41 0 0 2 3 05 040056 Often used for conversions. Cooper Washington State University Introduction Iterative Methods Sparsity Compressed Sparse Row (CSR) Matrix Sparse Storage 21 0 0 0 2 03 63 4 0 0 05 7 6 7 0 4 0 1 5 ::: 1 4 5 cols 60 6 7 0 0 07 6 7 ! 1 2 3 45 ::: 456 vals 60 0 8 9 0 07 6 7 02579 12 row offsets 41 0 0 2 3 05 040056 Also Compressed Sparse Column format, for multiplications. Cooper Washington State University Introduction Iterative Methods Sparsity Sparse Package Scipy has a subpackage called sparse that implements many of these formats. I Diagonal: dia_matrix() I Coordinate: coo_matrix() I CSR, CSC: csr_matrix(), csc_matrix() I . and others. Cooper Washington State University Introduction Iterative Methods Sparsity Using Sparse from scipy import * from scipy.sparse import csr_matrix A = csr_matrix([[-1,1,0,0],[0,-2,0,0],[0,-3,0,5],\ [0,0,1,1]]) x = array([1, 0, -1,0]) y = A.dot(x) print(y) print(A) Results in y=[-1,0,0,-1]. A prints in COO format. Note that dot(A,x) does not work. dot must be the method of the sparse object. Cooper Washington State University Introduction Iterative Methods Sparsity Example As a very simple example of the efficacy of the sparse matrix package in scipy, consider the PDE −∆x = 1; xj@Ω = 0; where the region Ω is the unit square. We solve this numerically using finite differences. Cooper Washington State University Introduction Iterative Methods Sparsity Matrix There are many ways to assemble the matrix. Here is one. N = 100 Nsq = N*N h = 1.0/float(N+1) offsets = [-N,-1,0,1,N] subDiag1 = ones(Nsq) subDiag1[N-1:Nsq:N] = 0. supDiag1 = ones(Nsq) supDiag1[0:Nsq:N] = 0. A = dia_matrix(([-ones(Nsq),-subDiag1,4.*ones(Nsq),\ -supDiag1,-ones(Nsq)],offsets),shape=(Nsq,Nsq)) Cooper Washington State University Introduction Iterative Methods Sparsity Conversion It is easy to convert to other formats: e.g. given our DIA format matrix A, we can get other formats using: Acsr = A.tocsr() Afull = A.toarray() Cooper Washington State University Introduction Iterative Methods Sparsity Solve We can solve the systems using various methods. Given: from scipy.linalg import solve as lsolve import scipy.sparse.linalg as sp Sparse Conjugate Gradient: soln = sp.cg(A,h*h*ones(Nsq)) Sparse LU: solnsp = sp.spsolve(Acsr,h*h*ones(Nsq)) Full LU: solnfull = lsolve(Afull,h*h*ones(Nsq)) Cooper Washington State University Introduction Iterative Methods Sparsity Results . for a 100 × 100 system of interior finite difference points (i.e. 10000 × 10000 matrix) Sparse CG: 0.037 seconds, 2.7e-7 difference from full Sparse LU: 0.056 seconds, 1.9e-13 difference from full Full LU: 15.2 seconds, 0 difference Of course, for a more serious problem we would precondition etc. Cooper Washington State University Introduction.