Sparse Matrices and Iterative Methods

Home , Sparse matrix

Iterative Methods Sparsity

K. Cooper1

1Department of Mathematics Washington State University

2018

Cooper Washington State University Introduction Iterative Methods Sparsity

Iterative Methods

Consider the problem of solving Ax = b, where A is n × n. Why would we use an iterative method?

I Avoid direct decomposition (LU, QR, Cholesky)

I Replace with iterated matrix multiplication 3 I LU is O(n ) ﬂops. . . 2 I . . . matrix-vector multiplication is O(n )...

I so if we can get convergence in e.g. log(n), iteration might be faster.

Cooper Washington State University Introduction Iterative Methods Sparsity

Jacobi, GS, SOR

Some old methods:

I Jacobi is easily parallelized. . .

I . . . but converges extremely slowly.

I Gauss-Seidel/SOR converge faster. . .

I . . . but cannot be effectively parallelized.

I Only Jacobi really takes advantage of sparsity.

Cooper Washington State University Introduction Iterative Methods Sparsity

Sparsity

When a matrix is sparse (many more zero entries than nonzero), then typically the number of nonzero entries is O(n), so matrix-vector multiplication becomes an O(n) operation. This makes iterative methods very attractive. It does not help direct solves as much because of the problem of ﬁll-in, but we note that there are specialized solvers to minimize ﬁll-in.

Cooper Washington State University Introduction Iterative Methods Sparsity

Krylov Subspace Methods

A class of methods that converge in n iterations (in exact arithmetic). We hope that they arrive at a solution that is “close enough” in fewer iterations. Often these work much better than the classic methods. They are more readily parallelized, and take full advantage of sparsity.

Cooper Washington State University Introduction Iterative Methods Sparsity

Possibilities

Sparse matrices are quite common in computation

I Finite differences for PDEs

I Finite element for PDEs

I Integral equations with localized kernels

Cooper Washington State University Introduction Iterative Methods Sparsity

Structures

Nonzero elements shown in blue.

Note: nz denotes number of nonzeros out of ∼17 million entries.

Cooper Washington State University Introduction Iterative Methods Sparsity

Some formats

There are a few ways to store sparse matrices that are obvious.

I Diagonals - 1+ entry per nonzero

I Coordinates - 3 entries per nonzero

I Row- or column-oriented coordinates - 2+ entries per nonzero

Cooper Washington State University Introduction Iterative Methods Sparsity

Diagonal (DIA)

Matrix Sparse Storage 1 0 0 0 2 0 1 3 1 0 3 4 0 0 05  4640      0 6 7 0 0 0 0 8 7 0   →   0 0 8 9 0 0 0 2 9 0     1 0 0 2 3 0 0 5 3 2 040056 0 0 65

Offsets: −3 −1 0 3

Cooper Washington State University Introduction Iterative Methods Sparsity

Coordinate (COO)

Matrix Sparse Storage 1 0 0 0 2 0 3 4 0 0 05    0 0 1 1 1 ... 5 5 5 rows 0 6 7 0 0 0   → 0 4 0 1 5 ... 1 4 5 cols 0 0 8 9 0 0   1 2 3 45 ... 456 vals 1 0 0 2 3 0 040056

Often used for conversions. . .

Cooper Washington State University Introduction Iterative Methods Sparsity

Compressed Sparse Row (CSR)

Matrix Sparse Storage 1 0 0 0 2 0 3 4 0 0 05    0 4 0 1 5 ... 1 4 5 cols 0 6 7 0 0 0   → 1 2 3 45 ... 456 vals 0 0 8 9 0 0   02579 12 row offsets 1 0 0 2 3 0 040056

Also Compressed Sparse Column format, for multiplications.

Cooper Washington State University Introduction Iterative Methods Sparsity

Sparse Package

Scipy has a subpackage called sparse that implements many of these formats.

I Diagonal: dia_matrix()

I Coordinate: coo_matrix()

I CSR, CSC: csr_matrix(), csc_matrix()

I . . . and others.

Cooper Washington State University Introduction Iterative Methods Sparsity

Using Sparse

from scipy import * from scipy.sparse import csr_matrix A = csr_matrix([[-1,1,0,0],[0,-2,0,0],[0,-3,0,5],\ [0,0,1,1]]) x = array([1, 0, -1,0]) y = A.dot(x) print(y) print(A)

Results in y=[-1,0,0,-1]. A prints in COO format. Note that dot(A,x) does not work. dot must be the method of the sparse object.

Cooper Washington State University Introduction Iterative Methods Sparsity

Example

As a very simple example of the efﬁcacy of the sparse matrix package in scipy, consider the PDE

−∆x = 1, x|∂Ω = 0,

where the region Ω is the unit square. We solve this numerically using ﬁnite differences.

Cooper Washington State University Introduction Iterative Methods Sparsity

Matrix

There are many ways to assemble the matrix. Here is one. N = 100 Nsq = N*N h = 1.0/float(N+1) offsets = [-N,-1,0,1,N] subDiag1 = ones(Nsq) subDiag1[N-1:Nsq:N] = 0. supDiag1 = ones(Nsq) supDiag1[0:Nsq:N] = 0. A = dia_matrix(([-ones(Nsq),-subDiag1,4.*ones(Nsq),\ -supDiag1,-ones(Nsq)],offsets),shape=(Nsq,Nsq))

Cooper Washington State University Introduction Iterative Methods Sparsity

Conversion

It is easy to convert to other formats: e.g. given our DIA format matrix A, we can get other formats using: Acsr = A.tocsr() Afull = A.toarray()

Cooper Washington State University Introduction Iterative Methods Sparsity

Solve

We can solve the systems using various methods. Given: from scipy.linalg import solve as lsolve import scipy.sparse.linalg as sp Sparse Conjugate Gradient: soln = sp.cg(A,h*h*ones(Nsq)) Sparse LU: solnsp = sp.spsolve(Acsr,h*h*ones(Nsq)) Full LU: solnfull = lsolve(Afull,h*h*ones(Nsq))

Cooper Washington State University Introduction Iterative Methods Sparsity

Results

. . . for a 100 × 100 system of interior ﬁnite difference points (i.e. 10000 × 10000 matrix) Sparse CG: 0.037 seconds, 2.7e-7 difference from full Sparse LU: 0.056 seconds, 1.9e-13 difference from full Full LU: 15.2 seconds, 0 difference

Of course, for a more serious problem we would precondition etc.

Cooper Washington State University Introduction