sign in sign up
<<
Home , Applied mathematics, Matrix analysis, Numerical linear algebra

Numerical Linear and Analysis

Higham, Nicholas J.

2015

MIMS EPrint: 2015.103

Manchester Institute for Mathematical School of

The University of Manchester

Reports available from: http://eprints.maths.manchester.ac.uk/ And by contacting: The MIMS Secretary School of Mathematics The University of Manchester Manchester, M13 9PL, UK

ISSN 1749-9097 1

exploitation of matrix structure (such as sparsity, sym- Numerical and metry, and definiteness), and the design of † Matrix Analysis to exploit evolving architectures. Nicholas J. Higham Throughout the article, uppercase letters are used for matrices and lower case letters for vectors and scalars. Matrices are ubiquitous in . Matrices and vectors are assumed to be complex, unless ∗ Ordinary differential (ODEs) and partial dif- otherwise stated, and A = (aji) denotes the conjugate ferential equations (PDEs) are solved numerically by of A = (aij ). An unsubscripted norm k · k finite difference or finite element methods, which lead denotes a general vector norm and the corresponding to systems of linear equations or matrix eigenvalue subordinate . Particular norms used here problems. Nonlinear equations and optimization prob- are the 2-norm k · k2 and the Frobenius norm k · kF . lems are typically solved using linear or quadratic The notation “i = 1: n” means that the integer models, which again lead to linear systems. i takes on the values 1, 2, . . . , n. Solving linear systems of equations is an ancient task, undertaken by the Chinese around 1AD, but the study 1 Nonsingularity and Conditioning of matrices per se is relatively recent, originating with Arthur Cayley’s 1858 “A Memoir on the Theory of Matri- Nonsingularity of a matrix is a key requirement in many ces”. Early research on matrices was largely theoret- problems, such as in the solution of n linear equations ical, with much attention focused on the development in n unknowns. For some classes of matrices, nonsingu- of canonical forms, but in the 20th century the practi- larity is guaranteed. A good example is the diagonally cal value of matrices started to be appreciated. Heisen- dominant matrices. The matrix A ∈ Cn×n is strictly berg used matrix theory as a tool in the development diagonally dominant by rows if of quantum mechanics in the 1920s. Early proponents X of the systematic use of matrices in applied mathemat- |aij | < |aii|, i = 1: n ics included Frazer, Duncan, and Collar, whose 1938 j6=i book Elementary Matrices and Some Applications to and strictly diagonally dominant by columns if A∗ is Dynamics and Differential Equations emphasized the strictly diagonally dominant by rows. Any matrix that important role of matrices in differential equations and is strictly diagonally dominant by rows or columns mechanics. The continued growth of matrices in appli- is nonsingular (a proof can be obtained by applying cations, together with the advent of mechanical and Gershgorin’s theorem in section 5.1). then digital computing devices, allowing ever larger Since data is often subject to uncertainty we wish problems to be solved, created the need for greater to gauge the sensitivity of problems to perturbations, understanding of all aspects of matrices from theory which is done using condition numbers. An appropriate to computation. for the matrix inverse is This article treats two closely related topics: matrix k(A + ∆A)−1 − A−1k analysis, which is the theory of matrices with a focus lim sup . ε→0 εkA−1k on aspects relevant to other , and k∆Ak6εkAk (also called matrix computa- This expression turns out to equal κ(A) = kAkkA−1k, tions), which is concerned with the construction and which is called the condition number of A with respect for solving matrix problems as to inversion. This condition number occurs in many well as related topics such as problem sensitivity and contexts. For example, suppose A is contaminated rounding error analysis. by errors and we perform a similarity transformation −1 −1 −1 Important themes that are discussed in this article X (A + E)X = X AX + F. Then kFk = kX EXk 6 include the matrix factorization paradigm, the use of κ(X)kEk and this bound is attainable for some E. Hence unitary transformations for their , the errors can be multiplied by a factor as large as κ(X). We therefore prefer to carry out similarity and

†. Author’s final version, before copy editing and cross-referencing, other transformations with matrices that are well con- of: N. J. Higham. Numerical linear algebra and matrix analysis. In N. J. ditioned, that is, ones for which κ(X) is close to its Higham, M. . Dennis, P. Glendinning, P. A. Martin, F. Santosa, and lower bound of 1. By contrast, a matrix for which is J. Tanner, editors, The Princeton Companion to Applied Mathematics, κ pages 263–281. Princeton University Press, Princeton, NJ, USA, 2015. large is called ill conditioned. For any X, 2

κ2(X) = 1, so in numerical linear algebra transforma- angular systems. It is then clear how to solve efficiently tions by unitary or orthogonal matrices are preferred several systems Axi = bi, i = 1: r , with different right- and usually lead to numerically stable algorithms. hand sides but the same coefficient matrix A: compute In practice we often need an estimate of the matrix the LU factors once and then re-use them to solve for condition number number κ(A) but do not wish to go each xi in turn. to the expense of computing A−1 in order to obtain This matrix factorization1 viewpoint dates from it. Fortunately, there are algorithms that can cheaply around the 1940s and has been extremely successful produce a reliable estimate of κ(A) once a factorization in matrix computations. In general, a factorization is a of A has been computed. representation of a matrix as a product of “simpler” Note that the , det(A), is rarely com- matrices. Factorization is a tool that can be used to puted in numerical linear algebra. Its magnitude gives solve a variety of problems, as we will see below. no useful information about the conditioning of A, not Two particular benefits of factorizations are unity least because of its extreme behavior under scaling: and modularity. GE, for example, can be organized det(αA) = αn det(A). in several different ways, corresponding to different orderings of the three nested loops that it comprises, 2 Matrix Factorizations as well as the use of different blockings of the matrix elements. Yet all of them compute the same LU factor- The method of (GE) for solving ization, carrying out the same mathematical operations a nonsingular linear system Ax = b of n equations in a different order. Without the unifying concept of a in n unknowns reduces the matrix A to upper trian- factorization, reasoning about these GE variants would gular form and then solves for x by substitution. GE be difficult. is typically described by writing down the equations Modularity refers to the way that a factorization (k+1) (k) (k) (k) (k) aij = aij − aik akj /akk (and similarly for b) that breaks a problem down into separate tasks, which can (1) (1) be analyzed or programmed independently. To carry describe how the starting matrix A = A = (aij ) changes on each of the n − 1 steps of the elimina- out a rounding error analysis of GE we can analyze the tion in its progress towards upper triangular form U. LU factorization and the solution of the triangular sys- Working at the element level in this way leads to a pro- tems by substitution separately and then put the analy- fusion of symbols, superscripts, and subscripts that ses together. The rounding error analysis of substitu- tend to obscure the mathematical structure and hin- tion can be re-used in the many other contexts in which der insights being drawn into the underlying process. triangular systems arise. One of the key developments in the last century was An important example of the use of LU factoriza- the recognition that it is much more profitable to work tion is in iterative refinement. Suppose we have used at the matrix level. Thus the basic above is GE to obtain a computed solution xb to Ax = b in (k+1) (k) floating-point . If we form r = b − Ax and written as A = MkA , where Mk agrees with the b except below the diagonal in the kth solve Ae = r , then in exact arithmetic y = xb + e is (k) (k) the true solution. In computing e we can reuse the LU column, where its (i, k) element is mik = −aik /akk , i = k + 1: n. Recurring the matrix equation gives factors of A, so obtaining y from xb is inexpensive. In (n) practice, the computation of r , e, and y is subject to U := A = Mn−1 ...M1A. Taking the Mk matrices over to the left-hand side leads, after some calculations, to rounding errors so the computed yb is not equal to x. the equation A = LU, where L is unit lower triangular, But under suitable assumptions yb will be an improved approximation and we can iterate this refinement pro- with (i, k) element mik. The prefix “unit” means that L cess. Iterative refinement is particularly effective if has ones on the diagonal. r can be computed using extra precision. GE is therefore equivalent to factorizing the matrix Two other key factorizations are: A as the product of a lower and an upper triangular matrix—something that is not at all • Cholesky factorization: for Hermitian positive def- obvious from the element-level equations. Solving the inite A ∈ Cn×n, A = R∗R, where R is upper tri- linear system Ax = b now reduces to the task of solving angular with positive diagonal elements, and this the two triangular systems Ly = b and Ux = y. factorization is unique. Interpreting GE as LU factorization separates the computation of the factors from the solution of the tri- 1. Or decomposition—the two terms are essentially synonymous. 3

m×n T • QR factorization: for A ∈ with m > n, A = (recall that Fn = Fn), though this was not realized QR where Q ∈ Cm×m is unitary (Q∗Q = I ) and when the methods were developed. Transposition also m h i m×n R1 R ∈ C is upper trapezoidal, that is, R = 0 plays an important role in automatic differentiation: n×n with R1 ∈ C upper triangular. the so-called reverse or adjoint mode can be obtained by transposing a matrix factorization representation of These two factorizations are related: if A ∈ Cm×n with the forward mode. m > n has full and A = QR is a QR factorization, The factorizations described in this section are in in which without loss of generality we can assume that “plain vanilla” form, but all have variants that incor- R has positive diagonal, then A∗A = R∗R, so R is the porate pivoting. Pivoting refers to row or column inter- Cholesky factor of A∗A. changes carried out at each step of the factorization as The Cholesky factorization can be computed by what it is computed, introduced either to ensure that the fac- is essentially a symmetric and scaled version of GE. The torization succeeds and is numerically stable or to pro- QR factorization can be computed in three main ways, duce a factorization with certain desirable properties one of which is the classical Gram–Schmidt orthogonal- usually associated with rank deficiency. For GE, partial ization. The most widely used method constructs Q as pivoting is normally used: at the start of the kth stage a product of Householder reflectors, which are unitary (k) of the elimination an element ar k of largest modulus in matrices of the form H = I −2vv∗/(v∗v), where v is a the kth column below the diagonal is brought into the nonzero vector. Note that H is a rank 1 perturbation of (k, k) (pivot) position by interchanging rows k and r . the identity and since it is Hermitian and unitary it is its (k) Partial pivoting avoids dividing by zero (if akk = 0 after own inverse, that is, it is involutory. The third approach the interchange then the pivot column is zero below the builds Q as a product of Givens rotations, each of which diagonal and the elimination step can be skipped). More  c s  is a 2 × 2 matrix −s c embedded into two rows and importantly, partial pivoting ensures numerical stabil- columns of an m×m identity matrix, where (in the real ity; see section8. The overall effect of GE with partial case) c2 + s2 = 1. pivoting is to produce an LU factorization PA = LU, The Cholesky factorization helps us to make the where P is a permutation matrix. most of the very desirable property of positive definite- Pivoted variants of Cholesky factorization and QR ness. For example, suppose A is Hermitian positive def- factorization take the form P TAP = R∗R and AP = h i = ∗ −1 R inite and we wish to evaluate the α x A x. Q 0 , where P is a permutation matrix and R satisfies We can rewrite it as x∗(R∗R)−1x = (x∗R−1)(R−∗x) = the inequalities ∗ −∗ z z, where z = R x. So once the Cholesky factoriza- j 2 X 2 tion has been computed we need just one triangular |rkk| > |rij | , j = k + 1: n, k = 1: n. solve to compute α, and of course there is no need to i=k h i explicitly invert the matrix A. R11 R12 If A is rank deficient then R has the form R = 0 0 A matrix factorization might involve a larger num- with R11 nonsingular, and the rank of A is the dimen- ber of factors: A = N1N2 ...N , say. It is immediate k sion of R11. Equally importantly, when A is nearly rank T = T T T that A Nk Nk−1 ...N1 . This factorization of the deficient this tends to be revealed by a small trailing transpose may have deep consequences in a particu- diagonal block of R. lar application. For example, the discrete Fourier trans- A factorization of great importance in a wide vari- form is the matrix–vector product y = Fnx, where ety of applications is the decomposition the n × n matrix Fn has (p, q) element exp(−2πi(p − (SVD) of A ∈ Cm×n: 1)(q − 1)/n); Fn is a complex, . The A = UΣV ∗, Σ = diag(σ , σ , . . . , σ ) ∈ Rm×n, (1) fast Fourier transform (FFT) is a way of evaluating y in 1 2 p 2 m×m n×n O(n log2 n) operations, as opposed to the O(n ) oper- where p = min(m, n), U ∈ C and V ∈ C are ations that are required by a standard matrix–vector unitary, and the singular values σi satisfy σ1 > σ2 > multiplication. Many variants of the FFT have been pro- ··· > σp > 0. For a square A (m = n), the 2-norm posed since the original 1965 paper by Cooley and condition number is given by κ2(A) = σ1/σn. m×n Tukey. It turns out that different FFT variants corre- The polar decomposition of A ∈ C with m > n m×n spond to different factorizations of Fn with k = log2 n is a factorization A = UH in which U ∈ C has sparse factors. Some of these methods correspond sim- orthonormal columns and H ∈ Cn×n is Hermitian posi- ply to transposing the factorization in another method tive semidefinite. The matrix H is unique and is given by 4

(A∗A)1/2, where the exponent 1/2 denotes the princi- be rewritten as the inequality k∆Ak/kAk < κ(A)−1, −1 pal square root, while U is unique if A has full rank. The where κ(A) = kAkkA k > 1 is the condition number polar decomposition generalizes to matrices the polar introduced in section1. It turns out that we can always representation z = r eiθ of a . The Her- find a perturbation ∆A such that A + ∆A is singular mitian polar factor H is also known as the matrix abso- and k∆Ak/kAk = κ(A)−1. It follows that the relative lute value, |A|, and is much studied in matrix analysis distance to singularity and . d(A) = min { k∆Ak/kAk : A + ∆A is singular } (2) One reason for the importance of the polar decom- −1 position is that it provides an optimal way to orthogo- is given by d(A) = κ(A) . This reciprocal relation nalize a matrix: a result of Fan and Hoffman (1955) says between problem conditioning and the distance to a that U is the nearest matrix with orthonormal columns singular problem (one with an infinite condition num- to A in any unitarily invariant norm (a unitarily invari- ber) is common to a variety of problems in linear alge- ant norm is one with the property that kUAVk = kAk bra and , as shown by James Demmel in for any unitary U and V; the 2-norm and the Frobe- the 1980s. nius norm are particular examples). In various appli- We may want a more refined test for whether A + ∆A cations a matrix A ∈ Rn×n that should be orthogonal is nonsingular. To obtain one we will need to make drifts from because of rounding or other some assumptions about the perturbation. Suppose ∗ errors; replacing it by the orthogonal polar factor U is that ∆A has rank 1: ∆A = xy , for some vectors x and then a good strategy. y. From the analysis above we know that A + ∆A will −1 −1 ∗ The polar decomposition also solves the orthogonal be nonsingular if A ∆A = A xy has no eigenvalue Procrustes problem, for A, B ∈ Cm×n, equal to −1. Using the fact that the nonzero eigenvalues of AB are the same as those of BA for any conformable min kA − BQk : Q ∈ Cn×n,Q∗Q = I , F matrices A and B, we see that the nonzero eigenvalues for which any solution Q is a unitary polar factor of of (A−1x)y∗ are the same as those of y∗A−1x. Hence B∗A. This problem comes from factor analysis and mul- A + xy∗ is nonsingular as long as y∗A−1x 6= −1. tidimensional scaling in , where the aim is to Now that we know when A + xy∗ is nonsingular we see whether two data sets A and B are the same up to might ask if there is an explicit formula for the inverse. an orthogonal transformation. Since A + xy∗ = A(I + A−1xy∗) we can take A = I Either of the SVD and the polar decomposition can without loss of generality. So we are looking for the be derived, or computed, from the other. Histori- inverse of B = I + xy∗. One way to find it is to guess cally, the SVD came first (Beltrami, in 1873), with the that B−1 = I + θxy∗ for some scalar θ and equate the polar decomposition three decades behind (Autonne, product with B to I, to obtain θ(1 + y∗x) + 1 = 0. Thus in 1902). (I + xy∗)−1 = I − xy∗/(1 + y∗x). The corresponding formula for (A + xy∗)−1 is 3 Distance to Singularity and Low-Rank (A + xy∗)−1 = A−1 − A−1xy∗A−1/(1 + y∗A−1x), Perturbations which is known as the Sherman–Morrison formula. The question commonly arises of whether a given per- This formula and its generalizations originate in the turbation of a nonsingular matrix A preserves nonsin- 1940s and have been rediscovered many times. The gularity. In a sense, this question is trivial. Recalling corresponding formula for a rank p perturbation is that a is nonsingular when all its - the Sherman–Morrison–Woodbury formula: for U,V ∈ values are nonzero, and that the product of two matri- Cn×p, ces is nonsingular unless one of them is singular, from (A + UV ∗)−1 = A−1 − A−1U(I + V ∗A−1U)−1V ∗A−1. A + ∆A = A(I + A−1∆A) we see that A + ∆A is non- singular as long as A−1∆A has no eigenvalue equal to Important applications of these formulae are in opti- −1. However, this is not an easy condition to check, mization, where rank-1 or rank-2 updates are made to and in practice we may not know ∆A but only a bound Hessian approximations in quasi-Newton methods and for its norm. Since any norm of a matrix exceeds the to matrices in the simplex method. More gener- modulus of every eigenvalue, a sufficient condition for ally, the task of updating the solution to a problem after A + ∆A to be nonsingular is that kA−1∆Ak < 1, which a coefficient matrix has undergone a low-rank change, is certainly true if kA−1kk∆Ak < 1. This condition can or has had a row or column added or removed, arises in 5 many applications, including , where 5 Eigenvalue Problems new data is continually being received and old data is The eigenvalue problem Ax = λx for a square matrix discarded. A ∈ Cn×n, which seeks an eigenvalue λ ∈ C and an The minimal distance in the definition (2) of the dis- eigenvector x 6= 0, arises in many forms. Depending on tance to singularity d(A) can be shown to be attained the application we may want all the eigenvalues or just for a rank-1 matrix ∆A. Rank-1 matrices often feature a subset, such as the 10 that have the largest real part, in the solutions of matrix optimization problems. and eigenvectors may or may not be required as well. Whether the problem is Hermitian or non-Hermitian 4 Computational Cost changes its character greatly. In particular, while a Her- mitian matrix has real eigenvalues and a linearly inde- In order to compare competing methods and predict pendent set of n eigenvectors that can be taken to their practical efficiency we need to know their com- be orthonormal, the eigenvalues of a non-Hermitian putational cost. Traditionally, computational cost has matrix can be anywhere in the complex plane and there been measured by counting the number of scalar arith- may not be a set of eigenvectors that spans Cn. metic operations and retaining only the highest order terms in the total. For example, using GE we can solve 5.1 Bounds and Localization a system of n linear equations in n unknowns with One of the first questions to ask is whether we can find n3/3 + O(n2) additions, n3/3 + O(n2) multiplications, a finite region containing the eigenvalues. The answer and O(n) divisions. This is typically summarized as is yes, because Ax = λx implies |λ|kxk = kAxk 6 2n3/3 flops, where a flop denotes any of the scalar kAkkxk, and hence |λ| 6 kAk. So all the eigenvalues lie operations +, −, ∗,/. Most standard problems involv- in a disc of radius kAk about the origin. More refined ing n×n matrices can be solved with a cost of order n3 bounds are provided by Gershgorin’s theorem. flops or less, so the interest is in the exponent (1, 2, or 3) and the constant of the dominant term. However, the Theorem 1 (Gershgorin’s theorem, 1931). The eigen- costs of moving data around a computer’s hierarchi- values of A ∈ Cn×n lie in the union of the n discs in cal memory and the costs of communicating between the complex plane different processors on a multiprocessor system can ( ) X be equally important. Simply counting flops does not Di = z ∈ C : |z − aii| 6 |aij | , i = 1: n. therefore necessarily give a good guide to performance j6=i in practice. An extension of the theorem says that if k discs form Seemingly trivial problems can offer interesting chal- a connected region that is isolated from the other discs lenges as regards minimizing arithmetic costs. For then there are precisely k eigenvalues in this region. matrices A, B, and C of any dimensions such that the The Gershgorin discs for the matrix product ABC is defined, how should we compute the  −1 1/3 1/3 1/3  product? The associative law for  −   3/2 2 0 0    (3) tells us that (AB)C = A(BC), but this mathematical  1/2 0 3 1/4  equivalence is not a computational one. To see why, 1 0 −1 6 n note that for three vectors a, b, c ∈ R we can write are shown in figure1. We can conclude that there is ∗ ∗ one eigenvalue in the disc centered at 3, one in the disc (ab ) c = a (b c) . | {z } | {z } centered at 6, and two in the union of the other two n×n 1×1 discs. Evaluation of the left-hand side requires O(n2) flops, as Gershgorin’s theorem is most useful for matrices there is an ab∗ and then a matrix–vector that are close to diagonal, such as those eventually pro- product to evaluate, while evaluation of the right-hand duced by the Jacobi for eigenvalues of side requires just O(n) flops, as it involves only vec- Hermitian matrices. Improved estimates can be sought tor operations: an inner product and a vector scaling. by applying Gershgorin’s theorem to a matrix D−1AD One should always be alert for opportunities to use the similar to A, with the D chosen in an associative law to save computational effort. attempt to isolate and shrink the discs. Many variants 6

2 20 1 10 10 0 • • • • 0 0 −1 −10 −2 −10 −4 −3 −2 −1 0 1 2 3 4 5 6 7 8 −20 Figure 1 Gershgorin discs for the matrix in (3); the −20 −10 0 0 20 40 eigenvalues are marked as solid dots. Figure 2 Fields of values for a pentadiagonal Toeplitz of Gershgorin’s theorem exist with discs replaced by matrix (left) and a (right), both of dimen- sion 32. The eigenvalues are denoted by crosses. other shapes. The spectral radius ρ(A) (the largest absolute value of any eigenvalue of A) satisfies ρ(A) 6 kAk, as shown y, respectively, then there is an eigenvalue λ + ∆λ of above, but this inequality can be arbitrarily weak, as ∗ ∗ 2 h i A+∆A such that ∆λ = y ∆Ax/(y x)+O(k∆Ak ) and 1 θ | |  the matrix 0 1 shows for θ 1. It is natural to ask so whether there are any sharper relations between the kyk kxk |∆λ| 2 2 k∆Ak + O(k∆Ak2). spectral radius and norms. One answer is the equality 6 |y∗x| k 1/k ρ(A) = lim kA k . (4) The term kyk kxk /|y∗x| can be shown to be an k→∞ 2 2 (absolute) condition number for λ. It is at least 1 and Another is the result that given any ε > 0 there is a tends to infinity as y and x approach orthogonality norm such that kAk 6 ρ(A) + ε; however, the norm depends on A. This result can be used to give a proof of (which can never exactly be achieved for simple λ), so the fact, discussed in the article on the Jordan canonical λ can be very ill conditioned. However if A is Hermitian form, that the powers of A converge to zero if ρ(A) < 1. then we can take y = x and the bound simplifies to 2 The field of values, also known as the numerical |∆λ| 6 k∆Ak + O(k∆Ak ), so all the eigenvalues of a range, is a tool that can be used for localization and Hermitian matrix are perfectly conditioned. many other purposes. It is defined for A ∈ Cn×n by Much research has been done to obtain eigenvalue  z∗Az  perturbation bounds under both weaker and stronger F(A) = : 0 6= z ∈ Cn . z∗z assumptions about the problem. Suppose we drop the The set F(A) is compact and convex (a nontrivial prop- requirement that λ is simple. Consider the matrix and erty proved by Toeplitz and Hausdorff) and it contains perturbation all the eigenvalues of A. For normal matrices it is the  0 1 0   0 0 0  convex hull of the eigenvalues. The normal matrices A A =  0 0 1  , ∆A =  0 0 0  . ∗ ∗     are those for which AA = A A, and they include the 0 0 0 ε 0 0 Hermitian, the skew-Hermitian, and the unitary matri- The eigenvalues of A are all zero and those of A + ∆A ces. For a Hermitian matrix F(A) is a segment of the real are the third roots of ε. The change in the eigenvalue axis while for a skew-Hermitian matrix it is a segment is proportional not to ε but to a fractional power of ε. of the imaginary axis. Figure2 illustrates two fields of values, the second of which is the convex hull of the In general, the sensitivity of an eigenvalue depends on eigenvalues because a circulant matrix is normal. the Jordan structure for that eigenvalue.

5.2 Eigenvalue Sensitivity 5.3 Companion Matrices and the Characteristic Polynomial If A is perturbed how much do its eigenvalues change? This question is easy to answer for a simple eigenvalue The eigenvalues of a matrix A are the roots of its charac- λ—one that has algebraic multiplicity 1. We need the teristic polynomial, det(λI −A). Conversely, associated notion of a left eigenvector of A corresponding to λ, with the polynomial which is a nonzero vector y such that y∗A = λy∗. n n−1 If λ is simple with right and left eigenvectors x and p(λ) = λ − an−1λ − · · · − a0 7

∗ ∗ is the companion matrix Special cases are λn = minx6=0 x Ax/(x x) and λ1 = ∗ ∗  an−1 an−2 ...... a0  maxx6=0 x Ax/(x x).  1 0 ...... 0  Taking x to be a unit vector ei in the previous formula    .   .  for λ1 gives λ1 > aii for all i. This inequality is just C =  0 1 . 0  ,  . .  the first in a sequence of inequalities relating sums of  . .. .   . . 0 .  eigenvalues to sums of diagonal elements, obtained by 0 ...... 1 0 Schur in 1923: and the eigenvalues of C are the roots of p. k k X X This relation means that the roots of a polynomial λi > aeii, k = 1: n, (5) can be found by computing the eigenvalues of an n × n i=1 i=1 matrix, and this approach is used by some computer where {aeii} is the set of diagonal elements of A codes, for example the roots of MATLAB. arranged in decreasing order: ae11 > ··· > aenn. There While standard eigenvalue algorithms do not exploit is equality for k = n, since both sides equal trace(A). the structure of C, this approach has proved competi- These inequalities say that the vector [λ1, . . . , λn] of tive with specialist polynomial root-finding algorithms. eigenvalues majorizes the vector [ae11,..., aenn] of diag- Another use for the relation is to obtain bounds for onal elements. roots of polynomials from bounds for matrix eigen- In general there is no useful formula for the eigen- values, and vice versa. values of a sum A + B of Hermitian matrices. How- Companion matrices have many interesting proper- ever, the Courant–Fischer theorem yields the upper and ties. For example, any nonderogatory n × n matrix lower bounds is similar to a companion matrix. Companion matri- λk(A) + λn(B) 6 λk(A + B) 6 λk(A) + λ1(B), ces therefore have featured strongly in matrix analysis | + − | and also in control theory. However, similarity trans- from which it follows that λk(A B) λk(A) 6 | | | | = k k formations to companion form are little used in prac- max( λn(B) , λ1(B) ) B 2. The latter inequality tice because of problems with ill conditioning and again shows that the eigenvalues of a Hermitian matrix numerical instability. are well conditioned under perturbation. Returning to the characteristic polynomial, p(λ) = The Cauchy interlace theorem has a different flavor. It n n−1 relates the eigenvalues of successive leading principal det(λI − A) = λ − an−1λ − · · · − a0, we know that submatrices Ak = A(1: k, 1: k) by p(λi) = 0 for every eigenvalue λi of A. The Cayley– n n−1 Hamilton theorem says that p(A) = A − an−1A − λk+1(Ak+1) 6 λk(Ak) 6 λk(Ak+1) · · · − a0I = 0 (which cannot be obtained simply by 6 ··· 6 λ2(Ak+1) 6 λ1(Ak) 6 λ1(Ak+1) putting “λ = A” in the previous expression!). Hence the nth power of A, and inductively all higher powers, are for k = 1: n − 1, showing that the eigenvalues of Ak expressible as a of I, A, ... , An−1. interlace those of Ak+1. Moreover, if A is nonsingular then from A−1p(A) = 0 it In 1962 Alfred Horn made a conjecture that a cer- follows that A−1 can also be written as a polynomial in tain set of linear inequalities involving real numbers A of degree at most n − 1. These relations are not use- αi, βi, and γi, i = 1: n, is necessary and sufficient for ful for practical computation because the coefficients the existence of n × n Hermitian matrices A, B, and C with eigenvalues the αi, βi, and γi, respectively, such ai can vary tremendously in magnitude and it is not possible to compute them to high relative accuracy. that C = A+B. The conjecture was open for many years but was finally proved to be true in papers published by 5.4 Eigenvalue Inequalities for Hermitian Matrices Klyachko in 1998 and Knutson and Tao in 1999, which exploit deep connections with algebraic , rep- The eigenvalues of Hermitian matrices A ∈ Cn×n, resentations of Lie groups, and quantum cohomology. which in this section we order λn 6 ··· 6 λ1, satisfy many beautiful inequalities. Among the most impor- tant are those in the Courant–Fischer theorem (1905), 5.5 Solving the Non-Hermitian Eigenproblem which states that every eigenvalue is the solution of a min-max problem over a suitable subspace S of Cn: The simplest method for computing eigenvalues, the x∗Ax power method, computes just one: the largest in mod- = λi min max ∗ . ulus. It comprises repeated multiplication of a starting dim(S)=n−i+1 06=x∈S x x 8

∗ vector x by A. Since the resulting sequence is liable to check that Hk+1 = Qk HkQk, so the QR iteration carries overflow or underflow in floating-point arithmetic one out a sequence of unitary similarity transformations. normalizes the vector after each iteration. Therefore Why the QR iteration works is not obvious but can one step of the power method has the form x ← Ax, be elegantly explained by analyzing the subspaces −1 x ← ν x, where ν = xj with |xj | = maxi |xi|. If A spanned by the columns of Qk. To produce a practi- has a unique eigenvalue λ of largest modulus and the cal and efficient various refinements of the starting vector has a component in the direction of the iteration are needed, which include corresponding eigenvector then ν converges to λ and x converges to the corresponding eigenvector. The power • deflation, whereby when an element on the first −1 method is most often applied to (A − µI) , where µ is subdiagonal of Hk becomes small, that element is an approximation to an eigenvalue of interest. In this set to zero and the problem is split into two smaller form it is known as and convergence is problems that are solved independently, to the eigenvalue closest to µ. We now turn to methods • a double shift technique for real A that allows that compute all the eigenvalues. two QR steps with complex conjugate shifts to be Since similarities X−1AX preserve the eigenvalues carried out entirely in real arithmetic and gives and change the eigenvectors in a controlled way, car- convergence to the real Schur form, rying out a sequence of similarity transformations to • a multishift technique for including m different reduce A to a simpler form is a natural way to tackle the shifts in a single QR iteration. eigenproblem. Some early methods used nonunitary X, but such transformations are now avoided because of A proof of convergence is lacking for all current shift numerical instability when X is ill conditioned. Since strategies. Implementations introduce a random shift the 1960s the focus has been on using unitary similar- when convergence appears to be stagnating. The QR ities to compute the Schur decomposition A = QTQ∗, algorithm works very well in practice and continues where Q is unitary and T is upper triangular. The diag- to be the method of choice for the non-Hermitian onal entries of T are the eigenvalues of A, and they can eigenproblem. be made to appear in any order by appropriate choice of Q. The first k columns of Q span an invariant subspace 5.6 Solving the Hermitian Eigenproblem corresponding to the eigenvalues t11, . . . , tkk. Eigen- vectors can be obtained by solving triangular systems The eigenvalue problem for Hermitian matrices is eas- involving T . ier to solve than that for non-Hermitian matrices and For some matrices the Schur factor T is diagonal; the range of available numerical methods is much these are precisely the normal matrices defined in sec- wider. tion 5.1. The real Schur decomposition contains only To solve the complete Hermitian eigenproblem we real matrices when A is real: A = QRQT , where Q is need to compute the spectral decomposition A = ∗ orthogonal and R is real upper quasi-triangular, which QDQ , where D = diag(λi) contains the eigenvalues means that R is upper triangular except for 2×2 blocks and the columns of the unitary matrix Q are the corre- on the diagonal corresponding to complex conjugate sponding eigenvectors. Many methods begin by unitary ∗ eigenvalues. reduction to tridiagonal form T = U AU, where tij = 0 The standard algorithm for solving the non- for |i − j| > 1 and the unitary matrix U is constructed Hermitian eigenproblem is the QR algorithm, which as a product of Householder matrices. The eigenvalue was proposed independently by John Francis and Vera problem for T is much simpler, though still nontriv- Kublanovskaya in 1961. The matrix A ∈ Cn×n is ial. The most widely used method is the QR algorithm, first unitarily reduced to upper Hessenberg form H = which has the same form as in the non-Hermitian case ∗ U AU (hij = 0 for i > j + 1), with U a product of but with the upper Hessenberg Hk replaced by the Her- Householder matrices. The QR iteration constructs a mitian tridiagonal Tk and the shifts chosen to acceler- sequence of upper Hessenberg matrices beginning with ate the convergence of Tk to diagonal form. The Her-

H1 = H defined by Hk − µkI =: QkRk (QR factorization, mitian QR algorithm with appropriate shifts has been computed using Givens rotations), Hk+1 := RkQk +µkI, proved to converge at a cubic rate. where the µk are shifts chosen to accelerate the con- Another method for solving the Hermitian tridiag- vergence of Hk to upper triangular form. It is easy to onal eigenproblem is the divide and conquer method. 9

This method decouples T in the form to x, that is, N(x). The LDL∗ factors of a tridiagonal  T 0  matrix can be computed in O(n) flops, so this bisec- T = 11 + αvv∗, 0 T22 tion process is efficient. An alternative approach can be where only the trailing diagonal element of T11 and the built by using properties of Sturm sequences, which are leading diagonal element of T22 differ from the corre- sequences comprising the characteristic polynomials sponding elements of T and hence the vector v has of leading principal submatrices of T − λI. only two nonzero elements. The eigensystems of T11 and T22 are found by applying the method recursively, 5.7 Computing the SVD ∗ ∗ yielding T11 = Q1Λ1Q and T22 = Q2Λ2Q . Then 1 2 m×n  ∗  For a rectangular matrix A ∈ C the eigenvalues of Q1Λ1Q1 0 ∗ h 0 A i T = ∗ + αvv the Hermitian matrix A∗ 0 of dimension m + n are 0 Q2Λ2Q 2 plus and minus the nonzero singular values of A along = diag diag Λ Λ + ∗ diag ∗ (Q1,Q2) ( 1, 2) αveve (Q1,Q2) , with m + n − 2 min(m, n) zeros. Hence the SVD can ∗ where ve = diag(Q1,Q2) v. The eigensystem of a rank- be computed via the eigendecomposition of this larger 1 perturbed diagonal matrix D + ρzz∗ can be found by matrix. However, this would be inefficient, and instead solving the secular equation obtained by equating the one uses algorithms that work directly on A and are characteristic polynomial to zero: analogues of the algorithms for Hermitian matrices. n 2 X |zj | The standard approach is to reduce A to bidiagonal f (λ) = 1 + ρ = 0. d − λ form B by Householder transformations applied on the j=1 jj left and the right and then to apply an adaptation of the Putting the pieces together yields the overall eigende- QR algorithm that works on the bidiagonal factor (and composition. implicitly applies the QR algorithm to the tridiagonal Other methods are suitable for computing just a por- matrix B∗B). tion of the spectrum. Suppose we want to compute the kth smallest eigenvalue of T and that we can some- 5.8 Generalized Eigenproblems how compute the integer N(x) equal to the number of eigenvalues of T that are less than or equal to x. The generalized eigenvalue problem (GEP) Ax = λBx, Then we can apply the bisection method to N(x) to with A, B ∈ Cn×n, can be converted into a standard find the point where N(x) jumps from k − 1 to k. eigenvalue problem if B (say) is nonsingular: B−1Ax = We can compute N(x) by making use of the following λx. However, such a transformation is inadvisable result about the inertia of a Hermitian matrix, defined numerically unless B is very well conditioned. If A and B by inertia(A) = (ν, ζ, π), where ν is the number of neg- have a common null vector z the problem takes on a dif- ative eigenvalues, ζ is the number of zero eigenvalues, ferent character because then (A − λB)z = 0 for any λ; and π is the number of positive eigenvalues. such a problem is called singular. We will assume that Theorem 2 (Sylvester’s inertia theorem). If A is Her- the problem is regular, so that det(A − λB) 6≡ 0. The mitian and M is nonsingular then inertia(A) = linear polynomial A − λB is sometimes called a pencil. inertia(M∗AM). It is convenient to write λ = α/β, where α and β are Sylvester’s inertia theorem says that the number not both zero, and rephrase the problem in the more of negative, zero, and positive eigenvalues does not symmetric form βAx = αBx. If x is a nonzero vector change under congruence transformations. By using GE such that Bx = 0 then, since the problem is assumed we can factorize2 T − xI = LDL∗, where D is diago- to be regular, Ax 6= 0 and so β = 0. This means that nal and L is unit lower bidiagonal (a bidiagonal matrix λ = ∞ is an eigenvalue. Infinite eigenvalues may seem is one that is both triangular and tridiagonal). Then a strange concept, but in fact they are no different in inertia(T − xI) = inertia(D), so the number of nega- most respects to finite eigenvalues. tive diagonal or zero elements of D equals the number An important special case is the definite general- of eigenvalues of T − xI less than or equal to 0, which ized eigenvalue problem, in which A and B are Hermi- is the number of eigenvalues of T less than or equal tian and B (say) is positive definite. If B = R∗R is a Cholesky factorization then Ax = λBx can be rewrit- ten as −∗ −1 · = , which is a standard eigen- 2. The factorization may not exist, but if it does not we can simply R AR Rx λRx −∗ − perturb T slightly and try again without any loss of numerical stability. problem for the Hermitian matrix C = R AR 1. This 10 argument shows that the eigenvalues of a definite prob- The standard approach for numerical solution of the lem are all real. Definite generalized eigenvalue prob- QEP mimics the conversion of the scalar polynomial lems arise in many physical situations where an energy root problem into a matrix eigenproblem described in minimization principle is at work, such as in problems section 5.3. From the relation in and .  A A   A 0   λx  L(λ)z ≡ 1 0 + λ 2 A generalization of the QR algorithm called the QZ I 0 0 −I x algorithm computes a generalization to two matrices  Q(λ)x  = of the Schur decomposition: Q∗AZ = T , Q∗BZ = S, 0 where Q and Z are unitary and T and S are upper tri- we see that the eigenvalues of the quadratic Q are the angular. The generalized Schur decomposition yields eigenvalues of the 2n×2n linear polynomial L(λ). This the eigenvalues as the ratios tii/sii and enables eigen- is an example of an exact linearization process—thanks vectors to be computed by substitution. to the hidden λ in the eigenvector! The eigenvalues of L The quadratic eigenvalue problem (QEP) Q(λ)x = can be found using the QZ algorithm. The eigenvectors 2 n×n  λx  (λ A2 + λA1 + A0)x = 0, where Ai ∈ C , i = 0: 2, of L have the form z = x , where x is an eigenvector arises most commonly in the dynamic analysis of struc- of Q, and so x can be obtained from either the first n tures when the finite element method is used to dis- (if λ 6= 0) or the last n components of z. cretize the original PDE into a system of second-order

ODEs A2q(t)¨ + A1q(t)˙ + A0q(t) = f (t). Here, the Ai 6 Sparse Linear Systems are usually Hermitian (though A is skew-Hermitian in 1 For linear systems coming from of dif- gyroscopic systems) and positive (semi)definite. Anal- ferential equations it is common that A is banded, ogously to the GEP, the QEP is said to be regular if that is, the nonzero elements lie in a band about the det(Q(λ)) 6≡ 0. The quadratic problem differs funda- main diagonal. An extreme case is a tridiagonal matrix, mentally from the linear GEP because a regular problem of which the classic example is the second-difference has 2n eigenvalues, which are the roots of det(Q(λ)) = matrix, illustrated for n = 4 by 0, but at most n linearly independent eigenvectors,     and a vector may be an eigenvector for two different −2 1 0 0 4 3 2 1  1 −2 1 0    eigenvalues. For example, the QEP with   −1 1  3 6 4 2  A =   ,A = −   .      0 1 −2 1  5  2 4 6 3  2 −1 −6 0 12   Q(λ) = λ I + λ + − 1 2 3 4 2 −9 −2 14 0 0 1 2 This matrix corresponds to a centered finite difference has eigenvalues 1, 2, 3, and 4, with eigenvectors  1 , 0 approximation to a second derivative: f 00(x) ≈ (f (x +  0 ,  1 , and  1 , respectively. Moreover, there is no 1 1 1 h)−2f (x)+f (x−h))/h2. Note that A−1 is a full matrix. Schur form for three or more matrices, that is, we can- For banded matrices, GE produces banded LU factors not in general find unitary matrices U and V such that ∗ and its computational cost is proportional to n times U AiV is triangular for i = 0: 2. 2 the square of the bandwidth. Associated with the QEP is the matrix Q(X) = A2X + n×n A matrix is sparse if advantage can be taken of the A1X + A0, with X ∈ C . From the relation zero entries, because of either their number or their dis- 2 2 Q(λ) − Q(X) = A2(λ I − X ) + A1(λI − X) tribution. A banded matrix is a special case of a sparse

= (λA2 + A2X + A1)(λI − X) matrix. Sparse matrices are stored on a computer not as a square array but in a special format that records only it is clear that if we can find a matrix X such that the nonzeros and their location in the matrix. This can Q(X) = 0, known as a solvent, then we have reduced be done with three vectors: one to store the nonzero the QEP to finding the eigenvalues of X and solving entries and the other two to define the row and column one n × n GEP. For the 2 × 2 Q above there are five sol- h i indices of the elements in the first vector. vents, one of which is 3 0 . The existence and enumer- 1 2 Sparse matrices help to explain the tenet: never solve ation of solvents is nontrivial and leads into the theory a linear system Ax = b by computing x = A−1 × b. The of matrix polynomials. In general, matrix polynomials reasons for eschewing A−1 are threefold: Pk i are matrices of the form i=0 λ Ai whose elements are polynomials in a complex variable; an older term for • Computing A−1 requires three times as many flops such matrices is λ-matrices. as solving Ax = b by GE with partial pivoting. 11

• GE with partial pivoting is backward stable for solv- 0 0 −1 ing Ax = b (see section8) but solution via A is 50 50 not. 100 100 • If A is sparse, A−1 is generally dense and so requires much more storage than GE with partial pivoting. 150 150

When GE is applied to a fill-in occurs 200 200 when the row operations cause a zero entry to become 0 100 200 0 100 200 nonzero during the elimination. To minimize the stor- nz = 3137 nz = 5041 age and the computational cost, fill-in must be avoided 0 0 as much as possible. This can be done by employing 50 50 row and column interchanges to choose a suitable pivot from the active submatrix. The first such strategy was 100 100 introduced by Markowitz in 1957. At the kth stage, 150 150 (k) with cj denoting the number of nonzeros in rows (k) 200 200 k to n of column j and ri the number of nonzeros in columns k to n of row i, the Markowitz strategy 0 100 200 0 100 200 (k)  (k)  nz = 3137 nz = 3476 finds the pair (r , s) that minimizes ri − 1 cj − 1 (k) over all nonzero potential pivots aij and then takes (k) ar s as the pivot. The quantity being minimized is a Figure 3 Sparsity plots of a symmetric positive definite bound on the fill-in. In practice, the potential pivots matrix (left) and its Cholesky factor (right) for original matrix (first row) and reordered matrix (second row). nz is must be restricted to those not too much smaller in the number of nonzeros. magnitude than the partial pivot, in order to preserve numerical stability. The result of GE with Markowitz pivoting is a factorization PAQ = LU, where P and Q than unknowns (m > n), and underdetermined systems, are permutation matrices. with fewer equations than unknowns (m < n). Since in The analogue of the Markowitz strategy for Hermi- general there is no solution when m > n and there are tian positive definite matrices chooses a diagonal entry many solutions when m < n, extra conditions must (k) (k) be imposed for the problems to be well-defined. These aii as the pivot, where ri is minimal. This is the mini- mum degree algorithm, which has been very successful usually involve norms and different choices of norms in practice. Figure3 shows in the first row a sparse and are possible. We will restrict our discussion mainly to banded symmetric positive definite matrix A of dimen- the 2-norm, which is the most important case, but other sion 225 followed to the right by its Cholesky factor. choices are also of practical interest. The Cholesky factor has many more nonzeros than A. The second row shows the matrix PAP T produced by 7.1 The Problem an approximate minimum degree ordering (produced When m > n the residual r = b − Ax cannot in general by the MATLAB symamd function) and its Cholesky fac- be made zero so we try to minimize its norm. The most tor. We can see that the permutations have destroyed common choice of norm is the 2-norm, which gives the the band structure but have greatly reduced the fill-in, linear least squares problem producing a much sparser Cholesky factor. As an alternative to GE for solving sparse linear sys- min kb − Axk2. (6) x∈Cn tems one can apply iterative methods, described in sec- This choice can be motivated by statistical consider- tion9; for sufficiently large problems these are the only ations (the Gauss–Markov theorem) or by the fact that feasible methods. the square of the 2-norm is differentiable, which makes the problem explicitly solvable. Indeed by setting the 7 Overdetermined and Underdetermined gradient of kb − Axk2 to zero we obtain the normal Systems 2 equations A∗Ax = A∗b, which any solution of the least Linear systems Ax = b with a rectangular matrix squares problem must satisfy. If A has full rank then A ∈ Cm×n are very common. They break into two cat- A∗A is positive definite and so there is a unique solu- egories: overdetermined systems, with more equations tion, which can be computed by solving the normal 12 equations using Cholesky factorization. For reasons of (It is certainly not obvious that these equations have numerical stability, it is preferable to use a QR fac- a unique solution.) In the case where A is square and h i R1 + −1 torization: if A = Q 0 then the normal equations nonsingular it is easily seen that A is just A . More- + ∗ −1 ∗ reduce to the triangular system R1x = c, where c is the over, if rank(A) = n then A = (A A) A , while if first n components of Q∗b. rank(A) = m then A+ = A∗(AA∗)−1. In terms of the When A is rank deficient there are many least squares SVD (7), solutions, which vary widely in norm. A natural choice A+ = V diag(σ −1, . . . , σ −1, 0,..., 0)U ∗, is one of minimal 2-norm, and in fact there is a unique 1 r + minimal 2-norm solution, xLS , given by where r = rank(A). The formula xLS = A b holds for r all m and n, so the pseudoinverse yields the minimal X ∗ xLS = (ui b/σi)vi, 2-norm solution to both the least squares (overdeter- i=1 mined) problem Ax = b and an underdetermined sys- where tem Ax = b. The pseudoinverse has many interesting ∗ + + = A = UΣV ,U = [u1, . . . , um], V = [v1, . . . , vn] (7) properties, including (A ) A, but it is not always true that (AB)+ = B+A+. is an SVD and r = rank(A). The use of this formula in Although the pseudoinverse is a very useful theoret- practice is not straightforward because a matrix stored ical tool it is rarely necessary to compute it explicitly in floating-point arithmetic will rarely have any zero (just as for its special case the matrix inverse). singular values. Therefore r must be chosen by desig- The pseudoinverse is just one of many ways of gen- nating which singular values can be regarded as negligi- eralizing the notion of inverse to rectangular matri- ble and this choice should take account of the accuracy ces, but it is the right one for minimum 2-norm solu- with which the elements of A are known. tions to linear systems. Other generalized inverses can Another choice of least squares solution in the rank- be obtained by requiring only a subset of the four deficient case is a basic solution: one with at most r nonzeros. Such a solution can be computed via the QR Moore–Penrose conditions to hold. factorization with column pivoting. 8 Numerical Considerations 7.2 Underdetermined Systems Prior to the introduction of the first digital comput- When m < n and A has full rank, there are infinitely ers in the 1940s, numerical computations were carried many solutions to Ax = b and again it is natural to out by humans, sometimes with the aid of mechanical seek one of minimal 2-norm. There is a unique such calculators. The human involvement in a sequence of ∗ ∗ −1 solution xLS = A (AA ) b, and it is best computed calculations meant that potentially dangerous events via a QR factorization, this time of A∗. A basic solu- such as dividing by a tiny number or subtracting two tion, with m nonzeros, can alternatively be computed. numbers that agree to almost all their significant digits As a simple example, consider the problem “find two could be observed, their effect monitored, and possible  x1  numbers whose sum is 5”, that is, solve [1 1] x2 = corrective action taken—such as temporarily increas- 5. A basic solution is [5 0]T while the minimal 2- ing the precision of the calculations. On the very early norm solution is [5/2 5/2]T . Minimal 1-norm solu- intermediate results were observed on a tions to underdetermined systems are important in cathode-ray tube monitor, but this became impossible compressed sensing. as problem sizes increased (along with available com- puting power). Fears were raised in the 1940s that algo- 7.3 Pseudoinverse rithms such as GE would suffer exponential growth The analysis in the previous two subsections can be of errors as the problem dimension increased, due unified in a very elegant way by making use of the to the rapidly increasing number of arithmetic opera- Moore–Penrose pseudoinverse A+ of A ∈ Cm×n, which tions, each having its associated rounding error. These is defined as the unique X ∈ Cn×m satisfying the fears were particularly concerning given that the error Moore–Penrose conditions growth might be unseen and unsuspected. The subject of rounding error analysis grew out AXA = A, XAX = X, of the need to understand the effect on algorithms ∗ ∗ (AX) = AX, (XA) = XA. of rounding errors. The person who did the most to 13 develop the subject was James Wilkinson, whose influ- upper triangular matrix Tb such that ential papers and 1961 and 1965 books showed how ∗ Qe (A + ∆A)Qe = T,b k∆AkF 6 p(n)ukAkF , backward error analysis can be used to obtain deep insights into numerical stability. We will discuss just where Qe is some exactly unitary matrix and p(n) is a two particular examples. cubic polynomial. The computed Schur factor Qb is not Wilkinson showed that when a nonsingular linear necessarily close to Qe —which in turn is not necessarily system Ax = b is solved by GE in floating-point close to the exact Q!— but it is close to being orthogo- ∗ nal: kQb Qb −IkF p(n)u. This distinction between the arithmetic the computed solution xb satisfies 6 different Q matrices is an indication of the subtleties (A + ∆A)xb = b, k∆Ak∞ 6 p(n)ρnukAk∞. of backward error analysis. For some problems it is not Here p(n) is a cubic polynomial, the growth factor clear exactly what form of backward error result it is possible to prove while obtaining useful bounds. How- max |a(k)| i,j,k ij ever, the purpose of a backward error analysis is always ρn = > 1 maxi,j |aij | the same: either to show that an algorithm behaves in a measures the growth of elements during the elimina- numerically stable way or to shed light on how it might tion, and u is the unit roundoff. This is a backward fail to do so and to indicate what quantities should be stability result: it says that the computed solution xb monitored in order to identify potential instability. is the exact solution of a perturbed system. Ideally, we would like k∆Ak∞ 6 ukAk∞, which reflects the 9 Iterative Methods uncertainty caused by converting the elements of A to In numerical linear algebra methods can broadly be floating-point numbers. The polynomial term p(n) is divided into two classes: direct and iterative. Direct pessimistic and might be more realistically replaced by methods, such as GE, solve a problem in a fixed num- its square root. The danger term is the growth factor ρn, ber of arithmetic operations or a variable number that and the conclusion from Wilkinson’s analysis is that a in practice is fairly constant, as for the QR algorithm for pivoting strategy should aim to keep ρn small. If no eigenvalues. Iterative methods are infinite processes pivoting is done, ρn can be arbitrarily large (e.g., for  ε 1  that must be truncated at some point when the approx- A = 1 1 with 0 < ε  1, ρn ≈ 1/ε). For partial pivot- n−1 imation they provide is “good enough”. Usually, iter- ing however, it can be shown that ρn 6 2 and that ative methods do not transform the matrix in ques- this bound is attainable. In practice, ρn is almost always tion and access it only through matrix–vector products; of modest size for partial pivoting (ρn 6 50, say); why this makes them particularly attractive for large, sparse this should be so remains one of the great mysteries of matrices, where applying a direct method may not be ! practical. One of the benefits of Wilkinson’s backward error We have already seen in section 5.5 a simple iterative analysis is that it enables us to identify classes of matri- method for the eigenvalue problem: the power method. ces for which pivoting is not necessary, that is, for The stationary iterative methods are an important class which the LU factorization A = LU exists and ρn is of iterative methods for solving a nonsingular linear nicely bounded. One such class is the matrices that system Ax = b. These methods are best described in are diagonally dominant by either rows or columns, for terms of a splitting which ρn 6 2. The potential instability of GE can be attributed to A = M − N, the fact that A is premultiplied by a sequence of non- with M nonsingular. The system Ax = b can be rewrit- unitary transformations, any of which can be ill con- ten Mx = Nx + b, which suggests constructing a ditioned. Many algorithms, including Householder QR sequence {x(k)} from a given starting vector x(0) via factorization and the QR algorithm for eigenvalues, use (k+1) = (k) + (8) exclusively unitary transformations. Such algorithms Mx Nx b. are usually (but not always) backward stable, essen- Different choices of M and N yield different methods. tially because unitary transformations do not magnify The aim is to choose M in such a way that it is inexpen- errors: kUAVk = kAk for any unitary U and V for the sive to solve (8) while M is a good enough approxima- 2-norm and the Frobenius norm. As an example, the QR tion to A that convergence is fast. It is easy to analyze algorithm applied to A ∈ Cn×n produces a computed convergence. Denote by e(k) = x(k) − x the error in the 14

kth iterate. Subtracting Mx = Nx + b from (8) gives where M1z = d1 and M2z = d2 are easy to solve. In (k+1) (k) M(x − x) = N(x − x), so this case it is natural to take W = diag(M1,M2) as the e(k+1) = M−1Ne(k) = · · · = (M−1N)k+1e(0). (9) . When A is Hermitian positive definite the preconditioned system is written in a way that pre- If ρ(M−1N) < 1 then (M−1N)k → 0 as k → ∞ (see serves the structure. For example, for the Jacobi pre- Jordan ) and so x(k) converges to x, at conditioner, D = diag(A), the preconditioned system a linear rate. In practice, for convergence in a reason- would be written D−1/2AD−1/2x = be, where x = D1/2x able number of iterations we need ρ(M−1N) to be suf- e e and be = D−1/2b. Here, the matrix D−1/2AD−1/2 has unit ficiently less than 1 and the powers of M−1N should diagonal and off-diagonal elements lying between −1 not grow too large initially before eventually decaying; and 1. in other words, M−1N must not be too nonnormal. Three standard choices of splitting are, with D = The most powerful iterative methods for linear sys- diag(A) and L and U denoting the strictly lower and tems Ax = b are the Krylov methods. In these methods (k) strictly upper triangular parts of A, respectively, each iterate x is chosen from the shifted subspace (0) (0) x + Kk(A, r ) where • M = D, N = −(L + U): Jacobi iteration; K (0) = { (0) (0) k−1 (0)} • M = D + L, N = −U: Gauss–Seidel iteration; k(A, r ) span r , Ar ,...,A r 1 1−ω • = + = − ∈ (k) M ω D L, N ω D U, where ω (0, 2) is a of dimension k, with r = is a relaxation parameter: successive overrelaxation b − Ax(k). Different strategies for choosing approxi- (SOR) iteration. mations from within the Krylov subspaces yield dif- ferent methods. For example, the conjugate gradient Sufficient conditions for convergence are that A is strictly diagonally dominant by rows for the (CG, for Hermitian positive definite A) and iteration and that A is symmetric positive definite for the full orthogonalization method (FOM, for general A) (k) the Gauss–Seidel iteration. How to choose ω so that make the residual r orthogonal to the Krylov sub- 0 −1 space K (A, r ( )), while the minimal residual method ρ(M N|ω) is minimized for the SOR iteration was k elucidated in the landmark 1950 PhD thesis of David (MINRES, for Hermitian A) and the generalized min- Young. imal residual method (GMRES, for general A) mini- The Google PageRank algorithm, which underlies mize the 2-norm of the residual over all vectors in the Google’s ordering of search results, can be interpreted Krylov subspace. How to compute the vectors defined as an application of the Jacobi iteration to a certain lin- in these ways is nontrivial. It turns out that CG can ear system involving the adjacency matrix of the graph be implemented with a recurrence requiring just one corresponding to the whole world wide web. However, matrix–vector multiplication and three inner products the most common use of stationary iterative methods per iteration, and MINRES is just a little more expen- is as within other iterative methods. sive. GMRES, being applicable to non-Hermitian matri- The aim of preconditioning is to convert a given lin- ces, is significantly more expensive, and it is also much ear system Ax = b into one that can be solved more harder to analyze its convergence behavior. For general cheaply by a particular iterative method. The basic idea matrices there are alternatives to GMRES that employ is to use a nonsingular matrix W to transform the sys- short recurrences. We mention just BiCGSTAB, which tem to (W −1A)x = W −1b in such a way that (a) the pre- has the distinction that the 1992 paper by Henk van conditioned system can be solved in fewer iterations der Vorst that introduced it was the most-cited paper than the original system and (b) matrix–vector multi- in mathematics of the 1990s. −1 plications with W A (which require the solution of a Theoretically, Krylov methods converge in at most linear system with coefficient matrix W ) are not signif- n iterations for a system of dimension n. However, in icantly more expensive than matrix–vector multiplica- practical computation rounding errors intervene and tions with A. In general, this is a difficult or impossible the methods behave as truly iterative methods not task, but in many applications the matrix A has struc- having finite termination. Since n is potentially huge, ture that can be exploited. For example, many elliptic a Krylov method would not be used unless a good PDE problems lead to a positive definite matrix A of the approximate solution was obtained in many fewer than form   n iterations, and preconditioning plays a crucial role M1 F A = T , here. Available error bounds for a method help to guide F M2 15

the choice of preconditioner, but care is needed in inter- D = diag(λi) containing the eigenvalues on its diag- preting the bounds. To illustrate this, consider the CG onal. In many respects, normal matrices have very pre- k k method for Ax = b, where A is Hermitian positive defi- dictable behavior. For example, kA k2 = ρ(A) and ∗ 1/2 tA α(tA) nite. In the A-norm, kzkA = (z Az) , the error on the k e k2 = e , where the spectral abscissa α(tA) is kth step satisfies the largest real part of any eigenvalue of tA. However, !k κ (A)1/2 − 1 matrices that arise in practice are often very nonnor- kx − x(k)k 2kx − x(0)k 2 , A 6 A 1/2 mal. The adjective “very” can be quantified in various κ2(A) + 1 ways, of which one is the Frobenius norm of the strictly where κ (A) = kAk kA−1k . If we can precondition A 2 2 2 upper triangular part of the upper triangular matrix T so that its 2-norm condition number is very close to 1 in the Schur decomposition A = QTQ∗. For example, then fast convergence is guaranteed. However, another h i the matrix t11 θ is nonnormal for θ 6= 0 and grows result says that if A has k distinct eigenvalues then 0 t22 increasingly nonnormal as |θ| increases. CG converges in at most k iterations. Therefore a bet- Consider the moderately nonnormal matrix ter approach might be to choose the preconditioner so that the eigenvalues of the preconditioned matrix are  −0.97 25  A = . (10) clustered into a small number of groups. 0 −0.3 Another important class of iterative methods is While the powers of A ultimately decay to zero, since multigrid methods, which work on a hierarchy of grids ρ(A) = 0.97 < 1, we see from figure4 that initially they that come from a discretization of an underlying PDE increase in norm. Likewise, since α(A) = −0.3 < 0 the (geometric multigrid) or are constructed artificially tA norm k e k2 tends to zero as t → ∞, but figure4 shows from a given matrix (algebraic multigrid). that there is an initial hump in the plot. In station- An important practical issue is how to terminate ary iterations the hump caused by a nonnormal iter- an iteration. Popular approaches are to stop when the ation matrix M−1N can delay convergence, as is clear residual r (k) = b − Ax(k) (suitably scaled) is small or from (9). In finite precision arithmetic it can even hap- when an estimate of the error x −x(k) is small. Compli- pen that, for a sufficiently large hump, rounding errors cating factors include the fact that the preconditioner cause the norms of the powers to plateau at the hump can change the norm and a possible desire to match the level and never actually converge to zero. error in the iterations with the discretization error in How can we predict the shape of the curves in fig- the PDE from which the linear system might have come ure4? Let us concentrate on kAkk . Initially it grows (as there is no point solving the system to greater accu- 2 k kk k racy than the data warrants). Research in recent years like A 2 and ultimately it decays like ρ(A) , the decay has led to good understanding of these issues. rate following from (4). The height of the hump is The ideas of Krylov methods and preconditioners can related to pseudospectra, which have been popularized be applied to problems other than linear systems. A by Nick Trefethen. n×n popular Krylov method for solving the least squares The ε- of A ∈ C is defined, for a problem (6) is LSQR, which is mathematically equiva- given ε > 0, to be the set lent to applying CG to the normal equations. In large- Λε(A) = { z ∈ C : z is an eigenvalue of A + E scale eigenvalue problems only a few eigenpairs are usually required. A number of methods project the for some E with kEk2 < ε }, (11) original matrix onto a Krylov subspace and then solve a and it can also be represented, in terms of the resolvent smaller eigenvalue problem. These include the Lanczos (zI − A)−1, as method for Hermitian matrices and the Arnoldi method −1 −1 for general matrices. Also of much current research Λε(A) = { z ∈ C : k(zI − A) k2 > ε }. interest are rational Krylov methods based on rational The 0.001-pseudospectrum, for example, tells us the generalizations of Krylov subspaces. uncertainty in the eigenvalues of A if the elements are known only to three decimal places. Pseudospectra pro- 10 Nonnormality and Pseudospectra vide much insight into the effects of nonnormality of Normal matrices A ∈ Cn×n (defined in section 5.1) matrices and (with an appropriate extension of the def- have the property that they are unitarily diagonaliz- inition) linear operators. For nonnormal matrices the able: A = QDQ∗ for some unitary Q and diagonal pseudospectra are much bigger than a perturbation of 16 the spectrum by ε. It can be shown that for any ε > 0, 3. there is a positive vector x such that Ax = ρ(A)x, ρ (A) − 1 ρ (A)k+1 4. ρ(A) is an eigenvalue of algebraic multiplicity 1. sup kAkk ε , kAkk ε , > ε 6 ε k>0 To illustrate the theorem consider the following two where the pseudospectral radius ρε(A) = max{ |λ| : irreducible matrices and their eigenvalues: −2 λ ∈ Λε(A) }. For A in (10) and ε = 10 these inequal-   8 1 6 √ ities give an upper bound of 230 for kA3k and a lower   A =  3 5 7  , Λ(A) = {15, ±2 6}, bound of 23 for sup kAkk, and figure5 plots the k>0 4 9 2 corresponding ε-pseudospectrum.  0 0 6  √ =  1  =  1 − ± B  2 0 0  , Λ(B) 1, 2 ( 1 3i) . 11 Structured Matrices 1 0 3 0 In a wide variety of applications the matrices have The Perron–Frobenius theorem correctly tells us that a special structure. The matrix elements might form ρ(A) = 15 is a distinct eigenvalue of A, and that it has a pattern, as for a Toeplitz matrix or a Hamiltonian a corresponding positive eigenvector, which is known matrix, the matrix may satisfy a nonlinear equation as the Perron vector. The Perron vector of A is the vec- such as A∗ΣA = Σ, where Σ = diag(±1), which yields tor of all ones, as A forms a magic square and ρ(A) is the pseudo-unitary matrices A, or the submatrices may the magic sum! The Perron vector of B, which is both a satisfy certain rank conditions (as for quasisepara- Leslie matrix and a companion matrix, is [6 3 1]T . There ble matrices). We discuss here two of the oldest and is one notable difference between A and B: for A, ρ(A) most studied classes of structured matrices, both of exceeds the other eigenvalues in modulus, but all three which were historically important in the analysis of eigenvalues of B have modulus 1. In fact, Perron’s orig- iterative methods for linear systems arising from the inal version of Theorem3 says that if A has all positive discretization of differential equations. elements then ρ(A) is not only an eigenvalue of A but is larger in modulus than every other eigenvalue. Note 11.1 Nonnegative Matrices that B3 = I, which provides another way to see that the eigenvalues of B all have modulus 1. A nonnegative matrix is a real matrix all of whose We saw in the section9 that the spectral radius plays entries are nonnegative. A number of important classes an important role in the convergence of stationary iter- of matrices are subsets of the nonnegative matrices. ative methods, through ρ(M−1N), where A = M − N is These include adjacency matrices, stochastic matrices, a splitting. In comparing different splittings we can use and Leslie matrices (used in population modeling). Non- the result that for A, B ∈ Rn×n, with |A| denoting the negative matrices have a large body of theory, which matrix (|aij |), originates with Perron in 1907 and Frobenius in 1908. To state the celebrated Perron–Frobenius theorem |aij | 6 bij ∀i, j ⇒ ρ(A) 6 ρ(|A|) 6 ρ(B). n×n we need the definition that A ∈ R with n > 2 is reducible if there is a permutation matrix P such that  A A  11.2 M-Matrices P T AP = 11 12 , 0 A22 A ∈ Rn×n is an M-matrix if it can be written in the form where A11 and A22 are square, nonempty submatrices, A = sI − B, where B is nonnegative and s > ρ(B). M- and it is irreducible if it is not reducible. A matrix with matrices arise in many applications, a classic one being positive entries is trivially irreducible. A useful char- Leontief’s input–output models in . acterization is that A is irreducible if and only if the The special pattern of an M-matrix—positive directed graph associated with A (which has n vertices, diagonal elements and nonpositive off-diagonal with an an edge connecting the ith vertex to the jth elements—combines with the spectral radius condi- vertex if aij 6= 0) is strongly connected. tion to give many interesting characterizations and Theorem 3 (Perron–Frobenius). If A ∈ Rn×n is nonneg- properties. For example, a nonsingular matrix A with ative and irreducible then nonpositive off-diagonal elements is an M-matrix if and only if A−1 is nonnegative. Another characterization, 1. ρ(A) > 0, which makes connections with section1, is that A 2. ρ(A) is an eigenvalue of A, is an M-matrix if and only if A has positive diagonal 17

34 16

32 14

30 12

10 k 28 ||A || tA 2 ||e || 2 8 26 6 24 4

22 2

20 0 0 5 10 15 20 0 2 4 6 8 10 k t

Figure 4 2-norms of powers and exponentials of 2 × 2 matrix A in (10).

open right half-plane. This means that M-matrices are 0.4 special cases of positive stable matrices, which in turn 0.3 are of great interest due to the fact that the stabil-

0.2 ity of various mathematical processes is equivalent to positive (or negative) stability of an associated matrix. 0.1 The class of matrices whose inverses are M-matrices

0 is also much-studied. To indicate why, we state a result about matrix roots. It is known that if A is an M-matrix −0.1 then A1/2 is also an M-matrix. But if A is stochastic (that 1/2 −0.2 is, it is nonnegative and has unit row sums), A may not be stochastic. However, if A is both stochastic and −0.3 the inverse of an M-matrix then A1/p is stochastic for −0.4 all positive integers p.

−1.2 −1 −0.8 −0.6 −0.4 −0.2 12 Matrix Inequalities Figure 5 Approximation to 10−2-pseudospectrum of A in (10) comprising eigenvalues of 5000 randomly perturbed There is a large body of work on matrix inequalities, matrices A + E in (11). The eigenvalues of A are marked by ranging from classical 19th and early 20th century white circles. inequalities (some of which are described in section 5.4) to more recent contributions, which are often moti- entries and AD is diagonally dominant by rows for vated by applications, notably in statistics, physics, some nonsingular diagonal matrix D. and control theory. In this section we describe just An important source of M-matrices is a few examples, chosen for their interest or practical of differential equations, and the archetypal example is usefulness. the second-difference matrix, described at the start of An important class of inequalities on Hermitian section6, which is an M-matrix multiplied by −1. For matrices is expressed using the Löwner (partial) order- this application it is an important result that when A ing in which for Hermitian X and Y , X > Y denotes that is an M-matrix the Jacobi and Gauss–Seidel iterations X − Y is positive semidefinite while X > Y denotes that for Ax = b both converge for any starting vector—a X − Y is positive definite. Many inequalities between result that is part of the more general theory of regular real numbers generalize to Hermitian matrices in this splittings. ordering. For example, if A, B, C are Hermitian and A Another important property of M-matrices is imme- commutes with B and C then diate from the definition: the eigenvalues all lie in the A > 0,B 6 C ⇒ AB 6 AC. 18

A+B A/2 B A/2 A function f is matrix monotone if it preserves the the related inequalities k e k 6 k e e e k 6 A B order, that is, A 6 B implies f (A) 6 f (B), where f (A) k e e k hold for any unitarily invariant norm. denotes a function of a matrix. Much is known about this class of functions, including that t1/2 and log t are 13 Library Software matrix monotone but t2 is not. From the early days of digital computing the benefits Many matrix inequalities involve norms. One exam- of providing library subroutines for carrying out basic ple is √ operations such as the addition of vectors and the for- k| | − | |k k − k A B F 6 2 A B F , mation of vector inner products was recognized. Over where A, B ∈ Cm×n and |·| is the matrix absolute value the ensuing years many matrix computation research defined in section2. This inequality can be regarded codes were published, including in the Linear Alge- as a perturbation result that shows the matrix absolute bra volume of the Handbook for Automatic Computa- value to be very well conditioned. tion (1971) and in the Collected Algorithms of the ACM. An example of an inequality that finds use in the Starting in the 1970s the concept of standardized sub- analysis of convergence of methods in nonlinear opti- programs was developed in the form of the Basic Lin- mization is the Kantorovich inequality, which for Her- ear Algebra Subprograms (BLAS), which are specifica- tions for vector (level 1), matrix–vector (level 2), and mitian positive definite A with eigenvalues λn 6 ··· 6 matrix–matrix (level 3) operations. The BLAS have been λ1 and x 6= 0 is widely adopted, and highly optimized implementations (x∗Ax)(x∗A−1x) (λ + λ )2 1 n . are available for most machines. The freely-available (x∗x)2 6 4λ λ 1 n LAPACK library of codes represents the current This inequality is attained for some x, and the left-hand state of the art for solving dense linear equations, least side is always at least 1. squares problems, and eigenvalue and singular value Many inequalities are available that generalize scalar problems. Many modern programming packages and inequalities for means. For example, the arithmetic– environments build on LAPACK. 1/2 1 geometric mean inequality (ab) 6 2 (a+b) for posi- It is interesting to note that the TOP500 list tive scalars has an analogue for Hermitian positive def- (http://www.top500.org) ranks the world’s fastest 1 inite A and B in the inequality A # B 6 2 (A + B), where computers by their speed (measured in flops per sec- A # B is the geometric mean defined as the unique Her- ond) in solving a random linear system Ax = b by GE. − mitian positive definite solution to XA 1X = B. The This benchmark has its origins in the 1970s LINPACK geometric mean also satisfies the extremal property project, a precursor to LAPACK, in which the perfor-   AX   mance of contemporary machines was compared by A # B = max X : X = X∗, 0 , XB > running the LINPACK GE code on a 100 × 100 system. which hints at matrix completion problems, in which the aim is to choose missing elements of a matrix in 14 Outlook order to achieve some goal, which could be to satisfy a Matrix analysis and numerical linear algebra remain particular matrix property or, as here, to maximize an very active areas of research. Many problems in applied objective function. Another mean for Hermitian posi- mathematics and scientific computing require the solu- tive definite matrices (and applicable more generally), tion of a matrix problem at some stage, so there is 1 is the log-Euclidean mean, exp( 2 (log A + log B)), where always a demand for better understanding of matrix log is the principal logarithm, which is used in image problems and faster and more accurate algorithms for registration, for example. their solution. As the overarching applications evolve, Finally, we mention an inequality for the matrix expo- new problem variants are generated, often involving nential. Although there is no simple relation between new assumptions on the data, different requirements eA+B and eA eB in general, for Hermitian A and B the on the solution, or new metrics for measuring the suc- A+B A B inequality trace(e ) 6 trace(e e ) was proved inde- cess of an algorithm. A further driver of research is pendently by S. Golden and J. Thompson in 1965. Orig- computer hardware. With the advent of processors with inally of interest in statistical mechanics, the Golden– many cores, the use of accelerators such as graphics Thompson inequality has more recently found use in processing units (GPUs), and the harnessing of vast random matrix theory. Again for Hermitian A and B, numbers of processors for parallel computing, the 19 standard algorithms in numerical linear algebra are having to be reorganized and possibly even replaced, so we are likely to see significant changes in the coming years.

15 Further Reading

Three must-haves for researchers are Golub and Van Loan’s influential treatment of numerical linear alge- bra and the two volumes by Horn and Johnson, which contain a comprehensive treatment of matrix analysis.

[1] Rajendra Bhatia. Matrix Analysis. Springer-Verlag, New York, 1997. xi+347 pp. ISBN 0-387-94846-5. [2] Rajendra Bhatia. Linear algebra to quantum cohomology: The story of Alfred Horn’s inequal- ities. Amer. Math. Monthly, 108(4):289–318, 2001. [3] Rajendra Bhatia. Positive Definite Matrices. Prince- ton University Press, Princeton, NJ, USA, 2007. ix+254 pp. ISBN 0-691-12918-5. [4] Gene H. Golub and Charles F. Van Loan. Matrix Computations. Fourth edition, Johns Hopkins Uni- versity Press, Baltimore, MD, USA, 2013. xxi+756 pp. ISBN 978-1-4214-0794-4. [5] Nicholas J. Higham. Accuracy and Stability of Numerical Algorithms. Second edition, Society for Industrial and Applied Mathematics, Philadelphia, PA, USA, 2002. xxx+680 pp. ISBN 0-89871-521-0. [6] Roger A. Horn and Charles R. Johnson. Topics in Matrix Analysis. Cambridge University Press, Cam- bridge, UK, 1991. viii+607 pp. ISBN 0-521-30587-X. [7] Roger A. Horn and Charles R. Johnson. Matrix Analysis. Second edition, Cambridge University Press, Cambridge, UK, 2013. xviii+643 pp. ISBN 978-0-521-83940-2. [8] Beresford N. Parlett. The Symmetric Eigenvalue Problem. Society for Industrial and Applied Mathe- matics, Philadelphia, PA, USA, 1998. xxiv+398 pp. Unabridged, amended version of book first pub- lished by Prentice-Hall in 1980. ISBN 0-89871-402- 8. [9] Yousef Saad. Iterative Methods for Sparse Linear Systems. Second edition, Society for Industrial and Applied Mathematics, Philadelphia, PA, USA, 2003. xviii+528 pp. ISBN 0-89871-534-2. [10] G. W. Stewart and Ji-guang Sun. Matrix Pertur- bation Theory. Academic Press, London, 1990. xv+365 pp. ISBN 0-12-670230-6. [11] Françoise Tisseur and Karl Meerbergen. The quad- ratic eigenvalue problem. SIAM Rev., 43(2):235– 286, 2001.