Solving Sparse, Symmetric, Diagonally-Dominant Linear Systems in Time ¢¡¤£¦¥¨§ © ¥ Daniel A. Spielman Shang-Hua Teng Department of Mathematics Department of Computer Science Massachusetts Institute of Technology Boston University and Akamai Technologies Inc. Abstract quires ST)U7V- space. In contrast, if is symmetric and has non-zero entries, then one can use the Conjugate Gradi- )N8- ent method, as a direct method, to solve for OJN' in We present a linear-system solver that, given an -by- time and )U- space! Until Vaidya’s revolutionary intro- symmetric positive semi-definite, diagonally dominant ma- duction of combinatorial preconditioners [24], this was the trix with non-zero entries and an -vector , pro- best complexity bound for the solution of general PSDDD duces a vector within relative distance of the solution systems. !#"%$'&()+*,!-/.!0213546- . to in time , where is the log of the ratio of the largest to smallest non-zero entry of The two most popular families of methods for solving lin- 879 :<;>= . If the graph of has genus or does not have a ear systems are the direct methods and the iterative meth- minor, then the exponent of can be improved to the min- ods. Direct methods, such as Gaussian elimination, perform ¥>?A@CB ¥I?B - imum of and EDF*CG¨-H . The key contribution of arithmetic operations that produce treating the entries of our work is an extension of Vaidya’s techniques for con- and symbolically. As discussed in Section 1.4, direct structing and analyzing combinatorial preconditioners. methods can be used to quickly compute if the matrix has special topological structure. Iterative methods, which are discussed in Section 1.5, com- pute successively better approximations to . The Cheby- 1. Introduction shev and Conjugate Gradient methods take time propor- Y[Z \-]"^$¨&[_Y[Z(`-/*C- tional to XW to produce approxima- Y#Z[`- Sparse linear systems are ubiquitous in scientific comput- tions to with relative error , where is the ratio of ing and optimization. In this work, we develop fast algo- the largest to the smallest non-zero eigenvalue of . These rithms for solving some of the best-behaved linear systems: algorithms are improved by preconditioning—essentially ¢ abJL' a those specified by symmetric, diagonally dominant matri- solving abJL! for a preconditioner that is Y EKc/ad- ces with positive diagonals. We call such matrices PSDDD carefully chosen so that Z is small and so that it is a as they are positive semi-definite and diagonally dominant. easy to solve linear systems in a . These systems in may Such systems arise in the solution of certain elliptic differ- be solved using direct methods, or by again applying itera- ential equations via the finite element method, the model- tive methods. ing of resistive networks, and in the solution of certain net- Vaidya [24] discovered that for PSDDD matrices one work optimization problems [23, 19, 15, 25, 26]. could use combinatorial techniques to construct matrices a While one is often taught to solve a linear system that provably satisfy both criteria. In his seminal work, a MJN by computing KJL and then multiplying by , this ap- Vaidya shows that when corresponds to a subgraph of Y Mc/aO- proach is quite inefficient for sparse linear systems—the the graph of , one can bound Z by bounding best known bound on the time required to compute OJL the dilation and congestion of the best embedding of the a 7¤ /PRQ JN - is ) [9] and the representation of typically re- graph of into the graph of . By using precondition- ers derived by adding a few edges to maximum spanning Partially supported by NSF grant CCR-0112487. spiel- trees, Vaidya’s algorithm finds -approximate solutions to [email protected] PSDDD linear systems of maximum valence e in time Partially supported by NSF grant CCR-9972532. [email protected] 1 P¡ L"%$'&[_Y[Z[`-/*C-/- 5e¨U-/! . When these systems have spe- result to general planar PSDDD linear systems. cial structure, such as having a sparsity graph of bounded Due to space limitations, some proofs of have been omitted, genus or avoiding certain minors, he obtains even faster al- but will appear in the on-line and full versions of the paper. gorithms. For example, his algorithm solves planar linear 7 /e¨U- "^$¨&(_Y `-/*C-/- systems in time Z . This paper fol- lows the outline established by Vaidya: our contributions 1.1. Background and Notation Y Mc/ad- are improvements in the techniques for bounding Z , a construction of better preconditioners, a construction that depends upon average degree rather than maximum degree, ,.-+/,10 A symmetric matrix is semi-positive definite if and an analysis of the recursive application of our algo- 2 for all vectors , . This is equivalent to having all eigenval- rithm. ues of non-negative. 2 As Vaidya’s paper was never published , and his manuscript In most of the paper, we will focus on Laplacian matri- lacked many proofs, the task of formally working out his re- ces: symmetric matrices with non-negative diagonals and 2 sults fell to others. Much of its content appears in the the- 5 465U/798 non-positive off-diagonals such that for all 3 , . sis of his student, Anil Joshi [16]. Gremban, Miller and However, our results will apply to the more general fam- Zagha[13, 14] explain parts of Vaidya’s paper as well as ex- ily of positive semidefinite, diagonally dominant (PSDDD) tend Vaidya’s techniques. Among other results, they found ;7<8 7=:>0 matrices, where a matrix is diagonally dominant if : ways of constructing preconditioners by adding vertices to 5 4 5? : @7<8 : for all 3 . We remark that a symmetric matrix is the graphs and using separator trees. PSDDD if and only if it is diagonally dominant and all of Much of the theory behind the application of Vaidya’s tech- its diagonals are non-negative. niques to matrices with non-positive off-diagonals is devel- In this paper, we will restrict our attention to the solution oped in [3]. The machinery needed to apply Vaidya’s tech- of linear systems of the form where is a PS- niques directly to matrices with positive off-diagonal ele- OJN DDD matrix. When is non-singular, that is when ments is developed in [5]. The present work builds upon OJN' exists, there exists a unique solution , to the lin- an algebraic extension of the tools used to prove bounds ear system. When is singular and symmetric, for every on Y(Z[Mc/aO- by Boman and Hendrickson [6]. Boman and BA \- A \- Span there exists a unique Span such Hendrickson [7] have pointed out that by applying one of that . If is the Laplacian of a connected graph, their bounds on support to the tree constructed by Alon, C ¢ then the null space of is spanned by . Karp, Peleg, and West [1] for the -server problem, one ob- There are two natural ways to formulate the problem of find- a Y EKcRaO- tains a spanning tree preconditioner with Z ing an approximate solution to a system . A vector § ¨¡© § ¨¡© § ¨¡© '- ¦¥ 0 ¤£ FE . They thereby obtain a solver for PS- D¤ GDIH #D'JD has relative residual error if . We DDD systems that produces -approximate solutions in time say that a solution is an -approximate solution if it is at !1354¨"^$¨&[EY(Z[\-R*,!- 8! . In their manuscript, they asked relative distance at most from the actual solution—that is, IE whether one could possibly augment this tree to obtain a DIH KD D if D . One can relate these two notions of better preconditioner. We answer this question in the af- approximation by observing that relative distance of to 7 R7L"%$'& )U-/- firmative. An algorithm running in time the solution and the relative residual error differ by a multi- has also recently been obtained by Maggs, et. al. [18]. Y `- plicative factor of at most Z . We will focus our atten- ! - The present paper is the first to push past the ) bar- tion on the problem of finding -approximate solutions. rier. It is interesting to observe that this is exactly the point Y `- The ratio Z is the finite condition number of . The at which one obtains sub-cubic time algorithms for solving L !MD DH/,#DU*ND=,#D norm of a matrix, D , is the maximum of , dense PSDDD linear systems. 7 and equals the largest eigenvalue of if is symmetric. ;QPSR `- DHMD Reif [21] proved that by applying Vaidya’s techniques re- For non-symmetric matrices, O and are typi- T: cursively, one can solve bounded-degree planar positive cally different. We let : denote the number of non-zero UWVYX \- U[Z\LE\- definite diagonally dominant linear systems to relative ac- entries in , and and denote the small- ) H1364 "%$'&(EY+`-/*,!-/- curacy in time . We extend this est and largest non-zero elements of in absolute value, re- spectively. 1 For the reader unaccustomed to condition numbers, we note that for The condition number plays a prominent role in the analy- an PSDDD matrix in which each entry is specified using bits of sis of iterative linear system solvers. When is PSD, it is precision, !#"¦"%$'&()%*+" . ¥ W Y \-]"^$¨&[ *,!- 2 Vaidya founded the company Computational Applications and Sys- known that, after Z iterations, the Cheby- tem Integration (http://www.casicorp.com) to market his linear sys- shev iterative method and the Conjugate Gradient method tem solvers. produce solutions with relative residual error at most . ? To obtain an -approximate solution, one need merely run from the observation in [3] that if , where sat- ? Y#Z] McRa -(HAY(Z EKc/ad- "^$¨&[EY(Z[\-/- times as many iterations. If has non-zero isfies the above condition, then .