Current Trends in Numerical Linear Algebra: from Theory to Practice

SOFSEM ’94 – Milovy, The Czech Republic, 27. 11. – 9. 12. 1994 Current Trends in Numerical Linear Algebra: From Theory to Practice Z. Strakoˇs, M. T˚uma Institute of Computer Science, Academy of Sciences of the Czech Republic Pod vodárenskou vˇeˇz´ı2, CZ–182 07 Prague, The Czech Republic Email: [email protected] [email protected] Abstract: Our goal is to show on several examples the great progress made in numerical analysis in the past decades together with the principal problems and relations to other disciplines. We restrict ourselves to numerical linear algebra, or, more specifically, to solving Ax = b where A is a real nonsingular n by n matrix and b a real n dimensional vector, and to computing eigenvalues of a sparse matrix A. We discuss recent− developments in both sparse direct and iterative solvers, as well as fundamental problems in computing eigenvalues. The effects of parallel architectures to the choice of the method and to the implementation of codes are stressed throughout the contribution. Keywords: numerical analysis, numerical linear algebra, linear algebraic systems, sparse direct solvers, iterative methods, eigenvalue computations, parallel architectures and algorithms 1 Motivation modelling and extensive use of the numerical analysis, mentioning just one important area Our contribution deals with numerical analy- of application. sis. What is numerical analysis? And what Nick Trefethen proposes in his unpublished is its relation to computer science (comput- note [38] the following definition: ing sciences)? It is hard to give a definition. Sometimes “pure” mathematicians do not con- Numerical analysis is the study of sider numerical analysis as a part of mathe- algorithms for mathematical prob- matics and “pure” computer scientists do not lems involving continuous variables consider it as a part of computer science. Nev- ertheless, hardly anyone would argue against The keywords are algorithm and continuous, the importance of this field. Dramatic devel- the last typically means real or complex. Since opment in high technology in the last decades real or complex numbers cannot be represented would not be possible without mathematical exactly on computers, the part of the business 2. Solving large sparse linear algebraic systems of numerical analysis must be to study round- the numerical method to the efficient and reli- ing errors. To understand finite algorithms or able code. We demonstrate that especially on direct methods, e.g. Gaussian elimination or the current trends in developing sparse direct Cholesky factorization for solving systems of solvers. linear algebraic equation, one have to understand the computer architectures, operation 2 Solving large sparse linear alge- counts and the propagation of rounding errors. braic systems This example, however, does not tell you all Although the basic scheme of the symmetric the story. Most mathematical problems involv- Gaussian elimination is very simple and can ing continuous variables cannot be solved (or be casted in a few lines, the effective algo- effectively solved) by finite algorithms. A clas- rithms which can be used for the really large sical example - there are no finite algorithms problems usually take from many hundreds for matrix eigenvalue problems (the same con- to thousands of code statements. The differ- clusion can be extended to almost anything ence in the timings can then be many orders nonlinear). Therefore the deeper business of of magnitude. This reveals the real complex- numerical analysis is approximating unknown ity of the intricate codes which are necessary quantities that cannot be known exactly even in to cope with the large real-world problems. principle. A part of it is, of course, estimating Subsection 2.1 is devoted to the sparse direct the precision of the computed approximation. solvers stemming from this symmetric Gaus- Our goal is to show on several examples the sian elimination. Iterative solvers, which are great achievements of the numerical analysis, based on the approaching the solution step by together with the principal problems and rela- step from some initial chosen approximation, tions to other disciplines. We restrict ourselves are discussed in subsection 2.2. A comparison to numerical linear algebra, or more specifi- of both approaches is given in subsection 2.3. cally, to solving Ax = b, where A is a real nonsingular n by n matrix and b a real n- 2.1 Sparse direct solvers dimensional vector (for simplicity we restrict This section provides a brief description of the ourselves to real systems; many statements ap- basic ideas concerning sparsity in solving large ply, of course, also to the complex case), and linear systems by direct methods. Solving the to computing eigenvalues of a square matrix A. large sparse linear systems is the bottleneck in Much of scientific computing depends on these a wide range of engineering and scientific com- two highly developed subjects (or on closely putations. We restrict to the symmetric and related ones). This restriction allows us to go positive definite case where most of the impor- deep and show things in a sharp light (well, at tant ideas about algorithms, data structures least in principle; when judging this contribu- and computing facilities can be explained. In tion you must take into account our - certainly this case, the solution process is inherently sta- limited - expertise and exposition capability). ble and we can thus avoid numerical pivoting We emphasize two main ideas which can be which would complicate the description other- found behind our exposition and which are es- wise. sential for the recent trends and developments Efficient solution of large sparse linear sys- in numerical linear algebra. First, our strong tems needs a careful choice of the algorith- belief is that any “software solution” without mic strategy influenced by the characteristics → a deep understanding of mathematical (physi- of computer architectures (CPU speed, mem- cal, technical, ...) background of the problem ory hierarchy, bandwidths cache/main mem- is very dangerous and may lead to fatal er- ory and main memory/auxiliary storage). The rors. This is illustrated, e.g., on the prob- knowledge of the most important architectural lem of computing eigenvalues and on charac- features is necessary to make our computations terizing the convergence of the iterative meth- really efficient. ods. Second, there is always a long way (with First we provide a brief description of the many unexpectably complicated problems) from Cholesky factorization method for solving of a 2. Solving large sparse linear algebraic systems sparse linear system. This is the core of the the column j by the column k at the lines (3)– symmetric Gaussian elimination. Basic nota- (5) is denoted by cmod(j, k). Vector scaling at tion can be found, e.g., in [2], [18]. the lines (8)–(10) is denoted by cdiv(j). We use the square root formulation of the The right-looking approach can be de- factorization in the form scribed by the following pseudo-code: (1) for k = 1 to n A = LDLT , (2) dk = ak where L is lower triangular matrix and D is (3) for i = k + 1 to n if aik = 0 6 diagonal matrix. Having L, solution x can be (4) lik = aik/dk computed using two back substitutions and one (5) end i diagonal scaling: (6) for j = k + 1 to n if akj = 0 6 1 T (7) for i = k + 1 to n if lik = 0 Ly¯ = b ; y = D− y¯ ; L x = y. 6 (8) aij = aij likakj − Two primary approaches to factorization (9) end i are as follows (we do not mention the row- (10) end j Cholesky approach since its algorithmic prop- (11) end k erties usually do no fit well with the modern computer architectures). The left-looking approach can be described In the right-looking approach, once a col- by the following pseudo-code: umn k is completed, it immediately generates (1) for j = 1 to n all contributions to the subsequent columns, i.e., columns to the right of it in the matrix. (2) for k = 1 to j 1 if akj = 0 − 6 A number of approaches have been taken in (3) for i = k + 1 to n if lik = 0 6 order to solve the problem of matching nonze- (4) aij = aij likakj − (5) end i ros from columns j and k at the lines (6)-(10) (6) end k of the pseudo-code as will be mentioned later. Similarly as above, operation at the lines (3)– (7) dj = ajj (8) for i = k + 1 to n (5) we denote cdiv(k) and column modification at the lines (7)–(9) is denoted by cmod(j, k). (9) lij = aij/ajj (10) end i Discretized operators from most of the ap- (11) end j plications such as structural analysis, computational fluid dynamics, device and process In this case, a column j of L is computed by simulation and electric power network prob- gathering all contributions from the previously lems contain only a fraction of nonzero ele- computed columns (i.e. the columns to the left ments: these matrices are very sparse. Explicit of the column j in the matrix) to the column consideration of sparsity leads to substantial j. Since the loop at the lines (3)–(5) in this savings in space and computational time. Usu- pseudo-code involves two columns, j and k, ally, the savings in time are more important, with potentially different nonzero structures, since the time complexity grows quicker as a the problem of matching corresponding nonze- function of the problem size than the space ros must be resolved. Vector modification of complexity (memory requirements). ∗ ∗ ∗∗∗∗∗ ∗ ∗ ¯ ∗ ∗ A = A = ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗∗∗∗∗ ∗ ∗ Figure 1.1: The arrow matrix and in the natural and reverse ordering. 2. Solving large sparse linear algebraic systems For a dense problem we have CPU time the diagonal.

Load more