From Numerical Analysis to Computational Science Bjorn¨ Engquist · Gene Golub 1. Introduction The modern development of numerical computing is driven by the rapid in- crease in computer performance. The present exponential growth approximately follows Moore’s law, doubling in capacity every eighteen months. Numerical computing has, of course, been part of mathematics for a very long time. Al- gorithms by the names of Euclid, Newton and Gauss, originally designed for computation “by hand”, are still used today in computer simulations. The electronic computer originated from the intense research and devel- opment done during the second world war. In the early applications of these computers the computational techniques that were designed for calculation by pencil and paper or tables and mechanical machines were directly implemented on the new devices. Together with a deeper understanding of the computational processes new algorithms soon emerged. The foundation of modern numerical analysis was built in the period from the late forties to the late fifties. It be- came justifiable to view numerical analysis as an emerging separate discipline of mathematics. Even the name, numerical analysis, originated during this pe- riod and was coined by the National Bureau of Standards in the name of its laboratory at UCLA, the Institute for Numerical Analysis. Basic concepts in numerical analysis became well defined and started to be understood during this time: • numerical algorithm • iteration and recursion • stability • local polynomial approximation • convergence • computational complexity The emerging capability of iteratively repeating a set of computational op- erations thousands or millions of times required carefully chosen algorithms. A theory of stability became necessary. All of the basic concepts were important but the development of the new mathematical theory of stability had the most immediate impact. The pioneers in the development of stability theory for finite difference approximations of partial differential equations were von Neumann, Lax and Kreiss, see e.g. the text by Richtmyer and Morton [16]. In numerical linear algebra the early analysis by Wilkinson was fundamental, [19] and the paper Page: 433 Engquist/Schmid (eds.) Mathematics Unlimited – 2001 and Beyond author: Engquis2 date/time: 29-Oct-2000 / 19:48 434 B. Engquist · G. Golub [7] by Dahlquist gave the foundation for stability and convergence theory of numerical methods for ordinary differential equations. The development of numerical computing has been gradual and based on the improvements of both algorithms and computers. Therefore, labeling different periods becomes somewhat artificial. However, it is still useful to talk about a new area, often called scientific computing, which developed a couple of decades after the foundation of numerical analysis. The SIAM Journal of Scientific and Statistical Computing was started in 1980. Numerical analysis has always been strongly linked to mathematics, appli- cations and the computer. It is a part of applied mathematics and its language is mathematics. Its purpose is to solve real world problems from basic physics to practical engineering. The tool used to obtain the solution is the computer. Thus its development is often driven by technology, both in terms of computer capacity and architecture and also by the many technological applications. With the term scientific computing we indicate a further strengthening of these links. It, therefore, became less viable to think of numerical computing as an isolated topic. As the mathematical models to be numerically approximated became more complex, more advanced mathematical analysis was necessary. The emphasis shifted from linear to nonlinear problems, and it became possible to handle more realistic applications. Yet, in order to construct efficient algorithms for these applications, a more detailed knowledge of their properties was needed. The computer architecture changed. Vector and parallel architectures required a rethinking of the basic algorithms. The powerful new computers also pro- duced enormous amounts of data and consequently visualization became more important in order to understand the results of the calculations. We use the label computational We use the label computational science for the latest shift in paradigms, that science for the latest shift in is now occuring at the turn of the century. The links to the rest of mathematics, to paradigms, that is now occuring application and the rest of the computer science are even further strengthened. at the turn of the century. However, the most significant change is that new fields of applications are being considered. Many important parts of numerical computing were established with little input from mathematicians specializing in numerical analysis or scientific com- puting. This includes simulations in large parts of physics, chemistry, biology and material science. The scientists in these fields often developed their own algorithms. It is clear that it is of mutual benefit for both the scientists and the experts in numerical analysis to initiate closer collaboration. The applied math- ematicians can analyze and improve the numerical methods and also adapt them to new areas of applications. This has happened in many fields in the past. Math- ematical analysis has been the basis for the derivation of new algorithms, which often have contributed more to the overall computational capability than the increase in computer power. Computational science has also been a stimulating source for new problems in mathematics. 2. Two Computational Subfields We shall discuss the two important fields of numerical linear algebra and com- putational fluid dynamics. The computer can only do logical and basic arith- Page: 434 Engquist/Schmid (eds.) Mathematics Unlimited – 2001 and Beyond author: Engquis2 date/time: 29-Oct-2000 / 19:48 From Numerical Analysis to Computational Science 435 metical operations. Therefore the core of all numerical computation will be to solve problems from linear algebra. Numerical linear algebra reached a mature state quite early in terms of theory and software. It is, however, still developing partly because it is very sensitive to changes in computer architecture. The field of computational fluid dynamics has been a significant engine for the development of numerical computing throughout the century. The reason has been the importance of applications such as weather prediction and aerody- namical simulations, but also the relative complexity of nonlinear mathematical models requiring careful analysis in the development of numerical algorithms. 2.1 Numerical Linear Algebra From the earliest days of modern electronic computers, the solution of partial differential equations and matrix equations have been of great interest and im- portance. Solving systems of equations was often a central issue in solving a sparse system associated with an approximation to a partial differential equa- tion. There was significant interest in the solution of linear equations for such problems which often arise in scientific and engineering applications, as well as in statistical problems. There has been from the earliest days a considerable effort in solving numerical problems arising in linear algebra, a pursuit which has involved such distinguished mathematicians as von Neumann and Turing. However, the number of people involved in this effort has always been relatively small, probably less than 150 people at any given time. This estimate is based on the attendance at the Householder meetings which is held approximately every three years and is devoted to numerical linear algebra. There are three important components that play a role in solving any prob- lem involving linear algebra. They are: perturbation theory, development and analysis of numerical algorithms and software. We shall briefly describe some of the developments in each of these areas which has taken place in the last fifty years. Let us consider a system of equations Ax = b + r where A is an m × n matrix, and b is a given vector. We desire to determine x so that the norm r=min. There are several “parameters” associated with the numerical solution of such a problem. We list some of these below. a) The relationship between m and n is of importance in any numerical proce- dure. When m ≥ n, then the system is overdetermined, but in many instances, the solution will be unique. b) The rank of the matrix A may or may not be known explicitly. The real difficulty arises when a small perturbation in A will change the rank. c) The numerical algorithms will be dependent on the norm chosen. For instance when the least squares solution is computed, it is only necessary to solve a system of linear equations, though this can have disastrous effects! d) The structure of the matrix A plays an important role in any solution method. If the matrix, for example, is sparse or symmetric, specialized methods may be used. Page: 435 Engquist/Schmid (eds.) Mathematics Unlimited – 2001 and Beyond author: Engquis2 date/time: 29-Oct-2000 / 19:48 436 B. Engquist · G. Golub e) Another important consideration is the origin of the problem. For instance, if one is solving the approximation of an elliptic equation, then often one can make use of this knowledge in developing an algorithm. The interplay of these parameters with the structure of the matrix plays an important role in the development of any numerical procedure. Inherent in the solution of any problem is the basic stability of the solution under perturbation to the data. We describe this for the special case of linear equations where the matrix A has full rank; thus m = n = rank(A). Consider the system of equations, Ax = b, (1) and the perturbed system, (A + ∆)y = b + δ. (2) How can we relate the solution of (1) and (2)? The perturbation theory gives us some sense of inherent accuracy of the problem. After a few simple matrix manipulations, it can be shown that if, ∆ δ ≤ ε, ≤ ε ρ< , A b and 1 then, x − y ε ≤ 2 κ, x 1 − ρ where, − − ρ =∆·A 1 and κ(A) =A·A 1. The quantity κ(A) is called the condition number with respect to linear systems.