ON PROGRAMMING PARALLEL COMPUTERS Leslie Lamport Massachusetts Computer Associates, Inc. .INTB,0DUCTION writing a program, which is an intermediate stage between the problem and the machine code. The In this paper, I will make some general ob- task of generating machine code for a problem is servations about how computers should be program- then broken into two steps: (1) programming -- med, and how programs should be compiled. I will the essentially human task of constructing a pro- restrict my attention to programming computers to gram which represents a solution to the problem, solve numerical analysis problems, although most and (2) compiling -- the essentially computer- of my remarks can be applied to other problem performed task of constructing a machine code areas as well. My primary concern is for large which correctly implements the program. I will parallel computers. However, I will show that represent this process with the following picture: parallelism is not a major issue, and I will not re- programming compiling strict the discussion to any single type of compu- PROBLEM • PROGRAM ~ MACHINE CODE ter architecture. The paper is a very general one, and I will We can think of the problem, the program, and the make few specific proposals for language or com- machine code as variables assuming values in piler design. Instead, I will examine the funda- their appropriate domains, so programming and mental nature of the programming/compiling pro- compiling are mappings between these domains. cess, how this process is done now, and how it The usefulness of this picture lies in the should be done in the future. Few of my observa- fact that it clearly shows the duality or comple- tions will be new or original. However, a careful mentarity of the programming and compiling tasks. analysis will lead to a somewhat surprising conclu- Suppose we consider the problem and the machine sion: a higher level programming language is code to be fixed and the program to be variable. needed both to reduce programming costs and also Choosing a program which makes the programming to obtain more efficient programs. The analysis task easier will tend to make the compiling task will also provide some useful ideas for designing more difficult, and vice-versa -- assuming that all a higher level language and compiling efficient else remains the same. (Requiring the programmer code from it. to be blindfolded will make programming more dif- ficult, but it will not make compiling any easier.) ON PROGRAMMING It is important to remember that this duality exists only when the program is varied, but the machine I will begin with a general discussion of code -- or the class of acceptable machine what programming is all about. I will not say any- codes -- is not. thing very profound, but I hope to clarify some of In going from the problem to the machine the problems, and provide a basis for the more code, the basic goal is to minimize the cost of specific material I will present later. solving the problem. This cost has three distinct We are given a domain of problems to be components: solved. We may think of these problems as speci- fied in some very abstract form -- perhaps as I Programming Cost -- the cost of generating "pure thoughts". I -#ill remain vague about the na- the program. Its major element is the ture of a problem specification, since it is very human effort required, including the debug- hard to define it precisely. For example, the ging effort. One seldom solves a single specification of a problem involving the simulation problem on a single computer. Therefore, of a physical system includes some sort of state- evaluating the effect of some policy on this ment about the likely range of certain physical cost requires considering the cost of mod- parameters. ifying the program for a different problem Given one of these problems, a necessary (program maintenance) or for a different task is to generate a suitable machine code for It, computer. where a machine code is a collection of operating II Compili~q Cost -- the cost of generating instructions for some specific computer. For ex- machine code from the program. Its major ample, a machine language (binary) program for a element is the computer resource used to PDP-10 and an input tape for some particular perform all the compilations needed to Turlng machine are both machine codes. write and debug the program. It also in- This task is usually accomplished by cludes part of the cost of developing the 25 compiler. Since this latter cost is shared transforming the problem into the PDP-10 program byall uses of the compiler, it is usually and must be put back by the decompiler. It is this negligible. vague something which I am calling "information". Ill Execution Cost -- the cost of executing the This concept of information is closely re- machine code on a computer in order to lated to the concept of entropy in physics. A liter solve the problem. It consists mainly of of blue ink and a liter of yellow ink are equivalent the cost of the computer resource used. to the two litersof green fluid obtained by mixing them, since the individual "ink molecules" are Observe that the first two costs are at- unchanged by the mixing process. However, tached to the arrows in our picture. Hence, they ergodic theory tells us that it is more difficult to are dual to each other. Decreasing the program- separate the two colors than it is to mix them. ming cost will tend to increase the compiling Without an ergodic theory for programming, we can- cost, and vice-versa -- assuming that the quality not precisely define the concept of information. of the machine code and all other factors remain However, it seems to be a valid concept, and we the same. The third cost is just attached to the can place some trust in our intuitive understanding machine code. It measures the quality of the end of it. product of the programming and compiling proces- Consider a specific program for a certain ses. problem. We say that it is a high level program if Let us now examine the compiling task more it contains much information about the problem, closely. It can be separated conceptually into and that it is a low level program if it contains two parts: (i) translation -- the process of trans- little information about the problem. In other forming a program into some correct machine code, words, we have lost less information about the and (ii) compiler optimization -- the process Of problem in writing a high level program than in choosing the best possible machine code. Al- writing a low level one. though it is impossible to completely separate Equivalently, a high level program allows these two aspects of compiling, many compilers do a great deal of freedom of choice in selecting a have a separate optimization phase which is de- machine code to implement it, and a low level pro- voted solely to performing this choosing. Of gram allows little choice of machine code imple- course, other phases of these compilers also per- mentation. For example, an assembly language form some compiler optimization, since they make program allows the compiler little choice in the choices among different possible machine codes. machine language it produces. It is only free to There is a dual separation of the program- choose such things as the absolute machine ad- ming task into two parts: (i) the process of trans- dresses of certain program segments. Hence, the forming the problem into some program, and (ii) assembly language program is a low level one. programmer optimizatlon -- the process of choos- Conversely, a Fortran program can be compiled into ing the best possible program. It is impossible to many different machine codes. For example, there actually separate them, but it is useful to remem- may be many different instruction sequences for ber that there are these two different aspects to the evaluating a single Fortran arithmetic expression. programming task. Hence, the Fortran program is a high level one. Let us return to our picture of the program- Another way of viewing this is to think of a ruing/compiling process. Moving from left to right problem as specifying what is to be done, and a in the picture represents a loss of information. machine code as specifying how it is done. A pro- Choosing a particular program destroys some infor- gram lies somewhere in between. The closer it mation about the problem. A machine code imple- comes to describing what is to be done rather than mentation of that program contains even less infor- how to do it, the higher level a program it is. mation about the problem than the program does. A high level programming language is a Information is the same as freedom of choice. language in which one usually writes high level There are many possible machine codes to solve a programs, and a low level programming language is problem. When we write a program for the prob- one in which low level programs are usually writ- lem, we lose the possibility of obtaining some of ten. However, it is important to remember that those machine codes. For example, writing a even with a single programming language, one can PDP-10 assembly language program for a matrix write higher and lower level programs for the same calculation problem undoubtedly destroys the in- problem. formation about the program which is needed to Now let us consider the programming/com- generate an efficient machine code for the ILLIAC piling duality in terms of the level of a program.