@) I AUTHOR'S COPY Decision Resources, Inc. Bay Colony Corporate Center 1100 Winter Street Information Systems Industry Waltham, Massachusetts 02154 Telephone 617.487.3700 The Outlook for Scalable Telefax 617.487.5750 Parallel Processing Gordon Bell Consultant to Decision Resources, Inc. It is likely that this decade will usher in the beginning Business Implications of an era in which general-purpose1 scalable parallel - A * Scalable, massively parallel processing computers computers assume most of the applications currently promise to become the most costeffective ap run on mainframes, supercomputers, and specialized proach to computing within the next decade, and scalable computers. A scalable computer is a com- the means by which to solve particular, difficult, puter designed from a small number of basic compo- large-scale commercial and technical problems. nents, without a single bottleneck component, so that The commercial and technical markets are funda- the computer can be incrementally expanded over its mentally different. Massively parallel processors designed scaling range, delivering linear incremental may be more useful for commercial applications performance for a well-defined set of scalable applica- because of the parallelism implicit in accessing a tions. General-purpose scalable computers provide a database through multiple, independent transac- wide range of processing, memory size, and 1/0 re- tions. Ease of programming will be the principal sources. Scalability is the degree to which perform- factor that determines how rapidly this class of ance increments of a scalable computer are linear. computer architecture will penetrate the general- Ideally, an application should be usable at all com- purpose computing market. puter size scales and operate with constant efficiency. Vendors that succeed in developing general-pur- Parallel computers are defined by their ability to share pose scalable parallel computers have the oppor- or communicate data among multiple processors. Fig- tunity, by early in the next decade, to be able to ure 1 shows the basic structure of a parallel computa- address the computer systems market, including tion. The computation starts with a sequential thread most of the traditional roles of mainframes and supercomputers and today's specialized scalable (1) that includes job scheduling and other serial com- computers. putation. A basic loop starts with supervisory schedul- ing (2) followed by the computation (3) and inter- The direction offering the most promise for scal- computer message (4) phases of a thread. Synchroni- able parallel processing computer development zation (5) occurs prior to returning to scheduling the involves the use of standard processing and net- next unit of parallel work (2). The length of time un- working elements and programming environ- til a computation thread must synchronize with an- ments and ensuring compatibility with traditional other parallel thread indicates the granularity of a multiprocessors, workstations, and PCs. ~araUelstructure. 1. Test for general purposeness: Can the computer efficiently process a wide range--. of jobs (includin~ a workload consisting of sequential to parallel processing, small to large job sizes, short to ibng runtimes, and interactive to batch response times) requiring a variety of proc- essing, memory, database, and I/O resources? Press Date: June 21, 1994 Figure 1 systems software and applications. The transition from The Basic Structure of Parallel Computation what currently exists to the scalable parallel computer Communication systems of the future will not be automatic, however, Cornputaton Overhead and Delays because of the difficulty in establishing standards for parallel processing, which enable applications to run efficiently on a range of parallel machines. Only when standards have been established, standards to which all manufacturers adhere, will software applications Startup Overhead for scalable parallel computing truly flourish and drive market growth. Scalable parallel computers have evolved from two in- dependent and distinct application directions based on two different sets of requirements: technical (i.e., scientific/engineering) and commercial. Scheduling synchronization Overhead Overhead Technical Applications Source: Gordon Bell. Technical applications are based on floating-point op- erations used in analysis, simulation, and design. Tech- The most basic parallelism is using multiprogramming nical applications focus on achieving the greatest at the workload level, where a common pool of compu- number of floating-point operations per second tational resources (processing, primary and secondary (FLOPS),although some technical applications, such memory, and networking) is available to trade off as genome sequencing, are fundamentally database-ori- among a large job mix with varying degrees of paral- ented. Most of the fundamental understanding about lelization (including completely scalar operations). parallelism has been derived from attempts to provide For peak performance of a single job, two forms of par- highly parallel technical computers. allelism may be required: Evolvability .is an essential pm$mty of Transparent (or implicit) parallelism in which the a scalable parallel cornputex computer breaks a job into parallel computational threads without intervention by the user, and Two basic programming paradigms are used for techni- Explicit multiprocess parallelism in which the user is cal computing: data parallel and multiprocess. In the required to formulate a job in terms of both func- data parallel approach, a FORTRAN dialect (such as tional and data parallelism. FORTRAN 90, High Performance FORTRAN [HPF] , or just FORTRAN 77) is used with multiple copies of a Evolvability (i.e., generation or technology scalability) single program that operate on multiple data items in is the ability to implement a follow-on computer of the parallel (called SPMD) . same family using faster components. Evolvability is an essential property of a scalable parallel computer The multiprocess approach, as in FORTRAN M, uses a because of the time and financial investment required program that is divided into subproblems and distrib- to develop parallel programs. It requires that all rate uted among the nodes that communicate by explicit and size metrics (such as processing, memory and 1/0 message passing. Multiprocess applications can be di- bandwidth, memory size, and especially interconnec- vided by function (i.e., different processes handle dif- tion bandwidth) increase proportionally from genera- ferent types of tasks) or by data (i.e., different proc- tion to generation. esses handle different data). Ordinary operating sys- tem mechanisms such as pipes, sockets, and threads facilitate parallelism by providing communication The Software Driver among and within processes. Programming environ- Computers that are used for a single problem, func- ments that operate on all computer structures, includ- tion, or workload can be built to scale over a range of ing networked PCs and workstations, have been several thousand processors; they are limited only by developed for multiprocessing. They include Oak SPECTRUM Information Systems Industry Scalable Parallel Processing Decision Resources, Inc. Press Date: June 21, 1994 Ridge National Laboratory's Parallel Virtual Machine memory that any processor can symmetrically access. (PVM) , Scientific Computing Associates' Linda, Para- Linda handles only the coordination functions, which soft's Express, and various programs (for example, include establishing the common memory space, proc- IBM's LoadLeveler) that can manage a computer clus- ess creation, interprocess communication, and con- ter as a single facility. trol. All objects can be run in parallel under the right controlling circumstances. The base language, such as Commercial Applications C and a FORTRAN dialect, acts in a normal fashion, while Linda adds four functions-in, out, read, and Commercial applications are usually database-cen- evalua&to the language. tered for transaction processing and database analysis. Transaction processing is implicitly parallel, and many User interface software, debuggers, performance moni- customer-specific applications are easily portable be- tors, and many other tools are part of these basic paral- cause of the nature of the interface and implicit paral- lel environments. New sets of tools that treat a cluster lelism. Once a database port has been made, many of workstations as a single entity and then allow users uses are possible because the database is parallel. Data to utilize the cluster in parallel for a variety of tasks analysis or "data mining" is organized to utilize the par- have been recently introduced by IBM, Platform Com- allel access to a single database. Because data analysis puting, and Scalable Technologies. is not typically considered mission-critical, it has been For multiprocessors, small degrees of parallelism are the entry point for parallel applications in commerce. supported through such mechanisms as multitasking The first parallel computers for the commercial mar- and Unix pipes in an explicit or direct user control ket were from Tandem and era data.^ In these sys- fashion. Linda extends this model to manage the crea- tems, a transaction-processing monitor operated on a tion and distribution of independent processes for par- number of independent transactions using a variety of allel execution in a shared address space. applications, which were distributed