Promoting High-Performance Computing and Communications

CONGRESS OF THE UNITED STATES CONGRESSIONAL BUDGET OFFICE CBO

PROMOTING HIGH-PERFORMANCE COMPUTING AND COMMUNICATIONS

The Congress of the United States Congressional Budget Office

For sale by the U.S. Gove~nmentPrinting OFfice Superintendent of Documenls. Mail Stop: SSOP, Washington, DC 20102-9328 ISBN 0-16-041828-3 NOTE

Cover photo shows a detail from a processing node of a parallel supercomputer. (Photo courtesy of Bolt Beranek and Newman, Inc., Cambridge, Massachusetts.) Preface

he High Performance Computing and Communications (HPCC) program embodies many concerns regarding federal technology policy: the T role of the federal government in promoting new industries, building infrastructure, supporting joint federal-private efforts, and encouraging the transfer of technology. This report focuses on the first of these concerns--the federal role in promoting new industries. As requested by the Senate Commit- tee on Commerce, Science, and Transportation, the report concentrates on the obstacles--mainly on the demand side--that might prevent the high-performance computing and data communications industries from growing and using the technology being developed under the HPCC program. In keeping with the Congressional Budget Office's (CBO's) mandate to provide nonparti- san analysis, no recommendations are made.

Philip Webre of CBO's Natural Resources and Commerce Division wrote this report under the supervision of Jan Paul Acton and Elliot Schwartz. Serdar Dinc, then with CBO, provided research assistance in early stages of the project. The National Coordinating Office for High Performance Comput- ing and Communications coordinated Executive Branch agency comments on earlier drafts. The author wishes to thank Fiona Branton, Alan Buzacott, Deborah Clay-Mendez, Kenneth Flamm, Stephen Gould, Mark Grabowitz, Peter Gregory, Anthony Hearn, Robert Kelly, Paul Muzio, W. Edward Steinmueller, Richard Van Atta, and Joan Winston for their helpful comments on earlier drafts.

Francis Pierce and Sherry Snyder edited the manuscript, with editorial assistance from Christian Spoor, and Donna Wood typed the many drafts. Martina Wojak-Piotrow prepared the report for publication.

Robert D. Reischauer Director

June 1993

Contents

SUMMARY ONE INTRODUCTION

Public Purposes for Funding the HPCC Program 2 The HPCC and Previous Federal Computer R&D Programs 4 Alternative Strategies for the HPCC Program 5 Competing Views of Commercialization 6 TWO THE HPCC PROGRAM

The HPCC Budget 9 High-Performance Computing Systems 10 Advanced Software Technology and Algorithms 12 National Research and Education Network 13 Basic Research and Human Resources 14 Agency Responsibilities 14 THREE HPCC AND THE DEVELOPMENT OF SUPERCOMPUTER MARKETS

HPCC and Supercomputers 17 Analysis of the Supercomputer Market 18 The Supercomputer Market and Technology Spinoffs 20 Obstacles to Widespread Commercial Use of HPCC Technology 24 FOUR HPCC AND COMPUTER NETWORK MARKETS 33

HPCC and Computer Networks 33 Analysis of the Markets for Computer Network Hardware, Software, and Services 35 Obstacles to Wider Use of HPCC Technology 39 vi PROMOTING HIGH-PERFORMANCE COMPUTING AND COMMUNICATIONS June 1993

FIVE POLICY DIRECTIONS AND CONCLUSIONS 45 Networks and Medicine 46 Networks and Education 48 Networks and Manufacturing 49 Networks and Libraries 49 Program Balance 50 Conclusions 50 APPENDIXES

A Supercomputer Component Technology Spinoffs 55 B Supercomputer Speed Calculations 61 C Network Speed Calculations 63 CONTENTS vii

TABLES S-1. Incremental Funding for the HPCC Program, Fiscal Years 1992-1996, by Program Component x 1. Original Five-Year Budget Plan for the HPCC Program 10 2. Incremental Funding for the HPCC Program, Fiscal Years 1992-1996, by Program Component 11 3. Agency Responsibilities Under the HPCC Program 15 A-1. Major Computer Technologies 56

FIGURES S-1. History of Internet Traffic, 1988-1993 xiv 1. Supercomputer Market Segments 19 2. Relative Computer Market Sizes 2 1 3. Simulated Performance of Three Supercomputers 2 7

4. Composition of Internet 36 5. History of Internet Traffic, 1988-1993 37

BOX 1. Segments of the Market for Networking Products

Summary

n 1991, the Administration and the transmitting a billion bits (a gigabit) of data Congress initiated the multiagency per second. In each case, the goal is to achieve High Performance Computing and sustained performance of between one and omm.unications (HPCC) program to further three orders of magnitude better than that of the development of U.S. supercomputer tech- current technology. With this additional pow- nology and high-speed computer network er, federal agencies hope to be able to solve technology. Although one reason for this leg- computational problems that have so far elud- islation was to improve the performance of ed solution in areas of federal interest such as federal missions, an important consideration changes in global climate, high-energy phys- was the benefits such technology might yield ics, and high-performance aircraft. These for private industry. Such benefits can only types of problems are referred to as "the grand be realized if the technology becomes widely challenges." used in industry--in short, if it is commercial- ized. The program is divided into four areas:

Commercial success, however, is only one of o Supercomputer hardware, which focuses several goals for HPCC. A full analysis of the primarily on so-called scalable massively program, which is not attempted in this study, parallel supercomputers that are often would take into account how HPCC technol- composed of hundreds and thousands of in- ogy will affect federal missions. Even if it has dividual microprocessors; only a small impact on commercial technologies, the expenditures may be justified by its o Supercomputer software, which centers benefits for federal agencies. primarily on creating programming models, tools, and applications for massively parallel supercomputers, which cannot use conventional supercomputer software;

The High Performance o Networking, which focuses on upgrading the interagency National Research and Computing and Educational Network and other agency Communications networks, as well as on advanced network Program research and development (R&D);and o Basic research and advanced education for The technical goals of the High Performance programmers. Computing and Communications program are to create supercomputers capable of being In many ways the HPCC program continues scaled to a trillion mathematical operations ongoing federal work in computer R&D, but per second and computer networks capable of with added emphasis. Thus, its budget has x PROMOTING HIGH-PERFORMANCE COMPUTING AND COMMUNICATIONS June 1993

two components: existing funds and additional funds. The program is intended to double the annual federal resources being devoted to the Supercomputer development of high-performance computers and computer networks over a five-year pe- Technology and Markets riod, going from $600 million in fiscal year The systems that now seem likely to be the 1992 to $1.2 billion in 1996. Funding over the first to develop the speeds sought by HPCC are five-year period is forecast to total $4.7 billion, massively parallel supercomputers. Conven- of which $1.9 billion will be incremental to the tional supercomputers are composed of a small previous levels of resources. number--usually less than 16--of very fast pro- Although HPCC is often described as a com- cessing units. The massively parallel systems puter network R&D program, 80 percent of use a large number--as many as several hun- the new funds are for R&D involving areas dred or thousand--of relatively slow processing other than networks. Supercomputer hard- units, all working in parallel on the same ware and software are to receive over two- problem. thirds of the incremental funds, while network HPCC is supporting research into a type of R&D will receive one-fifth of the new monies; such supercomputers called scalable, which the remainder goes to basic research and edu- means that the computers can be made more cation. Summary Table 1 presents the divi- powerful by simply adding more computing sion of additional spending by program area. nodes, unlike conventional supercomputers. The theoretical advantage of scalable archi- Summary Table 1. tectures is that software applications can be Incremental Funding for the HPCC Program, developed on smaller computers and used to Fiscal Years 1992-1996, by Program Component run smaller data sets on smaller computers. However, if the models or data become too Fundinq massive, the problem can be shifted to larger As a In Millions Percentage computers without having to rewrite the soft- Component of Dollars of Total ware.

High-Performance Most modern high-end supercomputers, in- Computing Systems 682 3 6 cluding conventional supercomputers, are par- Advanced Software allel in the sense of having more than one pro- Technology and cessor, and the trend clearly is to increase the Algorithms 662 35 number. Where the technology strategies dif- National Research and fer between conventional and massively par- Education Network 390 20 allel machines is in how fast to increase the Basic Research and number of processors over time. In order to Human Resourcesa 183 10 achieve its speed goals, HPCC has gone with Total 1,917 100 computer designers who would increase the number rapidly. SOURCE: Congressional Budget Office using data from Office of Science and Technology Policy, "The Federal High Performance Computing Program" (September 8, 1989). p. 46. Supercomputer Markets NOTES: HPCC = High Performance Computing and Commu- nications. Because most supercomputers are used for re- Incremental funding is the additional money appro- priated to the HPCC agencies above the level the search purposes, the market for them is small agencies were receiving at the start of the HPCC pro- relative to other parts of the computer market. gram for their activities in this area. Numbers may not add to totals because of rounding. Only a couple of hundred systems are sold a. In addition, 15 percent of each of the other three program each year. By comparison, mainframe com- components are to be devoted to this area. puter systems--the most similar type of system SUMMARY physically, organizationally, and in cost--sell the major thrusts of computer technology de- between 15,000 and 20,000 units a year. velopment have recently been elsewhere. Overall, only 600 supercomputer systems are installed worldwide. According to the Depart- Nevertheless, in certain areas supercom- ment of Commerce, supercomputer sales to- puters have been at the forefront of computer taled $1.8 billion worldwide in 1992. technology, most notably in software. As engineering workstations--high-performance Growth in the demand for supercomputers versions of the personal computer--have be- has slowed substantially in recent years. Al- come rapid enough to provide timely math- though the market for them grew rapidly dur- ematical solutions to engineering and scienti- ing most of the 1980s, the recent annual rate fic problems, software originally written for of growth has been only 6 percent. Since many supercomputers has been translated for work- supercomputer uses are military applications, stations. overall demand may be depressed for a number of years. Future Trends in Technology Spinoff Supercomputer Technology Spinoffs Unlike previous generations of supercomputer technology, the massively parallel supercom- Part of the reason for the focus on super- puters, such as those being developed in computer technology is the commonly held be- HPCC, use the same components as the engi- lief that supercomputer technology leads all neering workstations. In this circumstance, computer technology. New technologies, it is the likelihood of innovation in workstation often argued, will appear first in supercom- technology arising first in the supercomputer puters, where performance considerations out- market may increase, although at present the weigh cost and other considerations. After an technology is flowing the other way. Indeed, innovation has proved its worth, conventional the very existence of massively parallel supercomputers can adapt it to their use. The policy computers is predicated upon the availability corollary to this belief is that small invest- of high-performance technology from commer- ments in supercomputer technology will pro- cial workstations. At the computer component duce large benefits in all areas of the computer level, the technology flow is expected to con- market. tinue from workstations to parallel supercomputers. Although supercomputers and their high- performance predecessors originally led over- Two of the likeliest major spinoffs from the all computer technical development, this role massively parallel supercomputer to other has been reduced in the last 10 years. In fact, types of computers do not involve components. supercomputers played a minor role in many Rather, the novel challenges HPCC is con- of the most important developments in com- fronting in advance of the rest of the industry puter technology in the 1980s. Most signifi- involve creating parallel paths in solving soft- cantly, in the two central areas of computer ware problems and facilitating communica- technology development during the 1980s-- tion between processors. When researchers computer architecture and integrated circuit have rewritten software applications to make technology--conventional supercomputer tech- them parallel for the massively parallel super- nology was removed from the areas of rapid computer, they have discovered that the ap- advance. The needs of personal computers and plications also run better on conventional their high-powered cousins, the engineering supercomputers. workstations, drove most of the technology developments in these areas. Although some This effect--making software run more rap- technology has come from supercomputers, idly by making the parallels implicit in most xii PROMOTING HIGH-PERFORMANCE COMPUTING AND COMMUNICATIONS June 1993 problems more explicit--may apply more gen- ministration suggest that currently available erally. Similarly, discovering ways to en- massively parallel supercomputers do not hance communications among the supercom- seem to be able to run most existing software puter's massive number of processors may be as rapidly as the fastest conventional super- analogous to improving communications computers, despite their higher theoretical among conventional computers in computer peak speeds. Nevertheless, measures of cost networks. Transfer of these technical insights per unit of performance are favorable to mas- from supercomputer technology to conven- sively parallel supercomputers. tional computer technology, however, may be limited by the differences in cost and perfor- Massively parallel computers currently oc- mance objectives in the different markets. cupy a few different small markets, some with potential for significant growth. The market with the most potential involves data-base Obstacles to the Development of management. Roughly half of corporate main- Large Commercial Markets for frame use involves data-base manipulation, such as spotting sales trends. Such problems HPCC Supercomputer are also well suited to parallel architectures. Technology One major producer of massively parallel computers is already selling to this specialized The main obstacle to the rise of large commer- market. (Although its computer is parallel, cial markets for HPCC-developed supercom- this is not, properly speaking, a supercomput- puter technology is that cheaper workstations er as it lacks substantial number-crunching may preempt further substantial growth of abilities.) the supercomputer market as a whole. At present, an increasing share of commercial de- Some massively parallel supercomputer sign problems may be solved on workstations manufacturers are interested in entering the rather than on supercomputers. If the growth data-base market. Data-base manipulation, in the workstation market preempts further however, generally does not require the num- substantial growth in the supercomputer mar- ber crunching that is the strength of super- ket, economic benefit from HPCC technology computers. Whether supercomputers can re- may not be sufficiently widespread to justify position themselves into this market is not the large funds being devoted to it on purely clear. Massively parallel supercomputers also commercial grounds. This situation may be occupy a significant fraction of the supercom- analogous to the way the personal computer puter market, currently supplementing con- replaced the mainframe or minicomputer for ventional supercomputers in supercomputer many uses in the business world and reduced centers. demand for larger business computers.

The other obstacle to the widespread use of HPCC technology in the near term can best be described as economic inertia: conventional Computer Network supercomputer hardware is good and contin- Technology and Markets ues to improve, making conventional supercomputers hard to displace in the market. The National Research and Educational Net- Furthermore, close to 20 years' worth of soft- work (NREN) is central to the HPCC program, ware has been refined to run on that hard- both as part of the goal and as an enabling ware. The difficulty of writing software for technology for other components of the pro- massively parallel supercomputers will make gram. By allowing supercomputers and other them slow to supplant conventional supercom- research assets of the federal government to puters. Tests by the Department of Energy be linked over a very rapid computer network, and the National Aeronautics and Space Ad- the program planners hope that the different ... SUMMARY Xlll agencies will be able to bring their best tools estimated to be in the $3.5 billion to $7 billion to bear on solving the grand challenges. In ad- range annually. By contrast, the market in dition, NREN could demonstrate how other switched network services, comparable to segments of society could use such power. switched voice telephone service, is still in its infancy. The NREN component of HPCC has two major subcomponents: upgrading the interagency interim NREN, and research and de- The Internet and Development of velopment focused on computer networks of 1 the Network Services Market billion bits per second. The interagency interim NREN has several parts. The most com- The Interim NREN or Internet, which uses monly discussed is the effort to bring the Na- the existing NSFNET as its backbone, is a net- tional Science Foundation's NSFNET up to 45 work of computer networks serving the educa- million bits per second, which is largely com- tional and research communities, all of which pleted. NSF has begun the bidding process for use the same computer protocol to communi- the next upgrade. Less well known efforts in- cate. The very rapid growth of traffic on volve bringing specialized data bases devel- Internet during its five years under the cur- oped by federal agencies on line to make them rent contractor indicates that it is fulfilling a accessible from the NSFNET, and improving role in research and education (see Summary other federal agency networks. Figure 1). But its users are only a fraction of the potential numbers. A major question in The research and development component upgrading Internet is whether delivering en- of NREN largely focuses on so-called "gigabit hanced services to its most sophisticated users test beds." In these test beds, industry teams is more valuable than broadening its usage by and government researchers unite to develop less sophisticated users. and test different components, protocols, and other aspects of networking technology. As Because networks become more valuable as technology is developed it presumably will be more people connect to them, Internet is be- incorporated into the construction of the in- coming the core of a nationwide data network terim NREN and vendor equipment. for many of the key centers of science and technology in the United States. Not only universities and research institutions, but also Computer Network Markets many technology companies have Internet connections directly or indirectly though com- The United States has a very large and thriv- mercial service providers. ing data communications market. According to the Department of Commerce, the worldwide market for computer network hardware Obstacles to Developing a Large and software exceeded sales of $8 billion in 1991, much if not most of it provided by U.S. Commercial Market for HPCC companies to clients in the United States. Computer Network Technology

In addition, the Department of Commerce HPCC technology might be precluded from estimates that the U.S. market for value- having substantial effect on the current mar- added network services, including credit card ket for data communications if demand for validation and the like--perhaps the most com- high-speed data communication services does mon single type of commercial network not emerge as rapidly or in the directions en- service--reached $6 billion in 1991. The mar- visaged by HPCC leaders. First, network us- ket for another popular type of service, called ers may find costs to be too high because the outsourcing, in which one firm contracts with price of one of the most important network another to manage its computer network, is components, leased private telephone lines, is xiv PROMOTING HIGH-PERFORMANCE COMPUTING AND COMMUNICATIONS June 1993 not dropping rapidly. Stagnant network costs providing switched network services, their could slow demand growth. Second, most us- present commanding position in the leased ers may find their needs satisfied or mostly private line business gives local telephone satisfied by current or emerging technologies. companies few incentives to spread switched network services aggressively, leaving most Costs of Private Network Lines. One cause potential customers with expensive leased of the proliferation of computers has been the lines as their only option. rapid decline in computer prices. Prices of the metropolitan or regional area private tele- The long-distance carriers seem to be fol- phone lines used by many computer networks lowing the lead provided by the local tele- are under the control of local telephone com- phone companies in setting short-distance pri- panies, which have a near-monopoly on them. vate line prices. Although the prices posted The prices of these regulated lines, according with the Federal Communications Commis- to data filed before the Federal Communica- sion for private lines longer than 200 miles tions Commission, are falling by only a small have been declining since 1990, the prices of percentage a year, which is insignificant in leased private lines shorter than 100 miles comparison with the price declines in most have been rising. Thus, the average posted electronics hardware. Although potential price of a private 1,500,000 bit-per-second competitors have entered the market for local leased line provided by the major long dis- area lines, they represent only a fraction of to- tance carriers roughly the distance between tal capacity. Despite commitments by local New York and Chicago (700 miles) has been telephone companies to bring costs down by falling roughly 13 percent a year since 1990,

Summary Figure 1. History of Internet Traffic, 1988-1993

Millions of Packets of Computer Information

35'000 0

July January January January January January 1988 1989 1990 1991 1992 1993

SOURCE: Congressional Budget Office from data provided by MeritINSFNET, Ann Arbor, Michigan. SUMMARY while the price of the same capacity line 75 gued in their evaluation of HPCC that the miles long has been rising by 7 percent a year computing problems the program intends to since 1990. (The long-distance carriers also address, the so-called grand challenges, are provide time and quantity discounts, so no sin- too distant from the problems of daily life. In gle user may face precisely these prices.) Such their view, the program does not effectively price increases may affect the decisions of demonstrate ways in which high-performance small and medium-sized regional businesses, computer networks could actually help people. which are crucial in a growing market. In response to these arguments, the Con- By contrast, prices of most other compo- gress has been considering legislation to ex- nents of computer networks are falling. Con- pand the network component of HPCC to ad- sequently, overall computer network prices dress these problems. One approach would ex- are still coming down. As prices of other net- pand HPCC to create applications in four dis- work components fall, the prices of the leased tinct areas. To pay for these additional pro- lines will come to represent a larger fraction grams, the Congress would authorize an addi- at the whole network, which may slow the tional $1.0 billion to $1.6 billion over the next overall decline in network costs and slow the five years, targeted on four areas: medicine, growth in demand for network services. The education, manufacturing, and libraries. rapid rise in the number of computers and local area networks for computers, however, o In medicine, the bills would direct the Na- may serve to increase the demand for network tional Library of Medicine to develop net- services, even if prices do not fall. work applications for communicating medical images, such as X-rays and CAT Trade-offs Between Cost and Perfor- scans, by computer network; build test-bed mance. Above a certain technological level, networks to link hospitals and other medi- computer network users may find little advan- cal centers for the sharing of data and tage in higher performance speeds. The records; and help provide long-distance NSFNET switching facilities have been up- medical care. graded from 1.5 million bits per second to provide individual circuits capable of carrying 45 o In education, the National Science Foun- million bits per second. This new service will dation would be authorized to develop pilot make a difference in most current applica- projects to connect U.S. elementary and tions. Above this level, however, networks secondary schools to Internet, including reach a point of diminishing returns and ex- generating software and training materi- pansion from 45 million bits per second to 1 als. billion bits per second may not create notice- able benefits for the vast majority of users who o In manufacturing, the National Institute concentrate on electronic mail and simple file of Standards and Technology would transfers. Without new applications to justify develop and transfer electronically net- the cost, a mass market in such services may worked manufacturing applications, in- not develop. cluding development of standards.

o Both the National Science Foundation and the National Aeronautics and Space Ad- ministration would be made responsible Policy Directions for developing prototype digital libraries and the associated technology, including, The Computer Systems Policy Project, a group in the case of the National Aeronautics representing U.S. computer producers, has ar- and Space Administration, making spe- xvi PROMOTING HIGH-PERFORMANCE COMPUTING AND COMMUNICATIONS June 1993

, cialized government satellite data available over Internet. Conclusions Underlying the proposed expansion is an unresolved tension in the direction of HPCC. Is HPCC ultimately concerned with providing Most of the barriers to the commercialization networks for the federal missions that need to of HPCC technology discussed in this report link high-performance computers, or is its may be factors limiting the demand for these main objective to develop technology to make new technologies. New technologies often ubiquitous computer networks a reality? The founder on the demand side, when the benefits former implies high-speed links between im- they offer seem not to be worth the added ex- portant national research centers, while the pense for most potential users. Commercial latter emphasizes an inexpensive and easy-to- demand is, of course, not the only reason for use network available to a wider public. pursuing these technologies. In many cases, government agencies can use the capabilities Until now, the fact that parts of the HPCC of high-performance computing or commu- budget were dispersed among the "discretion- nications--indeed, that is one of the primary ary spending" portions of several agency bud- goals of HPCC. In the short term, however, gets mitigated this conflict among objectives. agency missions often conflict with commer- But given the cuts in discretionary spending cial requirements. mandated by the Budget Enforcement Act of 1990, policymakers may have to decide which The different elements of HPCC vary in aspects of HPCC are most critical. Since the their economic potential. Most obviously, the agencies centrally involved in HPCC--the Na- graduate students trained under the human tional Science Foundation and National Aero- resource programs are likely to find jobs in nautics and Space Administration--experi- teaching and in industry, where they will be enced real declines in their budget authority needed to educate the next generation of com- in 1993, they will have to make internal deci- puter programmers and to write commercial sions as to which of the different components computer programs. Other parts of the HPCC of HPCC will take precedence. program may not have as smooth a road. Chapter One Introduction

he federal government has fostered "grand challenges." These grand challenges the development of computer technol- include scientific problems such as studying ogy since World War 11.1 At the fron- ozone depletion and air pollution and model- tier of today's technology are high-perfor- ing the ocean. But they also include potential mance computers and the communications industrial applications such as designing networks that would allow more people to use drugs (rather than discovering them) and them. The Administration and the Congress modeling industrial chemical processes. Their have taken existing programs, provided addi- common ground is that the solutions to these tional funds, and molded a multiagency pro- problems should be both demanding on com- gram for research into and early development puter resources and valuable to the nation. of high-performance computers, the software they use, and the networks that link them. The program's mission, however, is not simply to devise the fastest computers and com- The High-Performance Computing Act of puter networks, but also to ensure that the re- 1991 established the High Performance Com- sulting technology is disseminated widely and puting and Communications (HPCC) program applied to national needs, including those re- to promote U.S. technological leadership in lated to economic competitiveness, national high-performance computing and data com- defense, education, and the environment. A munications. In programmatic terms, that key goal of the program is to have the technol- goal means developing computer architec- ogy become an integral part of the problem- tures--the design of a computer's functional solving process in U.S. firms, thus improving operations and logic system--that can perform U.S. productivity and competitiveness. Faster up to 1 trillion mathematical operations per and cheaper computer power could help design second, and computer networks that enable teams in their search for new drugs, airplane users to transmit or receive up to 1 billion bits designs, efficient and nonpolluting automobile of computer data per second. Concomitant engines, and so on. Program managers be- with this will be the development of software lieve that the improvements derived from this applications and the training of computer sci- new computer technology could allow U.S. entists and programmers who will use the new firms to compete more effectively in world technologies. markets.

The types of problems that require this level The HPCC program will clearly benefit the of computing power are referred to as the scientific and research communities. But will it have a major effect on the larger computing market beyond this initial user group? This question would not be as central if HPCC were 1. For a history of federal government involvement in com- solely a science program. The intent of the en- puter technology, see Kenneth Flamm, Creating the Computer: Government, Industry and High Technology abling legislation, however, is that HPCC will (Washington,D.C.: Brooking8 Institution, 1988). also help industry to create and sell better 2 PROMOTING HIGH-PERFORMANCE COMPUTING AND COMMUNICATIONS June 1993 computers and data communications equip- The principal federal uses of supercom- ment. Meeting the first goal--building a sci- puters have been in the design of nuclear entific instrument to be used by federal weapons and in intelligence. With the end of agencies--will be relatively easier to accom- the Cold War, the need for new nuclear weap- plish and provides a familiar role for the gov- ons has been substantially reduced. The intel- ernment. Meeting the second goal will be ligence function, especially the electronic sur- more difficult; past federal efforts in this direc- veillance that required supercomputers, has tion have had mixed success. also been affected, though less drastically. In addition, supercomputers are used in energy, This report focuses on potential obstacles environmental, and space missions, but these facing HPCC program managers in meeting have traditionally been secondary uses. Fed- the second goal, particularly on economic and eral activity in the fields of national security institutional barriers that may hamper the and pure science is outside the bounds of this creation of large commercial markets for the study, which focuses on the commercial as- technology developed in the HPCC program. pects of HPCC technology. Most of the barriers discussed in this report relate to the demand for this new technology. It Economists use two general arguments to is on the demand side, in fact, that many new justify financing R&D programs for technol- technologies founder: potential clients simply ogies that have potential commercial uses. do not consider the advantages provided by The first is the well-known tendency of private the new product or services worth the cost and firms to underinvest in R&D. Economists gen- effort of switching from their current way of erally argue that private firms, left to their doing things.2 Although there will also be own devices, tend to underfund R&D largely challenges on the supply side, HPCC is ad- because the innovating firm is unable to cap- dressing them directly through its research ture all the benefits from its R&D investment. and development (R&D) component. This report assumes that the program can produce The second argument invokes a traditional the desired technology. The question it seeks governmental role in the production of infra- to answer is, What happens when the R&D is structure. As computers become ubiquitous, successful? their ability to communicate through networks will benefit more and more of the economy, just as the proliferation of telephones early in this century made telecommunications an essential part of the U.S. economy. The Public Purposes for role of the federal government in helping to Funding the HPCC create that infrastructure--through R&D, demonstrations, regulation, and setting of Program standards--can be significant. (Questions of how the computer network infrastructure will The federal government is funding the HPCC be planned, financed, and provided are beyond program for two general reasons. First, such the scope of this study.) technology may help federal agencies to ac- complish some of their missions more effec- Governmental support for HPCC is also tively. Second, such funding is part of the driven by the perception that the development overall federal role of funding R&D. These ra- tionales, however, are shifting and are subject of computer technology has been good for the country and will continue to be so. Because to some qualification. the federal government played a significant role in the past development of the computer 2. Steven Schnaars, Megamistakes: Forecasting and the industry, many people believe that it can do so Myth of Rapid Technological Change (New York: Free Press, 1989). in the future. CHAPTER ONE INTRODUCTION 3

Commercialization Defined large percentage of all buyers, independent of the HPCC program. For this reason, and also In this study, the term "commercialization of because many statistical sources may not HPCC technology" refers to the development break out the relevant data into governmental of substantial nongovernmental markets for and nongovernmental sales, the definitions products incorporating technology developed used in this study have been broadened to in- under the HPCC program. A technology de- clude all sales--governmental and commercial. veloped for the HPCC program would be com- mercialized if it were sold directly in large numbers or if it were incorporated into various The HPCC Program as an products that were then sold in significant Investment quantities. Funding the HPCC program will be a substan- The HPCC program leadership takes a less tial investment and should be undertaken on- rigorous view of commercialization: it argues ly if the economic rewards are likely to provide that if a contractor to an HPCC agency devel- an acceptable rate of return to society as a ops technology for the government, the tech- whole. Improving computer technology itself nology is commercial because it is in the hands is not a sumcient rationale--computer technol- of a private party. Because most of the R&D ogy is already improving rapidly. Rather, the performed on hardware for supercomputers or HPCC program has to move computer technol- networks is being performed in the laborato- ogy ahead more rapidly than would otherwise ries of private contractors, the technology is by occur or in directions in which it would not this definition commercial, and the HPCC pro- otherwise move. gram has therefore succeeded in commercial- izing its technology. On previous occasions, federal agencies committed themselves to speeding up elec- The problem with this definition is that it tronic technology only to discover that com- ignores half of all markets--namely buyers. mercial electronic technology was moving Numerous technologies have been sold by con- even more rapidly.3 Improving technology is tractors to the government, but have had vir- not enough; the HPCC program must have a tually no commercial buyers. The National very good chance of substantially advancing it Aeronautics and Space Administration's con- more rapidly than would otherwise occur. tractors have failed to sell a single space shut- tle to nongovernmental buyers, for example. Similarly, the commercial benefits should Similarly, the Department of Energy's con- be in proportion to the size of the investment. tractors are not selling a lot of synchrotrons, A federal program that spends several billion although the expertise to build them is surely dollars to help create a market of a few hun- in private hands. Of course, weapons first de- dred million dollars a year may not provide a veloped for the Department of Defense have substantial social rate of return to the tax- been sold worldwide, but usually to other gov- payers. (On an annual basis, a few hundred ernments. million dollars would normally be a reason- able return on a few billion dollars, but these To judge the effect of the HPCC program on few hundred million dollars of sales would re- the development of computer and data com- quire substantially more than just the federal munications markets, which is the stated intent of the legislation, the appropriate analytic framework must involve private sales, 3. For a discussion of how civilian semiconductor technol- not just private supplies. In the case of super- ogy usually outpaces that of the military despite special initiatives, see Anna Slomovic, "An Analysis of Military computers and computer networks, however, and Commercial Microelectronics: Has DoD's R&D this definition may be too narrow. In both Funding Had the Desired Effect?" (Ph.D. dissertation, RAND Graduate School, Santa Monica, California. these markets, federal buyers represent a 1991). 4 PROMOTING HIGH-PERFORMANCE COMPUTING AND COMMUNICATIONS June 1993 investment. They would also require addi- $20 billion in 1970. Large, mature industries tional product development by private inves- are likely to be more difficult to stimulate tors, as well as investment in manufacturing technologically than small, developing ones. the devices. The return on HPCC would be net of the return on these subsequent invest- The federal role in computer R&D has been ments.14 decreasing for several decades. In the 19609, federal agencies spent between a quarter and a half of what private firms spent on R&D in the computer industry.7 But in the late 1970s and 1980s, federal agencies did not keep up The HPCC and Previous with the growth in private R&D in the computer industry: spending by federal agencies Federal Computer R&D was between 10 percent and 15 percent of the Programs private level.8 The absolute levels of the private R&D efforts rose throughout this period. Many economic studies of federal R&D pro- The National Science Foundation estimates grams have found that, unlike private R&D that U.S. computer manufacturers alone spent spending, most federal R&D funding does not roughly $10.5 billion of their own funds in provide a substantial economic rate of return.5 1989--the last year for which data are (Most federal R&D is mission-related and does available--on R&D for computers, compo- not result in increased economic output as nents, and software.9 By contrast, in 1975, measured by typical indexes.) By contrast, private firms spent $1.7 billion on such R&D. federal efforts in computer R&D have enjoyed Given the reduced share of federal R&D, as many technical and economic successes. well as the increased level of private R&D, one might assume that the federal role in promot- In the periods when the federal government ing technological innovation was also reduced. had many of its largest successes in computer R&D, the computer industry was in its in- Most important, the computer and data fancy. Now, however, that industry is very communications industries have become much large and quite sophisticated. U.S. producers of computers, software, and related services sold roughly $260 billion worth of goods and 7. Before 1972, the National Science Foundation had no separate R&D category for the computer industry. services worldwide in 1989.6 This amount Moreover, the NSF has not published federal spending compares with less than $5 billion in 1960 and for R&D in the computer industry for reasons of confi- dentiality since 1979. Thus, this analysis is based on the machinery industry, of which the computer industry is the largest component. The R&D and output statistics of 4. See Congressional Budget Office, "A Review of Edwin the machinery induetry are dominated by the computer Mansfield's Estimate of the Rate of Return from Aca- industry, according to the National Science Board, Sci- demic Research and Its Relevance to the Federal Budget ence and Engineering Indicators--1991 (1991),p. 424. Process," CBO Staff Memorandum (April 1993),for a dis- See Science Indicators for previous years. See also Na- cussion of rates of return on R&D. tional Science Foundation, "Research and Development in Industry," various years. Using these sources, CBO 5. For a review of this literature, see Congressional Budget was able to obtain a fairly consistent estimate of govern- Ofice, How Federal Spending for Infrastructure and ment and company R&D spending in the machinery in- Other Public Investments Affects the Economy (June dustry and, to a less extent, the computer industry. 1991), Chapter N. 8. As noted, these figures are for the larger machinery in- 6. CBEMA Industry Marketing Statistics Committee, The dustry; the computer industry R&D figures were prob- Information Technology Industry Data Handbook, 1960- ably closer throughout the entire period, while exhibit- 2001 (Washington, D.C.: Computer and Business Equip- ing the same downward trend. ment Manufacturers Association, 1991), p. 9. Other sources have a smaller, but still large, estimate of the 9. National Science Board, Science and Engineering Indi- U.S. computer industry. For example, McKinsey and cators--1991, p. 424. Software producers (federal and Company, the management consultants, estimate 1992 private) spent an additional $2.7 billion in 1989. See worldwide sales at $208 billion. See McKinsey and Com- Melissa Pollak, "Research and Development in the Ser- pany, 'The 1992 Report on the Computer Industry" (San vice Sector," The Service Economy (Washington, D.C.: Francisco, Calif., 1992). p. 2-13. Coalition of Service Industries, July 1991),p. 4. CHAPTER ONE INTRODUCTION 5 more competitive during the past decade. search and Education Network] component to Many different companies compete for a rela- include research and development on the tech- tively small part of this large market, and nologies needed to support broadly accessible many federal R&D and regulatory programs and affordable networks."lo have helped make this industry highly competitive. As a result, the pressures on firms in CSPP's specific concerns focused not so the industry are now quite different than much on computer power or network speed, when the computer industry was dominated but on those aspects that might impede the by one company and data communications by spread of network technology. These aspects another. In this new, more competitive mar- include security and privacy of data and indi- ket, the pressures on firms to introduce new viduals and issues of intellectual property technology are much greater than previously. rights, standards, and regulation. CSPP Consequently, federal policies to stimulate in- placed a great deal of emphasis on ensuring novation are less likely to play an important network compatibility and the broadest possi- role overall. ble access to the system.

The technical choices the HPCC program is making to serve the interests of federal computational science researchers may not trans- Alternative Strategies for late into features that the majority of commercial computer network users will find desir- the HPCC Program able. Although HPCC planners have ex- pressed concern about the solution of a broad The strategy of the HPCC program for com- range of problems, including some that could mercialization is consistent with a traditional contribute substantially to U.S. welfare, CSPP federal approach. It aims to develop technol- contends the initiative would directly contri- ogy to fulfill the federal mission by using a bute little to the lives of most citizens. The mix of in-house expertise and outside contrac- grand challenges are, in some sense, too tors. Like previous federal programs, it em- grand. phasizes a single aspect of the technology--in HPCC's case, speed. Demonstration projects CSPP would prefer to see more everyday ap- and other explicit efforts to transfer and com- plications of high-performance computing.11 mercialize the technology account for only a These uses would include giving broader ac- fraction of the resources devoted to the project. cess to electronic data bases, enhancing education through electronic field trips, and improving the life of housebound citizens. In the Development of Everyday manufacturing arena, CSPP places as much emphasis on the integrative functions of data Applications communications as on the "gee-whiz" aspects of designing products on computers. In this The Computer Systems Policy Project (CSPP), view, the HPCC program should not just aim an industry organization representing the to develop trillion-operation supercomputers chief executive officers of major U.S. producers of computer systems, presented a different strategy for HPCC. Its recommendations for 10. Computer Syetems Policy Project, "Expanding the Vi- sion of High Performance Computing and Communica- the network portion of the project included tions: Linking America for the Future" (Washington, suggestions to "ensure the HPCC serves as a D.C., December 3,1991), p. 11. stepping stone to broader future information 11. Computer Systems Policy Project, 'Terspectives on U.S. infrastructure. . . . This will require expanding Technology and Trade Policy: The CSPP Agenda for the activities under the NREN [National Re- 103rd Congress" (Washington, D.C., October 1, 19921, p. 8. 6 PROMOTING HIGH-PERFORMANCE COMPUTING AND COMMUNICATIONS June 1993 that talk to each other on billion-bit networks. those activities can pay the costs high-per- Rather, it should include a wide array of com- formance systems usually involve. Electronic puter resources, usable in a wide array of cir- field trips for schools or the disabled, as envi- cumstances, and applicable to a wider array of sioned by CSPP, will not be among the early national needs than originally envisioned. uses of HPCC technology. Ordinary citizens are most likely to encounter such technology In keeping with its alternative vision of the when they purchase a good or service that in- HPCC program, CSPP has recommended re- corporates it--for example, medical records arranging the program's budgetary priorities. and images that are stored and transmitted Whereas most of the current funding focuses electronically. on a single design of supercomputer, CSPP would like to see a broader range of hardware Indeed, HPCC technology is most likely to and software configurations. Second, the be commercially successful if it can first de- HPCC program's software budget contains velop a broadly based market of high-value re- funds to purchase a great deal of hardware, sources to store, distribute, and manipulate while CSPP would emphasize more software specialized information. By providing special- funding. Third, it would expand the compo- ized information equipment and other re- nents for developing basic research and hu- sources to organizations willing to pay for man resources. Finally, CSPP would shift the them, it may enable CSPP members and other program's balance from advancing technology firms to bring down costs to the point that to applying it. such equipment and resources become more widely used. Although the HPCC program is commonly viewed as a network project, only 20 percent of the new funds for HPCC is for networks. CSPP believes that the network component of HPCC has the greatest potential and sees its Competing Views of low share of funding as an example of the im- balance in the HPCC budget. Commercialization

In short, rather than being concerned about The differences in policy strategies discussed the program's meeting the grand challenges of above largely result from divergent views of the original HPCC documents, the CSPP is in- the process of technological change and the terested in ensuring that the technology has role of the government in it.12 One view fo- potential commercial use. Unless the new net- cuses on developing advanced technology, and work technology addresses the concerns of the other on diffusing that technology. their commercial customers, who represent the largest market share of demand for com- In the first view, the federal government de- puter network goods and services, CSPP mem- velops the technology for its own needs and bers feel it will not meet their needs. leaves its further application to others. This view recognizes that there are many institutional obstacles to creating a market for new technology and that such market development Development of Self-Sustaining is inherently risky. A basic tenet of this view Markets is that, other than in a few areas such as setting of standards, the federal government can To a degree, CSPP's vision contains contradic- tory notions. The earliest users of advanced computer technology are likely to be involved 12. For a more detailed discussion of different views of commercialization, see Congressional Budget Office, Us- in high-value activities, typically in business ing RhD Consortia for Commercial Innovation: and medicine or in government, since only SEMATECH, X-ray Lithography, and High-Resolution Systems (July 1990), pp. 5-10. CHAPTER ONE INTRODUCTION 7 do little to overcome these obstacles and parallel high-performance computers, the should leave their solution to the private sec- market success of which is still quite limited tor, which has profit as the motivation for cre- (under $300 million in yearly sales). Even ating markets and overcoming obstacles. with some of its successful projects, such as ARPANET--the first computer network-- In the case of HPCC, federal agencies will ARPA has sometimes had to continue support develop, or help industry to develop, very fast for years beyond its original intent before a computers and networks and software tools for self-sustaining community of users devel- application to agency missions, including ba- oped.14 sic university research. Their diffusion to the rest of the economy, however, is left to the computer industry and telecommunications The Comparative Advantage of providers. the Government

In the other vision of technological change, One trade-off that the Congress and the Ad- the federal government has a role to play in ministration face in the different facets of overcoming obstacles that could keep technol- HPCC is the choice between pursuing near- ogy from spreading throughout the economy. term commercial implementation of HPCC The government might help to develop specific technology and emphasizing the highest tech- uses that could speed the spread of technology nical performance. For example, federal re- broadly into the economy--for example, devel- searchers stand at a comparative advantage oping less expensive, rather than faster, data in basic research on new ways of conceptualiz- communications technology, or making soft- ing computer software. The reward system for ware easier to use. In this view, developing federal R&D is based on technical excellence, new markets and infrastructure is as filled often through peer review. By contrast, com- with risk, uncertainty, and socially beneficial mercial providers of software achieve success-- spillovers as is developing new technology. that is, profits--by being able to deliver soft- Given the costs, which are concentrated and ware that addresses users' immediate con- large, and the social benefits, which are dif- cerns. Thus, although providing the next gen- fuse but in the aggregate can be greater than eration of hardware or software may ensure the costs, the federal government may in this more commercial success for the technology view be uniquely positioned to sustain the in- developed under the HPCC program, federal vestment necessary to spread new technology researchers may not be in the best position to throughout the economy. do that.15 The federal track record in bringing new Conversely, however, because HPCC pro- computer technology to market is mixed, how- gram-sponsored supercomputers lack a large ever. The Advanced Research Projects Agency supply of commercial software, the sales of (ARPA) has certainly had much success in such computers are limited mainly to aca- many areas of computing and electronics.13 demic and government R&D centers. If the But it also has a 20-year history of promoting community of users is not sufficiently developed, the loss of federal backing could hurt the

13. The Defense Advanced Research Projects Agency (DARPA) recently changed its name to Advanced Re- search Projects Agency (ARPA), the name it was known by before 1972. For the role of ARPA in the development 14. Advanced Research Projects Agency, "The Advanced Re- of computer technologies, see Richard Van Atta, Sidney search Projecta Agency, 1958-1974" (produced under Reed, and Seymour Deitchman, DARPA Technical Ac- contract by Richard J. Barber Associates, Washington, complishments: A Historical Review of Selected DARPA D.C.,December 1975), pp. IX-57 through IX-59. Projects, Volumes I, ZZ and ZZZ (Alexandria, Va.: Institute for Defense Analysis, February 1990), especially Chap- 15. For a similar argument with regard to supercomputers, ters 18 through 23 of Volume I and Chapters 13-19 of see Willie Schatz, "DARPA Rigs the Playing Field," Up- Volume 11. side (May 1993), pp. 38-48. 8 PROMOTING HIGH-PERFORMANCE COMPUTING AND COMMUNICATIONS June 1993 efforts of private companies to create a market But ARPA did its work without being directly for their products. concerned with the sales of the resulting product. This pattern of federal comparative advantage can be seen in other computer R&D ef- Program Risks forts. In many of its more recent successes in high-performance computing, ARPA paid for By trying to be all things to all people, the university research to build a prototype and HPCC program might lose its focus and dissi- then left it to private initiative to bring this to pate its resources without effect. By focusing market. University researchers operating on on one fairly narrow range of computing ARPA funds originally designed much of the needs, the program may be more likely to ad- technology underlying the specialized graph- vance the technology. The risk is that the ics, software, integrated circuits, and high- technology will be irrelevant. performance computers now commonly used in scientific and engineering computing cir- No strategy the HPCC planners choose will cles.16 Without ARPA funding, it is fair to ar- be without risk. On the one hand, federal gue, high-performance computing would prob- R&D could be too near the marketplace, dis- ably be much less developed than it is today. torting choices and preempting private actions. On the other hand, the HPCC program could abandon such R&D prematurely to focus only on longer-term technology, leaving this 16. Van Atta, Reed, and Deitchman, DARPA Technical Ac- technology an orphan, and perhaps stunting complishments, Chapter 17 of Volume II. the growth of this market. Chapter Two The HPCC Program

olicymakers designed the High Per- cluded in the HPCC tallies. For instance, the formance Computing and Communica- original 1992 plan included only $17 million tions program to improve the technol- for the National Library of Medicine, but the ogy of supercomputers and data communica- 1993 request included other parts of the Na- tions. This program focuses on hardware and tional Institutes of Health, bringing the total software for supercomputers, data communi- to $45 million. The National Oceanic and At- cations systems, and basic research and train- mospheric Administration (NOAA) also in- ing. Several principal federal agencies are increased its participation. The National Secu- volved in developing different aspects of the rity Agency joined the HPCC program, adding technology. $22 million to the total.

The second set of changes in the plan have resulted from policy decisions to change funding. For example, to allow for growth in fund- The HPCC Budget ing for the space station within a constrained overall budget, the National Aeronautics and The HPCC program is essentially a continu- Space Administration (NASA) reduced its ation of ongoing federal work in the area of HPCC research and development for 1993 by computer research and development with ad- $26 million. The Advanced Research Projects ditional resources and more emphasis on co- Agency has cut its funding by $8 million below ordinating federal efforts. Thus, the HPCC the plan. The most significant change, how- budget includes both existing and incremental ever, has been in the National Science Foun- funds. dation's appropriations for research and related activities for 1993, in which its HPCC activities received only a 12 percent increase The 1992 Plan rather than the planned increase of 23 percent.

The program is intended to double the federal Despite these changes, no new baseline bud- resources being devoted annually to develop- get exists for HPCC as a whole. The agencies ing high-performance computers and com- regard their five-year budget projections for puter networks over the 1992-1996 period. the program as confidential. The original funding request for 1992 was $640 million, rising to $1.2 billion in 1996. Funding over the five years would total $4.7 Incremental Funding billion (see Table 1). Establishing a firm estimate of the costs of the Since 1992, two factors have intervened to different components over the duration of the change the budget, though not substantially. program is difficult. As noted above, this ini- First, other existing programs have been in- tiative includes many ongoing programs, some 10 PROMOTING HIGH-PERFORMANCE COMPUTING AND COMMUNICATIONS June 1993 new ones, and additional resources that have been diverted to existing programs. Agencies may choose which of their programs to include High-Performance in the HPCC. Specific agency participation is Computing Systems reflected in various pieces of authorizing legislation, if at all. Consequently, there is no sin- The goal of the HPCC program with regard to gle complete statement of the HPCC budget. computer hardware is to develop computer architectures that can achieve a trillion math- Details on the incremental funding that ematical operations per second--two or three was included in the 1992 plan are presented orders of magnitude above current perfor- by program area, not by agency. Not all pro- mance levels. The systems that seem most gram components received equally increased likely to develop this power first are called funding in the original plan. Table 2 presents massively parallel supercomputers. Conven- the level of additional funding discussed in the tional supercomputers have a small number-- HPCC planning documents. usually fewer than 16--of very fast processing units working on a problem. In contrast, the Incremental funds totaled $1.9 billion over massively parallel systems can have several five years. The programs for High-Perfor- hundred or even several thousand slower pro- mance Computing Systems (HPCS) and Ad- cessing units. vanced Software Technology and Algorithms (ASTA) each received slightly more than one- What distinguishes the scalable architec- third of the funds. The National Research and tures that the HPCC program is concentrating Education Network accounted for 20 percent, on from other massively parallel supercom- and Basic Research and Human Resources for puters is the idea that the same basic com- the rest. puter architecture should be able to be scaled

Table 1. Original Five-Year Budget Plan for the HPCC Program (Budget authority in millions of dollars, by fiscal year)

Agency 1992 1993 1994 1995 1996 1992-1996

Advanced Research Projects Agency

Department of Energy

Environmental Protection Agency 5 5 5 5 5 25

National Aeronautics and Space Administration 7 2 107 134 15 1 145 609

National Library of Medicine 17 17 17 17 17 85

National Institute of Standards and Technology 3 3 3 3 3 15

National Science Foundation 213 262 305 354 41 3 1,547

National Oceanic and Atmospheric Administration -2 -3 -3 -3 -3 14

Total 638 790 958 1,089 1,201 4,676 SOURCE: Congressional Budget Office using data from the Office of Management and Budget. NOTE: includes incremental funding. CHAPTER TWO THE HPCC PROGRAM 11

a few, many, and very many. Most important Table 2. software applications will be perfectly trans- Incremental Funding for the HPCC Program, portable between them, merely running faster Fiscal Years 1992-1996, by Program Component as more processing nodes are added to the Fundinq computers. As a In Millions Percentage Most modern U.S. top-of-the-line supercom- Component of Dollars of Total puters, including conventional supercomputers, are parallel in the sense of having more High-Performance than one processor, and the trend is clearly to Computing Systems 682 36 increase the number. Where the technology Advanced Software strategies differ between conventional and Technology and massively parallel machines is in how fast to Algorithms 662 3 5 increase the number of processors. To achieve National Research and the highest peak speeds, HPCC has opted to Education Network 390 20 increase it rapidly. Basic Research and Human Resourcesa 183 10 The type of hardware-scaling that HPCC is exploring can, to some extent, be replicated in Total 1.91 7 100 software today. Although many high-perfor- SOURCE: Congressional Budget Office using data from Office mance computer systems are not scalable, of Science and Technology Policy, "The Federal High Performance Computing Program" (September 8, they have software that is compatible with 1989), p. 46. that of more powerful computers. For exam- NOTES: HPCC = High Performance Computing and Commu- ple, the entry-level Cray Y-MP EL--a conven- nications. tional supercomputer--has roughly 1 percent Incremental funding is the additional money appro- priated to the HPCC agencies above the level the to 3 percent of the processor performance of agencies were receiving at the start of the HPCC pro- the top-of-the-line Cray Y-MP C90 at between gram for their activities in this area. Numbers may 1 percent and 3 percent of the cost. They both not add to totals because of rounding. can run the same software, but at different a. In addition, 15 percent of each of the other three program components are to be devoted to this area. speeds. Conventional supercomputers, however, cannot be upgraded easily. Buying a linearly until it is the right size for the task at new machine is easier and less costly than hand. Scaling refers to the addition of proces- making an existing machine work more rap- sors to the system so that the system's perfor- idly. Computers with scalable architecture, mance increases in proportion to the number therefore, allow users flexibility in tailoring of processors added. the hardware, and consequently the speed, to the task at hand. Computer design visionaries believe that the proliferation of computer designs using the same microprocessors is part of the new Components wave in computer architecture.1 In the future, they argue, computers will not be cate- The development of supercomputer hardware gorized as a microcomputer, minicomputer, focuses on four areas: mainframe, supercomputer, and so on, each with a different architecture. Rather, comput- o Research for future generations of computers will be characterized by the number of ing systems addresses the concepts under- standard microprocessors in the machine: one, lying advanced scalable architecture, components, packaging, and systems software. The goal of this research is to ensure that 1. David Patterson, "Expert Opinion: Traditional Main- frames and Supercomputers Are Losing the Battle," the requisite technology is available for ZEEE Spectrum (January 1992), p. 34. new systems and to provide a solid techno- 12 PROMOTING HIGH-PERFORMANCE COMPUTING AND COMMUNICATIONS June 1993

logical base for the next generation of high-performance computers. Advanced Software o System design tools focuses on the develop- Technology and ment of computer-aided design tools and frameworks for integrating multiple tools. Algorithms The tools include computer-aided design, analysis, simulation, and testing. The Because these massively parallel supercom- frameworks are to allow computer design- puters operate under a fundamentally differ- ers to take data or output from one com- ent architecture than other computers, they do puter design tool and integrate it as seam- not have the software library, developed over lessly as possible into other design tools. 40 years, that other computers inherit. This The intent of this effort is to speed up con- element of the HPCC program will produce ge- struction of prototypes. neric software and algorithms for research applications. It also contains funding to create o Advanced prototype systems encompasses complete supercomputer applications that can the development of experimental systems. work on supercomputers attached to net- The dual goals of these activities include works. The goals are to help solve the applica- both increasing the speed and reducing the tion problems posed by the grand challenges, size and cost of the scalable supercomputer make the software easier to use and more reli- hardware, thus making it accessible to a able, and apply the knowledge so gained to broader class of users. The federal govern- other computational problems. ment and the supercomputer vendors will share development costs. Components o Evaluation of early systems places experimental machines with experts who use Development of the software for high-perfor- and evaluate them. Such evaluations in- mance computers covers four areas: clude establishing universally agreed- upon performance criteria and widespread o Software support for grand challenges publication of results. Some of the early aims to develop high-performance soft- experimental systems may begin work on ware to address issues related to the grand the grand challenges discussed in Chapter challenges, including global climate 1, such as some experiments in alleviating change, advanced materials research, and pollution or in high-energy physics. biotechnology. The problems are being selected on the basis of their national importance, their potential to advance computa- Budget tional technology, their potential to yield spill-over benefits, and the extent to which The importance of the hardware component of other parties--particularly private firms-- the HPCC program is reflected in its share of might share the development costs. overall funding. In the original budget plan, hardware accounted for roughly 36 percent of o Software components and tools seeks to de- all additional funds, or $680 million over five velop applications for high-speed computer years. It was scheduled to increase steadily in networks and generic software for high- annual increments of between $30 million and performance computers. Such software in- $50 million through 1996. More than two- cludes that which currently is available for thirds of the hardware funding, roughly $460 conventional computers--for example, soft- million, was for the manufacture and testing ware that enables programmers to design of prototypes of advanced-design supercom- fast software or helps them find errors in puters. the software they have written. Because CHAPTER TWO THE HPCC PROGRAM 13

HPCC involves many agencies and private tions industry by using the research and edu- vendors, this software is intended to oper- cation communities as the first customers--the ate on different brands of supercomputers. communities in which the applications and other ancillary technologies are most likely to o Computational techniques funds research be developed. on algorithms, parallel languages, and numerical analyses. The advent of massively parallel supercomputers requires the de- Components velopment of techniques optimized in ways different from those of single-processor This portion of the HPCC program has two machines. components: o High-performance computing research cen- o The Interagency Interim National Re- ters supports the deployment of new HPCC search and Education Network is based on computer hardware to research centers the National Science Foundation Network and provides fresh opportunities for re- (NSFNET), the Department of Energy search as well as transfer of technology to (DOE) Energy Sciences Network, the Na- large numbers of researchers. The intent tional Aeronautics and Space Administra- is to provide these centers with the new tion's Science Internet, and other networks hardware and connect them through the supporting research and development. National Research and Education Net- This component includes a phased process work. Doing so will permit a wide range of of upgrading and expanding the U.S. por- software experiments. tion of the research-oriented Internet so that it can serve more of its intended community with, insofar as feasible, data Budget transmission rates of 1 billion bits (one gigabit) per second.2 The interagency net- The software element accounted for about 35 work has been upgraded from transmis- percent of the incremental HPCC funding, or sion speeds of 1.5 million bits per second, roughly $660 million. About half of that will commonly called T-1, to 45 million bits per be spent for R&D on tools and techniques, and second, commonly called T-3. The DOE is about 30 percent will fund the research cen- also upgrading its research network. ters. The grand challenges themselves account for less than a quarter of incremental o Gigabit research and development funds spending for software. R&D on network switches, protocols, and other hardware and software needed to de- ploy the initial gigabit network. Much of this R&D is to be accomplished through six joint privatelpublic research test beds. A National Research and key feature of this program is its close col- laboration with industry to encourage the Education Network transfer of the resulting technology to the U.S. telecommunications industry. The National Research and Education Net- work (NREN) is intended to make data communications services available to more researchers and to government and educational

institutions. It would also increase the speed 2. Internet is an international network of computer net- and quality of such services. A by-product of works all using a common communications protocol and roughly centered in and around research and educa- this effort would be the further development of tional institutions. For a more substantial discussion of the commercial high-speed data communica- Internet, see Chapter 4. 14 PROMOTING HIGH-PERFORMANCE COMPUTING AND COMMUNICATIONS June 1993

Budget Budget

In the plan, the network portion of the HPCC In the original plan, only 10 percent of all new program would receive only 20 percent--or funds for HPCC were directly assigned to ba- $390 million--of the incremental funding. sic research and human resources. However, This funding would be equally divided be- HPCC planning documents stipulated that 15 tween setting up the interim NREN and doing percent of HPCS, ASTA, and NREN funding R&D on the next generation of networks. would be devoted to this general area.

Basic Research and Agency Responsibilities Human Resources The HPCC program divides its tasks not only The central goal of this portion of the HPCC into the four elements, but also along agency program is to encourage the long-term growth lines. The program includes eight agencies of high-performance computing by ensuring with some level of responsibility for portions of that the infrastructure for basic research is de- HPCC R&D (see Table 3). Although the agen- veloped and that students now being trained cy count has been rising, there is no single as the next generation of programmers have lead agency. The Office of Science and Tech- the opportunity to pursue such research. nology Policy is responsible for coordinating the individual agency efforts.

Components The Advanced Research Projects Agency has major responsibility for R&D on hardware This element of the HPCC program has four and generic software tools for both scalable components: supercomputers and networks. ARPA is funding the development of scalable massively par- o Basic research aims to increase awards for allel supercomputers and much of the basic R&D on high-performance computing ini- software needed to operate these new systems tiated by individual investigators. Such successfully, such as tools for debugging and basic research may relate to one or more of compiling software. ARPA is also paying half the three main HPCC areas--massively the federal portion of the gigabit test beds, parallel supercomputer hardware, soft- where much of the network hardware R&D ware, and networking. will occur. o Research participation and training pro- The Department of Energy's mission and vides more advanced training through history has endowed it with much of the fed- workshops, fellowships, and other activi- eral government's expertise with supercom- ties at the postsecondary level. puters. DOE therefore has the task of evaluating the scalable supercomputer systems devel- o Infrastructure provides upgrades of both oped under the HPCC program. The depart- equipment and software for computer sci- ment is also the potential user of these super- ence research facilities at universities. computer systems for research into nuclear weapons, global warming, and other energy o Education, training, and curriculum in- matters and is responsible for developing soft- volves the preparation of materials and ware applications in these areas. As a poten- courses for improving computer education, tial user of high-speed computer networks, including more teacher training. DOE will begin a major upgrade of its Energy CHAPTER TWO THE HPCC PROGRAM 15

Sciences Network to link its facilities as part The other agency with responsibility for of the HPCC program and will probably be the overall hardware is the National Science first federal agency to provide near-gigabit Foundation (NSF). Its principal responsibility speeds over a network. is to coordinate deployment of the National

Table 3. Agency Responsibilities Under the HPCC Program

Advanced Software National High-Performance Technology Research and Basic Research Agency Computing Systems and Algorithms Education Network and Human Resources

Advanced Research Technology development Technology development Technology development University programs Projects Agency and coordination for for parallel algorithms and coordination for supercomputer systems and systems software gigabits networks

Department Technology development Energy applications Access to energy Basic research and of Energy Systems evaluation research centers research facilities education programs Energy grand challenge and data bases and computation Gigabits applications research research Software tools

National Aeronautics and space Software coordination Access to aeronautic and Research institutes Aeronautics application test beds Computation spaceflight research and university and Space Research in: centers block grants Administration Aerosciences Earth and space sciences

National Science Basic architecture Research in: Facilities coordination Programs in: Foundation research Software tools, and deployment Basic research Prototyping data bases Gigabits research Education experimental Grand challenges Traininglcurricula systems Computer access

National Institute Research in systems Research in: Coordinate performance of Standards instrumentation and Software indexing and assessment and standards and Technology performance measurement exchange Programs in protocols Research in interfaces Scalable parallel and security and standards algorithms

National Oceanic Ocean and atmospheric Oceanographicand and Atmospheric computation research atmospheric mission Administration Software tools facilities Techniques Data-baseaccess

Environmental Research in environ- State environmental Technology transfer Protection Agency mental computations, mission assimilation to states data bases, and appli- University programs cation test beds

National Institutes Systems evaluation Medical application Development of Basic Research of Health and performance test beds for intelligent gateways Internships for parallel analysis medical computation Access for academic algorithm development research medical centers Training and career development SOURCE: Congressional Budget Office based on data from the Federal Coordinating Council for Science, Engineering, and Technology. 16 PROMOTING HIGH-PERFORMANCE COMPUTING AND COMMUNICATIONS June 1993

Research and Education Network in several high-priority programs, but no one has the au- stages, depending on the pace of the R&D ac- thority to decide HPCC funding trade-offs complishments. NSF also houses the basic re- among agencies. search efforts in both hardware and software. Much of the high-speed data communica- The remaining agencies' responsibilities for tions work being done within each agency is developing hardware and related software ap- also not necessarily represented at FCCSET. plications are largely associated with their in- For example, the National Library of Medi- dividual missions: NASA, space; NOAA, cine represents the National Institutes of oceanographic and atmospheric research; the Health, even though much of the work regard- Environmental Protection Agency, the envi- ing medical imaging technology at NIH is ronment. Only the National Institute of Stan- done elsewhere and, in the past, had been dards and Technology, which sets some net- done independent of the HPCC program. work standards, has crosscutting responsibil- Similarly, efforts by the armed services to ity. transmit medical images at relatively high speeds have been undertaken outside HPCC. (Part of the problem in describing the HPCC Role of the Federal Coordinating budget comes from the fact that member agen- Council on Science, Engineering, cies continue to increase the portion of their and Technology activities described as belonging in the HPCC program.) The federal entity responsible for coordinating HPCC research and development is the Fed- The HPCC program has addressed this eral Coordinating Council for Science, Engi- problem by creating a National Coordination neering, and Technology (FCCSET) within the Office for High-Performance Computing and Office of Science and Technology Policy. Communications, which provides the Con- Agencies participating in HPCC meet regu- gress and the public with a point of entry into larly under the auspices of FCCSET. How- the initiative. The National Coordinating Of- ever, FCCSET has no budgetary power or oth- fice is responsible to a FCCSET committee and er authority to realign priorities or make bind- its director is a special assistant to the Direc- ing decisions. Agencies attend meetings and tor of the Office of Science and Technology present proposals for new R&D on data com- Policy. This new office is intended to coordi- munications or scalable supercomputers, but nate interagency budget and reporting re- FCCSET neither sets priorities nor decides quirements, but it has neither a budget nor a which programs should go forward. Thus the full-time staff. Furthermore, the director is National Aeronautics and Space Administra- also the director of the National Library of tion can decide whether to reduce HPCC fund- Medicine, which presumably will continue to ing to accommodate the space station or other occupy a substantial portion of his time. Chapter Three HPCC and the Development of Supercomputer Markets

ne intent of the High Performance puter hardware, HPCC will most likely de- Computing and Communications pro- liver a prototype of a supercomputer with a 0 gram is to ensure continued leadership peak performance of 1 trillion operations per of U.S. firms in the high-performance comput- second. Whether this machine will be able to ing arena, including supercomputers and oth- perform useful work for a wide variety of users er fast computers, in the belief that such lead- is more open to question. ership would enable these firms to increase their performance in the other segments of the computer market. Policymakers also ex- The Promises of New pect to make the benefits of additional com- Supercomputer Technology puter technology more widely available by stimulating development of more applications HPCC planners expect to produce supercom- for supercomputers. They hope that increas- puters that cost the same as today's supercom- ing access to high-performance computers and puters but deliver between a hundred and a increasing the quality of the software for thousand times the power. This added power those computers will accelerate the wide- would permit supercomputers to solve prob- spread diffusion of this technology throughout lems that are beyond the capabilities of cur- the economy. rent supercomputers. Such solutions might give U.S. industry a competitive edge by enhancing the design and analytic capabilities of U.S. firms. From a policy perspective, then, HPCC and HPCC will yield substantial benefits if the power of the scalable massively parallel super- Supercomputers computer expands the uses and, hence, the market for high-performance computers. In The stated goal of HPCC program officers is to this sense, HPCC is trying to shift both the apply leverage to federal monies by funding supply curve for supercomputers (that is, ex- only those areas where they feel private dol- pand the supply at a given price) and the de- lars are not likely to go and that appear to be mand curve by creating new uses for the im- on a path of rapid technological advance. Con- proved supercomputers. sequently, they are more likely to fund revolu- tionary than evolutionary advances. The new supercomputer technology could also directly strengthen the competitiveness of To date, the HPCC supercomputer program the U.S. computer industry. Planners believe has developed some early prototype systems that supercomputer technology leads all com- that operate at 100 billion operations per sec- puter technology--that is, improvements ap- ond. Given the rapid rate at which semicon- pear on these machines first, are tested and ductor technology advances and the intense improved, and then move onto other comput- competition among producers of supercom- ers. In this way, the program would further 18 PROMOTING HIGH-PERFORMANCE COMPUTING AND COMMUNICATIONS June 1993 enhance the lead of U.S. technology in com- cial data-base management as specialized puters overall. (single-purpose) machines. Half of all mainframe computer use involves data-base manipulation; even if their only sales were to cus- Unsettling Questions tomers with such uses, parallel architecture computers would have a substantial market. This vision of success is not in complete corre- Whether they can move into general-purpose spondence with the market for high-perfor- use is at present unclear. mance computers. Far from driving computer technology, supercomputers have largely been involved in areas of technology that are not growing rapidly. Even now, massively parallel supercomputer technology draws largely Analysis of the on mainstream computer component technol- Supercomputer Market ogy rather than leading it. But it would appear that HPCC has the potential to improve The supercomputer market represents only a software and processor communications tech- minute fraction of annual computer sales and nology in ways that might have wider applica- an even smaller fraction of installed comput- tion. ers. Roughly 10 million to 15 million personal computers are sold each year. Engineering HPCC's vision of the development of com- workstations account for 300,000 to 400,000 puter technology also ignores the most impor- units. Computer producers also sell 10,000 to tant development in high-performance com- 20,000 mainframe computers each year.1 puting of the 1980s: the engineering workstation. Sales of these technical computers--high- By contrast, only a few hundred supercom- performance versions of the personal computers are sold each year.2 These machines puter--have grown substantially, just as su- have a high value, however, totaling about percomputer sales have slowed, suggesting $1.8 billion in worldwide sales in 1992.3 As of that workstations are competing with super- 1990, slightly more than 600 supercomputers computers for many uses. If this analysis is were operational worldwide.4 correct, HPCC is attempting to increase computer speed at a time when the market, in- Growth in the demand for supercomputers cluding the scientific market, may be valuing has slowed substantially in recent years. Ac- other factors such as cost and convenience cording to one recent industry estimate, the more highly than raw speed. world market for supercomputers grew from sales of less than $90 million to more than $1 Finally, the massively parallel supercom- billion between 1980 and 1988. Since then, puters, after a decade of federal R&D support, have yet to demonstrate their utility as general-purpose computers. Translating ex- 1. Egil Juliueeen and Karen Juliueeen, The Computer In- isting software for them has proved difficult. dustry Almanac, 1989 (New York: Simon & Schueter, For a variety of reasons that will be explored 1988), pp. 3.4-3.5. below, they match the performance of conven- 2. John B. Jones and Theresa Liu, "Cray Computer Corpo- tional supercomputers only in the most spe- ration" (Montgomery Securitiee, San Francisco, Calif., cialized problems. They are generally cheap- July 23,19911, p. 4. er, however, than the high-end conventional 3. Department of Commerce, 1991 U.S.Industrial Outlook supercomputers and may gain a following on (1991), p. 26-6. that basis. 4. heAlamoe National Laboratory, Department of En- ergy, "High Performance Computing and Communica- tions: Investment in American Competitiveneee" (pre- Massively parallel computer systems al- pared under contract by Gartner Group, Stamford, ready have a substantial market in commer- Conn., March 15.1991), p. 75. CHAPTER THREE HPCC AND THE DEVELOPMENT OF SUPERCOMPUTER MARKETS 19 however, demand for supercomputers has grown only 6 percent annually.5 Demand for Figure 1. entry-level supercomputers, which are typi- Supercomputer Market Segments cally much less expensive and almost as powerful, has remained high.

The recent slowdown is attributable in part to the decline in federal demand brought about by the end of the Cold War and the re- cession, but also to the increased power of engineering workstations, which will be discussed below.6 Assuming that defense spending continues to decrease throughout the 19909, demand for supercomputers may not resume its upward climb for 5 to 10 years unless emerging environmental or other applica- Vector Supercomputers Minisupercomputers tions grow sufficiently to replace military de- Massively Parallel Vector Facilities mand. Supercomputers

SOURCE: Congressional Budget Office from Department of Commerce. Supercomputer Market Japanese and, until recently, U.S. firms.7 In Segments 1991, roughly 60 of these vector supercomputers were shipped worldwide.8 The supercomputer market has four general submarkets: vector supercomputers, mini- The minisupercomputer (or departmental supercomputers, massively parallel super- supercomputer) is sold to cost-sensitive users computers, and vector facilities (see Figure 1). or those with limited needs.

Vector supercomputers usually consist of The largely experimental massively par- between 1 and 16 very fast processors with allel machines using arrays of inexpensive special features that allow them to work on microprocessors have started to sell more strings of numbers (vectors) quite rapidly. widely. Currently, most such sales are for The market for large vector supercomputers is smaller versions of the massively parallel ma- traditionally associated with the name Cray chines with fewer than 50 or 100 microproces- Research, but is also occupied by a handful of sors. Such configurations allow purchasers to experiment with the technology at relatively low cost. 5. Los Alamos National Laboratory, "High Performance Computing and Communications," p. 73. Other analyses differ in both absolute size and rate of growth. However, At the rapidly growing end of the market is there seems to be more general agreement that growth the vector facility, which takes a mainframe in the demand for the largest supercomputers has slowed. See Department of Commerce, U.S.Industrial computer and supercharges it with special ca- Outlook 1991, p. 28-5. At least part of the difference in pabilities. Vector facilities can rapidly do size of total market comes from the imprecise nature of the definition of the term supercomputer. 6. Between 1984 and 1989,public agencies bought an aver- 7. Sometimes vector supercomputers are also called vector age of 17 of the top-of-the-line supercomputers per year, parallel supercomputers because many of them have be- compared with only 11 in 1990. Private purchases fell tween 4 and 16 processors. Most of the time each proces- from an average of 13 to 10. See Department of Com- sor is assigned to a different user--that is, they are not merce, Technology Administration, "Global Markets for used in parallel. Supercomputers: The Impact of the U.S.-Japan Su- percomputer Procurement Agreement" (October 1992), 8. Department of Commerce, 1992 U.S.Industrial Outlook p. 14. (1992),p. 27-6. 20 PROMOTING HIGH-PERFORMANCE COMPUTING AND COMMUNICATIONS June 1993 computations on strings (vectors) of numbers. flected price increases or increases in average The rest of the computer program remains un- computer size; unit sales rose only 8 percent. affected. Since programs that need supercom- Most demand is still in federal laboratories or puters typically have large strings of computa- supercomputer centers, where the machines tions, vector facilities can give a mainframe are available as part of the general computa- some supercomputer capabilities. The Depart- tional services the centers provide. Commer- ment of Commerce data do not break out the cial users are still relatively rare. Although cost of the vector facility from that of the there has been a great deal of discussion about mainframe computer. "massively" parallel machines, most units currently sold can best be described as "mod- erately" parallel--that is, they have fewer Sources of Supercomputer than 64 processing nodes. Their moderate Demand parallelism reduces the size of the initial investment and the complexity in testing this new technology. The government is the largest buyer of the large vector supercomputers and the massively parallel processor-based supercomputers. This demand has been driven largely by the need to design nuclear weapons at the De- partment of Energy's national laboratories. The Supercomputer Since the national laboratories were the first Market and Technology to adopt many of the new computing technologies, their use carried with it a stamp of ap- Spinoffs proval. In addition, the national laboratories served the socially useful function of being on Supercomputers are widely perceived to be the the so-called "bleeding edge" of technology: motivating force behind advances in computer they made the mistakes and absorbed the technology.10 According to this view, new costs of making the new models of supercom- computer technologies are introduced through puters usable. this small but crucial market at the apex of the technology pyramid; successful technol- At the opposite end of the spectrum are very ogies then trickle down to the broader market price-sensitive users of supercomputers. at the base. Although the market for super- These users, typically private firms, have a computers is small, it is so performance-driven more mundane use for the supercomputer: de- that it allows computer companies to test new signing better products. The cost-sensitive us- ideas, thus providing a platform for new tech- er currently has three options: the minisuper- nologies that mainstream computer users will computer, a vector facility attached to a main- see only later. frame computer, or a high-end workstation. Despite the presence of high-end worksta- A practical corollary of this view is that in- tions, the low end of the supercomputer mar- vestments in supercomputer technology may ket is growing more rapidly than the high end. provide society with a higher return than similar investments in other parts of computer The market for massively parallel super- technology because these investments will computers has been growing rapidly, up trickle down to other users. In this way, ad- roughly a third in 1992 to almost $300 mil- herents believe, federal investments in super- lion, according to Department of Commerce computer technology may benefit U.S. com- estimates.9 Most of this growth, however, re- puter manufacturing firms by providing them

10. For example, see Ofice of Technology Aeaeeement, Com- 9. Department of Commerce, 1993 U.S.Industrial Outlook peting Economies: America, Europe and the Pacific Rim (1993),p. 26-7. (October 1991),p. 253. CHAPTER THREE HPCC AND THE DEVELOPMENT OF SUPERCOMPUTER MARKETS 21 with access to technology before their foreign competitors. Figure 2. Relative Computer Market Sizes Contradicting this view, however, is the secondary role supercomputer technology has played in two major advances of the 1980s: improved integrated circuits, and reduced-instruction-set computer (RISC) architecture.11 These advances have been driven instead by the other end of the spectrum of computer technology--personal computers and engineering workstations, Of course, some new technology has come from supercomputers, but the major thrusts of computer technology development lie elsewhere. The workstation and personal computer markets have been at the heart of technology advances because the demand for such computers has grown rapidly enough to justify devoting resources to solving the problems these markets confront (see Figure 2). Around 1980, all three market segments--personal comput- SOURCE: Congressional Budget Office from Department of ers, workstations, and supercomputers--were Commerce data. small. Although the supercomputer market a. The Department of Commerce provided no estimate for has grown to $1.8 billion, it has been dramati- 1989 supercomputer shipments consistent with those of cally outpaced by growth in both the personal other years. computer and workstation markets, which in 1991 exceeded $30 billion and $8.8 billion, re- spectively.12 Given the levels and growth in The manufacture of supercomputers, on the sales, companies anxious to become market other hand, has not led to significant advances leaders have pushed every aspect of the tech- in general computer technology. The trade- nology, with the result that these markets see offs in cost and performance that supercom- whole turnovers in their technology every few puter producers face are different from those years. of most other computer producers. For example, vector supercomputers must often be equipped with specialized liquid cooling facilities. Even when the manufacturers use more conventional technology, its application is 11. In RISC architecture, the computer can perform fewer likely to differ substantially from the practices instructions, but it performs them rapidly. Complex instructions are then put together from simple individ- of most electronics equipment producers. In ual instructions. Conventional computer architecture, limited areas, however, supercomputers have by contrast, includes many more and complex instructions, which are performed more slowly. made a contribution to technology, most notably in software. As engineering workstations 12. Supercomputer sales are based on Department of Com- have become rapid enough to provide timely merce, U.S. Industrial Outlook 1992, p. 27-6. Personal computer sales are from International Data Corp, a mar- mathematical solutions to engineering and ket research firm, and appeared in Jim Seymour, "Plat- scientific problems, software originally writ- forms: How the PC Stacks Up," PC Afagazine (May 12, 1992), p. 115. Engineering workstation sales are based ten for supercomputers has been translated for on Dataquest estimates in Rick Whiting, "A Clash of the workstations. Giants Shakes the Low-End Workstation Market," Elec- tronic Business (April 27. 1992), p. 87. Because the workstation and personal computer markets overlap, the The trickle-down theory of technology may two numbers should probably not be added together. have corresponded to reality in the past. In 22 PROMOTING HIGH-PERFORMANCE COMPUTING A ND COMMUNICATIONS June 1993 the 1950s and 1960s, high-performance com- in speed between the integrated circuits used puters led the general advance in computer for small computers and those used for super- technology. For example, the IBM Stretch computers. computer, a high-performance predecessor to the famous IBM 360, pioneered a technique Computer Architecture. The most impor- called pipelining to speed up computers.13 tant advance in computer architecture has Now all modern computers have elements of been the proliferation of reduced-instruction- pipelining. But more recent years may pro- set computing designs; conventional super- vide a better indication of conditions in the computer design, on the other hand, has con- near future. Recent spinoffs from supercom- centrated on vector arrays. Although the Ad- puter technology have been concentrated in vance Research Projects Agency funded much the mainframe computer market, where sales of the early research on RISC as part of its ef- have leveled off and their relative market val- forts in high-performance computing, includ- ue has declined to roughly one-quarter of coming massively parallel processors, the rapid puter sales. development of this technology during the last decade is largely the result of private initiative in the workstation market. The most Recent Trends in Computer commercially successful massively parallel Technology Development supercomputers now incorporate the RISC microprocessors developed for commercial workstations. The more significant advances in technology have occurred in integrated circuits, computer architecture, printed circuit boards, and com- When RISC designs first emerged, designers copied many of the features of existing puter disks. Appendix A describes major areas of computer technology in greater detail. mainframe and supercomputer architectures. However, RISC design moved aggressively, Integrated Circuits. The most notable semi- surpassing existing computer designs in many areas. conductor advances of the 1980s--suchas those in computer memories and microprocessors-- were made with the complex but relatively Printed Circuit Boards and Other Pack- slow integrated circuits used in personal com- aging. Producers of conventional electronics equipment have had to accommodate fewer puters and workstations. By contrast, most vector supercomputers have used simple, fast numbers of relatively more complex inte- integrated circuits. Semiconductor manufac- grated circuits on their printed circuit boards. Conversely, supercomputer designers accom- turing technology has also been driven by the needs of the complex, slow integrated circuits. modate many relatively simple integrated cir- The economies of scale of these large markets, cuits on their printed circuit boards. Because combined with improved manufacturing tech- of the heat generated by many small but fast nology, allowed these integrated circuits to in- integrated circuits, supercomputer designers corporate more functions while becoming fast- have had to turn to water and even freon cool- er, making up most of the original difference ing, while most computers remain air-cooled. Computer Disks. Although supercomputer designers demanded speed from their disks, 13. Pipelining can best be thought of as an assembly line in the bulk of demand for computer disks focused which many different instructions are in various stages of completion at all times. See David Patterson and John on capacity, size, and price. The result has Hennessy, Computer Architecture: A Quantitative Ap- been a computer disk industry largely geared proach (San Mateo, Calif.: Morgan Kaufmann Publish- ers, 1990), p. 338. For a more complete hietory of the to serving the growing market for small com- Stretch computer and its contributions to computer tech- puters. Now the newest disk technology-- nology, see Kenneth Flamm, Creating the Computer: Government, Industry and High Technology (Washing- known as redundant arrays of inexpensive ton, D.C.: Brookings Institution, 1988), pp. 90-95. disks (RAID)--is coming to supercomputers, CHAPTER THREE HPCC AND THE DEVELOPMENT OF SUPERCOMPUTER MARKETS 23 after first being used in other computer mar- nent spinoffs flowing mainly from workstation kets. technology to supercomputer technology.

Some HPCC advocates argue that the price Future Directions of Technology of computer hardware will decrease substan- Spinoff s tially below recent trends because of massively parallel supercomputers. But mas- No one can predict the origins of commercially sively parallel supercomputers are likely to represent only a small fraction of the market successful ideas and technology. The history for most computer components, and hence are of technology development is replete with in- not likely to be the source of economies of scale novative tools and ideas originating from un- or of learning curves. Consequently, HPCC likely circumstances. In most cases, however, technology stays where it was invented; spin- will probably not result in a drastic drop in the price of computer hardware, as the expansion offs, though desirable, are the exception, not of the personal computer and computer work- the rule. If supercomputer technology remains true both to its own history and to the station markets did in the last decade. history of technology development in general, Areas of Likely Spinoff. Two of the likeliest it will probably not result in many component spinoffs into other parts of the computer in- areas of spinoff from the massively parallel dustry, especially the rapidly growing seg- supercomputer to other types of computers do not involve components but communication ments of that industry. between processors and the creation of parallel solution paths in software problems. Component Technology. Unlike previous generations of supercomputer technology, the massively parallel supercomputers, such as Most massively parallel supercomputers are composed of many RISC microprocessors. those being developed by HPCC, use many of the same components as engineering work- In getting these individual microprocessors to communicate and coordinate rapidly, HPCC stations. Indeed, the very existence of massively parallel supercomputers is predicated will have to confront the exigencies of proces- on the availability of high-performance com- sor communication to a much greater extent mercial semiconductor technology. Much of than does the computer market as a whole. the improvement in the performance of mas- Both the speed and frequency of communica- sively parallel supercomputers during the last tion between microprocessor nodes that de- decade can be attributed to the use of increas- signers of these systems must deal with ingly sophisticated workstation microproces- present greater challenges than computer de- sors.14 signers have had to face until now. As more computers are joined in computer networks, In fact, within the market for massively the problems of coordination will become more severe, and the experience gained in trying to parallel supercomputers, the use of microprocessor technology designed specifically for su- tie microprocessors together within a mas- percomputers is often regarded as a weakness. sively parallel supercomputer may become relevant. Such technology, because it is not continually pushed by the brutally competitive forces of the workstation market, is felt to be in danger The transfer of technology may be limited, of falling behind its commercial counterparts. however. Communication and coordination among microprocessors in a massively parallel Technologists in the field therefore see compo- supercomputer must be rapid and tight. By contrast, most computer network applications involve slower communication and looser co- 14. For this history of improvement, see John Van Zandt, Parallel Processing in Information Systems (New York: ordination. Whether the principles needed to John Wiley and Sons, 1992). solve problems of tight coordination and rapid 24 PROMOTING HIGH-PERFORMANCE COMPUTING AND COMMUNICATIONS June 1993 communication can be economically applied to Another obstacle to the widespread use of more general computer communications is not HPCC-sponsored supercomputer technology clear. As noted above, supercomputers push in the near term can best be described as eco- performance to such extremes that many of nomic inertia: current supercomputer hard- the solutions they devise are uneconomic or ware and software are very good and continue inappropriate for other types of computers. to improve. Furthermore, close to 20 years' worth of software has been written to run rap- The other area of substantial innovation in idly on that hardware. Given the difficult massively parallel supercomputer technology problems of writing software for massively involves making problems parallel. This pro- parallel supercomputers, one cannot expect cess involves rethinking problems and re- the massively parallel supercomputers to sup- arranging them so that their component ele- plant vector supercomputers soon; they do not ments can be worked on simultaneously. The yet seem able to run most software as rapidly methods of finding parallels in problems may as the vector supercomputers. Nevertheless, have wider applicability. For example, re- supercomputer centers are buying massively searchers who have rewritten software appli- parallel supercomputers as supplements to cations to make them parallel for the mas- their vector supercomputers so they can serve sively parallel supercomputer have discovered clients who need the additional power. This that these applications also run better on vec- market is still developing. tor supercomputers, sometimes faster than on the parallel system.15 This effort to make in- The most apparent area of near-term com- ternal parallelisms more explicit may be one mercial demand for massively parallel com- of the areas in which massively parallel puters is in data-base management. The mar- supercomputer researchers advance general ket for parallel special-purpose computers for computer technology. managing data bases is almost as large as the market for massively parallel supercomputers. It remains to be seen whether the producers of massively parallel supercomputers can reposition their products to serve this market, Obstacles to Widespread or whether the firms that currently serve this market can break into the general-purpose Commercial Use of HPCC computer market. Technology Most analysts believe that parallel supercomputers and, more generally, parallel com- The main obstacle to developing widespread puters, are indeed the wave of the future. Ex- commercial markets for HPCC-developed perts disagree on how near and how parallel supercomputer technology is not that a com- that future is and on whether the economy peting supercomputer technology will emerge, will get there through incremental steps or but rather that cheaper workstation technol- major leaps.16 By trying to leapfrog the devel- ogy will preempt further substantial growth of opment of intermediate technology, producers the supercomputer market as a whole. At of massively parallel supercomputer hardware present, more commercial design problems are risk leaving software producers and users being solved on workstations than on super- with no simple path for the evolution of their computers. products.

15. George Cybenko and David Kuck, "Revolution or Evolu- 16. Cybenko and Kuck, "Revolution or Evolution?" pp. 39- tion?" IEEE Spectrum (September 19921, p. 41. 41. CHAPTER THREE HPCC AND THE DEVELOPMENT OF SUPERCOMPUTER MARKETS 25

Growth in the Engineering the market for numerical analysis: the little Workstation Market May Slow problems go to workstations, the big problems go to supercomputers, and some analyses can Further Substantial Growth in be performed on either. the Supercomputer Market In fact, one might argue that, in the United As growth in the supercomputer market has States, computer tasks go to supercomputers slowed, the engineering workstation market only when the data needed for their solution has continued to grow at a very high rate, as exceed the ability of most workstations to re- shown in Figure 2. Since both types of com- trieve and use the data rapidly.18 These very puters are used for numerical analysis and often are tasks such as designing nuclear are--to some extent--interchangeable, work- weapons, sifting through electronic intelli- stations may be performing work that pre- gence, or other traditional supercomputer viously would have been performed on a su- problems where cost is not of paramount con- percomputer. Both markets were small in the cern. The market is more likely to favor tasks early 19809, but the workstation market has that could have been solved on a supercomput- grown to several times the size of the super- er had supercomputers been ubiquitous and computer market. cheap. Thus, in essence, the expanding need for numerical processing is more likely to be The increasing power of engineering work- satisfied on a workstation than on a supercom- stations has spurred their growth relative to puter. To some extent, the workstation may that of supercomputers. The speed and low do to the supercomputer what the personal cost of many high-end engineering worksta- computer did to the mainframe--limit its de- tions make them a logical substitute for a su- mand. percomputer. They are quite popular among engineers and designers; the workstation is al- Resolution of some of the software problems ready nearby and, at the margin, it is virtu- of the massively parallel supercomputers that ally free to use. Consequently, advances in HPCC is working on may inadvertently help workstation technology and speed may further workstations take over part of the supercom- limit the supercomputer market. puter market. As noted above, many of these massively parallel supercomputers are com- Even the positive developments in software posed of the same microprocessors that power for supercomputers can become a liability for the workstations; each microprocessor may supercomputer hardware sales. As supercom- have its own memory and some peripheral puter software becomes transferable to work- chips, just like workstations. In short, these stations, it establishes a self-reinforcing trend. massively parallel supercomputers are really Software programs that previously could run many "workstations" within one box. Soft- only on supercomputers have already been ware developments that permit many of these transferred to workstations, including some workstations to work together within one box programs that perform specialized numerical may also permit many freestanding worksta- analysis, which had been the forte of the su- tions to do the same. Indeed, HPCC is funding percomputer.17 Of course, not all problems research to speed such developments. that could once be solved only on supercomputers are going to workstations. Many prob- Software already exists that allows clusters lems are simply beyond the power of a work- of freestanding workstations on a network to station to solve economically. But the work- use each other's computational resources to station permits a much finer segmentation of produce a computing machine that mimics

17. Jeffrey Young, "Crashware," Forbes, April 13, 1992. pp. 18. Mark Furtney and George Taylor, "Of Workstations and 110-112. Supercomputers," IEEE Spectrum (May 1993). pp. 64-68. 26 PROMOTING HIGH-PERFORMANCE COMPUTING AND COMMUNICATIONS June 1993 much of the power of a supercomputer.19 With part-time use of a single resource--than are this software, problems are divided among dif- simple additions to hardware. ferent workstations, just as they are within a massively parallel supercomputer, while the HPCC is devoting substantial resources to different parts of the software communicate resolving the software problems involved in back and forth between different worksta- assigning tasks to groups of processors. It has tions. This type of software is currently in a joined workstation producers in funding much relatively primitive state and works best with of the research on workstation clusters. problems having parts that are fairly indepen- Should generic solutions be found, the work- dent. Clustered workstations are also multi- station market may benefit substantially. purpose: they can be used as stand-alone com- Thus, efforts by HPCC to develop network puters during the day and as parts of super- software and some classes of hardware may af- computers when not occupied. fect the commercialization of HPCC-supported hardware in other markets. But using workstations in this manner has serious limitations. Communication between freestanding workstations is a couple of orders Vector Supercomputing of magnitude slower than between the work- Technology May Limit the Near- stations inside a massively parallel supercomputer. Consequently, until network commu- Term Growth of the Massively nications technology becomes much faster, Parallel Supercomputer Market there will be whole classes of problems for which such clusters are inappropriate. The quest to make massively parallel supercomputers the dominant supercomputer ar- Advances in network technology also pro- chitecture faces a well-entrenched incumbent: vide the most likely route through which com- the vector supercomputer. Displacing vector puting resources will be "scaled." As was supercomputers from their current position is noted in Chapter 2, HPCC is developing high- not likely to be easily or quickly accomplished. performance computer hardware that can be It is not yet certain that the higher theoretical scaled linearly by simply adding more comput- peak speeds of the massively parallel super- ing nodes. The basic architecture will not computers will translate into features that the change--it will just grow faster. Advanced marketplace will value. The difficulty in pro- network technology, however, also allows us- gramming the massively parallel supercom- ers to create "virtual" scaling by simply add- puters as well as their lower performance on ing more computer resources as they are need- many kinds of problems may limit their share ed.20 Although virtual scaling is not likely to of the supercomputer market. Finally, many allow as fine increments in capabilities as massively parallel supercomputers are in- purely scalable hardware, it may be more eco- compatible not only among manufacturers, nomic--allowing more investigators to make but in some cases also between generations.

Hardware for Vector Supercomputers. Because of differences in design, massively parallel supercomputers can provide very high 19. Alexander Wolfe, "Parallel Tools Kick Applications into High Gear," Electronic Engineering Times (March 30, speeds for certain types of problems, and vec- 19921, pp. 30,60, and 82. tor supercomputers can provide higher speeds 20. For examples of adding computer resources to create a for most problems. The central point to re- larger computing resource through networks, see the member is that a vector supercomputer is com- presentations of Larry Smarr of the National Center for Supercomputing Applications, Univeraity of Illinois, and posed of a few--16 at most--extremely fast cen- Gene Feldman of the NASAIGoddard Space Flight Cen- tral processing units, while a massively par- ter at the Revolution in Supercomputing Symposium, January 27, 1993, Washington, D.C., sponsored by Sili- allel supercomputer is composed of hundreds con Graphics, Inc. or thousands of slower central processing CHAPTER THREE HPCC AND THE DEVELOPMENT OF SUPERCOMPUTER MARKETS 27

Figure 3. Simulated Performance of Three Supercomputers

Millions of Floating Point Operations per Second 12,500

10,000 -

7,500 - Massively Parallel Supercomputers

5,000 -

Vector Supercomputer

2,500 - -

I I 0 0 10 20 30 40 50 60 70 80 90 100 Percentage of Calculations Performed in Parallel

SOURCE: Congressional Budget Office based on Peter Gregory, "Will MPP Always Be Specialized?" Supercomputing Review (March 1992), pp. 28-31.

NOTE: The vector supercomputer simulation is based on Cray Research's C-90. The massively parallel supercomputer simulations are based on Thinking Machines Corporation's CM-5 and Intel's Paragon.

units. The high theoretical speed occurs only will be parallel and some will be nonparallel. when a problem can be divided into very many A supercomputer consultant recently ana- components.21 Then the massively parallel lyzed the performance of different types of supercomputer can take advantage of its supercomputers with different combinations many processing units. But when a problem of parallel and nonparallel computation.22 cannot be divided into many parallel compo- The analysis in Figure 3 compares the simu- nents, the speed of the vector supercomputer's lated performance of different supercomputers central processing unit becomes a winning ad- as the problem to be solved becomes more par- vantage. allel.

Most actual problems are not entirely mas- The analysis shows that, for several of the sively parallel or entirely composed of a few most common models of supercomputer, the components, however. Typically, some aspects

22. This section is based on Peter Gregory, 'Will MPP [Mas- sively Parallel Processing] Always Be Specialized?" 21. For a discussion of the limits to parallelism, see Jack Supercomputing Review (March 19921, pp. 28-31. Ap- Worlton, "Existing Conditions," in Supercomputers: Di- pendix B of this report presents the formula used in the rections in Technology and Applications (Washington, calculations. A similar point is made in Patterson and D.C.:National Academy Press, 1988) pp. 47-49. Hennessey. Computer Architecture. p. 576. 28 PROMOTING HIGH-PERFORMANCE COMPUTING AND COMMUNICATIONS June 1993 vector supercomputers are faster than mas- proved to be between 1 percent and 5 percent sively parallel supercomputers unless as much of theoretical peak efficiency.25 In most prob- as 99.2 percent--depending on the computer-- lems, the massively parallel supercomputers of the computations are performed in parallel. were able to obtain only a fraction of the speed That is, even if only 8 of every 1,000 computa- of the older vector supercomputer, in many tions performed in the course of solving a prob- cases not even matching the efforts of an older lem are not performed in parallel, then the Cray's processor. Interestingly, the latest gen- vector supercomputer will be faster. eration of massively parallel supercomputers has closed in on the performance of an older, Another major issue in the speed of parallel- single-processor CRAY Y-MP for several ism is the length of time required to commu- benchmarks, a feat that largely eluded the nicate between nodes versus the length of time earlier generations. it takes to perform the work. Problems that require either relatively little communication The NASA analysis found, however, that between nodes or communication only to a few because of the lower purchase price of mas- nodes are especially appropriate for parallel sively parallel supercomputers, the best of computation. But problems whose solution them exceeded the best vector supercomputers has many interactive parts are more difficult on a cost-performance basis. In addition, some for parallel computers because the computers Department of Energy researchers concluded spend their time communicating between the that at least part of the lower performance of nodes rather than computing within the the massively parallel supercomputers was at- nodes.23 tributable to their immature software tools, a weakness HPCC intends to correct.26 The difficulty of internode communication further complicates the issue of parallelism: The common wisdom in the supercomputer not only must the problems be massively par- industry is that a new computer system has to allel, but they must also have a certain type of be 5 to 10 times as fast (or cost-effective) as the parallelism, one in which the different compo- old one to justify the costs of adapting to the nents are fairly independent. One obstacle for new technology. In the absence of such a cost parallel supercomputers is that whole cate- differential, the massively parallel supercom- gories of problems most commonly solved on puters will replace vector supercomputers on- supercomputers have low degrees of parallel- ly slowly, if at all. Massively parallel super- ism and require a high degree of communica- computers exhibit this differential in only a tion between nodes.24 small number of applications.

Thus, theoretical maximum speed may not Software for Vector Supercomputers. The be a helpful guide to actual operations. In vast majority of problems solved today on su- benchmark tests conducted at NASA's Ames percomputers are solved on vector supercom- Research Laboratory, the actual performance puters, not massively parallel supercomput- of several brands of massively parallel super- ers. For the past 20 years, scientists and en- computers across a wide range of problems gineers have written and optimized their programs for these vector computers, so that there is now a large stable of mature software 23. Robert Pool, "Massively Parallel Machines Usher in Next Level of Computing Power," Science, April 3, 1992, p. 50. Technical Report RNR-92-002 (December 7, 1992). For 24. Furtney and Taylor, "Of Workstations and Super- similar results, see Olaf Lubeck, Margaret Simmons, computers." p. 68. and Harvey Waseerman, "The Performance Realities of Massively Parallel Processors: A Case Study" (LOB 25. See David Bailey, "Experience with Parallel Computers Alamos National Laboratory, Los Alamos, New Mexico, at NASA Ames," Technical Report RNR-91-007 (NASA June 19,1992). Amee Research Center. November 14, 1991); David Bai- ley and others, "NAS Parallel Benchmark Results," 26. Lubeck, Simmons, and Wasserman, "The Performance Realities of Massively Parallel Processors." CHAPTER THREE HPCC AND THE DEVELOPMENT OF SUPERCOMPUTER MARKETS 29 able to solve a wide variety of problems.27 computers. Even if they are faster than con- Most of this software is continuously being up- ventional supercomputers, they may be only graded and refined to improve its performance modestly faster--by a factor of two or some and make it easier to use. small number--which would appear to limit the incentives to rewrite software for them. If A recent survey at the Pittsburgh Super- rewriting several hundred thousand lines of computing Center indicated that their library code only halves computation time, such an in- had more than 300 third-party software pro- vestment would probably not be worthwhile grams for their vector supercomputer but only given the relative costs of computer time and two such software packages for their mas- programmer time. The current code, although sively parallel supercomputer.28 As a result, not perfect, may be "good enough." their vector supercomputer was oversub- scribed, and their massively parallel super- This obstacle should not be underestimated. computer was underused. In most other computer markets, a backlog of existing software usually gives incumbent Existing software would have to be rewrit- computer designs a substantial, if not insuper- ten in order to run on massively parallel su- able, advantage. For example, most users percomputers. Some of it can be easily rewrit- would not switch from an IBM-compatible per- ten, but much cannot. In fact, some of the sonal computer even though the leading alter- most popular software for supercomputers is native is commonly described as "friendlier." likely to be difficult to translate. The tech- In technical markets, an investment in cur- nical software written for supercomputers can rent software may weigh less heavily because involve thousands of lines of code. Moreover, so many users write their own software, al- those programs must not only be rewritten but though it is unlikely that this completely off- also optimized for the new computer architec- sets the advantage of having a library of exist- ture. (This rewriting assumes that HPCC's ing applications. software effort is successful in producing software tools and techniques for massively par- Economic Inertia and Technology Devel- allel supercomputers. If robust software tools opment. Technological conservatism is seen are not developed, such rewriting may not everywhere in the computer world. A familiar take place.29) example is the QWERTY keyboard, the conventional alphanumeric keyboard on virtually Even if the software code is rewritten, there every typewriter and computer terminal.30 is no guarantee that it will improve perfor- More productive keyboard layouts have been mance. For some kinds of problems the mas- available since the 1930~1,but have never sively parallel supercomputers are fast, but caught on, even though they could easily pay for others they are not as fast as vector super- for themselves and the retraining costs in enhanced typing productivity. Typists have little incentive to learn a new layout until it be- 27. For a list of more than 600 software applications for vec- comes prevalent, while any business that in- tor supercomputers, see "Directory of Applications Soft- ware for Cray Research Supercomputers" (Cray Re- troduced a new system would have to retrain search, Mendata Heighte, Minn., 1990). its entire staff. 28. Frank Wimberly, "Connection Machine Software Pack- ages Becoming Available," Pittsburgh Supercomputing Tendencies toward technological conserva- Center News (MayIJune 1992). p. 7. Third-party soft- tism may be gradually overcome, but only if ware ie software that has been written by neither the manufacturer nor the user of the supercomputer. Third- there is a clear reward for doing so. The origi- party software is typically intended for sale. nal CRAY 1 overcame such obstacles by pro- 29. The General Accounting Office recently argued that ARPA wae neglecting the development of software tools. See General Accounting Office, Advanced Research Projects Agency Should Do More to Foster Program Goals 30. Paul David, "Clio and the Economics of QWERTY," (May 17,1993). American Economic Review (May 1986), pp. 332-337. 30 PROMOTING HIGH-PERFORMANCE COMPUTING AND COMMUNICATIONS June 1993 viding fast performance when it was intro- shift from the technical market to the business duced in 1976. The massively parallel proces- market. Recently, a leading manufacturer of sors have not yet been able to match current massively parallel supercomputers--Thinking supercomputers for most applications, so that Machines Corporation--announced its interest there is no clear reward for investing in them. in providing "back end" power for mainframe For most applications, existing supercomput- computers.32 In such a system, a massively ers are adequate. parallel supercomputer would supplement the power of a conventional mainframe when ac- In some commercial applications, where cessing large data bases. Other, smaller pro- there may be clear rewards for rewriting soft- ducers of these supercomputers have posi- ware code, massively parallel supercomputer tioned their product similarly. As an adjunct technology may penetrate first. The simula- to a mainframe, massively parallel supercom- tion of oil reservoirs, for example, is more ac- puters would not conflict directly with conven- curate on parallel machines, and the problems tional supercomputers. But like the vector fa- involved in the simulations lend themselves to cilities discussed above, they might be able to such technology. Moreover, from an economic create a niche by exploiting weaknesses in cer- standpoint, coming up with an accurate esti- tain computer markets. mate of the size of an oil field carries a substantial payoff. The contracts for oil produc- So far, the commercial sales of massively tion can be in billions and tens of billions of parallel supercomputers have been largely for dollars. Even a small improvement in the es- a special or single purpose. In a widely publi- timate can easily pay for the rewriting ef- cized sale, American Express bought two fort.31 Another example involves financial CM-5 massively parallel supercomputers from calculations for certain investment instru- Thinking Machines Corporation to reduce the ments for which the rewards can be large and time it took to query its massive data base. immediate. Whether these isolated instances Even in this case, however, the CM-5s are not of commercial use can be turned into larger acting independently but in conjunction with markets is as yet unclear. a mainframe computer.33

Other applications, such as rational drug One special-purpose massively parallel design and computational chemistry, may pro- computer made by the Teradata unit within vide openings for massively parallel super- NCR has made substantial inroads into busi- computers because they are so new that vector ness markets over the last decade.34 (Because supercomputers have not yet been applied to Teradata computers lack specialized number- them to a great extent. Consequently, inves- crunching capacity, they are not properly con- tigators can start from scratch without worry- sidered supercomputers.) This line of ma- ing about leaving behind a lifetime's work. chines combines between several dozen and several hundred of the microprocessors that power IBM-compatible personal computers Potential Entry into Business (the so-called 80286, 80386, and 80486) into a Computer Markets facility that runs a specialized data-base application, again as an addition to a general- One way to move massively parallel technol- purpose mainframe computer. Teradata's ogy into general use despite the preemption of much of its potential market by the workstation and the vector supercomputer may be to 32. Johanna Ambrosio, "Super CPU Maker Widens Aim," Computerworld (October 19,19921, p. 6.

33. Dwight Davis, "Supercomputers Knock at IS Doors," 31. Ironically. simulation of oil reservoirs is often mentioned Datamation (December 1,19921, pp. 79-82. as a problem that is especially well suited for networks of engineering workstations, as discussed above. Thus, two 34. Dwight Davis, "Oracle's Parallel Punch for OLTP [On types of technology will be competing for the same set of Line Transaction Processing]," Datamation (August 1, problems. 19921, p. 67. CHAPTER THREE HPCC AND THE DEVELOPMENT OF SUPERCOMPUTER MARKETS 31 sales in this specialized market almost equal with massively parallel supercomputers the combined sales of all the massively par- would also apply to these parallel mainframes, allel supercomputer producers. IBM recently although data-base management is inherently announced its intention of entering this mar- very parallel. ket of specialized parallel data-base computers.35 A major software company, Oracle, has also devoted substantial efforts to rewriting Obstacles in Perspective its software for several brands of massively parallel supercomputer to serve this market. HPCC is not unique in facing obstacles in the For its part, Teradata has been trying to ex- marketplace. Most new products and many pand its market beyond its specialized one. new technologies fail to gain widespread ac- ceptance when first introduced. Even when It is not yet clear whether the data-base they finally succeed, it may be for reasons the market has a path that will lead to general- inventors could not have foreseen. For exam- purpose commercial computing. Such a path ple, the creation of small personal computers might emerge as more and more functions are stimulated the demand for large flat-panel incorporated into the specialized computers displays. These had been invented in the until eventually they become general-purpose 1960s to service a home TV market that did computers. This path might be more likely to not materialize. Their inventors could not penetrate the business markets than one di- have foreseen laptop computers. rectly through the supercomputer market because most U.S. firms do not use supercom- The designers of massively parallel super- puters in their business. A survey of Fortune computers hope to increase the demand for 500 firms found that only 15 percent of them supercomputers by making their hardware use supercomputers, and an additional 6 per- less expensive. But the software for these ma- cent consider them an option.36 By contrast, chines, when it exists, is difficult to write and data-base management accounts for 50 per- difficult to use. Viewed from the perspective cent of corporate mainframe use, and almost of the costs of a supercomputer over its life- all Fortune 500 firms have mainframes. time in actual use, the massively parallel supercomputer reduces the cost of the least ex- Thus, to the extent that parallel business pensive element (initial purchase price) while computers use HPCC technology, it might en- increasing the cost of the most expensive ele- ter mainstream commercial uses through al- ments (software and support). ternative avenues. Like the massively parallel supercomputers, however, the software Firms introducing products based on HPCC problems associated with turning these spe- supercomputer technology may be able to cial-purpose computers into general-purpose overcome these obstacles, although the result- computers are substantial. Also, because the ing market is not likely to experience the levels of technical competence are lower in the automatic growth of a new market. It is not business market than in the scientific and merely a case of waiting until sufficient ap- technical markets, the effect of these software plications are written for massively parallel problems on demand is likely to be more se- supercomputers and all the kinks are ironed vere. And many of the same questions regard- out. The target it seeks to hit is moving: both ing parallelism discussed above in connection workstation technology and vector supercomputer technology are improving.

35. Laurence Hooper, "IBM to Push New Generation of Mainframes," Wall Street Journal, December 22,1992, p. Most important, as noted above, most ana- A3. lysts agree that computer technology is likely be parallel in the future. They disagree about 36. Mohammad Amini. "The Status of Supercomputing in Corporate America," Supercomputing Review (Septem- how best to get there and how parallel is par- ber 1991), pp. 40-41. 32 PROMOTING HIGH-PERFORMANCE COMPUTING AND COMMUNICATIONS June 1993 allel. HPCC has positioned itself toward one demand within those potential markets. end of the spectrum. Overcoming some of these obstacles is a goal of the HPCC program. Other obstacles may be The presence of these obstacles does not overcome by the firms that actually sell pro- mean that there will be no market in the near ducts incorporating HPCC technology. How term for such massively parallel supercom- these firms position themselves in the market puters: the analysis found two potential mar- will be crucially important, but is beyond the kets. Rather, the obstacles will serve to limit scope of this report. Chapter Four HPCC and Computer Network Markets

n addition to its effect on the research society might use and benefit from such a net- and education communities, the High- work. I Performance Computing and Commu- nications program may also have a significant As noted in Chapter 2, the NREN compo- impact on the emerging data communications nent of HPCC has two major subcomponents: markets in the United States. The commer- upgrading the interagency interim NREN, cial markets currently consume most of the and promoting the development of computer billions of dollars worth of network equip- networks that transmit data at the rate of 1 ment and data communications services the billion bits per second. Upgrading interim U.S. industry produces every year. Conse- NREN has several parts. The main focus of quently, they are likely to be among the early the Internet program has been on bringing the consumers of the network technology devel- existing NSFNET up to 45 million bits per sec- oped under HPCC. ond speed. NSFNET is a computer network supported by the National Science Foundation that serves as the central link of Internet. Re- cently, the NSF published a solicitation to bid on upgrading the NSFNET to 155 million bits HPCC and Computer per second speed, with an eye toward eliminat- ing the federal subsidy on part of it. Less well Networks known are efforts to bring some specialized agency data bases on line to make them acces- The National Research and Educational Net- sible from the NSFNET. work, discussed in Chapter 2, is a central part of the HPCC, both as a goal in and of itself and In another part of the interim NREN, the as an enabling technology for other compo- Department of Energy is accelerating develop- nents of the program. The aim is to make the ment of the Energy Sciences Network--the network available to more researchers and al- computer network for the department's R&D so to enable it to transmit data at the rate of l facilities--beyond the capabilities of the rest of billion bits (1 gigabit) per second (it currently the interim NREN. DOE had selected a con- operates at about 45 million bits per second). tractor for the development of a 622 million By linking supercomputers and other research bit-per-second network for service among a assets of the government over very rapid com- dozen or so nodes on the Energy Sciences Net- puter networks, federal agencies should be work, using protocols that have only recently able to bring the best tools to bear on the entered commercial trials. However, the Gen- grand challenges. In addition, the NREN eral Accounting Ofice set this selection aside could demonstrate how other segments of and the contract will have to be rebid. 34 PROMOTING HIGH-PERFORMANCE COMPUTING AND COMMUNICATIONS June 1993

The research and development component Most computer power in the United States of the NREN largely focuses on so-called giga- exists in unconnected islands. Even where bit test beds.1 These test beds are a pub- computers are connected, the links are either liclprivate venture in which industry teams isolated, as with organizational networks, or and government researchers develop and test temporary, as with modem-based dial-up ser- components, protocols, and other aspects of vices. Computer networks have already dem- networking technology. As technology is onstrated that they can provide a competitive proved, it will be incorporated into the con- advantage to companies having them. Many struction of the interim NREN and into com- of the most successful retailing and service mercial offerings. In the test beds, the federal companies use computer networks as a source government has leveraged its investment sev- of competitive advantage.2 eral times over with private participants: while the federal cost is projected to be roughly $15 million, the private investment is Furthering U.S. reported to be many times that. Competitiveness Advocates of HPCC believe that a prolifera- Meeting Goals tion of network services could enhance economic performance and welfare. The competi- The network component of HPCC is more dif- tiveness of U.S. industry would be furthered in fuse than the supercomputer R&D component several ways. First, U.S. service and retail because part of the effort is involved in R&D companies, which already use computer net- and part in developing an existing network. works far more than other sectors, are likely Each needs to be judged by different criteria. to be the first to take advantage of any new The R&D component obviously needs to be network technology. Of course, if networks judged primarily by technical standards. became more widely used internationally, U.S. firms might lose some of the advantages Technical criteria, however, are not suffi- they currently enjoy relative to their foreign cient for evaluating a network; marketing and competition. demonstration criteria also need to be taken into account--for example, growth in the use of Second, U.S. firms manufacture most of the Internet. The combination of R&D programs large computer networks.3 Consequently, if with an operating network is intended to federal R&D and demonstration projects in- make computer networks faster and to dem- crease the demand for these networks, U.S. onstrate to more people that computer net- firms are likely to capture a large portion of works can contribute to their work, education, that demand. Even in the market for small and other aspects of their lives. networks, U.S. firms have a disproportionate influence, largely because they dominate the Network R&D and especially Internet are markets for personal computer hardware and intended to encourage the rapid spread of com- software. An offsetting factor is that computer puter networks throughout the economy. Like networks compete with mainframe computers, telephones and FAX machines, computer net- a market in which U.S. firms are also quite works become more valuable as more people dominant. If sales of U.S.-produced networks hook up to them. HPCC planners hope that as the capacity becomes more widely available, more sectors of the economy will become linked by computer and will invent and dis- 2. Thomas Malone and John Rockart. "Computers, Net- works and the Corporation," Scientific American (Sep- cover new ways of using the network. tember 1991), pp. 128-136.

3. National Research Council. Computer Science and Tele- communicatione Board, Keeping the U.S. Computer In- 1. Gary Stix, "Gigabit Connection," Scientific American dustry Competitive: Systems Integration (Washington, (October 1990), pp. 118 and 120. D.C.: National Academy Press, 1992), p. 7. CHAPTER FOUR HPCC AND COMPUTER NETWORK MARKETS 35 increased, some of this increase would come at Unlike the supercomputer component, the the cost of reduced sales by other U.S. corpora- vision for the network component of HPCC is tions, such as mainframe producers. in line with economic trends. There is a tradeoff, however, between performance and sales-- HPCC proponents argue that computer net- very fast networks are not likely to be cheap, works can help people in many different walks and expensive networks are not likely to be of life share scarce resources, reducing their widely used. All users would like faster, more cost and improving their quality. Examples of responsive networks, although most of them such specialized resources include services, do not even begin to use the maximum tech- such as medical expertise; or information, nical capabilities currently available. Advo- such as library catalogs; or people, such as cates argue that the very presence of such ca- teachers. pabilities will lead to new applications, but critics maintain that new applications may not be economically viable or may not find wide use outside a few specific sectors. Box 1. Regulation--particularly if the regulation is Segments of the Market dispersed among numerous state public utility for Networking Products authorities with many diverging objectives-- may retard the adoption of advanced network The market for computer network products technology. Although the general trend of and services can be divided into four overlap- regulatory intervention seems favorable in ping segments. While hardware and software are generally complementary, network ser- many ways, the regulated carriers will need vices are generally in distant competition time to adjust their behavior and to put the in- with hardware and software. frastructure in place.

Hardware

File Servers Cabling Network Interface Cards Analysis of the Markets Bridges and Routers for Computer Network Leased Lines Hardware, Software, and Software Services

Operating Systems The United States has a very large and thriv- Applications ing data communications market. Some analysts believe that data transmission amounts Services to a substantial part of all U.S. telecommunications. Many large companies are in the Value Added Networks business of providing the hardware, connect- Outsourcing ing lines, software, and services for computer Switched Networks networks.

Internetworking Market Segments Internet Commercial Providers The market for computer networks comprises four partially overlapping segments (see Box 1). The first two segments consist of providers SOURCE: Congressional Budget Ofice. of the hardware and software for computer 36 PROMOTING HIGH-PERFORMANCE COMPUTING AIND COMMUNICATIONS June 1993 networks, including telephone companies that work but a network of networks: it serves net- lease lines over which private firms run their work providers directly and final users indi- computer networks as well as firms that man- rectly through the networks to which they be- ufacture network hardware or write network long. software. According to the Department of Commerce, worldwide sales of computer network hardware and software exceeded $8 bil- Internet and the Development of lion in 1991.4 Dataquest, a market research the Network Services Market firm, estimates that spending for digital private lines increased from less than $1 billion Internet is a network of computer networks in 1986 to more than $3 billion by 1990--an serving the education and research communi- annual increase of roughly 25 percent.5 ties, all of which use the same protocol (TCP/ IP, for Transmission Control Protocolflnternet Firms in the third segment provide network Protocol) to communicate. This network is an services to companies, organizations, and indi- outgrowth of the ARPANET. Internet has viduals who do not wish to set up their own three tiers: national NSFNET links regional networks. This set of firms is in distant com- midlevel networks, which in turn link local re- petition with the first two, just as restaurants search and university networks. Other fed- are in distant competition with grocery stores. eral agencies' networks are also connected to The Department of Commerce estimates that many of the research and educational commu- the U.S. market for the most common single nities, but such connections are specific to dif- type of commercial network service--value ferent agency missions. The contractor that added networks--reached $6 billion in 1991.6 runs the facilities for the National Science The market for another popular type of ser- Foundation is a joint venture of IBM, MCI, vice, called outsourcing (in which one firm and Merit, the Michigan-based midlevel net- contracts with another to manage its com- work. Although the NSFNET provides the puter network), is estimated to be from $3.5 upper-level connections, the entire agglomer- billion to $7 billion annually. By contrast, the ation of connections and networks is com- market for simple switched network services, monly called Internet (see Figure 4). comparable with switched voice telephone service, is still in its infancy. Figure 4. The fourth segment has one principal Composition of Internet member--Internet, a federally funded group of networks associated mainly with universities Other NSFNET Backbone Agency and research organizations. Logically it is a Networks I I I subset of the third (network) segment, but it is I I I I now too large to be treated on a par with com- I interconnections j mercial providers of network services. Unlike those providers, Internet is not a single net- Mid-Level Regional Networks 4. Department of Commerce. 1992 U.S. Industrial Outlook (1992), p. 27-18. Spending on modems used by computers to connect to networks is not included in the network market data discussed here. The market for modems exceeded $900 million in 1991. 5. "Dataquest Market Statistics: Long Di~tanceTelephone University and Research Networks Services" (Dataquest, Inc., San Jose, Calif., May 1991), p. 2. Much of the private-line usage is for private voice networks, but this is decreasing relative to the whole. SOURCE: Congressional Budget Office. 6. Department of Commerce, 1992 US.Industrial Outlook, NOTE: NSFNET = National Science Foundation computer p. 28-8. The Congressional Budget Ofice found no es- network. timate of the total market for network services. CHAPTER FOUR HPCC AND COMPUTER NETWORK MARKETS 37

Figure 5. History of Internet Traffic, 1988-1993

Millions of Packets of Computer Information 35.000

July January January January January January 1988 1989 1990 1991 1992 1993

SOURCE: Congressional Budget Office from data provided by MeritINSFNET, Ann Arbor, Michigan.

Although the growth rate of traffic on wide has risen dramatically--from 350 in Jan- Internet indicates that it is currently fulfilling uary 1989 to 9,100 in 1993. The immediate a role in research and education, this is only a success of Internet is in marked contrast to fraction of its potential use. A major question that of other parts of HPCC. is whether delivering enhanced services to sophisticated users is more important than The uses of the NSFNET backbone are simi- broadening usage to the less sophisticated. lar to those of other computer networks: electronic mail and file exchanges account for al- Background. Internet has achieved a very most three-quarters of the data transmitted. high rate of growth during its five years under These have been a decreasing share of the to- the current contractor.7 When it began oper- tal, however, although no single other use is ating in July 1988, the NSFNET backbone displacing them. traffic was well under 100 million packets of computer information delivered electronical- Despite its popularity, Internet presents ob- ly. In August 1992, the NSFNET backbone stacles to some users. Individuals not affili- delivered 16.5 billion packets of computer in- ated with a research firm or university may formation (see Figure 5). Similarly, the num- have difficulty obtaining access to the net- ber of networks connected to NSFNET world- work. Security is also a concern to many potential members that are research firms. Po- tential users are put off by the unavailability 7, Information on growth and usage of NSFNET come from MERIT Network, Ann Arbor, Michigan. of addresses of potential correspondents: pro- 38 PROMOTING HIGH-PERFORMANCE COMPUTING AND COMMUNICATIONS June 1993 grams for electronic white pages are still be- sure to provide connections to Internet clients ing developed, as is a directory of services. only grows as more individuals, corporations, Within universities, many faculty and re- and organizations join the network. Although search staff members are unaware of having many questions remain about how best to access to Internet or, if aware, may be intimi- transfer the responsibility for Internet's non- dated by it (for example, users must learn yet R&D operations to private hands, it has had a another layer of software). Training on how to major influence on the development of nation- use Internet is limited. wide computer network markets.

Nevertheless, a remarkable number of us- Commercial Internetworking. Internet was ers log on regularly. Internet offers access to one of the first major providers of switched electronic message boards and journals. Pro- internetworking services--that is, providing fessional academic societies have mailing lists network capability on demand parallel to tele- that permit members to exchange information phone communication. But the commercial rapidly, often serving as a grapevine in "hot" development of switched internetworking ser- research areas. vices has been slow for several reasons. Be- cause the market for switched voice commu- Impact on Commercial Markets. Internet nications is much larger than that for has already had a substantial effect on the switched data communications, most of the po- commercial networking market. It specializes tential providers--the regional Bell operating in internetworking, that is, connecting differ- companies and the long-distance companies-- ent networks. When the first contract for the have been slow to enter it. Standards have al- NSFNET was signed in 1987, the internet- so been slow to develop. Finally, the number working market barely existed. Now, how- of computer networks needing to be intercon- ever, the number of networks has multiplied, nected has not been very large until recently.8 and companies and groups investing in them In these circumstances, Internet provided ser- want interconnections, rapidly creating a sub- vices for a stable pool of customers. stantial market for appropriate hardware and services. In fact, the market for internetwork- The most rapidly growing type of switched ing hardware exceeded $700 million in 1991. networking service, called frame relay, still As one of the first providers of internetwork- has quite a limited market, and most others ing services to a large community of users, are still in trials or even awaiting them.9 The Internet was in a position to help define stan- NSFNET contractor is in fact involved in some dards and protocols and develop the generic of these trials, separate from its contract with technology that would see wider use. NSFNET but using many of the same facilities. A very important contribution of Inter- Internet already provides network connec- net, particularly the NSFNET, may have been tions to a substantial fraction of the users and in allowing the contractor--especially its joint developers of technology in the United States. venture members, IBM and MCI--to develop In addition to universities and research insti- expertise in providing switched internetwork- tutions, many technology companies also have ing services ahead of other firms that did not Internet connections directly or indirectly have a built-in clientele. through a commercial service provider. Be- cause networks become more valuable as more 8. Elieabeth Horwitt, "Anewering Users' Interconnectivity people connect to them, this particular net- Needs," Computerworld (March 30,1992), pp. 69 and 72. work is becoming the core of a nationwide data 9. One recent survey found that ae of September 1992, network. The major public computer net- there were only 189 frame relay cuetomers in the United works--such as Compuserve and McI Mail-- Statee. Nathan Muller, "What Does Frame Relay Really already have Internet gateways, and the pres- Cost?" Datamation (December 1,1992),p. 58. CHAPTER FOUR HPCC AND COMPUTER NETWORK MARKETS 39

Telephone Companies and the Obstacles to Wider Use of Price of Leased Telephone Lines HPCC Technology The data communications market as a whole is growing quite rapidly, probably as a result What is the potential demand for switched ser- of the rapid proliferation of computers and the vices at speeds above 45 million bits per sec- substantial decline in the cost of network ond? Current commercial technology can equipment. Because of these changed circum- serve at speeds below that. No one expects the stances, new network services may achieve market to jump from 45 million to 1 billion the commercial success that eluded network bits per second. Some designs are already run- services in the 1970s and 1980s. It is not only ning at 155 million and 622 million bits per a question of new technical capabilities, but second. This additional capacity may be also of price. shared by a number of users, none of whom might require all of it. An analogy may be seen in the way falling computer prices stimulated the demand for computers. In 1970, when the IBM 370 was Unlike the massively parallel supercom- being sold for several million dollars, the de- puter technology discussed in the previous mand for it was large for the time, but minus- chapter, the technology being developed and cule in comparison with today's sales figures. tested in HPCC gigabit test beds has clear po- Currently, IBM-compatible personal comput- tential for influencing mainstream data com- ers, which have roughly the same computa- munications technology. The participating tional capabilities as the IBM 370, sell by the companies are reportedly testing.switches millions for under $2,500. There have been that may then become the basis for commer- some increases in computational power to be cial offerings, although probably operating at sure, but they are far outweighed, especially sub-gigabit speeds. at the lower end of the price spectrum, by decreases in price. Similarly, Internet has helped the demand for internetworking services to jell. As this If the HPCC program leads to similar de- market develops further, Internet technology creases in the cost of data communications, and modes of operation, not to mention Inter- more and more commercial uses will be found net itself, may become the heart of further de- for them. Until quite recently, communica- velopments. tions costs had not matched computation costs in their decline.10 One recent analysis sug- Most analysts believe that a large market gests that, through 1980, communications for rapid data communications will ultimately costs had declined between 3 percent and 5 develop. The disagreements are about the percent a year, but that computing costs were path and the pace. Is the large market more falling more than 20 percent per year during likely to be achieved by creating many net- this same period, even though major pieces of work applications that each need only a small computing equipment and communications network capacity and by allowing the evolu- switching equipment share many compo- tion of their needs to create a large demand for nents.11 rapid services? Or is it more likely to result from creating requirements, and equipment to 10. This analysis ie from Kenneth Flamm, "Innovation,Reg- ulation and Competition,"in Robert Crandall and Ken- serve the most demanding requirements, neth Flamm, ede., Changing the Rules: Technological which are relatively few in number? Change, International Competition, and Regulation in Communications (Waehington, D.C.: Brookinge Inetitu- tion, 1989),pp. 13-61. 11. bid., p. 29. 40 PROMOTING HIGH-PERFORMANCE COMPUTING A,ND COMMUNICATIONS June 1993

The trajectory of costs in the future, how- er declines, none had annual declines larger ever, will crucially depend on the market than 5 percent. structure. The above-cited analysis argued that it was not lack of investment or R&D that Local carriers still account for the vast ma- kept communications prices from falling as jority of private lines. According to one esti- rapidly as those of computers, but rather regu- mate, they provided 80 percent of all private lation and lack of competition. Providers of lines in 1989.13 Two-thirds of all data lines network hardware, however, are in a highly were local, at least 90 percent of which were competitive market. With the breakup of the provided by local exchange carriers. The local Bell system, there is now competition in the metropolitan fiber systems, satellite systems, long-distance market, and so some prices of and other systems still accounted for only a leased digital long-distance lines, as discussed small fraction of leased data lines. According below, have fallen. to FCC data, these alternative local carriers together served fewer than 6,000 customer lo- Local Leased Telephone Lines. The local cations nationwide at the end of 1991.14 telephone companies still have near-monopoly control over local access, which the nationwide Long-Distance Leased Lines. Interex- providers of leased lines and network services change (or long-distance) carriers as a group need in order to reach their customers. The accounted for roughly 20 percent of leased da- prices local telephone companies charge these ta circuits in 1989, according to the previously providers may not drop sufficiently to encour- cited survey of private lines. Their percentage age a large market. has probably risen since then. Though more competitive than the local exchange carriers, Nonetheless, some providers of alternative American Telephone and Telegraph still ac- services have bypassed local service providers counts for the bulk of this business. and may be in a position to force competitive reactions from the local telephone companies The interexchange carriers, in contrast in some areas. How much discretion the local with local exchange carriers, seem to have companies will be allowed will depend crucial- brought some of their prices on leased lines ly on the actions of regulatory agencies. The down substantially. Most important, in posted regulatory climate seems favorable for encour- prices filed before the FCC, there is no domi- aging competition overall, but a fully competi- nant single price trajectory. Posted prices for tive market may be slow to emerge and may short-range private lines, where the inter- take years to develop nationwide. The slow exchange carriers compete with the local ex- development of alternative providers may al- change carriers, have not fallen and in some low local providers a substantial degree of pro- instances may have risen since 1990. In the tection from the hazards of competition and highly competitive longer-distance (above 200 enable them to hold onto a sizable chunk of the entire commerce in this area.

As noted above, private leased lines are a 12. The data are taken from the individual company Tariff major element in the data communications Review Plans using the Special Access Basket lines market, especially at faster speeds. Local tele- detailing High Capacity and DDS (Dedicated Digital Service) indices. phone companies have not lowered their prices for these lines significantly. According to fil- 13. Allan Turnbill, "The Future of Private Lines" (Probe Re- search, Cedar Knolls, N.J.,1989). See Chapter 13, espe- ings with the Federal Communications Com- cially pp. 396 and 402. Private lines can carry either da- mission (FCC), the Bell operating companies' ta or voice traffic. Surveys and other techniques can be prices for data services averaged a 2 percent used to provide estimates of the use. annual decline between July 1990 and July 14. Jonathan Kraushaar, "Fiber Deployment Update, End of Year 1991" (Industry Analysis Division. Common Car- 1992 (the most recent data available).lz Al- rier Bureau, Federal Communications Commission, though some local exchange carriers had larg- March 13, 19921, Table 15. CHAPTER FOUR HPCC AND COMPUTER NETWORK MARKETS 41 miles) markets, prices have been falling, al- leased-line services or force those prices down though not by as much as computer prices. in competition.

Private-line prices have a fixed monthly The near-monopoly of local-exchange car- component and a mileage-based monthly com- riers on local circuits, both for purely local ponent. While the mileage price has been fall- transmissions and for access to interexchange ing dramatically--by more than half in the last carrier lines, is forecast to decline. Whether two years--the fixed component has undergone the decline will be fast enough or large enough a dramatic rise--20 to 50 percent since 1990. to cause prices to decline is not clear. If com- For short distances, the rise in the fixed competition is not sufficient to lower prices sub- ponent dominates the decrease in the mileage- stantially, then the market for data commu- based component. Thus, posted prices on pri- nications may not develop as quickly as the vate lines shorter than 200 miles have risen, computer market. while posted prices on longer private lines have fallen, with coast-to-coast posted prices Though central to the successful commer- falling most. By way of illustration, the post- cialization of HPCC technology, the issue of ed price for a 1,500,000 bit-per-second leased regulation of the telecommunications markets line roughly the distance from Chicago to New is beyond the scope of this analysis, other than York (700 miles) fell by an average of 13 per- to note the central role competition could play cent per year between 1990 and 1992, while in bringing the technology to market. prices for leased lines 75 miles long have risen by an average of 7 percent per year. Likely Effects of Price Changes. The creation of a substantial market usually involves The posted prices are subject to volume dis- shifts in both demand and supply. HPCC is at- counts, commitment discounts, and other dis- tempting to influence both. By subsidizing counts depending on private negotiation. early users through Internet, HPCC helps in- Such discounts serve the larger customers the crease the potential uses of network technol- best. The consumer who is least likely to ogy and thus shifts the demand curve so that benefit from such arrangements is a regional more people seek to use networks. The inde- business or organization that wishes to tie to- pendent proliferation of computers similarly gether only a few sites: neither the local- shifts the demand curve by increasing the exchange carriers nor the interexchange car- number of potential users. riers have reduced posted prices in that market substantially. For many components of networks, primarily hardware, the supply curve also has been Data communications prices may come shifting as the general progress in electronics down if consumers buy switched network ser- has lowered prices. Prices of network-related vices rather than lease private data lines. Un- electronic equipment have declined by a der this arrangement, customers would pay weighted average of 16 percent a year since only for the actual use, just as they do cur- 1990, according to industry data.16 rently for long-distance voice conversations.15 The local-exchange carriers have jointly com- Although the total cost of the system and mitted themselves to providing these services. the network services has fallen, the price of This service is completely new for the local-ex- one major component--the leased local data change carriers, however, and it is not clear how rapidly these services will displace their 16. Figure calculated using U.S. prices and world market ehares of the different types of network equipment. Price data from Dataquest, "Local Area Networks Mar- ket Share and Forecast" (May 1992); and "Networking Market Trends" (August 1992). Market share data from 15. Muller, "What Does Frame Relay Really Cost?" pp. 55- Department of Commerce, 1993 U.S.Industrial Outlook 62. (1993), pp. 24-26 and 25-26. 42 PROMOTING HIGH-PERFORMANCE COMPUTING AND COMMUNICATIONS June 1993 line--has not budged, although the number of different capacity transmitting files roughly such lines has increased. Such lines are cru- the size of the average FAX halfway across the cial if the widespread internetworking con- country.19 FAXes were chosen because they ceived by HPCC is to become a commercial re- are currently the most common form of data ality. But as the other components decrease in communications. cost, local data lines will come to represent an ever larger share of the total cost. At roughly 20 percent capacity use, a network of 1.5 million bits per second--roughly Under some circumstances, HPCC technol- the capacity of the older Internet trunks-- ogy could become widely used without major would transmit a FAX-sized file in a little less shifts in relative prices of leased lines. Com- than a second. This rate is much faster than puter networks may be subject to what eco- the current analog transmission speed of 9,600 nomic historian Nathan Rosenberg calls baud per second.20 If the capacity of the net- "learning by using": as people learn to use network was improved to 45 million bits per works, they rely more and more on them, and second--the capacity of the new trunks on the networks increase in value over time. Un- Internet--transmission would take one- der these circumstances, demand may expand twentieth of a second. If a 1.2 billion bit-per- without price decreases, provided the alterna- second network was transmitting the files, tive local telephone systems also expand to transmission would take one-eightieth of a match any capacity constraint of the Bell op- second. In this last case, a 26-fold increase in erating companies. capacity brings only a fourfold decrease in the system's response time. (Appendix C contains the formula used in calculating these results.) Demand for Very Fast Services Well before a network reaches the point at which there is no physical advantage to its increase in capacity, it is likely to reach the When the capacity of a network to carry data point at which there is no economic advantage files is initially increased, the user sees imme- either. For many users, the difference be- diate benefits.17 Large files can be trans- tween receiving files in one-eightieth of a sec- ferred much more rapidly, for instance. But as ond rather than one-twentieth of a second may the capacity of the network rises further, the not be worth the extra cost. speed of file transfers is limited not by the capacity of the network but by other-factors, If communications costs and transmission such as the speed of light or distance.18 Con- times decline, however, file size will probably sequently, there is a point of diminishing re- grow in response as people are able to include turns to increasing network capacity. For more information in their messages. If the modest-size files, the point of diminishing returns comes at a network capacity well below average file size grows by a factor of 10, the the gigabit-per-second level. benefits of the higher speed become more apparent: at 20 percent capacity, the 1.2 billion bit network is almost 15 times faster than the To analyze the relationship between capac- 45 million bit network. If the average file size ity and speed, the Congressional Budget Office simulated the behavior of three networks of

17. This analysis is taken from Leonard Kleinrock, "The La- 19. This analysis aasumes the average file on the network is tency1Bandwidth Tradeoff in Gigabit Networks," IEEE the size of a FAX, that is, 1 million bits. Most users Communications Magazine (April 19921,pp. 36-40. would not use FAX technology to transmit data through a computer network because it would take too long to 18. Light does not travel through fiber-optic networks at the print. Actual data files are likely to be much smaller. same velocity as in a vacuum. Most fibers have an index of refraction of roughly 1.5, which reduces the speed of 20. A baud is a measure of modern transmission speed and light by a third. does not correspond to bits of data per aecond. CHAPTER FOUR HPCC AND COMPUTER NETWORK MARKETS 43 grows by a factor of 25, the faster network is duce full-motion video over the copper tele- 20 times as fast. phone (twisted-pair) wires connecting most homes. Home trials are expected to begin Thus, the commercial success of the HPCC's soon. fast network technology will depend on the spread of applications that can make economic Compression technology has two closely re- use of the capacity. Except in some obvious lated advantages over accelerating transmis- areas, such as medical imaging and video sion speeds. First, it can use existing infra- conferencing, such uses may be slow to become structure. Second, it does not have to advance widely used. the whole system to achieve its end. Design- ers of compression technology can cater to selected final users without concerning them- Advances in Other Areas of selves much with the rest of the system. They Technology can target high-value and specialized niches much more rapidly than can providers of net- Rapid advances in computer technology in work infrastructure, who, by definition, must other areas can temporarily substitute for ac- compromise among users. To some degree, celeration of transmission speeds in several they can beat the network providers to the ways. Software and hardware designers are early, most valuable customers. designing systems that reduce the volume of The potential of compression technology to data that must be transmitted, using a technology called data compression or simply com- substitute for network technology has several limitations, however. The techniques using pression. The effective result is the same as the highest compression ratios--those that re- an increase in communications speed. If, for example, the size of a file to be transmitted is duce the file to be transmitted by the largest compressed to half its size, then the effective percentage--currently discard part of the data. Some economic uses, such as video transmis- speed of communication--from the user's point of view--is doubled. Advances in compression sions to the home, can accommodate these technology have also reduced the cost substan- losses, while others, such as medical images, tially. In 1992, AT&T announced a set of in- cannot. Eut even the compression techniques that lose data are limited in their ability to tegrated circuits costing $400 that would perform many of the same compression functions compress it. Although they may slow the mi- gration to higher network capacity, they can- previously performed by printed circuit boards not prevent it. costing $25,000.21

Electronic systems designers have substan- Parts of HPCC are concentrating on com- tial incentive to improve compression technol- pression R&D and presumably will play a role in increasing that technology's power and ogy and already use it to allow consumers to defer expensive investments or increase capa- availability. This particular trade-off between bilities. Several of the more popular pieces of different aspects of HPCC is not unique and personal computer software claim to expand can work in any direction. Depending on the the capacity of existing computer hard disks level of technology development, software by compression. Similarly, the growing field may be interchangeable with hardware, and of multimedia applications depends on com- communications capacity may be interchange- pression technology. Already some experi- able with computing power. Such substitut- ability is common in complex technology de- mental systems have used compression to pro- velopment programs.

A similar advance in electronic technology 21. John Keller, "AT&T Unveile a Chip Set That Lets PCs Be Used for Video Communication," Wall Street Journal, that reduces the need for high-speed networks April 3,1992, p. B2. is the splitting of electronic signals. This tech- 44 PROMOTING HIGH-PERFORMANCE COMPUTING AND COMMUNICATIONS June 1993 nique involves combining and splitting the As noted above, computer power can be sub- signals that come down the transmission line, stituted for communications power only to a substituting many small channels for a single limited extent. As computers become ubiqui- large one. tous, the need to link them rises. This linking is likely to be guided by economic consider- The technology is quite mature and has ations, which does not mean that it will inevi- been in use for a long time. When combined tably favor the consumer uses often described with some new technologies that shift signals by data communications forecasters. in time (called delay equalization) and with changes in pricing and service structures of long-distance carriers, it substitutes for high- Perspective on the Obstacles er transmission speeds. For instance, by com- to the Development of a bining seven 56,000 bit-per-second lines, a Commercial Market for HPCC video conference system can be transmitted Technology without the need for special lines. The long- distance carrier Sprint provides a commercial Obstacles to development are not insurmount- service (Healthcare Application Network De- able. If high-speed data transmission is costly, livery System) that uses this type of system to firms and individuals will be encouraged to transmit video conferences and medical im- find ways around the expensive component or ages. to make do with less. Although declining prices increase demand, so also does expansion in End users also employ these techniques to the pool of potential users. A case in point is split a large leased line into many small lines. the growth of local area networks that connect In fact, the leased lines described above are of- all the computers within a building or campus. ten used not for single large applications but Connecting two aggregations of computers to agglomerate many small applications, even and computer users is likely to be more valu- mixing voice and data transmissions. Some of able than connecting individual computers this equipment also has limited switching ca- that happen to be at great distances from each pacity. other. Thus, increases in the number of local area networks are likely to expand the de- Compression techniques and splitting pro- mand for larger networks. vide instances in which an advance in computational technology is substituted for an ad- These shaping forces are creating opportu- vance in communications technology. Since nities for entrepreneurs to make use of the the growth in computer technology does not type of technology that is being developed un- seem to be slowing, there is substantial reason der HPCC. In this perspective, the scenario to believe that these and other advances may for network technology looks better than that continue to delay the need for faster commu- for supercomputer R&D. This is not to say nications technology. In both compression that all the firms that use HPCC network technology and splitting technology, however, technology will be successful, or that none of market growth is hampered by lack of com- the firms attempting to market HPCC patibility and common standards. Much will supercomputer technology will be successful. depend on the extent to which producers can The outcome will depend on events that are reach agreement. beyond the scope of this study. Chapter Five Policy Directions and Conclusions

ommercial demand is not the only rea- apparent to anyone who drove a car or truck. son high-performance computing and The federal government's unique contribu- C communications technologies are be- tions were the financial resources to build the ing pursued. Federal agencies can use some highways and the legal power to get the land of the capabilities of high-performance com- and rights of way. In the HPCC case, many puting or communications--indeed those are private parties can afford to build major por- among the primary goals of HPCC. In the tions of the information infrastructure, but short term, however, agency missions are of- most people do not yet have a use for it. The ten in conflict with commercial success. Fed- appropriate federal role may be to help de- eral agencies differ from most potential users velop early applications of the new technology. of the new technologies in having more technical expertise and being somewhat less sen- In response to these concerns, the Congress sitive to costs. In setting policy for HPCC, is considering whether to expand the network policymakers must decide how much weight component of HPCC. The most commonly dis- to assign to each set of goals: the short-term cussed forms of this expansion would create goals of federal agencies and the long-term applications in four distinct areas under the goals of commercialization. rubric of an information infrastructure development program.1 The four areas are medi- The Computer Systems Policy Project cine, education, manufacturing, and libraries. (CSPP), in its evaluation of HPCC, argued that the computing problems the program in- o Medicine. The National Library of Medi- tended to address, the so-called grand chal- cine would develop network applications lenges, were too distant from those most peo- for communicating medical images, such ple encounter to provide much of a demonstra- as X-rays and CAT (computer-aided to- tion of the ways in which computer networks mography) scans, by computer network; could actually help people. They proposed build test-bed networks linking hospitals that the program focus on more mundane ap- and other medical centers to enable them plications. to share data and records; and develop network applications to provide long-distance Other commentators have suggested that medical care. the federal government should help create this new information infrastructure, just as it has o Education. The National Science Founda- played a role in creating other components of tion would be authorized to develop pilot infrastructure such as the Interstate Highway projects to connect U.S. elementary and System. In this case, however, the roles of the federal government and private parties might 1. Several versions of this option have been introduced in be reversed. When the Interstate Highway the 103rd Congress. S. 4, introduced in January, incor- System was built, its everyday benefits were porates such an option as Title VI; most of this legislation is unrelated to HPCC. H.R. 1757 and parts of H.R. 820 also incorporate much of this expansion. 46 PROMOTING HIGH-PERFORMANCE COMPUTING AND COMMUNICATIONS June 1993

secondary schools to Internet and to gen- erate software and training materials. Networks and Medicine o Manufacturing. The National Institute of Standards and Technology would develop The medical program is designed to improve and transfer electronically networked the use of information technology in manipu- manufacturing applications, including lating and transferring medical information, standards development. from centralized recordkeeping to medical imaging. The major questions regarding the o Libraries. Both the National Science Foun- sharing of patients' records are legal and or- dation and the National Aeronautics and ganizational as well as technical and eco- Space Administration are responsible for nomic: How does one organize a cost-effective developing prototype digital libraries and system that allows medical personnel timely the associated technology. NASA would access to useful information, yet preserves the make specialized government sensing data privacy of patients and satisfies the needs of available over Internet. medical organizations?

To pay for these programs, the proposals Medical imaging is seen as an early poten- would typically authorize an additional $1.0 tial user of high-speed data transmission. billion to $1.6 billion over the next five years. Hospitals and medical centers have to get X- If funded at the proposed level, total spending rays into the hands of radiologists and other for high-performance computing and commu- doctors for interpretation. Other medical im- nications technology would rise to between ages that doctors need to see on a regular basis $5.0 billion and $5.5 billion over the 1992- include angiograms, sonograms, CAT scans, 1996 time frame, depending on when the pro- and magnetic resonance imaging (MRI) scans. posal was passed, for an average of $1.0 billion (Cardiologists are already faxing electrocar- to $1.2 billion per year. diogram results to colleagues for diagnosis.)

Aside from these Congressional initiatives, Hospitals have been transmitting medical HPCC has shifted the focus of some research images over telephone lines to radiologists, and development to accommodate the CSPP who use personal computers in their offices or critique. For example, in choosing "grand homes. In this way a team of radiologists can challenge" supercomputing projects, the De- serve more than one hospital and make more partment of Energy has begun to select efficient use of their time. Because ordinary projects that will use the high speed of the X-rays contain so much data, they are difficult massively parallel supercomputer to provide to transmit over phone lines.2 Many other ra- answers to some applied research questions of diological images, for instance CAT scans and near-term economic significance. DOE has in- MRI scans, can be easily transmitted this way. cluded projects using computational chemistry Transmission time is slow, however, requiring to study various aspects of industrial pollution roughly half an hour to send a complete scan. and alternative ways of alleviating it. An- other project employs computational fluid dy- Because conventional telephone lines take namics to study industrial processes with the so long to transmit sophisticated images, aim of increasing energy efficiency and lower- medical centers cannot use them for most of ing the output of pollutants. DOE is also fund- their imaging needs. A large medical center ing R&D toward the modeling of petroleum reservoirs and groundwater. These projects 2. An ordinary X-ray can be converted by a laser scanner are in addition to the long-range ones to which into an array of data 2,000 numbers wide by 2,000 num- DOE is committed as part of its grand chal- bers high by 12 numbers deep (for shades of gray), or roughly 50 million bite of data. Even if the resolution is lenge problems, such as those involving high- reduced to 1,024 by 1,024, an X-ray still has over 12 mil- energy physics. lion bits of data. CHAPTER FIVE POLICY DIRECTIONS AND CONCLUSIONS 47 at the height of its work day might produce a switched service so that much of their radio- medical image every few seconds. For this logical work can be concentrated in regional reason, research in medical imaging has centers. A small fort or post may not have suf- turned to computer networks. ficient demand to justify a radiologist, and for a center that sends relatively few images (several dozen a day) the cost of using dedicated Types of Medical Imaging private lines would be exorbitant. Local ex- Communications change carriers are also beginning to offer some switched data services that allow users to pay only for the time the lines are in actual Medical imaging transmission needs can be divided into two types: for local areas within use. major medical centers and, for wider areas, be- Many radiological practices and individual tween medical centers of differing sizes or radiologists still use ordinary telephone lines, from medical centers and hospitals to individ- modems, and personal computers to transmit ual doctors and small practices. The two types medical images. Currently available technol- have different communications needs, and ogy has a lot of unused potential, often be- each is likely to find a different cost-effective cause the load does not warrant special invest- solution. ments. The ultimate transmission vehicle for small practices is still unclear. Local Communications. Several private demonstration projects have begun. For example, Washington University, which has a major radiology institute, has teamed up with Obstacles to Market private companies to develop a high-speed Development network to give radiologists rapid access to medical images.3 The objective is to create a By concentrating on medical imaging, HPCC network initially capable of delivering 155 may be able to avoid the fate of earlier million bits per second to 128 ports, and ulti- telemedicine demonstration projects. Earlier mately of delivering 620 million bits per sec- projects were unable to show that adopting so- ond to 256 ports phisticated broadband telecommunications equipment led to significant increases in de- Wide-Area Communications. Sprint has livered health care despite higher delivery demonstrated a system for transmitting medi- costs.4 Furthermore, many of the benefits cal images between medical centers that joins that did occur were easily obtained using a together Sprint's current switched services telephone. Most of these early experiments in- (56,000 bits per second). It allows customers volved replacing face-to-face contact between to join together temporarily several low-capac- doctor and patient with some electronically in- ity telephone lines to provide a single large- termediated contact, whereas much HPCC re- capacity telephone line without having to search will be concentrating on efficient use of build new capacity to accommodate the peak information within the health care delivery load. system.

This service permits communications be- Parallel with the need for transmission of tween centers that have a moderate amount of medical images is that of storing, cataloging, radiological- data to transmit. For instance, the armed services have contracted to have a 4. For a review of earlier effort0 to use sophisticated telecommunications technology in the provision of medical 3. Jerome Cox and Jonathan Turner, "Project Zeus: Design services, see David Conrath, Earl Dunn, and Christo- of a Broadband Network and its Application on a Uni- pher Higgins, Evaluating Telecommunications Technol- versity Campus" (Department of Computer Science, ogy in Medicine (Dedham. Mass.: Artech House, 1983), Washington University, St. Louis, Mo., 1991). pp. 197-217. 48 PROMOTING HIGH-PERFORMANCE COMPUTING AND COMMUNICATIONS June 1993 and retrieving these medical images. (These year of service and was growing rapidly. functions might fall under the purview of the Teachers, administrators, parents, and other library systems described below.) interested people connect to TENet an average of 3,000 times per day. TENet provides data None of the current systems has proved ade- bases to help teachers develop work plans for quate for all the uses to which it might be put. their classes. Some curriculum plans have For example, no software exists to help re- been developed cooperatively over the net- searchers search through X-rays for patterns work. Matchmaker services help teachers and to help them test theories about the develop- classes find correspondent teachers and ment of certain medical conditions. Such limi- classes to undertake joint projects--for exam- tations have prevented the electronic storage ple, comparing the cost of family groceries of medical images from becoming as widely across the state. In addition, TENet provides used as it might be, given the stage of hard- the usual computer network services--elec- ware development. Moreover, without soft- tronic mail, access to Internet, and file trans- ware that can electronically store, search, and fer. manipulate the images, the transmission of medical images becomes much less valuable. The Texas state legislature funded TENet To develop fully the market for electronic ex- only after a comprehensive plan had been change of medical images, these software drawn up to develop computer use for educa- problems will have to be addressed. tion in kindergarten through grade 12. To ensure success in reaching its broad target group, TENet piggybacked on the Texas High- er Education Network that serves the state universities. This move reduced TENet's costs Networks and Education and ensured the expertise needed to run such a system. TENet had a comprehensive train- Most versions of the education proposal would ing strategy to ensure that there would be a have the National Science Foundation develop trained representative in every school district pilot programs to connect primary and secon- in Texas. The system administrators also dary schools to NSFNET and Internet, and de- made every effort to ensure that the network velop software, systems, and training for included information that schoolteachers teachers and students. Although NSF could would find of immediate interest--curriculum develop materials and perhaps subsidize In- development materials, for example. They re- ternet connections, the ultimate responsibility wrote software to make it easier for teachers for setting up computer networks at the local to use, and made it affordable to encourage school level would most likely lie with state participation. Along with TENet, the State of and local authorities, who in fact have shown Texas also provided $100 million to fund tech- themselves quite willing to move rapidly. nology grants to local schools to encourage them to buy technology training, hardware (including calculators), and software or educa- Existing State Networking tional systems to help spread technology Programs throughout the state educational system.

Twenty to thirty states already have state- level computer networks dedicated to educa- Obstacles to Market tion. Although they vary in sophistication, Development the Texas Education Network (TENet) is often described as the most comprehensive. Started Whatever the efforts of HPCC in developing by a $1.2 million grant from the state legisla- applications, the trend toward extensive use of ture, TENet had 13,000 accounts in the first computer networks at the school classroom CHAPTER FIVE POLICY DIRECTIONS AND CONCLUSIONS 49 level will probably be limited, at least in the technology. They include funding for the near future, for a number of reasons. Only preparation of training materials, but not for one-third of elementary and secondary school training. teachers have had as much as 10 hours of computer training.5 Existing training focuses As noted above, entering Internet is a more on computer literacy than on how to use daunting experience. Without training or con- the computer in education. Most classrooms siderable experience, most teachers will have at the elementary and secondary level do not difficulty using any resource available even have telephone lines. Rewiring all the through the network, regardless of its quality. elementary and secondary schools would be Policymakers must decide whether the use of expensive, and most education budgets are al- technology in education would be advanced ready under pressure. Most of the existing more by spending several hundred million dol- computers in public schools are not appropri- lars to develop specialized network resources ate for network use and would require upgrad- or by training teachers to use existing net- ing or replacement. work resources.

Despite these limitations on near-term market development, federal agencies may be useful in providing leadership to the states' efforts in educational technology. Although Networks and many school districts are conducting sophisticated experiments, progress is uneven nation- Manufacturing wide, and every state and district has had to relearn expensive lessons. As the main spon- Federal agencies have been trying to promote sor of Internet, the federal government is also computer networking in manufacturing for in a unique position to ensure that this re- years without much success. The National In- source is made available to education. As use stitute of Science and Technology (NIST) has of computer networks becomes more wide- been very active in fostering several manufac- spread, schools may have to introduce stu- turing-oriented protocols for computer com- dents to computer networks in order to pre- munications. These protocols include GOSIP, pare them for their lives as adults in a net- the government version of the OSI (Open Sys- worked society. tems Interconnection) protocol, and MAP (Manufacturing Automation Protocol), but the The proposals under consideration suffer market for capital goods using these protocols from several shortcomings. First, they are de- has been very limited thus far. Commonly fined from the perspective of technology rath- cited reasons for the market's failure to take er than education. The National Science off include the inability of the vendors to agree Foundation, not the Department of Education, on standards, expensive and relatively primi- is the organizational locus. Policy seems to be tive technology, and cyclical problems in the focused on getting educators to use Internet major industries that would use the network. rather than using Internet to solve the problems educators face.

And, unlike the Texas educational technology endeavor, these proposals largely fail to Networks and Libraries address the program's main constraint--the lack of teacher training in advanced computer By contrast, libraries are readily taking to computer networks. The fit is quite natural: networks increase both the resources avail- 5. Lynn Olson, "Contrary to Predictions, Use of Technology able at any library and the potential user base in Schools Elueive," Education Week, Special Report (January 8,19921, p. 2. of any library. For example, one of the ser- 50 PROMOTING HIGH-PERFORMANCE COMPUTING AND COMMUNICATIONS June 1993 vices provided on the TENet allows Texas teachers on-line access to the U.S. Department of Education Research and Information Cen- Program Balance ter (ERIC) data base, giving teachers in distant communities access to much more infor- The proposals to expand the network compo- mation than would be available to them lo- nent of HPCC could provide additional fund- cally. Since many teachers use TENet in the ing for current programs, or they could replace evening, having libraries on-line means that current activities. Replacing current activi- they need to have fewer resources at home. ties with those suggested in these proposals For example, TENet makes encyclopedias would reduce HPCC's current focus on super- available. Many libraries also have their card computer R&D and result in a more even di- catalog on-line. vision between supercomputer R&D and network R&D. Similarly, the National Library of Medicine allows doctors to search the literature from re- The two options reflect the unresolved ten- mote locations. As in medical imaging, net- sion in the direction of HPCC. Is it ultimately works and medicine have begun to work well concerned with aiding federal missions by pro- together in this area. viding networks that link high-performance computers, or with developing technology to In addition to these public data bases, many make computing inexpensive, communica- private firms provide electronic data-base ser- tions technology easy to use, and computers vices, including many of the commercial net- therefore ubiquitous? work services discussed in Chapter 4. For example, Compuserve has information on airline Until now, the fact that the HPCC budget schedules, and various dial-up services sell da- was spread among different parts of the discre- ta for a fee. Most of these services, as well as tionary spending category mitigated this ten- the public data bases, are text retrieval ser- sion and prevented it from surfacing com- vices and lack experience with the much larg- pletely. Starting in fiscal year 1994, however, er data sets or protocol issues that come with the Budget Enforcement Act puts all discre- electronic imaging. tionary spending in a common pot. Policy- makers, for example, during the current bud- As noted in the medical section above, sub- get debate can remove supercomputer R&D stantial questions arise in making digital li- funds from the Advanced Research Projects braries useful to researchers. In the medical Agency to pay for network R&D in the NSF, area, for instance, the inability of currently NIST, and NASA. Given the cuts in discre- available software to sift through medical im- tionary spending mandated by the act, policy- ages according to predetermined criteria has makers may have to decide which part of limited the usefulness of networks to radiolog- HPCC is most critical. Furthermore, the ical research. agencies centrally involved in HPCC--NSF and NASA--experienced real declines in bud- HPCC is already substantially involved get authority in 1993 and will have to make with the creation of a national storage system internal decisions regarding the different com- as part of NSF efforts to integrate all the re- ponents of HPCC. sources at its supercomputer centers. This effort includes promoting standards for filing systems that would be accessible by network and creating systems for distributed mass storage--that is, permitting the integration of Conclusions data that are stored in different computers at different sites. Using such data requires uni- The different elements of HPCC have different versally accepted conventions and protocols. potentials. Clearly, graduate students trained CHAPTER FIVE POLICY DIRECTIONS AND CONCLUSIONS 51 under the human-capital part of HPCC are parallel processors on the limited number of likely to find jobs in academia where they will problems they are uniquely qualified to an- educate the next generation of computer pro- swer will provide a sufficient distinction in the grammers, and in industry where they will marketplace is not yet clear. write computer programs. It is less easy to foresee the outcome of other parts of HPCC. Network Technology

Supercomputer Technology Demand for computer communications, by contrast, is growing quite rapidly, and there is The scalable massively parallel supercom- no dominant incumbent technology. Further, puter systems that HPCC is supporting must a large component of the federal effort in this overcome very large obstacles before they can area is addressed not so much to technology succeed commercially. As noted in Chapter 3, development as to institution building, an ar- growth in the workstation market may pre- ea in which the federal government has a com- empt substantial growth in the supercom- parative advantage because it is perceived as puter market. If that occurs, firms may not be a disinterested party. Although Internet has willing to make the large investments in prod- done much to push back the frontiers of tech- uct development necessary to bring massively nology, its major achievement has been to cre- parallel supercomputer systems to market. If ate a network of individuals and organizations they do bring them to market, they may not who have learned how to work together in sell enough of them to pay for those invest- maintaining lines of communication. But ments. even though the field of data communications is growing rapidly, this growth will not nec- The supercomputer market is already oc- essarily translate into a demand for services cupied by a successful incumbent, the vector providing the types of speed HPCC is designed supercomputer, sales of which have riven from to provide. $100 million to $1.5 billion over the last decade. This incumbent is armed with a backlog The next arena for policy discussion is of well-written, fine-tuned, ready-to-run soft- whether to begin a second phase of HPCC that ware. Nor is it standing still; rather, it is ad- would develop more applications for its net- vancing in technology, even becoming more work capability. Members of Congress and in- parallel, albeit at a less rapid pace than the dustry have suggested developing and dem- massively parallel supercomputers. onstrating such applications in areas as di- verse as education, medicine, and manufactur- Dislodging incumbents in computer mar- ing. Legislation has been introduced that kets is not easy. New entrants usually go would authorize $1.0 to $1.6 billion over the around their competitors by introducing new next five years for this purpose. In some of the capabilities or sizes or price levels. The mini- areas the federal program could build on, computer avoided direct competition with the there is already substantial activity. In addi- mainframe computer, as the personal com- tion, federal, state, and local governments al- puter did with the minicomputer. Whether ready play a substantial role in both education the additional speed provided by massively and medicine.

Appendixes

Appendix A Supercomputer Component Technology Spinoffs

his appendix describes major areas of Thus, it is not surprising when features that computer technology and shows in appeared first on supercomputers eventually each case how supercomputer technol- appear elsewhere. When other computer mak- ogy requirements differ from the require- ers copy a popular feature, this is a response to ments of the rapidly growing parts of the com- perceived demand and not necessarily a spin- puter market. In general, it is these other off of the same technology. In fact, the ability parts of the market that are leading the devel- to provide the same feature in a more conve- opment of technology. Table A-1 outlines the nient package at lower cost is characteristic of areas of computer technology that are touched technological advance. on and notes how the requirements of the supercomputer market differ from those of other computer markets.

The word "technology" as used in this report Integrated Circuits means the ability to perform certain work at a The largest advances in integrated circuits given cost level: technology is defined in both have been found in complementary metal- technical and economic terms. That two com- oxide-semiconductor (CMOS) technology.2 puters or electronic components have the same This type of integrated circuit permits low capabilities or features does not mean they power usage with a high transistor count (or have the same technology. For example, in density) at relatively low cost, but it is rela- 1973 the XEROX Palo Alto Research Center tively slow. Throughout the 1980s, most of the invented the ALTO computer, which had microprocessors and memory chips at the many of the features now commonly seen in a heart of the personal computer increasingly state-of-the-art personal computer, including employed CMOS or its metal-oxide-semicon- graphical-user interfaces, networks, a point- ductor (MOS) relatives. The slowness of the ing device like the "mouse," and so forth. Al- chips did not matter to most personal com- though the technical features are the same, puter or workstation buyers given the rela- the technology of the computers is different: tively low cost and high number of the compo- the XEROX machine would have cost $25,000, nents. (A higher transistor count allows an in- while the personal computer with these fea- tegrated circuit to perform more tasks or re- tures now costs less than $1,000.1 The compo- tain more information in its memory.) nents have changed, the software has changed, the way they work together has By contrast, conventional supercomputer changed, and the way they are manufactured designers have had to work with emitter- has changed.

2. For a simple if dated description of CMOS and other semiconductor devices, see James D. Meindl. "Microelec- 1. Robert Cringely, Accidental Empires (New York: tronic Circuit Elemente," Scientific American (Septem- Addieon-Wesley Publiehing Co., 19921, pp. 83-84. ber 1977), pp. 70-81. 56 PROMOTING HIGH-PERFORMANCE COMPUTING AND COMMUNICATIONS June 1993

Table A-1. Major Computer Technologies

lnteqrated Circuits Charac- Transister Archi- Printed Circuit Boards Operating Type teristics Count tecture Packed Layers Cooling Disks Software

Supercomputers ECL Fast Relatively Vector Densely Many Liquid Fast Unix Few Arrays

Other Major CMOS Cheap Many RlSC Less Few Air Cheap Unix Computer Densely and Markets Other

SOURCE: Congressional Budget Office.

NOTES: ECL = emitter-coupled logic; CMOS = complementary metal-oxide-semiconductor; RISC = reduced-instruction-set computing. coupled logic (ECL), a fundamentally different supercomputer technology is beginning to integrated circuit technology that, although it benefit from the technology developed for the has fewer components and uses more power, is ubiquitous and inexpensive personal com- substantially faster. Since supercomputers puter and its more expensive cousin, the en- are optimized for speed, they usually have not gineering workstation. had the option of using the CMOS devices. Thus, the ECL markets and MOS markets Within their market segment, however, su- have been relatively separate; even the pro- percomputer designers have improved or been ducers of these different integrated circuits among the first to adopt new ECL technology are not necessarily the same.3 as it became available, and thus have influ- enced mainframe computer technology, often As the speed of workstations has increased, by pushing their vendors. For instance, Cray CMOS devices have proved too slow. The Research bought its ECL gate arrays from BiCMOS technology for integrated circuits vendors and pushed them to increase the com- was developed to meet the need for dense but plexity of the metal layers and to increase the fast components. This technology is much number of layers from three to four.6 The ven- faster than the conventional CMOS, but still dors subsequently made these improvements has quite a high transistor count. The engi- available to other ECL clients, primarily in neering workstation market has been the pri- mainframe computers. Before its current gen- mary driver of the development of this tech- eration of supercomputers, however, Cray Re- nology.4 Now both mainframe and supercom- search mainly bought its integrated circuits as puter producers are looking at BiCMOS tech- off-the-shelf components, preferring to con- nology, especially in memory, for their next centrate its research and development efforts generation of computers.5 In this sense, on novel packaging and interconnections.

5. David Patterson, "Expert Opinion: Traditional Main- 3. See Harold Dozier, "Superchips for Supercomputing," frames and Supercomputers Are Losing the Battle," IEEE Spectrum (September 19921, pp. 66-68. IEEE Spectrum (January 1992), p. 34.

4. Occasionally, a CMOS workstation microprocessor de- 6. A gate array is a kind of semicustom integrated circuit sign will be implemented in ECL technology, but such composed of a matrix of logic gates (or switches). The implementations are not driving the technology of work- layers of metal connecting these gates give each inte- station microprocessors. grated circuit its unique electronic character. APPENDIX A SUPERCOMPUTER COMPONENT TECHNOLOGY SPINOFFS 57

software programs, because they are composed of many simple instructions, typically take up Computer Architecture more memory than equivalent programs designed for CISC.) While many supercom- The other major innovation in computer tech- puters have had elements of RISC--mainly a nology of the last decade was reduced-instruc- simplified instruction set--in them since the tion-set computer (RISC) architecture, which 19609, supercomputers did not lead in RISC is at the heart of contemporary workstations design. The engineering workstation, which and allows them great speed despite the use of is the major market for RISC, was the primary CMOS components. Most mainframe and per- force behind software and hardware develop- sonal computers, by contrast, still use com- ment in this market. plex-instruction-set computer (CISC) architecture. As RISC architecture first developed in The central difference between the two microprocessors for engineering workstations, types of architecture is in the number and designers copied features from existing main- types of instructions a computer understands. frame and supercomputer designs. The high- A RISC machine understands relatively fewer performance features most commonly copied instructions, but executes them rapidly. A included pipelining, which, as discussed in CISC computer has a larger vocabulary with Chapter 3, involves creating an assembly line more complex instructions, but executes them of instructions, and cache memory, which more slowly. functions as a clipboard for frequently used items in the computer memory and gives fast The insight that led to RISC is that most access to these items. computer tasks actually involve a relatively small number of operations, such as adding However, as CMOS integrated circuit tech- two numbers, so it is advantageous to be able nology progressed, RISC designers were able to perform these common operations rapidly, to advance the use of these features very ag- rather than to be able to perform a wider array gressively. According to John Hennessy, one of operations more slowly. By way of compari- of the original RISC designers, "a number of son, when RISC was first introduced, the ma- [RISC] microprocessor-based machines have jor CISC families were capable of performing used pipelining and cache techniques that are between 200 and 300 instructions, while the more advanced than those in use in the big- early RISC computers had only between 40 gest machines."g The ease of redesigning and 120 instructions.7 large-scale integrated circuits, relative to redesigning conventional central processing IBM first started work on RISC architecture units, encouraged rapid development of RISC in the 1970~1,and other research was under- architecture. Thus, although RISC architec- taken at Berkeley and Stanford, although the ture inherited much of its technology from ex- latter two projects were unaware of the IBM isting high-performance computers, it moved efforts.8 RISC architecture became wide- at a much faster pace and soon began advanc- spread as the price of computer memory fell ing technology on its own. dramatically throughout the 1980s. (RISC The Defense Department's Advanced Re- search Projects Agency (ARPA) paid for much 7. David A. Patterson, "Reduced Instructions Set Comput- of the early R&D on RISC at Berkeley and ers," Communications of the ACM [Association of Com- puting Machinery] (January 1985). pp. 8-21. Stanford as part of its work on very-large- 8. David Patterson and John Hemeasy, Computer Archi- tecture: A Quantitative Approach (San Mateo, Calif.: Morgan Kaufmam Publishers, 19901, pp. 189-191. For the IBM contribution, see Ronald Jurgen, "John Cooke: 9. John Hennessy and Norman Jouppi, "Computer Tech- Vision with Enthusiasm," IEEE Spectrum (December nology and Architecture: An Evolving Interaction," 1991), pp. 33-34. Computer (September 1991). p. 22. 58 PROMOTING HIGH-PERFORMANCE COMPUTING A ND COMMUNICATIONS June 1993 scale integrated circuits in the late 1970s.10 A growing number of workstations and ARPA also paid for much of the early R&D in- even high-end personal computers use more volved in developing the first engineering than one microprocessor, in a trend reminis- workstation and funded much of the early cent of but independent from the development R&D in massively parallel supercomputers. of massively parallel supercomputers.l3 Sun This juncture is no coincidence: ARPA was Microsystems, Silicon Graphics, and others al- funding several different avenues of research ready make or have announced products con- to achieve the same goal--faster computers. In taining more than one processor. Thus, while the words of one technology analysis, supercomputer technology is advancing by linking massive numbers of microprocessors, Where RISC architects sought to workstation technology is similarly advancing achieve efficiencies by incorporating by linking small numbers of the same RISC within microprocessor structure the microprocessors. Because manufacturers of best possible juxtapositions of cache vector supercomputers have had substantial memory, registers, message routers, experience with multiprocessors, computer de- and logic units so that they would in- signers are likely to have drawn on that exper- teract in the most efficient way, par- ience in producing multiprocessor worksta- allel architects sought to use separate tions.14 processors in communication (intercon- nect) structures that would achieve those kinds of efficiencies.11

Having grown primarily in workstation de- Printed Circuit Boards sign, RISC architecture is now moving to other areas of the computer market. Mainframe and Other Packaging producers have announced their interest in such architecture. In addition, the next gen- Supercomputer producers have confronted eration of IBM-compatible personal computers problems in the area of printed circuit boards is likely be powered by an Intel microprocessor well ahead of other producers of electronics (the Pentium) that contains elements of RISC equipment. In order to make the supercom- architecture. Lastly, many, if not most, of the puters run faster, producers have placed the massively parallel supercomputers are based electronic components close together and have on RISC microprocessors originally developed had to design sophisticated multilayer boards for workstations. As noted in the main text, to accommodate them.15 The fast components much, if not most, of the improvement in the used by supercomputers put out more elec- performance of massively parallel supercomputers during the last decade can be attrib- 13. Norris Parker Smith, "Workstations and Supercom- uted to the use of ever more sophisticated puters," Supercomputing Review (January 1992), pp. 50- workstation RISC microprocessors.12 51. 14. For a history of multiprocessing in workstations, see Patterson and Hennessy, Computer Architecture, p. 588ff. 10. Richard Van Atta, Sidney Reed, and Seymour Deitch- man, "DARF'A Technical Accomplishrnenta: A Historical 15. Supercomputer Systems, an DM-backed Cray spinoff. Review of Selected DARF'A Projects," Volume 11 (Alex- announced that the printed circuit board for its forth- andria, Va.: Institute for Defense Analysis, February coming supercomputer would have 78 layers. Terry 1990),pp. 17-A-1through 17-A-4. Chapter 17 and its ap- Costlow, "SSI Supercomputer to Use Superboards," Elec- pendix~concentrates on DARPA's efl'orts in integrated tronic Engineering Times (July 20, 19921, pp. 1 and 64. circuit and computer design technology in the later (Since this announcement, the company has had to close 1970s and early 1980s. for reasons unrelated to its circuit boards.) By contrast, one printed circuit board for the next generation micro- 11. Ibid., p. 17-23. processor for IBM-compatible personal computere will have only six layers. Rick Boyd-Merritt and Alan Pat- 12. For this history of improvement, see John Van Zandt, terson, "Asians [personal computer producers] Get Parallel Processing in Information Systems (New York: Chance to Plug in Pentium," Electronic Engineering John Wiley and Sons, 1992). Times (January 4,19931, p. 60. APPENDIX A SUPERCOMPUTER COMPONENT TECHNOLOGY SPINOFFS 59 tronic noise than slower, low-power devices; station or personal computer markets. Again, consequently, the supercomputer designers the mainframe computer market seems posi- have become expert at isolating both the in- tioned as the most likely beneficiary. tegrated circuits and the circuits that connect them, using special materials and other sophisticated techniques.

As conventional electronic equipment Computer Disks grows in sophistication, its producers will confront many of the same problems. For in- In the past, supercomputers have been at the stance, as workstation microprocessors be- forefront in many aspects of magnetic disk come faster, they outrun the electronic signal storage technology. Specifically, they have de- speeds that can be transmitted easily through manded ever-increasing speed from suppliers conventional printed circuit boards. Conse- of hard disk drives. This has meant faster quently, some have begun to turn to multi- readJwrite heads, multiple platters, and mul- chip modules, where the central processing tiple spindles, all of which have the potential unit (or microprocessor) and several periph- to be carried over to the general computer eral chips are contained within specially de- market. signed modules.16 Vendors who have had to supply supercomputer producers may find However, as with microprocessors, the some of their expertise applicable in these new needs of the more numerous desktop systems areas. have dominated the industry. The small systems have emphasized the achievement of size Because supercomputers were designed to and price goals rather than absolute perfor- use large numbers of relatively simple and hot mance. In fact, parallel with the use of large integrated circuits, their producers have con- numbers of inexpensive microprocessors to fronted problems of packaging and heat dissi- create a supercomputer, computer designers pation long before most other computer pro- have now begun to provide redundant arrays ducers have. Although most computers can be of inexpensive disk (RAID) subsystems as an cooled by air and fans, supercomputers often alternative to the larger disk drives used by need water and refrigerant systems.17 As oth- most mainframe and supercomputer producer computers increase their component count ers.18 Arrays of disks have been available on and speed they will confront the same package smaller systems and are beginning to migrate density and cooling problems that supercom- to mainframes. ARPA funded some of the ear- puter producers have for years. ly R&D in this area.

However, conventional computer producers will be constrained in ways that supercomputer producers are not. For example, workstation makers will want to retain the small size Supercomputer Software and relative portability of their systems. Giv- en these constraints, it is not clear how much At present, the vector supercomputers and of the unique supercomputer packaging and minisupercomputers made in the United cooling technology will migrate to the work- States largely use the same UNIX operating software as the engineering and technical computers. This is the same software that 16. Philip Koopman, Jr. and Daniel Siewiorek, "ICs: The runs on the engineering workstations that Brains of a Workstation," IEEE Spectrum (April 1992), pp. 52-53.

17. In fact, so much waste heat is generated that it can be used for other purposes. Supercomputer waste heat is 18. Terry Costlow. "RAIDs are Underway," Electronic En- sometimes used to heat buil.dings. gineering Times (January 6,1992), pp. 1 and 50. 60 PROMOTING HIGH-PERFORMANCE COMPUTING AND COMMUNICATIONS June 1993 dominate the design and technical uses. Much of the software originally developed UNIX is as ubiquitous in that world as the for supercomputers has found its way into the IBM-originated operating systems (DOS, workstation market. The scientific visualiza- MVS) are in the world of business and finance. tion of it originated in supercomputer applica- For example, since the mid-1980~1,Cray Re- tions and only in recent years, as workstations search has used UNICOS, a Cray version of have increased in power, has the software be- UNIX. Because it is a version of UNIX, gun to migrate. Many of the numerical analy- UNICOS can run many UNIX programs, sis applications have similarly migrated from which makes using these supercomputers sim- supercomputers to workstations. pler for those who are already familiar with UNIX software.19

19. Cray Research, "Directory of Applications Software for Cray Research Supercomputers" (Mendota Heights, Minn., January 1990). Appendix B Supercomputer Speed Calculations

his appendix discusses the assump- S is the speed of the individual processor (in tions and calculations underlying the millions of floating point operations per sec- T comparison of vector supercomputer ond), speed with massively parallel supercomputer speed. The Congressional Budget Office FP is the fraction of operations that can be (CBO) replicated the calculations of Peter performed in parallel, and Gregory in "Will MPP [Massively Parallel Processing] Always be Specialized?" which N is the number of processors. appeared in the March 1992 edition of Super- computing Review.1 Numerical problems presented to supercomputers for solution typically have two types of The simulations are based on Amdahl's Law processes: scalar operations, which involve and published estimates of the speed of compo- single numbers, and vector operations, which nents of different supercomputers, not on ac- involve strings of numbers. Each processor tual performance. Amdahl's law states that has a different speed for each type of opera- "the performance improvement to be gained tion. Each problem has a unique combination from using some faster mode of execution is of scalar and vector operations. Thus, the for- limited by the fraction of the time the faster mula was used to create polar cases, where mode can be used."2 In other words, the problems were 100 percent vector and 100 per- speedup attributable to the use of a massively cent scalar. Actual performance will lie be- parallel processor will be limited by the per- tween. Either way, except for instances where centage of a problem that can use parallel cal- problems permitted parallel calculations that culations. were almost entirely parallel, the vector supercomputer outperformed the massively par- The formula used to construct Figure 3 on allel supercomputer. For purposes of expo- page 27 was the following: sitional simplicity, CBO only presented the vector calculations. The addition of scalar performance would not substantially change the conclusions. In only one of the existing computers did inclusion of scalar performance where shift the crossover point by more than 1 percentage point, and there it only served to re- P is computer performance (in millions of inforce the advantage of vector supercom- floating point operations per second), puters.

The computers Peter Gregory chose for com- 1. See pp. 28-31. parison were the Cray Research Y-MP C90, 2. David Patterson and John Hemeasy, Computer Archi- Thinking Machines CM-5, and the Intel Para- tecture: A Quantitative Approach (San Mateo, Calif.: Morgan Kaufmann Publishera, 19901, p. 8. gon. These machines are the premier ma- 62 PROMOTING HIGH-PERFORMANCE COMPUTING AND COMMUNICATIONS June 1993 chines of three leading supercomputer produc- vector performance is based on published peak ers. The assumptions about their speeds (in speed. millions of floating point operations per second) and processor counts are shown below: The variable FP is defined as the percentage of operations that can be performed in par- Computer Scalar Vector Processor allel over the entire application. It assumes Speed Speed Count that parallel means that the entire number of processors of each machine will be used. This Y-MP C90 29 1,000 16 assumption is more favorable to the massively CM-5 4 128 1,000 parallel supercomputer as it is easier to find Paragon 7 75 2,048 problems that are divisible 16 ways than 1,000 or 2,000 ways. It also assumes instanta- When CBO performed sensitivity tests, neous communication between nodes, an as- even doubling the scalar speeds of the mas- sumption that again favors the massively par- sively parallel supercomputers did not sub- allel supercomputer, since it has more nodes. stantially alter the central conclusion. The Appendix C Network Speed Calculations

his appendix discuses the assumptions D is propagation delay because of the and calculations underlying the dis- amount of time it takes light signals to trav- cussion of computer network response el between locations. time.1 The calculation of computer network response time is based on the following for- Unlike Kleinrock, the Congressional Bud- mula: get Office does not assume that light travels at the same speed in a fiber optic cable as in a T= O+D vacuum. Light travels through fiber at roughly two-thirds the speed it travels (1-P) through interstellar space. Consequently, where CBO assumes that it takes roughly 10.5 mil- T is response time (in milliseconds), liseconds for a light signal to propagate through a fiber halfway across the continental F is the average file size on a network (in United States. Kleinrock assumed it would 1,000 bits), travel across the continental United States in 15milliseconds. C is the network capacity (in million bits per second), CBO assumed that the average size of files on the computer network was 1,000,000 bits, p is capacity utilization (in percentage of to- roughly the size of a FAX.2 tal capacity), and

2. Lawrence Roberts, "Gigabit Networks: Emerging Com- 1. The discussion is based largely on Leonard Kleinrock, mercial Applications & Opportunities" (presentation at "The LatencyIBandwidth Tradeoff in Gigabit Networks," the 1992 conference of the Technology Transfer Insti- IEEE Communications Magazine (April 19921, pp. 36-40. tute, Santa Monica, Calif., July 8-29.19921.

CONGRESSIONAL FIRST-CLASS MAIL POSTAGE & FEES PAID BUDGET OFFICE C.B.O. Second and D Streets, S. W. WASHINGTON, D.C. PERMIT NO. G-70 Washington, D.C. 20515

- OFFICIAL BUSINESS PENALTY FOR PRIVATE USE, $300

ISBN 0-1 6-041 828-3