Grids, the Teragrid, and Beyond

COVER FEATURE Grids, the TeraGrid, and Beyond

To enable resource and data sharing among collaborating groups of science and engineering researchers, the NSF’s TeraGrid will connect multiple high- performance computing systems, large-scale scientiﬁc data archives, and high-resolution rendering systems via very high-speed networks.

Daniel A. magine correlating petabytes of imagery and The TeraGrid will be one of the largest grid-based Reed data from multiple optical, radio, infrared, and HPC infrastructures ever created. As Figure 1 shows, National Center for x-ray telescopes and their associated data initially, the TeraGrid will tightly interconnect the Supercomputing archives to study the initial formation of large- National Center for Supercomputing Applications Applications I scale structures of matter in the early universe. (NCSA) at the University of Illinois at Urbana- Imagine combining real-time Doppler radar data Champaign; the San Diego Supercomputer Center from multiple sites with high-resolution weather (SDSC) at the University of California, San Diego; models to predict storm formation and movement the Argonne National Laboratory (ANL) Mathe- through individual neighborhoods. matics and Computer Science Division; and the Imagine developing custom drugs tailored to spe- California Institute of Technology (Caltech) Center cific genotypes based on statistical analysis of single for Advanced Computing Research. The Pittsburgh nucleotide polymorphisms across individuals and Supercomputing Center (PSC) recently joined the their correlation with gene functions. TeraGrid as well. Although they involve disparate sciences, these When operational, the TeraGrid will help research- three scenarios share a common theme. They depend ers solve problems that have long been beyond the on joining a new generation of high-resolution sci- capabilities of conventional supercomputers. It will entific instruments, high-performance computing join large-scale computers, scientific instruments, and (HPC) systems, and large-scale scientific data archives the massive amounts of data generated in disciplines via high-speed networks and a software infrastruc- such as genomics, earthquake studies, cosmology, ture that enables resource and data sharing by col- climate and atmospheric simulations, biology, and laborating groups of distributed researchers. high-energy physics. Just as today’s Internet evolved from a few re- RESEARCH INFRASTRUCTURE search networks, like the early Arpanet, we expect The National Science Foundation’s TeraGrid the TeraGrid to serve as an accretion point for inte- (www.teragrid.org) is a massive research computing grated scientific grids. Conjoining discipline-specific, infrastructure that will combine five large comput- regional, and national grids promises to create a ing and data management facilities and support worldwide scientific infrastructure. This active info- many additional academic institutions and research sphere will enable international, multidisciplinary laboratories in just such endeavors. Scheduled for scientific teams to pose and answer questions by tap- completion in 2003, the TeraGrid is similar in spirit ping computing and data resources without regard to projects like the United Kingdom’s National e- to location. It creates a new paradigm in which sci- Science grid and other grid initiatives launched by entists can collaborate anytime, anywhere on the the academic community, government organiza- world’s most pressing scientific questions. tions, and many large corporations. All are creat- ing distributed information technology infra- TERAGRID COMPUTING ARCHITECTURE structures to support collaborative computational HPC clusters based on the Linux operating sys- science and access to distributed resources. tem and other open source software are the Tera-

62 Computer Published by the IEEE Computer Society 0018-9162/03/$17.00 © 2003 IEEE Figure 1. TeraGrid partners map. Initially, the TeraGrid will provide an infrastructure that interconnects ﬁve large computing and data management facilities and supports additional academic institutions and research facilities.

Grid’s fundamental building blocks. Intel’s 64-bit “Building the Next-Generation Itanium Processor” Itanium2 processors, commercially introduced in sidebar provides information about the develop- July 2002, power these Linux clusters. The terascale ment of these processors. computing capabilities of each cluster accrue by The major components of the TeraGrid will be aggregating large numbers of Itanium2 processors located at the TeraGrid sites and interconnected by to form a single, more powerful parallel system. The a 40 Gbps wide area network:

Building the Next-Generation Itanium Processor Intel’s Itanium processor family is at the heart of processor. The Intel Itanium2 processor has 3 the TeraGrid, expected to be one of the world’s Mbytes of on-die cache. It also provides the high largest scientific high-performance computing infra- levels of RAS—reliability, availability, and service- structures. Gadi Singer, vice president and general ability—that high-performance computing needs. manager of Intel’s PCA Components Group has this to say about the Itanium processor development To perform floating-point calculations at high process: speed, Intel had to position many execution units on the chip. The Itanium2 processor contains six When you link computers to perform a single task, integer execution units plus two floating-point you must look at the processor from the outside units; with these, it can perform up to six instruc- in—what it must do—and the inside out—how it tions every cycle and up to 20 operations. The Ita- does the job. nium2 processor’s bus—the pathway for input and output—can transfer 128 bits at 6.4 Gbps. The High-performance computing tasks require scien- architecture also provides parallelism—the ability tific computing support on the processor, and that to activate all these resources simultaneously. means floating-point calculations and the capabil- ity to transfer a lot of information very quickly. Once we designed these features, there was still the Because the data space for most scientific problems challenge of how to efficiently link different pieces is huge, high-performance computing requires a of software so they could operate together very large address space. It requires 64-bit address- smoothly. The operating system, compilers, and ing (rather than conventional 32-bit addressing) other pieces are all part of the Itanium architecture and a lot of memory that is accessible near the design effort.

January 2003 63 A Brief History of High-Performance Computing

The desire for high-speed computing can be traced to the origins of modern computing systems. Beginning with Babbage’s analytical engine, a mechanical calculating device, the insatiable appetite for systems that operate faster and that can solve ever-larger problems has been the enabler of new discoveries. • Caltech will provide online access to large sci- World War II brought the first of the large, electronic calculating sys- entific data collections and connect data-inten- tems, based on vacuum tubes. Universities, laboratories, and vendors con- sive applications to the TeraGrid. Caltech will structed a series of high-performance systems from the 1950s to the 1970s. deploy a 0.4 Tflop Itanium cluster and associ- Of these, one of the most famous was Illiac IV, developed at the Univer- ated secondary and tertiary storage systems. sity of Illinois and part of the inspiration for HAL in the movie 2001: A Space Odyssey. When the Pittsburgh Supercomputing Center In 1976, Seymour Cray introduced the Cray 1, a water-cooled vector joined the project in August 2002, it added more machine that became synonymous with supercomputing. When IBM than 6 Tflops of HP/Compaq clusters and 70 introduced the personal computer in the early 1980s, forces for massive Tbytes of secondary storage to the TeraGrid, com- change in high-performance computing were unleashed. Moore’s law, plementing the original four sites. which states that transistor density—and therefore processing power— When completed, this US$88 million project will approximately doubles every 18 to 24 months, led to the high-perfor- include more than 16 Tflops of Linux cluster com- mance microprocessors found in today’s desktops, workstations, and puting distributed across the original four TeraGrid servers. sites, more than 6 Tflops of HP/Compaq cluster As microprocessor performance increased, high-performance cluster computing at PSC, and facilities capable of man- computing based on “killer micros” became feasible. Clusters have been aging and storing more than 800 Tbytes of data on around since the early 1980s, but clusters started to get serious attention fast disks, with multiple petabytes of tertiary stor- in 1994 as a result of the Beowulf project. In this project, NASA age. High-resolution visualization and remote researchers Thomas Sterling and Don Becker built a cluster from 16 com- rendering systems will be coupled with these dis- mercial off-the-shelf processors linked by multiple Ethernet channels and tributed data and computing facilities via toolkits supported by Linux. for grid computing. These components will be Within two years, NASA and the Department of Energy had fielded tightly integrated and connected through a network computing clusters with a price tag of under US$50,000 that operated at that will initially operate at 40 Gbps, a speed four more than 1 Gflop per second. Today, thousands of clusters based on the times faster than today’s fastest research network. Beowulf design are in use in laboratories around the world. Development of the Intel Itanium architecture has set the stage for the TERAGRID PARTNERS AND INFRASTRUCTURE next generation of clustered supercomputing systems. The Itanium2 The TeraGrid sites are working with private sec- processor, the second generation in the Itanium processor family, uses 64- tor and consortium collaborators to assemble and bit addressing and support for the high-performance floating-point oper- deploy the TeraGrid infrastructure: ations central to scientific computing. The Itanium2 processors are at the heart of the TeraGrid. • IBM is providing cluster integration, storage, and software; • Intel will provide high-performance, 64-bit • NCSA will house the largest of the TeraGrid Itanium and Itanium2 processors; computing systems. NCSA’s TeraGrid cluster • Myricom is the vendor for internal cluster will consist of more than 10 Tflops of com- networking; puting capacity powered by Intel’s Itanium • Oracle will provide database management processor family, more than 200 Tbytes of and data mining software; high-performance disk storage, and a large ter- • Qwest will deploy the 40-Gbps network tiary storage system. connecting the Illinois and California • SDSC will house the TeraGrid’s primary data sites; and and knowledge management facilities. This • Sun Microsystems will provide metadata will consist of an Itanium cluster with a peak management engines. performance of more than 4 Tflops, an IBM Power4 cluster with a peak performance of Intel’s Itanium architecture provides the high per- more than 1 Tflop, 500 Tbytes of disk storage, formance needed for floating-point intensive sci- and a large tertiary storage system. A Sun entific calculations. Its scalability will help keep Microsystems high-end server will provide a pace with the complex calculations associated with gateway to grid-distributed data. the scientific and engineering research projects that • Argonne will deploy high-resolution render- will exploit the TeraGrid infrastructure. The “A ing and remote visualization capabilities, sup- Brief History of High-Performance Computing” ported by a 1.25 Tflops Itanium cluster with sidebar provides information about the evolution of parallel visualization hardware. high-speed computing.

64 Computer What Is a Grid?

In the world of high-performance computing, a grid is an infrastructure that enables the integrated, collaborative use of high-end computing systems, networks, data archives, and scientific instruments that multiple All of the TeraGrid’s components will be integrated organizations operate. Grid applications often involve large amounts of using open source software like Linux and grid mid- data or computing and require secure resource sharing across organiza- dleware such as the Globus toolkit.1 Linux commu- tional boundaries—something that today’s Internet and Web infrastruc- 1 nity software tools for cluster management form the tures do not easily support. software base for clusters. Globus features grid soft- The grid infrastructure for science and engineering research typically ware for intracluster interactions such as security and includes the following elements: authentication for remote resource access, resource scheduling and management, distributed data man- • Smart instruments. Advanced scientific instruments, such as tele- agement, and wide-area communication. scopes, electron microscopes, particle accelerators, and environmental monitors—coupled with remote supercomputers, users, and RESEARCH OPPORTUNITIES databases—to enable interactive rather than batch use, online com- Scientific and engineering research problems rang- parisons with previous runs, and collaborative data analysis. ing from high-energy physics to geosciences to biol- • Data archives. Scientific data produced by large-scale computations ogy are awaiting TeraGrid deployment. All of these or obtained from high-resolution scientific instruments. Mining, projects share several common characteristics: extracting, and correlating data from distributed archives opens the way for new scientific discoveries. • complex, often multidisciplinary computa- • Distributed collaboration. Shared access to data, computing, and tional models that require access to the world’s discussion, often via high-bandwidth, multiway audio-video con- most powerful computing systems; ferencing. The Access Grid uses high-resolution video and multicas- • remote access to distributed data archives con- ting to support distributed meetings. taining observational data from a new gener- • High-performance computing systems. Increasingly, these systems are ation of high-resolution scientific instruments based on large numbers of commercial microprocessors connected by and distributed sensors; high-speed local networks and integrated via Linux and other open • real-time access to instruments when time-crit- source software to function as high-performance computing systems. ical phenomena such as severe storms or super- Laboratory clusters typically contain tens of systems, whereas clustered novas occur; and supercomputing systems contain hundreds or thousands of processors. • national and international collaborations that require worldwide interactions among indi- The Globus Project (www.globus.org/about/faq/general.html) and viduals and organizations. TeraGrid (www.teragrid.org) Web sites provide more details about grid computing. A broad range of research projects have similar needs, including environmental modeling, weather Reference prediction and climate change research, astronomy, 1. I. Foster and C. Kesselman, eds., The Grid: Blueprint for a New Com- ecology, and physics. In industry, projects such as puting Infrastructure, Morgan Kaufmann, 1998. supply chain management, distributed decision making, and even the design of advanced heavy equipment engines can benefit from computational LIGO also involves challenging computational grids. searches such as correlating detected gravity waves with astrophysical objects. Gravitational wave detection Within the next year, scientists from the Cali- Earthquake simulation fornia Institute of Technology’s Laser Interferom- Earthquake engineers with the National Science eter Gravitational Wave Observatory (LIGO) will Foundation’s George E. Brown Jr. Network for begin collecting more than 100 Tbytes of new data Earthquake Engineering Simulation (NEES) hope annually to further validate Einstein’s theories by to fuse observational data from distributed geo- attempting to detect gravitational waves. Scientists physical sensors and shake tables that provide data believe these gravity waves result from interactions on the strength of building materials with simula- between massive objects such as orbiting black tions of building dynamics and responses to earth- holes. quakes and physical attacks (www.nees.org). Because it must correlate putative events from The NEESgrid (www.neesgrid.org) will let engi- two detectors, one in Hanford, Wash., and the neers simulate engineering problems over high- other in Livingstone, La., LIGO requires grid infra- speed networks, adjust simulation parameters in structure for data analysis. The “What Is a Grid?” real time, and develop buildings and infrastructures sidebar describes the grid computing infrastructure. that are better able to withstand earthquakes and

January 2003 65 physical attacks. Such an infrastructure, Systems biology Deep-sky together with sensors embedded in the build- Roughly one of every 1,000 nucleotides differs ing, might have enabled engineers to more across individual humans. These single nucleotide observations and quickly access the technical causes for the polymorphisms are responsible for the rich diver- whole-sky surveys September 2001 collapse of the World Trade sity of human characteristics. They also provide the have generated Center towers. source of variability in our response to viral and more data in the bacterial attacks and the efficacy of drugs. Large- Digital sky surveys past 10 years scale gene sequencing, microarrays for testing gene High-resolution charge-coupled devices and expression, and other biological data-capture facil- than in the the deployment of large-aperture telescopes ities have presaged a new era in our understanding entire history are generating both deep-sky observations and of such biological processes. Fully understanding of astronomy. whole-sky surveys of unprecedented scale. gene coding for proteins and protein control of Recent examples include the Two Micron All metabolic processes via enzymes is one of the 21st Sky Survey (2MASS) and the Sloan Digital Sky century’s great scientific challenges. Survey (SDSS). More data has been captured Building on this understanding to create models in the past 10 years from these surveys than in the of cells, organs, and organisms that researchers entire history of astronomy. Correlating such data to can use to develop more effective drugs or repair identify new phenomena and test theories motivated genetic defects will require comparative analysis of the recent creation of the United States National genomes across species and individuals. It will also Virtual Observatory (NVO) and the European require grid-based distributed access to genomic, Astrophysical Virtual Observatory. protein, and metabolic databases and computa- The NVO (www.us-vo.org) and other virtual tional modeling of biological processes. observatories will create a distributed information The common theme across these examples is infrastructure for astronomy based on federated the combination of complex computations; large, databases containing ground- and space-based distributed data archives and instruments; and observational data. A separate institution will man- collaborative research teams. Developing and sup- age each principal sky survey, which includes data porting such an infrastructure is the TeraGrid taken at different wavelengths—optical, infrared, team’s goal. radio, and x-ray—and ranging in size from 3 to 40 Tbytes. The TeraGrid will make it possible to link USING THE TERAGRID: AN EXAMPLE and analyze surveys and to generate important data Consider the scientific vision with which we points and statistics across the entire visible uni- began—correlating petabytes of imagery and data verse, fundamentally changing astronomy research. from multiple optical, radio, infrared, and x-ray telescopes and their associated data archives to High-energy physics study the initial formation of large-scale structures The Standard Model of Physics is arguably one of matter in the early universe. The NVO and the of the most successful scientific theories in human TeraGrid make this possible. Atop the TeraGrid, at history. Its predictions have been repeatedly vali- NCSA we deploy copies of radio astronomy data- dated experimentally, and it successfully explains bases from the National Radio Astronomy Ob- electromagnetism and the weak and strong nuclear servatory Very Large Array and the Berkeley- forces. However, physicists still seek a broader “the- Illinois-Maryland Association Millimeter-Wave ory of everything” that integrates gravity and Array. Meanwhile, Caltech hosts copies of the SDSS, explains the origins of mass. One of the elusive par- the Roentgen Satellite X-ray Observatory, and ticles being sought is the Higgs boson, believed to 2MASS. Similarly, SDSC hosts a copy of the be the mechanism via which particles acquire mass. Palomar Digital Sky Survey. New accelerators like the Large Hadron Collider A research astronomer in Boston poses a query (LHC) being constructed at CERN (European to construct an unbiased sample of galaxy clusters Nuclear Research Organization) and its ATLAS (A to test structure formation models. The goal is to Toroidal LHC Apparatus) and the Compact Muon compare theoretical models with observed mass and Solenoid detectors will generate events at 100 Mbps. luminosity functions over a cosmologically signifi- An international grid of data archives, with the US cant redshift range and study the evolutionary anchor at the Fermi National Laboratory, will be effects. This requires correlation of data from mul- created to store and analyze this data during the tiple archives, computationally intense clustering, search for new particles. and high-resolution visualization.

66 Computer NSF PACI Program

The National Science Foundation PACI Program supports two partnerships: the National Computational Science Alliance (Alliance) and the National Partnership for Advanced Computational Infrastructure Grid software extracts the relevant data from the (NPACI). Both partnerships operate high-performance computing facil- database archives, then transfers the data to an analy- ities supporting computational science research by the national research sis engine at NCSA, where tasks are launched to community. Each partnership consists of a leading-edge site, the National Center for Supercomputing Applications (NCSA) in Urbana-Champaign • identify the signatures of galaxy clusters from for the Alliance, and the San Diego Supercomputer Center (SDSC) in San x-ray emissions of hot gas, and Diego for NPACI. The two partnerships also include more than 60 geo- • convolve optical and infrared data to select graphically distributed academic and national laboratory partners. and identify clusters, based on the presence of The TeraGrid project complements and builds upon the work of the radio source morphology variation in the cos- two partnerships. Launched in 1997, the PACI partnerships—led by mological microwave background at millime- NCSA and SDSC—seek to build an advanced computational and infor- ter wavelengths. mation infrastructure for open research in science and engineering. NCSA, SDSC, and their partners have built many of the tools needed to Researchers use the large data sets to construct operate grid-based distributed computing and information infrastruc- simulated surveys as functions of galaxy cluster tures. These tools include middleware such as the Globus toolkit, secu- mass, density, redshift, and other properties. They rity systems, interfaces for accessing grid resources, and applications use correlation analysis to compare the results of capable of running effectively on distributed systems. the different techniques, understand selection effects, and generate predictions of observed sample properties based on various theoretical models. The first of these, the cluster toolkit, builds on Concurrently, a remote rendering task is sched- the intense interest among the scientific, commer- uled at Argonne to generate 3D visualizations of cial, and computing research communities in the the predicted and observed structures and correla- evolution and development of cluster computing tion results. Third-party data transfer software using industry-standard building blocks and open such as gridFTP2 moves the visualizations to the source software. The cluster toolkit was developed astronomer’s desk in Boston for visualization. Just as part of the Open Cluster Group (www.open as we use an electrical appliance without knowing clustergroup.org), a collaboration between indus- the location of generators, transformers, or trans- try and academia that produced the Open Source mission lines, grid software hides all the details of Cluster Applications Resources software package. data staging, computation scheduling, and user- SDSC and the National Partnerships for Computa- identity authentication on remote systems. tional Infrastructure have developed a comple- mentary toolkit called ROCKS (rocks.npaci.edu). BUILDING THE GRID: The cluster toolkits let research groups quickly A DO-IT-YOURSELF TOOLBOX deploy Linux clusters for scientific computing on Today’s Internet began as a few research net- Intel processors that interoperate with other research works, each dedicated to serving a specific research clusters and the TeraGrid software environment. project, discipline, or community. These networks The toolkits include step-by-step guidance on clus- expanded as new sites joined and interconnected ter software installation and configuration, along with other networks to enable cross-network shar- with open source numerical and message passing ing. We believe the TeraGrid and other national libraries, tools, and monitoring software. Atop this research grids will play a similar role as accretion base, researchers can then use Globus, Condor,3 and points for a new, international network of high- other grid tools, packaged via the NSF Middleware performance computing systems, data archives, col- Initiative (www.nsf-middleware.org), to configure laboration systems, and scientific instruments. the cluster as a fully functional node on the grid. Implicit in such a vision is the need for mecha- The rapidly increasing size and complexity of sci- nisms that will let research groups, universities, and entific imagery has created a need for high-resolu- institutions quickly deploy their own grid infra- tion, scalable displays. For example, the charge- structure for connection to the burgeoning inter- coupled-device detectors on today’s telescopes pro- national grid. To simplify creation of clusters, grids, duce 8,000 × 8,000 pixel images, with even larger and collaboration and visualization systems, the images to come. Human-scale displays let research National Computational Science Alliance and groups collaboratively examine and correlate data. NCSA have developed a set of ready-made “in-a- A scalable display wall, built using the display box” software toolkits. The “NSF PACI Program” toolkit, achieves multimegapixel resolution by sidebar provides more information about the part- tiling the output from a collection of lightweight nerships that have developed these toolkits. liquid crystal display or digital light processing pro-

January 2003 67 jectors into a single image. A Linux cluster, aug- Generalizing this theme leads to the emergence mented with graphics accelerator cards, drives the of a ubiquitous infosphere, which scientists, projectors. researchers, engineers, businesses, and the public Finally, the Access Grid is an ensemble of net- will use to find and interact with information, tech- work, computing, and interaction resources that nology, and one another without concern for the supports group-to-group human interaction across grid’s technical infrastructure. If truly successful, the grid. Originally developed by Argonne National the global grid will become invisible, empowering Laboratory and enhanced by the Alliance, Access and enriching the human experience. grid nodes include large-format multimedia dis- With its distributed data archives, support for plays, presentation and interactive software envi- multidisciplinary teams, and advanced, high- ronments, and interfaces to grid middleware and performance computing and communication remote visualization environments. resources, the TeraGrid is one of the first steps along this path. Not only will it provide the resources needed to tackle today’s most demand- rids connect distributed computing systems, ing scientific and engineering problems, it will also data archives, scientific instruments, and col- serve as a catalyst in the development of tomor- G laboration systems. However, their true power row’s grid-enabled world. I lies in reducing or eliminating the barriers of time and space that make it difficult for distributed research groups to collaborate naturally and intu- Acknowledgments itively. The TeraGrid and its expansion as the Extended Terascale Facility are the result of the vision, lead- ership, and dedication of many people. Fran Berman (SDSC), Rick Stevens (Argonne), Paul Messina (Caltech), Michael Levine (PSC), and Ralph Roskies (PSC) are all co-leaders of the effort, in addition to my contributions.

REACH References 1. I. Foster et al., “Grid Services for Distributed System Integration,” Computer, June 2002, pp. 37-46. 2. B. Allcock et al., “Data Management and Transfer in HIGHER High-Performance Computational Grid Environ- ments,” Parallel Computing J., May 2002, pp. 749- 771. 3. J. Basney and M. Livny, “Deploying a High-Through- Advancing in the IEEE Computer Society put Computing Cluster,” High-Performance Cluster can elevate your standing in the profession. Computing, R. Buyya, ed., Prentice Hall, 1999. Application to Senior-grade membership recognizes ✔ ten years or more of professional expertise Nomination to Fellow-grade membership Daniel A. Reed is a principal investigator for the recognizes National Science Foundation TeraGrid and serves ✔ exemplary accomplishments in as the project’s chief architect. He is also director of computer engineering the National Center for Supercomputing Applica- tions and the National Computational Science GIVE YOUR CAREER A BOOST Alliance. He holds the Edward William and Jane Marr Gutgsell Professorship in the Department of UPGRADE YOUR MEMBERSHIP Computer Science at the University of Illinois at Urbana-Champaign. Contact him at reed@ncsa. computer.org/join/grades.htm uiuc.edu.

68 Computer