Unsurpassed performance, ultrascalable computing

IBM Blue Gene Solution

Today Blue Gene ranks as the number one on the TOP500 list.1 BlueGene is poised to achieve an astonishing 360 peak teraflops when a 64 rack system with over 130,000 IBM PowerPC® processors is completely installed in 2005 at the Department of Energy’s/National Nuclear Security Administration’s Lawrence Livermore National Laboratory (LLNL). Initial instal- lation of 16 racks begins in November of 2004. The DOE entered into a part- nership with IBM in funding research and development of this machine in 2000 in order to explore designs for highly cost-effective computers. Supercomputing leadership Highlights The IBM® _`® Blue Gene® But Blue Gene is not just a super- Solution is the result of an IBM super- computer that delivers ultrascalable ■ Leadership performance in a computing project begun five years performance. It is also extremely effi- space-saving, power-efficient ago, dedicated to building a new cient. When compared with the package for the most demanding family of optimized for previous TOP500 leader, the Earth HPC applications bandwidth, scalability and the ability to Simulator, the 64 rack Blue Gene com- handle large amounts of data while plex will offer over 100 times the floor ■ Design innovations for advance- consuming a fraction of the power and space density, 25 times more perform- ment of science across a broad floor space required by today’s fastest ance per kilowatt of power, and nearly set of vital workloads systems. 40 times more memory per square foot.2 ■ Technology accessible to a wide range of scientists, researchers and developers With most new HPC application devel- Starting earlier this year with the avail- Widely accessible opment designed for clusters, efficient ability of operational hardware, hands- Blue Gene is a powerful multi-teraflop use of power and space now rivals on experiences with many differing system constructed from technology scalability as the biggest challenge. applications have shown that the that will lead to petaflop (1000 teraflops) Available in configurations ranging from Blue Gene architecture is applicable to performance, and many interested sci- one to 64 racks, Blue Gene is the inno- a broad set of parallel workloads found entists and engineers might conclude vative new solution from IBM that deliv- across a variety of disciplines. Today, that harnessing Blue Gene is simply ers an ultrascalable solution without IBM and its collaborators are exploring beyond their reach. However, IBM is sacrificing efficiency. a growing list of high performance planning steps that will make computing (HPC) applications including Blue Gene accessible to a wide range Broad workload applicability life sciences, financial modeling, of users. When chartered five years ago, the hydrodynamics, quantum chemistry, Blue Gene project had as its goal to molecular dynamics, astronomy and First, IBM plans to provide access to develop a massively parallel computer space research and climate modeling. Blue Gene through the Deep applied to the study of biomolecular Other promising areas of interest Computing Capacity on Demand phenomena such as protein folding. include Grid Computing, business (DCCoD) center. Clients with con- The effort would advance the under- intelligence, financial risk and strained budgets and limited require- standing of the mechanisms behind compliance, aerodynamics study and ments for access to a Blue Gene protein folding via large-scale simula- testing, and manufacturing processes. system could request time on the sys- tion, and explore novel ideas in mas- tem and just pay for the amount of sively parallel machine architecture As greater numbers of scientists and capacity reserved. In this way, clients and software. The level of performance researchers apply large-scale cluster could contract for variable capacity and provided by Blue Gene can enable a computing to a diverse set of complex services to help satisfy short term tremendous increase in the scale of problems and build a collective expert- planned or unplanned peak workloads. simulations beyond what is now possi- ise in parallel program development, the Remote access would be provided via ble with existing supercomputers. relevance of the Blue Gene architecture a dedicated Virtual Private Network Successful simulation studies of protein becomes clearer. The design innova- connection between the client site and folding on this scale are expected to tions of Blue Gene offer the promise IBM’s facility. Accessing Blue Gene advance the techniques, models, of advancing vital science across through the DCCoD center could help and algorithms used in biomolecular numerous disciplines. simulation. system to get started with Blue Gene. The system can be upgraded to 512 nodes if desired. The offer will be available only to members of the Consortium. For information about the Consortium, visit www.mcs.anl.gov/bgconsortium.

Innovative design for ultrascalable computing The Blue Gene system is built out of a very large number of compute nodes, each of which has a relatively modest clock rate contributing to both low power consumption and low cost. Blue Gene utilizes IBM PowerPC embedded processors, embedded clients quickly tap Blue Gene super- Finally, IBM intends to make available DRAM and system-on-a-chip tech- computing power while helping to lower-priced, reduced-sized Blue Gene niques that allow for integration of all reduce financial and technical risk. configurations. Under special arrange- system functions including compute ments with the Blue Gene Consortium, processor, communications processor, Second, Blue Gene systems can be to be led by Argonne National three cache levels, and multiple high leased and outright purchases can be Laboratory, IBM intends to offer for a speed interconnection networks with financed through IBM Global Financing limited time beginning in 2005, a sys- sophisticated routing onto a single (IGF). IGF can help clients control costs tem partially populated with 128 nodes ASIC. Because of a relatively modest with highly competitive rates, powerful for a special price. Offering an entry- processor cycle time, the memory is asset-management tools, and end- level price may enable clients that do close, in terms of cycles, to the proces- of-lease or end-of-life options that max- not need or cannot afford a full-sized sor. This is also advantageous for imize flexibility while minimizing risk. power consumption, and enables nodes, the system provides for a flexi- Blue Gene also requires a service node construction of dense packages in ble number of dual-processor I/O where the system administrator man- which 1024 dual-processor compute nodes which handle communication ages the complex, front-end nodes nodes can be placed within a single between compute nodes and other where end-users compile and launch rack. The current design goals of systems. jobs, and file servers for storing data. Blue Gene aim for a scalable super- computer having up to 65,536 com- The nodes are interconnected through Encompassing numerous design inno- pute nodes and target peak five networks: a 3-dimensional torus vations, Blue Gene can be scaled-up to performance of 360 Teraflops with network for point-to-point messaging previously unachievable levels of per- extremely cost-effective characteristics between compute nodes, a global col- formance while staying within the prac- and low power, cooling and floor space lective network for collective operations tical limitations clients face with power requirements. over the entire application, a global bar- consumption, thermal displacement, rier and network, a gigabit and available floor space. Blue Gene is built up as follows: two for machine control, and nodes per compute card, 16 compute another gigabit Ethernet network for Familiar software environment tuned cards per node board, 16 node boards connection to other systems. The 3D for Blue Gene per 512-node midplane and two mid- torus allows for each node to have low- Three fundamental principles were fol- planes in a 1024-node rack. Each latency, high-bandwidth interconnect lowed when the system software was processor can perform four floating with its six nearest neighbors and is designed for Blue Gene: simplicity, per- point operations per cycle. The majority useful on applications where locality of formance and familiarity. Driving toward of the nodes are applied to computa- computation is prevalent. The global simplicity in the software design has tion. Depending on the nature of the collective network is useful for speeding allowed development of software that application to be run on Blue Gene, the up commonly used MPI collective takes advantage of hardware features programmer may choose to employ communications constructs. And the to deliver high performance without both processors in a node for compu- global barrier network quickly synchro- compromising stability and security. tation or have one processor dedicated nizes state across all processors in the And by creating a programming and to handling message passing opera- system. administration environment based on tions. In addition to the compute familiar programming languages, libraries, job management tools, and written in , or C++. Many Further enhancing the Blue Gene soft- parallel file systems, clients benefit from of these routines will be tuned for the ware environment will be the inclusion the innovative design elements of Blue Gene architecture. of IBM General Parallel File System Blue Gene without facing a steep (GPFS). GPFS is a high-performance, learning curve. In support of parallel programming, shared-disk file system that will provide Blue Gene is offered with an MPI solu- fast data access from all nodes in a The front-end nodes of a Blue Gene tion that leverages the MPICH2 library Blue Gene complex. Applications can complex are the portals through which from Argonne National Laboratory readily access shared files using stan- programmers access the computational to produce an implementation that dard file system interfaces, and the core of the system. The front-end exploits the communication technology same file can be accessed concurrently nodes run a standard ® distribu- of the compute nodes. from multiple nodes. tion which provides a familiar platform from which users compile and debug For job submission and workload bal- The Blue Gene system also includes a programs and submit jobs. ancing, IBM LoadLeveler® will provide service node where the system admin- support for Blue Gene. LoadLeveler, istrator manages the complex through Blue Gene systems are supported by which has been used for years by the Core Management and Control standard IBM XL Fortran, C and C++ large-scale cluster clients provides a System (CMCS). The key functions of compilers for PowerPC that have been facility for building, submitting and pro- CMCS are system configuration, initial- augmented with a backend that takes cessing jobs and is designed to match ization, monitoring and operation. In the advantage of the dual floating-point unit application processing needs with future, these functions will be integrated that is unique to Blue Gene. available resources. into the IBM Cluster Systems Management product to provide addi- Programmers will be able to employ the In a Blue Gene environment tional management capabilities and a popular IBM Engineering and Scientific LoadLeveler will coordinate with a spe- single point of control for management Subroutine Library (ESSL), a state- cial scheduler function that selects a set across the platforms supported by of-the-art collection of over 400 mathe- of compute nodes to form a partition CSM. Also running on the service node matical subroutines that provide opti- that meets the size and shape is a DB2® relational database that is a mum performance for floating-point requirements specified by the user. repository for static and dynamic state engineering and scientific applications information. Blue Gene at a glance

Processor PowerPC 440 700 MHz; two per node

Memory 512 MB SDRAM-DDR per node

Networks 3D Torus—175 MB/sec in each direction Global Tree—350 MB/sec; 1.5 usec latency Global Barrier/Interrupt Gigabit Ethernet (JTAG) Gigabit Ethernet (outside)

Compute Nodes Dual processor; 1024 per rack

I/O Nodes Dual processor; 16 per rack (additional nodes optional)

Operating Systems Compute Node—Lightweight proprietary kernel I/O Node—Linux Front End and Service Nodes—SUSE LINUX Enterprise Server 9

Compilers IBM XL Fortran IBM XL C/C++

MPI Library MPICH2 tuned for Blue Gene

Performance Peak per rack (virtual node mode)—5.73 Teraflops Peak per rack (coprocessor mode)—2.86 Teraflops Linpack per rack (VN mode)—4.53 Teraflops

Power 28.14 kW power consumption per rack (maximum) 208 VAC 3-phase; 100 amp service per rack

Cooling Air conditioning 8 tons/rack (minimum) 2800 CFM (compute rack); 350 CFM (power supplies)

Acoustics 9.0 LwAD and 8.7 LwAm

Dimensions (includes air duct) Height—1958 mm Width—915 mm Depth—915 mm Weight—750 Kg

Price Pricing available IBM Engineering and Technology Services at .com/chips/templates/ets_contactus.html In summary, the Blue Gene system how-to information and defect fix lists The Blue Gene project has progressed software is a collection of end-user will be accessible via the Web. Optional far in the past five years. Hands-on components tuned for performance and on-site assistance will be available on a experience has clarified the extent to reliability that will also be familiar to per-incident basis. which Blue Gene can be applied to var- many existing cluster clients. In this ious computationally intensive work- way, clients will be able to become pro- Blue Gene—a commitment to Deep loads. Programs have been developed ductive on Blue Gene right away while Computing that expand the opportunities for benefiting from its architectural features. Blue Gene joins IBM’s broad portfolio of researchers to access the power of Deep Computing solutions that includes Blue Gene. Software packages that Backed by IBM POWER™ processor-based UNIX® have brought high value to IBM cluster Blue Gene is backed by a separately symmetric multiprocessor (SMP) sys- solutions will be enabled for Blue Gene, priced, comprehensive maintenance tems, Linux clusters, high-speed inter- helping to preserve the familiarity of the package that covers all hardware and connects, storage, workstations and an operating environment. software components. Maintenance extensive collection of software tools. and support will be provided Monday By leveraging the many choices, Collaborations with government labora- through Friday 8 A.M. to 5 P.M. local IBM has been the revenue marketshare tories, institutions of higher learning, time with a maximum two hour leader in the industry for large-scale applications and tools vendors, and response time objective from the local HPC computing every year since 19993. industrial clients are today producing IBM support team. The hardware main- The addition of an innovative solution new insights into the potential of tenance covers three years of parts like Blue Gene accompanied by a prod- Blue Gene. replacement. uct roadmap that stretches to petaflop performance before the end of the The outlook is positive that Blue Gene Clients will be able to call IBM or go decade demonstrates a significant will become a growing contributor to online to report Blue Gene problems. commitment to the Deep Computing the advancement of science in a way Technical documentation, descriptions community and the researchers who that is cost-effective, resource-efficient of Blue Gene problem resolutions, are pursuing solutions to the important and extremely scalable. challenges facing humankind. For more information To learn more about the IBM _` Blue Gene Solution, please contact your IBM marketing representative or visit the following Web site: © Copyright IBM Corporation 2004 IBM Corporation ibm.com/research/bluegene/ Integrated Marketing Communications Systems and Technology Group index.html Route 100 Somers, NY 10589 Produced in the United States November 2004 All Rights Reserved This publication was developed for products and/or services offered in the United States. IBM may not offer the products, features or services discussed in this publication in other countries. The information may be subject to change without notice. Consult your local IBM business contact for information on the products, features and services available in your area. All statements regarding IBM’s future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. IBM, the IBM logo, the e-business logo, _`, Blue Gene, DB2, LoadLeveler, POWER and PowerPC are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries or both. A full list of U.S. trademarks owned by IBM may be found at: ibm.com/legal/copytrade.shtml. UNIX is a registered trademark of The Open Group in the United States, other countries, or both. Linux is a trademark of Linus Torvalds in the United States, other countries, or both. Other company, product and service names may be trademarks or service marks of others. IBM hardware products are manufactured from new parts, or new and used parts. In some

1 Benchmark as of November 7, 2004 available at cases, the hardware product may not be new www..org. The Blue Gene results were and may have been previously installed. submitted to TOP500 organization on October Regardless, IBM warranty terms apply. 25, 2004. References in this publication to IBM products 2 “The System” Habata, et.al. or services do not imply that IBM intends to NEC Research & Development, Vol. 44, No. 1, make them available in all countries in which January 2003 IBM operates. 3 IDC High-Performance Computer QView Photographs show engineering and design published quarterly. IBM has been number 1 in models. Changes may be incorporated in terms of annual revenue every year since 1999, production models. in the Capability competitive segment, systems Copying or downloading the images contained configured and purchased to solve the largest, in this document is expressly prohibited without most demanding problems. the written consent of IBM.

DCS00764-USEN-00