CRAY XD1 DATASHEET Cray XD1™ Supercomputer Release 1.3 ■ Purpose-built for HPC — delivers exceptional application The Cray XD1 supercomputer combines breakthrough performance interconnect, management and reconfigurable computing ■ Affordable power — designed for a broad range of HPC technologies to meet users’ demands for exceptional workloads and budgets performance, reliability and usability. Designed to meet ■ Linux, 32 and 64-bit x86 compatible — runs a wide variety of ISV the requirements of highly demanding HPC applications applications and open source codes in fields ranging from product design to weather prediction to ■ Simplified system administration — automates configuration and scientific research, the Cray XD1 system is an indispensable management functions tool for engineers and scientists to simulate and analyze ■ Highly reliable — monitors and maintains system health faster, solve more complex problems, and bring solutions ■ Scalable to hundreds of compute nodes — high bandwidth and to market sooner. low latency let applications scale Direct Connected Processor Architecture Cray XD1 System Highlights The Cray XD1 system is based on the Direct Connected Processor (DCP) architecture, harnessing many processors into a single, unified system to deliver new levels of application � � � ���� � � � Compute� � � Processors performance. ���� � Cray’s implementation of the DCP architecture optimizes message-passing applications by 12 AMD Opteron™ 64-bit single directly linking processors to each other through a high performance interconnect fabric, or dual core processors run eliminating shared memory contention and PCI bus bottlenecks. Linux and are organized as six nodes of 2 or 4-way SMPs to deliver up to 106 GFLOPS* per chassis. Matching memory and I/O performance removes bottlenecks and maximizes processor performance. � � � ���� � � � R� �api� dArray™ Interconnect ���� � The industry’s fastest embedded switching fabric, the RapidArray interconnect uses 12 custom communications processors and a 96 GigaBytes (GB) per second nonblocking switching fabric per chassis to deliver 8 GB per second bandwidth �� between nodes with 1.7 microsecond MPI latency. Each � � chassis presents 24 RapidArray � ���� � � � � � � links externally with an � �� � � aggregate 48 GB per second bandwidth between chassis. � � � ���� � � � � � � Application Acceleration ���� � ������������������������������ Six Xilinx Virtex-4 Field Programmable Gate Arrays (FPGAs) per chassis attach to the RapidArray fabric for massively parallel execution of critical algorithm components, promising orders of magnitude performance improvement for target applications. � � � ���� � Highly modular, the Cray XD1 base unit is a chassis. Up to 12 chassis can be installed in a � � � � � Active Management � �� � cabinet. Multi-cabinet configurations integrate hundreds of processors into a single system. � A management processor Chassis Each Cabinet in each chassis, a realtime Compute Processors 12 144 operating system, distributed software and an independent Performance 106 GFLOPS* 1.27 TFLOPS* supervisory network operate Aggregate Switching Capacity 96 GB/s 1152 GB/s together to monitor, control and MPI Interprocessor Latency 1.7 µsec 2.0 µsec manage every aspect of the Aggregate Memory Bandwidth 77 GB/s 922 GB/s computer. The Cray XD1 system enables single system command Maximum Memory 96 GB 1.2 TB and control and provides Maximum Local Disk Storage 1.5 TB 18 TB extensive high availability features. * peak theoretical performance with 2.2 GHz Dual Core AMD Opteron processors CRAY XD1 DATASHEET The Cray XD1 HPC system is based on the � � � ���� � � � R� �api� dArray Interconnect DCP architecture. This innovative new computer ���� � unifies up to hundreds of processors into The Cray XD1 RapidArray interconnect a single, balanced computer, delivering directly connects processors over high-speed, exceptional performance, reliability and 1.7 low-latency pathways. This DCP architecture usability. The Cray XD1 architecture includes eliminates the memory contention and PCI microsecond MPI latency four key subsystems: bus bottlenecks. • Compute Environment • RapidArray Interconnect Each fully configured chassis includes: • Application Acceleration • 12 custom communications processors System Topologies • Active Management • a 96 GB per second nonblocking The Cray XD1 embedded switching fabric embedded switch fabric, which includes and 24 interchassis links enable a wide variety main and optional expansion fabrics � � of network topologies. Topologies initially � ���� � � � • Optimized communications libraries Compute� � � Environment ���� � supported include direct connect and fat tree. The Cray XD1 compute subsystem is composed A fat tree topology is ideal for larger systems of a high-performance Linux operating system (or small systems that are expected to grow) and AMD Opteron 64-bit processors. 96 GB/s that run applications that use all-to-one, one- to-all, or all-to-all communications. Systems up Direct Connect Bandwidth to 24 chassis can be supported with a single The AMD Opteron processor’s integrated nonblocking switching layer fat tree. Each half of the RapidArray fab- memory controller increases memory fabric per chassis ric forms a separate fat tree, with its own set bandwidth and reduces latencies. Three of external switches. HyperTransport™ links per processor provide up to 19.2 GB per second I/O bandwidth. RapidArray Communications Processors Lustre Global Parallel File System The Lustre parallel file system provides a high- Dual Core Processors Tightly coupled to the AMD Opteron performance, high-availability, object-based Dual Core AMD Opteron processors are processors and switching fabric, the storage architecture for the Cray XD1 system available as an option on the Cray XD1 RapidArray processors offload and accelerate and can scale to thousands of nodes. Lustre system. Two processing cores exist on a single communications functions from the Opteron runs natively across the Cray XD1 RapidArray die, doubling peak performance, without processor, freeing it to perform core compute interconnect, avoiding TCP/IP overheads. raising power consumption and heat levels. tasks and enabling concurrent computing and communication. The communications processors deliver up to � � � ���� � 8 GB per second bandwidth with 1.7 � � � � � Application Acceleration ���� � microsecond MPI latency between nodes. The communications processors also deliver The application acceleration subsystem Linux 1.1 microsecond latency on the randomly incorporates reconfigurable computing capabilities to deliver superlinear speedup for a wide variety ordered ring latency test, part of the HPC Challenge (HPCC) benchmark suite. With of targeted applications. of HPC codes interconnect bandwidth on par with memory Each Cray XD1 chassis can be configured bandwidth, a major system bottleneck is removed with six application acceleration processors, for many applications, improving performance Global Process Synchronization based on Xilinx Virtex-4 FPGAs, that can be and greatly simplifying software development. Optimizations to the Linux scheduler result in programmed to accelerate key algorithms. the synchronization of process execution RapidArray Embedded Switching Fabric Well suited to functions such as all_reduce across the system, minimizing time spent by operations, searching, sorting and signal applications at barriers, and increasing A 96 GB per second, nonblocking, crossbar switching fabric in each chassis provides processing, the application acceleration application performance as much as 50 subsystem acts as a coprocessor to the percent. four 2 GB per second links to each node and twenty-four 2 GB per second AMD Opteron processors, handling the interchassis links. computationally intensive and highly repetitive Standards Based algorithms that can be significantly The Linux operating system, combined with the accelerated through parallel execution. AMD Opteron processor’s support for 32 and RapidArray Communications Libraries 64-bit x86 compatible computing, allows users The RapidArray communications libraries The application acceleration processors to take advantage of a wide range of work in conjunction with the RapidArray are tightly integrated with Linux and the commercial and open-source applications. communications processors to streamline AMD Opteron processors and use standard communications protocols, bypassing the software programming APIs, removing a major Linux kernel where possible and optimizing obstacle to application speedup. the compute/communicate interaction. The RapidArray interconnect delivers outstanding performance for highly parallel applications built upon the MPI, shmem, and Global Arrays standards. Balance The Key to Exceptional Application Performance It takes more than simply adding processors to increase HPC application performance. MPI applications, such as simulation and modeling, are compute- and � � Self-Healing communicate- � ���� � � � � � � Active Management ���� � The Cray XD1 system provides extensive fault intensive. They must The active management subsystem delivers detection, isolation, and prediction capabilities, perform millions of coupled with automated proactive and reactive outstanding usability and reliability through calculations and then self-healing intelligence. single system command and control and self- exchange intermediate healing capabilities. results before beginning another iteration. Single System Command and Control The active management