NCCS Hardware

Presented by Jim Rogers Director of Operations National Center for Computational Sciences NCCS resources October 2007 Control network summary 1 GigE 10 GigE

routers UltraScience Network 7 Systems

CRAY XT4 XIE TRACK BLUE/GENE P IBM LINUX VISUALIZATION IBM JAGUAR PHOENIX II NSTG CLUSTER HPSS Supercomputers

51,110 CPUs 68 TB Memory 4512 quad-core 2048 quad-core processors @ processors @ (23,662) (1,024) 2.3 GHz 850 MHz (56) (128) Many storage 336 TFlops 2.6GHz 0.5 GHz 18,048 GB 4096 GB 3 GHz 2.2 GHz devices 45 TB Memory 2 TB Memory Memory Memory 76 GB Memory 128 GB Memory supported 900 TB 44 TB 4.5 TB 9 TB 5 TB Total shared disk 250.5 TB

Evaluation platforms Test systems Scientific visualization Backup • 144-processor Cray • 96-processor Cray XT3 lab storage 5 PB XD1 with FPGAs 35 megapixels data storage • SRC Mapstation Powerwall 5 PB • Clearspeed • BlueGene (at ANL)

2 Rogers_NCCS_Hardware_SC07 Hardware roadmap As it looks to the future, the National Center for Computational Sciences expects to lead the accelerating field of high-performance computing. Upgrades will boost Jaguar’s performance to 250 teraflops by the end of 2007, followed by installation of two separate petascale systems in 2009.

3 Rogers_NCCS_Hardware_SC07 Jaguar system specifications

54 TF 100 TF 250 TF 1.000 TF

Compute 5,212 Dual-core 11,508 Dual-core 7,816 Quad-core 24,000 Multi-core processors 2.6 GHz Opteron 2.6 GHz Opterons Opterons Opterons

SIO 82 Single-core 116 Dual-core 124 Dual-core 544 Quad-core processors 2.4 GHz Opteron 2.6 GHz Opteron Opteron Opteron

Memory per 4 GB / 4 GB / 8 GB / 32 GB / socket/total 20 TB total system 45 TB total system 63 TB total system 768 TB total system

Interconnect Seastar 1 Seastar 2 Seastar 2 bandwidth per Gemini socket 1.8 GB/s 4.0 GB/s 4.0 GB/s

Disk space 120 TB 900 TB 900 TB 5-15 PB

Disk 14 GB/s 5 GB/s Total 55 GB/s 240 GB/s bandwidth

4 Rogers_NCCS_Hardware_SC07 Phoenix – Cray X1E

CRAY X1E: 1,024 vector processors, 18.5 TF

• Ultra-high bandwidth • Globally addressable memory • Addresses large-scale problems that cannot be done on any other computer

5 Rogers_NCCS_Hardware_SC07 Jaguar – Cray XT4

Today:

• 120 TF Cray XT4 • 2.6 GHz dual-core AMD Opteron processors • 11,508 compute nodes • 900 TB disk • 124 cabinets • Currently partitioned as 96 cabinets of catamount and 32 cabinets of • November 2007: All cabinets compute node linux

6 Rogers_NCCS_Hardware_SC07 Jaguar – Cray XT4 architecture

Cray XT4 scalable architecture

• Designed to scale to 10,000s of processors • Measured MPI bandwidth of 1.8 GB/s • 3-D torus topology

7 Rogers_NCCS_Hardware_SC07 Phoenix – Cray X1E

Phoenix Utilization by Discipline Astro 100 Atomic 90 Biology Chemistry 80 Climate Fusion

) 70

% Industry ( 60 n Materials o i t

a 50 Other z i l i t

U 40

30 20

10

0 Oct Dec Feb Apr Jun Aug

8 Rogers_NCCS_Hardware_SC07 Jaguar – Cray XT4

Jaguar Utilization by Discipline Astro 100 Biology Chemistry 90 Climate Combustion 80 Fusion Industry ) 70 Materials % (

Nuclear

n 60 o i Other t

a 50 z i l i t

U 40 30

20

10

0 Oct Dec Feb Apr Jun Aug

9 Rogers_NCCS_Hardware_SC07 Contact

Jim Rogers Director of Operations National Center for Computational Sciences (865) 576-2978 [email protected]

10 Rogers_NCCS_Hardware_SC07