NCCS Hardware Presented by Jim Rogers Director of Operations National Center for Computational Sciences NCCS resources October 2007 Control network summary 1 GigE 10 GigE routers UltraScience Network 7 Systems CRAY XT4 CRAY XIE TRACK BLUE/GENE P IBM LINUX VISUALIZATION IBM JAGUAR PHOENIX II NSTG CLUSTER HPSS Supercomputers 51,110 CPUs 68 TB Memory 4512 quad-core 2048 quad-core processors @ processors @ (23,662) (1,024) 2.3 GHz 850 MHz (56) (128) Many storage 336 TFlops 2.6GHz 0.5 GHz 18,048 GB 4096 GB 3 GHz 2.2 GHz devices 45 TB Memory 2 TB Memory Memory Memory 76 GB Memory 128 GB Memory supported 900 TB 44 TB 4.5 TB 9 TB 5 TB Total shared disk 250.5 TB Evaluation platforms Test systems Scientific visualization Backup • 144-processor Cray • 96-processor Cray XT3 lab storage 5 PB XD1 with FPGAs 35 megapixels data storage • SRC Mapstation Powerwall 5 PB • Clearspeed • BlueGene (at ANL) 2 Rogers_NCCS_Hardware_SC07 Hardware roadmap As it looks to the future, the National Center for Computational Sciences expects to lead the accelerating field of high-performance computing. Upgrades will boost Jaguar’s performance to 250 teraflops by the end of 2007, followed by installation of two separate petascale systems in 2009. 3 Rogers_NCCS_Hardware_SC07 Jaguar system specifications 54 TF 100 TF 250 TF 1.000 TF Compute 5,212 Dual-core 11,508 Dual-core 7,816 Quad-core 24,000 Multi-core processors 2.6 GHz Opteron 2.6 GHz Opterons Opterons Opterons SIO 82 Single-core 116 Dual-core 124 Dual-core 544 Quad-core processors 2.4 GHz Opteron 2.6 GHz Opteron Opteron Opteron Memory per 4 GB / 4 GB / 8 GB / 32 GB / socket/total 20 TB total system 45 TB total system 63 TB total system 768 TB total system Interconnect Seastar 1 Seastar 2 Seastar 2 bandwidth per Gemini socket 1.8 GB/s 4.0 GB/s 4.0 GB/s Disk space 120 TB 900 TB 900 TB 5-15 PB Disk 14 GB/s 5 GB/s Total 55 GB/s 240 GB/s bandwidth 4 Rogers_NCCS_Hardware_SC07 Phoenix – Cray X1E CRAY X1E: 1,024 vector processors, 18.5 TF • Ultra-high bandwidth • Globally addressable memory • Addresses large-scale problems that cannot be done on any other computer 5 Rogers_NCCS_Hardware_SC07 Jaguar – Cray XT4 Today: • 120 TF Cray XT4 • 2.6 GHz dual-core AMD Opteron processors • 11,508 compute nodes • 900 TB disk • 124 cabinets • Currently partitioned as 96 cabinets of catamount and 32 cabinets of compute node linux • November 2007: All cabinets compute node linux 6 Rogers_NCCS_Hardware_SC07 Jaguar – Cray XT4 architecture Cray XT4 scalable architecture • Designed to scale to 10,000s of processors • Measured MPI bandwidth of 1.8 GB/s • 3-D torus topology 7 Rogers_NCCS_Hardware_SC07 Phoenix – Cray X1E Phoenix Utilization by Discipline Astro 100 Atomic 90 Biology Chemistry 80 Climate Fusion ) 70 % Industry ( 60 n Materials o i t a 50 Other z i l i t U 40 30 20 10 0 Oct Dec Feb Apr Jun Aug 8 Rogers_NCCS_Hardware_SC07 Jaguar – Cray XT4 Jaguar Utilization by Discipline Astro 100 Biology Chemistry 90 Climate Combustion 80 Fusion Industry ) 70 Materials % ( Nuclear n 60 o i Other t a 50 z i l i t U 40 30 20 10 0 Oct Dec Feb Apr Jun Aug 9 Rogers_NCCS_Hardware_SC07 Contact Jim Rogers Director of Operations National Center for Computational Sciences (865) 576-2978 [email protected] 10 Rogers_NCCS_Hardware_SC07.