HPCG Pleiades

National Aeronautics and Space Administration

HPCG Pleiades

Bob.Ciotti@nasa.gov John Baron [email protected] NASA Ames Research Center

www.nasa.gov Pleiades HW Environment • 11, 472 compute nodes 246,048 x86 cores • 1,968 Sandybridge • 5,400 Ivybridge • 2,088 Haswell • 2,016 Broadwell • 938TB Memory • FDR Infiniband – dual rail hypercube • Additional task specific nodes • GPU • Xeon Phi (KNC+KNL) • 1024/512 cpu large shared memory • Large memory data analysis nodes • Front Ends • hyperwall viz/data analysis • + a couple hundred administration/management nodes of various types. Pleiades SW Environment

• LINUX • SLES11/12 (most user facing systems) • Red Hat/Centos (lustre servers)

• Lustre • NFS

• Continuous Availability • *All* software can be updated without full system dedicated outage • ‘rolllling updates for compute nodes • Suspend/Resume for service nodes (lustre/NFS servers, rack leaders)

• Compute nodes added/removed without dedicated system down SGI ICE Dual Plane – Topology

n0 n1 n2 n3 n4

n5 n6 n7

2x 11d hypercube n8 full 11d == 2048 vertices ib1 ib0 Pleiades – partial 11d - 1296 vertices (2592 across both cubes) http://en.wikipedia.org/wiki/User:Qef/Orthographic_hypercube_diagram s 7+8+,-9:;<=6;>3,2=?@,A=BCDE

! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! " " " " " " " ! ! ! ! ! ! ! ! ! +,-./0,1123 # $ % & ' " ! ( ) * # $ % & ' "

FCE,+<>:; )6

! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! & & & ' ' ' ' ' ' ' ' ' ' " " " 4,-./0,1123 ' " ! ( ) * # $ % & ' " ! ( ) *

(6 /CC:,+<>:;

! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! % % % % % % % % % & & & & & & & /,-./0,1123 ) * # $ % & ' " ! ( ) * # $ % &

FCE,+<>:; )6

"!6 ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! # # # # # $ $ $ $ $ $ $ $ $ $ % 1,-./0,1123 % & ' " ! ( ) * # $ % & ' " ! (

92 racks – 2008 ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ) * * * * * * * * * * # # # # # 0,-./0,1123 565 teraflops ! ( ) * # $ % & ' " ! ( ) * # $ (6 FCE,+<>:; )6

! ! ! ! ! ! ! ! ! ! ! ! ( ( ( ) ) ) ) ) ) ) ) ) 5,-./0,1123 #3 Top500 ' " ! ( ) * # $ % & ' " :+;+,-<=>?@9>A3,2@BC,7@DEFG

! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! " " " " " " " ! ! ! ! ! ! ! ! ! +,-./0,1123 # $ % & ' " ! ( ) * # $ % & ' "

HEG,+?A=> )9

! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! & & & ' ' ' ' ' ' ' ' ' ' " " " 4,-./0,1123 ' " ! ( ) * # $ % & ' " ! ( ) *

(9 /EE=,+?A=>

! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! % % % % % % % % % & & & & & & & /,-./0,1123 ) * # $ % & ' " ! ( ) * # $ % &

HEG,+?A=> )9

"!9 ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! # # # # # $ $ $ $ $ $ $ $ $ $ % 1,-./0,1123 % & ' " ! ( ) * # $ % & ' " ! (

/EE=,+?A=> ""9

" " " " " " " " " " * # # # # # # # # # 5,-./0,6123 ! ( ) * # $ % & ' "

HEG,+?A=> )9

" " " " " " " " " " ) ) ) ) ) ) ) * * * 7,-./0,6123 # $ % & ' " ! ( ) * 112 racks – 2009 /EE=,+?A=> ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ) * * * * * * * * * * # # # # # 0,-./0,1123 683 teraflops ! ( ) * # $ % & ' " ! ( ) * # $ (9 HEG,+?A=> )9

! ! ! ! ! ! ! ! ! ! ! ! ( ( ( ) ) ) ) ) ) ) ) ) 8,-./0,1123 ' " ! ( ) * # $ % & ' " :+;+,-<=>?@9>A3,2@BC,7@DEFG

! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! " " " " " " " ! ! ! ! ! ! ! ! ! +,-./0,1123 # $ % & ' " ! ( ) * # $ % & ' "

IEG,+?A=> )9

! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! & & & ' ' ' ' ' ' ' ' ' ' " " " 4,-./0,1123 ' " ! ( ) * # $ % & ' " ! ( ) *

(9 /EE=,+?A=>

! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! % % % % % % % % % & & & & & & & /,-./0,1123 ) * # $ % & ' " ! ( ) * # $ % &

IEG,+?A=> )9

"!9 ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! # # # # # $ $ $ $ $ $ $ $ $ $ % 1,-./0,1123 % & ' " ! ( ) * # $ % & ' " ! (

/EE=,+?A=> ""9

" " " " " " " " " " * # # # # # # # # # 5,-./0,6123 ! ( ) * # $ % & ' "

IEG,+?A=> )9 ""9

" " " " " " " " " " ) ) ) ) ) ) ) * * * 7,-./0,6123 # $ % & ' " ! ( ) * 144 racks – 2010 /EE=,+?A=> ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ) * * * * * * * * * * # # # # # 0,-./0,1123 969 teraflops ! ( ) * # $ % & ' " ! ( ) * # $ (9 IEG,+?A=> )9

! ! ! ! ! ! ! ! ! ! ! ! ( ( ( ) ) ) ) ) ) ) ) ) 8,-./0,1123 ' " ! ( ) * # $ % & ' "

/EE=,+?A=>

" " " ! " " " " " " " " " " " " % % % % % & & & & & & & & & & ' .,-./0,6123 % & ' " ! ( ) * # $ % & ' " ! (

IEG,+?A=> )9

" " " " " " " " " " " " " " " " # $ $ $ $ $ $ $ $ $ $ % % % % % H,-./0,6123 ! ( ) * # $ % & ' " ! ( ) * # $ :+;+,-<=>?@9>A3,2@BC,7@DEFG

! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! " " " " " " " ! ! ! ! ! ! ! ! ! +,-./0,1123 # $ % & ' " ! ( ) * # $ % & ' "

IEG,+?A=> )9

! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! & & & ' ' ' ' ' ' ' ' ' ' " " " 4,-./0,1123 ' " ! ( ) * # $ % & ' " ! ( ) *

(9 /EE=,+?A=>

! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! % % % % % % % % % & & & & & & & /,-./0,1123 ) * # $ % & ' " ! ( ) * # $ % &

IEG,+?A=> )9

"!9 ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! # # # # # $ $ $ $ $ $ $ $ $ $ % 1,-./0,1123 % & ' " ! ( ) * # $ % & ' " ! (

/EE=,+?A=> ""9

" " " " " " " " " " " " " " " " * * * * * * * # # # # # # # # # 5,-./0,6123 # $ % & ' " ! ( ) * # $ % & ' "

IEG,+?A=> )9 ""9

" " " " " " " " " " " " " " " " ( ( ( ) ) ) ) ) ) ) ) ) ) * * * 7,-./0,6123 ' " ! ( ) * # $ % & ' " ! ( ) * 156 racks – 2010 /EE=,+?A=> ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ) * * * * * * * * * * # # # # # 0,-./0,1123 1.08 petaflops ! ( ) * # $ % & ' " ! ( ) * # $ (9 IEG,+?A=> )9

! ! ! ! ! ! ! ! ! ! ! ! ( ( ( ) ) ) ) ) ) ) ) ) 8,-./0,1123 ' " ! ( ) * # $ % & ' "

/EE=,+?A=>

" " " ! " " " " " " " " " " " " % % % % % & & & & & & & & & & ' .,-./0,6123 % & ' " ! ( ) * # $ % & ' " ! (

IEG,+?A=> )9

" " " " " " " " " " " " " " " " # $ $ $ $ $ $ $ $ $ $ % % % % % H,-./0,6123 ! ( ) * # $ % & ' " ! ( ) * # $ :+;+,-<=>?@9>A3,2@BC,7@DEFG

! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! " " " " " " " ! ! ! ! ! ! ! ! ! +,-./0,1123 # $ % & ' " ! ( ) * # $ % & ' "

JEG,+?A=> )9

! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! & & & ' ' ' ' ' ' ' ' ' ' " " " 4,-./0,1123 ' " ! ( ) * # $ % & ' " ! ( ) *

(9 /EE=,+?A=>

! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! % % % % % % % % % & & & & & & & /,-./0,1123 ) * # $ % & ' " ! ( ) * # $ % &

JEG,+?A=> )9

"!9 ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! # # # # # $ $ $ $ $ $ $ $ $ $ % 1,-./0,1123 % & ' " ! ( ) * # $ % & ' " ! (

/EE=,+?A=> ""9

" " " " " " " " " " " " " " " " * * * * * * * # # # # # # # # # 5,-./0,6123 # $ % & ' " ! ( ) * # $ % & ' "

JEG,+?A=> )9 ""9

" " " " " " " " " " " " " " " " ( ( ( ) ) ) ) ) ) ) ) ) ) * * * 7,-./0,6123 ' " ! ( ) * # $ % & ' " ! ( ) * 168 racks – 2011 /EE=,+?A=> ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ) * * * * * * * * * * # # # # # 0,-./0,1123 1.18 petaflops ! ( ) * # $ % & ' " ! ( ) * # $ (9 JEG,+?A=> )9

! ! ! ! ! ! ! ! ! ! ! ! ( ( ( ) ) ) ) ) ) ) ) ) 8,-./0,1123 ' " ! ( ) * # $ % & ' "

/EE=,+?A=>

" " " ! " " " " " " " " " " " " % % % % % & & & & & & & & & & ' ""9 .,-./0,6123 % & ' " ! ( ) * # $ % & ' " ! ( "$I,B@K=> JEG,+?A=> )9

" " " " " " " " " " " " " " " " # $ $ $ $ $ $ $ $ $ $ % % % % % "!9 H,-./0,6123 ! ( ) * # $ % & ' " ! ( ) * # $

/EE=,+?A=> "!I,B@K=>

' ' ' ' ' " " " " " " " ! ! ! ! ! ( ( ( ( ( ( ( I,-./0,6123 % & ' " ! ( ) * # $ % &

+99>9,%,L@BCA +99>9,),L@BCA !'M')M'!"" !'M"*M'!"" :+;+,-<=>?@9>A3,2@BC,7@DEFG

! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! " " " " " " " ! ! ! ! ! ! ! ! ! +,-./0,1123 # $ % & ' " ! ( ) * # $ % & ' "

JEG,+?A=> )9

! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! & & & ' ' ' ' ' ' ' ' ' ' " " " 4,-./0,1123 ' " ! ( ) * # $ % & ' " ! ( ) *

(9 /EE=,+?A=>

! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! % % % % % % % % % & & & & & & & /,-./0,1123 ) * # $ % & ' " ! ( ) * # $ % &

JEG,+?A=> )9

"!9 ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! # # # # # $ $ $ $ $ $ $ $ $ $ % 1,-./0,1123 % & ' " ! ( ) * # $ % & ' " ! (

/EE=,+?A=> ""9

" " " " " " " " " " " " " " " " * * * * * * * # # # # # # # # # 5,-./0,6123 # $ % & ' " ! ( ) * # $ % & ' "

JEG,+?A=> )9 ""9

" " " " " " " " " " " " " " " " ( ( ( ) ) ) ) ) ) ) ) ) ) * * * 7,-./0,6123 ' " ! ( ) * # $ % & ' " ! ( ) * 170 racks – 2011 /EE=,+?A=> ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ) * * * * * * * * * * # # # # # 0,-./0,1123 1.20 petaflops ! ( ) * # $ % & ' " ! ( ) * # $ (9 JEG,+?A=> )9

! ! ! ! ! ! ! ! ! ! ! ! ( ( ( ) ) ) ) ) ) ) ) ) 8,-./0,1123 ' " ! ( ) * # $ % & ' "

/EE=,+?A=>

" " " ! " " " " " " " " " " " " % % % % % & & & & & & & & & & ' ""9 .,-./0,6123 % & ' " ! ( ) * # $ % & ' " ! ( "$I,B@K=> JEG,+?A=> )9

" " " " " " " " " " " " " " " " # $ $ $ $ $ $ $ $ $ $ % % % % % "!9 H,-./0,6123 ! ( ) * # $ % & ' " ! ( ) * # $

/EE=,+?A=> ,)1 "!I,B@K=> ""1

' ' ' ' ' " " " " " " " ! ! ! ! ! ( ( ( ( ( ( ( I,-./0,6123 % & ' " ! ( ) * # $ % & "!1 JEG,+?A=>,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,)1 +99>9,%,L@BCA +99>9,),L@BCA !'M')M'!"" !'M"*M'!"" ' ' " ! :,-./0,6123 ! ( NASA (Pleiades) Rack Layout

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 A (ICE DDR) 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1

Hot Aisle 8d

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 3 3 2 2 2 2 2 2 2 2 2 2 1 1 1 B (ICE DDR) 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7

9d Cool Aisle

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4 4 4 4 4 4 4 4 4 3 3 3 3 3 3 3 C (ICE DDR) 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3

Hot Aisle 8d

10d 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 6 6 6 6 6 5 5 5 5 5 5 5 5 5 5 4 D (ICE DDR) 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9

Cool Aisle 11d

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 7 7 7 7 7 7 7 6 6 6 6 6 6 6 6 6 K (ICE QDR) 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1

Hot Aisle 8d 11d

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 9 9 9 8 8 8 8 8 8 8 8 8 8 7 7 7 L (ICE QDR) 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 182 racks – 2011 Cool Aisle 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 8 7 7 7 7 7 7 7 7 7 7 6 6 6 6 6 E (ICE DDR) 1.31 petaflops 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 9d Hot Aisle 8d

0 0 0 0 0 0 0 0 0 0 0 0 9 9 9 8 8 8 8 8 8 8 8 8 F (ICE DDR) 2 1 0 9 8 7 6 5 4 3 2 1

Cool Aisle

1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 4 4 4 4 4 3 3 3 3 3 3 3 3 3 3 2 11d I (ICE QDR) 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 15M cable Hot Aisle 8d

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 6 5 5 5 5 5 5 5 5 5 5 4 4 4 4 4 10d J (ICE QDR) 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5

Cool Aisle 10M cable 11D

2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 9 9 9 9 9 9 9 M (ICE QDR) 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 10D Hot Aisle 8D Added 4 racks Added 8 racks 02-28-20112 2 2 2 202-17-20112 2 2 2 2 1 1 1 1 1 1 1 1 1 0 N (ICE QDR) 8 7 6 5 4 3 2 1 0 9 11D cables from Row N to Gpgpu racks 223 and 224 Rack 205-218 altix ICE Row F may not arrive on 8400/EX westmere – new time. addition to pleiades NASA (Pleiades) Rack Layout

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 A (ICE DDR) 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1

Hot Aisle 8d

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 3 3 2 2 2 2 2 2 2 2 2 2 1 1 1 B (ICE DDR) 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7

9d Cool Aisle

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4 4 4 4 4 4 4 4 4 3 3 3 3 3 3 3 C (ICE DDR) 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3

Hot Aisle 8d

10d 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 6 6 6 6 6 5 5 5 5 5 5 5 5 5 5 4 D (ICE DDR) 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9

Cool Aisle 11d

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 7 7 7 7 7 7 7 6 6 6 6 6 6 6 6 6 K (ICE QDR) 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1

Hot Aisle 8d 11d

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 9 9 9 8 8 8 8 8 8 8 8 8 8 7 7 7 L (ICE QDR) 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 186 racks – 2011 Cool Aisle 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 8 7 7 7 7 7 7 7 7 7 7 6 6 6 6 6 E (ICE DDR) 1.33 petaflops 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 9d Hot Aisle 8d 8/9/2011- NASA planning to 0 0 0 0 0 0 0 0 0 0 0 0 remove rack 92 9 9 9 8 8 8 8 8 8 8 8 8 F (ICE DDR) from Pleiades and 2 1 0 9 8 7 6 5 4 3 2 1 use as test rack. Cool Aisle

1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 4 4 4 4 4 3 3 3 3 3 3 3 3 3 3 2 11d I (ICE QDR) 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 15M cable Hot Aisle 8d

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 6 5 5 5 5 5 5 5 5 5 5 4 4 4 4 4 10d J (ICE QDR) 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5

Cool Aisle 10M cable 11D

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 A (ICE DDR) 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1

Hot Aisle 8d

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 3 3 2 2 2 2 2 2 2 2 2 2 1 1 1 B (ICE DDR) 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7

9d Cool Aisle

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4 4 4 4 4 4 4 4 4 3 3 3 3 3 3 3 C (ICE DDR) 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3

Hot Aisle 8d

10d 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 6 6 6 6 6 5 5 5 5 5 5 5 5 5 5 4 D (ICE DDR) 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9

Cool Aisle 11d

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 7 7 7 7 7 7 7 6 6 6 6 6 6 6 6 6 K (ICE QDR) 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1

Hot Aisle 8d 11d

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 9 9 9 8 8 8 8 8 8 8 8 8 8 7 7 7 L (ICE QDR) 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 158 racks – 2012 Cool Aisle

*Note: Harpertown Racks O (ICE FDR) 1.15 petaflops Removed 3/21/2012 in preparation for SGI ICE X Racks installation. I/O 9d Hot Aisle deinstall Racks remain 12d P (ICE FDR)

Cool Aisle

1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 4 4 4 4 4 3 3 3 3 3 3 3 3 3 3 2 12d I (ICE QDR) 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 15M cable Hot Aisle 8d

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 6 5 5 5 5 5 5 5 5 5 5 4 4 4 4 4 10d J (ICE QDR) 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5

Cool Aisle 10M cable

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 A (ICE DDR) 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1

Hot Aisle 8d

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 3 3 2 2 2 2 2 2 2 2 2 2 1 1 1 B (ICE DDR) 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7

9d Cool Aisle

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4 4 4 4 4 4 4 4 4 3 3 3 3 3 3 3 C (ICE DDR) 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3

Hot Aisle 8d

10d 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 6 6 6 6 6 5 5 5 5 5 5 5 5 5 5 4 D (ICE DDR) 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9

Cool Aisle 11d

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 7 7 7 7 7 7 7 6 6 6 6 6 6 6 6 6 K (ICE QDR) 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1

Hot Aisle 8d 11d

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 9 9 9 8 8 8 8 8 8 8 8 8 8 7 7 7 L (ICE QDR) 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 182 racks – 2012 Cool Aisle 3 3 3 3 3 3 3 3 3 3 3 3 2 * Install – 3/30/2012 Note: 1 1 1 0 0 0 0 0 0 0 0 0 9 O (ICE FDR) 1.7 petaflops RK 299 and RK 300 are 2 1 0 9 8 7 6 5 4 3 2 1 9 RLC racks. Racks 301-312 and Racks 317-328 are 9d Hot Aisle 8d Intel E5 Processors 3 3 3 3 3 3 3 3 3 3 3 3 3 2 2 2 2 2 2 2 2 2 1 1 1 0 P (ICE FDR) 8 7 6 5 4 3 2 1 0 9 8 7 0 12d

Cool Aisle

1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 4 4 4 4 4 3 3 3 3 3 3 3 3 3 3 2 12d I (ICE QDR) 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 15M cable Hot Aisle 8d

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 6 5 5 5 5 5 5 5 5 5 5 4 4 4 4 4 10d J (ICE QDR) 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5

Cool Aisle 10M cable

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 A (ICE DDR) 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1

Hot Aisle 8d

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 3 3 2 2 2 2 2 2 2 2 2 2 1 1 1 B (ICE DDR) 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7

9d Cool Aisle

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4 4 4 4 4 4 4 4 4 3 3 3 3 3 3 3 C (ICE DDR) 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3

Hot Aisle 8d

10d 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 6 6 6 6 6 5 5 5 5 5 5 5 5 5 5 4 D (ICE DDR) 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9

Cool Aisle 11d

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 7 7 7 7 7 7 7 6 6 6 6 6 6 6 6 6 K (ICE QDR) 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1

Hot Aisle 8d 11d

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 9 9 9 8 8 8 8 8 8 8 8 8 8 7 7 7 L (ICE QDR) 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 64 rack deinstall Cool Aisle 3 3 3 3 3 3 3 3 3 3 3 3 2 * Install – 3/30/2012 Note: 1 1 1 0 0 0 0 0 0 0 0 0 9 O (ICE FDR) 2013 RK 299 and RK 300 are 2 1 0 9 8 7 6 5 4 3 2 1 9 RLC racks. Racks 301-312 and Racks 317-328 are 9d Hot Aisle 8d Intel E5 Processors 3 3 3 3 3 3 3 3 3 3 3 3 3 2 2 2 2 2 2 2 2 2 1 1 1 0 P (ICE FDR) 8 7 6 5 4 3 2 1 0 9 8 7 0 12d

Cool Aisle

1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 4 4 4 4 4 3 3 3 3 3 3 3 3 3 3 2 12d I (ICE QDR) 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 15M cable Hot Aisle 8d

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 6 5 5 5 5 5 5 5 5 5 5 4 4 4 4 4 10d J (ICE QDR) 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5

Cool Aisle 10M cable

2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 9 9 9 9 9 9 9 M (ICE QDR) 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 10D Hot Aisle 8D Added 4 racks Added 8 racks 2 2 2 02-28-20112 2 2 2 2 202-17-20112 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 0 N (ICE QDR) Gpgpu racks 219 and 220 2 1 0 9 8 7 6 5 4 3 2 1 0 9 but configured as rack 219. note switches on Note: Rack 221 will cable to on 11D to rack 92. There gpgpu are in rear of rack is no 11d for Rack 222. this is a problem. If we so cable lengths needs to remove rack 92 then we have issue with racks 221 & be adjusted to reflect this. 222. Note: 06/21/2013 -Rack 001- 004 are I/O racks for RLC and NASA (Pleiades) Rack Layout switches.. RowS A,B,C,D – 46 racks are proposed IVYB. They will connect via 10D to 4 4 4 4 4 4 4 4 0 A (ICE FDR) Row O and P. This will be 0 0 0 0 0 0 0 0 0 SGI Ice X – the 48 partial 9D and partial 10D. 8 7 6 5 4 3 2 1 1 port cable version

Note: Hot Aisle 8d 1st delivery : 8 racks in Row C 8 racks Row D 4 4 4 4 4 4 4 4 0 B (ICE FDR) SGI ICE 2nd delivery: 8 racks in Row B 2 2 2 2 2 1 1 1 0 X – the 48 port 8 racks in Row A 4 3 2 1 0 9 8 7 2 version 3rd delivery: add 8 racks to Cool Aisle row D 9D add 6 racks to Row C (ICE FDR) C 4 4 4 4 4 4 4 4 4 4 4 4 4 4 0 SGI ICE X – the 4 4 4 4 4 4 4 3 3 3 3 3 3 3 0 Rack 001-004 are the admin 6 5 4 3 2 1 0 9 8 7 6 5 4 3 3 48 port version racks that house the RLC and ethernet switches. There is Hot Aisle 8d one being added for Row A. D (ICE FDR) 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 0 SGI ICE X – the 6 6 6 6 6 5 5 5 5 5 5 5 5 5 5 4 0 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 4 48 port version

Cool Aisle K (ICE QDR) 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 RK 161-170 are Altix 8200 with QDR IB 7 7 7 7 7 7 7 6 6 6 6 6 6 6 6 6 SW/but DDR compute blade. RKS 172- 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 * Install – Rks 313-316 176 are Altix 8400EX with QDR are the pyramid with MIC Hot Aisle 8d 10D racks and are configured L (ICE QDR) RK 177-1186 are Altix 8200 with QDR IB as two racks of SGI ICE 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 SW/but DDR compute blade. RKS 187— X except there are only 9 9 9 8 8 8 8 8 8 8 8 8 8 7 7 7 192 are Altix 8400EX with QDR SW and 64 nodes per Rack. They 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 are virtually racks 313 blades 167 racks – 2013 and 314. They will not be Cool Aisle O (ICE FDR) delivered till Nov 2012 to SGI ICE X – the 48 port version – rk 301-312 NASA 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 are FDR/rk 313-316 uses 2U servers and they 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 use Mellanox 6036 unmanaged switches. 2.9 petaflops 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 There are 32 nodes/rack but arrange in hypercube like two virtual ICE X racks 9d Hot Aisle 8d P (ICE FDR) 3 3 3 3 3 3 3 3 3 3 3 3 3 3 2 SGI ICE X – the 48 port version – Rks I I I I This is the switch rack 3 2 2 2 2 2 2 2 2 2 2 1 1 1 9 317-330 are all FDR including the O O O O 0 9 8 7 6 5 4 3 2 1 0 9 8 7 9 12d compute blade.

Cool Aisle

1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 I (ICE QDR) rk 129-144 are Altix 4 4 4 4 4 3 3 3 3 3 3 3 3 3 3 2 12d ICE 8400 EX – all QDR 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 15M cable Hot Aisle 8d J (ICE QDR) 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 rk 145-260 are Altix ICE 8400 EX 6 5 5 5 5 5 5 5 5 5 5 4 4 4 4 4 10d – all QDR 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5

Cool Aisle 10M cable M (ICE QDR) 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 rk 193-208 are Altix ICE 8400 0 0 0 0 0 0 0 0 0 9 9 9 9 9 9 9 EX – all QDR 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 10D Hot Aisle 8D N (ICE QDR) Added 4 racks Added 8 racks rk 209-217 are Altix ICE 8400 EX – all QDR / 2 2 2 022-28-220121 2 2 202-217-220112 2 2 rk 219-220 is 2 racks with 32 servers each 2 2 2 1 1 1 1 1 1 1 1 1 1 0 configured in hypercube using 5025 sw QDR./ Gpgpu racks 219 and 220 2 1 0 9 8 7 6 5 4 3 2 1 0 9 rack 22-222 – Altix Ice 8400 EX all QDR SW but configured as rack and Blades. 219. note switches on Note: Rack 221 will cable to on 11D to rack 92. There gpgpu are in rear of rack is no 11d for Rack 222. this is a problem. If we so cable lengths needs to remove rack 92 then we have issue with racks 221 & be adjusted to reflect this. 222. NASA (Pleiades) Rack Layout as of 12/30/2013

4 4 4 4 4 4 4 4 4 4 4 4 0 A (ICE FDR) 1 1 1 0 0 0 0 0 0 0 0 0 0 RK 401-416 -SGI ICE X 2 1 0 9 8 7 6 5 4 3 2 1 1 (ivybridge) Prem SW

Hot Aisle 8d

4 4 4 4 4 4 4 4 0 B (ICE FDR) 2 2 2 2 2 1 1 1 0 RK417-432 SGI ICE X 4 3 2 1 0 9 8 7 2 (ivybridge) Prem SW 10D Cool Aisle 9D 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 0 C (ICE FDR) 4 4 4 4 4 4 4 4 4 3 3 3 3 3 3 3 0 RK 433-448 SGI ICE X 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 3 (ivybridge) prem Sw

Hot Aisle 8d

4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 0 D (ICE FDR) Rk 449- 6 6 6 6 6 5 5 5 5 5 5 5 5 5 5 4 0 464 SGI ICE X 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 4 (ivybridge) Prem SW

Cool Aisle K (ICE QDR) rks 161- 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 170 Altix ICE 8200 7 7 7 7 7 7 7 6 6 6 6 6 6 6 6 6 Nehamel + ICE QDR) 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 171-176 Altix ICE 8400 EX (Westmere) Hot Aisle 10D 8d L (ICE QDR) rks 177- 186 Altix ICE 8200 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Nehalem +(ICE QDR) 9 9 9 8 8 8 8 8 8 8 8 8 8 7 7 7 187-17192 Altix ICE 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 8400 EX (Westmere) Cool Aisle 160 racks – 2013 O (ICE FDR)/ RK 300- 312 SGI ICE X 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 Sandybridge Prem SW/ 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 RK 313-316 128 node 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 3.1 petaflops Pyramid in hypercube topology 9D Hot Aisle 8d

3 3 3 3 3 3 3 3 3 3 3 3 3 3 2 P (ICE FDR)/ RK 317- I I I I This is the switch rack 3 2 2 2 2 2 2 2 2 2 2 1 1 1 9 330 SGI ICE X SNB O O O O 0 9 8 7 6 5 4 3 2 1 0 9 8 7 9 12d Prem SW

Cool Aisle

1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 I (ICE QDR) RKS 129- 4 4 4 4 4 3 3 3 3 3 3 3 3 3 3 2 12d 144 Altix Ice 8400 EX 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9

Hot Aisle 8d

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 J (ICE QDR) RKS 145- 6 5 5 5 5 5 5 5 5 5 5 4 4 4 4 4 10d 160 ALTIX ICE 8400 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 EX

Cool Aisle 10M cable

2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 M (ICE QDR) RKS 193- 0 0 0 0 0 0 0 0 0 9 9 9 9 9 9 9 208 ALTIX ICE 8400 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 EX 10D Hot Aisle 8D N (ICE QDR) RKS 209- Added 4 racks Added 8 racks 218 ALTIX ICE 8400 EX/ 2 2 2 022-28-220121 2 2 202-217-220112 2 2 RK 219-220 Coyote based 2 2 2 1 1 1 1 1 1 1 1 1 1 0 Westmere with GPGPU 2 1 0 9 8 7 6 5 4 3 2 1 0 9 M2090 in hypercube. RK 221-222 Altix ICE 8400 EX NASA (Pleiades) Rack Layout as of 1/30/2014

4 4 4 4 4 4 4 4 4 4 4 4 0 A (ICE FDR) 1 1 1 0 0 0 0 0 0 0 0 0 0 RK 401-412 -SGI ICE X 2 1 0 9 8 7 6 5 4 3 2 1 1 (ivybridge) Prem SW

Hot Aisle 8d

4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 0 B (ICE FDR) 3 3 3 2 2 2 2 2 2 2 2 2 2 1 1 1 0 RK417-432 SGI ICE X 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 2 (ivybridge) Prem SW 10D Cool Aisle 9D 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 0 C (ICE FDR) 4 4 4 4 4 4 4 4 4 3 3 3 3 3 3 3 0 RK 433-448 SGI ICE X 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 3 (ivybridge) prem Sw

Hot Aisle 8d

4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 0 D (ICE FDR) Rk 449- 6 6 6 6 6 5 5 5 5 5 5 5 5 5 5 4 0 464 SGI ICE X 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 4 (ivybridge) Prem SW

Cool Aisle E (ICE FDR) rks 465- 1 1 1 1 1 1 468 SGI ICE X 7 7 7 7 7 7 Ivybridge Prem SW – K 6 5 4 3 2 1 ICE QDR) 171-176 Altix ICE 8400 EX Hot Aisle 10D 8d F (ICE FDR) rks 481- 9D 483 SGI ICE X 1 1 1 1 1 1 Ivybrodge Prem SW L- 9 9 9 8 8 8 ICE QDR) 187-17192 2 1 0 9 8 7 Altix ICE 8400 EX Cool Aisle 168 racks – 2013 O (ICE FDR)/ RK 300- 312 SGI ICE X 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 Sandybridge Prem SW/ 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 RK 313-316 128 node 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 3.2 petaflops Pyramid in hypercube topology 9D Hot Aisle 8d

3 3 3 3 3 3 3 3 3 3 3 3 3 3 2 P (ICE FDR)/ RK 317- I I I I This is the switch rack 3 2 2 2 2 2 2 2 2 2 2 1 1 1 9 330 SGI ICE X SNB O O O O 0 9 8 7 6 5 4 3 2 1 0 9 8 7 9 12d Prem SW

Cool Aisle

1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 I (ICE QDR) RKS 129- 4 4 4 4 4 3 3 3 3 3 3 3 3 3 3 2 12d 144 Altix Ice 8400 EX 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9

Hot Aisle 8d

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 J (ICE QDR) RKS 145- 6 5 5 5 5 5 5 5 5 5 5 4 4 4 4 4 10d 160 ALTIX ICE 8400 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 EX

Cool Aisle 10M cable

4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 0 A (ICE FDR) 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 RK 401-416 -SGI ICE X 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 1 (ivybridge) Prem SW

Hot Aisle 8d

4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 0 D (ICE FDR) Rk 449- 6 6 6 6 6 5 5 5 5 5 5 5 5 5 5 4 0 464 SGI ICE X 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 4 (ivybridge) Prem SW

Cool Aisle E (ICE FDR) rks 465- 1 1 1 1 1 1 4 4 4 4 0 468 SGI ICE X 7 7 7 7 7 7 6 6 6 6 0 Ivybridge Prem SW – K 6 5 4 3 2 1 8 7 6 5 5 ICE QDR) 171-176 Altix ICE 8400 EX Hot Aisle 10D 8d F (ICE FDR) rks 481- 9D 483 SGI ICE X 1 1 1 1 1 1 Ivybrodge Prem SW L- 9 9 9 8 8 8 ICE QDR) 187-17192 2 1 0 9 8 7 Altix ICE 8400 EX Cool Aisle 168 racks – 2014 O (ICE FDR)/ RK 300- 312 SGI ICE X 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 Sandybridge Prem SW/ 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 RK 313-316 128 node 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 3.3 petaflops Pyramid in hypercube topology 9D Hot Aisle 8d

3 3 3 3 3 3 3 3 3 3 3 3 3 3 2 P (ICE FDR)/ RK 317- I I I I This is the switch rack 3 2 2 2 2 2 2 2 2 2 2 1 1 1 9 330 SGI ICE X SNB O O O O 0 9 8 7 6 5 4 3 2 1 0 9 8 7 9 12d Prem SW

Cool Aisle

1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 I (ICE QDR) RKS 129- 4 4 4 4 4 3 3 3 3 3 3 3 3 3 3 2 12d 144 Altix Ice 8400 EX 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9

Hot Aisle 8d

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 J (ICE QDR) RKS 145- 6 5 5 5 5 5 5 5 5 5 5 4 4 4 4 4 10d 160 ALTIX ICE 8400 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 EX

Cool Aisle 10M cable

4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 0 A (ICE FDR) 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 RK 401-416 -SGI ICE X 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 1 (ivybridge) Prem SW

Hot Aisle 8d

4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 0 D (ICE FDR) Rk 449- 6 6 6 6 6 5 5 5 5 5 5 5 5 5 5 4 0 464 SGI ICE X 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 4 (ivybridge) Prem SW

Cool Aisle

4 4 4 4 4 4 4 4 0 E (ICE FDR) rks 465- 7 7 7 6 6 6 6 6 0 468 SGI ICE X 2 1 0 9 8 7 6 5 5 Ivybridge Prem SW

Hot Aisle 8d 10D 9D 4 4 4 0 F (ICE FDR) rks 481- 8 8 8 0 483 SGI ICE X 3 2 1 6 Ivybrodge Prem SW

Cool Aisle 170 racks – 2014 O (ICE FDR)/ RK 300- 312 SGI ICE X 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 Sandybridge Prem SW/ 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 RK 313-316 128 node 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 3.5 petaflops Pyramid in hypercube topology Hot Aisle 8d

3 3 3 3 3 3 3 3 3 3 3 3 3 3 2 P (ICE FDR)/ RK 317- I I I I This is the switch rack 3 2 2 2 2 2 2 2 2 2 2 1 1 1 9 330 SGI ICE X SNB O O O O 0 9 8 7 6 5 4 3 2 1 0 9 8 7 9 12d Prem SW

Cool Aisle

1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 I (ICE QDR) RKS 129- 4 4 4 4 4 3 3 3 3 3 3 3 3 3 3 2 12d 144 Altix Ice 8400 EX 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 15M cable Hot Aisle 8d

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 J (ICE QDR) RKS 145- 6 5 5 5 5 5 5 5 5 5 5 4 4 4 4 4 10d 160 ALTIX ICE 8400 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 EX

Cool Aisle 10M cable

4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 0 A (ICE FDR) 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 RK 401-416 -SGI ICE X 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 1 (ivybridge) Prem SW

Hot Aisle 8d

4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 0 D (ICE FDR) Rk 449- 6 6 6 6 6 5 5 5 5 5 5 5 5 5 5 4 0 464 SGI ICE X 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 4 (ivybridge) Prem SW

Cool Aisle

4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 0 E (ICE FDR) rks 465- 8 7 7 7 7 7 7 7 7 7 7 6 6 6 6 6 0 468 SGI ICE X 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 5 Ivybridge Prem SW

Hot Aisle 8d 10D 9D 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 0 F (ICE FDR) rks 481- 9 9 9 9 9 9 9 8 8 8 8 8 8 8 8 8 0 488 SGI ICE X 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 6 Ivybrodge Prem SW

Cool Aisle 168 racks – 2014 O (ICE FDR)/ RK 300- 312 SGI ICE X 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 Sandybridge Prem SW/ 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 3 RK 313-316 128 node 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 4.5 petaflops 3 Pyramid in hypercube topology Hot Aisle 8d

3 3 3 3 3 3 3 3 3 3 3 3 3 3 2 P (ICE FDR)/ RK 317- I I I I This is the switch rack 3 2 2 2 2 2 2 2 2 2 2 1 1 1 9 330 SGI ICE X SNB O O O O 0 9 8 7 6 5 4 3 2 1 0 9 8 7 9 12d Prem SW

Cool Aisle

1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 I (ICE QDR) RKS 129- 4 4 4 4 4 3 3 3 3 3 3 3 3 3 3 2 12d 144 Altix Ice 8400 EX 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 15M cable Hot Aisle 8d

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 J (ICE QDR) RKS 145- 6 5 5 5 5 5 5 5 5 5 5 4 4 4 4 4 10d 160 ALTIX ICE 8400 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 EX

Cool Aisle 10M cable

2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 M (ICE QDR) RKS 193- 0 0 0 0 0 0 0 0 0 9 9 9 9 9 9 9 208 ALTIX ICE 8400 Westmere Racks under warranty 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 EX maintenance 10D Racks 193-204 – maint till 3/14/2014 Hot Aisle 8D N (ICE QDR) RKS 209- Racks 205-212 – maint till 5/23/2014 Added 4 racks Added 8 racks 218 ALTIX ICE 8400 EX/ Racks 213-218 – maint till 5/26/2014 2 2 2 022-28-220121 2 2 202-217-220112 2 2 RK 219-220 Coyote based Racks 221-222 – maint till 9/19/2014 2 2 2 1 1 1 1 1 1 1 1 1 1 0 Westmere with GPGPU Racks 219-220 – maint till 6/28/2014 2 1 0 9 8 7 6 5 4 3 2 1 0 9 M2090 in hypercube. RK 221-222 Altix ICE 8400 EX NASA (Pleiades) Rack Layout

4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 0 A (ICE FDR) 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 RK 401-416 -SGI ICE X 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 1 (ivybridge) Prem SW

Hot Aisle 8d

4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 0 D (ICE FDR) Rk 449- 6 6 6 6 6 5 5 5 5 5 5 5 5 5 5 4 0 464 SGI ICE X 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 4 (ivybridge) Prem SW

Cool Aisle E (ICE FDR) rks 465- 5 5 5 5 5 5 5 5 4 4 4 4 4 4 4 4 0 472 SGI ICE X 8 7 7 7 7 7 7 7 7 7 7 6 6 6 6 6 0 Ivybridge Prem SW\RK 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 5 573-580 Haswell Prem SW Hot Aisle 8d 10D F (ICE FDR) rks 481- 9D 5 5 5 5 5 5 5 5 5 5 5 5 5 4 4 4 0 483 SGI ICE X 9 9 9 9 9 9 9 8 8 8 8 8 8 8 8 8 0 Ivybrodge Prem SW 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 6 Rks 584-590 Haswell Prem SW Cool Aisle 168 racks – 2015 O (ICE FDR)/ RK 300- 312 SGI ICE X 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 Sandybridge Prem SW/ 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 RK 313-316 128 node 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 5.4 petaflops Pyramid in hypercube 11D topology Hot Aisle 8d

3 3 3 3 3 3 3 3 3 3 3 3 3 3 2 P (ICE FDR)/ RK 317- I I I I This is the switch rack 3 2 2 2 2 2 2 2 2 2 2 1 1 1 9 330 SGI ICE X SNB O O O O 0 9 8 7 6 5 4 3 2 1 0 9 8 7 9 Prem SW

Cool Aisle

11/02/2014 – reomve 16 5 5 5 5 5 5 5 5 0 racks of WSM – rks 129- I (ICE FDR) RKS 509- 1 1 1 1 1 1 1 0 0 12d 144 and replace with 516 SGI ICE X - HSW 6 5 4 3 2 1 0 9 7 racks 509-516 haswell 15M cable Hot Aisle 10d 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 J (ICE QDR) RKS 145- 6 5 5 5 5 5 5 5 5 5 5 4 4 4 4 4 10d 160 ALTIX ICE 8400 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 EX

Cool Aisle 10M cable

10/20/2014 – note Row M M (ICE QDR) RKS 193- racks.193-208 turned off 208 ALTIX ICE 8400 EX 10D Hot Aisle 8D N (ICE QDR) RKS 209- Added 4 racks Added 8 racks 218 ALTIX ICE 8400 EX/ 2 2 2 022-28-220121 2 2 202-217-220112 2 2 RK 219-220 Coyote based 2 2 2 1 1 1 1 1 1 1 1 1 1 0 Westmere with GPGPU 2 1 0 9 8 7 6 5 4 3 2 1 0 9 M2090 in hypercube. RK 221-222 Altix ICE 8400 EX NASA (Pleiades) Rack Layout

4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 0 A (ICE FDR) 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 RK 401-416 -SGI ICE X 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 1 (ivybridge) Prem SW

Hot Aisle 8d

4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 0 D (ICE FDR) Rk 449- 6 6 6 6 6 5 5 5 5 5 5 5 5 5 5 4 0 464 SGI ICE X 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 4 (ivybridge) Prem SW

Cool Aisle E (ICE FDR) rks 465- 5 5 5 5 5 5 5 5 4 4 4 4 4 4 4 4 0 472 SGI ICE X 8 7 7 7 7 7 7 7 7 7 7 6 6 6 6 6 0 Ivybridge Prem SW\RK 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 5 573-580 Haswell Prem SW Hot Aisle 8d 10D F (ICE FDR) rks 481- 9D 5 5 5 5 5 5 5 5 5 5 5 5 5 4 4 4 0 483 SGI ICE X 9 9 9 9 9 9 9 8 8 8 8 8 8 8 8 8 0 Ivybrodge Prem SW 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 6 Rks 584-590 Haswell Prem SW Cool Aisle 162 racks – 2016 O (ICE FDR)/ RK 300- 312 SGI ICE X 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 Sandybridge Prem SW/ 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 RK 313-316 128 node 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 7.1 petaflops Pyramid in hypercube 11D topology Hot Aisle 8d

3 3 3 3 3 3 3 3 3 3 3 3 3 3 2 P (ICE FDR)/ RK 317- I I I I This is the switch rack 3 2 2 2 2 2 2 2 2 2 2 1 1 1 9 330 SGI ICE X SNB O O O O 0 9 8 7 6 5 4 3 2 1 0 9 8 7 9 Prem SW #13 Top500 Nov Cool Aisle 11/02/2014 – reomve 16 5 5 5 5 5 5 5 5 6 6 6 6 6 6 6 6 0 I (ICE FDR) RKS 509- racks of WSM – rks 129- 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 12d 516 SGI ICE X – HSW 144 and replace with #9 HPCG ISC 16 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 7 Racks 603-608 -BrdWL racks 509-516 haswell 15M cable Hot Aisle J (ICE QDR) RKS 145- 10d 160 ALTIX ICE 8400 1 1 1 1 1 1 1 1 6 6 6 6 6 6 0 EX Turn off Rks 145- 6 5 5 5 5 5 5 5 X X 2 2 2 1 1 1 0 10d 148 for RKS 603-608 0 9 8 7 6 5 4 3 2 1 0 9 8 7 8 Turn off RKS 149-152 for 601-602/617-622 >20 major upgrades Cool Aisle 10M cable

10/20/2014 – note Row M M (ICE QDR) RKS 193- racks.193-208 turned off 208 ALTIX ICE 8400 EX 10D N (ICE QDR) RKS 209- Hot Aisle 8D 218 ALTIX ICE 8400 EX/ Added 4 racks Added 8 racks RK 219-220 Coyote based 2 2 2 022-28-220121 02-17-2011 Westmere with GPGPU 2 2 2 1 1 1 X X X X X X X X M2090 in hypercube. RK 2 1 0 9 8 7 221-222 Altix ICE 8400 EX Turn off 209-212 for rkS 603-608 Turn off 213-216 for RKS 601-602/617-620 Highlights of SGI Optimized HPCG Code

• Lexicographical ordering for maximum data locality • Left and right data structures for full matrix representation • A variant of CSR storage format • Pure MPI • No overlapping of computation and communication • Additional tuning for contiguous memory, setup time and combined computations Heterogeneous considerations

Load balancing via number of ranks per node Broadwell E5-2680 v4 14-core 2.4 GHz • 2015 nodes, 12 ranks/socket, 0.85 GF/rank Haswell E5-2680 v3 12-core 2.5 GHz • 2080 nodes, 10 ranks/socket, 0.91 GF/rank Ivy Bridge E5-2680 v2 10-core 2.8 GHz • 5351 nodes, 9 ranks/socket, 0.86 GF/rank Sandy Bridge E5-2670 8-core 2.6 GHz • 1853 nodes, 7 ranks/socket, 0.92 GF/rank Pleiades results

NASA Pleiades HPCG Performance

200

180

160

140 Heterogeneous

120 BDW

100 HSW

80 IVB Performance (TF) 60 SNB

0 Aggregate performance is over 94% of the sum of the individual component results Credits to the Team

John Baron SGI Davin Chan CSRA Cheng Laio SGI Dale Talcott CSRA Michael Raymond SGI Jim Karella CSRA Jay Lan SGI Greg Matthews CSRA Scott Emery SGI Herbert Yeung CSRA Jennifer Fung SGI Mahmoud Hanafi CSRA Jose Rodriguez SGI Mike Hartman CSRA Matt Lepp SGI Jeff Becker CSRA Jason Inoue SGI Nathan Dauchy CSRA Rich Davila SGI Bill Thigpen NASA John Dugan SGI Mark Tangney NASA Bob Ciotti NASA