National Aeronautics and Space Administration

Performance Evaluation of Amazon EC2 for NASA HPC Applications!

Piyush Mehrotra! J. Djomehri, S. Heistand, R. Hood, H. Jin, A. Lazanoff, S. Saini, R. Biswas NASA Advanced Supercomputing (NAS) Division NASA Ames Research Center piyush.mehrotra@nasa.gov!

www.nasa.gov Cloud Computing!

• Cloud computing is becoming ubiquitous - Allows organizations to outsource hardware acquisition and maintenance - Satisfies burst computational demands at the cost of performance - Provides support for customized execution environments - Increasing catalog of useful software services • Extensively being used for enterprise/business tasks • Increasing use for scientific and engineering (S&E) tasks, e.g., bioinformatics, oil exploration, data analytics, etc. - Embarrassingly parallel or loosely coupled parallel tasks • To attract the High Performance Computing (HPC) market Amazon EC2 provides HPC instance types

Are Clouds ready for NASA’s HPC applications?

NASA High End Computing Capability 2 Outline!

• Clouds • Systems tested • Benchmarks and Applications used • Performance achieved • Conclusions

NASA High End Computing Capability 3 Systems Tested! • Pleiades: NASAʼs primary supercomputer - SGI Altix ICE – 4 generations of Intel Xeon processors: Harpertown (4c), Nehalem (4c), Westmere (6c), Sandy Bridge (8c) – total 11,776 nodes, 126,720 cores, peak 1.75 Pflops • Amazon EC2: premier public cloud computing environment Amazon HPC Pleiades cc1.4xlarge Intel Xeon X5570 Intel Xeon X5570 CPU Nehalem Nehalem Clock Speed 2.93 GHz 2.93 GHz No. of cores/node 8 8 RAM 24 GB 23 GB Hyper-threading Enabled Enabled 4 Network InfiniBand QDR 10 GigE Interconnect Topology Hypercube Unknown OS Type SLES11SP1 SLES11SP1 Virtualization No Yes Multi-tenancy No No Compiler Intel 12.0.4 Intel 12.0.4 MPI Library MPT 2.04 Open MPI 1.4.4 File Structure Lustre NFS NASA High End Computing Capability 4 Benchmarks/Applications!

• HPCC Benchmarks - DGEMM – floating point rate measurement - Bandwidth and latency tests • NPBs: • Applications: - Cart3D: high-fidelity inviscid CFD - Kernel benchmarks code; multiple cartesian grids; cpu- • MG: multi-grid on a sequence of meshes, long- and short-distance intensive and memory bound communication, memory intensive - DDSCAT: embarrassingly parallel • FT: discrete 3D FFTs, all-to-all code to calculate the absorption and communication scattering properties oftargets • IS: integer sort, random memory - ENZO: Simulation of cosmological structure formulation using adaptive access • CG: conjugate gradient, irregular mesh refinement memory access and communication - MITGcm: Global ocean circulation • EP: embarrassingly parallel with model; memory bound; latency bound large number of cores some communication - Compact Application benchmarks • BT: block tri-diagonal solver • SP: scalar penta-diagonal solver • LU: lower-upper Gauss Seidel

NASA High End Computing Capability 5

Performance Results!

6 NASA High End Computing Capability HPCC results (64 cores)!

Pleiades Amazon HPC EP DGEMM N=5773 (Gflops)(Gflops) 11.417 10.758 Single DGEMM N=5773 (Gflops) 12.252 11.062 Max Ping Pong Latency (usec)(usec) 2.574 73.999 Min Ping Pong Latency (usec) 0.238 0.626 Avg Ping Pong Latency (usec) 1.954 61.535 Naturally Ordered Ring Latency (usec)(usec) 1.597 58.293 Randomly Ordered Ring Latency (usec)(usec) 4.466 180.786 Max Ping Pong Bandwidth (Gbytes) 5.044 6.982 Min Ping Pong Bandwidth (Gbytes) 1.538 0.406 Avg Ping Pong Bandwidth ((Gbytes)Gbytes) 2.171 0.952 Naturally Ordered Ring Bandwidth (Gbytes) 1.768 0.299 Randomly Ordered Ring Bandwidth (Gbytes)(Gbytes) 0.283 0.070

NASA High End Computing Capability 7 NPB Results!

NASA High End Computing Capability 8 NPB Results (MG)!

EC2-HPC relative to Pleiades

NASA High End Computing Capability 9 NPB Results (contd.)!

NASA High End Computing Capability 10 Application Performance!

NASA High End Computing Capability 11 CART3D Performance!

EC2-HPC relative to Pleiades

NASA High End Computing Capability 12 12 DDSCAT Performance!

EC2-HPC relative to Pleiades

NASA High End Computing Capability 13 13 EC2-HPC Performance Relative to Pleiades!

NASA High End Computing Capability 14 Communication Impact!

NASA High End Computing Capability 15 Communication Impact (2)!

NASA High End Computing Capability 16 Virtualization Overhead!

Pleiades Amazon HPC Perf. Diff.

EP DGEMM N=5773 (Gflops) 11.417 10.758 5.8%

Single DGEMM N=5773 (Gflops) 12.252 11.062 9.7%

NASA High End Computing Capability 17 Performance Variability!

NASA High End Computing Capability 18 Summary! • Amazon EC2-HPC performance lags Pleiades supercomputer for most NASA HPC applications - Network technology - Virtualization overhead • Other issues with clouds for NASA scientists - Cost - Security - Usability - Reliability Cloud computing currently not a viable solution for NASA HPC applications … but they are catching up fast!!! NASA High End Computing Capability 19 Characteristics Cloud Environments NASA HPC Center For users with a credit For approved users with On demand self-service card allocation Worldwide Broad network access Worldwide (subject to NASA security constraints) Resource Pooling Yes Across the Agency Users can provision Rapid Elasticity Extreme multiple thousands of cores Users credit cards Users allocations Measured Service charged according to charged according to usage usage Service Model Mainly IaaS PaaS Deployment Model Public Private

NASA’s HECC facility is a private, PaaS, HPC cloud NASA High End Computing Capability 20 21 NASA High End Computing Capability