High Performance Computing Storage Options Selecting AWS Partner Network storage solutions for High Performance Computing on Amazon Web Services December 2018 © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Notices This document is provided for informational purposes only. It represents AWS’s current product offerings and practices as of the date of issue of this document, which are subject to change without notice. Customers are responsible for making their own independent assessment of the information in this document and any use of AWS’s products or services, each of which is provided “as is” without warranty of any kind, whether express or implied. This document does not create any warranties, representations, contractual commitments, conditions or assurances from AWS, its affiliates, suppliers or licensors. The responsibilities and liabilities of AWS to its customers are controlled by AWS agreements, and this document is not part of, nor does it modify, any agreement between AWS and its customers. Contents Introduction 1 HPC Storage 1 AWS Storage Services 3 Amazon S3 3 Amazon EBS 4 Amazon EC2 Instance Store 4 Amazon EFS 5 AWS Partner Solutions 5 WekaIO Matrix 5 Qumulo QF2 7 Spectrum Scale 8 Lustre 11 BeeGFS 12 Conclusion 13 Contributors 14 Further Reading 14 Document Revisions 14 Abstract High Performance Computing (HPC) is an important and complicated workload for many customers. Many HPC workloads incorporate large compute clusters and need to process a lot of data. HPC workloads require storage systems that can keep up with performance and capacity demands. This paper explores storage options when running HPC workloads on Amazon Web Services (AWS). Amazon Web Services – AWS Partner Network High Performance Computing Storage Options Introduction HPC can take many forms across many industries. Technical computing is present in everything from electronics design and manufacturing to genomics and film rendering. HPC often means massively parallel compute clusters used to analyze existing data or generate new data. In either case, data needs to be acted upon and persisted by each node in a cluster as fast as possible. Many HPC workloads run for a long time and take checkpoints. These checkpoints create large amounts of intermediate data, but enable snapshotting and restarting in case of interruption. This is particularly important on the cloud, where specialized offerings like Amazon Elastic Compute Cloud (EC2) Spot Instances may incorporate temporary resource shortages as a tradeoff to being extremely cost-effective. There are many benefits of running HPC on AWS, and for more information about the benefits see the HPC section of the AWS website1. In this document, we explore HPC from a storage perspective. Cluster management and other compute aspects are out of scope for this paper, but addressed in the introduction to HPC on AWS whitepaper2 for those interested in learning about schedulers and solvers on AWS. HPC Storage Programs that run in HPC clusters usually require shared access to storage. Any program on any compute node can read from and write to any file or parts of any file. Because the read or write operation could come from anywhere, storage architectures for HPC workloads aim to avoid potential performance bottlenecks. Distributed locking systems and parallel access to metadata and data help keep things working at high speed. Add to that low latency networking and high-performance media, and it’s a formula for success. AWS componentry is an easy way to get started with HPC storage on the cloud, and can be used in conjunction with integrated solutions from AWS Partner Network (APN) Technology Partners. For background, let’s briefly examine the anatomy of a distributed parallel cluster. A file system understands how to map a file hierarchy and its contents to physical media, such as a drive. Only a single host can request a file. Even if the host is a Network File System (NFS) server, the NFS server software arbitrates remote requests from clients to become local requests. The media could be a Page 1 Amazon Web Services – AWS Partner Network High Performance Computing Storage Options physical device or an aggregate of devices presented as a logical device, like in a Storage Area Network (SAN). A distributed file system spans more than one set of logical media, but requests are still terminating in one point. NFS servers are a good example of this, because an NFS client only knows how to access the NFS server without understanding the underlying physical topology or having any way to access it. A clustered file system allows multiple hosts to access the same data at the same time, fully supporting concurrent reads and writes and arbitrating any contention. In this model, every participating host can make requests, but hosts find responses on the same shared media. In clustered environments, participating hosts often export the shared namespace as a networked resource. Requests coming from remote clients always terminate on the limited number of participating hosts, like in the simple case of a single host acting as an NFS server. A clustered distributed file system builds on these concepts by breaking data into pieces and spreading file system data across multiple hosts and multiple media. This does a better job of equalizing capabilities, because each participating host doesn’t have to compete with other hosts for access to storage media. Requests are routed through specific hosts depending on which piece of data is needed. A clustered distributed system still might have a request- handling bottleneck if the data being requested doesn’t vary. This is referred to as a hot spot. In a scenario where participating hosts re-export their file system to the network, remote clients asking for the same piece of data would cause all requests to be routed through the same host irrespective of which of host terminates the client connection. To solve for this, parallel file systems support a many-to-many relationship between requests and responses. Every participating host can initiate or service requests, accessing data through many independent pathways. They work together to provide a single file system hierarchy, or namespace. This grid or mesh includes a special client software to decipher the file system namespace and inform even remote clients as to the topology of the data layout. By avoiding any form of request serialization, parallel file systems historically have provided the best performance for HPC workloads. Before the cloud, it was common to see parallel file systems talk to clients via a front-end network, and utilize a segregated back-end network for behind-the-scenes data access. Reliability was generally handled by underlying storage systems, often complex and expensive SAN topologies. Page 2 Amazon Web Services – AWS Partner Network High Performance Computing Storage Options The advent of the cloud and storage solutions designed for the cloud greatly simplifies the deployment and configuration process for HPC workloads. In addition to scaling performance and capacity, distributed parallel file system clusters in the cloud also take fault tolerance into consideration. Participating hosts are fully aware of device health and have mechanisms in place to minimize interruptions in the event of component failure, thanks to the client’s understanding of what is happening behind the scenes. AWS Storage Services AWS provides several storage services useful for HPC. Some can be used on their own, while others are building blocks for APN Partner solutions. What you choose depends largely on your workload attributes. The best way to validate your storage choice is to try your workload. Benchmarks and other synthetic measurements are rarely adequate in the tailored and ever-changing world of HPC. Amazon S3 Companies today need the ability to simply and securely collect, store, and analyze their data at a massive scale. Amazon Simple Storage Service (S3) is object storage built to store and retrieve any amount of data from anywhere – websites and mobile applications, corporate applications, and data from Internet of Things (IoT) sensors or devices3. Amazon S3 delivers 99.999999999% durability, and stores data for millions of applications used by market leaders in every industry. Amazon S3 also provides comprehensive security and compliance capabilities that meet even the most stringent regulatory requirements, giving customers flexibility in the way they manage data for cost optimization, access control, and compliance. Amazon S3’s query- in-place functionality allows you to run powerful analytics directly on your data at rest in Amazon S3. Lastly, Amazon S3 is the most supported cloud storage service available, with solutions available from a large community of third-party solution providers, systems integrator partners, and other AWS services. Amazon S3 is a highly scalable and high durable storage platform for HPC applications that support an object interface. Amazon S3 includes many features, such as lifecycle management, that allow you to move less frequently accessed down to lower storage tiers for more cost effective solution4. Even if your HPC application does not directly support Amazon S3, you can use Amazon S3 as a data repository to hold your dataset, which can be ingested into Page 3 Amazon Web Services – AWS Partner Network High Performance Computing Storage Options your processing file system. There also are APN Partner solutions that provide traditional filesystem access via SMB, NFS, and POSIX clients while using Amazon S3 as part of their solution for data storage and/or data protection. Amazon EBS Amazon Elastic Block Store (EBS) provides persistent block storage volumes for use with Amazon EC2 instances in the AWS Cloud. Each Amazon EBS volume is automatically replicated within its Availability Zone (AZ) to protect you from component failure, offering high availability and durability. Amazon EBS volumes offer the consistent and low-latency performance needed to run your workloads. With Amazon EBS, you can scale your usage up or down within minutes – all while paying a low price for only what you provision.