Amazon Fsx for Lustre?
Total Page:16
File Type:pdf, Size:1020Kb
FSx for Lustre Lustre User Guide FSx for Lustre Lustre User Guide FSx for Lustre: Lustre User Guide Copyright © Amazon Web Services, Inc. and/or its affiliates. All rights reserved. Amazon's trademarks and trade dress may not be used in connection with any product or service that is not Amazon's, in any manner that is likely to cause confusion among customers, or in any manner that disparages or discredits Amazon. All other trademarks not owned by Amazon are the property of their respective owners, who may or may not be affiliated with, connected to, or sponsored by Amazon. FSx for Lustre Lustre User Guide Table of Contents What is Amazon FSx for Lustre? ........................................................................................................... 1 Multiple deployment options ....................................................................................................... 1 Multiple storage options ............................................................................................................. 1 FSx for Lustre and data repositories ............................................................................................. 2 Amazon S3 integration ....................................................................................................... 2 On-premises data repositories ............................................................................................. 2 Accessing file systems ................................................................................................................. 2 Integrations with AWS services .................................................................................................... 3 Security and compliance ............................................................................................................. 3 Assumptions .............................................................................................................................. 3 Pricing for Amazon FSx for Lustre ................................................................................................ 4 Amazon FSx for Lustre forums ..................................................................................................... 4 Are you a first-time user of Amazon FSx for Lustre? ....................................................................... 4 Setting up ......................................................................................................................................... 5 Sign up for AWS ........................................................................................................................ 5 Create an IAM user ..................................................................................................................... 5 Adding permissions to use data repositories in Amazon S3 ...................................................... 7 Next step .................................................................................................................................. 7 Getting started .................................................................................................................................. 8 Prerequisites .............................................................................................................................. 8 Step 1: Create your Amazon FSx for Lustre file system .................................................................... 8 Step 2: Install the Lustre client .................................................................................................. 12 Step 3: Run your analysis .......................................................................................................... 13 (Optional) Step 4: Check Amazon FSx file system status ................................................................ 14 Step 5: Clean up resources ........................................................................................................ 14 File system deployment options ......................................................................................................... 15 Deployment options ................................................................................................................. 15 Scratch file systems .......................................................................................................... 15 Persistent file systems ...................................................................................................... 16 Using data repositories ..................................................................................................................... 18 Overview of data repositories .................................................................................................... 18 POSIX Metadata Support ................................................................................................... 19 Data repository lifecycle state ............................................................................................ 20 Attaching POSIX permissions to an S3 bucket .............................................................................. 20 Linking your file system to an S3 bucket ..................................................................................... 22 Creating file systems linked to an S3 bucket ........................................................................ 22 Working with server-side encrypted Amazon S3 buckets ....................................................... 25 Viewing a file system's export path .................................................................................... 26 Importing files from your data repository .................................................................................... 27 Automatically import updates from your S3 bucket .............................................................. 28 Preloading files into your file system .................................................................................. 31 Data repository tasks ................................................................................................................ 31 Types of data repository tasks ........................................................................................... 32 Task status and details ...................................................................................................... 32 Using data repository tasks ............................................................................................... 32 Working with task completion reports ................................................................................ 40 Troubleshooting failed tasks .............................................................................................. 40 Exporting changes to the data repository .................................................................................... 42 Using data repository tasks to export data and metadata changes .......................................... 42 Exporting files using HSM commands ................................................................................. 44 Releasing data from your file system .......................................................................................... 45 Using Amazon FSx with your on-premises data repository ............................................................. 45 Performance .................................................................................................................................... 46 How FSx for Lustre file systems work ......................................................................................... 46 iii FSx for Lustre Lustre User Guide Aggregate file system performance ............................................................................................ 47 Example: Aggregate Baseline and Burst Throughput ............................................................. 50 File system storage layout ......................................................................................................... 50 Striping data in your file system ................................................................................................ 50 Modifying your striping configuration ................................................................................. 51 Progressive file layouts ..................................................................................................... 52 Monitoring performance and usage ............................................................................................ 53 Performance tips ...................................................................................................................... 53 Accessing file systems ....................................................................................................................... 55 Installing the Lustre client ......................................................................................................... 55 Amazon Linux 2 and Amazon Linux .................................................................................... 55 CentOS and Red Hat ........................................................................................................ 56 Ubuntu ........................................................................................................................... 62 SUSE Linux ...................................................................................................................... 67 Mount from Amazon EC2 .........................................................................................................