AWS Storage Gateway Fred Hutch - Partly Cloudy Conference Jeff Bartley | Storage Solutions Architect | [email protected] October 25th, 2018 © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Enabling a spectrum of hybrid use cases Analytics | File Services | Production Tiering | Data Processing Data Distribution | Backup | DR | Archive | Migration Amazon Amazon Amazon Amazon Amazon Amazon AWS Amazon EMR Athena S3 Glacier EBS EC2 Lambda CloudFront AWS Storage Gateway Enterprise Data Centers Remote Offices Research Sites Small-Medium Businesses Multimedia content File servers Databases App. servers Devices Users Backup servers © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Storage Gateway Family Hybrid storage service enabling applications to seamlessly use AWS storage File Gateway Store and access objects in Amazon S3 from file-based applications with local caching Volume Gateway Block storage on-premises backed by cloud storage with local caching, EBS snapshots, and clones Tape Gateway Drop-in replacement for physical tape infrastructure backed by cloud storage with local caching © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Storage Gateway Family Files Amazon S3 (NFS or SMB) AWS Storage Gateway Amazon EBS Volumes snapshots (iSCSI) HTTPS Gateway Service Amazon Glacier Tapes AWS Identity and Access (iSCSI VTL) AWS CloudTrail Management (IAM) Amazon AWS Key Management Service Customer Premises CloudWatch (AWS KMS) Gateway provides applications Native storage in AWS • Protocol conversion and device emulation • Objects in S3 (file) • Caching (read-through / write-back) • Snapshots in EBS (volume) • Optimized data transfer • Archives in Glacier (tape) © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. File Gateway Store and access objects in Amazon S3 from file-based applications with local caching Customer Premises NFS or HTTPS SMB Application File Gateway Objects in your Server S3 bucket Reduce on-premises Fully managed local Durability, scalability, storage infrastructure cache for low latency and reliability of access Amazon S3 storage © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. File Gateway support for SMB SMB v2 or v3 File Gateway Objects in your client Amazon S3 bucket Store and access objects in Amazon S3 buckets from file-based Windows applications On-premises Active Share-level access can POSIX ACLs Objects created Directory (AD) or AWS be restricted to (compatible subset of directly in S3 inherit Directory services configured users and NTFS) for object-level ACLs from parent groups permissions folder © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Hybrid File Use Case – Backup to AWS Customer Premises AWS Region 1 2 3 Database File Gateway Objects in S3 Database server instance Amazon 1 Database server creates backup file CloudWatch Events 2 Gateway takes care of upload to S3 3 Automate restore or validation in-cloud or lifecycle to Glacier for archival © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Hybrid File Use Case – Content Distribution 1 2 Application Objects in your S3 Application File Gateway bucket File Gateway (read-only) San Diego Data Center AWS Region Sydney Data Center 1 Application in San Diego writes files which are uploaded to S3 by gateway 2 After refresh cache, files are visible to applications in Sydney. Local cache improves access performance. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Hybrid File Use Case – Active Archive 1 2 NAS Application File Gateway AWS Direct S3 Standard S3-Infrequent Amazon LTO Servers Connect Access Glacier Customer Data Center 1 Use Snowball to ship data from on-premises offline archives 2 Online access to all data through gateway, reduces on-premises storage cost, S3 lifecycle policies to lower costs further © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Enabling Cloud Workloads Easily migrate existing applications to the cloud “We have applications that don’t support S3 natively, and we can’t make changes to the applications, so being able to use native protocols is very important to us. Storage Gateway allows Celgene to get data to the cloud without making any changes to the application or environment.” – Lance Smith, IT Director, Celgene © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. File Gateway Deep Dive © 2018,© 2018, Amazon Amazon Web Web Services, Services, Inc. Inc. or itsor Affiliates.its Affiliates. All rightsAll rights reserved. reserved. Amazon Confidential and Trademark File Gateway – Control Over Data Storage and Access NFS or SMB File Gateway Objects in your S3 client bucket File share options Amazon S3 options per bucket • Restrict access by IP (NFS) or AD (SMB) • IAM role • Read-only/read-write • Storage class • Default ownership and permissions • Encryption with AWS KMS • User squashing (NFS) • Guess MIME type, requester pays, bucket owner ACL, … © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. File Gateway – Mapping Files to Objects \\{IP}\{Bucket1}\{Folder}\{File1} s3://{Bucket1}/{Prefix}/{Object1} \\{IP}\{Bucket1}\{Folder}\{File2} s3://{Bucket1}/{Prefix}/{Object2} \\{IP}\{Bucket1}\{Folder}\{File3} s3://{Bucket1}/{Prefix}/{Object3} SMB v2 or v3 client Objects in Bucket1 your Amazon S3 bucket NFS v3 or v4 File Gateway client Bucket2 {IP}:/{Bucket2}/{Folder}/{File1} s3://{Bucket2}/{Prefix}/{Object1} {IP}:/{Bucket2}/{Folder}/{File2} s3://{Bucket2}/{Prefix}/{Object2} {IP}:/{Bucket2}/{Folder}/{File3} s3://{Bucket2}/{Prefix}/{Object3} © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. File Gateway – Storage Architecture File Gateway File Cache share Disk NFS or SMB S3 Bucket client Customer Premises AWS Region © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. File Gateway – Storage Architecture File Gateway Data cache File Cache share Disk NFS or SMB Metadata S3 Bucket client cache Customer Premises AWS Region © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. File Gateway – Write Written data remains in cache File Gateway Data cache File share NFS or SMB Metadata S3 Bucket client cache Customer Premises AWS Region Writes to S3 are optimized © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. and encrypted File Gateway – Read from Cache File Gateway Data cache File share NFS or SMB Metadata S3 Bucket client cache Customer Premises AWS Region © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. File Gateway – Read from S3 File Gateway Data cache File share NFS or SMB Metadata S3 Bucket client cache Customer Premises AWS Region © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. File Gateway – Low Latency Access to Active Data Durable storage of all of your data File Gateway Data cache File share NFS or SMB Metadata S3 Bucket client cache Customer Premises AWS Region Fully managed cache of recent data © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. File Gateway – Metadata Cache Bucket can have billions of objects in it, gateway discovers them on- demand File Gateway Data cache File share NFS or SMB Metadata S3 Bucket client cache Customer Premises AWS Region Metadata is cached based on operations performed by the file client © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. File Gateway - File System Metadata • File system metadata (permissions, ownership, ctime, etc.) persisted in object user-metadata • Configurable defaults for objects that don’t have this metadata (i.e. objects that were already in the bucket) • Changing file metadata results in a copy- put operation © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. File Gateway - File system operations directories • Folder objects created in S3 using same pattern as S3 Console • Walking the file system can be expensive (ls -R, find, etc). • Caching of metadata locally reduces latency of directory level operations (such as “ls”). delete • Removes the object in S3 rename • Atomic for clients connected to the same gateway • Copy-put request on S3, so eventually consistent for S3 clients links • Hard links and symbolic links are not supported • No analog in S3, so returns “Operation not supported” sparse files • Object is zero-filled for sparse ranges © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Deployment On-Premises virtual environments Run as virtual machine on VMware ESXi and Microsoft Hyper-V Hardware Appliance Pre-installed with Storage Gateway software. Ideal for branch offices, warehouses, and “outpost” offices that lack dedicated IT resources. Amazon EC2 Run Storage Gateway in your AWS VPC environment © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Deployment Best Practices VM environments • Four virtual processors assigned to the VM. • 16 GiB of reserved RAM assigned to the VM. • 80 GiB of disk space for installation of VM image and system data. Cache Disk • One or more disks - 150 GB to 16 TB total cache • Use redundant storage © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon CloudWatch monitoring/alerting Metrics • Cache metrics • S3 read/write bytes • Share read/write bytes Events • Notify on file upload • Cache refresh © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. To Learn More… aws.amazon.com/storagegateway/ • Whitepapers • Webinars • Demos & Videos • Developer Resources • Recorded AWS re:Invent Sessions • Product Documentation • Pricing • FAQs • Customer Case Studies © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Thank you! © 2018,© 2018, Amazon Amazon Web Web Services, Services, Inc. Inc. or itsor Affiliates.its Affiliates. All rightsAll rights reserved. reserved. Amazon Confidential and Trademark.