IBM Spectrum Scale – Use Cases

IBM Spectrum Scale – Use cases –

Ulf Troppens, Tomer Perry

1. What is IBM Spectrum Scale? a. Evolution b. Key concepts

2. Primary Use Cases a. High performance computing (HPC) b. Data intensive application & workflows

3. Summary

Luca Bruno/AP

• 1993: Started as “Tiger Shark” research project at IBM Research Almaden as high performance filesystem for accessing and processing multimedia data

• Next 20 years: Grew up as General Parallel File System (GPFS) to power the world’s largest

supercomputers POSIX NFS • Since 2014: Transforming to IBM Spectrum GPFS Scale to support new workloads which need to process huge amounts of unstructured data

NVME SSD Disk Tape

• Based on GPFS, a robust, fast and mature parallel file system

• BUT: If you still just think GPFS, you miss: – Support for workflows which for example inject data via object, analyze results via Hadoop/Spark and view results via POSIX

– Storing and accessing large and small POSIX HDFS NFS SMB Swift/S3 objects (S3 and Swift) with low latency IBM Spectrum Scale – Automatic destaging of cold data to on premise or off premise object storage – Exchange of data between Spectrum Scale clusters via object storage in the cloud – Storing and starting OpenStack VMs without copying them from object storage NVMe SSD Disk Tape to local file system – GUI , REST API, Grafana Bridge – And many, many more

IBM Spectrum Scale: Use Cases / March 11, 2019 / © 2019 IBM Corporation 6 IBM Spectrum Scale 5.0.1 and 5.0.2 Summit: The world’s smartest and most powerful supercomputer IBM Spectrum Scale 5.0 The History of IBM Spectrum Scale IBM Spectrum Scale 4.2.2 IBM Spectrum Scale 4.2.1 and 4.2.2 IBM Spectrum Scale 4.2.2 and Capacity Pricing

IBM Spectrum Scale 4.2.1 w/ Transparent Cloud Tiering Spectrum Scale 4.2 w/ SWIFT & S3 This infographic is the genealogy of IBM Spectrum Scale, from its birth as a digital media server and HPC research project to its place as a foundational element in the IBM Spectrum Storage family. It highlights key 3,000 clients and over 100,000 systems milestones in the product history, usage, and industry to convey that Spectrum Scale may have started as GPFS, IBM Spectrum Scale 4.1 w/ OpenStack & Hadoop but it is so much more now. IBM has invested in the enterprise features that make it easy to use, reliable and Argonne Mira System 19PB and 400GB/s suitable for mission critical storage of all types. IBM joins OpenStack Foundation as Founding Member 2018 A Leader in the Gartner Magic Quadrant 2016, 2017, 2018* 10 Billion ﬁles in 43 minutes using ﬂash arrays

GPFS 3.4 2017 Over 2M licenses in use World-beating 2016 performance 1 Billion Files scanned in 3 hours! 2015 Cloud Data Hortonworks GPFS 3.1 Sharing certification 2014 ASCI Purple 10,000 disks, 2PB, 128GB/s iSCSI 2013 77 IBM ESS SC Bandwidth challenge record! 2012 Guided UI Transparent Object servers with 2011 Cloud Tiering GPFS 2.2 Sec17a-4 Spectrum Scale 5.0 Async SMB ASCI White 8192 disks, 110 TB, 7GB/s 2010 DR Spectrum Audit Log 2009 GUI Federated HDFS GPFS supports Linux LROC GPFS 1.1 Read Compression Hong Kong File Cloning Cache 2007 HAWC Telecom TCT 2006 SNMP LTFS Write Cache (tape) Bell Atlantic Federated HDFS Increased efficiency 2005 Declustered Sync DR SOFS/SONAS Share and performance for VOD field Trial RAID Self-tuning 2004 Nothing HPC and AI/ML/DL DMAPI Erasure Codes 2003 Clusters OpenStack Cloud AV Scans 2002 AFM (Active File ILM & Tiering HDFS 2001 Heterogeneous Mgmt.) 1993 1998 Clusters Metadata Scan ESS GUI Block & Policy Engine Snapshots Encryption Virtual & Secure Erase Vesta & POSIX NFS Tape Tiger Shark Server

UNIFIED STORAGE STORAGE TIERING DATA MANAGEMENT USABILITY * Gartner Magic Quadrant for Distributed File Systems and Object Storage

Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner's research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or ftness for a particular purpose.

IBM Spectrum Scale: Use Cases / March 11, 2019 / © 2019 IBM Corporation 7 IBM Spectrum Scale 5.0.1 and 5.0.2 Summit: The world’s smartest and most powerful supercomputer IBM Spectrum Scale 5.0 The History of IBM Spectrum Scale IBM Spectrum Scale 4.2.2 IBM Spectrum Scale 4.2.1 and 4.2.2 IBM Spectrum Scale 4.2.2 and Capacity Pricing

UNIFIED STORAGE STORAGE TIERING DATA MANAGEMENT USABILITY * Gartner Magic Quadrant for Distributed File Systems and Object Storage

Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of High Performance Computing Gartner's research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or ftness for a particular purpose. Data Intensive Applications Data Intensive Workflows Outline

1. What is IBM Spectrum Scale? a. Evolution b. Key concepts

2. Primary Use Cases a. High performance computing (HPC) b. Data intensive application & workflows

3. Summary

Management Monitoring Cloud-based Metadata-Driven Container & & Control Storage Data Insight VM APIs Management and Support

VM Data Copy Data Storage Archive Backup Services Availability Management

IBM’s comprehensive set of award-winning storage software delivered Infrastructure Scale-Out Virtualized Scale-Out Scale-Out Scale-Out across appliance, Software / NAS Block File Object Block converged and cloud. Software Defined

High IOPS Infrastructure All Flash Disk Cloud Servers Platforms

Highly scalable high-performance unified storage software for files and objects with integrated analytics

Remove data-related bottlenecks 2.5TB/s demonstrated throughput for a 250PB filesystem Enable global collaboration HDFS, files and object across sites Optimize cost and performance Automated data placement, movement and compression Ensure data availability, integrity and security End-to-end checksum, Spectrum Scale RAID, NIST/FIPS certification

IBM Spectrum Scale: Use Cases / March 11, 2019 / © 2019 IBM Corporation 11 Spectrum Scale environment Compute Nodes (NSD Clients) • Run applications to access and analyze data stored in one or more Spectrum Scale filesystems • Most nodes of a Spectrum Scale environment are Compute Nodes. Storage Nodes (NSD Server) • Provide the storage capacity for the Spectrum Scale filesystems Data Access Nodes (Remote & Local Access) • Access to Spectrum Scale filesystems using protocols like NFS, SMB, HDFS and Object Utility Nodes • Dedicated nodes for selected data management tasks such as backup, external tiering and hybrid cloud workflows. èThe Shared Data Network provides remote access to the Spectrum Scale environment. Management Nodes èThe Private Cluster Network connects all • Provides administration services components of the Spectrum Scale environment. (e.g., Spectrum Scale GUI, Performance Monitoring).

IBM Spectrum Scale: Use Cases / March 11, 2019 / © 2019 IBM Corporation 12 Data Center environment NFS&SMB Clients • Users and applications accessing data stored on a Spectrum Scale filesystem using NFS and/or SMB Other Clients • User and applications accessing data stored on a Spectrum Scale filesystem (e.g., Swift/S3, HDFS, Aspera, rsync, scp, etc.) • Administrative workstations (e.g. GUI client, REST API client, SSH client, etc.) Central Services • External infrastructure services required for the whole solution such as – Authentication and ID mapping (e.g. AD, LDAP), – Time synchronization (e.g., NTP), – Name resolution (e.g., DNS), etc. èThe Shared Data Network provides remote access to the Spectrum Scale environment. èThe Private Cluster Network connects all components of the Spectrum Scale environment.

Scaleable performance Unified data access – Billions of files and hundreds of petabytes – Proprietary NSD protocol for very high – Demonstrated 2.5TB/s aggregated throughput performance – Extend storage cache to compute for faster – Built-in NFS, SMB, HDFS and object for reads and writes application integration and end-user access Automated data management – Support for containers – Custom access nodes for integration of 3rd-party – Integration of NVMe, SSD, disk, tape and applications such as IBM Aspera, 0MQ, scp, etc. object in single filesystem – Policy-based data placement, data movement Flexible deployment options and compression to optimize costs – On-premise vs. cloud vs. hybrid – Integrated replication and scalable backup and – Single site vs. multi site restore for data protection – Reference Architectures vs. custom solutions – Audit logging, immutability, encryption and – IBM Elastic Storage Server vs. many other IBM checksums for compliance or 3rd-party storage systems

1. What is IBM Spectrum Scale? a. Evolution b. Key concepts

2. Primary Use Cases a. High performance computing (HPC) b. Data intensive application & workflows

3. Summary

• HPC is the Big Data of the 1980s/1990s. HPC always had the problem that it requires fast access to a lot of data. • Over the time IBM made enhancements to • Computer cluster (10s-1000s of nodes) Spectrum Scale to keep up to date with new • NFS and other protocols to ingest data technology (e.g. IB EDR, RoCE, NVMe, SSD) and and to access results new workloads (e.g. small files) to keep up to

date for customers computing needs. POSIX NFS • Nowadays Analytics/AI/ML/DL is everywhere. IBM Spectrum Scale It is a Big Data Problem, too. • Scaling and performance enhancements for HPC help Analytics and other use cases. • Enhancements for other use cases help HPC, e.g., the Spectrum Scale HDFS connector NVMe SSD Disk enables HPC customer to spin-up and terminate Hadoop or Spark clusters on their existing super computers like any other HPC job.

Imagine you need to meet these goals:

• 2.5 TB/sec single stream IOR as requested from ORNL • 1 TB/sec 1MB sequential read/write as stated in CORAL RFP • Single Node 16 GB/sec sequential read/write as requested from ORNL • 50K creates/sec per shared directory as stated in CORAL RFP • 2.6 Million 32K file creates/sec as requested from ORNL

IBM Spectrum Scale innovations have delivered these requirements

https://www.olcf.ornl.gov/summit/

World’s most powerful Summit System supercomputer World #2 supercomputer • 4608 nodes, each with: Sierra System • 2 IBM Power9 processors • 4320 nodes, each with • 6 Nvidia Tesla V100 GPUs • 2 IBM Power9 processors • 608 GB of fast memory • 4 Nvidia V100 GPUs • 1.6 TB of NVMe memory • 320 GB of node memory • 1.6 TB of NVMe memory • 200 petaflops peak IBM Spectrum Scale performance for modeling IBM Elastic Storage • IBM Spectrum Scale and simulation Server • IBM Elastic Storage Server • 3.3 ExaOps peak 2.5 TB/sec throughput performance for data to storage architecture 125 petaflops peak performance analytics and AI 250 PB HDD storage 154 PB HDD storage capacity capacity

1. What is IBM Spectrum Scale? a. Evolution b. Key concepts

2. Primary Use Cases a. High performance computing (HPC) b. Data intensive application & workflows

3. Summary

• Based on GPFS, a robust, fast and mature parallel file system • Type 1: Multiple tightly coupled instances of the same application running on multiple servers • Application farm that benefits from – Need: Fast shared filesystem for filesystem with scalable performance concurrent access to the same set of data • Data access is typically via applications – Examples:

- IBM DB2 POSIX NFS - SAS IBM Spectrum Scale • Type 2: Multiple isolated or loosely coupled instances of the same application running on multiple servers – Need: File system virtualization layer that flexibly provisions fast file storage to each

application instance NVMe SSD Disk - IBM Spectrum Protect - SAP HANA

• Based on GPFS, a robust, fast and mature parallel file system • Instruments and sensors like high-speed cameras, genome sequencers and super • Data intensive workflows from data microscopes generate huge amounts of data acquisition via analysis to archive that require HPC-like infrastructure to store • Integrate HPC for scalable analysis and analyze the acquired measured data • Spectrum Scale enables scientists to POSIX HDFS NFS SMB Swift/S3 seamlessly integrate HPC-like infrastructure IBM Spectrum Scale into their experiments and into their workflows to get timely insight in new data sets • The built-in support for multi-protocol eliminated the need to copy data for workflows that for instance ingest data via object, clean data via HDFS, analyze via POSIX and provide NVMe SSD Disk Tape results via NFS or SMB

https://www.youtube.com/ watch?v=JLCj4jQI3q8

IBM Spectrum Scale: Use Cases / March 11, 2019 / © 2019 IBM Corporation 22 IBM Spectrum Scale: Use Cases / March 11, 2019 / © 2019 IBM Corporation IBM 2019 / © 2019 11, / March Cases Use Scale: Spectrum IBM Typical Workflow for Data Intensive Science Intensive Data Workflowfor Typical f) e) d) c) b) a) è Long Storage Central of management Data processing) (offline analysis Deep Storage Central to Buffer Burst from movement Data real near and Visualization Real Scientists needaccesstodataduring each stageoftheworkflow - - time data ingest (data acquisition) term data archiving G S e u n p S o e D m e ( r C n

e e M a s

S o m i v c r e r e i s - o q c , r

time analysis (online processing) s u a … e c e , o n ) p c e e , r

Experiment (Real Experiment A c q D u a i s t a i t

i b o n

- 0 time) M Q 0 N S a M F M Q S B , ,

F ( O e B B e F n u u d a - f l r b f s NFS, SMB i s e n a t t

b r e c ) k Analysis (Iterative) Analysis c ( A S C O n t e f o a f n - r e

NFS, SMB l l a y t i n r g NSD s a e i e s l ) d f 23

A r c h i v e 23 Typical Workflow for Data Intensive Science (continued) a) Real-time data ingest (data acquisition) b) Visualization and near real-time analysis (online processing) c) Data movement from Burst Buffer to Central Storage d) Deep analysis (offline processing) e) Data management of Central Storage Fast Analysis f) Long-term data archiving Feedback (Off-line) (On-line)

b B Q B M M M

0 D S S

, S b , d S S N F F N N

a Device c f (Camera, Data 0MQ, Burst Central Super Microscope, NFS, Archive Genome Sequencer, Acquisition SMB Buffer Storage Sensors, …) e

IBM Spectrum Scale

è Scientists need access to data during each stage of the workflow è IBM Spectrum Scale has proven to support this workflow

IBM Spectrum Scale: Use Cases / March 11, 2019 / © 2019 IBM Corporation 24 24 IBM Spectrum Scale: Use Cases / March 11, 2019 / © 2019 IBM Corporation 25 IBM Spectrum Scale: Use Cases / March 11, 2019 / © 2019 IBM Corporation 26 IBM Spectrum Scale: Use Cases / March 11, 2019 / © 2019 IBM Corporation 27 Cloud infrastructures

• Pervasive Computing and Cloud is driving the development of new technologies such as object storage, virtual machines and containers All previous use cases • Those technologies get increasingly adopted in in a cloud architecture traditional enterprise data centers, in HPC environments and for Analytics/AI/ML/DL

• IBM makes enhancements in Spectrum Scale to POSIX HDFS NFS SMB Swift/S3 integrate in cloud architectures such as – Data access via object protocols IBM Spectrum Scale – Object storage as tier for cold data – Plug-ins to map directories into containers – Ready-to-use templates to run Spectrum Scale on AWS

NVMe SSD Disk Tape

1. What is IBM Spectrum Scale? a. Evolution b. Key concepts

2. Primary Use Cases a. High performance computing (HPC) b. Data intensive application & workflows

3. Summary

• Spectrum Scale is based on GPFS, a robust, fast and mature parallel file system. • The filesystem of the largest super computers are build on Spectrum Scale. • Spectrum Scale’s built-in parallelism enables a data layer that meets the performance and scaling requirements of data intensive applications and workflows such as Big Data, Analytics and AI/ML/DL. • Spectrum Scale’s built-in support for POSIX, NFS, SMB, HDFS and object accelerates workflows that require multiple access methods.