An Overview of On- Premise File and Object Storage Access Protocols Dean Hildebrand - Research Staff Member, IBM Research Bill Owen - Senior Engineer, IBM v1.2 Outline Introduction File and Object Discussion NFS Object Storage Introduction Swift S3 File to Object and Object to File Comparison Discussion Conclusion 2 Dean Hildebrand Bill Owen Research Staff Member Senior Engineer IBM Research IBM 3 Attendance Poll SysAdmin/ Storage Architect/ Developers Students Researchers Manager 4 Software Storage Market Growth 5 Accessing Data in On-Premise Storage Systems 6 Local vs Shared Storage 7 Local Storage ● Most common for laptops, desktops, mobile devices, server OS boot disks ● Typically formatted with a file system ○ E.g., Ext4, XFS, NTFS, HFS+, BtrFS, ZFS ● Invaluable to manage a single device (or maybe a few with LVM) ● Varying levels of availability, durability, scalability, etc, supported ● All limited to a single node ○ E.g., Cannot support VM or container migration, support 1000s of applications, etc, etc ● In your research, think about the real benefits of further optimizing local storage ○ How many pressing problems are left to be solved? Only incremental gains? ○ Common to be used as a building block in higher level storage systems 8 Shared Storage Supports any kind of client Independent Scaling of device clients Supports any type of network and network/file protocol Network Independent Scaling of storage bandwidth and SSD Fast Slow Tape Supports any kind of capacity Disk Disk storage device 9 Block Shared Storage ● Used to dominate, now mostly shrinking...except ○ FC continues to have very low iSCSI/FC/FCoE latency, and so finding new life (and others) with Flash storage systems FibreChannel/Ethernet ○ iSCSI still very popular for VMs Typically in pairs for H/A Fast Tape SSD Slow Disk Disk 10 Parallel and Scale-out File Systems ● Scalability (all dimensions) ● Performance (all dimensions) ● Support general applications and middleware ● Make managing billions of files, TB/s Proprietary File of bandwidth, and PBs of data *easy* Access Protocol Commercial Scale-out as Infiniband/Ethernet needed HPC ... Fast Tape SSD Slow Disk Disk 11 Distributed Access Protocols Wide variety of solutions Ethernet Infiniband/Ethernet Vast Range of Performance and Scalability Options ... Standard and Non- Slow Standard Protocols Disk Fast Tape SSD Slow Disk Disk 12 Distributed Access Protocols: Portability and Lock-In ● Standard APIs help ○ Maximize Application Portability ○ Minimize Vendor Lock-in ● Numerous benefits of standard protocols ○ Standard protocol clients ship in most OSs ○ Promote predictability of semantics and application behavior ○ Minimize changes to applications and system infrastructure when switching to new storage system (many times due to reasons out of your control) ○ Applications can move between on-premise and off-premise (cloud) systems ○ Wider and more broad user base makes it easier to find support and also hardens implementations 13 Distributed Access Protocols: Standards Are Not A Silver Bullet ● For File, while applications use POSIX, they are sensitive to implementation ○ No common set of commands guarantee crash consistency [***] ○ For distributed file systems, it becomes even more complicated ■ Different crash consistency semantics, cache consistency semantics, locking semantics, security levels/APIs/tools, etc ● For Object, each implementation varies w.r.t. level of eventual consistency, security, versioning, etc ● Even what we consider standards are not actually well defined ○ E.g., SMB, S3 Examples: ● CIFS/SMB does sequential consistency whereas NFS has close-to-open semantics ● Versioning is quite different between object protocols [***] - All File Systems Are Not Created Equal: On the Complexity of Crafting Crash-Consistent Applications, OSDI’14 14 Distributed Access Protocols: One Storage Protocol CANNOT Do It All ● There are so many vendors...each claiming they have *solved* data storage ○ or is it world hunger? ● Vendors sell what they have, not what you need ○ A storage seller takes what they have a makes it fit for practically any requirement and use case ○ Leads to many unsatisfied customers soon after deployment ● Many protocols have existed: DDN WOS, EMC ATMOS, CDMI, AFS, DFS, RFS, 9P, etc Tips ● Attend sessions like this to learn more about reality and not hype :) ● Dig into advertised feature support ○ How many customers use a feature, will the customer talk about it, in what context do they use it, etc ● Validate system on-premise using realistic workloads (do you know your workloads?) ○ Remember there is no guarantee for what you haven’t tried (x- and y-axis’ have an upper bound for a reason) ● Don’t buy H/W first and then expect any storage S/W vendor to support it efficiently 15 On Premise Data Access Protocols: NFS and now Swift, S3 Winners File Object NFS and SMB are the clear winners Industry appears to be centralizing around Swift and S3 SMB is being discussed in SNIA S3: Amazon + many, many tutorial this week, so we’ll focus on apps/tools NFS Swift: Open source + API + Note: HDFS also dominant for analytics 3 cloud vendors (or more) Easily repatriate apps due to cost 16 Tutorial Goals SysAdmin/ Storage Architect/ Developers Students Researchers Manager • Understand which • Understand how to ● Introduction to NAS and ● Understand challenges protocols are best for choose the best protocol Object history and of on-premise which applications for an application (and vendor landscape distributed data access consequences of • Understand tradeoffs choosing the wrong ● Introduction to ● Understand on-premise between protocols protocol) distributed data access data center challenges research potential ● Introduction to • Introduction to vendor landscape distributed data access research potential • Be able to determine which file-based applications are good candidates for using an object protocol 17 Outline Introduction File and Object Discussion NFS Object Storage Introduction Swift S3 File to Object and Object to File Comparison Discussion Conclusion 18 Fantasy File and Object Both Can Do Anything 19 Reality vs File and Object Each has their strengths and weaknesses 20 Reality vs File and Object Confusion Each has their strengths and weaknesses 21 Object vs File Summary Object Store File Target ‘cold’ data (Backup, Read-only, Archive) Target most workloads (except HPC) Low to Medium Performance Medium to High performance Typically Scales to Large Capacity Typically Scales to Medium Scalability Low cost Low to High Cost Global and Ubiquitous/Mobile Access Limited Capability for Global Distribution Data Access through REST APIs Standard File Data Access Immutable Objects and Versioning POSIX + Snapshots Loose/Eventual Consistency Strong(er) Consistency 22 Outline Introduction File and Object Discussion NFS Object Storage Introduction Swift S3 File to Object and Object to File Comparison Discussion Conclusion 23 NFS: A Little History... ● NFSv2 in 1983 ○ Synchronous, stable writes,...outdated ○ Finally removed from Fedora ● NFSv3 in 1995 ○ Still default on many distros... ● NFSv4 in 2003, updated 2015 ○ Default in RHEL... possibly others ● NFSv4.1 and pNFS in 2010 ○ Many structural changes and new features ● NFSv4.2 practically complete now… ○ Many new features, VM workloads specifically targeted ● Now going to try per-feature releases… 24 Deployment ● The beauty is that it is everywhere… (even Windows) ○ Well mostly...more on that later with object ● Most NFS servers are in-kernel or proprietary ○ But Ganesha is the first open-source user-level NFS server daemon ● For the enterprise, Scale-out NAS now a requirement for capacity and availability ● New clients and environments emerging ○ VMware announces support for NFSv4.1 as a client for storing VMDKs ○ Amazon announces support for NFSv4.0 in AWS Elastic File System (EFS) ○ OpenStack Manila is a shared file service with NFS as the initial file protocol ○ Docker has volume plugins that support NFS 25 NFS Caching Semantics ● Not POSIX… ○ But a single client with exclusive data access should see POSIX semantics ● v2 could not cache data ○ Sync writes ● v3 can cache data, but... ○ Weak cache consistency ○ Revalidates on open and periodically (30s in Linux) ○ Data must be kept in cache until ‘committed’ by server (just in case the server fails) ● v4 standardizes ‘close to open’ cache consistency ○ Similar to v3, but guarantees cache is revalidated on OPEN, flushed at CLOSE ○ Also checked periodically and at LOCK/LOCKU ○ Note granularity is typically on 1 second ○ Delegations reduces number of revalidations required... 26 NFSv3 ● Collection of protocols (file, mount, lock, status) ○ Each on their own port ● Stateless (mostly) ○ Locks add state ○ Server must keep request cache to prevent duplicate non-idempotent RPCs ● UNIX-centric, but seen in Windows too ● 32 bit numeric uids/gids ● UNIX permissions, but Kerberos also possible ● Works over UDP, TCP ● Needs a-priori agreement on character sets 27 NFSv4 New Features ● Security ● Finally standardized almost everything ○ NFSv4 ACLs (much more full featured than POSIX ACLs) ● Custom export tree with pseudo-name space ○ Use of named strings instead of 32-bit integers ● Mandatory use of congestion protocol (TCP) ○ Lofty goals with new GSS-API, but essentially benefit is that ● Delegations Kerberos is officially supported and easier to configure ■ Kerberos V5 must always supported (but not ○ Clients become server for a file, coordinating multi- necessarily used) threaded access ○ Less communication