#CLUS BRKDCN-2494 Deep Dive into NVMe™ and NVMe™ over Fabrics (NVMe-oF™) J Metz, Ph.D, R&D Engineer, Advanced Storage Board of Directors, NVM Express, Inc. Board of Directors, SNIA Board of Directors, FCIA @drjmetz #CLUS Agenda • Introduction • Who is NVM Express? • Storage Concepts • What is NVMe™ • NVMe Operations • The Anatomy of NVM Subsystems • Queuing and Queue Pairs • NVMe over Fabrics (NVMe-oF™) • How NVMe-oF Works • NVMe-oF™/RDMA • NVMe-oF™/FC • NVMe-oF™/TCP (Future) • Whats New in NVMe 1.3 • Additional Resources #CLUS BRKDCN-2494 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 3 What This Presentation Is… and Is Not • What it is: • A technology conversation • Deep dive (We’re going in, Jim!) • What it is not • A product conversation • Comprehensive and exhaustive #CLUS BRKDCN-2494 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 4 Goals • At the end of this presentation you should know: • What NVMe is and why it is important • How NVMe is extended for remote access over a network (i.e., “Fabrics”) • The different types of fabrics and their differences • Some of the differences between traditional SCSI- based storage solutions and NVMe-based solutions • What’s new in NVMe and NVMe over Fabrics #CLUS BRKDCN-2494 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 5 Prerequisites • You really should know… • Basics of Block-based storage • Basic terminology (initiator, target) • Helpful to know… • Basic PCIe semantics • Some storage networking #CLUS BRKDCN-2494 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 6 Note: Screenshot Warning! • Get your screenshots ready when you see this symbol: • Useful for getting more information about a topic, URLs, etc. #CLUS BRKDCN-2494 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 7 What We Will Not Cover (In Detail) • NVMe-MI (Management) • NVMe-KV (Key Value) • Protocol Data Protection features • Advances in NVMe features • RDMA verbs • Fibre Channel exchanges • New form factor designs #CLUS BRKDCN-2494 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 8 Who is NVM Express? NVM Express, Inc. 125+ Companies defining NVMe together Board of Marketing Subcommittee Directors NVMexpress.org, webcasts, 13 elected companies, stewards of tradeshows, social media, and press the technology & driving processes Co-Chairs: Janene Ellefson and Chair: Amber Huffman Jonmichael Hands Technical Management I/F Interop (ICC) Workgroup Workgroup Workgroup Out-of-band management over PCIe® Interop & Conformance Testing in Base specification and VDM and SMBus collaboration with UNH-IOL NVMe over Fabrics Chair: Peter Onufryk Chair: Ryan Holmqvist Chair: Amber Huffman Vice-Chair: Austin Bolen #CLUS BRKDCN-2494 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 10 About NVM Express (The Technology) • NVM Express (NVMe™) is an open collection of standards and information to fully expose the benefits of non-volatile memory in all types of computing environments from mobile to data center • NVMe™ is designed from the ground up to deliver high bandwidth and low latency storage access for current and future NVM technologies NVM Express Base Specification The register interface and command set for PCI Express attached storage with industry standard software available for numerous operating systems. NVMe™ is widely considered the defacto industry standard for PCIe SSDs. NVM Express Management Interface (NVMe-MI™) Specification The command set and architecture for out of band management of NVM Express storage (i.e., discovering, monitoring, and updating NVMe™ devices using a BMC). NVM Express Over Fabrics (NVMe-oF™) Specification The extension to NVM Express that enables tunneling the NVM Express command set over additional transports beyond PCIe. NVMe over Fabrics™ extends the benefits of efficient storage architecture at scale in the world’s largest data centers by allowing the same protocol to extend over various networked interfaces. #CLUS BRKDCN-2494 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 11 NVMe™ Adoption – Industry • NVMe™ displacing SAS and SATA SSDs in server/PC markets • PCIe NAND Flash SSDs primarily inside servers • Lower latency Storage Class Memory (i.e., 3D Xpoint™) SSDs - NVMe™-only • Extensive Client (i.e., laptop, tablet) use of smaller form factor SSDs – M.2 and BGA • NVMe™ ecosystem and recognition growing quickly • Many servers offer NVMe™ slots - different server configurations and form factors • Startups already shipping NVMe™ and NVMe over Fabrics™ (NVMe-oF™) solutions • Storage class NVMe™ SSDs emerging – enable high availability (HA) in storage arrays • NVMe-oF™ emerging as a solution to limited scale of PCIe as a fabric • Expanding ecosystem (i.e., Analyzers, NVMe-oF™ adapters) #CLUS BRKDCN-2494 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 12 The Basics of Storage and Memory The Anatomy of Storage • There is a “sweet spot” for storage • Depends on the workload and application type • No “one-size fits all” • Understanding “where” the solution fits is critical to understanding “how” to put it together • Trade-offs between 3 specific forces • “You get, at best, 2 out of 3” #CLUS BRKDCN-2494 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 14 The Anatomy of Storage • There is a “sweet spot” for storage • Depends on the workload and application type • No “one-size fits all” • Understanding “where” the solution fits is critical to understanding “how” to put it together • Trade-offs between 3 specific forces • “You get, at best, 2 out of 3” • NVMe goes here #CLUS BRKDCN-2494 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 15 Storage Solutions - Where Does It Fit? • Different types of storage apply in different places #CLUS BRKDCN-2494 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 16 Storage Solutions - Where Does NVMe Fit? • Different types of storage apply in different places • NVMe is PCIe-based • Local storage, in-server #CLUS BRKDCN-2494 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 17 Storage Solutions - Where Does NVMe Fit? • Different types of storage apply in different places • NVMe is PCIe-based • Local storage, in-server • PCIe Extensions (HBAs and switches) extend NVMe outside the server #CLUS BRKDCN-2494 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 18 Storage Solutions - Where Does NVMe-oF Fit? • Different types of storage apply in different places • NVMe is PCIe-based • Local storage, in-server • PCIe Extensions (HBAs and switches) extend NVMe outside the server • NVMe-oF inherits characteristics of underlying transport #CLUS BRKDCN-2494 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 19 But First… SCSI • SCSI is the command set used in traditional storage • Is the basis for most storage used in the data center • Obviously, is the most obvious starting point for working with Flash storage • These commands are transported via: • Fibre Channel • Infiniband • iSCSI (duh!) • SAS, SATA • Works great for data that can’t be accessed in parallel (like disk drives that rotate) • Any latency in protocol acknowledgement is far less than rotational head seek time #CLUS BRKDCN-2494 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 20 Evolution from Disk Drives to SSDs The Flash Conundrum • Flash • Requires far fewer commands than SCSI provides • Does not rotate (no latency time, exposing latency of a one command/one queue system) • Thrives on random (non-linear) access • Both read and write • Nearly all Flash storage systems use SCSI for access • But they don’t have to! #CLUS BRKDCN-2494 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 21 So, NVMe… • Specification for SSD access via PCI Express (PCIe), initially flash media • Designed to scale to any type of Non Volatile Memory, including Storage Class Memory • Design target: high parallelism and low latency SSD access • Does not rely on SCSI (SAS/FC) or ATA (SATA) interfaces: New host drivers & I/O stacks • Common interface for Enterprise & Client drives/systems: Reuse & leverage engineering investments • New modern command set • 64-byte commands (vs. typical 16 bytes for SCSI) • Administrative vs. I/O command separation (control path vs. data path) • Small set of commands: Small fast host and storage implementations • Standards development by NVMe working group. • NVMe is de facto future solution for NAND and post NAND SSD from all SSD suppliers • Full support for NVMe for all major OS (Linux, Windows, ESX etc.) • Learn more at nvmexpress.org #CLUS BRKDCN-2494 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 22 What is NVMe? NVMe Operations What’s Special about NVMe? Slower Faster . HDD: varying speed conveyer belts . Flash: all data blocks available at the . Flash: all data blocks available at the carrying data blocks (faster belts = same seek time & latency same seek time & latency lower seek time & latency) . SCSI / SAS: pick & place robot . SCSI /SAS: pick & place robot . NVMe / PCIe: pick & place robot with w/tracked, single arm executing 1 w/single arm executing 1 command at 1000s of arms, all processing & command at a time, 1 queue a time, 1 queue executing commands simultaneously, with high depth of commands #CLUS BRKDCN-2494 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 25 Technical Basics • 2 key components: Host and NVM Subsystem (a.k.a. storage target) • “Host” is best thought of as the CPU and NVMe I/O driver for our purposes • “NVM Subsystem” has a component - the NVMe Controller - which does the communication work #CLUS BRKDCN-2494 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 26 Technical Basics • Memory-based deep queues (up to 64K commands per queue, up to 64K queues) • Streamlined and simple command set (13; only 3 required commands) • Command completion interface optimized for success (common case) • NVMe Controller: SSD element that processes NVMe commands #CLUS BRKDCN-2494 © 2018 Cisco and/or its affiliates.