Networking Flash Technology “Showdown”

Rob Davis, J Metz, Motti Beck, Brandon Hoff, Peter Onufryk

Flash Memory 2018 Santa Clara, CA 1 Why Network Flash Based Storage?

. There are advantages to shared storage • Better utilization: – capacity, rack space, power PCIe InfiniBand Network • Scalability • Manageability • Fault isolation . Shared storage requires a Network

Flash Memory Summit 2017 Santa Clara, CA 2 Agenda

• PCIe Networked Flash Storage • Peter Onufryk, Microsemi(Microchip), NVM Solutions Fellow • InfiniBand Networked Flash Storage • Motti Beck, Mellanox, Sr. Dir. Enterprise Market Development • Fibre Channel Networked Flash Storage • Brandon Hoff, Principle Software Architect, Connectivity Division, Broadcom • Ethernet Networked Flash Storage • J Metz, Cisco, R&D Engineer, Advanced Storage, Office of the CTO, UCS Systems Group • Panel Discussion

Flash Memory Summit 2018 Santa Clara, CA 3 Faster Storage Needs a Faster Network

Flash 24 HDDs to SSDs fill 10GbE move the Only 2 SSDs fill Bottleneck 10GbE from the One SAS SSD fills Disk to the 10GbE Network

Flash Memory Summit 2018 Santa Clara, CA 4 Eliminate the Bottleneck

Networked Storage Protocol and Network 100% Network

100 Storage

PM Sec)

Media -

Storage Media SSD

Storage Media HDD 15 Access Time (micro

? 0.01 HDDHD SSD NVMPM FC, TCP RDMA RDMA+ 50%

Storage Protocol (SW) Network Flash Memory Summit 2018 Santa Clara, CA 5 PCIe® Fabric

Peter Onufryk Microsemi Corporation

Flash Memory Summit 2018 Santa Clara, CA 6 PCIe Fabric . Storage Functions NVMe NVMe NVMe NVMe NVMe NVMe NVMe NVMe NVMe Host Host Host Host Host Host Host Host Host • Dynamic partitioning (drive-to-host mapping) • NVMe shared I/O (shared storage) PCIe Switch PCIe Switch PCIe Switch . Host-to-Host Communications N N N N N N N N N N N N N N N V V V V V V V V V V V V V V V

M M M M M M M M M M M M M M M • RDMA e e e e e e e e e e e e e e e

S S S S S S S S S S S S S S S

S S S S S S S S S S S S S S S • Ethernet emulation D D D D D D D D D D D D D D D . Manageability

N N N N N N N N N N N N N N N • NVMe controller-to-host mapping V V V V V V V V V V V V V V V M M M M M M M M M M M M M M M

e e e e e e e e e e e e e e e • PCIe path selection

S S S S S S S S S S S S S S S S S S S S S S S S S S S S S S

D D D D D D D D D D D D D D D • NVMe management . Fabric Resilience PCIe Switch PCIe Switch PCIe Switch • Supports link failover • Supports fabric manager failover NVMe NVMe NVMe NVMe NVMe NVMe NVMe NVMe NVMe Host Host Host Host Host Host Host Host Host

Flash Memory Summit 2018 Santa Clara, CA 7 The PCIe Latency Advantage

Flash SSD Media + Controller ~80mS

Fabric Overhead

Next Generation NVM SSD Next Generation NVM SSD Media + Controller ~10mS Media + Controller ~2mS

Fabric Overhead Fabric Overhead

Latency data from Z. Guz et al., “NVMe-over-Fabrics Performance Characterization and the Path to Low-Overhead Flash Disaggregation” in SYSTOR ‘17 8 The PCIe Advantage

HBA PCIe Network Network PCIe NVMe Host HCA Switch Interface SSD NIC

Other Flash Storage Networks

PCIe PCIe PCIe NVMe Host Switch SSD < 200ns

PCIe Fabric

Flash Memory Summit 2018 Santa Clara, CA 9 PCIe Fabric Characteristics

Ideal PCIe Property Notes Characteristic Fabric Cost Free Low • PCIe built into virtually all hosts and NVMe drives

Complexity Low Medium • Builds on existing NVMe ecosystem with no changes • PCIe fabrics are an emerging technology • Requires PCIe SR-IOV drives for low-latency shared storage Performance High High • High bandwidth • The absolute lowest latency Power consumption None Low • No protocol translation

Standards-based Yes Yes • Works with standard hosts and standard NVMe SSDs

Scalability Infinite Limited • PCIe hierarchy domain limited to 256 numbers • PCIe has limited reach (cables) • PCIe fabrics have limited scalability (less than 256 SSDs and 128 hosts)

10 InfiniBand Networked Flash Storage

Superior Performance, Efficiency and Scalability

Motti Beck – Sr. Director Enterprise Market Development,

Flash Memory Summit 2018 Santa Clara, CA The Need for Intelligent and Faster Interconnect Faster Data Speeds and In-Network Computing Enable Higher Performance and Scale

CPU-Centric (Onload) Data-Centric (Offload)

CPU GPU CPU GPU

GPU CPU GPU CPU Onload Network In-Network Computing CPU GPU CPU GPU

GPU CPU GPU CPU

Must Wait for the Data Analyze Data as it Moves! Creates Performance Bottlenecks Higher Performance and Scale Flash Memory Summit 2018 Santa Clara, CA InfiniBand Technical Overview

. What is InfiniBand? • InfiniBand is an open standard, interconnect protocol developed by the InfiniBand® Trade Association: http://www.infinibandta.org/home • First InfiniBand specification was released in 2000

. What does the specification includes? • The specification is very comprehensive • From physical to applications

. InfiniBand SW is open and has been developed under OpenFabrics Alliance • http://www.openfabrics.org/index.html 13

Flash Memory Summit 2018 Santa Clara, CA RDMA enabled Networking Powers Modern Storage Platforms

Higher Performance, Higher Efficiency and Higher Scalability NVMe over Fibre Channel

Brandon Hoff Principal Software Architect Emulex Connectivity Division, Broadcom Inc.

Flash Memory Summit 2018 Santa Clara, CA 15 Enterprise Storage Market

Block Storage Capacity Shipped • Fibre Channel storage shows strong growth in capacity 60,000

– Fibre Channel Storage capacity d 50,000 e Fibre Channel shipped is larger than all other types of

p

p external storage combined i 40,000

h

S

• The adoption of All Flash Arrays

s 30,000

and NVMe storage will drive the e

t need for faster networks y 20,000 iSCSI

b

a • iSCSI is the dominate technology t e 10,000

over Ethernet P Others - • Logical transition 2017 2018 2019 2020 • SCSI FCP transitions to NVMe/FC • iSCSI transitions to NVMe/TCP Other Includes: FICON, FCoE, Infiniband, External SAS IDC WW Capacity Shipped, 2016 Dual protocol SANs enable low risk NVMe adoption

• Get NVMe performance benefits Gen 5 HBAs while migrating incrementally SCSI “as-needed” SCSI NVMe NVMe SCSI

• Migrate application volumes 1 by 1 with easy rollback options

• Interesting dual-protocol use cases

NVMe Traffic • Full fabric awareness, visibility and manageability with existing SCSI Traffic Brocade Fabric Vision technology

17 NVMe over Fibre Channel Performance on a A700s single node Ethernet-Networked Flash Storage

J Metz, Ph.D R&D Engineer, Advanced Storage @drjmetz Flash Memory Summit 2018 Santa Clara, CA 19 Storage Perspective • There is a “sweet spot” for storage • Depends on the workload and application type • No “one-size fits all” • What is the problem to be solved? • Deterministic or non-deterministic? • Highly scalable or highly performant? • Level of manageability? • Understanding “where” the solution fits is critical to understanding “how” to put it together Flash Memory Summit 2018 Santa Clara, CA 20 Ethernet Roadmap • How to go faster • Different modulation techniques • Different data rate/lanes chosen • New Signaling methods • Pulse Amplitude Modulation 4 vs. Non Return to Zero (NRZ) • New Form Factors • Multi-lane interfaces

Flash Memory Summit 2018 Santa Clara, CA Source: ethernetalliance.org 21 Summary

• Ethernet • General Purpose network designed to solve many, many problems and do it well • Flexible for all but the most extreme conditions • Largest ecosystem of developers, vendors, and users • From the smallest system to the largest, there is no other networking technology more suited, or best understood

Flash Memory Summit 2018 Santa Clara, CA 22