NVMe® Technology Powering the Connected Universe Amber Huffman Fellow & Chief Technologist of IP Engineering Group, Intel Corporation President, NVM Express, Inc. Agenda
Fixing the Memory & Storage Hierarchy
Refactoring for the Next Decade of Growth
NVMe® Architecture Advancements
What’s Next: Computational Storage
Intel Corporation 2 Agenda
Fixing the Memory & Storage Hierarchy
Refactoring for the Next Decade of Growth
NVMe® Architecture Advancements
What’s Next: Computational Storage
Intel Corporation 3 Intel Corporation 4 Memory and Storage Hierarchy Gaps
COMPUTE 100s MB, ~1ns CACHE
IN- PACKAGE 1s GB, ~10ns MEMORY
MEMORY 10s GB, <100ns Memory 100s GB, <1usec CAPACITY GAP
STORAGE PERFORMANCE GAP 1s TB, <10µsecs
10s TB, <100µsecs Storage SECONDARY STORAGE COST- PERFORMANCE GAP
TERTIARY STORAGE 10s TB, <10 msecs
Intel Corporation 5 Memory and Storage Hierarchy Gaps
COMPUTE SRAM CACHE Bandwidth
IN- PACKAGE DRAM (HBM) MEMORY
MEMORY DRAM
3D NAND Flash
SECONDARY STORAGE HDD
TERTIARY STORAGE
Capacity For illustrative purposes only
Intel Corporation 6 Types of Memory
DRAM 3D XPOINT INTEL 3D NAND
One DRAM memory cell = 1 bit One 3D XPoint memory cell = 1 bit One 3D NAND memory cell = 1-4 bits
Intel Corporation 7 Types of Memory Compared
DRAM 3 D XPOINT INTEL 3 D NAND
One DRAM memory cell = 1 bit One 3D XPoint memory cell = 1 bit One 3D NAND memory cell = 1-4 bits
Intel Corporation 8 3D NAND Roadmap
128L Industry Target
96L QLC Gen 3
64L QLC Gen 2
32L TLC Gen 1 128L
LAYERS 96L 64L
32L
2016 2017 2019 2020
Intel Corporation 9 3D NAND Roadmap 144L QLC Gen 4
~50% Bit Density 96L QLC Increase Gen 3 ~50% 64L QLC Bit Density Gen 2 Increase
32L TLC 133% Gen 1 Bit Density Increase 144L
LAYERS 96L 64L
32L
2016 2017 2019 2020
Intel Corporation 10 3D XPoint Memory Roadmap
2nd Gen 4-Deck
1 st Gen 2 - Deck Multiple Millions of IOPS on 2nd Generation Intel® Optane™ SSD
2017 2020*
*Target Production
Intel Corporation 11 Media Innovation Realized in Product
Intel Optane™ SSD Innovation Powered by
Intel Optane ™ Memory H10 Optane + 3D QLC NAND Fast
Simple Intel 3D NAND SSD Scalable
Intel Corporation 12 Memory and Storage Hierarchy Gaps
COMPUTE SRAM CACHE
Bandwidth Rambo Cache
IN- PACKAGE DRAM MEMORY
DRAM Intel Optane™ Persistent Memory MEMORY
PERSISTENT Intel Optane™ SSDs MEMORY
PERFORMANCE 3D NAND SSDs STORAGE
4-Deck 3D XPoint SECONDARY STORAGE HDD
QLC 3D NAND TERTIARY STORAGE
Capacity For illustrative purposes only Courtesy 2020 Intel Architecture Day
Intel Corporation 13 Memory and Storage Hierarchy Gaps Solutions
COMPUTE SRAM CACHE
Bandwidth Rambo Cache Innovation Powered by IN- PACKAGE DRAM MEMORY
DRAM Intel Optane™ Persistent Memory MEMORY
PERSISTENT Intel Optane™ SSDs MEMORY
PERFORMANCE 3D NAND SSDs STORAGE
4-Deck 3D XPoint SECONDARY STORAGE HDD
QLC 3D NAND TERTIARY STORAGE
Capacity For illustrative purposes only Courtesy 2020 Intel Architecture Day
Intel Corporation 14 NVMe® Technology Powers the Connected Universe
Average Capacity (GB) Units (Ku) 2016 2017 2018 2019 2020* 2021* 2016 2017 2018 2019 2020* 2021* 6000 Enterprise 364 749 1,069 2,045 4,067 5,554 4000 Cloud 2,051 3,861 10,369 12,276 18,982 21,999 2000
Client 33,128 48,951 82,587 143,236 202,348 258,791 0 Client Enterprise Cloud * Projections provided by Forward Insights Q2’20
. NVMe technology grew from 3 Petabytes to 29 PB shipped per year from 2016 to 2019 . For 2020, the projection is 54 PB . NVMe technology demand projected to remain strong in a post COVID world
Intel Corporation 15 Agenda
Fixing the Memory & Storage Hierarchy
Refactoring for the Next Decade of Growth
NVMe® Architecture Advancements
What’s Next: Computational Storage
Intel Corporation 16 The Evolution of NVMe® Technology
ENABLE Deployments INNOVATION
SCALE OVER FABRICS
UNIFY PCIE* SSDS
2010 2020
Intel Corporation 17 Driving Simplicity in a World of Complexity
Zoned Namespaces
NVMe/New Original NVMe Technology Key Value Base Spec TOO DEPENDENT TOO NVMe over COMPLEX Fabrics Spec Computational ON HUMANNVMe/TCP GLUE Storage More expansion NVMe/RDMA
Intel Corporation 18 Computational Driving Simplicity in Key Value a World of Storage Complexity
Block I/O Zoned NVMe Namespaces . Back to core values… Fast, Simple, Scalable Base Specification with Fabrics . Foster areas of innovation AND avoid impact to broadly deployed solutions . Create an extensible infrastructure that will take us through the next decade of growth NVMe/PCIe* NVMe/TCP
NVMe/RDMA
Intel Corporation 19 Specification Families
THE CORE COMMAND SETS TRANSPORT LAYERS
. The core of NVMe and NVMe over Fabrics integrated into a base specification . Modular command set specifications (Block, Zoned Namespaces, Key Value, etc) . Modular transport layer specifications (PCI Express*, RDMA, TCP) . Maintain Management Interface as separate modular specification
Intel Corporation 20 Agenda
Fixing the Memory & Storage Hierarchy
Refactoring for the Next Decade of Growth
NVMe Architecture® Advancements
What’s Next: Computational Storage
Intel Corporation 21 NVM Express Technology Specification Roadmap
2015 2016 2017 2018 2019 2020 2021
Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4
NVMe 2.0* NVMe 1.3 May’17 NVMe 1.4 June’19 NVMe NVMe 1.2.1 May’16 Spec • Sanitize • IO Determinism (NVM Sets) NVMe Base Spec • Streams • Persistent Event Log, Rebuild Assist • Virtualization • Persistent Memory Region (PMR) • Asymmetric Namespace Access (ANA) • Merged w/Fabrics • Namespace Types • Alternate Cmd Sets NVMe-oF 1.0 May’16 NVMe-oF NVM-oF 1.1 July’19 NVMe Transport • Transport and protocol • Enhanced Discovery Spec(s) Spec • RDMA binding • TCP Transport Binding NVMe Cmd Set Spec(s)
NVMe-MI 1.2 Spec NVMe-MI 1.0 Nov’15 NVMe-MI 1.1 May’19 NVMe-MI • Out-of-band management • Device discovery • Enclosure Management Spec • Health & temp monitoring • In-band Mechanism • Firmware Update • Storage Device Extension
Released NVMe specification Planned release
Intel Corporation 22 Namespace Types . Discover Command Set supported for a Namespace Enable Alternate . Command Sets: Block I/O, Key Value, Zoned, < future > . Extensible approach for future innovation Command Sets
NVMe NVMe NVMe NVMe I/O Command Set #1 Controller 0 Controller 1 Controller 2 Controller 3
I/O Command Set #2 NSID 1 NSID 2 NSID 1 NSID 2 NSID 1 NSID 2 NSID 1 NSID 2 NSID 3
NS NS NS NS NS NS NS NS NS I/O Command Set #3 A B C D E F G H I
NVM Subsystem
Intel Corporation 23 Zoned Zone 0 Zone 1 ... Zone x-1 Namespaces LBA LBA LBA LBA LBA LBA LBA LBA LBA LBA LBA LBA
. NVMe ® technology evolving to address underlying media changes with larger Erase Blocks and more . Zoned Namespaces require Logical Blocks to be written sequentially in a Zone . Reduces write amplification and overprovisioning
Intel Corporation 24 Endurance Media Controller Groups Chan 0 Chan 1 Chan 2 Chan 3
NVM Set NVM Set NVM Set NVM Set Media Media Media Media Unit Unit Unit Unit . Flexible capacity management model Media Media Media Media . Create appropriate groupings based on access Unit Unit Unit Unit
pattern, media type(s), and more Media Media Media Media Unit Unit Unit Unit
Media Media Media Media Unit Unit Unit Unit
Media Media Media Media Unit Unit Unit Unit
Media Media Media Media Unit Unit Unit Unit
Endurance Group Endurance Group Endurance Group
Intel Corporation 25 Domains and Partitions
Port v Port w Port x Port y Port z
Domain 1 Communication Communication Domain 3 Communication Domain 5 Communication boundary boundary boundary boundary
Fabric Fabric Fabric Fabric Fabric NVMe Controller NVMe Controller NVMe Controller NVMe Controller NVMe Controller
NSID 1 NSID 2 NSID 3 NSID 2 NSID 2 NSID 4 NSID 4 NSID 2 NSID 4 NSID 2
NS NS A C
NS NS D E NS Domain 2 B Domain 4 Domain 6
. Supports subdividing LARGE scale solutions – enabling partial operations & maintenance flows
Intel Corporation 26 Building Highways for Client, Cloud, & Enterprise
INFRASTRUCTURE ENHANCEMENTS Simple Copy Command Command Group Control Feature Controller Memory Buffer Write Elasticity Status Namespace Attachment Limits Multiple Controller Firmware Update Telemetry Enhancements Command and Effect Log Enhancements Non-”Maximum Data Transmit Size” Command Size Limits
. Use cases for NVMe® architecture continue to expand across Client, Cloud, and Enterprise . Enhancements in Management, Telemetry and large storage systems meet the growing needs
Intel Corporation 27 Agenda
Fixing the Memory & Storage Hierarchy
Refactoring for the Next Decade of Growth
NVMe® Architecture Advancements
What’s Next: Computational Storage
Intel Corporation 28 Compressed, Encrypted, Stored Table Arbitrary Format
The Complex NAME STATE Bruce Banner NH Database Universe Bruce Wayne NY Diana Prince NJ Jean Grey NH . Data Warehouses (Presto, SparkSQL, AWS Jessica Jones NY Redshift AQUA, …) store LOTS of data Peter Parker NY . Data stored in LOTS of (arbitrary) formats Selina Kyle CA . Data stored compressed and encrypted Tony Stark CA . Formats and data constantly evolving
Intel Corporation 29 Finding the Needle in the Haystack
FILTER Select Name where State=“NY” Header metadata data Bruce Banner Bruce Wayne Tuple Set Diana Prince metadata Bruce Wayne data Jean Grey Jessica Jones 3 decrypt decompress Jessica Jones filter aggregate ‘names’ ‘names’ ‘names’ Peter Parker (count) Tuple Set Peter Parker metadata data Selina Kyle Tony Stark Tuple Set Footer
Compressed, Encrypted, Arbitrary Format
Intel Corporation 30 Programs as Computational Storage Offloads
3
2 . Programs invoked and used in standard way
. Programs in hardware agnostic bytecode and 1 Fixed Program downloaded from host for later execution 0 eBPF Program . Device may offer fixed function programs . Programs operate on data in on-device memory Program Slots
Computational Saving Power Program Memory Namespace NVMe Controller Increasing Performance
Intel Corporation 31 The Evolution of NVMe® Technology
ENABLING Deployments INNOVATION
SCALE OVER FABRICS
UNIFY PCIE* SSDS
2010 2020
Intel Corporation 32 Intel Corporation 33