NVM Express: “Memory Class” Capabilities?

Flash Drives: Block Storage or Memory? Tony Roug, Virident Director Solution Architecture

• Storage Class Memory: The Bridge • Access Models: The Architecture • Access Capabilities: The Value

Storage Class Memory (SCM) CPU CPU

Disk SATA/SAS PCIe Memory- SSDs SSDs class SCM DRAM DRAM DRAM DRAM

SATA/SAS PCIe Memory-class Attribute Disk DRAM SSDs SSD SCM

Capacity (GB) 100’s-1000’s 100’s 100’s-1000’s 100’s-1000’s 10’s-100’s

Read 10’s ms, 100’s us, 5010 –’ s75 us, us, 200100 – ’400s ns, ns, performance ~100 MB/s ~100’s MB/s 21 –’s 4GB/s GB/s 21 ’–s 5 GB/s GB/s ~100 ns, Write 10’s ms 100’s us ~25~10 –’ s50 us, us, 1~5’s us, 10’s GB/s performance ~100 MB/s ~100 MB/s ~0.5~1 ’–s 1+GB/s GB/s 12’-s5 GB/sGB/s

PCIe SSDs established, Memory-class emerging

• Storage Class Memory: The Bridge • Access Models: The Architecture • Access Capabilities: The Value

Datacenter SCM-aware Applications cache cache SCM Applications opt opt cache

File File SCM File SCM Direct Direct System System System Direct OS

Block Drivers Block Drivers SCM Drivers

Storage SATA, SAS, FC PCIe Hardware PCIe, Memory

Optimization required for applications to realize memory class benefit

Datacenter flashcach SCM-aware Applications cache cache SCM Applications e opt opt cache bcache File File SCM SNIAFile NVMeSCM Direct Direct System System ProgrammingSystem TWGDirect OS NVMExpres

s Block Drivers Block Drivers NVMExpressSCM Drivers SCSIExpres

s Storage SATA, SAS, FC Hardware PCIe SFF -8639 PCIe, Memory

Industry agreement for optimized architecture in place

• Storage Class Memory: The Bridge • Access Models: The Architecture • Access Capabilities: The Value

• File open and close, file read and write, raw read and write User Block IO • Multiple threads through libaio libaio O_Direct File System • Direct DMA from user to board/board to user through o_direct • CLI: monitor, diag, beacon, secure erase Kernel Block IO

Kernel Level Kernel Block IO Virtual to • block register: register, unregister, open, release, ioctrl MetaData Wear Level • block transfer: request, queue Physical • DMA scatter/gather: blk_rq_map_sg Rate Limit DMA Setup GC • rate limit: guaranteed minimum performance at 100% capacity • wear leveling: identification/adaptation to app access patterns PCIe I/F

Board Level PCIe I/F • partition options: 512 and 4K sector size DMA Engines Req/Resp Queues • write acceleration mode: maxperformance, maxcapacity RAID 5 • reliability option: raid5 enabled/disabled SRAM ECC SRAM ECC Flash I/F … Flash I/F FLASH FLASH FLASH FLASH

PCIe SSD optimizations within a block level interface

9 NVM Express: “memory class” capabilities?

Function NVMExpress 1.0 Capability Block Compatibility Basic Features Read, Write Read, Write Flush Sync Cache Format Format ACID Fused Operations Compare and Write Operations Atomic Write Unit New ACID Capabilities Write Atomicity Performance, Queues/PRP Endurance, Data Set Management: Incompressible, Access Size, New Performance and Efficiency Write Prepared, Sequential Read/Write, Access Freq, Features Access Latency Capabilities Compare Namespaces Deallocate TRIM/UNMAP Power Management Start Stop Data End-to-end Protection + Metadata DIF/DIX protection Format Secure Erase Secure Erase Write uncorrectable Write Long Write protect WRPROTECT

1. streamline the application access path: e.g. RDMA access SCM-aware 2. offer improved semantics: e.g. atomic-write SCM Applications 3. simplify caching implementations: e.g. key-value cache store, User 4. construct cluster-level solutions: e.g. synchronize cross-server activities and SCM File SCM orchestrate server to SAN data movement System Direct 5. emulate memory: e.g. mmap and malloc 6. expose internals of flash management: e.g. Kernel optimized file system with snapshot, cross-card data transfers, trading capacity vs performance NVM Express Driver

NVM Express

Create industry standard capabilities enabling memory class access

• Storage Class Memory: • First level value enabled by block mode • Access Models: • NVM Express enables memory class usage models • Access Capabilities: • Industry agreement on application enabling technology launched • SNIA Programming TWG

Tony Roug Phone: 4085829647 Email: [email protected]