(Nvme-MI™) and Drivers Update
Total Page:16
File Type:pdf, Size:1020Kb
Architected for Performance NVMe™ Management Interface (NVMe-MI™) and Drivers Update Sponsored by NVM Express® organization, the owner of NVMe™, NVMe-oF™ and NVMe-MI™ standards Speakers Uma Parepalli Austin Bolen Myron Loewen Lee Prewitt Dave Minturn Suds Jain Jim Harris 2 Agenda • Session Introduction – Uma Parepalli, Marvell (Session Chair) • NVMe Management Interface - Austin Bolen, Dell EMC and Myron Loewen, Intel • NVMe Driver Updates • NVMe Driver Ecosystem and UEFI Drivers - Uma Parepalli, Marvell • Microsoft Inbox Drivers - Lee Prewitt, Microsoft • Linux Drivers - Dave Minturn, Intel • VMware Drivers -Suds Jain, VMWare • SPDK Updates - Jim Harris, Intel 3 Architected for Performance NVMe™ Management Interface (NVMe-MI™) Workgroup Update Austin Bolen, Dell EMC Myron Loewen, Intel Agenda . NVMe-MI™ Workgroup Update . NVMe-MI 1.0a Overview . What’s new in NVMe-MI 1.1 . In-band NVMe-MI . Enclosure Management . Managing Multi NVM Subsystem Devices . Summary 5 NVM Express®, Inc. 120+ Companies defining NVMe™ together 6 NVM Express™ Roadmap 7 NVMe-MI™ Ecosystem . Commercial test equipment for NVMe-MI . NVMe-MI 1.0a compliance testing program has been developed . Compliance testing started in the May 2017 NVMe™ Plugfest conducted by the University of New Hampshire Interoperability Laboratory (UNH-IOL) . 7 devices from multiple vendors have passed compliance testing and are on the NVMe-MI Integrators List . Servers are shipping that support NVMe-MI 8 What is the NVMe™ Management Interface 1.0a? A programming interface that allows out-of-band management of an NVMe Storage Device Field Replaceable Unit (FRU) 9 Out-of-Band Management and NVMe-MI™ . Out-of-Band Management – Management that operates with hardware resources and components that are independent of the host operating system control Host Processor Management Controller (BMC) Host Operating System BMC Operating System . NVMe™ Out-of-Band Management Application Application BMC Operating System Interfaces NVMe Driver NVMe-MI Driver . SMBus/I2C PCIe Root PCIe Root PCIe Port SMBus/I2C . PCIe Vendor Defined Messages (VDM) Port Port PCIe PCIe VDM SMBus/I2C Bus PCIe Bus PCIe Port SMBus/I2C NVMe NVM Subsystem 10 NVMe-MI™ Out-of-Band Protocol Layering Management Applications (e.g., Remote Console) Application Management Controller Layer (BMC) Protocol NVMe Management Interface Management ManagementLayer Applications (e.g., Applications (e.g., Remote Console) Management Component Transport Protocol (MCTP) Remote Console) Transport MCTP over MCTP over Layer SMBus/I2C Binding PCIe VDM Binding Physical SMBus/I2C PCIe Layer PCIe SSD 11 NVMe™ Storage Device in 1.0a VPD . NVMe Storage Device – One NVM Subsystem with one or more ports, vital product data (VPD), and an optional SMBus/I2C interface 12 In-Band Management and NVMe-MI™ Host Processor Management Controller (BMC) . In-band mechanism allows application to tunnel Host Operating System BMC Operating System NVMe-MI commands through NVMe™ driver Application Application . Two new NVMe Admin commands BMC Operating System – NVMe-MI Send NVMe Driver NVMe-MI Driver – NVMe-MI Receive . Benefits PCIe Root PCIe Root PCIe Port SMBus/I2C Port Port . Provides management capabilities not available in-band via NVMe commands PCIe PCIe VDM SMBus/I2C – Efficient NVM Subsystem health status Bus PCIe Bus reporting PCIe Port SMBus/I2C – Ability to manage NVMe at a FRU level – Vital Product Data (VPD) access NVMe NVM Subsystem – Enclosure management 13 NVMe-MI™ over NVMe-oF™ Host Processor Management Controller (BMC) Host Operating System BMC Operating System Application Application BMC Operating System Flash Appliance Fabric Fabric NVMe-oF NVMe Software NVMe-MI Driver PCIe Root PCIe Root PCIe Port SMBus/I2C Port Port PCIe PCIe VDM SMBus/I2C Bus PCIe Bus PCIe Port SMBus/I2C NVMe NVM Subsystem Plumbing in place for NVMe-MI over NVMe-oF 14 Enclosure Management ... NVMe Enclosure . SES Based Enclosure Management Power NVM Subsystem Supplies NVMe Mgmt. Technical proposal developed in NVMe-MI™ Controller Ep. Cooling Objects workgroup Cntrl. Mgmt Intf. Temp. While the NVMe™ and SCSI architectures differ, the Sensors elements of an enclosure and the capabilities required Enclosure Other to manage these elements are the same Services Process Objects – Example enclosure elements: power supplies, fans, Slot Slot Slot Slot display or indicators, locks, temperature sensors, current sensors, voltage sensors, and ports NVMe NVMe NVMe NVMe Storage Storage Storage ... Storage Device Device Device Device . Comprehensive enclosure management that leverages SCSI Enclosure Services (SES), a standard developed by T10 for management of enclosures using the SCSI architecture 15 Multi NVM Subsystem Management 16 NVMe-MI™ 1.0a NVMe™ Storage Device . NVM Storage Device – One NVM Subsystem with one or more ports and an optional SMBus/I2C interface VPD Single Ported PCIe SSD Dual Ported PCIe SSD with SMBus/I2C 17 NVMe™ Storage Device with Multiple NVM Subsystems M.2 Carrier Board from Amfeltec ANA Carrier Board from Facebook PCIe SSD PCIe Switch NVM NVM NVM NVM Subsystem Subsystem Subsystem Subsystem NVM NVM PCIe SSD Subsystem Subsystem 18 SMBus Topology for NVMe-MI™ 1.0 Host 3Eh Subsystem 1 A6h VPD 19 19 Multiple NVM Subsystems on a single SMBus Port • Describe topology in new VPD MultiRecord • Add UDID types for additional devices like Mux Host Host ARP E8h 3Ah Subsystem 1 Subsystem 1 ARP 3Ah A6h Subsystem 2 A6h Subsystem 2 Mux VPD ARP VPD 3Ah Subsystem 3 Subsystem 3 ARP 3Ah Subsystem 4 Subsystem 4 20 20 Support Expansion Connectors • New VPD address to avoid conflicts with plugged in devices • Optional Labels for each connector to assist technicians Host Host ARP E8h 3Ah Connector 1 Connector 1 ARP 3Ah A4h Connector 2 A4h Connector 2 Mux VPD ARP VPD 3Ah Connector 3 Connector 3 ARP 3Ah Connector 4 Connector 4 21 21 A Connection Graph Between Element Types NVMe™ Subsystem PCIe Host Switch PCIe NVMe Subsystem Expansion Host Connector Power Mux SMBus SMBus Expansion Connector 22 Single Port Example (35 bytes of 256B EEPROM) CPU BMC Record: Record Record Record Header Version Rsvd Element Header Element Format Length Chcksm Chcksm Number Count 0Dh 82h 23h 34h 75h 00h 00h 03h Root Type: Element Form SMBus Link Link 0 Link 0 Link 0 Element Host Length Factor Dest Options Width Start Dest. complex 0 01h 08h 12h 02h 00h 84h 00h 02h Type: Element Thermal Vaux Rail Rail 12V 12V Element Power Length Load Load Options Voltage initial max 1 02h 08h 0Fh 32h 00h 78h 08h 0Fh NVM Type: Element MCTP SMBus PCIe Port 0 Port 0 Total NVM Capacity Element NVMe Length Address speed Ports Speed Flags (MSB first) Subsystem 2 09h 13h 3Ah 01h 12h 0Fh 01h 000000000000000000000000h MCTP VPD 23 23 2 NVM Subsystems Dual Port with Expansion with Mux (82) Connectors (78) CPU BMC CPU CPU BMC Root Root Root complex complex complex VPD VPD PCIe PCIe PCIe Switch Mux Switch Switch Bay Bay Bay 1 2 3 NVM NVM NVM NVM NVM Subsystem Subsystem Subsystem Subsystem Subsystem MCTP MCTP VPDMCTP VPDMCTP VPDMCTP 24 Summary . NVMe-MI™ 1.0a is gaining market acceptance and is available in shipping products . NVMe-MI 1.1 is nearing completion . Significant new features – In-band mechanism – Enclosure management – Support for multi NVM subsystem management . It is time to start thinking about anchor features for NVMe-MI 1.2 25 Additional Material on NVMe-MI™ • BrightTALK Webinar o https://www.brighttalk.com/webcast/12367/282765/the-nvme-management-interface-nvme-mi-learn-whats-new • Flash Memory Summit 2017 o Slides: https://www.flashmemorysummit.com/English/Collaterals/Proceedings/2017/20170808_FA12_PartA.pdf o Video: o https://www.youtube.com/watch?v=daKL7tIvNII o https://www.youtube.com/watch?v=Daqj-XqlCo8 • Flash Memory Summit 2015 o Slides: https://www.flashmemorysummit.com/English/Collaterals/Proceedings/2015/20150811_FA11_Carroll.pdf • Flash Memory Summit 2014 o Slides: https://www.flashmemorysummit.com/English/Collaterals/Proceedings/2014/20140804_SeminarF_Onufryk_Bolen.pdf • NVMe-MI Specification o https://nvmexpress.org/resources/specifications/ 26 References MCTP Overview: http://dmtf.org/sites/default/files/standards/documents/DSP2016.pdf MCTP Base Spec: https://www.dmtf.org/sites/default/files/standards/documents/DSP0236_1.3.0.pdf MCTP SMBus/I2C Binding: https://www.dmtf.org/sites/default/files/standards/documents/DSP0237_1.1.0.pdf MCTP PCIe VDM Binding: https://www.dmtf.org/sites/default/files/standards/documents/DSP0238_1.0.2.pdf IPMI Platform Management FRU Information Storage Definition: https://www.intel.la/content/www/xl/es/servers/ipmi/ipmi-platform-mgt-fru-infostorage-def-v1-0-rev-1-3- spec-update.html 27 Architected for Performance Architected for Performance UEFI NVMe™ Drivers Update Uma Parepalli, Marvell NVMe™ Driver Ecosystem Robust drivers available on all major platforms 30 NVM Express® Website – Drivers Home Page 31 UEFI NVMe™ Drivers – What is new • UEFI drivers available for a while on Intel platforms. • ARM processor based systems now have built-in NVMe specification compliant UEFI driver and boot to Windows and Linux Operating Systems. 32 Linux NVMe™ over Fabrics Drivers Supporting NVMe over RDMA, Fibre Channel, TCP and iWARP NVMe Host Software NVMe Host Software RDMA Transport Fibre Channel Transport iWARP InfiniBand RoCE Fibre Channel RDMA Fabric FC Fabric iWARP InfiniBand RoCE Fibre Channel RDMA Transport Fibre Channel Transport NVM Subsystem NVM Subsystem Marvell (former Cavium) contributed the NVMe-oF™ Drivers to the Linux Upstream 33 33 Architected for Performance Architected