Storage over PCIe protocol analysis and traffic generation techniques Isaac Livny Field Applications Engineer Teledyne LeCroy Corporation 1 Agenda . Storage over PCI Express® architecture • SATA Express / AHCI • SCSI Express / SOP – PQI • NVM Express . Command queue generation example . Emulating an SSD controller . Emulating an SSD host . Command Validation 2 Layered protocols analysis over PCIe as transport mechanism 3 Layered Protocols Support in Analysis Tools . Hierarchical view display capability with multi layer expansion into sub-layers . Multi-view capabilities . Processing capability of upper layer through scripting to adopt to specification changes . Tooltip feature to highlight specification details . Performance and statistical analysis per instruction, by segment and overall trace . Compacting of repetitive traffic . Compacting of multiple 32 bit transactions into 64 bit upper layer commands 4 Agenda . Storage over PCI Express® architecture • SATA Express / AHCI • SCSI Express / SOP – PQI • NVM Express . Command queue generation example . Emulating an SSD controller . Emulating an SSD host . Command Validation 5 SATA Express 6 ATA command over Serial ATA Read DMA 7 SATA Express . The Serial ATA International Organization (SATA-IO) developed the specification ATA command layer AHCI . This protocol combines the SATA AHCI software specification with the Transaction PCIe host interface Link . SATA Express enables new devices Physical to be developed that utilize the faster Logical PCIe interface and maintain compatibility with a broad base of Electrical existing SATA applications . Data Rate Support • PCIe 2.x at x2 link for 8GT/s data rate • PCIe 3.0 at x2 link for a 16GT/s data rate 8 AHCI HBA Registers HBA Memory Registers 1. Port Control 2. Generic Host Control(GHC) AHCI PCIe Configuration Space Registers CAP: Host Capabilities GHC: Global Host Control IS: Interrupt Status PI: Ports Implemented 9 SATA Express Port Control Setup Port control address of Command list setup Port control address of Received FIS setup 10 AHCI System Memory Host Port 0-31 Control System Registers Port 0 Each Port CTL Register points to two sections of System Memory Memory PxFB: FIS Base Addr PxCLB: Cmd List Base Address PxFBU: FIS Base Addr Up32b PxCLBU: CLB Addr Up32b Command 0 Command Command 1 Table Received Command 2 Address FIS Structure Physical Region Descriptor(PRD) Table Address Command 31 Data payload is sent and received through Command List the PRDT (Physical Region Descriptor Structure Table) in Command List 11 ATA command: Set Features 12 READ FPDMA QUEUED 13 ATA Command: READ BUFFER 14 Agenda . Storage over PCI Express® architecture • SATA Express / AHCI • SCSI Express / SOP – PQI • NVM Express . Command queue generation example . Emulating an SSD controller . Emulating an SSD host . Command Validation 15 PCIe Architecture Queuing Interface (PQI) 16 SAS Hierarchical View SCSI Command Decode Copyright © 2013, PCI-SIG, All Rights Reserved 17 SCSI Over PCI Express(SOP/SOX) . Developed by T10 Committee . Compliance with SCSI SCSI Over PCIe Architectural Model PQI . Proposed support for SOP Transaction target ports interfacing to flash Link devices, RAID controllers, and Physical Logical other SCSI peripheral device types Electrical . Targeting PCIe 3.0 Preliminary Protocol Stack 18 SCSI Express SOP/PQI IQ(Inbound Queue) PQI Host Elements are written to the element array by a First element in producer and read from the element array by a the array consumer 1 Last element in 0 the array n-1 IQ Local Working Copy of Copy of IQ Pl IQ Pl IQ CI PCIe Link Inbound Working Local IQ PI IUs Copy of Copy of PQI Device IQ Cl IQ Cl PQI Device 19 SCSI Express SOP/PQI OQ(Outbound Queue) PQI Host Elements are written to the element First element in the array array by a producer and read from the element array by a consumer 1 Last element in 0 the array OQ Working Local Copy of Copy of OQ PI OQ Cl OQ Cl PCI Express Link Outbound Local Working IUs OQ CI Copy of Copy of OQ Pl OQ Pl PQI Device 20 Creating Administrative and Operational Queues Creating an Operational Inbound Queue 1 1 0 0 n-1 IQ n-1 IQ Response Creating an Operational Outbound Queue 1 1 0 0 n-1 OQ n-1 OQ Response Admin Queue Operational Queue 21 SCSI Express Initialization Creating Administrative IQ and OQ Queues Creating Operational OQ Queue 22 SCSI Express SOP Transfer Packet Advancing Producer in Inbound Queue Advancing Consumer in Inbound Queue Advancing Producer in Outbound Queue Courtesy SanDIsk 2012 Advancing Consumer in Outbound Queue 23 PQI initialization 24 PQI: Create Operational Queues 25 PQI: Create Op Queue detailed 26 SCSI command: WRITE (10) 27 SCSI command: Report LUNS 28 Agenda . Storage over PCI Express® architecture • SATA Express / AHCI • SCSI Express / SOP – PQI • NVM Express . Command queue generation example . Emulating an SSD controller . Emulating an SSD host . Command Validation 29 NVM Express 30 NVM Express . The NVMHCI Workgroup released the NVM Express 1.0 specification on March 1, NVM Express 2011 and is available at Transaction www.nvmexpress.org Link . NVMe is a standardized high Physical performance queuing Logical interface and command set Electrical optimized for PCIe SSDs . NVMe is scalable from client to enterprise applications 31 Submission and Completion Queues Submission Completion Queue Queue Tail Head Pointer Host Memory Tail Completion Head Queue Submission Queue Head Tail Submission Queue Head Pointer 32 Circular Queuing Interface QueueQueue ProcessProcess Command Host 7 CompletionCompletion 1 Submission Completion Queue Host Memory Tail Queue Head Ring Doorbell Ring Doorbell New Tail New Head 2 PRP 8 Head Tail . PCIe TLP . PCIe TLP PCIe TLP PCIe TLP PCIe TLP PCIe TLP PCIe TLP PCIe TLP PCIe TLP. PCIe TLP PCIe TLP . Submission Queue 3 4 Completion Queue Tail Doorbell 5 6 Head Doorbell Fetch Process Queue Generate Command Command Completion Interrupt NVMe Controller 33 Viewing the NVM Express 1.0c Initialization Phase Physical Region Page pointer Set submission and completion queues base addresses Controller capabilities Courtesy SanDIsk 2012 Command Completion Capabilities sent to Host memory 34 Viewing the NVM Express 1.0c Identify Namespace Capabilities command Namespace capabilities Courtesy SanDIsk 2012 Command Completion 35 Viewing the NVM Express 1.0c Creating an I/O queue and Read command example Create IO Submission Queue Courtesy SanDIsk 2012 Physical Region Page pointer Data Transfer Command Completion Courtesy SanDIsk 2012 Data 36 Viewing the NVM Express 1.0c NVMe Multiple Pointer Based Transactions Physical Region Page(PRP) PRP List of pointers to memory addresses Data 37 Agenda . Storage over PCI Express® architecture • SATA Express / AHCI • SCSI Express / SOP – PQI • NVM Express . Command queue generation example . Emulating an SSD controller . Emulating an SSD host . Command Validation 38 NVMe Device script Submission Queue Setup . The host creates a command for execution # NVM 0 within the appropriate Submission Queue Wait=TLP { TLPType=MWr32 . Admin submission queue base register Length = 1 } 0x7F55A000 is written to controller register, #NVM 1 later used by controller to fetch the read Wait=TLP { command from host memory TLPType=MWr32 Length = 2 } 39 NVMe Device script Command fetch . The controller fetches the packet = TLP { command(s) in the Submission TLPType = MRd64 Queue from host memory Length = 0x10 . RequesterId = (8:0:0) The Admin submission queue Tag = 0x0 base address register LastDwBe = 0xF FirstDwBe = 0xF 0x7F55A000is read from AddressHi = ( FROM_MEM32_A, 0x28 ) implemented memory resource # 0x1 through field substitution AddressLo = ( FROM_MEM32_A, 0x2C ) # 0xEF224000 } 40 Agenda . Storage over PCI Express® architecture • SATA Express / AHCI • SCSI Express / SOP – PQI • NVM Express . Command queue generation example . Emulating an SSD controller . Emulating an SSD host . Command Validation 41 Testing NVM Express . NVMe Registers emulation • Setup Admin Queue • Doorbell Registers . Admin Commands • Delete I/O Submission Queue • Create I/O Submission Queue • Delete I/O Completion Queue • Create I/O Completion Queue • Get Log Page • Identify • Abort • Set Features Summit Z3-16 Protocol Exerciser • Get Features • Format NVM • Extensible for Vendor Specific Commands 42 Pre-Silicon Device emulation Summit Z3 RTL Design RTL Test Vectors SimPass CATC Trace Summit T3 PE Tracer Export to Generation script Z3 Exerciser Generation Script 8/2/2014 Company Confidential 43 Loading Config Space and Implementing Memory Space 44 SSD Drive Emulation 45 Setting up Controller Registers Submission queue size Submission queue BAR Completion queue BAR Enable doorbell execution Submission queue tail doorbell 46 Identify Command Execution 47 Identify Command Execution System Memory command NVMe Controller registers Address Data Register Address Data ASQB 0x28 3D61B000 0x3D61B000 0x3D2DA160 NVMe Device memory space System Memory Data Address Data Address Data 0x3D2DA160 0x70157015 0x3D61B000 3D2DA160 48 Emulated NVMe Device Shown in Device Manager 49 Emulated Drive Shown in Disk Management 50 Agenda . Storage over PCI Express® architecture • SATA Express / AHCI • SCSI Express / SOP – PQI • NVM Express . Command queue generation example . Emulating an SSD controller . Emulating an SSD host . Command Validation 51 Testing NVM Express . NVM Commands Write Read Compare Extensible for Vendor Specific Commands . Queue Management . Come up in Device Manager . Extensible Vendor Specific Features (for Get/Set Features) . Complete