Accelerating Innovation Through Next Generation Storage Technologies REBECCA WEEKLY SR

Accelerating innovation through next generation storage technologies REBECCA WEEKLY SR. PRINCIPAL ENGINEER & SR. DIRECTOR CLOUD PLATFORMS GROUP Source: April 2019 Raconteur Data Gravity: Challenge or Opportunity? Storage for the Data Challenge

Devices | Things Access | Edge Core Data Center | Cloud Storage Technology Trends

In Market

NVMe Storage

NVMe over Fabrics

Computational Storage Data Storage Paradigms

Block Storage File Storage Object Storage

1 0 1 1 1 0 0 1 0 0 0 0 1 1 0 1 0 1 0 0 1 0 1 1 0 1 0 1 0 1 1 0 0 1 1 0 0 1 0 0 1 0 1 0 0 1 0 1

1 0 1 1 1 0 0 1 0 0 0 0 1 1 0 1 0 1 0 0 1 0 1 1 0 1 0 1 0 1 1 0 0 1 1 0 0 1 0 0 1 0 1 0 0 1 0 1 1 0 1 1 1 0 0 1 0 0 0 0 1 1 0 1 1 0 1 1 1 0 0 1 0 0 0 0 1 1 0 1 1 0 1 1 1 0 0 1 0 0 0 0 1 1 0 1 0 1 0 0 1 0 1 1 0 1 0 1 0 1 1 0 0 1 0 0 1 0 1 1 0 1 0 1 0 1 1 0 0 1 0 0 1 0 1 1 0 1 0 1 0 1 1 0 0 1 1 0 0 1 0 0 1 0 1 0 0 1 0 1 0 1 1 0 0 1 0 0 1 0 1 0 0 1 0 1 0 1 1 0 0 1 0 0 1 0 1 0 0 1 0 1

• Enterprise databases • Hierarchical tree structures • Flat key / value structure • Mission critical core business • Applications write to file • Metadata provides context applications system over network (NAS) • Application keeps track of object • File system tracks state location (“URL”) (metadata) of objects (files) • Unstructured Big Data and • Active Documents Archives What Drives Storage Choice?

Unstructured Data Workloads

Capacity requirement in PB range

Data not coupled to application

Applications don’t require strong consistency

Concurrent/Distributed access to content

Granular security and multi-tenancy

Source: http://www.gartner.com/technology/reprints.do?id=1-1R78PJ9&ct=140226&st=sb00 Object Storage Workloads

REST/HTML API access Core High tolerance for latency Functions Support high concurrency Eventual consistency

VM Templates, ISO Images, etc. Specific Use Disk Volume Snapshots Cases Backup/Archive Image/Video Repository

Static Content, low change rate IO Sequential R/W Workloads Lower IOPS w/High throughput Modern Use Cases for Object Storage

Decoupling the data created from the application enables an entirely new paradigm for data management.

S3 Select Applications S3A SQL

Streaming Data Machine Learning Events S3 Select Intel® AVX-512 Disaggregate MKL-DNN Logs S3A Sensor Data Data Lake Social Media SQL Transactions Intel® AVX-512 Erasure Coding (HDFS, MinIO, RGW) S3 Select Data Processing

S3A Intel® AVX-512 Erasure Coding

Source: https://minio.io Source: https://minio.io 10 Distributed Asynchronous Object Storage

3rd Party Workflow Benefits Applications ▪ For persistent memory Apache Rich Data POSIX I/O HDF5 SQL … Arrow ▪ End-to-end OS bypass Models ▪ Low-latency I/O Storage DAOS Storage Engine Platform Open Source Apache 2.0 License ▪ True zero-copy I/O ▪ Non-blocking Data Plane Control Plane ▪ Scalable communications & I/Os

Network Libfabric TCP/IP TLS

OPA Sockets RoCE GNI Infiniband iWARP

Learn about the architecture and features of Distributed Asynchronous Object Storage (DAOS). This open source object store is based on the Persistent Memory Development Kit (PMDK) for massively distributed non-volatile memory applications. Persistent Storage for Cloud Apps

FaaS or Microservice architectures are not stateless –> Data is created or read in at many points in the architecture.

Image Source: https://microservices.io/patterns/microservices.html Optimizing for Performance and Value

Industry-Standard Server-Based Scale-out Storage Solution Intel Technology at all Clients Clients Clients levels of the stack

Client Network Intel® Xeon® SP processor- based servers HTTP Proxy Gateways Storage Performance Proxy Network Development Kit (SPDK) Persistent Memory Development Kit (PMDK) Object Storage Object Storage Object Storage Server Server Server Monitoring and Management Intel® Optane™ DCPMM Intel® Xeon SP® Intel® Xeon SP® Intel® Xeon SP® Processor Processor Processor Authentication

Intel® SSD Intel® SSD Intel® SSD Intel® SSD Data Center Family Proxy Gateway Servers Intel® Intel® Intel® Ethernet Ethernet Ethernet 10/25/40/50/100 Storage Node1 Storage Node2 Storage Node2 Scalable Cluster Framework Gigabit Intel® Ethernet Network Adapters

Replication/Cluster Network

Scale-out Performance and Efficiency - Intel® Processors

Intel® AVX-512 is designed to improve both Generational Cycle/Byte Comparison (higher is better) latency and throughput 250%

200%

Provides over 2X-3X performance boost 150% over previous generation processors for 100% storage functions: 50% 0% ▪ High speed, high bandwidth, vector pipeline integer XOR Gen (16+1) PQ Gen (16+2) Reed Solomon EC (10+4) operations 350% 300% ▪ XOR operations from ISA-L 250% 200%

▪ Hashing 150% ▪ Erasure Codes 100% 50% Intel® Xeon® Processor E5-2650v3 0% Intel® Xeon® Processor E5-2650v4 Multihash Multihash Multihash Multibuffer Multibuffer Multibuffer Multibuffer SHA-1 SHA-1 SHA-256 SHA-1 SHA-256 SHA-512 MD5 Intel® Xeon® Platinum 8180 Processor Murmur

Performance results are based on testing as of September 2019 and may not reflect all publicly available security updates. See configuration disclosure on slide 27 for details. No product can be absolutely secure. For more complete information about performance and benchmark results, visit www.intel.com/benchmarks.

Intel® Xeon® Processor E5-2600v3, E5-2650v3, 10C, 2.3 GHz, M1, Aztec City CRB, 4x8 GB DDR4 2133 MT/s ECC RDIMM. Intel® Xeon® Processor E5-2600v4, E5-2650v4, 12C, 2.2 GHz, M0, Aztec City CRB, 4x8 GB DDR4 2400 MT/s ECC RDIMM Intel® Xeon® Processor Scalable Family, Platinum 8180 Processor, 28C, 2.5 GHz, H0, Neon City CRB, 6x16 GB DDR4 2666 MT/s ECC RDIMM BIOS Configuration: P-States: Disabled, Turbo: Disabled, Speed Step: Disabled, ,C-States: Disabled, Power Performance Tuning: Disabled, ENERGY_PERF_BIAS_CFG: PERF, Isochronous: Disabled, Memory Power Savings: Disabled ISA-L 2.19 Performance and Scale: Intel Storage

MEMORY DRAM INTEL® OPTANE™ DC PERSISTENT MEMORY HOT TIER • DATA PERSISTENCE ENABLING MEMORY-CENTRIC APPLICATIONS (APP-DIRECT) • INCREASED MEMORY SIZE (MEMORY MODE) FOR PERFORMANCE PERSISTENT • IMPROVED TCO MEMORY INTEL® OPTANE™ SSDDC D4800X DUAL-PORT • PERFORMANCE + RESILIENCY FOR CRITICAL ENTERPRISE IT APPS • DUAL PORT CONNECTIONS ENABLE 24X7 DATA AVAILABILITY WITH REDUNDANT, HOT SWAPPABLE DATA PATHS STORAGE Intel® 3D Nand SSDs INTEL® SSDD-5 P4326 E1.L • COST-OPTIMIZED, ENABLES GREATER WARM STORAGE • E1.L FORM FACTOR SCALABLE TO ~1PB (IN 1U)

HDD / TAPE COLD TIER

INTEL.COM/OPTANE Compute Express Link (CXL) Enabled Computing CXL enables a more fluid and flexible memory model Single, common, memory address space across processors and devices

• Create shared memory pools CPU-attached Memory (OS Managed) Writeback PCIe DMA • Enhance movement of Memory Memory operands and results between Load/Store accelerators and target devices CPU CPU GPU FPGA AI NIC NIC

• Enable efficient resource … … … sharing Memory Load/Store • Significant latency reduction Writeback PCIe DMA to enable disaggregated Memory Accelerator-Attached Memory memory (Runtime managed cache)

CXL consortium - Currently 83 companies and growing Learn more at www.ComputeExpressLink.org Performance and Efficiency: Intel Tools

SPDK OCF PMDK Intel® VTune™ Storage Performance Development Kit Open Cache Acceleration Software Framework Persistent Memory Development Kit Amplifier

5X 5X 8X 2.2X 1 Cassandra FIO for NVME CEPH Workload, OCF 1 with OptaneCache1 With Native Persistence1 Netflix GBE

Performance results are based on testing as of September 2019 and may not reflect all publicly available security updates. See configuration disclosure on slide 29-35 for details. No product can be absolutely secure. For more complete information about performance and benchmark results, visit www.intel.com/benchmarks. Criticality of Connectivity for Storage

Inter-DC Links (DCI) PERFORMANCE Router Intel® Silicon Spine Switch Photonics

Optical Links (Interconnect) scalability Programmable End-of-Row Switches Switch (EOR)

Top-of-Rack Intel® efficiency Switch (TOR) Ethernet Rack Servers & Storage Accelerator Pools 18 All Ethernet NVMe-oF Protocols in a Single Adapter with Intel® Ethernet 800 Series

NVMe-oF* (Non-Volatile Memory Express over Fabrics)

RDMA (Remote Direct Memory Access)

Ethernet-based

iWARP* RoCE* v2 NVMe*/TCP (RDMA over Converged Ethernet (Transmission Control Protocol) ver. 2)

Intel® Omni-Path Architecture Infiniband* Fibre Channel (Intel® OPA) Future Fabrics = Supported in Intel® Ethernet 800 Series (“Columbiaville”) *Other names and brands may be claimed as the property of others. Performance and Scale: Intel Networking Application Device Queues (ADQ) Applying ADQ to NVMe/TCP An open technology designed to improve application predictability, latency and throughput

Without ADQ Application traffic intermixed with other traffic types (Lower is better)

ADQ Improvement

With ADQ Application traffic to a dedicated set of queues adq adq

Adding the Intel® Ethernet 800 Series with ADQ to NVMe/TCP narrows the performance gaps with RDMA NVMe-oF solutions linuxkernel updates NVMe/TCP FOR ADQ Released for comments Performance results are based on testing as of September 2019 and may not reflect all publicly available security updates. See configuration disclosure on slide 28 for details. No product can be absolutely secure. For more complete information about performance and benchmark results, visit www.intel.com/benchmarks. Example: Folded Clos Network Topology – Facebook

Source: http://firstclassfunc.com/facebook-fabric-networking Tiered Structure Results in more E-W BW congestion. Streaming Inference tends to be done in the same rack as the web-tier nodes to avoid congestion. Often done with batch sizes of 1-2 because folded topology prevents taking advantage of scale. Example: Fully Routed CloS – Google, Azure, IaaS trend

Provider diagram Provider diagram

Very high E-W BW and uniform latencies across any two nodes within a much larger zone. Facilitates distributed functions for inference. Eg – Delegate through RPC call to TPU node. Scale can allow streaming requests to be batched on Inference node for increased efficiency Standards bring calm to the chaos (NVMEOF, NVME OVER RDMA, NVME OVER TCP, ETC)

SNIA Board of Directors: Jim Pappas, Vice Chairman and Executive Committee Technical Council: Alan Bumgarner Solid State Storage Initiative (SSSI): Jenni Dietz, Co-Chair Solid State Drive Special Interest Group (SSD SIG): Jonmichael Hands, Co-Chair PM/NVDIMM Special Interest Group: Jim Pappas & Jenni Dietz Networking Storage Forum (NSF): Christine McMonigal NVM Programming TWG: Alan Bumgarner, Co-Chair Computational Storage TWG: Nick Adams, Co-Chair SFF TA TWG: Anthony Constantine Swordfish: Barry Kittner Architecting Storage for the Data Challenge

Devices | Things Access | Edge Core Data Center | Cloud Visit other Intel Sessions Date Time Speaker Name Room Session Title Nick Adams NVM Express Specifications: Mastering Today’s Architecture and Preparing for 9/23 8:30 am Cypress J Metz (Cisco) Tomorrow’s Jim Harris 9/23 9:30 am Cypress Squeezing Compression into SPDK Paul Luse 9/23 10:35 am Piotr Wysocki Stevens Creek Scalable Storage Management with NVMe and NVMe-oF 9/23 2:30 pm Benjamin Walker Cypress 10 Million I/Ops From a Single Thread Changpeng Liu 9/24 1:00 pm Lafayette/San Tomas Introduction of SPDK vhost FUSE Target to Accelerate File Access in VM and Containers Xiaodong Liu Alan Bumgarner 9/24 2:00 pm Stevens Creek Nonvolatile Memory Programming TWG - Remote Persistent Memory Tom Talpey (Microsoft) 9/24 3:05 pm Haodong Tang Stevens Creek Spark-PMoF: Accelerating big data analytics with Persistent Memory over Fabric Lisa Li 9/24 4:05 pm Lafayette/San Tomas A Crash-consistent Client-side Cache for Ceph Tushar Gohad 9/24 7:00 pm Fred Zhang Winchester BOF - Considerations in NVMe-oF Storage Transport Protocols 9/25 9:00 am Peter Onufryk Santa Clara Ballroom NVMe State of the Union 9/25 2:00 pm Usha Upadhyayula Stevens Creek Volatile Use of Persistent Memory 9/25 4:05 pm Nick Adams Cypress What Happens when Compute Meets Storage? – Computational Storage TWG 9/26 8:30 am Michael Strassmaier Stevens Creek Intel® Optane™ DC Persistent Memory Performance Review Andrzej Jakowski 9/26 8:30 am Lafayette/San Tomas Data-At-Rest Protection at Data Center Scale with NVMe* and Opal* Adrian Pearson Dave Minturn 9/26 9:30 am Winchester Selecting an NVMe over Fabrics Ethernet Transport, RDMA or TCP Anil Vasudevan 9/26 9:30 am Andy Rudoff Stevens Creek Persistent Memory Programming Made Easy with pmemkv

9/26 11:35 am Ziye Yang Winchester SPDK based user space NVMe over TCP Transport Solution Vishal Verma 9/26 3:35 pm Stevens Creek Improved Storage Performance Using the New Linux Kernel I/O Interface John Kariuki Join Birds-of-a-Feather NVMe-oF session Today 7:00 PM in Winchester Room and attend “Selecting NVMe-oF Ethernet Transport RDMA or TCP” presentation Thurs 9:30 AM in Winchester Room to learn more

26 NOTICES & DISCLAIMERS

Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Performance varies depending on system configuration. Performance results are based on testing as of September 2019 and may not reflect all publicly available security updates. See configuration disclosure for details. No product can be absolutely secure. Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. For more complete information about performance and benchmark results, visit http://www.intel.com/benchmarks . Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information visit http://www.intel.com/benchmarks . Intel® Advanced Vector Extensions (Intel® AVX)* provides higher throughput to certain processor operations. Due to varying processor power characteristics, utilizing AVX instructions may cause a) some parts to operate at less than the rated frequency and b) some parts with Intel® Turbo Boost Technology 2.0 to not achieve any or maximum turbo frequencies. Performance varies depending on hardware, software, and system configuration and you can learn more at http://www.intel.com/go/turbo. Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. Cost reduction scenarios described are intended as examples of how a given Intel-based product, in the specified circumstances and configurations, may affect future costs and provide cost savings. Circumstances will vary. Intel does not guarantee any costs or cost reduction.

Intel does not control or audit third-party benchmark data or the web sites referenced in this document. You should visit the referenced web site and confirm whether referenced data are accurate.

© 2019 Intel Corporation. Intel, the Intel logo, and Intel Xeon are trademarks of Intel Corporation in the U.S. and/or other countries. *Other names and brands may be claimed as property of others. NVME/TCP WITH ADQ ACCELERATION TESTING CONFIGURATION

SUT (Host) Client (Initiator) Test by Intel Intel Test date 09/17/19 09/17/19 Platform Dell R740XD Dell R740XD # Nodes 1 1 # Sockets 2 2 CPU Intel® Xeon® Platnium 8168 (33M cache 2.70GHz) Intel® Xeon® Platnium 8168 (33M cache 2.70GHz) 48 cores/socket 2 threads/socket Cores/socket, Threads/socket 48 cores/socket 2 threads/socket

Microcode 0x200005a 0x200005a HT Enabled Enabled Turbo Enabled Enabled BIOS version Dell 2.1.8 Dell 2.1.8 System DDR Mem Config: slots / cap / run-speed 4 slots / 32GB / 2666 MT/s 8 slots / 16GB / 2666 MT/s System DCPMM Config: slots / cap / run-speed N/A N/A Total Memory/Node (DDR+DCPMM) 128GB DDR4-2666 RDIMM 128GB DDR4-2666 RDIMM Storage - boot 128GB SATA3 SSD 128GB SATA3 SSD Storage - application drives 6x Intel® Optane SSD DC P4800X Series (375GB, 2.5in PCIe 3.1) N/A NIC Intel E810-C Intel E810-C Platform Chipset Intel Corporation C620 Series Chipset Family Intel Corporation C620 Series Chipset Family Other HW (Accelerator) N/A N/A

OS Red Hat Enterprise Linux 7.6 Red Hat Enterprise Linux 7.6 Kernel 5.2.1 5.2.1 IBRS (0=disable, 1=enable) 1 1 eIBRS (0=disable, 1=enable) 0 0 Retpoline (0=disable, 1=enable) 1 1 IBPB (0=disable, 1=enable) 1 1 PTI (0=disable, 1=enable) 1 1 Mitigation variants (1,2,3,3a,4, L1TF) 1,2,3,L1TF 1,2,3,L1TF Workload & version Fio-3-7 Fio-3-7 Compiler RDMA driver: ice-0.12.0_rc3 (irdma-0.12.113), firmware-version: 0x800018f7 RDMA driver: ice-0.12.0_rc3 (irdma-0.12.113), firmware-version: 0x800018f7 NIC Driver TCP driver: ice-0.12.0_rc3, firmware-version: 0x800018f7 TCP driver: ice-0.12.0_rc3, firmware-version: 0x800018f7 TCP(ADQ) driver: ice-0.11.2_rc3_adq_isv, firmware-version: 0x80001563 TCP(ADQ) driver: ice-0.11.2_rc3_adq_isv, firmware-version: 0x80001563 27 HARDWARE CONFIGURATION FOR

SYSTEM LEVEL PERFORMANCEIntel® Optane™ DC Persistent Memory SSDs Component Single DIMM Config Intel-tested: Measured using FIO 3.1. Common Configuration Test by Intel - Intel 2U Server System, OS CentOS 7.5, kernel 4.17.6- 1.el7.x86_64, CPU 2 x Intel® Xeon® 6154 Gold @ 3.0GHz (18 Test date 02/20/2019 cores), RAM 256GB DDR4 @ 2666MHz. Configuration – Intel® Platform NeonCity Optane™ SSD DC P4800X 375GB and Intel® SSD DC P4610 Chipset LBG B1

3.2TB. Intel Microcode: 0x2000043; System BIOS: CPU CLX B0 28 Core (QDF QQYZ) 00.01.0013; ME Firmware: 04.00.04.294; BMC Firmware: DDR Speed 2666 MT/s 1.43.91f76955; FRUSDR: 1.43. AEP QS Tranche3, 256GB, 18W

32GB DDR4 (per socket) Memory Config 128GB AEP (per socket)

AEP FW 5336

BIOS 573.D10

BKC version WW08 BKC The benchmark results may need to be revised as additional testing is conducted. Performance results are based on testing as of November 15, 2018 and may not Linux OS 4.20.4-200.fc29 reflect all publicly available security updates. See configuration disclosure for Spectre/ Patched details. No product can be absolutely secure. Meltdown (1,2,3, 3a)

Performance Turning QoS Disabled, IODC=5(AD) SPDK SYSTEM CONFIGURATION

Performance results are based on testing by Intel as of 2/26/2019 and may not reflect all publicly available security updates. See configuration disclosure for details. No product or component can be absolutely secure. Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information visit www.intel.com/benchmarks. Intel(R) Xeon(R) Platinum 8280L CPU @ 2.70GHz + P4610: Tested by Intel on 4/12/2019, S2600WFT Platform with 12 x 16GB 2666MHz DDR4 (total 192GB), Storage: Intel® SSD DC S3700 800GB, Storage drives: 20x Intel® SSD DC P4610 (2TB), SPDK: (16x P4610s), URING: (4x P4610s), AIO: (2x P4610s), Bios: SE5C620.86B.0D.01.0250.112320180145, ucode: 0x4000010 (HT=ON, Turbo=ON), OS: Fedora 29, Kernel: 5.0.0-rc6+, Benchmark: bdevperf, QD= 32 (for SPDK), QD= 64 (for URING), QD=128 (for AIO), runtime = 300s, SPDK commit: b62dca930, SPDK compiled with LTO, PGO gcc compiler options, for URING (tuning: echo 0 > /sys/block/$dev/queue/iostats, echo 0 > /sys/block/$dev/queue/rq_affinity, echo 2 > /sys/block/$dev/queue/nomerge, echo 0 > /sys/block/$dev/queue/io_poll_delay) Results: 4K 100% Random Reads (100%) SPDK = 8.15M IOPS Results: 4K 100% Random Reads (100%) URING = 1.56M IOPS Results: 4K 100% Random Reads (100%) AIO = 0.614M IOPS

29 OCF FOOTNOTES/SYSTEM CONFIGURATIONS

Tests document performance of components on a particular test, in specific systems. Differences in hardware, software, or configuration will affect actual performance. Consult other sources of information to evaluate performance as you consider your purchase. For more complete information about performance and benchmark results, visit www.intel.com/benchmarks. Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Performance varies depending on system configuration. No computer system can be absolutely secure. Check with your system manufacturer or retailer or learn more at intel.com.

System Configuration for slides titled “CAS + Intel® Optane™ SSD Accelerating MySQL” (pages 27-28) and for performance claim “MySQL up to 5.1X as fast w/CAS + Intel® Optane™ SSD” (pages 6, 8, 15) and for performance claim “MySQL* accelerated 5.11X” (pages 10, 26) System configuration –Red Hat Enterprise Linux 7.3, Kernal 3.10.0-514.el7.x86_64 #1 SMP Wed Oct 19 11:24:13 EDT 2016, Purley Silver Wolf Pass S2600WFQ, BIOS Version: SE5C620.86B.0X.01.0107.122220170349, BIOS Release Date: 12/22/2017, Skylake H0 (2 Processors)(24 cores each processor, hyper-threading is enabled in BIOS so thread count per processor is 48) Intel® Xeon® Platinum 8160T CPU @ 2.10GHz, Intel(R) Rapid Storage Technology enterprise PreOS Version : 5.3.0.1052, 256GB Physical RAM installed but set to 128GB in the grub2 configuration, Intel 82574L Gigabit Ethernet Adapter, VMD enabled in BIOS and VROC HW key (Premium) installed and activated., Package C-State set to C6(non retention state) and Processor C6 set to enabled in BIOS, P-States set to default in BIOS and SpeedStep and Turbo are enabled, BMC version: 1.43.33e8d6b4 ME version: 4.00.04.309 SDR Package version: 1.43, fio version: fio-3.5-86-gcefd2, (VROC) mdadm - v4.0 - 2017-09-22 Intel build: RSTe_5.3_WW38.5, kmod-md-rste-5.3- 514_4.el7_3.x86_64

System Configuration for slides “Accelerating Ceph* using HDD Backing Store” (page 33 - 34), performance claims “Ceph* Reads up to 4.9 X Faster with CAS + Intel® Optane™ SSD” and “Ceph* Writes up to 4.8 X Faster with CAS + Intel® Optane™ SSD” (pages 6, 8, 15) and “Ceph* reads 4.9X faster, Ceph writes 4.8X faster” (pages 12, 31) Baseline 4-Node Cluster: HDD OSD Drives with Journals on Intel S4600 SSD’s: 3x OSD 1x Mon/RGW Nodes: Server Intel S2600GZ (Grizzly Pass), CPUs 2x Intel® Xeon® Ivy Bridge E5-2660v2 @ 2.20GHz, 64GB Mem, SATA Boot SSD 1 x 800GB Intel® SSD DC S3700, OSD HDD 7 x 4TB WD* WDC_WD4003FZEX (excl. Mon/RGW), SATA Journal SSD 1 x 2TB Intel® SSD DC S4600, Network 2 x Intel® X540-AT2 10Gbe NICs; Ceph journal size: 10GB x 7. Value 4-Node Cluster: HDD OSD Drives with Journals on Optane, with/without CAS: Same as Baseline except NVMe Journal and cache 2 x 375GB Intel P4800x Optane; Ceph Journal size: 10GB x 7, Cache Size: 320GB x 2. Software: Ceph Luminous v12.2.3, RHEL 7.4 Updated, COSBench 0.4.2.c4, Intel CAS 3.5.1 (Value)

30 Parameter NVMe DCPMM CONFIGURATION Test by Intel/Java Performance Team Intel/Java Performance Team Test date 22/02/2019 22/02/2019 Platform S2600WFD S2600WFD SUMMARY # Nodes 1 1 # Sockets 2 2 CPU 8280L 8280L Cores/socket, Threads/socket 28/56 28/56 ucode 0x4000013 0x4000013 HT On On Turbo On On PMDK - Hardware BIOS version SE5C620.86B.0D.01.0286.011120190816 SE5C620.86B.0D.01.0286.011120190816 DCPMM BKC version NA WW52 -2018 Configuration DCPMM FW version NA 5318 Diagram System DDR Mem Config: slots / cap / run-speed 12 slots / 16GB / 2666 12 slots / 16GB / 2666 System DCPMM Config: slots / cap / run-speed - 12 slots / 512GB Total Memory/Node (DDR, DCPMM) 192GB, 0 192GB, 6TB Storage - boot 1x Intel 800GB SSD OS Drive 1x Intel 800GB SSD OS Drive Storage - application drives 4x P4610 1.6TB NVMe 12x512GB DCPMM NIC 1x Intel X722 1x Intel X722 Software OS Red Hat Enterprise Linux Server 7.6 Red Hat Enterprise Linux Server 7.6 Kernel 4.19.0 (64bit) 4.19.0 (64bit) Mitigation log attached Yes Yes DCPMM mode NA App Direct, Persistent Memory Run Method 5 minute warm up post boot, then start 5 minute warm up post boot, then start performance recording performance recording Iterations and result choice 3 iterations, median 3 iterations, median Dataset size Two 1.5 Billion Partitions (Insanity schema) Two 1.5 Billion Partitions (Insanity schema) Workload & version Read Only, Mix 80% Read/20% Updates, Read Only, Mix 80% Read/20% Updates, Updates only Updates only Compiler ANT 1.9.4 compiler for Cassandra ANT 1.9.4 compiler for Cassandra Libraries NA PMDK 1.5, LLPL (latest as of 2/20/1019) Other SW (Frameworks, Topologies…) NA NA

PMDK - HARDWARE CONFIGURATION DIAGRAM

Optane Optane

DRAM

DRAM Optane

Optane CLX

Optane Optane

DRAM

DRAM PM PM Server

S0 S1

DRAM

Optane

Optane Optane

DRAM

Optane Optane

NVMe NVMe NVMe NVMe P4610 P4610 P4610 P4610

10Gb network

10Gbit switch

10Gb network 10Gb network

Client running Client running cassandra-stress cassandra-stress

Intel Confidential-CNDA Required Intel Confidential 32 PMDK - SOFTWARE CONFIGURATION DIAGRAM

Server 1

Socket 1 Persistent Memory Client 1 Namespace or 2 NVME cassandra-stress 1 Cassandra App 1 Database 1 cassandra-stress 2

Socket 2 Persistent Memory Client 2 Namespace or 2 NVME cassandra-stress 3 Cassandra App 2 Database 2 cassandra-stress 4

Intel Confidential-CNDA Required Intel Confidential 33 ISA-L FOOTNOTES/SYSTEM CONFIGURATIONS

CLX:

Intel(R) Xeon(R) Platinum 8280L, 28C, 2.7 GHz, H0, Neon City CRB, 12x16 GB DDR4 2933 MT/s ECC RDIMM, Micron MTA18ASF2G72PDZ-2G9E1TG, NUMA Memory Configuration, Red Hat Enterprise Linux Server 7.5 64-bit OS, kernel 3.10.0-957.1.3.el7.x86_64, BIOS ENERGY_PERF_BIAS_CFG: PERF, Disabled: P-States, Turbo, Speed Step, C-States, Power Performance Tuning, Isochronous, Memory Power Savings, ISA-L 2.25

CLX:

Intel(R) Xeon(R) Gold 6230, 20C, 2.1 GHz, H0, Neon City CRB, 12x16 GB DDR4 2933 MT/s ECC RDIMM, Micron MTA18ASF2G72PDZ-2G9E1TG, NUMA Memory Configuration, Red Hat Enterprise Linux Server 7.5 64-bit OS, kernel 3.10.0-957.1.3.el7.x86_64, BIOS ENERGY_PERF_BIAS_CFG: PERF, Disabled: P-States, Turbo, Speed Step, C-States, Power Performance Tuning, Isochronous, Memory Power Savings, ISA-L 2.25

SKX:

Intel(R) Xeon(R) Gold 6126, 12C, 2.6 GHz, H0, Neon City CRB, 12x16 GB DDR4 2666 MT/s ECC RDIMM, Micron MTA36ASF2G72PZ-2G6B1QI, NUMA Memory Configuration, Red Hat Enterprise Linux Server 7.4 64-bit OS, kernel 3.10.0-693.21.1.el7.x86_64, BIOS ENERGY_PERF_BIAS_CFG: PERF, Disabled: P-States, Turbo, Speed Step, C-States, Power Performance Tuning, Isochronous, Memory Power Savings, ISA-L 2.23 vs ISA-L 2.25

BDX:

Intel(R) Xeon(R) E5-2650v4, 12C, 2.2 GHz, B0, Aztec City CRB, 8x8 GB DDR4 2400 MT/s ECC RDIMM, Samsung M393A1G43DB0, NUMA Memory Configuration, Red Hat Enterprise Linux Server 7.4 64-bit OS, kernel 3.10.0-693.21.1.el7.x86_64, BIOS ENERGY_PERF_BIAS_CFG: PERF, Disabled: P-States, Turbo, Speed Step, C-States, Power Performance Tuning, Isochronous, Memory Power Savings, ISA-L 2.23

34 See page 21 in “Notices and Disclaimers Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information visit http://www.intel.com/performance.