DOCSLIB.ORG

Intel® Omni-Path Fabric Host Software — User Guide

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text

Load more

Intel® Omni-Path Fabric Host Software User Guide Rev. 16.0 January 2020 Doc. No.: H76470, Rev.: 16.0 You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document. All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest Intel product specifications and roadmaps. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Performance varies depending on system configuration. No computer system can be absolutely secure. Check with your system manufacturer or retailer or learn more at intel.com. Intel, the Intel logo, Intel Xeon Phi, and Xeon are trademarks of Intel Corporation in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others. Copyright © 2015–2020, Intel Corporation. All rights reserved. Intel® Omni-Path Fabric Host Software User Guide January 2020 2 Doc. No.: H76470, Rev.: 16.0 Revision History—Intel® Omni-Path Fabric Revision History Date Revision Description January 2020 16.0 Updates to this document include: • Updated links to Intel® Omni-Path documentation set in Intel® Omni-Path Documentation Library. • Renamed section from IB Bonding to IPoIB Bonding. • Updated instructions for Create RHEL* IB Bonding Interface Configuration Script and Create SLES* IB Bonding Interface Configuration Script • Updated Running MPI Applications with Intel® MPI Library as TMI has been deprecated in Intel MPI 2019. • Removed information for running MPI over IPoIB in Running Open MPI Applications. • Updated publication link in Intel® Omni-Path Driver and PSM2 Support for GPUDirect*. • Added link to Configuring Non-Volatile Memory Express* (NVMe*) over Fabrics on Intel® Omni-Path Architecture Application Note in Non-Volatile Memory (NVM) Express* over Fabrics. October 2019 15.0 Updates to this document include: • Updated Preface to include new Best Practices. • Removed Chapter Using Sandia* OpenSHMEM and globally removed SHMEM throughout document; it is no longer supported. • Updated multi-rail information in Multi-Rail Support in PSM2. • Updated opaconfig to add ARPTABLE_TUNING question to -- answer option. June 2019 14.0 Updates to this document include: • Added new Troubleshooting section: Viewing IPoIB Hardware Addresses. March 2019 13.0 Updates to this document include: • Updated CLI command opaconfig to remove -r root option. • In the Update Existing Exports File and RDMA Configuration File section, corrected the instructions for updating the rdma.conf file, correcting the port number information in Step 2 to 20049. February 2019 12.0 Updates to this document include: • Updated the following CLI commands: — opaautostartconfig — opaconfig December 2018 11.0 Updates to this document include: • Added new CLI command: opasystemconfig • Updated opaconfig • Updated opasaquery September 2018 10.0 Updates to this document include: • Global change: Removed Verbs-only builds for Open MPI and MVAPICH2 (gcc) as well as those built with the Intel compiler for Open MPI and MVAPICH2. • Changed all textual references of OpenMPI to Open MPI • Updated Intel Omni-Path Architecture Overview. continued... Intel® Omni-Path Fabric Host Software January 2020 User Guide Doc. No.: H76470, Rev.: 16.0 3 Intel® Omni-Path Fabric—Revision History Date Revision Description • Updated the following CLI commands: — opaconfig — opaportconfig — opaportinfo — opasaquery — opasmaquery April 2018 9.0 Updates to this document include: • Removed section SHMEM Description and Configuration • Minor text updates in Introduction • Updated Installed Layout • Updated Configuring IPoIB Driver • Updated IB Bonding Interface Configuration Scripts and instructions found in Create RHEL* IB Bonding Interface Configuration Script and Create SLES* IB Bonding Interface Configuration Script • Updated IBM* Platform* MPI • Added Using PSM2 Features for GPUDirect*. • Updated section Multiple Bandwidth / Message Rate Test • Added new CLI command: hfi1_eprom • Updated the following CLI commands: — opaautostartconfig — opaconfig — opapacketcapture — opapmaquery — opaportconfig — opasaquery — opasmaquery • Removed opatmmtool. October 2017 8.0 Updates to this document include: • Added IPv4/IPv6 Dual Stack Support • Added new chapter Using Sandia* OpenSHMEM • Added Descriptions of Command Line Tools including the following CLI tools: opa-arptbl-tuneup, opaautostartconfig, opa- init-kernel, opa_osd_dump, opa_osd_exercise, opa_osd_perf, opa_osd_query, opacapture, opaconfig, opafabricinfo, opagetvf, opagetvf_env, opahfirev, opainfo, opapacketcapture, opapmaquery, opaportconfig, opaportinfo, oparesolvehfiport, opasaquery, opasmaquery, and opatmmtool • Updated the following CLI commands: — opaconfig — opasaquery August 2017 7.0 Updates to this document include: • Updated section PSM2 Compatibility • Updated section Intel MPI Library • Updated appendix hfidiags User Guide April 2017 6.0 Updates to this document include: • Added Intel Omni-Path Documentation Library to Preface. • Added Intel Omni-Path Architecture Overview • Updated sections Intel Omni-Path Software Overview and Host Software Stack • Updated Intel Distributed Subnet Administration • Updated Multi-Rail Support in PSM2 continued... Intel® Omni-Path Fabric Host Software User Guide January 2020 4 Doc. No.: H76470, Rev.: 16.0 Revision History—Intel® Omni-Path Fabric Date Revision Description • Updated Configuring IPoIB Driver • Updated Running MPI on Intel Omni-Path Host Fabric Interfaces • Updated Using Other MPIs • Added IBM* Platform* MPI • Added Using NFS Over RDMA with Intel OPA Fabrics • Added Intel Omni-Path Driver and PSM2 Support for GPUDirect* • Added Non-Volatile Memory (NVM) Express* over Fabrics • Default MTU size is changed from 8K to 10K • Globally, updated the following filepaths: — from /usr/lib/opa-fm/samples/ to /usr/share/opa-fm/ samples/ — from /usr/lib/opa-fm/etc/opafm* to /usr/lib/opa- fm/bin/opafm* — from /etc/sysconfig/opafm.xml to /etc/opa-fm/ opafm.xml — from /etc/sysconfig/* to /etc/* — from /usr/lib/opa/ to /usr/share/opa/ — from /usr/lib/opa/src/* to /usr/src/opa/* December 2016 5.0 Updates to this document include: • Added sections Intel Omni-Path Architecture Overview, Intel Omni- Path Software Overview, and Host Software Stack. • Multi-Rail Support in PSM2: clarified behavior when PSM2_MULTIRAIL is not set in multiple fabrics. Added Multi-Rail Overview section. • Added hfidiags User Guide (moved from Software Installation Guide to this document). • Globally, updated the following filepaths: — from /opt/opa to /usr/lib/opa — from /var/opt/opa to /var/usr/lib/opa — from /opt/opafm to /usr/lib/opa-fm — from /var/opt/opafm to /var/usr/lib/opa-fm • Added Cluster Configurator for Intel Omni-Path Fabric to Preface. August 2016 4.0 Updates to this document include: • Added Setting up Open MPI with SLURM. • Removed pre-built PGI* MPIs from "Installing University of Houston OpenSHMEM" section. The PGI* compiler is supported, however, pre-built PGI* MPIs are not included in the software package. • Added Intel Omni-Path Routing Features and Innovations. May 2016 3.0 Global updates to document. February 2016 2.0 Global updates to document. November 2015 1.0 Global updates to document. Intel® Omni-Path Fabric Host Software January 2020 User Guide Doc. No.: H76470, Rev.: 16.0 5 Intel® Omni-Path Fabric—Contents Contents Revision History..................................................................................................................3 Preface............................................................................................................................. 12 Intended Audience..................................................................................................... 12 Intel® Omni-Path Documentation Library.......................................................................12 How to Search the Intel® Omni-Path Documentation Set........................................ 14 Cluster Configurator for Intel® Omni-Path Fabric.............................................................15 Documentation Conventions........................................................................................ 15 Best Practices............................................................................................................ 16 License Agreements....................................................................................................16 Technical Support.......................................................................................................16 1.0 Introduction................................................................................................................17 1.1 Intel® Omni-Path Architecture Overview.................................................................. 17 1.1.1 Host Fabric Interface................................................................................. 19 1.1.2 Intel® OPA Switches..................................................................................19
Recommended publications
  • Rapidio® Connections

    Rapidio® Connections

    ® RapidIO Connections Q4 2011 Forward: RapidIO has had an amazing year with many new product announcements, the launch of the RapidIO 10xN roadmap--with serial lanes up to 10 Gbps and port data rates that go up to 160 Gbps--and production shipments of RapidIO Gen2 based systems. Don’t miss this issue’s links to many of the recent news releases and published articles. RapidIO is seeing excitement beyond its traditional wireless and defense/aerospace markets with high interest in the Cloud Computing market, both in the data center and in server systems being pushed to the base station in wireless. The introduction of new Gen 2 RapidIO products, and a number of single board computers with RapidIO and the Intel® Core i7 processor family. -Tom Cox. In This Issue Association News RapidIO Trade Association Releases Roadmap to 100 Gbps+ per port with RapidIO® 10xN Press Release - July 2011 Pushing the boundaries of performance for peer-to-peer embedded processing for the wireless, server, imaging, medical, industrial, defense and aerospace markets, the RTA announced its roadmap beyond RapidIO Gen2 in July of 2011. The new spec is S-RIO 10xN and it delivers 10 Gbps per serial lane and beyond. The S-RIO 10xNwill actually scale to speeds per serial lane up to 25 Gbps in the next revision of the standard. In x4 configuration, it means 40-100 Gbps per port. http://www.rapidio.org/news/pr/view?item_key=6ff6de5e4d4d09ba26a7242c34b85c28c9dd4af6 Industry Insights The OpenFabrics Alliance and OFED By OpenFabrics Alliance The OpenFabrics Alliance (OFA) is an open source community with a charter to develop, test and license high performance networking software for servers and storage systems.
  • Storage Administration Guide Storage Administration Guide SUSE Linux Enterprise Server 12 SP4

    Storage Administration Guide Storage Administration Guide SUSE Linux Enterprise Server 12 SP4

    SUSE Linux Enterprise Server 12 SP4 Storage Administration Guide Storage Administration Guide SUSE Linux Enterprise Server 12 SP4 Provides information about how to manage storage devices on a SUSE Linux Enterprise Server. Publication Date: September 24, 2021 SUSE LLC 1800 South Novell Place Provo, UT 84606 USA https://documentation.suse.com Copyright © 2006– 2021 SUSE LLC and contributors. All rights reserved. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or (at your option) version 1.3; with the Invariant Section being this copyright notice and license. A copy of the license version 1.2 is included in the section entitled “GNU Free Documentation License”. For SUSE trademarks, see https://www.suse.com/company/legal/ . All other third-party trademarks are the property of their respective owners. Trademark symbols (®, ™ etc.) denote trademarks of SUSE and its aliates. Asterisks (*) denote third-party trademarks. All information found in this book has been compiled with utmost attention to detail. However, this does not guarantee complete accuracy. Neither SUSE LLC, its aliates, the authors nor the translators shall be held liable for possible errors or the consequences thereof. Contents About This Guide xii 1 Available Documentation xii 2 Giving Feedback xiv 3 Documentation Conventions xiv 4 Product Life Cycle and Support xvi Support Statement for SUSE Linux Enterprise Server xvii • Technology Previews xviii I FILE SYSTEMS AND MOUNTING 1 1 Overview
  • Mac Labs. More Creativity

    Mac Labs. More Creativity

    Mac labs. More creativity. More engagement. More learning. Every Mac comes ready for 21st-century learning. Mac OS X Server creates the ultimate lab environment. Creativity and innovation are critical skills in the modern Mac OS X Server includes simplified tools for creating wikis and workplace, which makes them critical skills to teach today’s blogs, so students and staff can set up and manage them with students. Apple offers comprehensive lab solutions for your little or no help from IT. The Spotlight Server feature makes it school or university that will creatively engage students and easy for students collaborating on projects to find and view motivate them to take an interest in their own learning. The content stored anywhere on the network. Of course, access iLife ’09 suite of applications comes standard on every Mac, controls are built in, so they only get the search results they’re so students can immediately begin creating podcasts, editing authorised to see. Mac OS X Server also allows for real-time videos, composing music, producing photo essays and more. streaming of audio and video content, which lets everyone get right to work instead of waiting for downloads – saving major Mac or PC? Have both. storage across your network. Now one lab really can solve all your computing needs. Every Mac is powered by an Intel processor and features Launch careers from your Mac lab. Mac OS X – the world’s most advanced operating system. As an Apple Authorised Training Centre for Education, your Mac OS X comes with an amazing dual-boot feature called school can build a bridge between students and real-world Boot Camp that lets students run Windows XP or Vista natively careers.
  • Nvmf Based Integration of Non-Volatile Memory in A

    Nvmf Based Integration of Non-Volatile Memory in A

    14th ANNUAL WORKSHOP 2018 NVMf based Integration of Non-volatile Memory in a Distributed System - Lessons learned Jonas Pfefferle, Bernard Metzler, Patrick Stuedi, Animesh Trivedi and Adrian Schuepbach IBM Zurich Research April 2018 Outline . Target Distributed System: Apache Crail . NVMf • Basic operation • Available implementations . The Crail NVMf Tier • Integration with Crail • Implementation details • A new NVMf host library . Measurements • Microbenchmarks • TeraSort • TPC-DS workload . Summary & Outlook 2 OpenFabrics Alliance Workshop 2018 The Target System: Apache Crail (Incubating) . Targets I/O acceleration for Big Data frameworks due to user level and asynchronous I/O . Implements efficient distributed store for ephemeral data at native hardware speeds . Unifies isolated, incompatible efforts to integrate high performance I/O with BigData processing frameworks Data Processing Framework . Enables dramatically faster job (Spark, TensorFlow, λ Compute, …) completion 10th of GB/s . Applicable to Spark, Flink, < 10 µsec FS Streaming KV HDFS TensorFlow, Caffee, … Apache Crail . Flexibly makes best use of available I/O infrastructure TCP RDMA NVMf SPDK . Apache Incubator Project Fast Network (100Gb/s RoCE, etc.) (since 11/17) DRAM NVMe PCM … GPU 3 OpenFabrics Alliance Workshop 2018 NVMf in a Nutshell . Extends low latency NVMe interface over distance NVM read command • Retains BW and latency (< 5us additional delay) • Allows to retain user level I/O capability NVMf RDMA RDMA NVMf NVMe SQ/CQ QP • Message based NVMe operations QP SQ/CQ Device • Host/Target model IO cmd . Defined transports: RDMA (IB, RoCE, post_send iWarp), or FC SEND • Mapping of I/O Submission + Completion Queue recv comp to RDMA QP model NVMe read . Fabrics + Admin Commands • Discover, Connect, Authenticate, … CMD comp .
  • For Immediate Release

    For Immediate Release

    an ellisys company FOR IMMEDIATE RELEASE SerialTek Contact: Simon Thomas, Director, Sales and Marketing Phone: +1-720-204-2140 Email: [email protected] SerialTek Debuts PCIe x16 Gen5 Protocol Analysis System and Web Application New Kodiak™ Platform Brings SerialTek Advantages to More Computing and Data Storage Markets Longmont, CO, USA — February 24, 2021 — SerialTek, a leading provider of protocol test solutions for PCI Express®, NVM Express®, Serial Attached SCSI, and Serial ATA, today introduced an advancement in the PCIe® test and analysis market with the release of the Kodiak PCIe x16 Gen5 Analysis System, as well as the industry’s first calibration-free PCIe x16 add-in-card (AIC) interposer and a new web-based BusXpert™ user interface to manage and analyze traces more efficiently than ever. The addition of PCIe x16 Gen5 to the Kodiak analyzer and SI-Fi™ interposer family brings previously unavailable analysis capabilities and efficiencies to computing, datacenter, networking, storage, AI, and other PCIe x16 Gen5 applications. With SerialTek’s proven calibration-free SI-Fi interposer technology, the Kodiak’s innovative state-of-the-art design, and the new BusXpert analyzer software, users can more easily set up the analyzer hardware, more accurately probe PCIe signals, and more efficiently analyze traces. Kodiak PCIe x16 Gen5 Analysis System At the core of the Kodiak PCIe x16 analyzer is an embedded hardware architecture that delivers substantial and unparalleled advancements in capture, search, and processing acceleration. Interface responsiveness is markedly advanced, searches involving massive amounts of data are fast, and hardware filtering is flexible and powerful. “Once installed in a customer test environment the Kodiak’s features and benefits are immediately obvious,” said Paul Mutschler, CEO of SerialTek.
  • Integrated Step-Down Converter with SVID Interface for Intel® CPU

    Integrated Step-Down Converter with SVID Interface for Intel® CPU

    Product Order Technical Tools & Support & Folder Now Documents Software Community TPS53820 SLUSE33 –FEBRUARY 2020 Integrated Step-Down Converter with SVID Interface for Intel® CPU Power Supply 1 Features 3 Description The TPS53820 device is D-CAP+ mode integrated 1• Single chip to support Intel VR13.HC SVID POL applications step-down converter for low current SVID rails of Intel CPU power supply. It provides up to two outputs to • Two outputs to support VCCANA (5.5 A) and power the low current SVID rails such as VCCANA P1V8 (4 A) (5.5 A) and P1V8 (4 A). The device employs D-CAP+ • D-CAP+™ Control for Fast Transient Response mode control to provide fast load transient • Wide Input Voltage (4.5 V to 15 V) performance. Internal compensation allows ease of use and reduces external components. • Differential Remote Sense • Programmable Internal Loop Compensation The device also provides telemetry, including input voltage, output voltage, output current and • Per-Phase Cycle-by-Cycle Current Limit temperature reporting. Over voltage, over current and • Programmable Frequency from 800 kHz to 2 MHz over temperature protections are provided as well. 2 • I C System Interface for Telemetry of Voltage, The TPS53820 device is packaged in a thermally Current, Output Power, Temperature, and Fault enhanced 35-pin QFN and operates between –40°C Conditions and 125°C. • Over-Current, Over-Voltage, Over-Temperature (1) protections Table 1. Device Information • Low Quiescent Current PART NUMBER PACKAGE BODY SIZE (NOM) • 5 mm × 5 mm, 35-Pin QFN, PowerPAD Package TPS53820 RWZ (35) 5 mm × 5 mm (1) For all available packages, see the orderable addendum at 2 Applications the end of the data sheet.
  • Introduction to Intel® FPGA IP Cores

    Introduction to Intel® FPGA IP Cores

    Introduction to Intel® FPGA IP Cores Updated for Intel® Quartus® Prime Design Suite: 20.3 Subscribe UG-01056 | 2020.11.09 Send Feedback Latest document on the web: PDF | HTML Contents Contents 1. Introduction to Intel® FPGA IP Cores..............................................................................3 1.1. IP Catalog and Parameter Editor.............................................................................. 4 1.1.1. The Parameter Editor................................................................................. 5 1.2. Installing and Licensing Intel FPGA IP Cores.............................................................. 5 1.2.1. Intel FPGA IP Evaluation Mode.....................................................................6 1.2.2. Checking the IP License Status.................................................................... 8 1.2.3. Intel FPGA IP Versioning............................................................................. 9 1.2.4. Adding IP to IP Catalog...............................................................................9 1.3. Best Practices for Intel FPGA IP..............................................................................10 1.4. IP General Settings.............................................................................................. 11 1.5. Generating IP Cores (Intel Quartus Prime Pro Edition)...............................................12 1.5.1. IP Core Generation Output (Intel Quartus Prime Pro Edition)..........................13 1.5.2. Scripting IP Core Generation....................................................................
  • ISS 2016 Bootcamp: Intel Omni-Path Architecture

    ISS 2016 Bootcamp: Intel Omni-Path Architecture

    Intel® Scalable System Framework A Configurable Design Philosophy Extensible to a Wide Range of Workloads Small Clusters Through Supercomputers Compute Memory/Storage Compute and Data-Centric Computing Fabric Software Standards-Based Programmability Intel Silicon On-Premise and Cloud-Based Photonics Intel® Xeon® Processors Intel® Solutions for Lustre* Intel® Omni-Path Architecture HPC System Software Stack Intel® Xeon Phi™ Processors Intel® SSDs Intel® True Scale Fabric Intel® Software Tools Intel® Xeon Phi™ Coprocessors Intel® Optane™ Technology Intel® Ethernet Intel® Cluster Ready Program Intel® Server Boards and Platforms 3D XPoint™ Technology Intel® Silicon Photonics Intel® Visualization Toolkit 1 YANG YANGUO May 2016 Intel Confidential Agenda Quick Overview: HPC Fabrics What is Intel® 100Gb Omni-Path Architecture(OPA)? Why is Intel 100Gb OPA Summary Intel® Solutions Summit 2016 Intel Confidential 3 Intel Confidential What is Different Between Networks and Fabrics? Network: Universal interconnect designed to HPC Fabric: Optimized interconnect allows allow any-and-all systems to communicate many nodes to perform as a single system Intel® Omni-Path Architecture or Infiniband Key NETWORK (Ethernet) Attributes: Key FABRIC Attributes: . Flexibility for any application . Targeted for specific applications . Designed for universal communication . Optimized for performance and efficiency . Extensible configuration . Engineered topologies . Multi-vendor components . Single-vendor solutions Intel® Solutions Summit 2016 Intel Confidential 5 Fabric: InfiniBand* and OPA InfiniBand/OPA is a multi-lane, high-speed serial interconnect (Copper or Fiber) . Typically presented as a 4x solution . Speeds: 40Gb/s (M & Intel QDR), 56Gb/s (M FDR), 100Gb/s (EDR & Intel OPA) High bandwidth, low latency HPC interconnect for commodity servers . Ethernet switch latency is typically measured in μs, but InfiniBand/OPA is in nanoseconds .
  • Introducing the New 11Th Gen Intel® Core™ Desktop Processors

    Introducing the New 11Th Gen Intel® Core™ Desktop Processors

    Product Brief 11th Gen Intel® Core™ Desktop Processors Introducing the New 11th Gen Intel® Core™ Desktop Processors The 11th Gen Intel® Core™ desktop processor family puts you in control of your compute experience. It features an innovative new architecture for reimag- ined performance, immersive display and graphics for incredible visuals, and a range of options and technologies for enhanced tuning. When these advances come together, you have everything you need for fast-paced professional work, elite gaming, inspired creativity, and extreme tuning. The 11th Gen Intel® Core™ desktop processor family gives you the power to perform, compete, excel, and power your greatest contributions. Product Brief 11th Gen Intel® Core™ Desktop Processors PERFORMANCE Reimagined Performance 11th Gen Intel® Core™ desktop processors are intelligently engineered to push the boundaries of performance. The new processor core architecture transforms hardware and software efficiency and takes advantage of Intel® Deep Learning Boost to accelerate AI performance. Key platform improvements include memory support up to DDR4-3200, up to 20 CPU PCIe 4.0 lanes,1 integrated USB 3.2 Gen 2x2 (20G), and Intel® Optane™ memory H20 with SSD support.2 Together, these technologies bring the power and the intelligence you need to supercharge productivity, stay in the creative flow, and game at the highest level. Product Brief 11th Gen Intel® Core™ Desktop Processors Experience rich, stunning, seamless visuals with the high-performance graphics on 11th Gen Intel® Core™ desktop
  • Proposal Regarding USENIX and Openfabrics Alliance Ellie Young, 9/18/2008

    Proposal Regarding USENIX and Openfabrics Alliance Ellie Young, 9/18/2008

    Proposal Regarding USENIX and OpenFabrics Alliance Ellie Young, 9/18/2008 For the past two months a group from the OpenFabrics Alliance and USENIX management have been working together to reach an agreement on how best to explore the feasibility and desirability of OFA and USENIX combining forces in a way that would work to the advantage of both organizations. Now the USENIX Board must decide whether we should take the next step: sending a MOU to provide services to OFA as a 501(c)(6) under our 501(c)(3). We realized last week that in order for us to announce this at the obvious juncture, the upcoming Supercomputing Conference in November, we need to have a decision from OFA by September 29. In the past month USENIX has been given a consulting membership in OFA. Some of us have been participating in marketing, legal, and executive working group meetings. I think we have now spent enough time "getting to know one another." Background Last year, in response to the Board's request that we reach out to open source software projects and look beyond the academic and research community we currently serve, Clem Cole introduced us to several members of the OpenFabrics Alliance. We made a proposal to organize their annual Sonoma Workshop this past spring, but they decided to continue organizing this year’s event on their own. We were contacted again this summer. At a meeting in August, part of which Matt Blaze attended, we discussed the current state of OFA. Our notes on that meeting include: Software development: Development is the most valuable activity for OFA, but it's on its own track (Sonoma workshops, as well as workshops alongside Supercomputing, although there were none this year) and goes pretty smoothly for two software releases per year.
  • NVM Express Over Fabrics

    NVM Express Over Fabrics

    NVM Express Over Fabrics Dave Minturn, Intel Corp. OFADevWorkshop NVMe in Fabric Environments • A primary use case for NVMe PCIe SSDs is in an all flash appliance • Hundreds or more SSDs may be attached – too many for PCIe based attach scale-out • Concern: Today remote SSD scale-out over a fabric attach uses SCSI based protocols: Requiring protocol translation Desire best performance and latency from NVMe SSD investment over fabrics like Ethernet with RDMA (iWARP,RoCE), InfiniBand™, and Intel® Omni-Path Architecture Realizing Benefit of Next Gen NVM over Fabrics • PCIe NVMe SSD latency may be < 10 µs with Next Generation NVM • Using a SCSI-based protocol for remote NVMe access ads over 100 µs* in latency • Usage models require efficient write mirroring of PCIe Next Gen NVMe SSDs over fabrics *Source: Intel measurements. Concern: Low latency of Next Gen NVM lost in (SCSI) translation. Why NVMe over Fabrics? Simplicity, Efficiency and End-to-End NVMe Model • Simplicity of protocol enables hardware automated I/O Queues – NVMe transport bridge • No translation to or from another protocol like SCSI (in firmware/software) • Inherent parallelism of NVMe multiple I/O Queues is exposed to the host • NVMe commands and structures are transferred end-to-end • Maintains the NVMe architecture across a range of fabric types • Maintains architecture and software consistency between fabric types by standardizing a common abstraction and encapsulation definition Performance Goal: Make remote NVMe access over fabrics equivalent to local PCIe attached NVMe,
  • THE STORAGE PERFORMANCE DEVELOPMENT KIT and NVME-OF Paul Luse Intel Corporation

    THE STORAGE PERFORMANCE DEVELOPMENT KIT and NVME-OF Paul Luse Intel Corporation

    14th ANNUAL WORKSHOP 2018 THE STORAGE PERFORMANCE DEVELOPMENT KIT AND NVME-OF Paul Luse Intel Corporation Apr 2018 AGENDA Storage • What is SPDK? Performance • The SPDK Community Development Kit • Why are so many storage companies using it? • How is it being used? 2 OpenFabrics Alliance Workshop 2018 WHAT IS SPDK? 3 OpenFabrics Alliance Workshop 2018 WHAT IS SPDK? Scalable and Efficient Software Ingredients Storage . User space, lockless, polled-mode components Performance . Up to millions of IOPS per core Development Kit . Designed to extract maximum performance from non- volatile media Storage Reference Architecture . Optimized for latest generation CPUs and SSDs . Open source composable building blocks (BSD licensed) http://SPDK.IO 4 OpenFabrics Alliance Workshop 2018 SPDK ARCHITECTURE 5 OpenFabrics Alliance Workshop 2018 THE SPDK COMMUNITY 6 OpenFabrics Alliance Workshop 2018 THE SPDK COMMUNITY Full Transparency 7 OpenFabrics Alliance Workshop 2018 WHY ARE SO MANY STORAGE COMPANIES USING SPDK? 8 OpenFabrics Alliance Workshop 2018 SOFTWARE IS BECOMING THE BOTTLENECK SPDK Unlocks New Media Potential 9 OpenFabrics Alliance Workshop 2018 SPDK BENEFITS . Up to 10X MORE IOPS/core for NVMe-oF* vs kernel Storage . Up to 8X MORE IOPS/core for NVMe vs kernel Performance . Up to 50% BETTER tail latency w/RocksDB workloads Development Kit . FASTER TTM w/less RESOURCES vs from scratch . Provides FUTURE PROOFING as technologies evolve Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary.