white paper

Implementing Storage in ® Omni-Path

Architecture Fabrics Rev 2

Executive Overview A rich ecosystem of The Intel® Omni-Path Architecture (Intel® OPA) is the next-generation fabric storage solutions architected to deliver the performance and scaling needed for tomorrow’s high performance computing (HPC) workloads. supports Intel Omni- Path A rich ecosystem of storage offerings and solutions is key to enabling the building of high performance Intel OPA-based systems. This white paper

describes Intel OPA storage solutions and discusses the considerations involved in selecting the best storage solution. It is aimed at solution architects and those interested in understanding native Intel OPA high performance storage or connecting legacy storage solutions.

Storage Overview A system with an Intel OPA-based network fabric often requires connectivity to a parallel file system, enabling the Intel OPA connected compute nodes to access the file system storage in the most optimal way. These storage solutions involve storage servers, routers, block storage devices, storage networks, hierarchical storage management and parallel file systems. This overview discusses the components that make up the storage solutions and describes some typical configurations.

2

Table of Contents Storage Components Executive Overview 1 This section describes the terminology that will be used in this paper to discuss the Storage Overview ...... 1 components of a storage solution. Storage Components ...... 2 Storage Configurations ...... 2 Intel OPA Software and Storage Considerations ...... 3 Intel OPA HFI coexistence with Mellanox* Client Side Storage Storage Storage InfiniBand HCA...... 4 Connection Servers Connection Devices IB, Intel® OPA, Running *, IB, FC, SAS, etc Intel OPA Storage Solutions ...... 4 Ethernet, etc GPFS, IBM Spectrum Interoperability with Existing Storage ...... 5 Scale*, NFS, etc Dual-homed Storage ...... 6 Figure 1: Storage Components LNet Router ...... 6 Scoping LNet Routers ...... 7 The client side connections are the connections to the compute cluster fabric.

IP Router ...... 8 The storage servers run the file system server software, such as Lustre* Object

Storage Server (OSS) software, IBM Spectrum Scale* / General Parallel File System (GPFS) Network Shared Disk (NSD) software. Storage servers can take different forms. The storage

servers are often implemented as standalone * servers with adapter cards for connectivity to both the client side connections and the storage connections. These are sometimes productized by storage vendors, and in some cases the storage servers are integrated into an appliance offering.

Storage Servers Storage Appliance

Figure 2: Storage Server Configurations

The storage connection and storage devices generally take the form of a block storage device offered by the storage vendors. It is expected that there will be block storage devices with Intel OPA storage connections; however, this is not critical to enabling Intel OPA storage solutions. The client side connection is the focus for Intel OPA enablement.

Storage Configurations

The connections from the storage servers to the Intel OPA fabric can take different forms, depending on the requirements of the system installation.

• Direct attached – the storage servers are directly attached to the Intel OPA fabric with Intel OPA adapter cards in the storage servers.

• Dual-homed – the storage servers are directly attached to the Intel OPA fabric and to another fabric, typically InfiniBand* (IB) or Ethernet. Adapter cards for both fabrics are installed in the storage servers.

• Routed – the storage servers are connected to the Intel OPA fabric through routers that carry traffic between the Intel OPA fabric to the client side connection of the storage servers, typically InfiniBand or Ethernet. Implementing Storage in Intel® Omni-Path Architecture Fabrics 3

The direct attached solution is usually found with new system acquisitions where the best option is to provide a native fabric interface between compute and storage. The dual-homed and routed configurations are typically used to provide connectivity to legacy storage or to share storage across multiple clusters with different fabrics.

Direct-attached OPA Storage File Sys Server Routed Storage Connection

Legacy Legacy Storage Cluster File Sys Intel® OPA Server Fabric Router

Dual-homed File Sys Storage Server

CN 0 CN 1 CN 2 CN 3 CN 4 CN n

Figure 3: Storage Configurations Overview

Intel OPA Software and Storage Considerations Intel OPA Host Software applications to “just work.” ISVs may to incorporate Intel OPA support into over time choose to implement changes future releases. Prior to being in-box Intel’s host software strategy is to to take advantages of the unique with these distributions, Intel will utilize the existing OpenFabrics Alliance capabilities present in Intel OPA to release a delta package to support interfaces, thus ensuring that today’s further optimize their offerings. Intel OPA. The Intel software will be application software written to those available on the Intel® Download interfaces runs with Intel OPA with no All of the Intel Omni-Path host software Center: code changes required. This is open source. Intel is working with https://downloadcenter.intel.com immediately enables an ecosystem of major operating system vendors

Table 1: Intel® OPA Linux* Support LINUX* DISTRIBUTION VERSIONS SUPPORTED

RedHat RHEL 6.7, RHEL 7.2 or newer

SuSE SLES 12 SP1 or newer

CentOS CentOS 6.7, CentOS 7.2 or newer

Scientific Linux Scientific Linux 7.2 or newer Note: Check with your OS Vendor to insure CPU support.

Table 2: Intel® OPA Lustre* Support LUSTRE* DISTRIBUTION VERSIONS SUPPORTING INTEL OPA

Community 2.8 or newer

Intel Foundation Edition 2.7.1 or newer

Intel Enterprise Edition • 2.4 (client support only) • 3.0 or newer (client and server) 4

Table 3: Intel® OPA & IBM Spectrum Scale (formerly GPFS) Support IBM Spectrum Scale SOFTWARE VERSION SUPPORTING INTEL OPA

IBM Spectrum Scale (formerly GPFS) over Supported IP IBM Spectrum Scale (GPFS) over RDMA Version 4.2 and beyond with Intel OPA

Intel OPA HFI coexistence with * and SUSE*, to ensure that With the Lustre versions prior to 2.9.0, Mellanox* InfiniBand HCA Intel OPA support is integrated into the software doesn’t yet have the ability their OFA implementation. At the time to manage more than one set of Lustre In a dual-homed file system server, of writing, Red Hat 7.3 and SLES 12 SP2 network settings in a single node. There or in a Lustre Networking (LNet) or IP have integrated OFA with OPA support, is a patch to address this capability that router, a single OpenFabrics Alliance simultaneously supporting Mellanox is being tracked by (OFA) software environment InfiniBand and Intel OPA. The Mellanox https://jira.hpdd.intel.com/browse/LU- supporting both an Intel OPA HFI OFED drivers do not co-exist with other 7101. With QDR and FDR InfiniBand, and a Mellanox* InfiniBand HCA is OFA supported hardware, as a result there are settings that work well for both required. The OFA software stack is OS distrtibution OFA or Intel OFA Delta IB and Intel OPA. With Enhanced Data architected to support multiple release should be used. Rate (EDR) InfiniBand, there isn’t a set of targeted network types. Currently, settings that work well for both the the OFA stack simultaneously Even when support is present, it may still InfiniBand and the Intel OPA devices. supports iWARP for Ethernet, RDMA be advantagous to update the OFA Therefore, coexistence of Intel OPA and over Converged Ethernet (RoCE), software to resolve critical issues. Linux EDR InfiniBand isn’t recommended until InfiniBand networks, and the Intel distribution support is provided by the that Lustre patch is available. The patch OPA network has been added to operating system vendor. Operating has been encorporated into Lustre that list. As the OS distributions system vendors are expected to provide source builds, and also included in all implement their OFA stacks, it will the updates necessary to address issues future Intel Enterprise Edition Lustre be validated to simultaneously with the OFA stack that must be version 3.0.1 releases as well as support both Intel OPA Host Fabric resolved prior to the next official Linux community releases. Adapters and Mellanox Host distribution release. This is the way that Channel Adapters. software drivers for other interconnects,

such as Ethernet, work as well. Intel is working closely with the major Linux distributors, including

Intel OPA Storage Solutions Intel OPA Direct Attached Storage High performance file systems with connectivity to an Intel OPA compute fabric, including Lustre*, BeeGFS* and IBM Spectrum Scale* (formerly GPFS), are a core part of end-to-end Intel OPA solutions. When an Intel OPA based system requires new storage, there are several options available. The Omni-Path Fabric Builders catalog contains details on the ecosystem of partner offerings as well as partner contact names, including storage and storage solution providers. https://fabricbuilders.intel.com • OEM/Customer built – OEMs or end customers put together the file system, procuring block storage from storage vendors, selecting an appropriate server, and obtaining the file system software either from the open source community or from vendors that offer supported versions. This option is straightforward with Intel OPA. See the Intel OPA Host Software section above for information about OS and file system software versions compatible with Intel OPA. • Storage vendor offering – In some cases, the complete file system solution is provided by a system or storage vendor. These complete solutions can take the form of block storage with external servers running the file system software or fully integrated appliance-type solutions. Implementing Storage in Intel® Omni-Path Architecture Fabrics 5

Interoperability with If the existing file system can be cases, the file system can be upgraded Existing Storage upgraded to support Intel OPA to be dual homed by adding Intel OPA connections, this is often the best host adapters to the file system servers. In some cases, when a new Intel solution. Again keeping in mind the hardware, OS OPA-based system is deployed, and software requirements for all there are requirements for access to For cases where the existing storage components. existing storage. There are three will only be accessed by the Intel OPA main options to be considered: based system, this upgrade can take In some cases, dual-homing the storage is not possible. This can happen when • Upgrade existing file system to the form of replacing existing fabric the file servers are older and cannot Intel OPA adapter cards with Intel OPA adapter cards if supported by the server and support the newer hardware or the • Dual-home the existing file system storage vendor. Software and effort, risk and downtime required to to Intel OPA Operating System upgrades may also update the file system software • LNet and IP Router solutions be required. Contact your storage outweigh the benefits provided by dual- vendor to help with this upgrade and homing the solution. Site specific scenarios, such as to insure existing storage can be through-put requirements, how long In these cases, a router-based solution upgraded to support Intel OPA. the existing storage will remain in can solve the interoperability challenge. production and even distance In other cases, the existing file system For Lustre file systems, the LNet router between solutions need to be will need to continue to be accessed component of Lustre can be used for reviewed. from an existing (non-Intel OPA) cluster this purpose. For other file systems, and also will require access from the such as IBM Spectra Scale*, GPFS, or new Intel OPA-based system. In these NFS the Linux IP router can be used.

Table 4: Storage Options Considerations

DIRECT-ATTACHED DUAL HOMED ROUTED SOLUTION

Pros • Excellent bandwidth • Excellent bandwidth • Easy to add to existing storage • Predictable performance • Predictable performance • Minimal to no downtime of existing • No additional system complexity storage

Cons • Legacy cluster may need a • Downtime to update OS, driver stack and • Bandwidth requirements may mean that router to access. install hardware multiple routers are required • Legacy hardware may not support OPA • Complexity to manage extra • Updates may be viewed as too risky by pieces in the system storage administrators Implementing Storage in Intel® Omni-Path Architecture Fabrics

Dual-homed Storage In the dual-homed approach, an Intel OPA connection is provided directly from the file system server, providing the best possible bandwidth and latency solution. This is a good option when: • The file system servers have a PCIe slot available to add the Intel OPA adapters and meet Intel OPA hardware requirements • The file system servers utilize OS and file system software versions compatible with Intel OPA, or can be upgraded to do so For OEMs and customers who have built the file system themselves, this solution will be supported through the OS and file system software arrangements that are already in place. When the file system solution was provided by a storage vendor, that vendor can be engaged to perform and support the upgrade to dual-homed.

LNet Router The LNet router is a standard component of the Lustre stack. It is specifically designed to route traffic natively from one network type to another with the ability to perform load-balancing and failover. To facilitate the implementation of Lustre routers in Intel OPA deployments, a validated reference design recipe is provided in the Storage Router Design Guide. This recipe provides instructions on how to implement and configure LNet routers to connect Intel OPA and InfiniBand fabrics. Some vendors have also developed their own LNet router, while others may redistribute an Intel LNet router.

The Lustre software supports dynamic load sharing between multiple targets. This is handled in the client, which has a router table and does periodic pings to all of its end points to check status. This capability is leveraged to provide load balancing across multiple LNet routers. Roundrobin load sharing is performed transparently. This capability also provides for failover because in the event of an LNet router failure, the load is automatically redistributed to other available routers.

Implementing Storage in Intel® Omni-Path Architecture Fabrics

Table 5: LNet Router Hardware Recipe HARDWARE RECOMMENDATION

CPU Intel® Xeon® E5-2640 v4 (2.40 GHz), 10 core – Hyperthreading disabled

Memory 16 to 32 GByte RAM per node

Server Platform 1U rack server or equivalent form factor with two x16 PCIe* slots

Intel® OPA connection One x16 HFI

IB/Ethernet connection • Mellanox FDR • Mellanox EDR • Ethernet

Table 6: LNet Router Software Recipe SOFTWARE RECOMMENDATION

Base OS • RHEL 7.2 + Intel® OPA delta distribution OR • SLES 12 SP1 + Intel OPA delta distribution

Lustre* • Community version 2.8 OR • IEEL 2.4 or newer OR • FE 2.7.1 or newer Note: The LNet Router Hardware Recipe (Table 5) and the LNet Router Software Recipe (Table 6) information is preliminary and based upon configurations that have been tested by Intel to date. Further optimization of CPU and memory requirements is planned.

LNet CNo Routers Intel® OPA Compute Existing Storage Infrastructure CNn Fabric OPA IB Storage Servers

Figure 7: LNet Router

Scoping LNet Routers Some scenarios will require more than a single LNet router to provide appropriate throughput to the existing storage system. Whether developing your own LNet Router configuration or making use of packaged solutions available on the market today, there is a few considerations and rules of thumb that can be used for architecting an appropriate solution.  An LNet Router is capable of approximately 80% of the slowest network types throughput.  In practice LNet routers trend to linear scaling when routers are designated as equal priority and equal hops in the client fabric. Note: this does rely upon the backend file systems total capability, bulk data transfers and large messages.  Attempt to have the network cards use the same PCIe bus sub-system Intel benchmarked numerous LNet router configurations including changing CPU frequency from 2.6Ghz to 1.2 Ghz to measure the effect it had on performance, while also providing a baseline for individual LNet router performance. The setup used to collect these results had FDR and EDR compute nodes connecting through an LNet Router to an OPA based Lustre solution. The router remained the point of network contention and the results should hold true if the storage was InfiniBand based and the OPA compute nodes were attempting to read/write to storage. Some of the IOR results can be found in Table 7 below for a server with Intel® Xeon® E5-2697A v4, 2.6 GHz, 16 cores with the CPU clock speed changed to 2.0GHz. With the OPA/FDR results the PCIe Adapter were connected to the same socket, for the OPA/EDR results the PCIe adapters were connected to Implementing Storage in Intel® Omni-Path Architecture Fabrics

different sockets. What we found was the CPU speed had only minor effects on performance and did not play a direct role on the throughput the LNet router was capable of providing.

Table 7: LNet Router IOR Performance LNET ROUTER NETWORKS1 WRITE PERFORMANCE (IOR) READ PERFORMANCE (IOR)

OPA and Mellanox FDR2 6.1 GB/s 5.3 GB/s

9.3 GB/s saturated bandwidth with 4 clients 10.1 GB/s saturated bandwidth with 4 clients OPA and Mellanox EDR3 6.7 GB/s with 1 client node 7.2 GB/s with 1 client node Note: This table should be used as guidance, each individual solution will vary slightly. When a solution requires higher read and write I/O than is available from one LNet router, the additional bandwidth can be achieved by instantiating multiple LNet routers. To scope the number of LNet Routers required, a solution architect should understand the capability of the existing storage and the customers throughput requirements between the legacy storage and the new fabric systems, typically both in GB/s. Using this information and knowledge of rough LNet Router performance the appropriate number of routers can be determined. Solution architects may wish to err on the side of additional LNet Routers, as an additional router will provide fault tolerance and compensate for non-bulk traffic found in some application I/O communication patterns. As an example; if a customer would like to have 20GB/s read and write with a legacy FDR Lustre* storage solution that is capable of this level of performance. Based on our own findings found in table 7 above we know a single FDR LNet router is capable of 6 & 5 GB/s write/read respectively. Proposing 4 LNet routers would cover the requirements in best case scenarios. It may be beneficial to propose a 5th LNet Router as it could offer an additional level of cushion for redundancy and performance. Support for the LNet router solution is provided through the customer’s Lustre support path, as it is a standard part of the Lustre software stack. IP Router The IP router is a standard component in Linux. When configured to support routing between an Intel OPA fabric and a legacy fabric, it provides IP based routing that can be used for IP traffic from GPFS, NFS, IP based LANs and other file systems that use IP based traffic. To facilitate the implementation of IP routers in Intel OPA deployments, a validated reference design recipe is provided. The recipe is available at the Intel download center host software documentation, titled Intel® Omni-Path Storage Router Design Guide, provides instructions on how to implement and configure IP routers to connect Intel OPA and InfiniBand or Ethernet networks. The router design guide focuses on storage TCP/IP routing but the fundamentals hold true for general TCP/IP routing. The Virtual Router Redundancy Protocol (VRRP) v3 software in Linux is used to enable failover and load balancing with the IP routers. The VRRP is a computer networking protocol that provides for automatic assignment of available Internet Protocol (IP) routers to participating hosts. This increases the availability and reliability of routing paths via automatic default gateway selections on an IP subnetwork. IP routers can be configured for high availability using VRRP. This can be done with an active and a passive server. In a system configured with multiple routers, routers can be configured to be master on some subnets and slaves on the others, thus allowing the routers to be more fully utilized while still providing resiliency. The load-balancing capability is provided by VRRP using IP Virtual Server (IPVS). IPVS implements transport-layer load balancing, usually called Layer 4 LAN switching, as part of the Linux kernel. IPVS is incorporated into the Linux Virtual Server (LVS), where it runs on a host and acts as a load balancer in front of a cluster of real servers. IPVS can direct requests for TCP- and UDP- based services to the real servers, and make services of the real servers appear as virtual services on a single IP address. Implementing Storage in Intel® Omni-Path Architecture Fabrics

A weighted round-robin algorithm is used and different weights can been added to distribute load across file system servers that have different performance capabilities.

IP CNo Routers Existing Intel® OPA Storage Compute Infrastructure Fabric CNn ( IB or Ethernet ) OPA Ethernet GPFS or NFS or IB Storage Servers Figure 8: IP Router

Table 8: IP Router Hardware Recipe HARDWARE RECOMMENDATION

CPU Dual-socket current or future generation Intel® Xeon® Processors Example: • Intel® Xeon® E5-2643 v4 (3.40 GHz), 6 core

Memory 16 or 32 GByte RAM per node

Server Platform 1U rack server or equivalent form factor with two x16 PCIe slots

Intel® OPA connection One x16 HFI

IB/Ethernet connection • Mellanox FDR and EDR (Other generations are expected to work, performance will vary) • Ethernet

Table 9: IP Router Software Recipe SOFTWARE RECOMMENDATION

Base OS • RHEL 7.2 + Intel® OPA delta distribution OR • SLES 12 SP1 + Intel OPA delta distribution Note: The IP Router Hardware Recipe (Table 7) and the IP Router Software Recipe (Table 8) information is preliminary and based upon configurations that have been tested by Intel to date. Further optimization of CPU and memory is planned.

The target peak aggregate forwarding rate is 35 Gbps per IP router server with either EDR or FDR IB when following the above recipe. Performance of IP Router functionality is dependent on the TCP/IP stack in Linux which has proven to be sensitive to CPU frequency speeds. Higher CPU frequencies should be considered for optimal performance, lower frequency solution can be used but as with all solutions they should be benchmarked to fully understand the IPoIB throughput capabilities. Other generations of InfiniBand are expected to work with this recipe, however performance will vary. Ethernet connectivity and routing to storage or other networks is also supported by the recipe. Additional bandwidth can be achieved by instantiating multiple IP routers. Support for the IP router solution is provided through the customer’s Linux support path, as it is a standard part of the Linux software stack.

For more information about Intel Omni-Path Architecture and next-generation fabric technology, visit: www.intel.com/hpcfabrics www.intel.com/omnipath

Implementing Storage in Intel® Omni-Path Architecture Fabrics

Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products.

THE INFORMATION PROVIDED IN THIS PAPER IS INTENDED TO BE GENERAL IN NATURE AND IS NOT SPECIFIC GUIDANCE. RECOMMENDATIONS (INCLUDING POTENTIAL COST SAVINGS) ARE BASED UPON INTEL’S EXPERIENCE AND ARE ESTIMATES ONLY. INTEL DOES NOT GUARANTEE OR WARRANT OTHERS WILL OBTAIN SIMILAR RESULTS.

INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS AND SERVICES. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL’S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS AND SERVICES, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS AND SERVICES INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. Results have been estimated or simulated using internal Intel analysis or architecture simulation or modeling, and provided to you for informational purposes. Any differences in your system hardware, software or configuration may affect your actual performance. Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Performance varies depending on system configuration. No computer system can be absolutely secure. Check with your system manufacturer or retailer or learn more at intel.com. All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest Intel product specifications and roadmaps. Copyright © 2017 Intel Corporation. All rights reserved. Intel, Intel Xeon, and the Intel logo are trademarks of Intel Corporation in the U.S. and other countries. * Other names and brands may be claimed as the property of others. 1 - Lustre • Intel® Enterprise Edition for Lustre 46TB file system (IEEL version 2.7.16.4) • Meta Data Subsystem • 2xMDS Dual socket E5-2699 v3 BDW-EP, 256 GB/node 2133 MHz DDR4, RHEL7.2 • 2x 480GB Intel Haleyville SSD per node • Intel R2224WTTYSR Wildcat Pass • Object Storage Subsystem • 4xOSS Dual socket E5-2699 v3 BDW-EP, 256 GB/node 2133 MHz DDR4, RHEL7.2 • 24x 480GB Intel Haleyville SSD per node • Intel R2224WTTYSR Wildcat Pass • SAS Controller 47 Lnet Router • Intel® Xeon® processor E5-2697A v4, 2.60 GHz, 16 cores • Intel Enterprise edition for Lustre version 2.7.16.11 • Intel® Turbo Boost Technology disabled, Intel® Hyper-Threading Technology disabled • BIOS settings: Snoop hold-off timer = 9, Early snoop disabled, Cluster on die disabled, IOU Non-posted prefetch disabled • OS: Red Hat Enterprise Linux* Server release 7.2 (Maipo) • Kernel: 3.10.0-327.36.3.el7.x86_64 2 - Tests performed on Intel® Xeon® processor E5-2697A v4, 2.60 GHz, 16 cores. Intel® Turbo Boost Technology enabled, Intel® Hyper-Threading Technology enabled. RHEL 7.2. BIOS settings: IOU non-posted prefetch disabled. Snoop timer for posted prefetch=9. Early snoop disabled. Cluster on Die disabled. Intel Fabric Suite 10.2.0.0.158. Intel Corporation Device 24f0 – Series 100 HFI ASIC (B0 silicon). OPA Switches: Series 100 Edge Switch – 48 port (B0 silicon). Mellanox EDR based on internal measurements: MLNX_OFED_LINUX-3.2-2.0.0.0 (OFED-3.2- 2.0.0). Mellanox EDR ConnectX-4 Single Port Rev 3 MCX455A HCA. Mellanox SB7700 - 36 Port EDR Infiniband switch. IOR benchmark version 2.10.3. Transfer size=1 MB, File size=256 GB, 16ppn, unique file created per process. EDR parameters: -genv I_MPI_FABRICS=shm:dapl 3 - Through-put achieved with 4 aggregate EDR clients reading/writing to OPA based Lustre Storage. Tests performed on Intel® Xeon® processor E5-2697A v4, 2.60 GHz, 16 cores. Intel® Turbo Boost Technology enabled, Intel® Hyper-Threading Technology enabled. RHEL 7.2. BIOS settings: IOU non-posted prefetch disabled. Snoop timer for posted prefetch=9. Early snoop disabled. Cluster on Die disabled. Intel Corporation Device 24f0 – Series 100 HFI ASIC (B0 silicon). OPA Switches: Series 100 Edge Switch – 48 port (B0 silicon). Mellanox EDR based on internal measurements: MLNX_OFED_LINUX-3.2-2.0.0.0 (OFED- 3.2-2.0.0). Mellanox EDR ConnectX-4 Single Port Rev 3 MCX455A HCA. Mellanox SB7700 - 36 Port EDR Infiniband switch. IOR benchmark version 2.10.3. Transfer size=1 MB, File size=256 GB, 16ppn, unique file created per process. EDR parameters: -genv I_MPI_FABRICS=shm:dapl

Printed in USA 1115/EON/HBD/PDF Please Recycle 333509-001US