DOCSLIB.ORG

Infiniband and 10-Gigabit Ethernet for Dummies

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text

Load more

Designing Cloud and Grid Computing Systems with InfiniBand and High-Speed Ethernet A Tutorial at CCGrid ’11 by Dhabaleswar K. (DK) Panda Sayantan Sur The Ohio State University The Ohio State University E-mail: [email protected] E-mail: [email protected] http://www.cse.ohio-state.edu/~panda http://www.cse.ohio-state.edu/~surs Presentation Overview • Introduction • Why InfiniBand and High-speed Ethernet? • Overview of IB, HSE, their Convergence and Features • IB and HSE HW/SW Products and Installations • Sample Case Studies and Performance Numbers • Conclusions and Final Q&A CCGrid '11 2 Current and Next Generation Applications and Computing Systems • Diverse Range of Applications – Processing and dataset characteristics vary • Growth of High Performance Computing – Growth in processor performance • Chip density doubles every 18 months – Growth in commodity networking • Increase in speed/features + reducing cost • Different Kinds of Systems – Clusters, Grid, Cloud, Datacenters, ….. CCGrid '11 3 Cluster Computing Environment Compute cluster Frontend LAN LAN Storage cluster Compute Meta-Data Meta Node Manager Data Compute I/O Server L Data Node A Node Compute N I/O Server Node Node Data Compute I/O Server Node Node Data CCGrid '11 4 Trends for Computing Clusters in the Top 500 List (http://www.top500.org) Nov. 1996: 0/500 (0%) Nov. 2001: 43/500 (8.6%) Nov. 2006: 361/500 (72.2%) Jun. 1997: 1/500 (0.2%) Jun. 2002: 80/500 (16%) Jun. 2007: 373/500 (74.6%) Nov. 1997: 1/500 (0.2%) Nov. 2002: 93/500 (18.6%) Nov. 2007: 406/500 (81.2%) Jun. 1998: 1/500 (0.2%) Jun. 2003: 149/500 (29.8%) Jun. 2008: 400/500 (80.0%) Nov. 1998: 2/500 (0.4%) Nov. 2003: 208/500 (41.6%) Nov. 2008: 410/500 (82.0%) Jun. 1999: 6/500 (1.2%) Jun. 2004: 291/500 (58.2%) Jun. 2009: 410/500 (82.0%) Nov. 1999: 7/500 (1.4%) Nov. 2004: 294/500 (58.8%) Nov. 2009: 417/500 (83.4%) Jun. 2000: 11/500 (2.2%) Jun. 2005: 304/500 (60.8%) Jun. 2010: 424/500 (84.8%) Nov. 2000: 28/500 (5.6%) Nov. 2005: 360/500 (72.0%) Nov. 2010: 415/500 (83%) Jun. 2001: 33/500 (6.6%) Jun. 2006: 364/500 (72.8%) Jun. 2011: To be announced CCGrid '11 5 Grid Computing Environment Compute cluster Compute cluster LAN Frontend Frontend LAN LAN LAN Storage cluster WAN Storage cluster Compute Meta-Data Meta Compute Meta-Data Meta Node Manager Data Node Manager Data Compute L I/O Server Compute I/O Server Data L Data Node A Node Node A Node Compute N I/O Server Compute N I/O Server Data Node Node Node Node Data Compute I/O Server Compute I/O Server Data Node Node Node Node Data Compute cluster Frontend LAN LAN Storage cluster Compute Meta-Data Meta Node Manager Data Compute I/O Server L Data Node A Node Compute N I/O Server Node Node Data Compute I/O Server Node Node Data CCGrid '11 6 Multi-Tier Datacenters and Enterprise Computing Enterprise Multi-tier Datacenter Routers/ Application Database Servers Server Server Routers/ Application Database Servers Server Server Switch Switch Switch Routers/ Application Database Servers Server Server . Routers/ Application Database Servers Server Server Tier1 Tier2 Tier3 CCGrid '11 7 Integrated High-End Computing Environments Compute cluster Storage cluster Compute Meta-Data Meta Node Manager Data Compute I/O Server L Data Node A Node Frontend LAN Compute N I/O Server Node Node Data Compute I/O Server LAN/WAN Node Node Data Enterprise Multi-tier Datacenter for Visualization and Mining Routers/ Application Database Servers Server Server Routers/ Application Database Server Servers Switch Switch Server Switch Routers/ Application Database Servers Server Server . Routers/ Application Database Servers Server Server Tier1 Tier2 Tier3 CCGrid '11 8 Cloud Computing Environments VM VM Physical Machine Meta Meta-Data Local Storage Data VM VM Virtual FS Virtual I/O Server Data Physical Machine Local Storage LAN I/O Server Data I/O Server Data VM VM Physical Machine I/O Server Data VM VM Local Storage Physical Machine Local Storage CCGrid '11 9 Hadoop Architecture • Underlying Hadoop Distributed File System (HDFS) • Fault-tolerance by replicating data blocks • NameNode: stores information on data blocks • DataNodes: store blocks and host Map-reduce computation • JobTracker: track jobs and detect failure • Model scales but high amount of communication during intermediate phases CCGrid '11 10 Memcached Architecture System System Area Area Network Network Internet Web-frontend Memcached Database Servers Servers Servers • Distributed Caching Layer – Allows to aggregate spare memory from multiple nodes – General purpose • Typically used to cache database queries, results of API calls • Scalable model, but typical usage very network intensive CCGrid '11 11 Networking and I/O Requirements • Good System Area Networks with excellent performance (low latency, high bandwidth and low CPU utilization) for inter-processor communication (IPC) and I/O • Good Storage Area Networks high performance I/O • Good WAN connectivity in addition to intra-cluster SAN/LAN connectivity • Quality of Service (QoS) for interactive applications • RAS (Reliability, Availability, and Serviceability) • With low cost CCGrid '11 12 Major Components in Computing Systems • Hardware components P0 – Processing cores and memory Core0 Core1 Memory subsystem Core2 Core3 Processing – I/O bus or links Bottlenecks – Network adapters/switches P1 • Software components Core0 Core1 I Memory – Communication stack Core2 Core3 / O I/O Interface • Bottlenecks can artificially BottlenecksB u limit the network performance s the user perceives Network Adapter Network NetworkBottlenecks Switch CCGrid '11 13 Processing Bottlenecks in Traditional Protocols • Ex: TCP/IP, UDP/IP • Generic architecture for all networks P0 • Host processor handles almost all aspects Core0 Core1 Memory Core2 Core3 Processing of communication Bottlenecks P1 – Data buffering (copies on sender and Core0 Core1 I Memory receiver) Core2 Core3 / O – Data integrity (checksum) B u – Routing aspects (IP routing) s Network Adapter • Signaling between different layers – Hardware interrupt on packet arrival or Network Switch transmission – Software signals between different layers to handle protocol processing in different priority levels CCGrid '11 14 Bottlenecks in Traditional I/O Interfaces and Networks • Traditionally relied on bus-based P0 Core0 Core1 Memory technologies (last mile bottleneck) Core2 Core3 – E.g., PCI, PCI-X P1 Core0 Core1 – One bit per wire I Memory Core2 Core3 / – Performance increase through: O I/O Interface BottlenecksB • Increasing clock speed u • Increasing bus width s Network Adapter – Not scalable: • Cross talk between bits Network Switch • Skew between wires • Signal integrity makes it difficult to increase bus width significantly, especially for high clock speeds PCI 1990 33MHz/32bit: 1.05Gbps (shared bidirectional) 1998 (v1.0) 133MHz/64bit: 8.5Gbps (shared bidirectional) PCI-X 2003 (v2.0) 266-533MHz/64bit: 17Gbps (shared bidirectional) CCGrid '11 15 Bottlenecks on Traditional Networks P0 Core0 Core1 Memory • Network speeds saturated at Core2 Core3 around 1Gbps P1 Core0 Core1 I Memory – Features provided were limited Core2 Core3 / O – Commodity networks were not B u considered scalable enough for very s large-scale systems Network Adapter Network NetworkBottlenecks Switch Ethernet (1979 - ) 10 Mbit/sec Fast Ethernet (1993 -) 100 Mbit/sec Gigabit Ethernet (1995 -) 1000 Mbit /sec ATM (1995 -) 155/622/1024 Mbit/sec Myrinet (1993 -) 1 Gbit/sec Fibre Channel (1994 -) 1 Gbit/sec CCGrid '11 16 Motivation for InfiniBand and High-speed Ethernet • Industry Networking Standards • InfiniBand and High-speed Ethernet were introduced into the market to address these bottlenecks • InfiniBand aimed at all three bottlenecks (protocol processing, I/O bus, and network speed) • Ethernet aimed at directly handling the network speed bottleneck and relying on complementary technologies to alleviate the protocol processing and I/O bus bottlenecks CCGrid '11 17 Presentation Overview • Introduction • Why InfiniBand and High-speed Ethernet? • Overview of IB, HSE, their Convergence and Features • IB and HSE HW/SW Products and Installations • Sample Case Studies and Performance Numbers • Conclusions and Final Q&A CCGrid '11 18 IB Trade Association • IB Trade Association was formed with seven industry leaders (Compaq, Dell, HP, IBM, Intel, Microsoft, and Sun) • Goal: To design a scalable and high performance communication and I/O architecture by taking an integrated view of computing, networking, and storage technologies • Many other industry participated in the effort to define the IB architecture specification • IB Architecture (Volume 1, Version 1.0) was released to public on Oct 24, 2000 – Latest version 1.2.1 released January 2008 • http://www.infinibandta.org CCGrid '11 19 High-speed Ethernet Consortium (10GE/40GE/100GE) • 10GE Alliance formed by several industry leaders to take the Ethernet family to the next speed step • Goal: To achieve a scalable and high performance communication architecture while maintaining backward compatibility with Ethernet • http://www.ethernetalliance.org • 40-Gbps (Servers) and 100-Gbps Ethernet (Backbones, Switches, Routers): IEEE 802.3 WG • Energy-efficient and power-conscious protocols – On-the-fly link speed reduction for under-utilized links CCGrid '11 20 Tackling Communication Bottlenecks with IB and HSE • Network speed bottlenecks • Protocol processing bottlenecks • I/O interface bottlenecks CCGrid '11 21 Network Bottleneck Alleviation: InfiniBand (“Infinite Bandwidth”) and High-speed Ethernet (10/40/100 GE) • Bit serial
Recommended publications
  • What Is It and How We Use It

    What Is It and How We Use It

    Infiniband Overview What is it and how we use it What is Infiniband • Infiniband is a contraction of "Infinite Bandwidth" o can keep bundling links so there is no theoretical limit o Target design goal is to always be faster than the PCI bus. • Infiniband should not be the bottleneck. • Credit based flow control o data is never sent if receiver can not guarantee sufficient buffering What is Infiniband • Infiniband is a switched fabric network o low latency o high throughput o failover • Superset of VIA (Virtual Interface Architecture) o Infiniband o RoCE (RDMA over Converged Ethernet) o iWarp (Internet Wide Area RDMA Protocol) What is Infiniband • Serial traffic is split into incoming and outgoing relative to any port • Currently 5 data rates o Single Data Rate (SDR), 2.5Gbps o Double Data Rate (DDR), 5 Gbps o Quadruple Data Rate (QDR), 10 Gbps o Fourteen Data Rate (FDR), 14.0625 Gbps o Enhanced Data Rate (EDR) 25.78125 Gbps • Links can be bonded together, 1x, 4x, 8x and 12x HDR - High Data Rate NDR - Next Data Rate Infiniband Road Map (Infiniband Trade Association) What is Infiniband • SDR, DDR, and QDR use 8B/10B encoding o 10 bits carry 8 bits of data o data rate is 80% of signal rate • FDR and EDR use 64B/66B encoding o 66 bits carry 64 bits of data Signal Rate Latency SDR 200ns DDR 140ns QDR 100ns Hardware 2 Hardware vendors • Mellanox o bought Voltaire • Intel o bought Qlogic Infiniband business unit Need to standardize hardware. Mellanox and Qlogic cards work in different ways.
  • Cray XD1™ Supercomputer Release 1.3

    Cray XD1™ Supercomputer Release 1.3

    CRAY XD1 DATASHEET Cray XD1™ Supercomputer Release 1.3 ■ Purpose-built for HPC — delivers exceptional application The Cray XD1 supercomputer combines breakthrough performance interconnect, management and reconfigurable computing ■ Affordable power — designed for a broad range of HPC technologies to meet users’ demands for exceptional workloads and budgets performance, reliability and usability. Designed to meet ■ Linux, 32 and 64-bit x86 compatible — runs a wide variety of ISV the requirements of highly demanding HPC applications applications and open source codes in fields ranging from product design to weather prediction to ■ Simplified system administration — automates configuration and scientific research, the Cray XD1 system is an indispensable management functions tool for engineers and scientists to simulate and analyze ■ Highly reliable — monitors and maintains system health faster, solve more complex problems, and bring solutions ■ Scalable to hundreds of compute nodes — high bandwidth and to market sooner. low latency let applications scale Direct Connected Processor Architecture Cray XD1 System Highlights The Cray XD1 system is based on the Direct Connected Processor (DCP) architecture, harnessing many processors into a single, unified system to deliver new levels of application � � � ���� � � � Compute� � � Processors performance. ���� � Cray’s implementation of the DCP architecture optimizes message-passing applications by 12 AMD Opteron™ 64-bit single directly linking processors to each other through a high performance interconnect fabric, or dual core processors run eliminating shared memory contention and PCI bus bottlenecks. Linux and are organized as six nodes of 2 or 4-way SMPs to deliver up to 106 GFLOPS* per chassis. Matching memory and I/O performance removes bottlenecks and maximizes processor performance.
  • Merrimac – High-Performance and Highly-Efficient Scientific Computing with Streams

    Merrimac – High-Performance and Highly-Efficient Scientific Computing with Streams

    MERRIMAC – HIGH-PERFORMANCE AND HIGHLY-EFFICIENT SCIENTIFIC COMPUTING WITH STREAMS A DISSERTATION SUBMITTED TO THE DEPARTMENT OF ELECTRICAL ENGINEERING AND THE COMMITTEE ON GRADUATE STUDIES OF STANFORD UNIVERSITY IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY Mattan Erez May 2007 c Copyright by Mattan Erez 2007 All Rights Reserved ii I certify that I have read this dissertation and that, in my opinion, it is fully adequate in scope and quality as a dissertation for the degree of Doctor of Philosophy. (William J. Dally) Principal Adviser I certify that I have read this dissertation and that, in my opinion, it is fully adequate in scope and quality as a dissertation for the degree of Doctor of Philosophy. (Patrick M. Hanrahan) I certify that I have read this dissertation and that, in my opinion, it is fully adequate in scope and quality as a dissertation for the degree of Doctor of Philosophy. (Mendel Rosenblum) Approved for the University Committee on Graduate Studies. iii iv Abstract Advances in VLSI technology have made the raw ingredients for computation plentiful. Large numbers of fast functional units and large amounts of memory and bandwidth can be made efficient in terms of chip area, cost, and energy, however, high-performance com- puters realize only a small fraction of VLSI’s potential. This dissertation describes the Merrimac streaming supercomputer architecture and system. Merrimac has an integrated view of the applications, software system, compiler, and architecture. We will show how this approach leads to over an order of magnitude gains in performance per unit cost, unit power, and unit floor-space for scientific applications when compared to common scien- tific computers designed around clusters of commodity general-purpose processors.
  • What Is a Storage Area Network?

    What Is a Storage Area Network?

    8646_Barker_01_d.qxd 9/20/01 10:21 AM Page 3 CHAPTER1 What Storage Networking Is and What It Can Mean to You n this chapter we’ll learn more about: II What a storage area network is I What properties a storage area network must have, should have, and may have I The importance of software to storage area networks I Why information services professionals should be interested in storage networking I Information processing capabilities enabled by storage area networks I Some quirks in the vocabulary of storage networking What Is a Storage Area Network? According to the Storage Networking Industry Association (and who should know better?): A storage area network (SAN) is any high-performance network whose primary pur- pose is to enable storage devices to communicate with computer systems and with each other. We think that the most interesting things about this definition are what it doesn’t say: I It doesn’t say that a SAN’s only purpose is communication between computers and storage. Many organizations operate perfectly viable SANs that carry occa- sional administrative and other application traffic. I It doesn’t say that a SAN uses Fibre Channel or Ethernet or any other specific interconnect technology. A growing number of network technologies have archi- tectural and physical properties that make them suitable for use in SANs. 3 8646_Barker_01_d.qxd 9/20/01 10:21 AM Page 4 4 STORAGE AREA NETWORK ESSENTIALS I It doesn’t say what kind of storage devices are interconnected. Disk and tape drives, RAID subsystems, robotic libraries, and file servers are all being used pro- ductively in SAN environments today.
  • Fibre Channel Solution Guide

    Fibre Channel Solution Guide

    Fibre Channel Solutions Guide 2019 fibrechannel.org FIBRE CHANNEL Powering the next generation private, public, and hybrid cloud storage networks ABOUT THE FCIA The Fibre Channel Industry Association (FCIA) is a non-profit international organization whose sole purpose is to be the independent technology and marketing voice of the Fibre Channel industry. We are committed to helping member organizations promote and position Fibre Channel, and to providing a focal point for Fibre Channel information, standards advocacy, and education. CONTACT THE FCIA For more information: www.fibrechannel.org • [email protected] TABLE OF CONTENTS Foreword .........................................................................................3 FCIA President Introduction.............................................................4 The State of Fibre Channel by Storage Switzerland .......................6 Fibre Channel New Technologies: FC-NVMe-2 ............................... 7 The 2019 Fibre Channel Roadmap ................................................. 8 Fibre Channel’s Future is Bright in Media and Entertainment ......10 Securing Fibre Channel SANs with End-to-End Encryption ..........12 FOREWORD By Rupin Mohan, FCIA Marketing Chairman; Director R&D and CTO, Hewlett-Packard Enterprise It’s 2019, and Fibre Channel continues to remain the premier storage fabric connectivity protocol in today’s data centers. Fibre Channel is deployed by thousands of customers in their data centers around the world and 80–90% of all All-Flash storage arrays are connected to servers via Fibre Channel. Customers have recently made a considerable investment in Gen 6 (32GFC), and given the 4-5-year depreciation cycle, this equipment will continue to run critical business applications requiring reliable, fast and scalable storage infrastructure. The NVMe over Fibre Channel (FC-NVMe) standard is published, and we see products being announced and released in the market across the board.
  • IBM Flex System IB6131 Infiniband Switch User's Guide

    IBM Flex System IB6131 Infiniband Switch User's Guide

    IBM Flex System IB6131 InfiniBand Switch User’s Guide ii IBM Flex System IB6131 InfiniBand Switch User’s Guide IBM Flex System IB6131 InfiniBand Switch User’s Guide Note: Before using this information and the product it supports, read the general information in , ʺAppen‐ dix B: Noticesʺ on page 33, the Safety Information and Environmental Notices and Userʹs Guide docu‐ ments on the IBM Notices for Network Devices CD, and the Warranty Information document that comes with the product. First Edition February 2012 © Copyright IBM Corporation 2012. US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Contents Safety . vii Chapter1. Introduction . 1 Related documentation. 1 Notices and statements in this document . 3 Features and specifications. 3 Specifications . 4 Major components of the switch . 5 Chapter2. Installing the switch and basic setup. 7 Installing the IBM Flex system blade . 7 CMM . 7 Serial port access (Method 1) . 7 Configuration . 9 Configuration Wizard (Method 2) . 9 Cabling the switch . 15 Chapter3. LEDs and interfaces. 17 Port LEDs . 17 Switch status lights . 17 Power LED . 18 Fault LED . 18 Unit identification switch identifier LED . 18 RS‐232 interface through mini connector. 18 RJ‐45 Ethernet connector . 19 Configuring the IBM Flex System IB6131 InfiniBand switch . 19 Rerunning the Wizard. 19 Updating the switch software . 20 Chapter4. Connecting to the switch platform . 25 Starting an SSH connection to the switch (CLI) . 25 Starting a WebUI connection to the switch . 25 Managing the IBM Flex System IB6131 InfiniBand switch . 26 Chapter5. Solving problems . 27 Running POST .
  • Pubtex Output 2006.05.15:1001

    Pubtex Output 2006.05.15:1001

    Cray XD1™ Release Description Private S–2453–14 © 2006 Cray Inc. All Rights Reserved. Unpublished Private Information. This unpublished work is protected to trade secret, copyright and other laws. Except as permitted by contract or express written permission of Cray Inc., no part of this work or its content may be used, reproduced or disclosed in any form. U.S. GOVERNMENT RESTRICTED RIGHTS NOTICE The Computer Software is delivered as "Commercial Computer Software" as defined in DFARS 48 CFR 252.227-7014. All Computer Software and Computer Software Documentation acquired by or for the U.S. Government is provided with Restricted Rights. Use, duplication or disclosure by the U.S. Government is subject to the restrictions described in FAR 48 CFR 52.227-14 or DFARS 48 CFR 252.227-7014, as applicable. Technical Data acquired by or for the U.S. Government, if any, is provided with Limited Rights. Use, duplication or disclosure by the U.S. Government is subject to the restrictions described in FAR 48 CFR 52.227-14 or DFARS 48 CFR 252.227-7013, as applicable. Autotasking, Cray, Cray Channels, Cray Y-MP, GigaRing, LibSci, UNICOS and UNICOS/mk are federally registered trademarks and Active Manager, CCI, CCMT, CF77, CF90, CFT, CFT2, CFT77, ConCurrent Maintenance Tools, COS, Cray Ada, Cray Animation Theater, Cray APP, Cray Apprentice2, Cray C++ Compiling System, Cray C90, Cray C90D, Cray CF90, Cray EL, Cray Fortran Compiler, Cray J90, Cray J90se, Cray J916, Cray J932, Cray MTA, Cray MTA-2, Cray MTX, Cray NQS, Cray Research, Cray SeaStar, Cray S-MP,
  • Global MV Standards English

    Global MV Standards English

    www.visiononline.org www.emva.org www.jiia.org www.china-vision.org www.vdma.com/vision Member-supported trade associations promote the growth of the global vision and imaging industry. Standards development is key to the success of the industry and its trade groups help fund, maintain, manage and promote standards. In 2009, three leading vision associations, AIA, EMVA and JIIA began a cooperative initiative to coordinate the development of globally adopted vision standards. In 2015 they were joined by CMVU and VDMA-MV. This publication is one product of this cooperative effort. Version: April 2016 Copyright 2013, AIA, EMVA and JIIA. All rights reserved. Data within is intended as an information resource and no warranty for its use is given. Camera Link (including PoCL and PoCL-Lite), Camera Link HS, GigE Vision and USB3 Vision are the trademarks of AIA. GenICam is the trademark of EMVA. CoaXPress and IIDC2 are the trademarks of JIIA. FireWire is the trademark of Apple Inc. IEEE 1394 is the trademark of The 1394 Trade Association. USB is the trademark of USB Implementers Forum, Inc. All other names are trademarks or trade names of their respective companies. This is a comprehensive look at the various digital hardware and software interface standards used in machine vision and imaging. In the early days of machine vision, the industry adopted existing analog television standards such as CCIR or RS-170 for the interface between cameras and frame grabbers. The defining In the 1990s, digital technology became characteristics of today’s prevalent and a multitude of proprietary hardware and software interface solutions were used.
  • Infiniband Trade Association Integrators' List

    Infiniband Trade Association Integrators' List

    InfiniBand Trade Association Integrators’ List April 2019 © InfiniBand Trade Association SPACE IBTA InfiniBand Integrators' List April 2019 Manufacturer Description Model Type Speed FW SW ConnectX®-3 Pro VPI, FDR IB (56Gb/s) and 40/56GbE MLNX_OFED_LINUX 4.4- Mellanox MCX354A-FCCT HCA FDR 2.42.5000 Dual-port QSFP, PCIe3.0 x8 2.0.7.0 ConnectX®-4 VPI, EDR IB (100Gb/s) and 100GbE MLNX_OFED_LINUX 4.5- Mellanox MCX456A-ECAT HCA EDR 12.24.1000 Dual-port QSFP28, PCIe3.0 x16 1.0.1.0 ConnectX®-5 VPI, EDR IB (100Gb/s) and 100GbE MLNX_OFED_LINUX 4.5- Mellanox MCX556A-EDAT HCA EDR 16.24.1000 Dual-port QSFP28, PCIe4.0 x16 1.0.1.0 ConnectX®-6 VPI adapter card, HDR IB (200Gb/s) and 200GbE, MLNX_OFED_LINUX 4.5- Mellanox MCX653A-HCAT HCA HDR 20.25.0330 dual-port QSFP56, Socket Direct 2x PCIe3.0 x16 1.0.1.0 SwitchX®2 InfiniBand to Ethernet gateway, 36 QSFP ports, Mellanox MSX6036G-2SFS Switch FDR 9.4.5070 3.6.8010 Managed Switch Switch-IB 2 based EDR InfiniBand 1U Switch, 36 QSFP28 ports, Mellanox MSB7800-ES2F Switch EDR 15.1910.0618 3.7.1134 x86 dual core Mellanox® Quantum(TM) HDR InfiniBand Switch, 40 QSFP56 Mellanox MQM8700-HS2F Switch HDR 27.2000.1016 3.7.1942 ports, 2 Power Supplies (AC), x86 dual core Software Versions Diagnostic Software Operating System Cent OS 7.5.1804 ibutils2 MLNX OFED 4.5-1.0.1.0 Mellanox OFED MLNX OFED 4.5-1.0.1.0 Compliance Test Suite v. 1.0.52 Open MPII Open MPI 3.1.2 Benchmark Intel MPI Benchmarks Benchmarks Performed Test Plan Software Forge IBTA MOI Suite PingPong Gather Duration 2-5 Minutes PingPing Gatherv Sendrecv Scatter Conditions for Passing Testing Exchange Scatterv Link Width Link width is @ expected width - i.e.
  • The Gemini Network

    The Gemini Network

    The Gemini Network Rev 1.1 Cray Inc. © 2010 Cray Inc. All Rights Reserved. Unpublished Proprietary Information. This unpublished work is protected by trade secret, copyright and other laws. Except as permitted by contract or express written permission of Cray Inc., no part of this work or its content may be used, reproduced or disclosed in any form. Technical Data acquired by or for the U.S. Government, if any, is provided with Limited Rights. Use, duplication or disclosure by the U.S. Government is subject to the restrictions described in FAR 48 CFR 52.227-14 or DFARS 48 CFR 252.227-7013, as applicable. Autotasking, Cray, Cray Channels, Cray Y-MP, UNICOS and UNICOS/mk are federally registered trademarks and Active Manager, CCI, CCMT, CF77, CF90, CFT, CFT2, CFT77, ConCurrent Maintenance Tools, COS, Cray Ada, Cray Animation Theater, Cray APP, Cray Apprentice2, Cray C90, Cray C90D, Cray C++ Compiling System, Cray CF90, Cray EL, Cray Fortran Compiler, Cray J90, Cray J90se, Cray J916, Cray J932, Cray MTA, Cray MTA-2, Cray MTX, Cray NQS, Cray Research, Cray SeaStar, Cray SeaStar2, Cray SeaStar2+, Cray SHMEM, Cray S-MP, Cray SSD-T90, Cray SuperCluster, Cray SV1, Cray SV1ex, Cray SX-5, Cray SX-6, Cray T90, Cray T916, Cray T932, Cray T3D, Cray T3D MC, Cray T3D MCA, Cray T3D SC, Cray T3E, Cray Threadstorm, Cray UNICOS, Cray X1, Cray X1E, Cray X2, Cray XD1, Cray X-MP, Cray XMS, Cray XMT, Cray XR1, Cray XT, Cray XT3, Cray XT4, Cray XT5, Cray XT5h, Cray Y-MP EL, Cray-1, Cray-2, Cray-3, CrayDoc, CrayLink, Cray-MP, CrayPacs, CrayPat, CrayPort, Cray/REELlibrarian, CraySoft, CrayTutor, CRInform, CRI/TurboKiva, CSIM, CVT, Delivering the power…, Dgauss, Docview, EMDS, GigaRing, HEXAR, HSX, IOS, ISP/Superlink, LibSci, MPP Apprentice, ND Series Network Disk Array, Network Queuing Environment, Network Queuing Tools, OLNET, RapidArray, RQS, SEGLDR, SMARTE, SSD, SUPERLINK, System Maintenance and Remote Testing Environment, Trusted UNICOS, TurboKiva, UNICOS MAX, UNICOS/lc, and UNICOS/mp are trademarks of Cray Inc.
  • Infiniband Technology Overview

    Infiniband Technology Overview

    InfiniBand Technology Overview Dror Goldenberg, Mellanox Technologies SNIA Legal Notice The material contained in this tutorial is copyrighted by the SNIA. Member companies and individuals may use this material in presentations and literature under the following conditions: Any slide or slides used must be reproduced without modification The SNIA must be acknowledged as source of any material used in the body of any document containing material from these presentations. This presentation is a project of the SNIA Education Committee. InfiniBand Technology Overview 2 © 2008 Storage Networking Industry Association. All Rights Reserved. Abstract InfiniBand Technology Overview The InfiniBand architecture brings fabric consolidation to the data center. Storage networking can concurrently run with clustering, communication and management fabrics over the same infrastructure, preserving the behavior of multiple fabrics. The tutorial provides an overview of the InfiniBand architecture including discussion of High Speed – Low Latency, Channel I/O, QoS scheduling, partitioning, high availability and protocol offload. InfiniBand based storage protocols, iSER (iSCSI RDMA Protocol), NFS over RDMA and SCSI RDMA Protocol (SRP), are introduced and compared with alternative storage protocols, such as iSCSI and FCP. The tutorial further enumerates value-add features that the InfiniBand brings to clustered storage, such as atomic operations and end to end data integrity. Learning Objectives: Understand the InfiniBand architecture and feature set. Understand the benefits of InfiniBand for networked storage. Understand the standard InfiniBand storage protocols. InfiniBand Technology Overview 3 © 2008 Storage Networking Industry Association. All Rights Reserved. Agenda Motivation and General Overview Protocol Stack Layers Storage Protocols over InfiniBand Benefits InfiniBand Technology Overview 4 © 2008 Storage Networking Industry Association.
  • Opencapi, Gen-Z, CCIX: Technology Overview, Trends, and Alignments

    Opencapi, Gen-Z, CCIX: Technology Overview, Trends, and Alignments

    OpenCAPI, Gen-Z, CCIX: Technology Overview, Trends, and Alignments BRAD BENTON| AUGUST 8, 2017 Agenda 2 | OpenSHMEM Workshop | August 8, 2017 AGENDA Overview and motivation for new system interconnects Technical overview of the new, proposed technologies Emerging trends Paths to convergence? 3 | OpenSHMEM Workshop | August 8, 2017 Overview and Motivation 4 | OpenSHMEM Workshop | August 8, 2017 NEWLY EMERGING BUS/INTERCONNECT STANDARDS Three new bus/interconnect standards announced in 2016 CCIX: Cache Coherent Interconnect for Accelerators ‒ Formed May, 2016 ‒ Founding members include: AMD, ARM, Huawei, IBM, Mellanox, Xilinx ‒ ccixconsortium.com Gen-Z ‒ Formed August, 2016 ‒ Founding members include: AMD, ARM, Cray, Dell EMC, HPE, IBM, Mellanox, Micron, Xilinx ‒ genzconsortium.org OpenCAPI: Open Coherent Accelerator Processor Interface ‒ Formed September, 2016 ‒ Founding members include: AMD, Google, IBM, Mellanox, Micron ‒ opencapi.org 5 5 | OpenSHMEM Workshop | August 8, 2017 NEWLY EMERGING BUS/INTERCONNECT STANDARDS Motivations for these new standards Tighter coupling between processors and accelerators (GPUs, FPGAs, etc.) ‒ unified, virtual memory address space ‒ reduce data movement and avoid data copies to/from accelerators ‒ enables sharing of pointer-based data structures w/o the need for deep copies Open, non-proprietary standards-based solutions Higher bandwidth solutions ‒ 25Gbs and above vs. 16Gbs forPCIe-Gen4 Better exploitation of new and emerging memory/storage technologies ‒ NVMe ‒ Storage class memory (SCM),