InfiniBand Architecture: Illuminata® Bridge Over Troubled Waters That is a bad bridge which is shorter than the stream. — German proverb Research Note The PCI peripheral expansion bus has had a long and illustrious history. Since its inception in 1991, system vendors and users have embraced it like few technical stan- dards before or since. PCI provides a substantial volume of the I/O bandwidth and peripheral connectivity across the range of RISC to CISC; PC to enterprise server; proprietary to commodity. User requirements at the advent of the 21st century, David Pendery however, have rapidly evolved. Not only has computer performance advanced enor- Jonathan Eunice mously, the very landscape of IT use and connectivity has changed. The PCI standard 27 April 2000 we have converged and relied upon for close to a decade is being rapidly outstripped by the demands of ever larger databases, transaction loads, and network user bases. The bridge is beginning to look shorter than the stream. Fortunately, help is on the way. The InfiniBand™ Architecture is the industry’s answer to the growing I/O problem. InfiniBand replaces the bus-based PCI with a high-band- width (multiple gigabytes per second) switched network topology, and shifts I/O control responsibility from processors to intelligent I/O engines commonly known as channels. These approaches have long enabled the world’s largest servers, now brought down to address virtually every server. InfiniBand is not yet a product, nor even really a standard. The first full specification won’t be available until this summer, with the first products appearing in 2001. Initial indications, however, are greatly encouraging. InfiniBand is the right technological advance, emerging at the right time and for the right reasons. To employ a bit of adolescent patois, InfiniBand rocks. ! ON OPENING SO Copyright © 2000 Illuminata, Inc. Illuminata, Inc. 187 Main Street Nashua, NH 03060 Licensed to InfiniBand Trade Ass'n 603.598.0099 603.598.0199 f www.illuminata.com Web Use Only - Do Not Reproduce 2 Presto, Change-O ISA’s maximum 10 MBps and EISA’s 33 MBps data transfer rates. As a commodity standard, it minimizes When PCI established itself in the early 1990s, 66 MHz cost to achieve high shipment volumes. Even so, PCI processors and 10 Mbps networks were fast. 0.8 micron has neatly outperformed virtually all alternatives, CMOS semiconductor fabrication was state of the art. including those quite proprietary and specialized. Early transaction processing benchmarks churned out a Then, in a classic case of volume sales providing the whopping 54 transactions per minute.1 Data ware- investment dollars needed to move a heretofore housing had just been invented. Client-server applica- commodity product upmarket, PCI has dramatically tions and deployments were increasing, but only the extended its reach. Enhanced versions have doubled digerati had email, and the Internet as we know it was both clock speed and bus width, making 264 MBps still years distant. easily achieved today, with 500+ MBps options avail- What a difference a decade makes! Today, multi- able. The HotPlug PCI extension made PCI suitable for terabyte databases running on clustered servers, if not high availability servers, and its CompactPCI deriva- exactly commonplace, are a reality in many shops. tive has driven into embedded systems and telco gear. Storage has been decoupled from the server, and often Although its attributes promised it a long life, PCI’s extended over a storage-optimized network (SAN). very architecture is ultimately limiting. Intel’s Pentium III Xeons, now the workhorse of PCI is built upon that simple connectivity structure, servers not just PCs, are fabbed at 0.18 micron and run the parallel bus. The simple, economical bus structure at 800 MHz; 0.13 micron, 1 GHz chips are on the way. has been at the base of so many electronic products for The top TPC-C server does 135,815 transactions per so long that it’s virtually taken for granted. Yet busses minute, and the Internet is now the workshop of IT. have inherent drawbacks: These are the new reality, driving ever-higher user • Disorderly contention for resources by periph- expectations. Fast, unencumbered I/O is the lifeblood erals, memory, and CPUs. Disorder breeds ineffi- of this evolving corpus. Never has such variety of I/O ciency and suboptimal performance. been required to link such scale of hardware and soft- • Vexing failure modes. Not only is the bus a poten- ware in such transparent and accelerated ways. And tial single point of system failure, failure isolation yet, never before have the incumbent I/O technologies is difficult or impossible. If one attached card fails, been so outstripped by processor capabilities. it can cause the entire system to fail. Worse, discovering which card caused the failure is at best PCI = Problematic Computing Interface? a hit-or-miss proposition—a misery in a world Introduced in response to a morass of incompatible needing high availability. peripheral connectivity and I/O options of a decade • Severe physical stipulations and limitations. As ago, PCI has been a blessing. Over time it expunged the bus length increases to accommodate more, or alphabet-soup that was AT/ISA, EISA, HP-PB, MCA, more widely dispersed, expansion devices, VME, NuBus, SBus, and TurboChannel, among others. signaling properties become less stable. The same It ushered in a long period of wide industry acceptance thing is true for clock rates. The faster the modu- of a single standard, and thus a stability and predict- lation, the shorter the feasible bus, and the fewer ability that made both product development and selec- peripheral interconnects are possible. In the tion pleasingly straightforward. extreme case, the 133 MHz defined for PCI-X, there can only be a single connector per bus! PCI not only standardized I/O attributes, it enabled 2 high bandwidth. Its initial 133 MBps may seem PCI’s shared structure cannot keep up on a perfor- modest today, but it greatly outpaced then-standard mance basis, nor are its manageability and availability 1. The first TPC-C result, published in 1992. attributes acceptable. As next-generation computing 2. Megabytes per second. Bandwidth figures are nominal, platforms are being planned and implemented, PCI will not typical. Such naive peak rates don’t consider practical gradually be left behind, as antiquated as 66 MHz slowdowns such as contention and protocol overhead. microprocessors and 40 MB disk drives. Licensed to InfiniBand Trade Ass'n Web Use Only - Do Not Reproduce 3 Incremental Upgrades community. But this is to be expected. The free market is contentious by nature. And, as they say, you can’t One could continue to improve PCI a bit, or work make an omelette without breaking a few eggs. At the around its limitations. Servers needing both high band- end of the day, these participants know that the width and large numbers of expansion slots, for customer uptake rate for their next-generation servers example, are often outfitted with multiple, independent depends on solving I/O bottlenecks, and on not PCI buses. This comes at a cost, of course, but averts an creating a divisive standards war. Thus while disagree- immediate capacity crisis. ments they may have, they are all highly motivated to The latest PCI-X revision goes further, cleaning up the find a common and standard solution. electrical signal definitions to drive towards 1 GBps IBTA leaders (officially, “Steering Members”) IBM, 3 (133 MHz x 64 bits). It’s a significant and promising Intel, Compaq, Hewlett-Packard, Dell, Microsoft, and extension that will extend PCI’s life by several years. Sun Microsystems reason that it’s better to have a Even improvements as extensive as PCI-X, however, smaller group get something practical and effective out have ever diminishing returns. The writing is on the the door than to hear everyone’s wishlist. In addition to wall. Despite PCI’s notable run of success, and the fact the Steering Members, Sponsoring Members include that it will remain with us for years to come, its ulti- 3Com, Adaptec, Cisco, Fujitsu-Siemens, Hitachi, mate headroom is limited. Bus architectures are funda- Lucent, NEC and Nortel Networks. It’s a potent brain mentally outpaced by our users’ and applications’ trust, among them the owners of the best I/O technol- voracious need for data, and thus high rates of I/O. ogies and intellectual property in the industry. Rather than more patches, what we now need is a jump as dramatic as PCI was when it was first introduced. As The Goods Mitch Shults, Intel’s point man on I/O strategies says, “the industry has got to move to some fundamentally InfiniBand is the cavalry to the rescue, the I/O standard new architecture.” Enter InfiniBand. and workhorse emerging for the new generation. So what exactly is it? On the Way to IBTA InfiniBand is a network approach to I/O. A system The road to a future I/O standard has been rocky. Even connects to the I/O “fabric” with one or more Host for PCI, vendors were reluctant to give up their favored Channel Adapters (HCAs). Devices, such as storage and proprietary options. Sun for example, while it has network controllers, would attach to the fabric with a supported PCI, to this day favors its own SBus design Target Channel Adapter (TCA). InfiniBand adapters in its premium servers. But the vastly better economics (generically, CAs) are addressed by IPv6 addresses, just of a single standard, both for IT producers and as any other network node might be. consumers, has won the day. The “fabric” concept may seem abstract to someone The once-divergent groups such as NGIO (Next who’s used to fitting a card in a slot, but it’s exactly Generation I/O, led by Intel) and Future I/O (led by what happens on any other network, whether of the IBM, Compaq, and HP) cast their fates together in traditional LAN/WAN/Internet variety, or the storage August 1999, a move that led to the foundation of the area networks (SANs) now rapidly entering data InfiniBand Trade Association (IBTA).