Infiniband FPGA Project

ISU SENIOR DESIGN – TEAM MAY09-04

InfiniBand FPGA Project Final Report

Matthew Wall CprE, Team Lead Ryan Schwarzkopf EE, Communication Coordinator Tim Prince CprE Alex Burds CprE Xiang Li EE Rachel Ayoroa CprE

Troy Benjegerdes Client Dr. Morris Chang Faculty Advisor

May 4, 2009

DISCLAIMER: This document was developed as part of the requirements of an electrical and computer engineering course at Iowa State University, Ames, Iowa. The document does not constitute a professional engineering design or a professional land surveying document. Although the information is intended to be accurate, the associated students, faculty, and Iowa State University make no claims, promises, or guarantees about the accuracy, completeness, quality, or adequacy of the information. Document users shall ensure that any such use does not violate any laws with regard to professional licensing and certification requirements. Such use includes any work resulting from this student prepared document that is required to be under the responsible charge of a licensed engineer or surveyor. This document is copyrighted by the students who produced the document and the associated faculty advisors. No part may be reproduced without the written permission of the senior design course coordinator. CONTENTS

InfiniBand FPGA Project 2 NFR1. INTRODUCTION

he InfiniBand architecture is a multi-gigabit, low-latency serial communications interconnect developed by the InfiniBand Trade TAssociation, a consortium of IBM, Cisco, Intel, and other corporations. It is used primarily in high-performance computing, and includes create an open-source hardware facilities for centralized management, data implementation of the InfiniBand rates as fast as 96 gigabits per second, and architecture. quality of service, among others. Last year’s project succeeded in laying the foundation for the implementation of the 1.1 PROBLEM STATEMENT InfiniBand link layer. This year’s project sought to complete the link layer Currently, only proprietary implementations implementation and develop the physical of InfiniBand hardware exist, limiting users to layer. a handful of vendors and limiting further research into the InfiniBand architecture by independent researchers.

Our client, Troy Benjegerdes, of the Scalable Computing Laboratory (SCL) at Ames Laboratory would like to be able to perform experiments on existing InfiniBand hardware, such as injecting errors into the data stream, as well as experiment with the effect of various modifications to portions of the specification.

1.2 APPROACH

To address the lack of open InfiniBand implementations, this project endeavors to InfiniBand FPGA Project 3 NFR2. PROJECT PLANNING 2.1.1 LINK LAYER uring the previous semester, our primary concern was designing The link layer provides a number of services our system and planning the for the upper layers, including data development cycle. This section verification and error detection, flow control Ddescribes the objectives of our project, our and buffering, among others. The previous implementation schedule, project resources, project provided the groundwork for this and so on. layer and our project built upon their foundation.

2.1 SYSTEM DESCRIPTION 2.1.2 PHYSICAL LAYER

Error: Reference source not found depicts The physical layer is primarily responsible for the top-level block diagram of our system. converting the symbols sent by the link layer This project initially planned to implement a into physical signals for communicating with complete InfiniBand switch, which would other InfiniBand devices. To that end, it is include the layers in the diagram, as well as a also responsible for configuring the link third switch controller layer. Through the between InfiniBand devices, maintaining the development process the immensity of link state, detecting symbol errors, and so on. completing this task became apparent and our scope was scaled back to include just the The detailed design of the link and link and physical layers. physical layers is discussed in sections 4.2 and 4.3, respectively.

InfiniBand FPGA Project 4 2.2 REQUIREMENTS ANALYSIS NFR7. The system shall be distributed under an open-source One of the first tasks addressed during the license. planning phase was to perform a requirements analysis of our system. The Rationale: To facilitate research, we would majority of our requirements focus on like the system to be as open as possible. following the InfiniBand specifications, and Using an open-source license ensures that the ensuring that our end product is as free and system can be used and modified freely. open as possible. NFR8. The system should use only 2.2.1 FUNCTIONAL REQUIREMENTS use standard VHDL libraries.

NFR3. The system shall provide Rationale: Using non-standard VHDL libraries InfiniBand compliant input and output has the effect of limiting development to a signals. specific vendor. By using only standard VHDL libraries, we can remain vendor-agnostic and Rationale: This requirement follows from the ensure that the system is as open as possible. ultimate objective of this project: to provide a complete hardware implementation of the NFR9. The system should be InfiniBand architecture. compatible with open-source VHDL simulators. NFR4. The system shall be capable of transmitting and receiving packets Rationale: By attempting to maintain according to the InfiniBand compatibility with open-source simulators, we can further distance ourselves from being specifications. tied to a particular vendor. Rationale: See rationale for FR01.

NFR5. The system should recognize and handle all errors defined by the InfiniBand specifications.

Rationale: See rationale for FR01.

5.1.1 NON-FUNCTIONAL REQUIREMENTS

NFR6. The system shall be implemented using VHDL.

Rationale: The client requested that the system be written in VHDL. Furthermore, using VHDL greatly simplifies the design of digital circuits as compared to schematic- based design. InfiniBand FPGA Project 5 9.1 PROJECT RESOURCES 9.1.2 OTHER RESOURCES Our project has had access to a number of resources which aided in our development In addition to our human resources, we also process. Those resources are listed here. had access to laboratories, equipment and software provided by the department. The 9.1.1 HUMAN RESOURCES department labs provided us with Xilinx ISE for development and synthesis, and The table below lists all the human resources ModelSim PE for simulation testing. available to our project. In addition to these resources, Dr. Name Role Jones graciously provided us with access to a Matthew Wall Project lead, link Virtex-5-based Xilinx ML507 development layer developer board to replace our original Virtex-II-based Alex Burds Link layer developer boards. The department also allowed us Ryan Schwarzkopf Communications access to a giga-sample oscilloscope and SMA coordinator, tester cables and connectors which proved Tim Prince Physical layer invaluable for debugging our gigabit developer transceivers. Xiang Li Tester Rachel Ayoroa Link layer developer Troy Benjegerdes Client 9.2 SCHEDULE Dr. Morris Chang Faculty advisor Jason Boyd Lab coordinator In the previous semester, our project Dr. Ahmed Kamal Senior design identified a possible schedule for instructor development. During this semester, as work Dr. Phillip Jones Equipment progressed, it became apparent that the contributor original schedule was infeasible. To that end, we constructed a new schedule with milestones based on the different tasks we identified to complete and the milestones we had already completed.

InfiniBand FPGA Project 6 9.2.1 WORK BREAKDOWN

When we constructed our original schedule, we broke the work down into a number of Figure 1 – Schedule milestones tasks. Error: Reference source not found shows how work was broken down into tasks.

When the original schedule was dropped, we constructed a new milestone- based schedule, with each milestone building upon the previous. The revised milestone schedule is listed below.

9.2.2 MILESTONES

When the new schedule was created, we identified a series of milestones and approximate feasible dates for the completion of those milestones. The new schedule is given on the right in Figure 1.

InfiniBand FPGA Project 7 9.3 DELIVERABLES

The deliverables for our project are:

1. A CD containing:

a. Source code

b. Compilation instructions

c. User’s manual

2. Code submitted to OpenCores.org and Google Code.

InfiniBand FPGA Project 8 NFR10. SYSTEM DESIGN we only had one board to work with at this time. he design of our system consists of several major components shown in Error: Reference source not found. The bulk of the project was found in 10.1.1 SYSTEM INTERFACE Tthe InfiniBand HDL Implementation The IP core has two external interfaces—the containing the link and physical layers. These physical interface and the link layer interface. layers make up the IP core of described by InfiniBand specification. Sections Link Layer The physical layer interface consists of Architecture and Physical Layer Architecture two low-voltage differential signals (LVDS) will detail the design and functionality of which carry the transmit and receive signals these layers in greater detail. from RocketIO.

The Xilinx FPGA chip is the The link layer interface consists of two programmable Virtex-5 board that we loaded virtual lanes, number VL0 and VL15, which our VHDL design onto. This specific board provide packet buffering between the link that is used for loading our InfiniBand and upper layers. implementation should be easily interchangeable so that our open-source InfiniBand HDL Implementation is usable by many different people and many different boards.

The Xilinx RocketIO transceiver is the adapter we used to establish connections between boards programmed with our InfiniBand HDL Implementation, or in our case, establish a loopback to itself. The RocketIO transceiver has a specific wrapper allowing it to interface with our InfiniBand HDL Implementation, but this wrapper could be rewritten to accommodate interfacing with other transceivers to make this IP core more usable for a wide range of developers. The RocketIO transceiver and wrapper will be examined in further detail in section RocketIO Interface.

The fabric just represents the external connections that exist between this FPGA chip and any other InfiniBand implementations or devices that could conceivably be connected to ours. Within the scope this project, this fabric was limited to a serial loopback to our Virtex-5 board since

InfiniBand FPGA Project 9 10.2 LINK LAYER ARCHITECTURE

As described in section 3.1.1, the link layer is responsible for providing error detection, flow control, buffering, and other services to the upper layers. This section describes how these services are designed and implemented. These components are described in 10.2.1 TOP-LEVEL ARCHITECTURE detail in the sections that follow.

A top-level design of the link layer is shown in Error: Reference source not found. There are 10.2.2 LINK STATE MACHINE nine main components of the link layer: The link state machine is used to control  Link State Machine which state the link layer is in. In order for either link packets or data packets to be  Packet Receiver State Machine processed, the link state machine must be initialized and pass all checks (enables,  Data Packet Check Machine triggers, etc) before processing a packet. Error: Reference source not found shows the  Link Packet Check Machine state transitions and what functionality is enabled when the link layer is brought to  Virtual Lanes each state. For a complete description of the link state machine, see [ 1 ].  Virtual Lane Selector 10.2.2.1 LINK STATE MACHINE INPUTS  Link Packet Generator  Reset – When asserted, link layer will  Packet Priority Selector return to initialize state. RemoteInit – A link packet with the flow control initialize OP code has been received and has passed the checks of the link packet check machine.

 CPortState – A value that indicates commands from management to change the port state. There are three valid states.

• Down – Connection between physical layer and link layer current inactive.

• Arm – Connection between physical layer and link layer has been initialized. InfiniBand FPGA Project 10 • Active – Ongoing connection between physical layer and link layer.

 ActiveEnable – Prevents Active state of CPortState from occurring too quickly. ActiveEnable will be asserted when a link packet with a normal OP code is received.

 ActiveTrigger – Allows the transition between Arm and Active states. ActiveTrigger only deals with non- VL15 packets that pass the VCRC check.

 LinkDownTimeout – A timeout that indicates that the Link State Machine has been down for a period of time that causes the link to transition to the LinkDownState.

 PhyLink – When asserted, notified link layer that the physical layer is active. When not asserted, link layer stays in initial state. packets received from the physical 10.2.2.2 LINK STATE MACHINE OUTPUTS layer are discarded.  DataPktXmitEnable - Indicates the link layer’s action with respect to  SMPEnable - Indicates the link layer’s transmission of non Subnet action with respect to transmission Management Packets (SMP) data and reception of subnet management packets. When true, transmission of packets. When true, transmission and non-SMP data packets is enabled. reception of SMPs are enabled. When When False, non-SMP data packets false, SMPs submitted to link layer for submitted to link layer for transmission or reception is transmission are discarded. discarded.

 DataPktRcvEnable - Indicates the link  LinkPktEnable - Indicates the link layer’s action with respect to layer’s action with respect to reception of non-SMP data packets transmission and reception of link from the physical layer. When True, packets. When true, transmission and reception of non-SMP data packets is reception of link packets are enabled. enabled. When false, non-SMP data When false, link packets are not

InfiniBand FPGA Project 11 generated by the link layer and any  Arm – Responsible for receiving link packets received are discarded. packet info

 PortState - The value of the PortState  Active – Responsible for processing component of the PortInfo attribute. packet info Valid values are “Down”, "Initialize”, “Arm” and “Active”. 10.2.4 DATA PACKET CHECK MACHINE

10.2.3 PACKET RECEIVER STATE MACHINE The data packet check machine performs checks on incoming data packets. The first The packet receiver state machine check is the variant and invariant CRC to determines the state of the incoming data determine if the link packet maintained data stream from the physical layer. The module integrity during transmission. The next check determines the type of packet incoming so determines if a packet marked for virtual lane that the packet check machines and other 15 is in fact a subnet management packet. The modules can respond to the packet type. If an length check determines if the packet length error occurs in the data stream or a stream is matches the length field in the header. marked bad by the physical layer, an error is Finally, the destination local identifier check indicated and the incoming packet is determines if the packet was destined for the dropped. The state diagram is shown in current port. If any errors are indicated by Error: Reference source not found. For a the module, the system will drop the packet. complete description of the packet receiver For a complete description of the data packet state machine, see [ 1 ]. check machine, see [ 1 ].

10.2.3.1 PACKET RECEIVER STATE MACHINE 10.2.5 LINK PACKET CHECK MACHINE INPUTS  Reset – An internal signal to reset the The link packet check machine performs checks on incoming link packets. The first interface check is the CRC to determine if the link  PhyLink – The physical link status packet maintained data integrity during from the physical layer transmission. The next check is the length check to see if the packet is six bytes long.  RcvStream – Input from physical layer Finally, the virtual lane check determines if with packet type information the destination virtual lane of the packet exists on the port. If any errors are indicated  PortState – Value of PortState by the module, the system will drop the component of PortInfo packet. For a complete description of the link packet check machine, see [ 1 ].  Down – Indicates to the link layer that the physical layer is down

 Initialize – Indicates there are packets to be received and initializes all starting values

InfiniBand FPGA Project 12 10.2.6 VIRTUAL LANES

A virtual lane (VL) provides buffering for the link layer. There can be up to 16 virtual lanes in an InfiniBand port, 15 data virtual lanes numbered 0-14 and one reserved for subnet management packets numbered 15. There must be a minimum of one data virtual lane and VL15. A virtual lane consists of a circular buffer with three indexes; read, write, and temp write. The temp write index indicates where bytes from the incoming data stream are written to. The write index is set to the temp write index once the incoming packet is confirmed valid by the data packet check machine. If the packet has an error then the temp write index is set to the write index, dropping the packet. When a virtual lane receives a link packet, it is stored on a separate buffer until the packet is validated and the packet can be consumed by the virtual lane. The procedure to using a link packet is described in the next section.

10.2.6.1 FLOW CONTROL Flow Control is implemented in the link layer to ensure that no packets need to be Let CR represent the total blocks sent retransmitted due to buffer overflow on the since link initialization plus the number of receiving end. VL15 is not subject to flow blocks in the data packet to be transmitted. control. Each virtual lane keeps track of the Let CL represent the last FCCL received in a Adjusted Blocks Received (ABR) and Flow link packet. If (CL-CR) modulo 4096 <= 2048, Control Total Blocks Sent (FCTBS). FCTBS is then the data packet may be transmitted by the number of blocks (64 bytes) sent by the the virtual lane. If not, the packet must wait virtual lane since link initialization. ABR is set until the condition becomes true. For a to the value of FCTBS for each link packet complete description of flow control, see [ 1 ]. received by the virtual lane, and then incremented by number of blocks received 10.2.7 VIRTUAL LANE SELECTOR for each data packet received by the virtual lane. When a link packet is transmitted, the The virtual lane selector routes the incoming Flow Control Credit Limit (FCCL) is calculated data stream to the appropriate virtual lane by taking the ABR plus the number of empty based on the packet header. The InfiniBand blocks left on the virtual lane up to 2048. specifications dictate that the first four bits of an InfiniBand packet is the virtual lane number. When a packet start delimiter is

InfiniBand FPGA Project 13 received from the physical layer, the virtual packet length indicated in the packet header. lane selector will read the virtual lane Finally, the module sends a packet end number from the first byte received and delimiter. enable write on the virtual lane indicated until a packet end delimiter is received. If the virtual lane indicated by the incoming packet does not exist on the port, then the packet is ignored. Link packets are routed to all data virtual lanes on the port.

10.2.8 LINK PACKET GENERATOR

The link packet generator creates a flow control packet to be sent to the device on the other end of the link. Each data virtual lane has a link packet generator. A link packet is generated whenever indicated to by the associated virtual lane, which is at least every 65,536 clock cycles. The generator receives the FCCL and FCTBS fields from the associated virtual lane along with the VL number. The link packet is then appended with a 16 bit CRC. Once a packet is generated the module indicates to the packet priority selector that a packet is ready to be transmitted. Link packets are not generated if not enabled by the link state machine.

10.2.9 PACKET PRIORITY SELECTOR

The packet priority selector feeds transmitting packets to the physical layer from multiple sources. If no sources have data to transmit the module remains idle. If one or more sources have data to transmit then the packet priority selector first selects which source to service. Link packets have first priority followed by subnet management packets and finally data packets. If more than one virtual lane exists in the architecture, then they are serviced in a round robin manner. Once a source is selector the module sends a start packet delimiter to the physical layer. The module then sends the source data to the physical layer. The packet priority selector counts the number of bytes sent and stops sending once the count matches the InfiniBand FPGA Project 14 10.3 PHYSICAL LAYER ARCHITECTURE

As described in section 3.1.2, the physical layer is responsible for maintaining the state of the link and converting data from the link layer into physical signals compatible with other InfiniBand devices. This section describes how these services are implemented.

10.3.1 TOP-LEVEL ARCHITECTURE

A top-level diagram of the physical layer is given in Error: Reference source not found. There are three main components of the physical layer:

 Link/Physical layer

 RocketIO wrapper

 RocketIO transceiver.

These components are described in detail in the sections that follow.

InfiniBand FPGA Project 15 10.3.2 LINK/PHYSICAL LAYER 10.3.2.1 CONTROL SEQUENCE GENERATORS The control sequence generators are three The Link/Physical layer constitutes the components which generate the various majority of the logic in the physical layer. It physical control sequences defined by the includes the logic for configuring the link InfiniBand specifications. These control between InfiniBand devices and maintaining sequences include: the link once configured.  TS1 control sequence The internals of the link/physical layer are depicted in the diagram in Error:  TS2 control sequence Reference source not found. The majority of the logic is implemented by the link training  skip sequence state machine (LTSM), which keeps track of the current link status and determines which  heartbeat sequence. data should be sent at any given time, and controls which control sequence generators All four of these sequences are currently are active. implemented by a simple lookup table, except for the heartbeat sequence, which is not currently implemented since we do not implement the high-speed signaling option. For a detailed description of each control sequence, see [ 1 ].

10.3.2.2 CONTROL SEQUENCE DETECTORS For each implemented sequence generator, there is also a corresponding sequence detector. Since our current implementation only implements single data rate communication and 1X links, these components also amount to a simple lookup table to verify that the received packet matches what is expected.

10.3.2.3 LINK TRAINING STATE MACHINE The link training state machine is like the control center of the link physical layer. It uses the signals provided by RocketIO and the control sequence detectors to maintain the state of the link. Error: Reference source not found gives a top-level state diagram of the state machine, and the states are described in the following sections. For a complete description of the LTSM, see [ 1 ].

InfiniBand FPGA Project 16 10.3.2.3.1 POLLING SUPER-STATE FPGAs. It is instantiated in Xilinx ISE using their coregen software. However, RocketIO is In this super-state, the link/physical layer a proprietary IP core. We also developed the periodically sends TS1 control sequences to RocketIO wrapper to avoid tying ourselves to the other endpoint, and waits for incoming a specific vendor. TS1 sequences. When a TS1 sequence is detected, the LTSM moves on to configuring 10.3.3.2 ROCKETIO WRAPPER the link. The RocketIO wrapper serves to abstract the proprietary Xilinx modules out of the physical 10.3.2.3.2 CONFIGURATION SUPER-STATE layer, allowing them to be replaced by a In this super-state, the link/physical layer similar module provided by an alternate negotiates with the other endpoint to achieve vendor. the maximum possible bandwidth. This is done by exchanging TS1 and TS2 control The current iteration of the sequences with the appropriate information. transceiver interface is likely still very Xilinx Once the link is configured, and idle data is specific. This semester, we lacked access to an received, the system moves into the link-up Altera development board with a gigabit state. transceiver; however, in the future, the interface will likely require modification to work with Altera transceivers. 10.3.2.3.3 LINK-UP STATE In the link-up state, data may be sent over the link by upper layers. Skip sequences are periodically inserted to allow for correction of clock skew, and idle data is sent whenever the link layer is idle.

10.3.2.3.4 RECOVERY STATE In this state, the link/physical layer attempts to regain connection with the other endpoint and recover from any major errors that may occur. This entails sending TS1 and TS2 control sequences to resynchronize communications and ensure agreement upon the configuration of the link.

10.3.3 ROCKETIO INTERFACE

The RocketIO transceiver is a critical part of the physical layer. It provides the physical layer with facilities for clock recovery, 8b/10b encoding, and gigabit transmissions.

10.3.3.1 ROCKETIO TRANSCEIVER The RocketIO transceiver itself is a physical core available on the Xilinx Virtex line of

InfiniBand FPGA Project 17 NFR11. TEST STRATEGY 11.2 UNIT TESTING

The first step of our test plan was unit testing. his section details our testing and The unit testing we performed was to verification strategy. This semester, simulate each module in ModelSim to ensure we focused on testing our designs to that it performed as we intended when coded. ensure that they worked according The emphasis of this step in the test plan was Tto what we expected of them. In a future not to verify each module against the project, further verification against the InfiniBand specifications, rather to ensure specification will be necessary, so this section that a working module had been designed to also details the test harness, which is match the designer’s interpretation of the designed to simplify the task of verifying our InfiniBand specification. Verification of the code against the specifications. system against the InfiniBand specification will be performed in the later phases of our 11.1 TEST ENVIRONMENT testing.

The environment for our testing was ModelSim for simulation of the VHDL 11.3 PHYSICAL TESTING modules we coded for our design. For our The second stage of our test plan was testing integration testing we planned to program the physical to physical connection. The goal our layers onto the Virtex-5 FPGA and test the of this testing was to ensure that we could system as a whole on the board. Shown below establish and maintain a connection between in Error: Reference source not found is an two physical layers as well as transfer a data example of a waveform from the ModelSim stream between the two. We only possessed simulator, this waveform shows the value of one Virtex-5 board to program the physical the signals of the physical layer as it moves layer on, so, in order to test the design we from the initial to link-up state. implemented a serial loopback to the same board for testing purposes. InfiniBand FPGA Project 18 11.4 TEST HARNESS 11.5.1 PLB INTERFACE

The InfiniBand specification is a two-part The PLB interface provides handles the document consisting of several thousand translation of PLB signals into a format pages. As one might expect, verifying that a compatible with the test harness interface. particular implementation meets these The test harness appears to the PLB as three specifications can be a difficult, time memory mapped registers. consuming, and expensive process. In order to facilitate full scale verification of the 11.5.2 RX AND TX FIFOS system, we have developed a test harness. The test harness will allow a future The RX and TX FIFOs allow the user to inject development team to verify and expand upon and extract data to and from various points our implementation in a straightforward on the data path. manner. To inject data into a point on the data 11.4.1 SYSTEM ARCHITECTURE path, the user first fills the FIFO buffer with the data to be injected, and then sets the TX Error: Reference source not foundError: tap point in the control register. The data Reference source not found provides a top from the FIFO will be injected into the data level diagram of the test harness architecture. path at a rate of one word per cycle until the A PowerPC 440 core on a Virtex-5 FPGA is FIFO is empty. Once the FIFO is empty, the TX interfaced to one or more InfiniBand test tap point is automatically reset so that no harnesses over the IBM Processor Local Bus data is injected into the data path. While data (PLB). Each test harness contains an instance is being read from the FIFO, the user may of the InfiniBand link and physical layers, as continue to add data to the FIFO. well as an instance of the Xilinx RocketIO To extract data from the data path, the gigabit transceiver IP core. user configures the RX tap point to the appropriate position. Data from the data path 11.5 TEST HARNESS ARCHITECTURE will then be written to the RX FIFO at a rate of one word per cycle until the RX FIFO is full, at The test harness itself will include an instance which point the oldest data in the FIFO is of the InfiniBand link and physical layers, a ejected. The end result is that the RX FIFO PLB control interface, and a variety of contains the last 8k values of the connected programmable interconnects to provide the bus. maximum possible level of flexibility for the user.

Error: Reference source not found depicts the basic architecture of the test harness. The PLB interface connects the control interface of the test harness to the PLB, the control register allows the user to program the various interconnects and control the operation of the test harness, and the TX and RX FIFOs allow the user to inject and extract data into and out of the data path at various points. InfiniBand FPGA Project 19 NFR12. CONCLUSION successful. Following completion of the integration of the link layer, integration of the link layer and the physical layer can then be his section details the end result of completed. our efforts, including what is done, what remains to be completed, and the lessons we learned through our 12.1.4 TESTING Twork on this project. The testing of the VHDL modules within the ModelSim simulator was completed on the 12.1 ACCOMPLISHMENTS link and physical layers of the design, although some simulation still may be The progress made towards an required on the link layer when all modules implementation of a working IP core that are fully integrated together. The test follows InfiniBand specifications is outlined harness is only partially complete: the link in detail in this section. layer has not yet been integrated, and the PLB interface is incomplete. 12.1.1 LINK LAYER We were able to test a connection Currently, all of main components of the link between two physical layers by programming layer shown in link layer block diagram have the physical layer onto the Virtex-5 FPGA and been implemented. This includes all of the running a serial loopback connection. We sub-modules required for each of these main were able to demonstrate that a connection components. Integration of the modules of was able to be established between the the link layer requires simulation testing and physical layers and data stream was able to detailed verification against the InfiniBand be transmitted over the connection. specifications.

12.2 CHALLENGES 12.1.2 PHYSICAL LAYER Some of the challenges this project faced All components of the physical layer have included an extensive amount of work coding been unit tested successfully. Integration has in VHDL and a lack of experience and been completed and tested. We have knowledge among some of the group successfully transferred data at a line rate of members working on the project. This 1.5 gigabits over RocketIO. The task of challenge was compounded with the loss of a verifying the physical layer against InfiniBand team member with VHDL experience specifications remains to be completed. following the first semester design of the project. 12.1.3 INTEGRATION Another challenge encountered was The physical layer has been fully integrated with the Virtex-II FPGA we intended to use and has been successful in maintaining a link with our project. The Virtex-II boards in the and sending data over a serial loopback lab worked with computers running an older connection. The link layer still requires all of version Xilinx ISE. The Virtex-II boards also the working modules from the block diagram used an older version of the RocketIO of the link layer to be integrated and verify transceiver, because of these challenges we that through testing that the integration was used a newer Virtex-5 board to solve these InfiniBand FPGA Project 20 problems. However, because we only had the link layer modules, and then integration access to one Virtex-5 board, we were unable of the physical layer with the link layer. to test our project by communicating between two boards, and instead had to Once this work has been performed, perform a serial loopback to our board for the next step in the project would be testing purposes. verification of the design against the InfiniBand specifications in detail. Another option for continued work could be porting 12.3 LESSONS LEARNED the design to transceivers other than RocketIO. By working on this project, our team learned the importance of planning and evaluation on Finally, some work could be done to the success of a project. This project was very create and open hardware development complex with a very detailed specification platform. Currently, we are tied to using that it needed to follow, and effective and development boards designed by Xilinx and accurate evaluation of the difficulty and Altera which may or may not have all the complexity of the project would help to components necessary for physically identify project risk and greatly increase the implementing a complete InfiniBand chances of success. endpoint.

Another important lesson from the work on this project was the importance of 12.5 ACKNOWLEDGEMENTS decomposing the complex and difficult tasks to make them more feasible, and then setting As a final note, we’d like to express our strict deadlines for the completion of these thanks to all those involved who helped to tasks. make our project a success.

Finally, the final goal and expectation  Troy Benjegerdes and the Ames for what we hoped to accomplish with this Laboratory SCL project at the end of the semester was  Dr. Morris Chang somewhat ambiguous and providing a more detailed, specific end goal would allow for a  Dr. Phillip Jones less subjective measure of project success.  Jason Boyd

12.4 FUTURE WORK  Dr. Ahmed Kamal

Subsequent work required to continue this  The ECpE Department project include complete integration of all of

InfiniBand FPGA Project 21 NFR13. BIBLIOGRAPHY

[1] InfiniBand Trade Assosciation, InfiniBand Architecture Specification, Volumes 1 and 2, Release 1.2. InfiniBand Trade Assosciation, 2004.

[2] InfiniBand Trade Assosciation. (2009, Jan.) InfiniBand Trade Assosciation Website. [Online]. http://www.infinibandta.org/about/about_infiniband

InfiniBand FPGA Project 22