The IEEE 1355 Standard: Developments, Performance and Application in High Energy Physics
Total Page:16
File Type:pdf, Size:1020Kb
The IEEE 1355 Standard: Developments, Performance and Application in High Energy Physics Thesis submitted in accordance with the requirements of the University of Liverpool for the degree of Doctor of Philosophy by Stefan Haas December 1998 The IEEE 1355 Standard: Developments, Performance and Applications in High Energy Physics Stefan Haas The data acquisition systems of the next generation High Energy Physics experiments at the Large Hadron Collider (LHC) at CERN will rely on high-speed point-to-point links and switching networks for their higher level trigger and event building systems. This thesis pro- vides a detailed evaluation of the DS-Link and switch technology, which is based on the IEEE 1355 standard for Heterogeneous InterConnect (HIC). The DS-Link is a bidirectional point-to-point serial interconnect, operating at speeds up to 200 MBaud. The objective of this thesis was to study the performance of the IEEE 1355 link and switch technology and to dem- onstrate that switching networks using this technology would scale to meet the requirements of the High Energy Physics applications. The performance and reliability of the basic point-to-point interconnect technology over elec- trical and fibre optic media were examined. These studies were carried out while the IEEE 1355 standard was still being finalised and have therefore provided valuable input to the standards working group. In order to validate the fibre optic physical layer proposed for the IEEE 1355 standard, an implementation demonstrator of a DS-Link interface for fibre optics, employing a new line encoding scheme, has been designed and characterised. This interface allows the link length for DS-links to be extended, which is important in the HEP context, where the cable length from the detector to the electronics can be up to 200 meters. A large switching network testbed of variable topology based on IEEE 1355 point-to-point serial links and high-valency crossbar switches has been designed and constructed. The net- work testbed consists of up to 1024 end nodes which are connected to a reconfigurable switching fabric constructed from 32-way crossbar switches. The end nodes are loaded with a predetermined traffic pattern and the response of the network in terms of throughput and latency is measured. The testbed allows the network performance of various topologies to be studied as a function of the network size and the traffic conditions, including those expected in HEP trigger and event building systems. This system is believed to be unique in its ability to measure network performance under well controlled and well defined conditions. The results from the Macramé network demonstrate that large IEEE 1355 DS-Link networks can be built and that they scale very well. Furthermore, it has been shown that per-link flow control, together with well designed hardware, can result in very reliable systems. It was also demonstrated, that a switching fabric based on the IEEE 1355 technology could meet the required network performance of the ATLAS second level trigger. Acknowledgements I would like to thank my supervisors Bob Dobinson and Erwin Gabathuler for the opportunity to carry out the work presented in this thesis and for their guidance and advice throughout the thesis. Many thanks also to the other members of the Macramé team: Brian Martin, David Thornley and Minghua Zhu. They all played a crucial role in the successful comple- tion of the project. I am grateful for the support of the European Union through the Macramé project (Esprit project 8603), without which this thesis would not have been possible, and to CERN for hosting this project. Finally, I would also like to thank my parents for their encouragement and support. Table of Contents Chapter 1: Introduction 1.1 Motivation . 1 1.2 Context. 2 1.3 Outline of the Thesis . 2 1.4 Author’s Work . 3 Chapter 2: IEEE 1355 Technology 2.1 Overview of the IEEE 1355 Standard . 5 2.1.1 The Protocol Stack . 6 2.1.1.1 Physical Media. 6 2.1.1.2 Signal Layer . 7 2.1.1.3 Character Layer . 7 2.1.1.4 Exchange Layer . 7 2.1.1.5 Packet Layer. 7 2.1.2 Advantages of IEEE 1355 . 7 2.2 Data/Strobe Links (DS-Links). 8 2.2.1 Signal Layer. 8 2.2.2 Character Layer . 9 2.2.3 Exchange Layer . 10 2.2.3.1 Flow Control . 10 2.2.3.2 Link Start-up . 10 2.2.3.3 Error Detection and Handling . 11 2.2.4 Packet Layer . 11 2.2.5 Higher Level Protocols . 12 2.3 IEEE 1355 Integrated Circuits. 12 2.3.1 The STC101 Parallel DS-Link Adapter . 13 2.3.1.1 Functional Description. 13 2.3.2 The STC104 Packet Switch. 14 2.4 Theoretical Performance . 15 2.4.1 Unidirectional Link Bandwidth. 15 2.4.2 Bidirectional Link Bandwidth . 16 2.4.3 Effect of Link Length on Bandwidth. 17 2.5 Summary . 19 Chapter 3: Electrical DS-Link Transmission 3.1 Single-Ended DS-Links (DS-SE) . 21 3.2 Differential DS-Links (DS-DE). 21 3.2.1 Limitations of Cable Transmission . 22 3.2.1.1 Crosstalk. 22 3.2.1.2 Skew. 22 3.2.1.3 Jitter . 23 I TABLE OF CONTENTS 3.2.1.4 Effect of Cable Attenuation. 23 3.3 Evaluation of Twisted-Pair Cable Transmission . 24 3.3.1 Test Setup . 24 3.3.2 Eye-Diagram . 24 3.3.3 Bit Rate versus Cable Length . 25 3.3.4 Bit Error Rate Test . 26 3.3.5 Summary of DS-DE Link Evaluation . 27 3.4 Susceptibility to Electromagnetic Interference . 27 3.4.1 Interference Problems with DS-Links . 27 3.4.2 Packaging . 28 3.4.3 IEC 801 Standard . 28 3.4.4 Test Setup . 29 3.4.5 The Failure Mechanism . 30 3.4.6 Test Board. 31 3.4.7 Results and Recommendations . 32 3.4.8 Summary on EMI Susceptability of DS-Links . 33 3.5 Summary and Conclusions . 34 Chapter 4: Fibre-Optic DS-Link Transmission 4.1 Fibre-Optic Transmission System . 35 4.2 Reliability of Fibre Optic Transmission . 36 4.3 The Transmission Code. 38 4.3.1 TS Transmission Code Definition . 38 4.3.1.1 TS-Code Symbols . 38 4.3.1.2 TS-Code Control Characters . 39 4.3.1.3 Longitudinal Parity . 40 4.3.1.4 Character Synchronisation. 40 4.3.1.5 Link Start-up . 40 4.3.1.6 Error Handling. 41 4.3.2 Flow Control . 41 4.3.3 TS-Link Bandwidth . 42 4.4 DS-Fibre Optic Link Interface Design . 42 4.4.1 Hardware Overview . 42 4.4.2 PCI Interface Board . 43 4.4.3 Mezzanine Board . 45 4.4.4 VHDL Structure . 45 4.5 Measurements and Results . 46 4.5.1 Fibre Optic Transceiver Test . 46 4.5.1.1 Fibre Optic Transceiver Test Results. 47 4.5.2 TS-Link Test . 48 4.6 Summary and Conclusions . 48 Chapter 5: Switches and Networks 5.1 Introduction . 51 5.2 Switch Architecture. ..