Researching methods for efficient hardware specification, design and implementation of a next generation communication architecture Inauguraldissertation zur Erlangung des akademischen Grades eines Doktors der Naturwissenschaften der Universität Mannheim vorgelegt von Dipl.-Inf. Patrick R. Haspel aus Mannheim Mannheim, 2006 Dekan: Professor Dr. M. Krause, Universität Mannheim Referent: Professor Dr. U. Brüning, Universität Mannheim Korreferent: Professor Dr. V. Lindenstruth, Universität Heidelberg Tag der mündlichen Prüfung: 4. Mai 2007 Abstract The objective of this work is to create and implement a System Area Network (SAN) ar- chitecture called EXTOLL embedded in the current world of systems, software and stan- dards based on the experiences obtained during the ATOLL project development and test. The topics of this work also cover system design methodology and educational issues in order to provide appropriate human resources and work premises. The scope of this work in the EXTOLL SAN project was: • the Xbar architecture and routing (multi-layer routing, virtual channels and their arbi- tration, routing formats, dead lock aviodance, debug features, automation of reuse) • the on-chip module communication architecture and parts of the host communication • the network processor architecture and integration • the development of the design methodology and the creation of the design flow • the team education and work structure. In order to successfully leverage student know-how and work flow methodology for this research project the SEED curricula changes has been governed by the Hochschul Didaktik Zentrum resulting in a certificate for "Hochschuldidaktik" and excellence in university ed- ucation. The complexity of the target system required new approaches in concurrent Hardware/Soft- ware codesign. The concept of virtual hardware prototypes has been established and exces- sively used during design space exploration and software interface design. Zusammenfassung Das Ziel dieser Arbeit ist der Entwurf sowie die Implementierung einer System Area Net- work (SAN) Architektur namens EXTOLL, die in die heutige Welt der Rechensysteme, Software und Standards eingebettet ist. Diese Arbeit beinhaltet auch die Aspekte der Systementwurfsmethodik und Ausbildungs- fragen um eine geeignete Arbeitsumgebung sowei ein kompetentes Team zu gewährleisten. Die Beiträge zu der EXTOLL SAN Architektur sind: • Die XBar Architektur sowie Routing (multi-layer routing, Virtuelle Kanäle sowie deren Arbitrierung, Routingformate, Deadlockvermeidung, Testeigenschaften, Automa- tisierung der Wiederverwendung) • Die on-chip Modulkommunikationsarchitektur und Teile der Hostkommunikation • Netzwerkprozessorarchitektur und Integration • Die Entwicklung einer Entwurfsmethodik und die Erstellung eines Designflows Um den erforderlichen Ausbildungsstand der Studenten sowie eine geeignete Arbeit- sabläufe für dieses Forschungsprojekt sicherzustellen wurden die Änderungen des SEED Lehrplanes von dem Hochschuldidaktikzentrum begleitet und dabei als Ergebnis das Zer- tifikat für Hochschuldidaktik und Exzellenz in der Lehre verliehen. Die Komplexität des Zielsystems erforderte neue Ansätze im simultanen Hardware/Soft- ware Codesign. Das Konzept virtueller Hardwareprototypen wurde geschaffen und intensiv während der Entwurfraumanalyse und dem Entwurf der Softwareschnittstellen genutzt. Inaugural Dissertation Patrick R. Haspel 1 Introduction ................................................................................................................. 1 1.1 Objectives ............................................................................................................ 1 1.2 Topics of this work .............................................................................................. 2 1.2.1 Creating the work/design environment ................................................... 2 1.2.2 Architecture definition ............................................................................ 3 1.2.3 Hardware/Software design methodology ................................................ 3 1.3 Thesis organization .............................................................................................. 3 2 Cluster Computing and Implementations .................................................................... 5 2.1 Communication demands of distributed and parallel computing ........................ 5 2.2 State of the art hardware support ......................................................................... 7 2.2.1 QsNetII by Quadrics ................................................................................ 8 2.2.2 Myrinet by Myricom ............................................................................. 17 2.2.3 Pathscale ................................................................................................ 19 2.2.4 PCI-ASI (Advanced Switching Interconnect) ....................................... 21 2.2.5 IBM Blue Gene BG/L ........................................................................... 23 2.2.6 Mellanox Infiniband .............................................................................. 25 2.2.7 Cray XT3 ............................................................................................... 26 2.2.8 10GigEthernet by Chelsio ..................................................................... 28 3 SANs in general ......................................................................................................... 30 3.1 Functions, Features and important quality metrics of a SAN ............................ 31 3.1.1 Latency .................................................................................................. 31 3.1.2 Bandwidth ............................................................................................. 37 3.1.3 Network Topology ................................................................................ 40 3.1.4 Supported Parallel Programming Model ............................................... 48 3.1.5 Cost ........................................................................................................ 50 3.1.6 Communication to / Location of the Network Interface Controller ...... 50 4 Methods - Approach .................................................................................................. 53 4.1 Providing the basis for efficiency in hardware design ....................................... 53 4.1.1 Education ............................................................................................... 53 4.1.2 EDA tooling .......................................................................................... 57 4.1.3 Realization - SEED Project ................................................................... 58 4.1.4 The System Realisation Bi-Cone .......................................................... 60 4.2 Hardware/Software Codesign and Cosimulation ............................................... 63 4.2.1 Concurrent development of hardware and related low level software .. 70 Table of Contents Page I Inaugural Dissertation Patrick R. Haspel 4.2.2 Seamless hardware/software interfacing ............................................... 70 4.2.3 Integrity and completeness of software required functionality ............. 71 4.3 FPGA based ASIC prototyping ......................................................................... 71 4.4 Physical design impact of UDSM designs ......................................................... 74 4.4.1 Creating a leading-edge design flow ..................................................... 74 4.4.2 UDSM characteristics ............................................................................ 75 5 Architecture and Function Scope of Building Blocks ............................................... 78 5.1 Top-Level Architecture Decisions ..................................................................... 78 5.2 NPU (Network Processing Unit) ....................................................................... 80 5.2.1 NPU Features and Overview ................................................................. 81 5.2.2 Reasons for a dedicated compute resource in a SAN ............................ 83 5.2.3 Instruction Set enhancements ................................................................ 85 5.2.4 Implementation details .......................................................................... 86 5.3 Network Switch ................................................................................................. 89 5.3.1 Design Space Exploration - Approach and Methods ............................ 91 5.3.2 Functional Enhancements ...................................................................... 95 5.4 Hostinterface and EXTOLL block level communication ................................ 114 5.4.1 Intermodule Communication Architecture .......................................... 115 5.4.2 The Slave interface Unit ...................................................................... 119 5.4.3 The Master Interface Unit ................................................................... 126 6 Conclusion and Outlook .......................................................................................... 135 7 References ............................................................................................................... 137 Table of Contents Page II Inaugural Dissertation Patrick R. Haspel 1 Introduction System Design is a highly complex challenge requiring the breath and depth of more than one