Department of Computer Architecture and Technology Performance Evaluation of Interconnection Networks using Simulation: Tools and Case Studies PhD Dissertation Javier Navaridas Palma Advisor : Prof. José Miguel-Alonso September, 2009 To the memory of my beloved Mother To my girlfriend for her patience and support To family and friends Acknowledgements There are so many people that helped and supported the author that an acknowledgements list citing them one by one would be almost infinite. Consequently, I will try to make this list as brief as possible: To all the members of my research group, particularly to my advisor Prof. José Miguel-Alonso for his guidance. To all the colleagues from several different institutions that helped developing collaborative works: The University of Cantabria, The University of Burgos, The Technical University of Catalonia, The University of Adelaide, The University of Manchester and Zurich IBM Research Lab. Author is especially grateful to the people from the University of Manchester and to the people from Zurich IBM Research Lab for giving me the invaluable experience of working in international environments, as well as for the received human support. Most of the research work carried out for this dissertation was supported by the Spanish Ministry of Education and Science, grants TIN2004-07440-C02-02 and TIN2007-68023-C02-02, and by the Basque Government grant IT- 242-07. The author was supported by a doctoral grant of the University of the Basque Country UPV/EHU. - 5 - - 6 - Abstract This dissertation focuses on the performance evaluation of interconnection networks. It briefly introduces supercomputing and shows three different classifications of computing systems, which are used to locate the performed researching work. Moreover, the most common methodologies for the performance evaluation of such systems are discussed showing their advantages and limitations. This dissertation thoroughly describes the simulation environment developed within the author’s research group and all the related tools. Remarkable contributions to this environment are the trace-driven engine and the application-kernels that allow the evaluation of interconnection networks using realistic loads. This environment is used to perform several researches in the area of interconnection networks that are shown in the form of four case studies. The first one is the evaluation of the twisted torus topology, a variation of the standard torus which offers better topological properties. In this study we show a pitfall of the derivation of the theoretical throughput from the bisection bandwidth. Furthermore, the two topologies are confronted in order to assess if the better topological characteristics of the twisted torus lead to better-performing execution of applications. The second case study evaluates the thin-tree topology, an alternative to the over-dimensioned k-ary n-tree topology which is shown to be more cost-effective. The third case study evaluates the interconnection network of SpiNNaker, a large- scale system-on-chip-based architecture with severe restrictions in terms of power consumption and chip area; this evaluation is specifically devised to evaluate a bespoke router mechanism focusing on the stability and the fault- tolerance of the system. The last case study measures the influence that job and task allocation policies have on the execution time of parallel applications. It is showed to be significant and dependent on the size of the system and the number of concurrent applications. The conclusion of this study is that allocating applications into isolated subnetworks leads to faster execution of applications. These evaluations have been carried out using mainly simulation, although some results have also been mathematically derived. - 7 - - 8 - Javier Navaridas Palma Table of Contents Table of Contents Acknowledgements...............................................................................................................................................................5 Abstract .................................................................................................................................................................................7 Table of Contents..................................................................................................................................................................9 List of Figures .....................................................................................................................................................................11 List of Tables.......................................................................................................................................................................13 Chapter 1. Introduction ....................................................................................................................................................15 1.1 Classification of Computer Systems ...................................................................................................................16 1.1.1 System Architecture .....................................................................................................................................................16 1.1.2 Objectives of the Computing System ...........................................................................................................................16 1.1.3 Interconnection Network ..............................................................................................................................................17 1.2 Position of this Research Work ...........................................................................................................................18 1.3 Structure of this Dissertation...............................................................................................................................18 Chapter 2. Evaluation Methodologies..............................................................................................................................21 2.1 Analysis...............................................................................................................................................................21 2.1.1 Topological Characteristics ..........................................................................................................................................21 2.1.2 Markov Chains .............................................................................................................................................................23 2.1.3 Queueing Theory..........................................................................................................................................................24 2.1.4 Petri Nets......................................................................................................................................................................25 2.1.5 Other Tools for Analytical Modeling ...........................................................................................................................27 2.2 Simulation ...........................................................................................................................................................27 2.2.1 Simulation Engine ........................................................................................................................................................27 2.2.2 Simulation Frameworks................................................................................................................................................28 2.2.3 Levels of Detail ............................................................................................................................................................29 2.2.4 Examples in the Literature............................................................................................................................................32 2.3 Empirical Evaluation...........................................................................................................................................33 2.3.1 Benchmarks..................................................................................................................................................................33 2.3.2 Examples in the Literature............................................................................................................................................35 2.4 Conclusions .........................................................................................................................................................36 Chapter 3. Simulation Environment................................................................................................................................37 3.1 FSIN ....................................................................................................................................................................37 3.1.1 Simulation Engines.......................................................................................................................................................37 3.1.2 Modeled Routers ..........................................................................................................................................................39 3.1.3 Topologies....................................................................................................................................................................44