DECENTRALIZED MODULAR ROUTER ARCHITECTURES Doctoral Thesis Markus Hidell Laboratory for Communication Networks School of Electrical Engineering KTH, Royal Institute of Technology Stockholm 2006 Decentralized Modular Router Architectures A dissertation submitted to the Royal Institute of Technology (KTH) in partial fulfillment of the requirements for the Doctor of Philosophy degree. Akademisk avhandling som med tillstånd av Kungliga Tekniska Högskolan framlägges till offentlig granskning för avläggande av teknologie doktorsexamen fredagen den 22 september 2006 i Salongen, KTHB, KTH, Stockholm. © Markus Hidell, 2006 Royal Institute of Technology (KTH) School of Electrical Engineering Laboratory for Communication Networks SE-100 44 Stockholm Sweden TRITA-EE 2006:036 ISSN 1653-5146 ISRN KTH/EE—06/036—SE ISBN 91-7178-424-1 Printed by Universitetsservice US-AB, Stockholm 2006 ABSTRACT The Internet grows extremely fast in terms of number of users and traffic volume, as well as in the number of services that must be supported. This development results in new requirements on routers—the main building blocks of the Internet. Existing router designs suffer from architectural limitations that make it difficult to meet future requirements, and the purpose of this thesis is to explore new ways of building routers. We take the approach to investigate distributed and modular router designs, where routers are composed of multiple modules that can be mapped onto different processing elements. The modules communicate through open well-defined interfaces over an internal network. Our overall hypothesis is that such a combination of modularization and decentralization is a promising way to improve scalability, flexibility, and robustness of Internet routers—properties that will be critical for new generations of routers. Our research methodology is based on design, implementation, and experimental verification. The design work has two main results: an overall system design and a distributed router control plane. The system design consists of interfaces, protocols, and internal mechanisms for physically separation of different components of a router. The distributed control plane is a decomposition of control software into independent modules mapped onto multiple distributed processing elements. Our design is evaluated and verified through the implementation of a prototype system. The experimental part of the work deals with two key issues. First, transport mechanisms for communication of internal control information between processing elements are evaluated. In particular, we investigate the use of reliable multicast protocols in this context. Results regarding communication overhead as well as overall performance of routing table dissemination and installation are presented. The results show that even though there are certain costs associated with using reliable multicast, there are large performance gains to be made when the number of processing elements increases. Second, we present performance results of processing routing information in a distributed control plane. These results show that the processing time can be significantly reduced by distributing the workload over multiple processing elements. This indicates that considerable performance improvements can be made through the use of the distributed control plane architecture proposed in this thesis. 1 2 ACKNOWLEDGMENTS First, I would like to thank my main advisor Prof. Gunnar Karlsson for his guidance, support, and thorough reading of this thesis. Gunnar has provided an inspiring environment and also given me the opportunity to build up an experimental laboratory at LCN. This environment has been a great platform for the thesis work. The most important person during this work has, without doubt, been my close friend and colleague, Dr. Peter Sjödin. I have had the pleasure to work with Peter during more or less all my professional life so far, and he has also had the role of assistant advisor throughout the thesis work. With his intelligence, generosity, and kindness, Peter has given invaluable support at the professional and the personal level. I owe a great debt of gratitude also to Dr. Olof Hagsand. Olof is an impressive software architect and researcher, and very professional in all the ways I can think of. The opportunity to work with Olof was one of the main reasons for me to join LCN. I have really enjoyed working together with Olof and I am very happy to have him among my personal friends. In addition, I would like to thank Tomas Klockar for the chance to collaborate with him during parts of thesis work. I am also grateful to all other people at LCN that I have learned to know over the years and who have let me take part of their professional experience during seminars and discussions. My research has been made possible by generous grants from SSF through the Winternet research program, and from the Wallenberg Foundation. I would like to express my gratitude to these organizations. Last, but not least, I would like to thank my family for their love and understanding. The unlimited support given by my parents goes beyond words. More than ever, I would like to express my love to the most important persons in my life: my lovely wife, Marie, and my wonderful children, Jonna and Jakob. 3 4 TABLE OF CONTENTS Abstract ........................................................................................................1 Acknowledgments........................................................................................3 Table of Contents .........................................................................................5 1. Introduction........................................................................................9 1.1. The IP Router Concept .............................................................11 1.2. IP Router Generations...............................................................12 1.2.1. First Generation of Routers...............................................13 1.2.2. Second Generation of Routers ..........................................14 1.2.3. Third Generation of Routers .............................................15 1.2.4. Further Distribution—the Fourth Router Generation .......16 1.3. Motivations, Problem Statements and Hypotheses...................17 1.3.1. Overall System Design .....................................................20 1.3.2. Internal Network and Transport Mechanisms...................22 1.3.3. Distributed Control ...........................................................23 1.4. Contributions of this Thesis......................................................24 2. Distributed Modular Router Architectures ......................................27 2.1. Related Work ............................................................................28 2.1.1. Modular Routers ...............................................................28 2.1.2. ForCES..............................................................................31 2.1.3. Router and Switch Architectures ......................................36 2.2. Motivations for a Decentralized Modular Design.....................37 2.2.1. Scalability .........................................................................38 2.2.2. Robustness ........................................................................39 2.2.3. Flexibility..........................................................................40 2.2.4. Meeting the Requirements ................................................41 2.3. An Abstract Model of a Decentralized Modular Router...........42 2.4. System Design ..........................................................................44 2.5. The Internal Protocols—Forz ...................................................48 2.5.1. Association........................................................................50 2.5.2. Configuration....................................................................51 2.5.3. Data Transfer ....................................................................53 2.6. Internal Network.......................................................................54 2.7. Control Element Design and Implementation...........................56 2.7.1. Application-level Interface Representation ......................59 2.7.2. OS-level Interface Representation ....................................61 2.8. Supporting Multiple Control Elements.....................................64 2.8.1. Network Interfaces—Link Objects ...................................65 2.8.2. IP Addresses—Address Objects .......................................65 5 2.8.3. Routing Table Entries—Route Objects.............................65 2.9. Forwarding Element Design and Implementation ....................66 2.9.1. User-space FE Implementation.........................................68 2.9.2. Kernel-based FE Implementation .....................................68 2.9.3. Intel IXP-based FE Implementation .................................69 2.9.4. Xelerated X10-based FE Implementation.........................71 2.10. Conclusions and Further Work .............................................72 3. Transport Mechanisms for Internal Control.....................................75 3.1. Reliable Multicast .....................................................................76 3.2. Analysis of Reliable Multicast Protocols..................................78 3.2.1. Sender-initiated Protocols.................................................79 3.2.2. Receiver-initiated Protocols..............................................80