University of Amsterdam White-label open-source networking Authors Łukasz Makowski [email protected] Paola Grosso [email protected] January 22, 2018 Table of Contents 1 Motivation for open-networking 4 2 Architecture background 5 2.1 NPU communication ............................. 5 2.2 ASIC control module ............................. 6 3 Network Operating Systems (NOS) 8 3.1 Open Networking Linux (ONL) ....................... 8 3.2 OpenSwitch (OPX) .............................. 8 3.3 SONiC ..................................... 10 3.4 Broadcom NPU interface access ....................... 13 4 Control-plane stacks 14 4.1 Quagga/FRR ................................. 14 4.2 BIRD ..................................... 14 4.3 Flexswitch ................................... 14 5 Method 15 5.1 Preliminary phase ............................... 15 5.2 Feature tests ................................. 16 5.3 FIB installation latency ........................... 20 6 Results: feature tests 23 6.1 Configuration and management ....................... 23 6.2 Layer 2 (L2) .................................. 27 6.3 Layer 3 (L3) : OSPF ............................. 30 7 Results: FIB installation latency 32 8 Discussion 33 8.1 Preliminary phase ............................... 33 8.2 Feature test .................................. 33 8.3 FIB installation latency accuracy ...................... 34 9 Conclusions 36 10 Future work 37 2 11 Acknowledgments 38 Bibliography 39 3 1 Motivation for open-networking Traditionally, networking equipment used to be a locked-in hardware with vendor’s pro- prietary software installed. However, recently the trend of opening this ecosystem is becoming more and more popular. Networking vendors allow the installation of other operating systems and software on their devices. This gives more power to network administrators, allowing the customiza- tion of the device according to their specific needs. Secondly, it also opens the possibility of running unified network operating system and networking daemons across the devices from various vendors. There are a couple of commercial NOS distributions targeting white-label switches (e.g. Cumulus Linux1 or PicaOS2). Generally, they are based on Linux operating system and for the purposes of control-plane also open-source software could be used. Never- theless, the NOS itself remains proprietary and closed-source. In the opposition to this trend, there are also open-source NOS projects available. They constitute an alternative for users wanting to exchange and adjust the system to their requirements. Moreover, allowing to avoid the licensing costs of using commercial NOSes. In this research we aim to answer the following questions: 1. Is it feasible to use a white-label, open-source based switching stack for a traditional deployment requiring basic L2 and L3 functionalities? 2. Are there any performance related differences between open-source NOS imple- mentations? 1https://cumulusnetworks.com/products/cumulus-linux/ 2http://www.pica8.com/products/picos 4 2 Architecture background In this section, we discuss the components which can be identified in the white-label network stack, where Figure 1 gives the overview on this. Starting from the bottom, there is a hardware layer representing the actual device i.e. Network Processing Unit (NPU), interfaces, sensors, etc. It later gets discovered and handled by a lower-level layer, which signifies the drivers and other components residing in the space of network operating system. Next, for the actual interaction with the NPU device, the abstraction layer is placed. Lastly, to realize the network devices features (e.g. routing protocol operations, configuration) the management and control-plane applications are used. Figure 1: White-label networking stack logical structure 2.1 NPU communication To program the NPU, there can be two generic approaches possible: (1) using NPU’s SDK, (2) using a “proxy” component which will handle the NPU communication on behalf of the application. Generally, the former is not present in the design of open- source white label stacks, as it would require re-adjusting the applications to every new change in SDK. Following the second approach, there have been two ways established: 5 • Switch Abstraction Interface (SAI) • switch-dev SAI SAI develops an abstracted interface over SDKs provided by NPU vendors used in white- label switches (e.g. Broadcom, Mellanox). This approach allows the NOS developers to focus on a compatibility with SAI’s API without the need to adjust to SDK changes or even a completely new NPU vendor. switch-dev Switch-dev is a project aiming to contribute open versions of NPU drivers to the Linux kernel code. The vision is to perform all NPU communication inside the kernel space, so that the standard Linux interfaces (CLI, netlink API) can be used to change NPU state. 2.2 ASIC control module The role of ASIC control module could be characterized as three activities: (1) interfac- ing with control-plane apps, (2) maintaining the state of the desired configuration, (3) communicating with an NPU (for example using SAI). On the second page of SAI speci- fication [1] it can be seen that authors explicitly define the requirement for suchcompo- nent, additionally specifying rudimentary features it should implement. The examples of ASIC control module implementations can be found in open-networking systems such as OpenSwitch (OPX) and SONiC. Those are implemented as sets of processes cooperating with each other according to the publish/subscribe communication scheme. Message passing architecture The concept of message-based asynchronous process communication is a key design prin- ciple for the analyzed network operating systems. It is also one of the attributes of Erlang1 programming language which has been designed with a telecom-grade high reli- ability in mind. Erlang provides language level paradigms for message passing, enforcing 1http://www.erlang.org/ 6 process communication to be realized only in this manner. Such approach allows for high process isolation and as a consequence, it results in a fact that “a software error in a con- current process should not influence processing in the other processes in the system” [2]. A similar solution appears to be empowering Arista EOS2 network operating system. A central database component (Sysdb) is used for inter-process message exchange between EOS individual components [3]. The analysis of middleware subsystems of Azure SONiC and OpenSwitch reveals that Redis3 message-queue is used to realize similar objectives as outlined for Erlang and EOS. 2https://eos.arista.com/ 3https://redis.io/ 7 3 Network Operating Systems (NOS) 3.1 Open Networking Linux (ONL) ONL is a project hosted by the Open Compute Project1. As the name implies, it is a Linux distro suited for installation on the network devices. Authors add the support for a specific hardware used in a particular model. This provides the foundation for starting to use the given platform, however, it lacks higher-level components which are the key part of fully usable network device. Most importantly, it is left to the user how to handle the communication with the NPU. 3.2 OpenSwitch (OPX) The current version of OpenSwitch is actually the second generation of this project. The first one, sponsored mainly by HPE (Hewlett-Packard Enterprise) has been discontinued. The two main contributors are Dell (providing the actual NOS layer) and SnapRoute (FlexSwitch control-plane). At the moment of our evaluation, there were no official project images available. However, the images offering Dell S6000-ON switch support were available at one of the repository pages2. The inspection of the images revealed that the control-plane stack was not included as the project work was wrapping up. However, it was possible to access FlexSwitch source code (opx-flxl* repositories in OPX github project). 1http://www.opencompute.org/ 2https://github.com/open-switch/opx-docs/wiki/Install-OPX-Base-on-Dell-S6000-ON-platform 8 Figure 2: NAS design3. OPX is based on Debian operating system combined with the abstraction layer compo- nents and utilities developed by OPX project participants. As illustrated in Figure 2, the component ingesting the intended configuration is Control Plane Service (CPS). Next, Network Adaptation Service (NAS) is responsible for translating the configuration to the set of NPU commands and communicating those using SAI interface. Table 1 (taken from [4]) lists the available features. The important fact is that not all of them are configurable with the means of using standard Linux network utilities. The operations which are simply not present there need to be performed through CPS API. 3https://github.com/open-switch/opx-docs/raw/master/images/nas_design.png 9 Configure with Linux Commands Networking Feature Configure with CPS API or Open Source Application Interfaces Physical Yes Yes Link Aggregation (LAG) Yes (Bond) Yes VLAN Yes Yes Fanout (4x10G) No Yes (script) Layer 2 Bridging LLDP Yes No MAC address table No Yes STP Yes Yes VLAN Yes Yes Layer 3 Routing ECMPv4 Yes Yes ECMPv6 Yes Yes IPv4 Yes Yes IPv6 Yes Yes Unicast routing Yes Yes QoS No Yes ACLs No Yes Monitoring Mirroring No Yes sFlow No Yes Port and VLAN statistics No Yes Table 1: Selected supported Dell OS10 networking features. 3.3 SONiC SONiC is an abbreviation for Software for Open Networking in the Cloud. It is a NOS by Microsoft Azure used to empower the network operations, however, the scale of its deployment is non-public. Table 2 outlines