Zebra 2.0 and Lagopus: newly-designed routing stack on high-performance packet forwarder Kunihiro Ishiguro∗, Yoshihiro Nakajimay, Masaru Okiz, Hirokazu Takahashiy ∗ Hash-Set, Tokyo, Japan y Nippon Telegraph and Telephone Corporation, Tokyo, Japan z Internet Initiative Japan Inc, Tokyo, Japan e-mail: [email protected], [email protected], [email protected], [email protected] Abstract First GNU Zebra architecture and its issues Zebra 2.0 is the new version of open source networking When we designed the first GNU Zebra, the biggest ambition software which is implemented from scratch. Zebra 2.0 was to make multi-process networking software work. The is designed to supports BGP/OSPF/LDP/RSVP-TE and co- first GNU Zebra is made from a collection of several dae- working with Lagopus as fast packet forwarder with Open- mons that work together to build the routing table. There may Flow API. In this new version of Zebra, it adapts new archi- be several protocol-specific routing daemons and Zebra’s ker- tecture which is mixture of thread model and task completion nel routing manager. Figure 1 shows the architecture of the model to achieve maximum scalability with multi-core CPUs. first GNU Zebra. RIB (Routing Information Base) / FIB (For- Zebra has separate independent configuration manager that warding Information Base) and the interface manager are sep- supports commit/rollback and validation functionality. The configuration manager understand YANG based configuration arated into an isolated process called ’zebra’. All of protocol model so we can easily add a new configuration written in handling is also separated to it’s own process such as ’ripd’, YANG. Lagopus is an SDN/OpenFlow software switch that is ’ospfd’, ’bgpd’ and so on. This makes easier to add a new designed to achieve high-performance packet processing and protocol modules without re-building the whole staff. It in- high-scalable flow handling leveraging multi-core CPUs and creases the system stability - one process crash does not af- DPDK on commodity servers and commodity NICs. Lagopus fect to other process. Most importantly it extend possibility of supports match/action-based packet forwarding and process- efficient use of multi-core CPU and / or distributed environ- ing as well as encapsulation/decapsulation operation for MPLS ments resources. The architecture succeeded and we believe and VxLAN. The inter working mechanism of the user space that it shows the architecture works in various operational en- data plane of Lagopus and the network stack in kernel allows vironments. At the same time there was several problems to easy integration with Zebra 2.0 and its OpenConfigd. be solved. One of them is a performance issue. All of protocol mod- Keywords ules are single threaded because matured threading library did not exist when we designed the first GNU Zebra. Therefore Routing stack, BGP, OSPF, LDP, RSVP-TE, Packet for- there must be careful thoughts about event management and warder, YANG, OpenConfig. non-preemptive manner of task execution. This is really an- noying overhead to the programmer. Introduction Another issue is to support various virtualized environ- ments, such as VRF (Virtual Routing and Forwarding), VM Zebra 2.0 is a sequel to GNU Zebra [5] / Quagga [13] which is (Virtual Machine) and Containers. When we designed the widely used an open source routing software. Since first GNU original Zebra architecture, virtualized environment was not Zebra was designed in 1996, the architecture start showing so common as today. Thus we did not take into account of the it’s age. In past 20 years, there were a lot of technology possibility of the software runs on a virtualized environment. evolution. Here we have re-designed a 21st century routing Adding to that in past couple of years there are several software from the ground up to match to the today’s industry new packet forwarding architectures and technologies, such needs. as DPDK (Data Plane Development Kit)[3], OF-DPA (Open- In this paper, we describe the design and implemenation of Flow Data Plane Abstraction)[9] emerged to industry. That Zebra 2.0 and its integration to a high-performance software needs tight integration with networking software today. switch called Lagopus. In the next section, we mention important issues to be Design and Implementation solved in the first GNU Zebra. In Section III, we describe the basic design and implementation of Zebra 2.0. In Section We have designed Zebra 2.0 to achieve high-performance net- IV, we present Lagopus software switch and its integration to work protocol handling and high-extensible for various plat- Zebra 2.0. Finally, we conclude the paper. form. Figure 2 shows the architecture overview of Zebra 2.0. Proceedings of netdev 1.1, Feb 10-12, 2016, Seville, Spain zAPI RSVP- BGP OPSF LDP bgpd ripd ospfd zebra TE UNIX Kernel routing table FEA Figure 1: The architecture of the first GNU Zebra. Figure 2: The architecture of Zebra 2.0. New process and thread model Zebra 2.0 Lagopus The biggest bottle neck of the original design was protocol handling affect to packet in/out and keep-alive timer. For example, a large SPF (Shortest Path First) calculation is re- API API quired in a large-scale network on OSPF, OSPF can’t handle packet in/out and keep-alive timer without voluntary yield of the execution of SPF calculation because it is single threaded. OpenConfigd DB This lead OSPF to Hello time out. That was the typical sce- nario of the performance issue. completion show config Zebra 2.0 employs totally new process and thread model to achieves drastically improved performance and stability. In Zebra 2.0, the protocol handler and packet in/out (including confsh keep alive timer) handler are executed in a separate thread. By default, when a new protocol module is instantiated, two Figure 3: The architecture of the OpenConfigd. threads are created. One is for the protocol handling, the an- other one is for packet in/out and time out handling. In ad- dition, a lock-free queue between these two threads is intro- duced. automatically generated by OpenConfigd with YANG model. When a control packet comes in, the packet is handled by a OpenConfigd integrates OpenConfig[11] model, that aims to packet handler thread. The packet handler thread is supposed develop programmatic interfaces and tools for managing net- to update keep-alive timer and to validate the packet, then the works in a dynamic and vendor-neutral way. thread puts into the shared lock free queue. With this architec- OpenConfigd has user interface shell called ’confsh’. It ture, protocol handling, such as SPF calculation, large scale communicates the configuration manager to show possi- BGP table updates, do not interfere with keep-alive timers. ble completion and show to execute the command. In As we learned from nginx[7] and node.js[8] achievement, Zebra 2.0, all of the protocol modules has configuration the task completion model with single thread has a lot of set/unset/validate callback functions which will be invoked advantage over to multi-thread model in performance stand from OpenConfigd. OpenConfigd use gRPC for transport to point. In Zebra 2.0 we combine the best part of task comple- allow easy micro-service integration. tion model and multi thread model. Forwarding Engine Abstraction Separation of Configuration Manager In past couple of years, new forwarding technology was in- In Zebra 2.0, configuration manager is not part of the vented. DPDK is one of the example. It delivers the packet implementation. Today’s industry require having a sin- directly from device buffer to user-space packet forwarder. gle integrated configuration manager which supports com- To handle the architecture, we need to have well designed mit/rollback with well designed configuration database. The FEA (Forwarding Engine Abstraction). Other example is OF- configuration manager handles all of the configuration on the DPA that wraps hardware functionality provided by merchant box or even boxes. To do that, we decouple configuration switch silicon with OpenFlow like APIs. With OF-DPA, manager from Zebra 2.0 then we introduce its own indepen- hardware forwarder configuration is much easier than it used dent software package called ‘OpenConfigd’. to be. Software forwarder and hardware forwarder is pretty The OpenConfigd generate configuration schema de- different. Therefore it require different API calls with dif- scribed in YANG[1]. Several interfaces and APIs such as CLI ferent kind of objects as arguments. In addition, sometimes (Command Line Interface), NETCONF and REST API are it require different API call sequences. Much worse, there Proceedings of netdev 1.1, Feb 10-12, 2016, Seville, Spain OpenFlow OVSDB CONF NET CLI L3 L2 SNMP might be a situation of supporting both software forwarder agent API API and hardware forwarder at the same time to handle slow path. 1.3 JSON IF In Zebra 2.0, new packet forwarding abstraction layer switch configuration datastore called FEA is introduced. FEA has sets of API which can (configuration/stats API, SW DSL) switch manager/ switch HAL automatically handle various different configuration under- Lagopussoftware dataplane neath of packet forwarders. FEA provides primitive con- Packet processing pipeline trol and configuration API for packet forwarder including in- Flow table Flow table Flow table flow Flow table GroupFlow table meterFlow table terface/port management, bridge management, routing table, table table table ARP table. A protocol modules must attach FEA and call queue/ flow lookup flow cache FEA APIs for packet forwarder controls. policer DPDK libs raw-socket / Kernel DPDK-enabled NIC Standard NIC Virtualization Figure 4: The design of Lagopus switch. Virtualization support is inevitable to today’s networking in- dustry for an execution environment of Zebra 2.0.