Modern Packet Switch Design White Paper Northforge Innovations Inc. September 2016 1 Introduction This white paper surveys the architectural evolution of Ethernet packet switches, featuring an overview of early packet switch design and the forces that have driven modern design solutions. These forces, which include increased interface speed, increased number of ports, Software-Defined Networking (SDN), Deep Packet Inspection (DPI), and Network Functions Virtualization (NFV) have had a major impact on the architecture and design of modern packet switches. All Packet Switches Are Divided Into Three Parts The networking industy has been building LAN-based packet switches (including under this rubric, LAN Bridges, MPLS Switches, and IP Routers) since the early-mid 1980s. The three functional components of a packet switch have been stable since the beginning. • Data plane – The primary job of a packet switch is to move packets from an input interface to an output interface. Moving the data from input to output is the job of the data plane (sometimes called the forwarding plane). • Control plane – The data plane decides which output port to select for each arriving packet based on a set of tables that are built by the control plane. For an Ethernet Switch, the control plane includes the process that learns MAC addresses, the spanning tree protocol, etc. For an IP router, the control plane includes the various IP routing protocols (e.g., OSPF and IS-IS). • Management plane – All packet switches require some configuration. In addition, they include mechanisms for fault detection and reporting, statistics collection, and troubleshooting. This is all done by the management plane. The management plane includes the CLI, structured management information (e.g., MIBs), and protocols for accessing management information (e.g., SNMP). The Early Years For the first 20 years or so, although LANs got faster, there were only two major changes in packet switch design and architecture. The first change occurred when LANs moved from shared coaxial cable to twisted pair (10BASE-T). When LANs were based on shared media, the packet switches had a relatively small number of interfaces (2 to 8) because each connected LAN could have dozens of systems connected. The packet switch connected networks together. When LANs moved to twisted pair cable and each switch interface connected to a single system (which might be another switch), the packet switch was the network. At that point we started seeing 8, 16, 32, 64 port switches. A related architectural change was the move from software-based implementations (based on standard CPUs, frequently Motorola 68000-series and occasionally Intel x86), often with some hardware assist, to ASIC-based implementations. Fortunately both of these changes happened around the same time (from about 1990 to 1995) since as the number of ports increased, software-based implementations became impractical. 1 Packet performance is usually based on minimum-sized packets (64 bytes). Although some aspects of forwarding performance may be limited by the actual data rate (which is higher with larger packets), per-packet functions such as address lookups tend to be the limiting factor, and minimum-sized packets have the highest packet rate. 2 Early Switch with Software Data Plane Early Switch with Switching SI Management Plane IF IF IF IF Control Plane CPU Data Plane Data Plane CPU Switch Control ASIC Management Optional Hardware Forwarding IF IF Assist The move to ASIC-based switching was critical to the success of packet switching. A 10Mbps Ethernet carries ~15,000 packets per second, so a packet switch with two 10Mbps Ethernet interfaces processes as many as 30,000 pps. This was feasible on a single CPU in 1990. A packet switch with two 100Mbps Ethernet interfaces processes 300,000 pps. This was feasible on a single CPU in the late 1990s. However, by then, it was common to deploy packet switches with 16 ports or more. In 1998 gigabit Ethernet was standardized increasing the packet rate on each interface by another factor of ten. Clearly, packet switching now depends on an ASIC-based forwarding plane. Today, switches with 64 or more 10Gbps ports are common. A 10Gbps Ethernet can source 15,000,000 pps, so these switches process a billion packets per second or more! During this period, very little happened in the control and management planes. The control plane migrated to newer protocols — at layer 2, Spanning Tree moved to Rapid and Multiple Spanning Tree and at layer 3, distance-vector protocols such as RIP and IGRP moved to link state protocols such as OSPF and IS-IS. MPLS was introduced during this period adding a new set of control protocols, a new set of data plane tables, and a new data plane forwarding algorithm. None of these improvements required substantial architectural change. The Later Years Ethernet keeps getting faster and the number of connected stations keeps increasing, so the data plane of packet switches continues to get bigger and faster. In the last 10 years there have been major changes to the functionality and implementation in all of the planes some of which have resulted in changes to the architecture and design of packet switches. The Modern Management Plane In the management plane there has been a migration towards a new way of expressing management information, moving away from ASN.1 and MIBs to a new data modeling language called YANG. YANG information is communicated using a new protocol called NETCONF (rather than SNMP). This is a major software change but does not result in any architectural changes. Another change in the management plane is the inclusion of Fault Management (FM) based on standards IEEE 802.1Q(ag) and ITU-T Y.1731 and Performance Monitoring (PM) based on ITU-T Y.1731. Although found primarily in carrier-based packet switches, they can 3 be used in enterprise networks also. These protocols require substantial software support and the regular exchange of protocol messages. The number of messages that need to be sent and received can be quite large (especially for PM), and therefore call for enhanced CPU capabilities or external hardware assist (e.g., an FPGA). Increasingly these capabilities are built right into the switch ASIC. The Modern Control Plane As noted above, the IP interior gateway protocols have migrated over the years from distance vector protocols to link state protocols. IP routers also frequently run BGP. At layer 2, the Spanning Tree Protocol migrated to Rapid (and Multiple) Spanning Tree Protocol. Newer layer 2 protection protocols such as ITU-T G.8032 Ethernet Ring Protection and G.8031 Ethernet Linear Protection became popular primarily because of their faster switching time. Also, MPLS became a staple on packet switches (especially those that supported IP), which brought along additional control protocols, primarily the Label Distribution Protocol (LDP) and RSVP-TE. None of these changes required substantial architectural changes in the switch. Some new control tables were needed and depending on the protocols supported and the design center with respect to network size, there was an effect on CPU power and the amount of memory needed for the control tables. A divergence has also developed over the past several years between the control protocols used in carrier networks (and enterprise networks) and those used in data centers. Data centers pose a unique challenge in that they have a very large number (hundreds or more) of colocated systems that must be interconnected with high bandwidth. This has resulted in development of new techniques and protocols. Two competing solutions are used in these environments, either the IEEE 802.1Q(aq) protocol Shortest Path Bridging (SPB), or the IETF protocol, Transparent Interconnection of Lots of Links (TRILL, several RFCs based on RFC6325). Over the past several years, however, a major new approach to packet switch control planes has garnered increasing interest – Software Defined Networking or SDN. The Control Plane is a body of software (firmware) that builds tables to drive the activity of the data plane. Control plane protocols are used to exchange information with the other systems in the network in order to derive the information necessary to build the tables. This is a distributed control plane since each node is performing its own computations to populate its forwarding tables. Software Defined Networking is a technique for centralizing the control plane. With SDN, a server communicates with the network elements to collect system and interface state information. The server then computes forwarding table information for each network element (NE) and distributes the new/updated information to each NE. With SDN, the control plane in each NE consists of a protocol processor that can receive the forwarding table updates and apply them to the data plane. The most common protocol for encoding and encapsulating this information is OpenFlow. There are several perceived2 benefits of SDN over the traditional distributed control plane: • Reduced reliance on proprietary protocol implementations – software embedded in network elements is developed, owned, and maintained by the equipment vendor. Improvements, enhancements, bug fixes, overall product direction, and, of course, cost are all based on the product strategy and direction of the vendor. With SDN, control software runs on an external server and therefore the network operator can purchase (or develop) software as needed. This can result in lower cost as well as faster adoption of new capabilities and features. • Lower equipment and network cost – with a distributed control plane each network element has “X” computes (generic unit of computation power) to apply to running the control plane protocols. In a network with 50 network elements there are 50X computes deployed for control plane protocols. If the processing is centralized, each network element requires substantially less horsepower for the control plane so each network element can cost less.