Rethinking Packet Forwarding Hardware Martin Casado∗x Teemu Koponeny Daekyeong Moonz Scott Shenkerzx 1 Introduction should not be tied to particular networking hardware, For routers and switches to handle ever-increasing band- and networking hardware should not be restricted width requirements, the packet “fast-path” must be to particular protocols. General-purpose network handled with specialized hardware. There have been two processors have largely failed the first requirement, approaches to building such packet forwarding hardware. while the current generation of commodity network- The first is to embed particular algorithms in hardware; ing chips fail the second requirement. this is what most commodity forwarding chips do (e.g., • Hardware simplicity: In order to scale to increasing those from Broadcom, Marvell, and Fulcrum). These speeds, the hardware functionality should be of chips have led to amazing increases in performance and very limited complexity. Again, both current reductions in cost; for instance, one can now get 24 ports approaches to hardware acceleration fail to satisfy of gigabit ethernet for under $1000. this requirement. Unfortunately, this approach offers only very rigid func- • Flexible and efficient functionality: The resulting tionality; one can’t change protocols or add new features software-hardware combination should be capable that require hardware acceleration without redoing the of realizing a wide variety of networking functions chip. This forces network forwarding enhancements to at high-speed and low-cost, while having short evolve on hardware design timescales, which are glacially development cycles. Commodity chips work at slow compared to the rate at which network applications high-speed and low-cost, but have long development and requirements are changing. cycles for new functionality. General-purpose To counter this inflexibility, several vendors have network processors are not yet competitive on speed taken a different approach by introducing more flexible or cost. “network processors”. These have not been as successful In this paper we propose a different approach to as anticipated, for at least two reasons. First, designers hardware packet forwarding that has the potential to were never able to find a sweet-spot in the tradeoff realize all of these goals. Our approach assigns a between hardware simplicity and flexible functionality, completely different role to hardware. Traditionally, so the performance/price ratios have lagged well behind hardware implementations have embodied the logic commodity networking chips. For another, the interface required for packet forwarding. That is, the hardware provided to software has proven hard to use, requiring had to capture all the complexity inherent in a packet protocol implementors to painstakingly contort their forwarding decision. code to the idiosyncrasies of the particular underlying In contrast, in our approach all forwarding decisions hardware to achieve reasonable performance. These two are done first in software, and then the hardware merely problems (among others) have prevented general-purpose mimics these decisions for subsequent packets to which network processors from dislodging the more narrowly that decision applies (e.g., all packets destined for the targeted commodity packet forwarding hardware that same prefix). Thus, the hardware does not need to dominates the market today. understand the logic of packet forwarding, it merely In this paper, we step back from these two well-known caches the results of previous forwarding decisions (taken approaches and ask more fundamentally: what would we by software) and applies them to packets with the want from packet forwarding hardware, how might we same headers. The key task is to match incoming achieve it, and what burden does that place on networking packets to previous decisions, so the required hardware software? is nothing more than a glorified TCAM, which is simple Ideally, hardware implementations of the forwarding to implement (compared to other networking hardware) paths should have the following properties: and simple to reason about. We describe this approach in more detail in Section2. • Clean interface between hardware and software: One key challenge in this approach is scaling to The hardware-software interface should allow each high speeds. Achieving high forwarding rates is not a to evolve independently. Protocol implementations matter of the number of clock cycles needed to make ∗Nicira Networks Inc. a forwarding decision, but instead is one of cache hit yHelsinki Institute for Information Technology (HIIT) rates; what fraction of packet forwarding can be done zUC Berkeley, Computer Science Division based on the forwarding of previous packets? This xInternational Computer Science Institute (ICSI) question is central to the viability of this approach, and 1 we address it in Section3. Another key challenge but other rules (such as “packets with TTL=1 should be is deciding to which packets a particular forwarding dropped”) can be applied to all ingress ports. decision applies (i.e., which fields in the packet header Matching rule: The software specifies the fields in the must match in order to apply a decision), and when this packet header that must match for the flow entry to decision has been rendered obsolete by changing network apply. This could be the entire header (i.e., requiring state (e.g., changes in forwarding tables); these system a perfect match), or only some fields (e.g., perhaps only design issues are addressed in Section4. the destination address to match). This approach has been motivated by a long line Forwarding action: The forwarding action is the set of of flow-oriented approaches to networking hardware, output ports to which the packet should be sent, along from [5] to [2] (and many others). What makes our with (optionally) the rewritten packet headers for each approach different from traditional connection-oriented port. Dropping the packet is represented by a special null architectures (e.g., ATM) is that our approach does output port. not require any change to networking protocols. Our State dependence: This represents the local state on approach (similar to the references [2, 5]) only changes which the forwarding decision is based (such as entries in the interface between hardware and software, but does a routing table or ACL database). not affect protocols or the Internet architecture. Like [3] When a packet arrives, it is compared to the table of we seek to provide a flexible software model with flow entries corresponding to its ingress port. If its packet hardware forwarding speeds, however in our approach we header does not match any of the entries, then it is sent propose to fully decouple the software from the hardware, to software for processing. If it matches a flow entry, the while [3] provides direct access to hardware resources packet is handled according to the specification of the such as the TCAM and special purpose ASICs. We view forwarding action in the flow entry. Whenever a piece this as a necessary and complimentary step for integrating of local state changes, such as an update to an ACL or with existing hardware. routing table, then all flow entries that depended on that state are invalidated and removed from the flow table. 2 Our Approach Before proceeding to examples, we mention a few more 2.1 General Description detailed points. First, the matching rules must be defined so a packet matches no more than one rule. Since we Packet forwarding decisions deterministically depend on assume that the software deterministically forwards each the header of the arriving packet and on some local state in packet based on its header fields and particular system 1 the router/switch (such as a routing table). For example, state, it should be possible to generate non-conflicting the header/state combination for ethernet forwarding is entries at runtime. the destination MAC address and the L2 learning table, Second, we view stateful tunneling (e.g., IPsec in whereas for IP forwarding it is the destination IP address which a sequence number is updated for every packet) as and the FIB. virtual output ports rather than part of the forwarding The forwarding decision includes both the output logic. This is similar to how they are handled in port(s) on the network device as well as possible modifi- standard software IP forwarding engines in which they are cations to the packet header, such as the label swapping in modeled as networking interfaces. If we did not use this MPLS or packet encapsulation/decapsulation. As long as technique then no flow entry would be recorded for such the relevant pieces of local state have not changed, then flows, because the state on which the forwarding actions all packets with the same packet header should receive the depends (which includes per-packet modifications) would same forwarding behavior. This statement applies to any be updated on every new packet arrival. forwarding protocol that depends only on packet headers Third, there are a number of possible system designs and local state. that could determine the headers and system state used To leverage this fact, we treat packet forwarding as a in a decision, such as a special-purpose programming matching process, with all packets matching a previous language, a specialized compiler or even hardware decision handled by the hardware, and all non-matching support. We discuss these further in Section4. However, packets handled by the software. We assume that when a the primary goal of this paper is not to give a specific and packet is handled by software, we can infer a flow entry comprehensive system design (we are working on this for that packet which contains four pieces of information: and plan to discuss it in future work), but rather to explore Ingress port(s): This specifies to which ingress ports a the fundamental viability of our general approach. flow entry applies. Many rules will only apply to the ingress port of the packet that generated the flow entry, 2.2 Examples 1We are ignoring forwarding decisions based on deep-packet To demonstrate how our proposal operates in practice, we inspection.