Charactersing the Limits of the OpenFlow Slow-Path Richard Sanger, [email protected] Brad Cowie, [email protected] Matthew Luckie, [email protected] Richard Nelson, [email protected] University of Waikato, New Zealand 28 November 2018 The Question How slow is the slow-path? © THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 2 Contents • Introduction to the Slow-Path • Motivation • Test Suite • Test Methodology • Results • Conclusions © THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 3 OpenFlow Packet-in and Packet-out To move packets between the controller and network, packets are encapsulated in OpenFlow packet-in and packet-out messages and sent via the slow-path. © THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 4 The Fast-Path ASIC OpenFlow Agent Ingress Egress OpenFlow Switch Network © THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 5 The Slow-Path (Packet In) ASIC OpenFlow Agent Packet in OpenFlow Switch Network Control-Plane Network OpenFlow Application NIC Controller © THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 6 Motivation: Control Traffic Requirements Control traffic is sensitive to bandwidth and latency Latency • Keep-alives • Flow Establishment (Reactive control) Bandwidth • Initial route exchange (BGP etc.) • Capture (Network debugging) • DoS (Misconfiguration, ICMP, etc.) © THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 7 Motivation: Control Traffic Requirements Control traffic requirements must be met simultaneously. Example: consider the requirement of link detection probing. • Typical Bidirectional Forwarding Detection (BFD) requirements • < 50ms • 2,880pps (48 port switch) © THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 8 Motivation: Shared Resource The slow-path is shared with all other OpenFlow messages. • Adding flows • Flow or port statistics • Failure notifications Messages from the controller should take priority over those generated from the network. • Packet-in should not delay a flow installation (reactive) © THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 9 Questions to answer • Can the slow-path support my control traffic? • What bandwidth can the slow-path support? • What latency can the slow-path incur, and when? • Will packet-in or packet-out messages affect flow installation rate? © THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 10 Testing Framework: OFLOPS-Turbo Based on OFLOPS-Turbo • Modular OpenFlow switch testing framework • Data packet generation • Data packet capture • Control channel (OpenFlow 1.0) • SNMP polling • Time Manager (events) OFLOPS-Turbo: Rotsos, Charalampos, et al. "OFLOPS-Turbo: Testing the next-generation OpenFlow switch." © THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 11 Testing Framework: Modifications OFLOPS-Turbo Modifications • Support OpenFlow 1.3 • Support encrypted control channel Libraries • Libfluid maintains the control channel • Revised OpenFlow Library (ROFL) - message construction Available: https://github.com/wandsdn/oflops-turbo/ ROFL: https://github.com/bisdn/rofl-common Libfluid: http://opennetworkingfoundation.github.io/libfluid/ OFLOPS-Turbo: Rotsos, Charalampos, et al. "OFLOPS-Turbo: Testing the next-generation OpenFlow switch." © THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 12 Testing framework © THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 13 Tested Variations • 5 Switches + OvS • OpenFlow 1.0 vs 1.3 • Packet size 64 byte vs 1500 byte • Encrypted vs unencrypted control channel • Packet-in buffering (off vs 56 bytes) • 5 different test modules Each test was run 5 times, we report mean results. © THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 14 Base Variation • OpenFlow 1.3 • Packet size 64 byte • Unencrypted control channel • Packet-in buffering off © THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 15 Switches Tested AT-x930 P3780 HP3500 MLX4 NoviSwitch1248 Open vSwitch © THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 16 Switches Tested Switch Type CPU ASIC AT-x930 Top-of-Rack PowerQUICC III Broadcom Allied Telesis 4x1GHz P3780 Top-of-Rack PowerQICC III Broadcom Pica8 MPC8548 1GHz Trident+ E3500 Top-of-Rack PowerQICC III HP HP MPC8540 ProVision 666MHz MLX4 Chassis PowerPC G4 Linecard Extreme/Brocade MPC 7447 MI-MLX-1Gx20- 916MHz SFP NoviSwitch 1248 Top-of-Rack Intel 64 Mellanox NPU NoviFlow i7-620LE EZchip NP-4 2x2.0GHz Open vSwitch Software Intel 64 Linux 3.16 Linux Foundation E5-2307v2 Kernel Datapath 2x4x2.4GHz © THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 17 Packet-in baseline test • Fixed-rate generation • Record packet-in messages Packet Timestamp Sequence # Switch OFLOPS Under pcap generation Test OFLOPS Packet-in control channel Timestamp Sequence # © THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 18 Results: Packet-Inbandwidth Mean Packet In Processing Rate Per Second © THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO • WANANGA TE WHARE OF WAIKATO © THE UNIVERSITY 10000 12000 2000 4000 6000 8000 Packet in processing rate with an input rate of 10,000pps of rate 10,000pps input an with rate processing in Packet 0 x930 100 P3780 4788 E3500 4776 Switch MLX-4 1008 NS1248 5145 10009 OvS 19 Results: Packet-In Latency Mean packet-in latency 600 500 400 x930 ) s P3780 m ( E3500 y 300 c n MLX-4 e t NS1248 a 200 L OvS 100 0 100pps 1,000pps 10,000pps Packet-in rate (pps) © THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 20 Results: Packet-In MLX4 Timer 10,000pps © THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 21 Packet-out baseline test • Fixed-rate generation • Record data-plane packets Packet-out Timestamp Sequence # Switch OFLOPS Under control channel Test OFLOPS Packet pcap capture Timestamp Sequence # © THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 22 Results: Packet-Out Bandwidth Mean Packet Out Processing Rate Per Second © THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO • WANANGA TE WHARE OF WAIKATO © THE UNIVERSITY 10000 12000 2000 4000 6000 8000 Packet out processing rate with an input rate of 10,000pps of rate 10,000pps input an with rate processing out Packet 0 x930 9921 P3780 4692 E3500 8562 Switch MLX-4 3010 NS1248 10000 10000 OvS 23 Results: Packet-Out Latency Mean packet-out latency 450 400 350 300 x930 ) s P3780 m ( 250 E3500 y c n 200 MLX-4 e t NS1248 a 150 L OvS 100 50 0 100pps 1,000pps 10,000pps Packet-out rate (pps) © THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 24 Results: Packet-Out E3500 Timer 100pps 1000 0.2 900 0.18 800 0.16 ) 700 0.14 s d s n t o e 600 0.12 c k c e a S ( P 500 0.1 l y a c t 400 0.08 n o e T t 300 0.06 a L 200 0.04 100 0.02 0 0 0 2 4 6 8 10 Test Duration (Seconds) © THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 25 Results: Packet-Out MLX-4 When overloaded • Sends TCP a zero-window size • Resumes with a window size less than MSS • The Linux kernel waits for a 200ms timer © THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 26 Flow installation baseline test • As fast as possible (match an IPv4 address) • Check using the data-plane Flow-mod Match IPv4 Forward to OFLOPS Switch OFLOPS Under control channel Test OFLOPS Packets matching flows pcap generation OFLOPS pcap capture Default Drop © THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 27 Flow Installation results 500 flows x930 P3780 E3500 MLX-4 NS1248 OvS 48 105 229 2,849 3,782 16,536 © THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 28 Reactive control traffic • Packet-in triggers packet-out Packet Timestamp Sequence # Switch OFLOPS Under pcap generation Test Packet-in Packet-out OFLOPS control channel Packet OFLOPS pcap capture © THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 29 Results: Reactive control traffic • Decrease in packet-ins (Priority given to packet outs) • E3500 36% • NS1248 29% • P3780 processes only 5% of packet-outs (Priority given to packet-ins) • MLX-4 and x930 are limited by packet-in rate © THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 30 Reactive control installation • Install flows and generate packet-ins simultaneously • Flow installation not tied to packet-in Switch Packet OFLOPS Under pcap generation Test Packet-in OFLOPS Flow-mod control channel OFLOPS Packets matching flows pcap generation OFLOPS pcap capture Default Drop © THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 31 Results: Reactive control installation • Priority given to installing flows • x930 • E3500 • Priority given to packet-in • P3780 (73% decrease in installations, 25% in packet-in) • NS1248 (only one test result) • No result • MLX-4 (timers) © THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 32 Other factors Switch x930 P3780 E3500 MLX-4 NS1248 Encryption Ð -37% Ð Ð 0% 1500B Packets 0% -36% -67% -28% -83% OpenFlow 1.0 0% +13% 0% -52% Ð Pkt In Buffering 0% -8% Ð Ð 0% TABLE V: Percentage change in mean packet-in rate compared to the baseline 64B 10,000pps test (Table II). Missing results are due to unsupported features. © THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 33 Results: Overview • All switches failed to meet <50ms of latency in some situations • Packet-in rate differ from 100 to 5,145pps • Switches have timers which can add 250ms delay • Switches may give priority to packet-in messages and starve other operations • SDN applications need to rate limit packet-outs • Consider alternatives to packet-in and packet-out messages © THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 34 Alternatives to the slow-path (App Dev) • Route directly to the controller (Atrium) • Fails for ARP • Tunnel (VLAN, MPLS etc.) • Can conflict with other tunnels • Lose ingress port etc. (Mazu IPID) • VLAN tunnel (CacheFlow) • Proxy into existing OpenFlow application © THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 35 Alternatives to the slow-path (Vendor) • Data-plane encapsulation • OpenFlow 1.3+ auxiliary channel • Channel for only packet-in, packet-out messages • Limited state when using UDP • Feasible data-plane (ASIC) implementation © THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 36 Questions Code: https://github.com/wandsdn/oflops-turbo/ © THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 37.