Charactersing the Limits of the OpenFlow Slow-Path
Richard Sanger, [email protected] Brad Cowie, [email protected] Matthew Luckie, [email protected] Richard Nelson, [email protected]
University of Waikato, New Zealand
28 November 2018 The Question
How slow is the slow-path?
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 2 Contents
• Introduction to the Slow-Path • Motivation • Test Suite • Test Methodology • Results • Conclusions
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 3 OpenFlow Packet-in and Packet-out
To move packets between the controller and network, packets are encapsulated in OpenFlow packet-in and packet-out messages and sent via the slow-path.
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 4 The Fast-Path
ASIC OpenFlow Agent Ingress
Egress OpenFlow Switch
Network
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 5 The Slow-Path (Packet In) ASIC OpenFlow Agent
Packet in
OpenFlow Switch Network Control-Plane Network
OpenFlow Application NIC
Controller
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 6 Motivation: Control Traffic Requirements
Control traffic is sensitive to bandwidth and latency Latency • Keep-alives • Flow Establishment (Reactive control) Bandwidth • Initial route exchange (BGP etc.) • Capture (Network debugging) • DoS (Misconfiguration, ICMP, etc.)
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 7 Motivation: Control Traffic Requirements
Control traffic requirements must be met simultaneously. Example: consider the requirement of link detection probing. • Typical Bidirectional Forwarding Detection (BFD) requirements • < 50ms • 2,880pps (48 port switch)
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 8 Motivation: Shared Resource
The slow-path is shared with all other OpenFlow messages. • Adding flows • Flow or port statistics • Failure notifications
Messages from the controller should take priority over those generated from the network. • Packet-in should not delay a flow installation (reactive)
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 9 Questions to answer
• Can the slow-path support my control traffic? • What bandwidth can the slow-path support? • What latency can the slow-path incur, and when? • Will packet-in or packet-out messages affect flow installation rate?
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 10 Testing Framework: OFLOPS-Turbo
Based on OFLOPS-Turbo • Modular OpenFlow switch testing framework • Data packet generation • Data packet capture • Control channel (OpenFlow 1.0) • SNMP polling • Time Manager (events)
OFLOPS-Turbo: Rotsos, Charalampos, et al. "OFLOPS-Turbo: Testing the next-generation OpenFlow switch."
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 11 Testing Framework: Modifications
OFLOPS-Turbo Modifications • Support OpenFlow 1.3 • Support encrypted control channel Libraries • Libfluid maintains the control channel • Revised OpenFlow Library (ROFL) - message construction
Available: https://github.com/wandsdn/oflops-turbo/
ROFL: https://github.com/bisdn/rofl-common Libfluid: http://opennetworkingfoundation.github.io/libfluid/ OFLOPS-Turbo: Rotsos, Charalampos, et al. "OFLOPS-Turbo: Testing the next-generation OpenFlow switch."
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 12 Testing framework
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 13 Tested Variations
• 5 Switches + OvS • OpenFlow 1.0 vs 1.3 • Packet size 64 byte vs 1500 byte • Encrypted vs unencrypted control channel • Packet-in buffering (off vs 56 bytes) • 5 different test modules
Each test was run 5 times, we report mean results.
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 14 Base Variation
• OpenFlow 1.3 • Packet size 64 byte • Unencrypted control channel • Packet-in buffering off
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 15 Switches Tested
AT-x930 P3780
HP3500 MLX4
NoviSwitch1248 Open vSwitch
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 16 Switches Tested
Switch Type CPU ASIC AT-x930 Top-of-Rack PowerQUICC III Broadcom Allied Telesis 4x1GHz
P3780 Top-of-Rack PowerQICC III Broadcom Pica8 MPC8548 1GHz Trident+ E3500 Top-of-Rack PowerQICC III HP HP MPC8540 ProVision 666MHz MLX4 Chassis PowerPC G4 Linecard Extreme/Brocade MPC 7447 MI-MLX-1Gx20- 916MHz SFP NoviSwitch 1248 Top-of-Rack Intel 64 Mellanox NPU NoviFlow i7-620LE EZchip NP-4 2x2.0GHz Open vSwitch Software Intel 64 Linux 3.16 Linux Foundation E5-2307v2 Kernel Datapath 2x4x2.4GHz
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 17 Packet-in baseline test
• Fixed-rate generation • Record packet-in messages
Packet Timestamp Sequence # Switch OFLOPS Under pcap generation Test
OFLOPS Packet-in control channel Timestamp Sequence #
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 18 Results: Packet-In bandwidth
Packet in processing rate with an input rate of 10,000pps
12000 e
t 10009 a 10000 R
g n i
s 8000 s d e n c o o r c 6000 P e
5145 S n 4788 4776
I r
t e e P
k 4000 c a P
n 2000
a 1008 e
M 100 0 x930 P3780 E3500 MLX-4 NS1248 OvS Switch
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 19 Results: Packet-In Latency
Mean packet-in latency
600
500
400 x930 )
s P3780 m ( E3500 y 300 c
n MLX-4 e t NS1248 a 200 L OvS
100
0 100pps 1,000pps 10,000pps Packet-in rate (pps)
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 20 Results: Packet-In MLX4 Timer 10,000pps
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 21 Packet-out baseline test
• Fixed-rate generation • Record data-plane packets
Packet-out Timestamp Sequence # Switch OFLOPS Under control channel Test
OFLOPS Packet pcap capture Timestamp Sequence #
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 22 Results: Packet-Out Bandwidth
Packet out processing rate with an input rate of 10,000pps
12000 e t
a 9921 10000 10000
R 10000
g 8562 n i s
s 8000 e d c n o r o c P
6000 e t u S
4692 r O
e t
e P 4000
k 3010 c a P 2000 n a e
M 0 x930 P3780 E3500 MLX-4 NS1248 OvS Switch
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 23 Results: Packet-Out Latency
Mean packet-out latency
450 400 350 300 x930 )
s P3780 m
( 250 E3500 y c
n 200 MLX-4 e t NS1248 a 150 L OvS 100 50 0 100pps 1,000pps 10,000pps Packet-out rate (pps)
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 24 Results: Packet-Out E3500 Timer 100pps
1000 0.2 900 0.18 800 0.16 )
700 0.14 s
d s n t o e 600 0.12 c k c e a S (
P 500 0.1
l y a c t
400 0.08 n o e T t
300 0.06 a L 200 0.04 100 0.02 0 0 0 2 4 6 8 10 Test Duration (Seconds)
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 25 Results: Packet-Out MLX-4
When overloaded • Sends TCP a zero-window size • Resumes with a window size less than MSS • The Linux kernel waits for a 200ms timer
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 26 Flow installation baseline test
• As fast as possible (match an IPv4 address) • Check using the data-plane
Flow-mod Match IPv4 Forward to OFLOPS Switch OFLOPS Under control channel Test
OFLOPS Packets matching flows pcap generation
OFLOPS pcap capture Default Drop
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 27 Flow Installation results 500 flows
x930 P3780 E3500 MLX-4 NS1248 OvS 48 105 229 2,849 3,782 16,536
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 28 Reactive control traffic
• Packet-in triggers packet-out
Packet Timestamp Sequence # Switch OFLOPS Under pcap generation Test Packet-in
Packet-out OFLOPS control channel Packet
OFLOPS pcap capture
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 29 Results: Reactive control traffic
• Decrease in packet-ins (Priority given to packet outs) • E3500 36% • NS1248 29% • P3780 processes only 5% of packet-outs (Priority given to packet-ins) • MLX-4 and x930 are limited by packet-in rate
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 30 Reactive control installation
• Install flows and generate packet-ins simultaneously • Flow installation not tied to packet-in
Switch Packet OFLOPS Under pcap generation Test Packet-in OFLOPS Flow-mod control channel
OFLOPS Packets matching flows pcap generation
OFLOPS pcap capture Default Drop
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 31 Results: Reactive control installation
• Priority given to installing flows • x930 • E3500 • Priority given to packet-in • P3780 (73% decrease in installations, 25% in packet-in) • NS1248 (only one test result) • No result • MLX-4 (timers)
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 32 Other factors
Switch x930 P3780 E3500 MLX-4 NS1248 Encryption — -37% — — 0% 1500B Packets 0% -36% -67% -28% -83% OpenFlow 1.0 0% +13% 0% -52% — Pkt In Buffering 0% -8% — — 0%
TABLE V: Percentage change in mean packet-in rate compared to the baseline 64B 10,000pps test (Table II). Missing results are due to unsupported features.
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 33 Results: Overview
• All switches failed to meet <50ms of latency in some situations • Packet-in rate differ from 100 to 5,145pps • Switches have timers which can add 250ms delay • Switches may give priority to packet-in messages and starve other operations • SDN applications need to rate limit packet-outs • Consider alternatives to packet-in and packet-out messages
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 34 Alternatives to the slow-path (App Dev)
• Route directly to the controller (Atrium) • Fails for ARP • Tunnel (VLAN, MPLS etc.) • Can conflict with other tunnels • Lose ingress port etc. (Mazu IPID) • VLAN tunnel (CacheFlow) • Proxy into existing OpenFlow application
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 35 Alternatives to the slow-path (Vendor)
• Data-plane encapsulation • OpenFlow 1.3+ auxiliary channel • Channel for only packet-in, packet-out messages • Limited state when using UDP • Feasible data-plane (ASIC) implementation
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 36 Questions
Code: https://github.com/wandsdn/oflops-turbo/
© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 37