Charactersing the Limits of the OpenFlow Slow-Path

Richard Sanger, [email protected] Brad Cowie, [email protected] Matthew Luckie, [email protected] Richard Nelson, [email protected]

University of Waikato, New Zealand

28 November 2018 The Question

How slow is the slow-path?

© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 2 Contents

• Introduction to the Slow-Path • Motivation • Test Suite • Test Methodology • Results • Conclusions

© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 3 OpenFlow Packet-in and Packet-out

To move packets between the controller and network, packets are encapsulated in OpenFlow packet-in and packet-out messages and sent via the slow-path.

© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 4 The Fast-Path

ASIC OpenFlow Agent Ingress

Egress OpenFlow Switch

Network

© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 5 The Slow-Path (Packet In) ASIC OpenFlow Agent

Packet in

OpenFlow Switch Network Control-Plane Network

OpenFlow Application NIC

Controller

© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 6 Motivation: Control Traffic Requirements

Control traffic is sensitive to bandwidth and latency Latency • Keep-alives • Flow Establishment (Reactive control) Bandwidth • Initial route exchange (BGP etc.) • Capture (Network debugging) • DoS (Misconfiguration, ICMP, etc.)

© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 7 Motivation: Control Traffic Requirements

Control traffic requirements must be met simultaneously. Example: consider the requirement of link detection probing. • Typical Bidirectional Forwarding Detection (BFD) requirements • < 50ms • 2,880pps (48 port switch)

© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 8 Motivation: Shared Resource

The slow-path is shared with all other OpenFlow messages. • Adding flows • Flow or port statistics • Failure notifications

Messages from the controller should take priority over those generated from the network. • Packet-in should not delay a flow installation (reactive)

© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 9 Questions to answer

• Can the slow-path support my control traffic? • What bandwidth can the slow-path support? • What latency can the slow-path incur, and when? • Will packet-in or packet-out messages affect flow installation rate?

© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 10 Testing Framework: OFLOPS-Turbo

Based on OFLOPS-Turbo • Modular OpenFlow switch testing framework • Data packet generation • Data packet capture • Control channel (OpenFlow 1.0) • SNMP polling • Time Manager (events)

OFLOPS-Turbo: Rotsos, Charalampos, et al. "OFLOPS-Turbo: Testing the next-generation OpenFlow switch."

© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 11 Testing Framework: Modifications

OFLOPS-Turbo Modifications • Support OpenFlow 1.3 • Support encrypted control channel Libraries • Libfluid maintains the control channel • Revised OpenFlow Library (ROFL) - message construction

Available: https://github.com/wandsdn/oflops-turbo/

ROFL: https://github.com/bisdn/rofl-common Libfluid: http://opennetworkingfoundation.github.io/libfluid/ OFLOPS-Turbo: Rotsos, Charalampos, et al. "OFLOPS-Turbo: Testing the next-generation OpenFlow switch."

© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 12 Testing framework

© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 13 Tested Variations

• 5 Switches + OvS • OpenFlow 1.0 vs 1.3 • Packet size 64 byte vs 1500 byte • Encrypted vs unencrypted control channel • Packet-in buffering (off vs 56 bytes) • 5 different test modules

Each test was run 5 times, we report mean results.

© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 14 Base Variation

• OpenFlow 1.3 • Packet size 64 byte • Unencrypted control channel • Packet-in buffering off

© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 15 Switches Tested

AT-x930 P3780

HP3500 MLX4

NoviSwitch1248 Open vSwitch

© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 16 Switches Tested

Switch Type CPU ASIC AT-x930 Top-of-Rack PowerQUICC III Broadcom Allied Telesis 4x1GHz

P3780 Top-of-Rack PowerQICC III Broadcom Pica8 MPC8548 1GHz Trident+ E3500 Top-of-Rack PowerQICC III HP HP MPC8540 ProVision 666MHz MLX4 Chassis PowerPC G4 Linecard Extreme/Brocade MPC 7447 MI-MLX-1Gx20- 916MHz SFP NoviSwitch 1248 Top-of-Rack 64 Mellanox NPU NoviFlow i7-620LE EZchip NP-4 2x2.0GHz Open vSwitch Software Intel 64 Linux 3.16 Linux Foundation E5-2307v2 Kernel Datapath 2x4x2.4GHz

© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 17 Packet-in baseline test

• Fixed-rate generation • Record packet-in messages

Packet Timestamp Sequence # Switch OFLOPS Under pcap generation Test

OFLOPS Packet-in control channel Timestamp Sequence #

© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 18 Results: Packet-In bandwidth

Packet in processing rate with an input rate of 10,000pps

12000 e

t 10009 a 10000 R

g n i

s 8000 s d e n c o o r c 6000 P e

5145 S n 4788 4776

I r

t e e P

k 4000 c a P

n 2000

a 1008 e

M 100 0 x930 P3780 E3500 MLX-4 NS1248 OvS Switch

© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 19 Results: Packet-In Latency

Mean packet-in latency

600

500

400 x930 )

s P3780 m ( E3500 y 300 c

n MLX-4 e t NS1248 a 200 L OvS

100

0 100pps 1,000pps 10,000pps Packet-in rate (pps)

© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 20 Results: Packet-In MLX4 Timer 10,000pps

© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 21 Packet-out baseline test

• Fixed-rate generation • Record data-plane packets

Packet-out Timestamp Sequence # Switch OFLOPS Under control channel Test

OFLOPS Packet pcap capture Timestamp Sequence #

© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 22 Results: Packet-Out Bandwidth

Packet out processing rate with an input rate of 10,000pps

12000 e t

a 9921 10000 10000

R 10000

g 8562 n i s

s 8000 e d c n o r o c P

6000 e t u S

4692 r O

e t

e P 4000

k 3010 c a P 2000 n a e

M 0 x930 P3780 E3500 MLX-4 NS1248 OvS Switch

© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 23 Results: Packet-Out Latency

Mean packet-out latency

450 400 350 300 x930 )

s P3780 m

( 250 E3500 y c

n 200 MLX-4 e t NS1248 a 150 L OvS 100 50 0 100pps 1,000pps 10,000pps Packet-out rate (pps)

© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 24 Results: Packet-Out E3500 Timer 100pps

1000 0.2 900 0.18 800 0.16 )

700 0.14 s

d s n t o e 600 0.12 c k c e a S (

P 500 0.1

l y a c t

400 0.08 n o e T t

300 0.06 a L 200 0.04 100 0.02 0 0 0 2 4 6 8 10 Test Duration (Seconds)

© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 25 Results: Packet-Out MLX-4

When overloaded • Sends TCP a zero-window size • Resumes with a window size less than MSS • The Linux kernel waits for a 200ms timer

© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 26 Flow installation baseline test

• As fast as possible (match an IPv4 address) • Check using the data-plane

Flow-mod Match IPv4 Forward to OFLOPS Switch OFLOPS Under control channel Test

OFLOPS Packets matching flows pcap generation

OFLOPS pcap capture Default Drop

© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 27 Flow Installation results 500 flows

x930 P3780 E3500 MLX-4 NS1248 OvS 48 105 229 2,849 3,782 16,536

© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 28 Reactive control traffic

• Packet-in triggers packet-out

Packet Timestamp Sequence # Switch OFLOPS Under pcap generation Test Packet-in

Packet-out OFLOPS control channel Packet

OFLOPS pcap capture

© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 29 Results: Reactive control traffic

• Decrease in packet-ins (Priority given to packet outs) • E3500 36% • NS1248 29% • P3780 processes only 5% of packet-outs (Priority given to packet-ins) • MLX-4 and x930 are limited by packet-in rate

© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 30 Reactive control installation

• Install flows and generate packet-ins simultaneously • Flow installation not tied to packet-in

Switch Packet OFLOPS Under pcap generation Test Packet-in OFLOPS Flow-mod control channel

OFLOPS Packets matching flows pcap generation

OFLOPS pcap capture Default Drop

© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 31 Results: Reactive control installation

• Priority given to installing flows • x930 • E3500 • Priority given to packet-in • P3780 (73% decrease in installations, 25% in packet-in) • NS1248 (only one test result) • No result • MLX-4 (timers)

© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 32 Other factors

Switch x930 P3780 E3500 MLX-4 NS1248 Encryption — -37% — — 0% 1500B Packets 0% -36% -67% -28% -83% OpenFlow 1.0 0% +13% 0% -52% — Pkt In Buffering 0% -8% — — 0%

TABLE V: Percentage change in mean packet-in rate compared to the baseline 64B 10,000pps test (Table II). Missing results are due to unsupported features.

© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 33 Results: Overview

• All switches failed to meet <50ms of latency in some situations • Packet-in rate differ from 100 to 5,145pps • Switches have timers which can add 250ms delay • Switches may give priority to packet-in messages and starve other operations • SDN applications need to rate limit packet-outs • Consider alternatives to packet-in and packet-out messages

© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 34 Alternatives to the slow-path (App Dev)

• Route directly to the controller (Atrium) • Fails for ARP • Tunnel (VLAN, MPLS etc.) • Can conflict with other tunnels • Lose ingress port etc. (Mazu IPID) • VLAN tunnel (CacheFlow) • Proxy into existing OpenFlow application

© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 35 Alternatives to the slow-path (Vendor)

• Data-plane encapsulation • OpenFlow 1.3+ auxiliary channel • Channel for only packet-in, packet-out messages • Limited state when using UDP • Feasible data-plane (ASIC) implementation

© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 36 Questions

Code: https://github.com/wandsdn/oflops-turbo/

© THE UNIVERSITY OF WAIKATO • TE WHARE WANANGA O WAIKATO 37