Multipath TCP: from Theory to Practice

MultiPath TCP - Architecture MultiPath TCP - Receive-Buﬀer MultiPath TCP - Congestion Control MultiPath TCP - Live-Demo

MultiPath TCP: From Theory to Practice

S´ebastienBarr´e Christoph Paasch Olivier Bonaventure

9 mai 2011

http ://inl.info.ucl.ac.be/mptcp/

SébastienBarré- Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 1 / 17 MultiPath TCP - Architecture MultiPath TCP - Receive-Buffer MultiPath TCP - Congestion Control MultiPath TCP - Live-Demo MultiPath TCP

Mobile devices can connect to the Internet via diﬀerent interfaces 3G

Internet

WiFi Data-centers have a large redundant infrastructure

Dual-Homed Servers

SébastienBarré- Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 2 / 17 MultiPath TCP - Architecture MultiPath TCP - Receive-Buffer MultiPath TCP - Congestion Control MultiPath TCP - Live-Demo MultiPath TCP

Mobile devices can connect to the Internet via diﬀerent interfaces 3G

Internet

WiFi Data-centers have a large redundant infrastructure

Dual-Homed Servers

State of the Art TCP is used for 95% of the Internet communications A single TCP connection cannot be used across diﬀerent interfaces.

MultiPath TCP (short MPTCP) MPTCP allows a single data-connection to use several interfaces simultaneously. Allows failover from one interface to another (e.g., mobile client). Increases the bandwidth due to resource pooling.

SébastienBarré- Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 3 / 17 MultiPath TCP - Architecture MultiPath TCP - Receive-Buffer MultiPath TCP - Congestion Control MultiPath TCP - Live-Demo

MultiPath TCP : From Theory To Practice S. Barr´e,C. Paasch, O. Bonaventure

We implemented MultiPath TCP in the Linux Kernel. Solved several design challenges that needed to be considered. Evaluated and measured the impact of our design choices in a testbed. Live-Demo of MPTCP at the end of this presentation.

SébastienBarré- Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 4 / 17 MultiPath TCP - Architecture MultiPath TCP - Receive-Buffer MultiPath TCP - Congestion Control MultiPath TCP - Live-Demo MultiPath TCP - Architecture - From Theory

Application Layer

standard Socket API

Transport Layer

MultiPath TCP

TCP TCP TCP subflow subflow subflow

Network Layer

WiFi Wired 3G

SébastienBarré- Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 5 / 17 MultiPath TCP - Architecture MultiPath TCP - Receive-Buffer MultiPath TCP - Congestion Control MultiPath TCP - Live-Demo MultiPath TCP - Architecture - To Practice

Sending packets over MultiPath TCP

Application Layer

standard Socket API

Transport Layer

Meta-socket MultiPath TCP send-queue

Master Slave Slave subsocket subsocket subsocket TCP TCP TCP subflow subflow subflow

Network Layer

SébastienBarré- Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 6 / 17 MultiPath TCP - Architecture MultiPath TCP - Receive-Buffer MultiPath TCP - Congestion Control MultiPath TCP - Live-Demo MultiPath TCP - Architecture - To Practice

Sending packets over MultiPath TCP

Application Layer

standard Socket API

Transport Layer

Meta-socket MultiPath TCP send-queue

MultiPath TCP Scheduler

Master Slave Slave subsocket subsocket subsocket TCP TCP TCP subflow subflow subflow

Network Layer

SébastienBarré- Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 7 / 17 MultiPath TCP - Architecture MultiPath TCP - Receive-Buffer MultiPath TCP - Congestion Control MultiPath TCP - Live-Demo MultiPath TCP - Architecture - To Practice

Receiving packets over MultiPath TCP Packets can be reordered at the data-level due to delay-diﬀerences.

out-of-order queue 8 7 6 5 4 M low-delay path M subflow 1 subflow 1 P P T T C subflow 2 subflow 2 C P high-delay path P 3 2 1 receive-queue

SébastienBarré- Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 8 / 17 MultiPath TCP - Architecture MultiPath TCP - Receive-Buffer MultiPath TCP - Congestion Control MultiPath TCP - Live-Demo MultiPath TCP - Architecture - To Practice

Receiving packets over MultiPath TCP A loss at the subﬂow-level (or network-reordering) can also cause reordering at the subﬂow-level

dropped packet out-of-order queue 9 10 8 7 6 5 4 M low-delay path M subflow 1 subflow 1 P P T T C subflow 2 subflow 2 C P high-delay path P 3 2 1 receive-queue

SébastienBarré- Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 9 / 17 MultiPath TCP - Architecture MultiPath TCP - Receive-Buffer MultiPath TCP - Congestion Control MultiPath TCP - Live-Demo MultiPath TCP - Architecture - To Practice

Receiving packets over MultiPath TCP Subﬂow-level out-of-order queues are necessary to handle the retransmission at the subﬂow-level

retransmitted packet subflow out-of-order queue 8 out-of-order queue 7 6 5 4 10 9 M low-delay path M subflow 1 subflow 1 P P T T C subflow 2 subflow 2 C P high-delay path P 3 2 1 receive-queue

SébastienBarré- Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 10 / 17 MultiPath TCP - Architecture MultiPath TCP - Receive-Buffer MultiPath TCP - Congestion Control MultiPath TCP - Live-Demo MultiPath TCP - Architecture - To Practice

Receiving packets over MultiPath TCP

Application Layer

standard Socket API

Transport Layer

Meta-socket MultiPath TCP send-queue receive-queue ofo-queue

Master Slave Slave subsocket subsocket subsocket TCP TCP TCP subflow subflow subflow ofo-queue ofo-queue ofo-queue

Network Layer

SébastienBarré- Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 11 / 17 MultiPath TCP - Architecture MultiPath TCP - Receive-Buffer MultiPath TCP - Congestion Control MultiPath TCP - Live-Demo MultiPath TCP - Architecture - To Practice

Interconnected testbed with two separate paths at 1Gbps

2000

1500

1Gbps 1000

1Gbps iperf goodput (Mbps ) 500

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 MSS (bytes )

SébastienBarré- Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 12 / 17 MultiPath TCP - Architecture MultiPath TCP - Receive-Buffer MultiPath TCP - Congestion Control MultiPath TCP - Live-Demo MultiPath TCP - Architecture - To Practice

regular TCP can only use one single path

2000

1500

1Gbps regular TCP 1000

1Gbps iperf goodput (Mbps ) 500

regular TCP 0 1000 2000 3000 4000 5000 6000 7000 8000 9000 MSS (bytes )

MultiPath TCP uses both paths simultaneously

2000

1500

1Gbps MPTCP 1000

1Gbps iperf goodput (Mbps ) 500

regular TCP MultiPath TCP 0 1000 2000 3000 4000 5000 6000 7000 8000 9000 MSS (bytes )

Regular TCP : receive-buﬀer should be twice the BDP Support a fast-retransmit Support network-level reordering MultiPath TCP : Higher reordering possible due to delay-diﬀerences of the paths. out-of-order queue 8 7 6 5 4 M low-delay path M subflow 1 subflow 1 P P T T C subflow 2 subflow 2 C P high-delay path P 3 2 1 receive-queue

SébastienBarré- Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 13 / 17 MultiPath TCP - Architecture MultiPath TCP - Receive-Buffer MultiPath TCP - Congestion Control MultiPath TCP - Live-Demo MultiPath TCP - Receive-Buffer - To Practice

⇒A subﬂow on a slow path (high RTT) may block a subﬂow on a fast path from transmitting.

RTTmax

SébastienBarré- Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 14 / 17 MultiPath TCP - Architecture MultiPath TCP - Receive-Buffer MultiPath TCP - Congestion Control MultiPath TCP - Live-Demo MultiPath TCP - Receive-Buffer - To Practice

⇒A subﬂow on a slow path (high RTT) may block a subﬂow on a fast path from transmitting.

⇒We need to cope with one fast-retransmit and network-level reordering on this slow path

RTTmax ∗ 2

⇒A subﬂow on a slow path (high RTT) may block a subﬂow on a fast path from transmitting.

⇒We need to cope with one fast-retransmit and network-level reordering on this slow path

⇒During this time, all other subﬂows should be able to transmit at full speed (BWi = bandwidth of subﬂow i)

X RTTmax ∗ 2 ∗ BWi i∈subﬂows

Regular TCP only gets 100Mbps of goodput

200

180

160

100Mbps 140 regular TCP 120

100 100Mbps

+ 0ms iperf goodput (Mbps ) 80

60 TCP (0 ms ec) 40 2 4 6 8 10 12 14 16 rcvbuf (MB)

SébastienBarré- Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 15 / 17 MultiPath TCP - Architecture MultiPath TCP - Receive-Buffer MultiPath TCP - Congestion Control MultiPath TCP - Live-Demo MultiPath TCP - Receive-Buffer - To Practice

MultiPath TCP doubles the goodput

200

180

160

100Mbps 140 MPTCP 120 100 100Mbps

+ 0ms iperf goodput (Mbps ) 80 MPTCP (0 ms ec, 0 ms ec)

60 TCP (0 ms ec) 40 2 4 6 8 10 12 14 16 rcvbuf (MB)

10ms and 100ms of diﬀerence has not a huge inﬂuence

200

180

160

100Mbps 140 MPTCP 120 100 100Mbps

+ 10ms iperf goodput (Mbps ) 80 MPTCP (0 ms ec, 0 ms ec) MPTCP (0 ms ec, 10 ms ec) 60 TCP (0 ms ec) 40 2 4 6 8 10 12 14 16 rcvbuf (MB)

10ms and 100ms of diﬀerence has not a huge inﬂuence

200

180

160

100Mbps 140 MPTCP 120 100 100Mbps

+ 100ms iperf goodput (Mbps ) 80 MPTCP (0 ms ec, 0 ms ec) MPTCP (0 ms ec, 10 ms ec) 60 MPTCP (0 ms ec, 100 ms ec) TCP (0 ms ec) 40 2 4 6 8 10 12 14 16 rcvbuf (MB)

500ms of difference affects the goodput significantly

200

180

160

100Mbps 140 MPTCP 120 100 100Mbps

+ 500ms iperf goodput (Mbps ) 80 MPTCP (0 ms ec, 0 ms ec) MPTCP (0 ms ec, 10 ms ec) 60 MPTCP (0 ms ec, 100 ms ec) MPTCP (0 ms ec, 500 ms ec) TCP (0 ms ec) 40 2 4 6 8 10 12 14 16 rcvbuf (MB)

Several subﬂows will ”eat up” regular TCP

100

80 regular TCP shared Bottleneck 33% 60

66% 40 MPTCP

2 subflows 20 Average Goodput in Mbps TCP MPTCP-Reno 0 1 2 3 Number of MPTCP-Subflows

SébastienBarré- Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 16 / 17 MultiPath TCP - Architecture MultiPath TCP - Receive-Buffer MultiPath TCP - Congestion Control MultiPath TCP - Live-Demo MultiPath TCP - Congestion Control

Our implementation of the Coupled Congestion Control 1 is fair to regular TCP across shared bottlenecks and avoids the paths with congested links.

100

80 regular TCP shared Bottleneck 50% 60

50% 40 MPTCP

2 subflows 20 Average Goodput in Mbps TCP MPTCP-CCC 0 1 2 3 Number of MPTCP-Subflows

1. D. Wischik, C. Raiciu, A. Greenhalgh, M. Handley. ”Design, implementation and evaluation of congestion control for multipath TCP”. NSDI’2011 SébastienBarré- Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 16 / 17 MultiPath TCP - Architecture MultiPath TCP - Receive-Buffer MultiPath TCP - Congestion Control MultiPath TCP - Live-Demo MultiPath TCP - Live-Demo

Access http ://inl.info.ucl.ac.be/mptcp to download our implementation (source-code, Live-CD,...). Contributions are always welcome ! ! !

Setup for the Live-Demo :

SébastienBarré- Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 17 / 17 MultiPath TCP - Architecture MultiPath TCP - Receive-Buffer MultiPath TCP - Congestion Control MultiPath TCP - Live-Demo MultiPath TCP - Congestion Control - Theory

Coupled Congestion Control 2

For each ack on subﬂow i :

cwndi = cwndi + min(α/cwndtot , 1/cwndi )

2 cwndi ∗mssi maxi ( 2 ) RTTi α = cwndtot (P cwndi ∗mssi )2 i RTTi

2. D. Wischik, C. Raiciu, A. Greenhalgh, M. Handley. ”Design, implementation and evaluation of congestion control for multipath TCP”. NSDI’2011 SébastienBarré- Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 17 / 17 MultiPath TCP - Architecture MultiPath TCP - Receive-Buffer MultiPath TCP - Congestion Control MultiPath TCP - Live-Demo MultiPath TCP - Congestion Control - Practice

Congestion Control in the Linux Kernel Limitations : Congestion window is expressed in packets Linux Kernel cannot handle ﬂoating point numbers

Coupled Congestion Control implementation Count the number of acknowledged packets per subﬂow in cwnd cnti . Increase congestion window by one, if cwnd cnti > max(totcwnd /α, 1/cwndi ) Calculate alpha by minimizing the number of divisions and scaling the remaining ones :

cwndmax ∗ scalenum α = cwndtot (P rttmax ∗cwndi ∗scaleden )2 i rtti

S´ebastienBarr´e- Christoph Paasch - Olivier Bonaventure MultiPath TCP: From Theory to Practice 17 / 17