PCC: Re-architecting Congestion Control for Consistent High Performance Mo Dong and Qingxi Li, University of Illinois at Urbana-Champaign; Doron Zarchy, Hebrew University of Jerusalem; P. Brighten Godfrey, University of Illinois at Urbana-Champaign; Michael Schapira, Hebrew University of Jerusalem https://www.usenix.org/conference/nsdi15/technical-sessions/presentation/dong This paper is included in the Proceedings of the 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’15). May 4–6, 2015 • Oakland, CA, USA ISBN 978-1-931971-218 Open Access to the Proceedings of the 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’15) is sponsored by USENIX PCC: Re-architecting Congestion Control for Consistent High Performance Mo Dong*, Qingxi Li*, Doron Zarchy**, P. Brighten Godfrey*, and Michael Schapira** *University of Illinois at Urbana-Champaign **Hebrew University of Jerusalem Abstract a very difficult task within TCP’s rate control architec- ture, which we refer to as : certain TCP and its variants have suffered from surprisingly poor hardwired mapping predefined packet-level events are hardwired to certain performance for decades. We argue the TCP family has predefined control responses. TCP reacts to events that little hope of achieving consistent high performance due can be as simple as “one packet loss” (TCP New Reno) to a fundamental architectural deficiency: hardwiring or can involve multiple signals like “one packet loss and packet-level events to control responses. We propose RTT increased by x%” (TCP Illinois). Similarly, the con- Performance-oriented Congestion Control (PCC), a new trol response might be “halve the rate” (New Reno) or a congestion control architecture in which each sender more complex action like “reduce the window size w to continuously observes the connection between its ac- f ( RTT)w” (Illinois). The defining feature is that the tions and empirically experienced performance, enabling ∆ control action is a direct function of packet-level events. it to consistently adopt actions that result in high perfor- mance. We prove that PCC converges to a stable and A hardwired mapping has to make assumptions about fair equilibrium. Across many real-world and challeng- the network. Take a textbook event-control pair: a packet ing environments, PCC shows consistent and often 10 loss halves the congestion window. TCP assumes that × performance improvement, with better fairness and sta- the loss indicates congestion in the network. When bility than TCP. PCC requires no router hardware support the assumption is violated, halving the window size can or new packet format. severely degrade performance (e.g. if loss is random, rate should stay the same or increase). It is fundamentally hard to formulate an “always optimal” hardwired map- 1 Introduction ping in a complex real-world network because the actual optimal response to an event like a loss (i.e. decrease In the roughly 25 years since its deployment, TCP’s rate or increase? by how much?) is sensitive to network congestion control architecture has been notorious for conditions. And modern networks have an immense di- degraded performance. TCP performs poorly on lossy versity of conditions: random loss and zero loss, shallow links, penalizes high-RTT flows, underutilizes high queues and bufferbloat, RTTs of competing flows vary- bandwidth-delay product (BDP) connections, cannot ing by more than 1000 , dynamics due to mobile wire- handle rapidly changing networks, can collapse under × less or path changes, links from Kbps to Gbps, AQMs, data center incast [24] and incurs very high latency with software routers, rate shaping at gateways, virtualization bufferbloat [28] in the network. layers and middleboxes like firewalls, packet inspectors As severe performance problems have accumulated and load balancers. These factors add complexity far be- over time, protocol “patches” have addressed prob- yond what can be summarized by the relatively simplistic lems in specific network conditions such as high BDP assumptions embedded in a hardwired mapping. Most links [31,52], satellite links [23,42], data center [18,55], unfortunately, when its assumptions are violated, TCP wireless and lossy links [38,39], and more. However, the still rigidly carries out the harmful control action. fact that there are so many TCP variants suggests that each is only a point solution: they yield better perfor- We propose a new congestion control architecture: mance under specific network conditions, but break in Performance-oriented Congestion Control (PCC). PCC’s others. Worse, we found through real-world experiments goal is to understand what rate control actions improve that in many cases these TCP variants’ performance is performance based on live experimental evidence, avoid- still far away from optimal even in the network condi- ing TCP’s assumptions about the network. PCC sends at tions for which they are specially engineered. Indeed, a rate r for a short period of time, and observes the re- TCP’s low performance has impacted industry to the ex- sults (e.g. SACKs indicating delivery, loss, and latency tent that there is a lucrative market for special-purpose of each packet). It aggregates these packet-level events high performance data transfer services [1, 2, 11, 13]. into a utility function that describes an objective like Thus, the core problem remains largely unsolved: “high throughput and low loss rate”. The result is a sin- achieving consistently high performance over complex gle numerical performance utility u. At this point, PCC real-world network conditions. We argue this is indeed has run a single “micro-experiment” that showed send- 1 USENIX Association 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’15) 395 TCP Packet Level ing at rate r produced utility u. To make a rate control Events Hardwired decision, PCC runs multiple such micro-experiments: it Action 1 Network Action 2 tries sending at two different rates, and moves in the di- PCC Packet-level rection that empirically results in greater performance Events Performance objective Move to Sending Aggregate U1 Y rate R1 utility. This is effectively A/B testing for rate control rate R1 Network and is the core of PCC’s decisions. PCC runs these Performance U1>U2? objective Sending Aggregate micro-experiments continuously (on every byte of data, Network U2 N Move to rate R2 rate R2 not on occasional probes), driven by an online learn- ing algorithm that tracks the empirically-optimal send- Figure 1: The decision-making structure of TCP and PCC. ing rate. Thus, rather than making assumptions about f might traverse a shallow buffer on a high-BDP • the potentially-complex network, PCC adopts the actions link, with the loss due to bad luck in statistical mul- that empirically achieve consistent high performance. tiplexing rather than high link utilization. Then, PCC’s rate control is selfish in nature, but surpris- backing off a little is sufficient. ingly, using a widely applicable utility function, com- There may be a higher-rate competing flow. Then, • peting PCC senders provably converge to a fair equi- f should maintain its rate and let the other back off. librium (with a single bottleneck link). Indeed, experi- There may be random non-congestion loss some- • ments show PCC achieves similar convergence time to where along the path. Then, f should maintain or TCP with significantly smaller rate variance. Moreover, increase its rate. the ability to express different objectives via choice of Classically, TCP assumes a packet loss indicates non- the utility function (e.g. throughput or latency) provides negligible congestion, and that halving its rate will im- a flexibility beyond TCP’s architecture. prove network conditions. However, this assumption With handling real-world complexity as a key goal, is false and will degrade performance in three of the we experimentally evaluated a PCC implementation in four scenarios above. Fundamentally, picking an opti- large-scale and real-world networks. Without tweak- mal predefined and hardwired control response is hard ing its control algorithm, PCC achieves consistent high because for the same packet-level events, a control re- performance and significantly beats specially engineered sponse optimal under one network scenario can deci- TCPs in various network environments: (a.) in the wild mate performance in even a slightly different scenario. on the global commercial Internet (often more than 10 × The approach taken by a large number of TCP vari- the throughput of TCP CUBIC); (b.) inter-data center ants is to use more sophisticated packet-level events and networks (55.23 vs. TCP Illinois); (c.) emulated satel- × control actions. But this does not solve the fundamen- lite Internet links (1717 vs TCP Hybla); (d.) unreliable × tal problem, because the approach still hardwires pre- lossy links (1010 37 vs Illinois); (e.) unequal RTT of − × determined events to predetermined control responses, competing senders (an architectural cure to RTT un- thus inevitably embedding unreliable assumptions about fairness); (f.) shallow buffered bottleneck links (up to the network. When the unreliable assumptions are vi- 45 higher performance, or 13 less buffer to reach × × olated by the complexity of the network, performance 90% throughput); (g.) rapidly changing networks (1414 × degrades severely. For example, TCP Illinois [38] uses vs CUBIC, 5.6 vs Illinois). PCC performs similar to × both loss and delay to form an event-control mapping, ICTCP [55] in (h.) the incast scenario in data centers. but its throughput collapses with even a small amount of Though it is a substantial shift in architecture, PCC can random loss, or when network conditions are dynamic be deployed by only replacing the sender-side rate con- ( 4). More examples are in 5. § § trol of TCP. It can also deliver real data today with a user- Most unfortunately, if some control actions are indeed space implementation at speedier.net/pcc. harming performance, TCP can still blindly “jump off 2 PCC Architecture the cliff”, because it does not notice the control action’s actual effect on performance.