Evaluating the Impact of Path Brokenness on TCP Options Korian Edeline Benoit Donnet Université de Liège, Montefiore Institute, Belgium Université de Liège, Montefiore Institute, Belgium [email protected] [email protected] ABSTRACT purposes other than packet forwarding [3]). Firewalls and In-path network functions enforcing policies like firewalls, deep packet inspection (DPI) boxes, deployed for security IDSes, NATs, and TCP enhancing proxies are ubiquitous. purposes, TCP accelerators for performance enhancement, They are deployed in various types of networks and bring and network address translation (NATs) boxes have put an obvious value to the Internet. end to this paradigm [8]. Unfortunately, they also break important architectural Today, middleboxes proliferates in large numbers, in vari- principles and, consequently, make the Internet less flexible ous type of networks. In enterprise networks, middleboxes by preventing the use of advanced protocols, features, or op- are as numerous as regular network equipment [36]. Tier-1 tions. In some scenarios, feature-disabling middlebox policies ASes are deploying more and more middleboxes [7]. Cellular can lead to a performance shortfall. Moreover, middleboxes networks are extensively deploying Carrier-Grade NATs (CG- are also prone to enforce policies that disrupt transport con- NATs) [40]. Besides, recent progresses in virtualization (i.e., trol mechanisms, which can also have direct consequences hardware virtualization, containerization) and the introduc- in term of Quality-of-Service (QoS). tion of network function virtualization (NFV) are facilitating In this paper, we investigate the impact of the most preva- middlebox deployment [1, 11]. Overall, at least 2% of public lent in-path impairments on the TCP protocol and its features. network devices are TCP/IP middleboxes, mostly deployed at Using network experiments in a controlled environment, we AS borders, and they affect more than one third of network quantify the QoS decreases and shortfall induced by feature- paths [7, 9]. breaking middleboxes, and show that even in the presence Although they have made themselves indispensable, by of a fallback mechanism, TCP QoS remains affected. violating the end-to-end semantics, middleboxes have rad- ically changed the transport paradigm. Generic examples CCS CONCEPTS of such policies are shown in Fig. 1. As a side effect, they have also introduced a wide variety of impairments to pro- • Networks ! Middle boxes / network appliances; Net- tocols and features, from connectivity, to performance and work simulations; Transport protocols. security issues. Establishing TCP connections with Explicit 1 INTRODUCTION Congestion Notification (ECN) enabled can lead to connec- tivity blackouts [26]. Mobile carriers using middleboxes to The Internet landscape is constantly evolving. From the impose aggressive timeout value for idle TCP connections original end-to-end TCP/IP architecture, which ensured that increase mobile devices battery consumption. Careless TCP all packets exchanged across the Internet would stay un- middleboxes can facilitate certain network attacks, and even touched in-transit from the transport layer perspective, the bring new attack vectors [40]. Overall, at least 6.5% of net- last decade has witnessed a progressive introduction of mid- work paths are crossing a middlebox that potentially harms dleboxes (i.e., network appliances manipulating traffic for TCP traffic [7, 9]. Permission to make digital or hard copies of all or part of this work for Moreover, middleboxes forbids transport innovation [12]. personal or classroom use is granted without fee provided that copies Often referred to as the ossification of the network infras- are not made or distributed for profit or commercial advantage and that tructure, this phenomenon consists in middleboxes apply- copies bear this notice and the full citation on the first page. Copyrights ing modify or drop policies to packets, and limiting the for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or set of authorized features to a restricted subset. In conse- republish, to post on servers or to redistribute to lists, requires prior specific quences, alternatives transport protocols that do not rely on permission and/or a fee. Request permissions from [email protected]. TCP nor UDP, such as Datagram Congestion Control Proto- ANRW ’20, July 27–30, 2020, Online (Meetecho), Spain col (DCCP) [25], or Stream Control Transmission Protocol © 2020 Copyright held by the owner/author(s). Publication rights licensed (SCTP) [37], despite being standardized, fail to be deployed to ACM. at large scale. The situation within TCP is similar, with new ACM ISBN 978-1-4503-8039-3/20/07...$15.00 https://doi.org/10.1145/3404868.3406662 ANRW ’20, July 27–30, 2020, Online (Meetecho), Spain Edeline and Donnet (a) Direct (b) Indirect (a) feature.blocked (b) feature.removed (c) feature.changed Figure 2: Measurement Setups. TG = Traffic Generator. NS = Network Simulator. Arrows are physical connec- Figure 1: Path Conditions. tions. features being stripped or packets discarded, hampering with TCP innovation [31]. In this paper, we investigate the impact of middlebox- is used to recreate existing middlebox traffic impairments. induced TCP feature brokenness that were observed in the nsim is a simple network delay simulator implemented as a wild, in term of Quality-of-Service (QoS). We chose to mimic VPP plugin. It simply adds delay and shapes traffic by pro- existing middlebox impairments in a controlled environment, cessing packet vectors. By tuning mmb and nsim, the NS can because it requires control on both endpoints and on inter- be configured to simulate realistic scenarios of networks mediary devices, with the use of mmb [10, 11], a Vector Packet with path-breaking middleboxes. The NS runs VPP 18.10, Processing (VPP) [4] plugin that allows to build various state- DPDK [20] 18.08 with 10 1-GB huge pages, mmb 0.4, and less and stateful classification and rewriting middlebox poli- nsim, and the TGs run iperf3 [38]. The NS device is config- cies, and analyze their impact on the TCP traffic. We focus ured to maximize its performance, to make sure that it is not on three basic and widely used features, Explicit Congestion the measurements bottleneck. Notification (ECN), Selective ACKnowledgment (SACK), and The TG devices are configured to handle properly Long TCP Window Scaling (WScale), and highlight traffic disrupt- Fat Networks (LFNs) [21] scenarios (e.g., high bandwidth and ing policies affecting each feature. Finally, we make all data high delay), by increasing the TCP receive and send buffers generated and our Python Notebook freely available to the sizes to their maximum value (i.e., 2GB). Research Community1. We configured our testbed into two different setups: A di- The remainder of this paper is organized as follows: Sec. 2 rect client-to-server communication setup, shown in Fig. 2a, describes our experimental testbed hardware and configura- is used to evaluate bandwidth baselines and to rule out tion; Sec. 3 details the tested features, the chosen experiments sender-bounded experiments. An indirect setup, Fig. 2b, in and discusses the results; Sec. 4 presents the related works; which the NS forwards traffic between sender and receiver, finally, Sec. 5 concludes this paper by summarizing itsmain and applies the desired network conditions. achievements. As mentioned above, we generate traffic using iperf. In preliminary, we compute the TGs baseline bandwidth in the 2 EXPERIMENTAL SETUP direct setup, and the NS overhead in the indirect setup, to ensure that the processing time of the NS is not a bottleneck For quantifying the impact of path brokenness on QoS, of the measurements. To this end, we run a single pair of we deploy a testbed consisting of three machines with Intel iperf client-server using the direct setup, and we add iperf Xeon CPU E5-2620 2.10GHz, 16 Threads, 32GB RAM, run- client-server pairs until the bandwidth reaches the maximum ning Debian 9.0 with 4.9 kernels. Two of these machines play capacity. We found that at least 2 iperf client-server pairs are the role of Traffic Generators (TGs), while one is the Net- needed to reach a consistent 37.7 Gbps bandwidth, which is work Simulator (NS). Each machine is equipped with an Intel the closest that iperf can get to the maximum capacity of the XL710 2x40GB NIC connected to a Huawei CE6800 switch NICs. The experiment is repeated in the indirect setup, and using one port each for TGs and both for the NS. Traffic found a similar bandwidth of 37.4 Gbps. exchanged by TGs has to go through the NS first. For the following experiments, we will use a single TCP The NS relies on Vector Packet Processing (VPP) [4], a high- flow. We will vary different parameters, including thenet- performance userspace packet processing stack, and on the work conditions and the middlebox TCP interference, ob- mmb and nsim plugins to simulate realistic network condi- serve their effect on TCP, and attribute performance dete- tions and middlebox interference. mmb is a middlebox plugin rioration to the parameters. Each experiment lasts for 20 for VPP that allows to build various stateless and stateful seconds and packets are sized according to Ethernet MTU. classification and rewriting middlebox policies [10, 11]. It All NICs distribute packets to the RX rings by hashing both 1https://github.com/ekorian/vpp-quantify IP addresses and ports. Each experiment result is averaged Evaluating the Impact of Path Brokenness on TCP Options ANRW ’20, July 27–30, 2020, Online (Meetecho), Spain Consequences Path Conditions Affected Paths [9] BT DF ND DT tcp.seqnum.changed 5.5 % 7773 tcp.opt.sackok.removed 0.8 % 7337 tcp.opt.ws.changed 0.02 % 7733 tcp.opt.ws.removed 0.02 % 7377 tcp.ecn.blocked 0.01 % 3377 tcp.ecn.changed 0.01 % 7377 ip.ecn.changed 0.01 % 7373 Table 1: Middlebox Impairments Overview.