Observing the Evolution of QUIC Implementations Maxime Piraux, Quentin De Coninck, Olivier Bonaventure UCLouvain fmaxime.piraux,quentin.deconinck,[email protected] ABSTRACT MUST study start The QUIC protocol combines features that were initially 250 MUST NOT found inside the TCP, TLS and HTTP/2 protocols. The SHOULD 200 SHOULD NOT IETF is currently finalising a complete specification of this protocol. More than a dozen of independent implementations 150 have been developed in parallel with these standardisation activities. 100 We propose and implement a QUIC test suite that interacts Number of appearances 50 with public QUIC servers to verify their conformance with key features of the IETF specification. Our measurements, 0 gathered over a semester, provide a unique viewpoint on the 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 evolution of the QUIC protocol and of its implementations. Draft version They highlight the introduction of new features and some Figure 1: Keywords in draft-ietf-quic-transport. regressions among the different implementations. ACM Reference Format: The QUIC IETF working group was chartered in late 2016. It Maxime Piraux, Quentin De Coninck, Olivier Bonaventure. 2018. is currently finalising the specifications for the first standards- Observing the Evolution of QUIC Implementations. In Workshop track version of QUIC. During this period, the IETF working on the Evolution, Performance, and Interoperability of QUIC group published fourteen versions of the main protocol draft (EPIQ'18), December 4, 2018, Heraklion, Greece. ACM, New York, and more than a dozen of QUIC implementations are publicly NY, USA, 7 pages. https://doi.org/10.1145/3284850.3284852 available. The efforts of the QUIC designers and implementers is 1 INTRODUCTION probably unique in the history of protocol design and im- Internet transport protocols usually evolve slowly. Any sig- plementation. Although many IETF protocols have been nificant evolution to TCP, the dominant transport protocol, designed, very few have attracted so many implementers takes years of efforts to be widely deployed. There are several while the protocol was still being developed. As an illustra- factors that explain this slow evolution [19]. On one hand, tion of the complexity of the QUIC protocol, Figure 1 plots TCP is usually implemented inside the operating system the number of RFC2119 keywords (i.e. \MUST", \MUST kernel and upgrading kernels remains costly and slow. On NOT", \SHOULD", \SHOULD NOT") in the different ver- the other hand, there are a growing number of middleboxes sions of draft-ietf-quic-transport. We can observe that on the Internet that block new TCP extensions [13]. QUIC, the number of these indicators has more than doubled since initially proposed by Google [15] addresses this ossification the first version of the specification. in several ways. First, QUIC runs above UDP, which implies Implementing network protocols is not a trivial task and that it can be distributed as a userspace library that can be several authors have proposed techniques to test and validate easily upgraded. Second, QUIC encrypts most of the con- protocol implementations. Some of these techniques rely on trol information and payload in order to prevent middlebox formal methods to automatically derive the test suite from interferences. the implementation [7, 12]. However, it is difficult to apply The results obtained by Google with QUIC [15] combined them to Internet protocols since their specifications are in- with its security features have convinced the IETF to stan- formal. Researchers have proposed different solutions to test dardise a new protocol starting from Google's initial design. and validate protocol implementations. Some have proposed techniques to validate complete TCP implementations [1, 17]. Permission to make digital or hard copies of all or part of this work Paxson et al. proposed specific tools to validate the confor- for personal or classroom use is granted without fee provided that mance of TCP implementations [20, 21]. With TBIT, Padhye copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first and Floyd developed techniques to infer the characteristics of page. Copyrights for components of this work owned by others than TCP implementations by interacting with them with specially ACM must be honored. Abstracting with credit is permitted. To copy crafted packets [18]. These tools have played an important otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions role in improving TCP implementations. from [email protected]. The dozen QUIC implementations [8] that are actively EPIQ'18, December 4, 2018, Heraklion, Greece being developed will likely face similar problems as TCP © 2018 Association for Computing Machinery. ACM ISBN 978-1-4503-6082-1/18/12. $15.00 implementations during the last decades [21]. The interoper- https://doi.org/10.1145/3284850.3284852 ability tests that are regularly organised by the QUIC IETF EPIQ'18, December 4, 2018, Heraklion, Greece Maxime Piraux, Quentin De Coninck, Olivier Bonaventure working group have helped to identify some ambiguities in Produce Reads Test scenarii Traces Dissector the specifications and solve interoperability problems. In this paper, we contribute to this implementation effort with a Call Calls publicly available and detailed test suite for QUIC. To our knowledge, this is the first public test suite for this new QUIC toolbox Web site protocol. Exchange QUIC packets We first describe the architecture of our test suite inSec- Consult tion 2. Section 3 analyses results collected during a 6-month over the Internet period using our test suite on the public servers that already Implementations Implementers implement QUIC. Section 4 concludes this paper by review- ing the future prospects for our work and assessing how it can be freely improved, reused and extended. Figure 2: Tools forming the test suite. 2.2 Architecture 2 THE QUIC TEST SUITE Figure 2 illustrates the architecture of our test suite. Akin to TBIT [18], our test suite is constituted of several self- In this section we first describe both the approach and the contained scenarios. Each scenario is self-contained in the architecture of our test suite. We then depict the test scenarios sense that it establishes a new QUIC connection to perform that constitute it. the test. A separate connection increases the isolation between two scenarios, in an attempt to accurately test a specific mechanism. Each test addresses a particular feature of the 2.1 Testing approach QUIC protocol. Combining several scenarios within a single Network protocol testing approaches can be categorised ac- connection is left as future work. cording to two dimensions, black-box versus white-box testing We implement the scenarios on top of a high-level API, and passive versus active testing. The first dimension defines i.e. the QUIC toolbox, that allows easily manipulating QUIC the perspective chosen to evaluate an implementation, i.e. packets. Pieces of QUIC client behaviour are implemented an external or internal perspective. Because we want to test as asynchronous message passing objects, called agents. We all QUIC implementations without relying on source code implemented 9 different agents responsible for, e.g. issuing availability, we chose the black-box approach. Our test suite ACK frames in response of received packets, retransmitting lost only observes their external behaviours, i.e. the packets sent data, interacting with TLS, decrypting and parsing QUIC and receiver, to evaluate them. packets, bundling frames into packets and performing 1-RTT The second dimension defines how the tool behave with handshake. respect to the implementation under test (IUT). The first ap- This increases modularity by defining the behaviour ofa proach is passive testing. It has been used in earlier works [14, test without having to reimplement mechanisms shared by 20, 24], but is of limited use with QUIC given that most of several tests. For instance, the address validation scenario, the packets are encrypted. which tests whether a server validates the client address be- The second approach is active testing, in which the tool fore sending significant amount of data to it, does not send used for experiments actively exchanges messages with the acknowledgements but retransmits lost data and derive ses- IUT. It requires the IUT to be available when using the sion keys from the handshake to decrypt all received packets. tool. Several studies have applied this approach to TCP. Therefore, it disables the agent responsible for acknowledge- TBIT [18] was one of the pioneering work in this domain. It ments, but enables the agents interacting with TLS and has been extended in later works [4, 27]. Conducting active sending retransmissions. tests with TCP is becoming more difficult today given the The QUIC toolbox depends on picotls [5] for using TLS 1.3 deployment of various types of middleboxes that may interfere through a Go binding we wrote [23]. The interaction between with active tests [11, 13, 16]. By encrypting most of the TLS and the toolbox is isolated in a separate 115-line long control information and payload, QUIC exposes a smaller agent. One can thus replace the TLS stack used by imple- surface subject to ossification. menting a new agent providing equivalent functionalities. The objective of our test suite is to check the conformance We synchronise the different agents using specific events of QUIC server-side implementations by only exchanging inside a connection, e.g. a packet has been received or sent out, packets with them. Given that the IETF specifications are a new encryption scheme is available. This paradigm allows still being developed, we only cover a subset of them. We attaching new behaviours upon reception of these events want the tool to be autonomous in two manners. It must be without requiring extra coordination with other agents or able to create QUIC packets on its own to perform the tests.