A First Look at QUIC in the Wild

Jan Rüth1, Ingmar Poese2, Christoph Dietzel3, Oliver Hohlfeld1

1: RWTH Aachen University 2: Benocs GmbH 3: TU Berlin / DE-CIX London / IETF-101 http://comsys.rwth-aachen.de/ Berlin / PAM, March 2018 The Internet is subject to a major change

— TCP is difficult to extend „ Only a couple of bytes left to modify „ Middleboxes ossify the protocol evolution — QUIC is UDP + TCP-like CC + loss recovery + TLS 1.3 „ Everything is encrypted à Goodbye middleboxes „ QUIC guarantees evolution — There is Google QUIC and IETF QUIC „ gQUIC evolves to IETF QUIC when standardization progresses

How much gQUIC is already out there?

Jan Rüth, Ingmar Poese, Christoph Dietzel, Oliver Hohlfeld 2 https://netray.io Measuring the QUIC Internet

— What infrastructures support QUIC? „ Perform ZMap scans over IPv4

— Is it (practically) used by any website? „ Scan full .com / .net / .org zones „ Scan Alexa Top 1M for “popular” domains

— How much QUIC traffic is there? „ in a university network „ in a major European Tier-1 ISP „ in a major European mobile network

„ in a major European IXP Picture by Seth Stoll (CC BY-SA 2.0): https://www.flickr.com/photos/sethstoll/4079897275

Jan Rüth, Ingmar Poese, Christoph Dietzel, Oliver Hohlfeld 3 https://netray.io Enumerating QUIC IPs with ZMap

— QUIC has a version negotiation feature (evolution!) „ Send version in first Client Hello (CHLO/Initial) „ If supported, server continues handshake „ Otherwise, sends version negotiation packet

— Use ZMap to test for gQUIC support + version support „ Send a valid CHLO packet „ Include an unused version number that the server does not support „ Module available on github

Jan Rüth, Ingmar Poese, Christoph Dietzel, Oliver Hohlfeld 4 https://netray.io QUIC in IPv4 600K 35..30QUIC 35..34 36..31 400K 36..32 36..34 37..34 # Hosts # Hosts 200K 37..35 38..34 38..35 0.0 39..35 39..37,35 40..37,35 Other 20.Aug16.Sep 2016 14.Oct 2016 11.Nov 2016 09.Dec 2016 06.Jan 2016 03.Feb 2017 03.Mar 2017 31.Mar 2017 28.Apr 2017 26.May 2017 201730.Jun 201728.Jul25.Aug 2017 22.Sep 2017 2017 — Number of QUIC IPs tripled to 617K IPs „ Classification using AS, certificate data, and reverse DNS „ 330K (53.53%) IPs can be attributed to Google „ Akamai: 983 in Aug., 44K in Nov. 2016, 251K (40.71%) October 2017 — HTTP via TCP on the remaining IPs reveal „ many timeouts and further Google and Akamai servers „ 7.34K using LightSpeed webserver „ 356 Caddy webservers

Jan Rüth, Ingmar Poese, Christoph Dietzel, Oliver Hohlfeld 5 https://netray.io QUIC in IPv4 600K 35..30QUIC 35..34 36..31 400K 36..32 36..34 37..34 # Hosts # Hosts 200K 37..35 38..34 38..35 0.0 39..35 39..37,35 40..37,35 Other 20.Aug16.Sep 2016 14.Oct 2016 11.Nov 2016 09.Dec 2016 06.Jan 2016 03.Feb 2017 03.Mar 2017 31.Mar 2017 28.Apr 2017 26.May 2017 201730.Jun 201728.Jul25.Aug 2017 22.Sep 2017 2017 — Number of QUIC IPs tripled to 617K IPs „ Classification using AS, certificate data, and reverse DNS „ 330K (53.53%) IPs can be attributed to Google „ Akamai: 983 in Aug., 44K in Nov. 2016, 251K (40.71%) October 2017 — HTTP via TCP onWeekly the remaining raw data available IPs reveal on https://quic.netray.io „ many timeouts and further Google and Akamai servers „ 7.34K using LightSpeed webserver „ 356 Caddy webservers

Jan Rüth, Ingmar Poese, Christoph Dietzel, Oliver Hohlfeld 6 https://netray.io QUIC website support

— There are ~150M domains in .com/.net/.org „ There are no tools to investigate QUIC

— We build upon quic-go to build a scanner „ We modify the library to trace the whole handshake „ We enable to dump all connection parameters that are exchanged ¾Certificates, server config, buffer settings, … „ Also available on github

— We can efficiently scan TLDs for QUIC support „ We can further analyze the connection parameters ¾Certificates valid?

Jan Rüth, Ingmar Poese, Christoph Dietzel, Oliver Hohlfeld 7 https://netray.io Fig. 3. Number of hosts giving out the same certificate on the y-axis. First listed common names for the 10 certificates with the highest coverage shown on the log x-axis. different certificates for the probed 617.59 K QUIC IPs. The heavy-tailed distribution shows the top-five (ten) certificates already represent 95.41% (99.28%) of the IPs, most prominently Google and Akamai. We validated that these IPs actually belong to both companies by requesting content via TCP and HTTP on port 80 on the same hosts. We next assess QUIC support among domain names. Probing complete domain lists. Presenting a non-existing SNI name in our previous measurement will miss any server that enforces to present a valid hostname, thus we next assess the QUIC support by probing complete domain name lists. That is, we probe all domains in the .com/.net/.org zone files and in the Alexa Top 1M list. These zones are available at Verisign [19] (.com/.net) and PIR [14] (.org). Together they contain more than 150M domains, i.e., about 46 % of the domain space [20]. We use zDNS to resolve the domains and for each successful resolution, we use our tool to check for QUIC support and to grab all parameters from the connection establishment. The whole process takes roughly 15 h and is thus feasible to run on a daily basis. Yet, as QUIC CHLO packets are padded to nearly fill the MTU, the scan easily saturates a 1Gbit link. Table 1 shows the QUIC-support in the .com/.net/.org zones as well as in the Alexa Top 1M list. We define QUIC-enabled domains as being able to initiate a QUIC hand- shake. A domain is tagged as Timeout when we received no response to our initial QUIC CHLO within 12 seconds, e.g., in the absence of QUIC support. We furthermore show some specific errors as well as DNS-failures. Overall QUIC-support is very low. Depending on the zone, 0.06%—0.1% domains QUICare in hosted TLDs on QUIC-enabled hosts. Only 1.6% – 2.44% of these domains present a valid X509 certificate. This questions how many domains actually deliver content via QUIC. 06. Oct 2017 03. Oct 2017 04. Oct 2017 08. Oct 2017 .com .net .org Alexa 1M # Domains 129.36 M (100.0%) 14.75 M (100.0%) 10.37 M (100.0%) 999.94 K (100.0%) QUIC-enabled 133.63 K (0.1%) 8.73 K (0.06%) 6.51 K (0.06%) 11.97 K (1.2%) Valid Certificate 2.14 K (0.0%) 181 (0.0%) 159 (0.0%) 342 (0.03%) Timeout 114.63 M (88.61%) 10.80 M (73.23%) 8.09 M (78.06%) 826.67 K (82.67%) Version-failed 29 (0.0%) 6 (0.0%) 1 (0.0%) 5 (0.0%) Protocol-error 606 (0.0%) 222 (0.0%) 0 (0.0%) 1 (0.0%) Invalid-IP 322.24 K (0.25%) 59.24 K (0.4%) 40.15 K (0.39%) 15.42 K (1.54%) DNS-failure 13.76 M (10.64%) 2.40 M (16.26%) 1.18 M (11.41%) 49.34 K (4.93%) „TableTimeout 1. QUIC à supportNo inQUIC different support TLDs and in the Alexa Top 1M list. Weekly data is available at https://quic.comsys.rwth-aachen.de. — In total: 161K domains „ 10% do not deliver any data „ 2.8K with valid certificate „ Only a fraction presents QUIC discovery headers via HTTP(s)

— Is there any QUIC traffic in the Internet then?

Jan Rüth, Ingmar Poese, Christoph Dietzel, Oliver Hohlfeld 8 https://netray.io QUIC Traffic Shares – Classification

— Classifying QUIC traffic „ Detecting QUIC on sampled (header) data is hard

— We rely on a port-based classification „ UDP Port 443 à QUIC „ TCP Port 443 à HTTPS „ TCP Port 80 à HTTP

— Depending on the data source we have AS-level information available

Jan Rüth, Ingmar Poese, Christoph Dietzel, Oliver Hohlfeld 9 https://netray.io QUIC Traffic Shares – Mawi Backbone

— Mawi: University uplink to ISP „ Open dataset: http://mawi.nezu.wide.ad.jp/mawi/ (samplepoint-F) „ Capped packet dumps of 15 minutes at 14h each day „ Source and destination have been anonymized

100 QUIC 80 HTTPS HTTP 60 Other 40 20

Traffic Share [%] 0 01.01 01.02 01.03 01.04 01.05 01.06 01.07 01.08 01.09 Month in 2017 „ No QUIC traffic in January last year MAWI 6.7% ¾Google said activation in January for most customers „ 5.2% QUIC in March, 6.7% in September

Jan Rüth, Ingmar Poese, Christoph Dietzel, Oliver Hohlfeld 10 https://netray.io QUIC Traffic Shares – Mawi Backbone

— Mawi: University uplink to ISP „ Open dataset: http://mawi.nezu.wide.ad.jp/mawi/End of paper (samplepoint-F) data „ Capped packet dumps of 15 minutes at 14h each day „ Source and destination have been anonymized

100 QUIC 80 HTTPS HTTP 60 Other 40 20

Traffic Share [%] 0 01.01 01.02 01.03 01.04 01.05 01.06 01.07 01.08 01.09 Month in 2017 „ No QUIC traffic Dailyin January updates last year available on MAWI 6.7% ¾Google said activationhttps://quic.netray.io in January for most customers „ 5.2% QUIC in March, 6.7% in September

Jan Rüth, Ingmar Poese, Christoph Dietzel, Oliver Hohlfeld 11 https://netray.io QUIC Traffic Shares – European Tier-1 ISP

— Anonymized Netflows of all boarder routers „ 1 full day in August 2017 (a Friday to Saturday) „ Netflows aggregated to 5 minute bins „ Upstream and Downstream „ IPs have been replaced by AS numbers „ Contains: edge (DSL,..), cellular, and transit backbone traffic

Akamai QUIC Google QUIC Other QUIC Akamai HTTPS Google HTTPS Other HTTPS bit/s Akamai HTTP Google HTTP Other HTTP Other Trac 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 00 01 Time of day on a Fri to Sat in Aug 2017 [h] Jan Rüth, Ingmar Poese, Christoph Dietzel, Oliver Hohlfeld 12 https://netray.io QUIC Traffic Shares – European Tier-1 ISP

Akamai QUIC Other QUIC Google HTTPS Akamai HTTP Other HTTP Google QUIC Akamai HTTPS Other HTTPS Google HTTP Other Traffic 100 80 60 [%] 40 bit/s 20 0 03 05 07 09 11 13 15 17 19 21 23 01 03 05 07 09 11 13 15 17 19 21 23 01 Time of day on a Fri to Sat in Aug 2017 [h]

— QUIC share is stable over the day — QUIC at around 7.8% (±1%) — HTTP (~38%) and HTTPS (40%) dominate the shares

— 98% of the QUIC traffic is from/to Google AS MAWI 6.7% „ Of all Google traffic QUIC accounts for ~39% peaking at ~42% ISP 7.8% — Almost no QUIC traffic to Akamai (0.1%) „ Still Akamai causes ~15% of all traffic à Potential QUIC traffic Jan Rüth, Ingmar Poese, Christoph Dietzel, Oliver Hohlfeld 13 https://netray.io QUIC Traffic Shares – Mobile ISP

Akamai QUIC Other QUIC Google HTTPS Akamai HTTP Other HTTP Google QUIC Akamai HTTPS Other HTTPS Google HTTP Other Traffic 100 80 60 [%] 40 bit/s 20 0 03 05 07 09 11 13 15 17 19 21 23 01 03 05 07 09 11 13 15 17 19 21 23 01 Time of day on a Fri to Sat in Aug 2017 [h] — ISP told us how to extract mobile traffic from trace — Mobile traffic pattern differs from classic daily pattern — QUIC share slightly larger 9.1% (±1.4%)

— Google dominates again MAWI 6.7% „ 34% of their traffic via QUIC ISP 7.8% MOBILE 9.1%

Jan Rüth, Ingmar Poese, Christoph Dietzel, Oliver Hohlfeld 14 https://netray.io QUIC Traffic Shares – European IXP

— Sampled flow data for the same day in August „ Flows annotated by customer port „ Aggregated to 5 minute bins

Akamai QUIC Other QUIC Google HTTPS Akamai HTTP Other HTTP Google QUIC Akamai HTTPS Other HTTPS Google HTTP Other Traffic 100 80 60 [%] 40 bit/s 20 0 03 05 07 09 11 13 15 17 19 21 23 03 05 07 09 11 13 15 17 19 21 23 Time of day on a Fri in Aug 2017 [h]

— QUIC only 2.6% overall MAWI 6.7% — Akamai accounts for ~60% of the QUIC traffic ISP 7.8% „ 33% by Google MOBILE 9.1% — Different traffic engineering strategies? IXP 2.6% Jan Rüth, Ingmar Poese, Christoph Dietzel, Oliver Hohlfeld 15 https://netray.io Summary

— QUIC is on the rise! „ A zoo of versions exist! Future compatibility? „ More and more infrastructure is enabled „ Some domains map to this infrastructure, only few can actually use it

— Non-negligible fraction of Internet traffic is QUIC „ Hard to detect in sampled data „ Very vantage point dependent „ Single companies have potential to drastically increase QUIC share „ How does QUIC impact Internet traffic? e.g., Fairness?

Jan Rüth, Ingmar Poese, Christoph Dietzel, Oliver Hohlfeld 16 https://netray.io We provide our tools and measurement data, visit https://quic.netray.io

Thank you!

Any questions?

PAM paper on arXiv: https://arxiv.org/abs/1801.05168

Thanks to RWTH Aachen ITC for enabling our measurements

This research is funded by the DFG SPP 1914

Jan Rüth, Ingmar Poese, Christoph Dietzel, Oliver Hohlfeld 17 https://netray.io