Measuring HTTP/3: Adoption and Performance

Martino Trevisan†, Danilo Giordano†, Idilio Drago ‡, Ali Khatouni? †Politecnico di Torino, ‡University of Turin, ?Shopify first.last@{polito.it, unito.it, shopify.com}

Abstract—The third version of the Hypertext Transfer real state of deployment nor the performance benefits of Protocol (HTTP) is currently in its final standardization phase HTTP/3 have been measured yet. by the IETF. Besides better security and increased flexibility, In this paper, we fill this gap by running a large-scale it promises benefits in terms of performance. HTTP/3 adopts a more efficient header compression schema and replaces TCP measurement study. Firstly, we collect data to understand to with QUIC, a transport protocol carried over UDP, originally what extent the web ecosystem already adopted HTTP/3. proposed by and currently under standardization too. Secondly, we run an experimental campaign to measure Although HTTP/3 early implementations already exist and the benefits introduced by HTTP/3 on the users’ Quality some websites announce its support, it has been subject to of Experience (QoE), and how HTTP/3 helps in different few studies. In this work, we provide a first measurement study on network conditions. HTTP/3. We testify how, during 2020, it has been adopted Using the open-source HTTPArchive Dataset, we find by some of the leading Internet companies such as Google, thousands of websites supporting HTTP/3, most of them and Cloudflare. We run a large-scale measurement owned by a handful of Internet hyper-giants, i.e., Facebook, campaign toward thousands of websites adopting HTTP/3, Google, and Cloudflare. We automatically visit the HTTP/3 aiming at understanding to what extent it achieves better performance than HTTP/2. We find that adopting websites websites under diverse network conditions to measure the often host most web page objects on third-party servers, which performance benefits in terms of QoE-related metrics. We support only HTTP/2 or even HTTP/1.1. Our experiments visit 14 707 websites while emulating artificial latency, show that HTTP/3 provides sizable benefits only in scenarios , and limiting the . In total, we with high latency or very poor bandwidth. Despite the run 735 350 visits over a period of one month. We find adoption of QUIC, we do not find benefits in case of high packet loss, but we observe large diversity across website that HTTP/3 benefits emerge only on particular network providers’ infrastructures. conditions and strongly differ across website providers. Our Index Terms—HTTP/3 ; Performance ; Measurements key findings are the following: • Google, Facebook and Cloudflare are the early I.INTRODUCTION adopters of HTTP/3, representing almost the totality The Hypertext Transfer Protocol (HTTP) is the king of of currently HTTP/3 enabled websites. web protocols and is used to access a plethora of services • For most of them, the majority of web page objects available on the Internet, from websites to social networks are still hosted on non-HTTP/3 third-party servers. and collaborative platforms. HTTP was born in the early • We observe sizable performance benefits only in sce- 90s, and its first version 1.1 was standardized in 1997 [1]. narios with high latency or poor bandwidth. For more than a decade, this version remained unchanged. • The performance speed-up largely depends on the Only in 2014, HTTP/2 [2] was proposed, with substantial infrastructure hosting the website. changes in the framing mechanisms. HTTP/3 is the third • Websites requiring few connections to load are those version of HTTP and is currently in the final standard- benefiting the most. ization phase at the IETF [3]. It promises performance The remainder of the paper is organized as follows: benefits and security improvements compared to version Section II describes HTTP/3 and illustrates related work. arXiv:2102.12358v1 [cs.NI] 24 Feb 2021 2. As a major change, HTTP/3 replaces TCP as transport Section III presents the datasets we use and how we collect layer in favor of QUIC, a UDP-based protocol originally them, and Section IV illustrates our results on HTTP/3 proposed by Google and currently being standardized by adoption and performance. Finally, Section V states the the IETF [4]. Furthermore, it introduces a more effective limitations of our measurements and poses the basis for header compression mechanism and exploits TLS 1.3 [5] future work, while Section VI concludes the paper. or higher to improve the level of security. HTTP/3 is expected to supplant HTTP/2 in the next II.BACKGROUND years, and some of the leading Internet companies already A. HTTP/3 announced its support during 2020 (e.g., the CloudFlare 1 2 The HTTP/3 protocol is the third version of the well- CDN and Facebook ). However, currently neither the known Hypertext Transfer Protocol, born in the 90s to transfer multimedia content and hyper-textual documents 1https://blog.cloudflare.com/http3-the-past-present-and-future/ 2https://engineering.fb.com/2020/10/21/networking-traffic/ over the nascent Internet. With its version 1.1, it has how-facebook-is-bringing--to-billions/ been the king of web protocols for more than 20 years, TABLE I: Description of the employed datasets. HTTP/1.1 HTTP/2 HTTP/3 Dataset Visits Goal TLS TLS QUIC HTTPArchive 53 107 185 (optional) (+TLS 1.3) HTTP/3 Adoption BrowserTime 735 350 HTTP/3 Performance TCP TCP UDP

IP Google firstly proposed QUIC in 2012 and, as such, has been the subject of many studies. For example, Wolsing et Fig. 1: for different HTTP versions. al. [12] show that QUIC still outperforms TCP thanks to the fast connection setup. Manzoor et al. [13] show how QUIC performs worse than TCP in Wireless Mesh Networks. superseded only by its second version HTTP/2 in 2014. Wolsing et al. [14] found that QUIC reduces the overall HTTP/2 implemented several novel features, especially page retrieval time compared with standard HTTP. Kakhi et to improve how the data is framed and transported. It al. [15] run a large-scale measurement campaign on QUIC, promised to make the web faster even if its benefits have finding that it outperforms TCP in most cases. However, not universally accepted [6], [7]. these works target older Google’s QUIC versions, while the HTTP/3 is the third version of HTTP. It is currently current standard proposed at the IETF has made significant in the final standardization phase, reaching the 34th draft progress [16]. Moreover, they focus uniquely on QUIC as version [3] and making it stable and usable for real de- the , neglecting the improvement introduced ployments. The main improvements from version 2 include in HTTP/3 and TLS 1.3, that we aim at measuring in this more efficient header compression, advanced security fea- work. tures based on TLS 1.3, and, especially, the use of QUIC at the transport layer. The resulting protocol stack is thus III.DATA COLLECTION heavily modified, as we show in Figure 1. QUIC, initially We rely on two datasets to study (i) the adoption developed by Google, is a transport protocol based on UDP of HTTP/3 and (ii) its performance on diverse network and is currently in the standardization phase too [4]. QUIC conditions. We summarize them in Table I. revisits TCP, moving congestion control in user space and allowing faster handshakes. Moreover, it solves the long- A. HTTPArchive standing issue of head-of-line blocking, allowing multiple We study the adoption of HTTP/3 using the independent streams within the same connection. Indeed, HTTPArchive, an open dataset available online.3 The QUIC allows independent retransmission for sub-streams dataset contains metadata coming from visits to a list and decouples it from congestion control. This operation of more than 5 M URLs provided by the Chrome User is expected to improve users’ QoE with faster website Experience Report.4. The list of URL is built using the responsiveness, especially in scenarios with poor network navigation data of real Chrome users and contains a conditions. HTTP/3 also mandates the use TLS 1.3 [5], representative view of the most popular website and web directly incorporated at the QUIC layer. Besides the dep- services accessed worldwide.5 Each month, all URLs are recation of older cipher suites, it allows 1-RTT handshakes visited using the browser from a US data and 0-RTT resumption, further reducing session setup time. center, and the resulting navigation data is made public. For each visit, the dataset contains information about the B. Related Work page performance and characteristics as well as details on all the HTTP transactions in the HAR format6, including Given its recent conception, few works already targeted request and response headers. HTTP/3. Saif et al. [8] run experiments controlling both Fundamental for our analyses, the details of HTTP client and server accessing a single web page. They study responses indicate the eventual Alt-Svc header, which the effect of delay, packet loss and throughput but do not is used by servers to announce support to HTTP/3. By measure any major impact on performance. In contrast setting the Alt-Svc header, the server has the possibility to them, we run a large-scale measurement campaign, to inform the client to make subsequent connections using controlling only the client and targeting thousands of HTTP/3 and may specify the support to specific draft HTTP/3 websites residing on their original servers. Marx et versions (e.g., 27 or 29). al. [9] compare 15 HTTP/3 implementations, finding a We download the HTTPArchive dataset starting from large heterogeneity in how congestion control, prioritiza- November 2019, when we observe the first websites offer- tion and packetization work. They only run single file ing support to HTTP/3. We use the data to study the trend downloads, but their results call for extensive in-the-wild measurements. Cloudflare benchmarks its own draft 27 3https://httparchive.org/,lastvisit:February4th2021 HTTP/3 implementation in [10], finding it to be 1 4% 4https://developers.google.com/web/tools/ − slower than HTTP/2. However, their experiments are lim- chrome-user-experience-report 5 ited to the blog.cloudflare.com website. Guillen et HTTPArchive used to adopt the Alexa top-1M website list, but switched to the Chrome User Experience Report when Alexa discontinued al. [11] proposed a control algorithm for adaptive streaming the rank in July 2018. tailored for HTTP/3. 6http://www.softwareishard.com/blog/har-12-spec/ TABLE II: Impairment levels for controlled experiments. 23 25 27 29 24 26 28 32 Parameter Impairment levels 5 Latency [ms] Native, 50, 100, 200 Loss [%] Native, 1, 2, 5 4

Bandwidth [Mbit/s] Native, 1, 2, 5 3

2 of HTTP/3 adoption. The data sum up to 6.6 TB, and we Websites [%] store them on a medium-sized Hadoop cluster for scalable 1 processing. Since we are interested in studying the adoption of HTTP/3 on websites, we discard all visits to internal 0 pages (less than half of the totality) and keep only visits 2019/102019/112019/122020/012020/022020/032020/042020/052020/062020/072020/082020/092020/102020/112020/12 to home pages. We refer to this dataset as HTTPArchive. B. Controlled Experiments Fig. 2: Websites announcing support to HTTP/3, separately We use the most recent snapshot at the time of writing by IETF draft. (December 2020) to build the list of websites currently supporting HTTP/3. We find 14 707 websites announcing 104 support to HTTP/3. Next, we visit them with all three ver- sions of HTTP (1.1, 2, and 3) to quantify the performance 103 improvements. To this end, we rely on BrowserTime, a dockerized tool to run automatic visits to web pages with a 102 large set of configurable parameters.7 We use BrowserTime Websites 101 to instrument Google Chrome to run visits by using a specific HTTP version. We achieve this using the command 100 switches of Google Chrome. Important for our goal, GSE gws Fly CaddyApache Other cloudflare gtranslate Google Chrome allows specifying a set of domains to be Not specified contacted with HTTP/3 on the first visit, i.e., without prior Google Frontend indication via Alt-Svc header. We cannot find a similar possibility for , and, as such, our experiments are Fig. 3: Server in HTTP response (December 2020). limited to Chrome. We are interested in studying the impact of HTTP/3 under different network conditions and, as such, we run our time at which visible portions of the page are dis- measurements enforcing different network configurations. played. It is computed by capturing the video of the Each network configuration is defined by three parameters: browser screen and tracking the visual progress of the (i) extra-latency, (ii) packet loss, and (iii) bandwidth limit. page during rendering. To enforce impairments during the visits, we rely on the We run our experiments using two high-end servers well-known tc tool. For each parameter, we use connected to the Internet via 1 Gbit/s cable and 4 different impairment levels, reported in Table II. In located in our university campus premises. In total, we case of latency, we impose it on the uplink, while packet run 735 350 visits over a period of one month. The visit loss and bandwidth limit are enforced on both up and metadata account for 189 GB, and we refer to this dataset downlink. For each network configuration, we visit each with BrowserTime. website (i) enabling uniquely HTTP/1.1, (ii) with HTTP/1.1 IV. RESULTS and HTTP/2, and (iii) enabling HTTP/3 too. All visits to the same website are run consecutively and repeated 5 We first provide an overview of the HTTP/3 adoption times to get reliable results. Hence, we visit each website offering a historical perspective and analyzing the current 43 3 5 = 960 times. status. Then, we study how HTTP/3 affects QoE-related ∗ ∗ BrowserTime collects several statistics for each visit, performance metrics by analyzing our experiments under including details on all HTTP transactions as well as controlled network conditions. performance metrics. We track two of these that have been A. Adoption shown to be correlated with users’ QoE [17]: We first study to what extent HTTP/3 has been adopted • onLoad: The time at which the browser fires the since its first proposal. To this end, we profit from the onLoad event – i.e., when all elements of the page, HTTPArchivedataset, offering a historical perspective. The including images, style sheets and scripts, have been first IETF draft is dated January 2017, but we observe the downloaded and parsed; first websites adopting HTTP/3 only in late 2019. Since • SpeedIndex: Proposed by Google8, represents the then, the number of enabled websites started to grow, and 7https://www.sitespeed.io/documentation/browsertime/ Figure 2 shows the trend for the last months of 2019 and 8https://web.dev/speed-index/ the entire 2020. Looking at the Alt-Svc header, we can 1.0 100 Objects 0.8 Volume 75 0.6 50 0.4

CDF (Websites) 25 Objects 0.2 HTTP/3 Share [%] Volume 0.0 0 0 20 40 60 80 100 Cloudflare Facebook Google Other Objects/volume served on HTTP/3 [%] (12455) (445) (1094) (731)

Fig. 4: Share of objects/volume served on HTTP/3 on Fig. 5: Share of objects/volume served on HTTP/3, sepa- enabled websites. rately by provider.

observe the HTTP/3 draft version supported by the server, belong to popular CDNs providers such as Akamai or Ama- shown with different colors in the figure. In case a website zon CloudFront, meaning they have not started deploying offers more than one version, we considered the earliest for HTTP/3 yet. the figure. In the first four months, the number of enabled Next, we study to what extent objects of enabled web- websites increased slowly, reaching 0.7 % of the totality, sites are served using HTTP/3. Indeed, if a website sup- and we notice that, at that time, only Google and Facebook ports HTTP/3, it does not entail that all objects are sent used to offer HTTP/3 on their websites. In February 2020, through HTTP/3. Objects may lay on external CDNs, cloud the number of enabled websites exploded. This is due to providers or third parties not supporting the same protocol. CloudFlare, which enabled HTTP/3 on most of the websites This is the case, for example, of ads and trackers typically it hosts. The share of enabled websites reaches and passes hosted on different third-party infrastructures. We now use 4%, with its maximum in October 2020, with 4.8% and the BrowserTimedataset, which allows us to observe the 203 k websites. In November, the number of websites protocol used for delivering each object composing the web suddenly dropped to less than 0.1 %(4 024 in absolute page we visit. Again, we focus on the 14 707 HTTP/3- terms). We observe that this was caused by CloudFlare enabled websites. In Figure 4, we consider all visits run suspending support to HTTP/3 due to performance issues, with HTTP/3 enabled, and, for each, we compute the share as declared online.9 On December, CloudFlare re-enabled of objects served over HTTP/3. As each website is accessed HTTP/3 on a subset of websites. On that date, 14 707 multiple times, we average the values across visits. Clearly, websites were announcing support to HTTP/3, and, in the at least the main HTML document is always sent with following, we consider these for our analyses. HTTP/3, but the remaining objects of the page may not. The majority of the 14 707 HTTP/3-enabled websites The figure shows the distribution of the percentage of belong to large companies running their own objects served over HTTP/3 (solid red line) and also depicts applications. We breakdown this in Figure 3, which in- the byte-wise percentage – i.e., weighting each object by dicates the 10 most popular servers as indicated on the its size. We first notice that in 18% of cases, all objects HTTP Server header. We extrapolate this figure using are delivered over HTTP/3, meaning that the web page again the HTTPArchive dataset, containing details on all only contains elements hosted on HTTP/3 enabled servers. HTTP headers of responses. CloudFlare, as expected, hosts The websites having 90% or more of objects (volume) on most of the HTTP/3-enabled websites (notice the log y- HTTP/3 are 36 (41) % and only 9 (28) % have less than scale). Google is in the second position, with GSE (Google 20 %. Interestingly, we notice that 51% of websites still Servlet Engine), Google Frontend and GWS (Google Web have one or more objects retrieved using HTTP/1.1. Server). Indeed, GSE is used on the Blogspot platform, represented by 809 websites in our list. For 575 websites, Next, we dissect the above analysis by provider – i.e., there is no server indication on HTTP responses, and the company/CDN hosting the website. We obtain it by we find that 445 of these belong to the Facebook company looking at the server HTTP header, website name and – e.g., facebook.com and instagram.com domains server IP address, which allow easy identification. As – revealing the social network is a prominent example discussed for Figure 3, we notice that HTTP/3 has been of an HTTP/3 early adopter. The remaining websites adopted mostly by (i) Cloudflare CDN, (ii) the Facebook run popular open-source servers (nginx, Apache) or more and (iii) Google companies. The remaining 595 websites peculiar HTTP ones (e.g., ), which offer HTTP/3 (i.e., Other) belong mostly to self-hosted websites running support in their earlier versions. Interestingly, we do not updated versions of the nginx web server. Figure 5 shows find any websites whose server header or IP address the share of objects and volume served over HTTP/3, separately by provider. Cloudflare websites tend to be more 9https://community.cloudflare.com/t/community-tip-http-3-with-quic/ heterogeneous, with half of the objects retrieved via non- 117551, visited on 2/20/2021. HTTP/3 servers in median. Moreover, only 24% of the websites’ volume is served by using HTTP/3. This is likely 12 due to the provider’s nature; indeed, Cloudflare offers its HTTP/1.1 10 hosting service to a very large plethora of websites. These HTTP/2 websites may use complex web pages composed of several 8 HTTP/3 third-party objects stored on external servers not relying on HTTP/3 yet. Conversely, Facebook and Google show 6 a very different situation. Almost all the websites’ objects 4 and volumes are served with HTTP/3. Indeed Facebook Page Load Time [s] and Google use their CDNs mostly to offer their own 2 services. Looking at Google, the long tail of the distribution 0 is due to Blogspot websites in which the creator may 6 add content from external sources. Finally, considering the Other category, almost all the objects and volumes are 5 served by using HTTP/3. These websites tend to be simple, 4 composed of a few objects stored in the same self-hosted servers together with the main HTML document. 3 2

B. Performance Speed Index [s] 1 We now study the impact of HTTP/3 on web page per- formance. To this end, we use the BrowserTime dataset, 0 Native 50 100 200 14 707 in which the have been visited multiple times under Extra Latency [ms] different emulated network conditions. Using tc-netem, we enforce extra latency, packet loss and limit bandwidth. We then contrast page QoE-related performance indicators Fig. 6: onLoad (top) and SpeedIndex (bottom) with extra (onLoad and SpeedIndex), (i) showing their absolute value latency, separately for HTTP/1.1, 2 and 3. and (ii) computing the speed-up. Given a website and a given network scenario, we obtain the speed-up as the relative deviation of the metric when using HTTP/3 (h3) onLoad in median in 6.4, 5.8 and 5.4 s with HTTP version instead of HTTP/2 (h2). As we always run 5 visits for each 1.1, 2 and 3, respectively. case, we consider the median value. As such, the speed-up To better catch differences between HTTP/3 and 2, we for a website w in scenario s is computed as follows: now study the speed-up in Figure 7, where we show the distribution over the 14 707 websites for both onLoad (top row) and SpeedIndex (bottom row). The three columns median(w, s, h3) median(w, s, h2) speed-up(w, s) = − refer to scenarios with additional latency, limited band- ( ( 3) ( 2)) max median w, s, h , median w, s, h width and packet loss, respectively. The solid red lines By definition, speed-up(w, s) is bound in [ 1, 1] and represent the native case – i.e., without imposing any − is negative when a website loads faster under HTTP/3, network impairment. Dashed lines represent scenarios with positive otherwise. We compute it for both onLoad and network impairments, as indicated in Table II. SpeedIndex. Starting from latency, we confirm what already emerged We illustrate how the metric values vary when imposing from Figure 6. When latency is high, HTTP/3 gives sizable different network impairments, focusing firstly on extra la- benefits compared to HTTP/2. In the native case, we tency in Figure 6. Using boxplots, we show the distribution observe no benefit. Looking at the solid red lines, we notice of onLoad (top) and SpeedIndex (bottom), separately by that approximately in 50 % of cases websites load faster HTTP version (colored boxes). The boxes span from the with HTTP/3 and in the remaining cases slower. If we first to the third quartile, while whiskers report the 10th and impose extra latency of 50 ms, 69 (74) % of websites have the 90th percentiles; black strokes represent the median. lower onLoad time (SpeedIndex), meaning that they load When no extra latency is imposed (native case), we observe faster. They increase to 76 (81) % with 100 ms latency. that onLoad time is in median around 2s, while SpeedIndex With 200 ms, they reach 81 (87) %, and the median speed- around 1s, without significant differences across HTTP up is 0.08 (0.12). In a few words, HTTP/3 shows sizable versions. When adding extra latency, the websites load benefits in case of high latency. slower as more time is needed to download the page Focusing on experiments with bandwidth limitation (cen- objects, requiring in median 6 seconds with 200 ms of tral plots in Figure 7), different considerations hold. We additional latency. Not shown here for brevity, also packet observe sizable benefits only for onLoad time with the loss and limited bandwidth cause similar degradation of bandwith limited to just 1 Mbit/s, where 69 % of websites performance indicators. In the following, we compare to load faster with HTTP/3. Notice that this benefit cannot what extent HTTP/3 provides benefits in such scenarios. be introduced by indirect higher latency due to queuing Figure 6 shows that HTTP/1.1 has the worst performance delay (also called bufferbloat), as we limit the machines’ with high latency, while HTTP/3 shows the greatest ben- queue to as little as 32 KB. In other cases, no clear efits. Considering additional latency of 200 ms, websites trend emerges, but we only notice a larger variability 1.0 Native Native 0.8 50 ms 1% 0.6 100 ms 2% 200 ms Native 5% CDF 0.4 5 Mbit/s 0.2 2 Mbit/s 1 Mbit/s 0.0 -0.5 -0.3 -0.1 0 0.1 0.3 0.5 -0.5 -0.3 -0.1 0 0.1 0.3 0.5 -0.5 -0.3 -0.1 0 0.1 0.3 0.5 Speed-up (onLoad) Speed-up (onLoad) Speed-up (onLoad) 1.0

0.8

0.6

CDF 0.4

0.2

0.0 -0.5 -0.3 -0.1 0 0.1 0.3 0.5 -0.5 -0.3 -0.1 0 0.1 0.3 0.5 -0.5 -0.3 -0.1 0 0.1 0.3 0.5 Speed-up (SpeedIndex) Speed-up (SpeedIndex) Speed-up (SpeedIndex) (b) Bandwidth. (c) Packet loss. (a) Latency. Fig. 7: Speed-up of HTTP/3 on different scenarios. onLoad (top) and SpeedIndex (bottom). of the speed-up measure introduced by the constrained loaded faster with HTTP/3 than with HTTP/2. Cloudflare setup. For example, in case of SpeedIndex, 56, 49, 58 (red solid line) shows the smallest benefits, with only % websites load faster with HTTP/3 with 5, 2 and 1 72% of websites loading faster. Google and the remaining Mbit/s bandwidth, respectively. Similar considerations hold websites sit in the middle. Similar considerations hold for for packet loss (right-most plots in Figure 7). Despite a SpeedIndex, not shown here for brevity. larger variability, we cannot identify any obvious trend, With limited bandwidth (Figure 8b), we observe a very and the speed-up values are equally distributed above and different situation. Here, Facebook has in general worst below 0. In summary, we do not testify any performance performance with HTTP/3 with 91% of websites loading benefit of HTTP/3 in scenarios with high packet loss. We faster with HTTP/2. Conversely, Google (green dashed only observe improvements with very poor bandwidth and line) shows the best improvements, with median speed-up limited to onLoad. 0.14 and 79% of websites loading faster with HTTP/3. − Cloudflare and the remaining websites exhibit no clear C. Dissecting performance speed-up trend with roughly half of the websites loading faster with We now focus on the performance speed-up as defined HTTP/3. Considering the scenario with artificial packet previously and discuss how it is impacted by (i) the loss, we do not observe differences across providers, as provider hosting the website and (ii) characteristics of they all approximately behave as shown in Figure 7c. the web pages and load process. As we observed sizable performance benefit for HTTP/3 only in cases of high 2) The impact of page characteristics: latency or (very) poor bandwidth, we restrict our analyses Now we focus on how web page characteristics impact to those cases. performance. To this end, we compute various metrics describing the web page load process and contrast them 1) Performance by Provider: to understand they relationship to the HTTP/3 speed-up. Figure 8 shows the distribution of performance speed-up For each visit to the 14 707 websites in our dataset, in ad- for onLoad, separately by provider. We focus on scenarios dition to the HTTP/3 speed-up, we compute the following with 200 ms extra latency and 1 Mbit/s bandwidth limit. metrics: Similarly to Figure 5, we study Cloudflare, Facebook • Number of connections issued by the browser to load and Google, being the companies behind the majority of the website when using HTTP/3. the HTTP/3 enabled websites currently. We observe the • Number of domains contacted while loading the web speed-up largely differs by provider. Focusing on latency page, thus including third-pary domains). (Figure 8a), Facebook websites show the highest perfor- • Share of objects on the largest connection: to mance gain ( 0.13 in median), represented in the figure measure to what extent the objects composing the − with the blue dashed line. Moreover, 95% of websites are web page are spit into many domains, we compute 1.0 6 H3 Faster 5 0.8 H3 H2 ≈ 4 H2 Faster 0.6 Cloudflare 3 CDF . 0 4 Facebook 2 Google 0.2 Normalized Value 1 Other 0.0 0 0.5 0.4 0.3 0.2 0.1 0.0 0.1 0.2 0.3 0.4 0.5 − − − − − Page size Speed-up (onLoad) Number of conn. HTTP/3 share NumberObjects of Domains on largest conn. (a) Latency. (a) Latency. 1.0 Cloudflare 6 0.8 H3 Faster Facebook 5 H3 H2 Google ≈ 0.6 4 H2 Faster Other CDF 0.4 3 2 0.2

Normalized Value 1 0.0 0.5 0.4 0.3 0.2 0.1 0.0 0.1 0.2 0.3 0.4 0.5 0 − − − − − Speed-up (onLoad) Page size Number of conn. HTTP/3 share NumberObjects of Domains on largest conn. (b) Bandwidth. Fig. 8: onLoad speed-up by website provider for scenarios (b) Bandwidth. with extra-latency and bandwidth limit. Fig. 9: Visit characteristics vs. HTTP/3 speed-up class (normalized values). the share of objects carried over the connection where most objects have been requested, divided by the total number of objects. Remind that the best practices We first focus on the left-most box group of Fig- of HTTP/3 recommend avoiding domain sharding to ure 9, showing the (normalized) number of connections increase performance. the browser issued to load the web page. Green boxes • Share of objects served on HTTP/3: our goal is hint that websites issuing fewer connections (smaller metric to understand how the fraction of the website objects values) are faster with HTTP/3 than in HTTP/2. This true served on HTTP/3 (shown in Figure 4) has an impact in both scenario, low latency and poor bandwidth. Similar on the speed-up. considerations hold if we focus on the second box group, • Page Size: to breakdown performance for small and representing the number of contacted domains. Indeed, we large web pages. notice that the number of connections per website and the In Figure 9, we compare the distribution of the aforemen- number of contacted domains are 0.91-correlated. The third tioned metrics, grouping websites based on classes defined box group offers a similar perspective, measuring how web by the onLoad speed-up: page objects are split on multiple connections/domains. • H3 Faster: websites loading faster with HTTP/3, i.e., The websites benefiting the most from HTTP/3 are those onLoad speed-up < 0.1. which tend to mass objects on a single connection – see − • H3 H2: websites having a similar loading the highest position of the green boxes, meaning more ≈ time in HTTP/2 and HTTP/3, i.e., onLoad speed- objects are on a single connection. This is very clear up [ 0.1, 0.1]. with limited bandwidth (Figure 9b), rather than with high ∈ − • H2 Faster: websites loading faster with HTTP/2, i.e., latency (Figure 9a). onLoad speed-up > 0.1. Serving most objects with HTTP/3 (rather than with In the figure, boxes with different colors represent these HTTP/2) has a positive impact too, as we notice from the three classes. The y-axis represents the metrics normalized fourth box group in Figure 9. Again, this is evident espe- by scaling to unit variance to ease the visualization and cially with bandwidth limit (Figure 9b), while with extra the comparison. Again, we study the scenarios with extra latency (Figure 9a) it is hard to find a clear trend. Finally, latency (top row) and limited bandwidth (bottom row) since interesting is the case of the web page size (last box group). they provide the most interesting insights. In case of extra In scenarios with high latency, the websites benefiting from latency, H3 Faster websites are 32%, H2 Faster 11% and HTTP/3 are small, while large ones typically perform better H3 H2 are 57%. With limited bandwidth, they are 38%, with HTTP/2. When bandwidth is scarce, we observe an ≈ 25% and 37%, respectively. opposite picture: even if moderately, website loading faster with HTTP/3 are the large ones. packet loss HTTP/3 and 2 perform roughly the same. In summary, websites taking benefits from HTTP/3 Moreover, we found large performance diversity depending are those limiting the number of connections and third- on the infrastructure hosting the website. In general, we party domains, and fully adopting HTTP/3 on all web observed that websites taking benefits from HTTP/3 are page objects. Page size has diverse implications depending those loading objects from a limited set of third-party on the network conditions. These considerations hold in domains, thus limiting the number of issued connections. scenarios with high latency or limited bandwidth, while we ACKNOWLEDGEMENTS do not observe any clear trend in case of optimal network conditions or high packet loss, where metric distributions This work has been supported by the EU H2020 research mostly overlap. and innovation programme under grant agreement No. 644399 (MONROE) and by the SmartData@PoliTO center V. MEASUREMENT LIMITATIONS AND FUTURE WORK on Big Data and Data Science. We dissect the performance of HTTP/3 under diverse REFERENCES network conditions, showing the impact of network latency and bandwidth and differences across website providers. [1] R. T. Fielding, H. Nielsen, J. Mogul, J. Gettys, and T. Berners-Lee, “Hypertext Transfer Protocol – HTTP/1.1.” RFC 2068, Jan. 1997. However, we run measurements uniquely using Google [2] M. Belshe, R. Peon, and M. Thomson, “Hypertext Transfer Protocol Chrome, as, to the best of our knowledge, it is the only Version 2 (HTTP/2).” RFC 7540, May 2015. browser that can be instrumented to use HTTP/3 on the [3] M. Bishop, “Hypertext Transfer Protocol Version 3 (HTTP/3),” Internet-Draft draft-ietf-quic-http-33, Internet Engineering Task first connection – i.e., without the need to observe the Force, Feb. 2021. Work in Progress. Alt-Svc header previously. Moreover, we uniquely visit [4] J. Iyengar and M. Thomson, “QUIC: A UDP-Based Multiplexed websites with a fresh browser profile with an empty cache and Secure Transport,” Internet-Draft draft-ietf-quic-transport-33, Internet Engineering Task Force, Dec. 2020. Work in Progress. and no pre-existing connections. This limits the scope [5] E. Rescorla, “The (TLS) Protocol Version of our study as we cannot measure how HTTP/3 affects 1.3.” RFC 8446, Aug. 2018. performance on subsequent visits or with a warm HTTP [6] M. Varvello, K. Schomp, D. Naylor, J. Blackburn, A. Finamore, and K. Papagiannaki, “Is the web http/2 yet?,” in International cache. Conference on Passive and Active Network Measurement, pp. 218– We measure a large set of websites, but we are limited 232, Springer, 2016. only to the subset of those already adopting HTTP/3. [7] M. Rajiullah, A. Lutu, A. S. Khatouni, M.-R. Fida, M. Mellia, A. Brunstrom, O. Alay, S. Alfredsson, and V. Mancuso, “Web Moreover, we include only a fraction of those hosted on experience in mobile networks: Lessons from two million page the CloudFlare CDN, as its HTTP/3 support is partially visits,” in The World Wide Web Conference, pp. 1532–1543, 2019. disabled at the time of writing this article. Our measure- [8] D. Saif, C.-H. Lung, and A. Matrawy, “An early benchmark of quality of experience between http/2 and http/3 using lighthouse,” ments need to be run continuously to thoroughly study how arXiv preprint arXiv:2004.01978, 2020. the web ecosystem will react to HTTP/3 in the near future, [9] R. Marx, J. Herbots, W. Lamotte, and P. Quax, “Same standards, adopting the protocol and modifying the web page structure different decisions: A study of quic and http/3 implementation diver- sity,” in Proceedings of the Workshop on the Evolution, Performance, to follow the new guidelines, in particular concerning and Interoperability of QUIC, pp. 14–20, 2020. domain sharding. [10] S. Tellakula, “Comparing HTTP/3 vs. HTTP/2 Performance.” : //blog.cloudflare.com/http-3-vs-http-2/, 2020. Finally, HTTP/3 and QUIC are not yet definitive IETF [11] L. Guillen, S. Izumi, T. Abe, and T. Suganuma, “Sand/3: Sdn- standards. Although recent modifications to the IETF draft assisted novel qoe control method for dynamic adaptive streaming only concerned minor protocol features, it will be funda- over http/3,” Electronics, vol. 8, no. 8, p. 864, 2019. [12] K. Wolsing, J. Ruth,¨ K. Wehrle, and O. Hohlfeld, “A performance mental to provide a similar analysis once the final standard perspective on web optimized protocol stacks: Tcp+ tls+ http/2 versions are approved. Similarly, we did not explore how vs. quic,” in Proceedings of the Applied Networking Research different endpoint configurations affect how objects are Workshop, pp. 1–7, 2019. [13] J. Manzoor, L. Cerda-Alabern,` R. Sadre, and I. Drago, “On the sent, eventually retransmitted and congestion control algo- performance of quic over wireless mesh networks,” Journal of rithms operate. This calls for fine-grained measurements, Network and Systems Management, vol. 28, no. 4, pp. 1872–1901, better suited to be run on fully-controlled lab experiments. 2020. [14] G. Carlucci, L. De Cicco, and S. Mascolo, “Http over udp: An ex- Moreover, we only explore scenarios with uniform and perimental investigation of quic,” in Proceedings of the 30th Annual random packet loss, while it is of certain interest to study ACM Symposium on Applied Computing, SAC ’15, (New York, NY, other loss schemes (e.g., bursty loss). USA), p. 609–614, Association for Computing Machinery, 2015. [15] A. M. Kakhki, S. Jero, D. Choffnes, C. Nita-Rotaru, and A. Mislove, VI.CONCLUSIONS “Taking a long look at quic: an approach for rigorous evaluation of rapidly evolving transport protocols,” in Proceedings of the 2017 In this paper, we provided a first study on HTTP/3, mea- Internet Measurement Conference, pp. 290–303, 2017. suring its adoption and dissecting performance benefits. We [16] M. Kosek, T. Shreedhar, and V. Bajpai, “Beyond quic v1–a first look at recent transport layer ietf standardization efforts,” arXiv preprint testified how some of the Internet leading companies started arXiv:2102.07527, 2021. deploying it during 2020, although most of the adopter [17] D. N. da Hora, A. S. Asrese, V. Christophides, R. Teixeira, and website still host the majority of objects on HTTP/2 third- D. Rossi, “Narrowing the gap between qos metrics and web qoe us- ing above-the-fold metrics,” in International Conference on Passive party servers. With a large-scale measurement campaign, and Active Network Measurement, pp. 31–43, Springer, 2018. we studied the performance of HTTP/3 under different network conditions, targeting thousands of websites. We found performance benefits emerging in scenarios with high latency or poor bandwidth, while in case of high