Building Reproducible Video Streaming Traffic Generators

Building Reproducible Video Streaming Traffic Generators Calvin Ardi Alefiya Hussain Stephen Schwab USC/ISI USC/ISI USC/ISI Marina del Rey, CA, USA Marina del Rey, CA, USA Marina del Rey, CA, USA [email protected] [email protected] [email protected] ABSTRACT In this paper, we present our preliminary work towards building Video streaming traffic dominates Internet traffic. However, there video streaming traffic generators that are both representative and is a dearth of tools to generate such traffic on emulation-based reproducible. We implement in our generators both the client and testbeds. In this paper we present tools to create representative and server endpoints, and each individual endpoint can be used on reproducible video streaming traffic to evaluate the next generation its own to test additional components within the network. Our of traffic classification, Quality of Service (QoS) algorithms and generators produce representative traffic on-the-wire by using freely traffic engineering systems. We discuss 27 different combinations of available or permissively-licensed videos and streaming them across streaming video traffic types in this preliminary work, and illustrate a variety of transport protocols. We enable reproducible experiments the diversity of network-level dynamics in these protocols. by emulating the process of “watching” videos with a systematic and well-defined methodology. We discuss in § 2 current state-of- ACM Reference Format: art video streaming protocols and how they can be used to recreate Calvin Ardi, Alefiya Hussain, and Stephen Schwab. 2021. Building Repro- representative and reproducible scenarios on an emulation-based ducible Video Streaming Traffic Generators. In Cyber Security Experimenta- tion and Test Workshop (CSET ’21), August 9, 2021, Virtual, CA, USA. ACM, testbed. Our tools support 27 video streaming protocol and software New York, NY, USA, 5 pages. https://doi.org/10.1145/3474718.3474721 combinations, and can be fully automated. Further, they can be configured to run completely end-to-end within a self-contained 1 INTRODUCTION environment on the testbed or reach-out to remote servers and services over the Internet to provide even more realism. Video streaming traffic has and continues to be the majority of Inter- We discuss in § 3 the varied network-level traffic patterns gen- net traffic, accounting for roughly 58 % of global Internet use (2020, erated from a sample of the 27 protocols on an emulation-based [14]) and is predicted to be 82 % of all traffic by 2022 [6]. Delivering testbed. We analyze the underlying video streaming performance video across the Internet is supported by a diverse set of infras- and behavior on both a local network and the Internet, highlight- tructure, client/server software, encoding algorithms, and network ing the differences in bandwidth throughput profiles between dif- protocols. Although many parts of the end-to-end video delivery ferent streaming and transport protocols. Our goal is to enable on the Internet has been studied in great detail [1, 3, 9, 11], there representative and reproducible experiments for video streaming are no readily available tools for video traffic to support testbed- traffic and this initial set of generators provides a wide spectrum based experiments. Prior work in traffic generation and simulation of protocols. We make our generators and tools available at https: typically focus on maximizing throughput for benchmarks and //mergetb.org/projects/searchlight/ in order to enable research in stress testing [12] or on simulating the behavior of the underlying this domain. protocols (TCP, UDP, and others) [2, 15, 16]. While experimenters can iterate on and test their network system components over the Internet, often coupling them with popular 2 REPRODUCIBLE VIDEO STREAMING content providers, there are a lack of tools to support principled TRAFFIC GENERATORS experimentation with video traffic on testbeds and the Internet. In this section, we discuss the design decisions for representa- Experimenters need these video traffic tools in order to evaluate tive protocols and software to enable reproducible experiments the next generation of systems for network traffic classification and on emulation-based testbeds. engineering and QoS algorithms. These tools should allow experimenters to test and evaluate these systems and their individual 2.1 Using Representative Protocols components during its development, and include both client and There are many formats and protocols for video streaming. We con- server endpoints used for watching and delivering video. These sider 27 combinations, that include different streaming protocols, tools should also send traffic responsibly on the Internet, in order the encapsulation format for video streaming, and “transport” pro- to minimize impact on third-party services. tocols, the underlying application-level protocols that move data Permission to make digital or hard copies of all or part of this work for personal or between client and server. We focus only on desktop browsers in classroom use is granted without fee provided that copies are not made or distributed this paper, and plan on addressing mobile and other application- for profit or commercial advantage and that copies bear this notice and the full citation specific clients as future work. on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. 0This work was sponsored by Sandia National Laboratory (SNL) under PO2160586. CSET ’21, August 9, 2021, Virtual, CA, USA This research was developed with funding from the Defense Advanced Research © 2021 Copyright held by the owner/author(s). Publication rights licensed to ACM. Projects Agency (DARPA). The views, opinions and/or findings expressed are those of ACM ISBN 978-1-4503-9065-1/21/08...$15.00 the authors and should not be interpreted as representing the official views or policies https://doi.org/10.1145/3474718.3474721 of Sandia National Laboratory, the Department of Defense, or the U.S. Government. CSETCSETCSET ’21, ’21, August ’21, August August 9, 2021,9, 9, 2021, 2021, Virtual, Virtual, Virtual, CA, CA, CA, USA USA USA CalvinCalvin Ardi, Ardi, Alefiya Alefiya Hussain, and Stephen Schwab Schwab CSET ’21, August 9, 2021, Virtual, CA, USA Calvin Ardi, Alefiya Hussain, and Stephen Schwab

n n n n n n n n TableTableTable 1: 1: 1: Supported Supported Supported Protocol Protocol Protocol and and Software Software Combinations DASHDASHDASH HLSHLSHLS Table 1: Supported Protocol and Software Combinations DASH HLS n n n n n n HTML5

n n ideo Streaming HTML5HTML5 ideo Streaming ideo Streaming

V HTML5 V ideo Streaming V n n V softwaresoftwaresoftware type type HTTP/1.1 HTTP/1.1 HTTP/2 HTTP/2 HTTP/3 HTTP/3 n n n n n n software type HTTP/1.1 HTTP/2 HTTP/3 HTTP/1.1HTTP/1.1HTTP/1.1HTTP/2HTTP/2HTTP/2 HTTP/3HTTP/3HTTP/3 HTTP/1.1 HTTP/2 HTTP/3 clearclearclear + + + TLS TLS TLS clear clear clear + + TLS TLS TLS TLS n n n n n n n n TLSTLSTLS TLS QUICQUICQUICQUIC ChromeChromeChromeChrome client client client ransport" ransport" ransport" ransport" "T "T "T n n n n n n n n "T TCPTCPTCP FirefoxFirefoxFirefoxFirefox client client client TCP UDPUDPUDP UDP CaddyCaddyCaddyCaddy server server server n n n n n n n n IP IPIP IP a nginxnginxnginxnginxaaa serverserverserver a a Apache2Apache2Apache2Apache2aa serverserverserver FigureFigureFigure 1:Figure 1: Network 1: Network Network 1: Network Topology Topology Topology Topology FigureFigureFigureFigure 2: 2: 2:Protocols Protocols Protocols 2: Protocols a Implementation:Implementation:Implementation:Implementation: == full, full,= full, == partial,= partial, partial, === none= none none nonea foraa comparisonforforfor comparison comparison purposes purposes purposes StreamingStreamingStreaming protocols. protocols. protocols.WeWeWe focus focusWe focus on focus on on Dynamic Dynamic Dynamicon Dynamic Adaptive Adaptive Adaptive Adaptive Streaming Streaming Streaming Streaming Streaming protocols. 2.22.2 Enabling Enabling Reproducible Reproducible Experiments Experiments overoverover HTTP HTTPover HTTP (DASH HTTP (DASH (DASH), (DASH HTTP),), HTTP HTTP), Live HTTP Live Live Streaming Streaming Live Streaming Streaming (HLS (HLS (HLS), (HLS), and), and and), basic and basic basic basicHTML5HTML5HTML5HTML5 2.22.2 Enabling Enabling Reproducible Reproducible Experiments Experiments video as our representative streaming protocols. While there are We are building our video traffic generators to enable reproducible videovideovideo as as our as our our representative representative representative streaming streaming streaming protocols. protocols. protocols. While While While there there there are are are WeWeWe are are are building building building our our our video video video traffic traffic generatorsgenerators to enable reproducible reproducible many different streaming protocols available, prior work [1] has experiments for building and testing traffic engineering systems, manymanymany different different different streaming streaming streaming protocols protocols protocols available, available,available, prior priorprior work [1work work[1] [1] has] has has experimentsexperimentsexperiments for for for building building building and and and testing testing traffic traffic engineering systems,systems, found that DASH and HLS accounted for 83 % of video playback and client or server video applications. Our generators can also be foundfound that thatDASHDASHandandHLSHLSaccountedaccounted for for 83 83 % % of of video video playback playback andandand client client client or or or server server server video video video applications. applications. applications. Our Our generators can also also be be found thattimeDASH (2018),and thusHLS makingaccounted the inclusion for 83 % of of both video protocols playback in our used to generate traffic in a self-contained manner, within a testbed, timetime (2018), (2018), thus thus making making the the inclusion inclusion of of both both protocols protocols in in our our usedused to to generate generate traffic traffic in in a self-contained manner, within a a testbed, testbed, time (2018),initial thus work making a straightforward the inclusion choice. of both protocols in our usedor to to generate remote, Internet traffic services. in a self-contained manner, within a testbed, initialinitial work work a a straightforward straightforward choice. choice. oror to to remote, remote, Internet Internet services. services. initial workWe a straightforward additionally add HTML5 choice. video to act as a baseline for eval- or to remote,Our traffic Internet generators services. provide the end-to-end application traffic We additionally add HTML5 video to act as a baseline for eval- OurOur traffic traffic generatorsgenerators provide the end-to-end application traffic traffic WeWe additionallyuation additionally and addcomparison. addHTML5HTML5 Althoughvideovideo to to actHTML5 act as as a a baselinevideo baseline lacks for foreval- many eval- of Ournecessary traffic for trafficgenerators engineering provide the systems. end-to-end Using application simple and well- traffic uation and comparison. Although HTML5 video lacks many of necessary for traffic engineering systems. Using simple and well- uationuation andthe and complexities comparison. comparison. and Although Although advancesHTML5 inHTML5 streamingvideovideo technology lacks lacks many many (discussed of of necessarynecessarydefined for for generators traffic traffic engineeringengineeringenable us to systems. systems.create reproducible Using simple experiments and well- the complexities and advances in streaming technology (discussed defined generators enable us to create reproducible experiments thethe complexities complexitiesnext), it is and broadly and advances advances supported in in streaming streaming in popular technology technology web browsers (discussed (discussed and easily defineddefinedthat system generatorsgenerators builders enable canenable use tousus validate to to createcreate and reproducible rapidly iterate experiments over next), it is broadly supported in popular web browsers and easily that system builders can use to validate and rapidly iterate over next),next), itincluded itis is broadly broadly by supported website supported authors. in in popular popular We plan web web to browsers includebrowsers other and and easilystreaming easily thatthattheir system system designs. builders builders can can use use to to validate validate and and rapidly iterate over included by website authors. We plan to include other streaming their designs. includedincludedprotocols by by website website not authors. mentioned authors. We We here plan plan as tofuture to include include work. other other streaming streaming theirtheir designs.We designs. build simplified, static websites for our video streaming traf- protocols not mentioned here as future work. We build simplified, static websites for our video streaming traf- protocolsprotocols notFor not mentioned better mentioned reproducibility, here here as as future future we work.fix work. the video resolution for the WeficWe build buildgenerators simplified, simplified, to maximize staticstatic the websiteswebsites signal-to-noise for our videoratio streaming on the wire. We traf- For better reproducibility, we fix the video resolution for the fic generators to maximize the signal-to-noise ratio on the wire. We Forentire better playback reproducibility, duration we to fix480p, the 720p, video or 1080p. resolution Although for the the ficficfocus generatorsgenerators on the video to to streamingmaximizemaximizethe contentthe signal-to-noisesignal-to-noise and exclude mostratio of the on fram-the wire. We Forentire better playback reproducibility, duration to we 480p, fix 720p,the orvideo 1080p. resolution Although thefor the focus on the video streaming content and exclude most of the fram- entireentire playbackprimary playback benefit duration duration of to tousing 480p, 480p, 720p,Adaptive 720p, or or 1080p.Bitrate 1080p.ABR ( Although) Although streaming the the proto- focusfocusing on on support the the video video that streaming generallystreaming surrounds content content and and the exclude exclude main content most of on the today’s fram- primary benefit of using Adaptive BitrateABR ( ) streaming proto- ingwebpages. support Whilethat generally content like surrounds ads, comments, the main or content other dynamically on today’s primaryprimarycols benefit benefit (DASH ,of HLSof using using) is their Adaptive Adaptive ability toBitrate dynamicallyBitrateABR ( ABR ( )) streaming streaming switch resolutions proto- proto- inging support support that that generally generally surrounds surrounds the the main main content content on today’s cols (basedDASH on, HLS network) is their conditions, ability to our dynamically initial goal switch is to generate resolutions consis- webpages.loaded sections While (“Comingcontent like Up ads, Next”, comments, etc.) are prevalentor other dynamically today, they colscols (DASH (DASH, HLS, HLS) is) is their their ability ability to to dynamically dynamically switch switch resolutions resolutions webpages.webpages. While While content content like like ads, ads, comments, comments, or other dynamically basedbased ontent, on networknetwork and thus conditions, conditions,reproducible, our our traffic initial initial goal goal on isthe is toto generatenetwork. generate Fixed consis- consis- playback loadedloadedadd an sections sections additional (“Coming (“Coming layer of Up Up complexity Next”, Next”, etc.) that are is prevalent not needed today, today, at early they they based on network conditions, our initial goal is to generate consis- loadedadd an sections additional (“Coming layer of Up complexity Next”, etc.) that are is prevalent not needed today, at early they tent,tent, andresolution, and thus thus reproducible, reproducible, or Constant traffic Bitrate traffic (CBR on on )the streaming,the network.network. is Fixedthe Fixed default playbackplayback be- addstages an additional in the design layer process. of complexity Our design that has enabled is not needed others to at focus early tent, and thus reproducible, traffic on the network. Fixed playback addstages an additional in the design layer process. of complexity Our design that has enabled is not needed others to at focus early resolution,resolution,havior or ofor ConstantHTML5 Constantvideo. Bitrate Bitrate We (CBR ( planCBR) to) streaming, streaming, provide a iswider is the the rangedefault default of be- video be- stagestheir in efforts the design on prototyping process. Our designand deploying has enabled an early others traffic to focus engineer- resolution, or Constant Bitrate (CBR) streaming, is the default be- stagestheir inefforts the design on prototyping process. Our and design deploying has enabled an early others traffic to focus engineer- haviorhaviorresolutions of ofHTML5HTML5 asvideo.video. well We as We resolution plan plan to to provide provide switching a a wider wider that range range is both of of dynamicvideo video theiring efforts system without on prototyping other irregularities—later, and deploying we an plan early to introduce traffic engineer- haviorresolutions of HTML5 asvideo. well as We resolution plan to provide switching a wider that isrange both of dynamic video theiringadditional systemefforts without optionson prototyping other in our irregularities—later, servers and todeploying increase we the an plan complexity early to introduce traffic and engineer- resolutionsand reproducible. as well as resolution switching that is both dynamic ing system without other irregularities—later, we plan to introduce resolutionsand reproducible. as well as resolution switching that is both dynamic ingadditionalrealism system that without options is more other in reflective our irregularities—later, servers of to today’s increase websites. we the plan complexity to introduce and and reproducible.Transport protocols. Our selection of streaming protocols (DASH, additional options in our servers to increase the complexity and and reproducible. additionalrealismFinally, that options our is more traffic in reflective our serversgenerators of to today’s increase are designed websites. the complexity to be self-contained and TransportHLS, HTML5 protocols.) operatesOur selection over HTTP of streaming(shown inprotocols Fig. 2). (DASH We have, realism that is more reflective of today’s websites. Transport protocols. Our selection of streaming protocols (DASH, realismwithinFinally, that a testbedour is more traffic environment reflective generators orof used today’s are with designed livewebsites. Internet to be self-contained services. TransportHLS,consideredHTML5 protocols.) operates otherOur protocols selection over HTTP like of streamingRTMP(shown, but in protocols leave Fig. 2). its( supportWeDASH have, for Finally, our traffic generators are designed to be self-contained HLS, HTML5) operates over HTTP (shown in Fig. 2). We have withinFinally,We currently a testbed our traffic limit environment using generators our generators or used are with ondesigned Internet live Internet services, to be self-contained services. taking HLSconsidered, HTML5future) work: otheroperates prior protocols over workHTTP like foundRTMP(shown that,RTMP but in leave Fig.andothers its 2). support We [1 have] are for not within a testbed environment or used with live Internet services. considered other protocols like RTMP, but leave its support for withinWecare currently a to testbed remain limit environmentwithin using the our typical generators or used Internet with on noise Internet live threshold. Internet services, We services. taking plan consideredfuturewidely work: other deployed prior protocols work today. found like RTMP that RTMP, but leaveand others its support [1] are for not We currently limit using our generators on Internet services, taking future work: prior work found that RTMP and others [1] are not Wecareto currently carefully to remain limit explore within using how the our typical we generators can Internet emulate on noiseInternet large threshold. amounts services, of We traffic taking plan futurewidely work: deployedWe prior support work today. for foundmultiple that versionsRTMP ofandHTTP others, including [1] are HTTP/1.1, not care to remain within the typical Internet noise threshold. We plan widelyHTTP/2, deployed and today. HTTP/3 (currently an IETF draft standard) both in caretoon carefully to the remain Internet explore within without how the typicalundue we can impact Internet emulate to noise other large usersthreshold. amounts andservices. of We traffic plan widelyWe deployed support today. for multiple versions of HTTP, including HTTP/1.1, to carefully explore how we can emulate large amounts of traffic Wecleartext support forwhen multiple possible versions and with of encryption.HTTP, including Table 1 shows HTTP/1.1, the var- toon carefully the Internet explore without how undue we can impact emulate to other large users amounts and services.of traffic WeHTTP/2, support and for HTTP/3 multiple (currently versions of anHTTP IETF, draft including standard) HTTP/1.1, both in on the Internet without undue impact to other users and services. HTTP/2,ious and software HTTP/3 and (currently supported HTTP an IETFcombinations draft standard) and Fig. both 2 shows in 2.3 Implementation HTTP/2,cleartext and when HTTP/3 possible (currently and with an encryption. IETF draft Table standard) 1 shows both thein var- on the Internet without undue impact to other users and services. cleartextboth when the possiblerelationship and and with choices encryption. between Table the 1 streaming shows the (green) var- We implement our video streaming traffic generators using the cleartextious software when possible and supported and withHTTP encryption.combinations Table 1 and shows Fig. the 2 shows var- 2.3 Implementation ious software and supported HTTP combinations and Fig. 2 shows 2.3 Implementation bothand the transport relationship (blue) and protocols. choices betweenThe IETF the QUIC streaming Working (green) Group is 2.3WePlaywright implement Implementation [13 our] browser video automation streaming frameworktraffic generators to coordinate using and the ious softwareactively and finalizing supported standardsHTTP combinations for HTTP/34 [ ] and and QUIC Fig. 2 [10 shows], both of script the client behavior and the Caddy [5] web server with config- bothand the transport relationship (blue) andprotocols. choices The between IETF QUIC the streaming Working Group (green) is WePlaywright implement [13] our browser video automation streaming framework traffic generators to coordinate using and the both thewhich relationship offer benefits and choices in more between effective the streamingbandwidth (green)use, security, and Weurations implement that ourcontrol video and streaming implementtraffic our chosen generators protocols (§ 2.1).using the andactively transport finalizing (blue) protocols.standards Thefor IETF HTTP/3 QUIC4 [ ] and Working QUIC [10 Group], both is of Playwrightscript the client [13] behaviorbrowser automation and the Caddy framework [5] web server to coordinate with config- and and transportflexibility (blue) in protocols. protocol evolution. The IETF As QUIC HTTP/3’s Working prevalence Group continues is PlaywrightFor rapid [13] prototyping browser automation and testing, framework we instrument to coordinate and automate and activelywhich finalizingoffer benefits standards in more for effective HTTP/34 [ ] andbandwidth QUIC [10 use,], both security, of and scripturations the that client control behavior and and implement the Caddy our [5 chosen] web server protocols with (§ config- 2.1). actively finalizingto grow, our standards tools will enable for us HTTP/3 to test4 [ ] and our QUIC algorithms [10], and both systems of scriptthe the web client browser behavior as our and video the application Caddy [5] web client. server Using with the config- Play- whichflexibility offer benefitsin protocol in evolution. more effective As HTTP/3’s bandwidth prevalence use, security,continues and urationsFor rapid that prototyping control and and implement testing, ourwe instrument chosen protocols and automate (§ 2.1). whichflexibility offerwith realisticbenefitsin protocol traffic in evolution. more and effective in As a HTTP/3’scontrolled, bandwidth prevalence systematic use, security,continues manner. and urationswrightFor rapid that framework, prototyping control and we andimplementinstrument testing, theour we Chromeinstrument chosen protocols and and Firefox automate (§ web 2.1). flexibilityto grow, We ourin also protocol tools support will evolution. enable cleartext us options toAs test HTTP/3’s our when algorithms possible.prevalence While and continues systems the Inter- theForbrowsers web rapid browser using prototyping JavaScript as our and video to testing, emulate application we an end-user’s instrument client. simpleUsing andautomatebehavior the Play- towith grow, realistic our tools traffic will enable and us in to a testcontrolled, our algorithms systematic and systems manner. thewright web framework, browser aswe our instrument video application the Chrome client. and Using Firefox the Play- web towith grow, realisticnet our is tools moving traffic will towards enable and usopportunistically in to a testcontrolled, our algorithms encrypting systematic and data-in-transit systems manner. thewrightof web watching framework, browser video. as ourwe For instrumentexample, video application we the have Chrome several client. automations and Using Firefox the Play-that web with realisticWe(for also example, traffic support HTTP/3 cleartext and will in options alwaysa controlled, when encrypt), possible. systematic using While cleartext manner.the has Inter- been wrightbrowsersemulate framework, using a user JavaScript watching we instrument toa video emulate for the ana specified end-user’sChrome and simpleamount Firefox behavior of time web on netWe is also moving support towards cleartext opportunistically options when encrypting possible. While data-in-transit the Inter- browsersof watching using video. JavaScript For example, to emulate we have an end-user’s several automations simple behavior that We alsouseful support for debugging cleartext and options protocol when analysis. possible. In some While cases, the Inter-either the browsersboth local using and JavaScript remote services to emulate (like anYouTube): end-user’s our simplescripts will behavior first net(for is example, moving towards HTTP/3 opportunistically will always encrypt), encrypting using cleartext data-in-transit has been ofemulate watching a user video. watching For example, a video we for have a specified several automationsamount of time that on net is movingserver towards or client opportunistically software does not encrypting implement adata-in-transit particular feature of watchingnavigate to video. a specified For example, internal we or have external severalURL, select automations the specified that (foruseful example, for debugging HTTP/3 will and alwaysprotocol encrypt), analysis. using In some cleartext cases, eitherhas been the emulateboth local a user and remotewatching services a video (like for YouTube): a specified our scriptsamount will of time first on (for example,of HTTP. HTTP/3 For example, will always while encrypt), HTTP/2 using cleartext cleartext is implemented has been in emulateresolution, a user press watching “play” a and video “watch” for a specified the video, andamount finally of closetime on usefulserver for or debugging client software and protocol does not analysis. implement In some a particular cases, either feature the bothnavigate local to and a specified remote services internal (like or YouTube):external URL, our select scripts the specified will first useful forsome debugging web servers, and protocol the major analysis. browsers In dosome not cases, support either it. the boththe local browser. and remote To retrieve services ground (like truth, YouTube): we also our capture scripts network- will first serverof HTTP. or client For example, software while does notHTTP/2 implement cleartext a particular is implemented feature in navigateresolution, to a press specified “play” internal and “watch” or external theURL video,, select andfinally the specified close server or client software does not implement a particular feature navigate to a specified internal or external URL, select the specified ofsome HTTP. web For servers, example, the while major HTTP/2 browsers cleartext do not support is implemented it. in resolution,the browser. press To retrieve “play” and ground “watch” truth, the we video, also capture and finally network- close ofsome HTTP. web For servers, example, the while major HTTP/2 browsers cleartext do not support is implemented it. in resolution,the browser. press To retrieve “play” and ground “watch” truth, the we video, also capture and finally network- close some web servers, the major browsers do not support it. the browser. To retrieve ground truth, we also capture network- Building Reproducible Video Streaming Traffic Generators CSET ’21, August 9, 2021, Virtual, CA,USA

(PCAPs) and HTTP application-level traffic for both online and one connection (‘0’). In contrast to DASH, video content chunks offline analysis. for HLS contain both audio and video segments (and thus would On the server-side, we use the Caddy web server as it supports not necessarily benefit from parallel connections). We believe that a variety of HTTP protocols (§ 2.1) and is easily extensible for connection ‘1’ was opened (and subsequently closed shortly after) our monitoring infrastructure (not covered in this paper). For eas- as a browser-level performance optimization. ier analysis and validation on the wire, we build simple, static Although not shown here, we have also observed that DASH and webpages incorporating dash.js (DASH,[8]), HLS.js (HLS,[7]), or HLS are capable of buffering enough video to continue playback plain HTML5 for videos, and serve them at unique URL endpoints. against a temporary connection disruption. We did not observe For example, 1080p video playback using DASH is available at this behavior with HTML5. We next look at the profiles of video https://example.internal/dash-1080p.htm, and so on. To fur- streaming on different HTTP versions. ther increase performance, we also pre-compute video content into relevant formats and chunks for ABR streaming as opposed to 3.2 Handling Multiple Client Generators dynamically transcoding on-demand. We look at the network throughput profiles of running multiple We use and support our implementation on emulation-based generators simultaneously. Our setup is the same as in § 3.1, except Linux testbeds and standalone computers (Linux, macOS), with with eight Google Chrome clients streaming from one server for support on Microsoft Windows planned. Our traffic generators, 250 s on a 100 Mbps link: the network topology is shown in Fig. 1, tools, and analysis are available at https://mergetb.org/projects/ with clients and server located on the nodes labeled ‘n’. searchlight/. Fig. 4 shows three graphs of data transferred over time, binned 3 EXPERIMENTATION by 1 s. Each graph contains a line for each of the eight client-server node pairs, and the top-most line is the sum across all pairs. In this section, we present a series of initial experiments designed We see that the total bandwidth use over time across all three to demonstrate our traffic generators and illustrate some of the underlying video streaming behavior on the network. We use our protocols initially maxes out the link at 100 Mbps, then exhibits experiments to understand the dynamics on the network for the steady-state behavior similar to what we saw earlier in § 3.1, with various streaming protocols (§ 3.1), how our video streaming traffic periodic spikes. We focus on the total throughput profile as the generators handle multiple clients (§ 3.2), and the network through- individual profile of any one client-server pair is hard to distinguish. put between different HTTP versions (§ 3.3). The differences at the start of each graph in Fig. 4 showthat the initial buffer fill behavior is much more varied across protocols 3.1 Differences between Video Streaming when multiple client generators are used. HTML5 (Fig. 4a) uses all Protocols the available bandwidth for 30 s before decreasing its use. DASH (Fig. 4b) and HLS (Fig. 4c) sustain maximum bandwidth for 90 s and Our experiments indicate that each video streaming protocol has 110 s, respectively. unique bandwidth-throughput characteristics at the network level. The difference in initial throughput between HTML5 and ABR We run our traffic generators on one Google Chrome client, con- protocols (DASH, HLS) is again likely due to the difference in buffer necting directly to a server and watching video over 3 streaming filling strategies, with HTML5 keeping a minimal playback buffer protocols at 1080p for roughly 90–120 s on a 10 Mbps link. (we observed earlier that HTML5 suffers early on from a disrupted Fig. 3 shows three graphs of data transferred over time, binned connection). While the available bandwidth to any one client is by 1 s. Each graph contains a line for each TCP connection (most limited due to contention, it is sufficient to maintain continuous browsers support up to six parallel connections in HTTP/1.1) and playback. Once all clients have reached steady-state in ABR stream- the top-most line (sometimes overlapping with other lines) is the ing, we see the usual bursty behavior as each client can quickly sum across all connections. Once we start playing video at roughly refill its buffer due to the lack of contention. 5–15 s, we see that each streaming protocol can effectively use Finally, we note that we are not necessarily able to observe the all available bandwidth. We observe that DASH (Fig. 3b) and HLS video streaming Quality of Experience (QoE) with throughput pro- (Fig. 3c) use bandwidth more efficiently as shown in the various files alone. For example, one of our eight clients might experience “valleys” in each line: this behavior is likely due to a more intelligent more rebuffering events, with slight pauses in playback, than others buffering strategy. as that client constantly attempts to retrieve more data. We plan DASH also takes advantage of multiple connections by stream- on developing instrumentation for client browsers to report QoE ing audio and video in separate connections, as seen in Fig. 3b. The measurement data as future work. DASH protocol separates audio and video file chunks, in contrast to HTML5 and HLS, in which each file or file chunk contains both, respectively. Initially, connection ‘1’ downloads video and ‘2’ down- 3.3 Differences between HTTP Versions loads audio chunks, switching once at roughly 30 s. We do not have Lastly, we look at the network throughput profiles of various trans- an explanation for why a connection switches at this point, but port protocols on a remote video streaming service. We run our traf- hypothesize that connection management decisions are made at fic generators on one Google Chrome client, connecting to YouTube the underlying browser-level rather than at the video player. and watching video over 3 different HTTP versions, all encrypted Finally, HLS in Fig. 3c seems to make the most efficient use of under TLS, at 1440p for roughly 80 s on a 200 Mbps link. bandwidth and number of connections. Both the supporting content Fig. 5 shows three graphs of data transferred over time, binned (HTML, CSS, JavaScript) and video content are handled with only by 1 s. While our client connects to multiple servers for supporting CSET ’21, August 9, 2021, Virtual, CA, USA Calvin Ardi, Alefiya Hussain, and Stephen Schwab CSET ’21, August 9, 2021, Virtual, CA, USA Calvin Ardi, Alefiya Hussain, and Stephen Schwab

1,309,793 1,301,350 1,294,414

1,200,000 1,200,000 1,200,000 1,309,793 1,301,350 1,294,414

1,200,000 1,200,000 1,200,000 1,000,000 1,000,000 1,000,000

1,000,000 1,000,000 1,000,000 800,000 800,000 800,000

800,000 800,000 800,000 600,000 600,000 600,000 data (bytes) data (bytes) data (bytes)

600,000 tcpstream 600,000 600,000 data400,000 (bytes) data (bytes) 400,000 data (bytes) 400,000 0 tcpstream 1 0 tcpstream 400,000 tcpstream 400,000 400,000 200,000 20 200,000 tcpstream1 200,000 0 31 02 tcpstream1 200,000 total2 200,000 1total 200,000 0 total 0 0 0 3 2 1 0 20 40 60 80 0 20 40 60 80 100 120 0 20 40 60 80 100 total total total 0 time (seconds) 0 time (seconds) 0 time (seconds) 0 20 40 60 80 0 20 40 60 80 100 120 0 20 40 60 80 100 time (seconds) time (seconds) time (seconds) (a) HTML5 (b) DASH (c) HLS Figure 3: Streaming(a) HTML5 video from one server to one client over(b) multiple DASH video streaming protocols via HTTP/1.1(c) HLS cleartext, 1080p resolution,Figure 3: Streaming 10 Mbps linkvideo speed. from one server to one client over multiple video streaming protocols via HTTP/1.1 cleartext, 1080p resolution, 10 Mbps link speed.

12,705,468 12,752,476 12,759,462 12,000,000 12,000,000 12,000,000 12,705,468 12,752,476 12,759,462 12,000,000 12,000,000 12,000,000 10,000,000 10,000,000 10,000,000

10,000,000 10,000,000 10,000,000 8,000,000 8,000,000 8,000,000 node pair node pair node pair 8,000,000 10.0.0.100 10.5.1.100 8,000,000 10.0.0.100 10.5.1.100 8,000,000 10.0.0.100 10.5.1.100 6,000,000 ↔ 6,000,000 ↔ 6,000,000 ↔

data (bytes) 10.0.1.100node pair10.5.1.100 data (bytes) 10.0.1.100node pair 10.5.1.100 data (bytes) 10.0.1.100node pair 10.5.1.100 ↔ ↔ ↔ 10.1.0.10010.0.0.100 10.5.1.10010.5.1.100 10.0.0.10010.1.0.100 10.5.1.10010.5.1.100 10.0.0.10010.1.0.10010.5.1.10010.5.1.100 6,000,000 ↔↔ 6,000,000 ↔↔ 6,000,000 ↔ ↔

4,000,000data (bytes) 10.1.1.10010.0.1.100 10.5.1.10010.5.1.100 data (bytes) 4,000,000 10.0.1.10010.1.1.100 10.5.1.10010.5.1.100 data (bytes) 4,000,000 10.0.1.10010.1.1.10010.5.1.10010.5.1.100 ↔↔ ↔↔ ↔ ↔ 10.2.0.10010.1.0.100 10.5.1.10010.5.1.100 10.1.0.10010.2.0.100 10.5.1.10010.5.1.100 10.1.0.10010.2.0.10010.5.1.10010.5.1.100 ↔↔ ↔↔ ↔ ↔ 4,000,000 10.2.1.10010.1.1.100 10.5.1.10010.5.1.100 4,000,000 10.1.1.10010.2.1.100 10.5.1.10010.5.1.100 4,000,000 10.1.1.10010.2.1.10010.5.1.10010.5.1.100 ↔↔ ↔↔ ↔ ↔ 2,000,000 10.3.0.10010.2.0.100 10.5.1.10010.5.1.100 2,000,000 10.2.0.10010.3.0.100 10.5.1.10010.5.1.100 2,000,000 10.2.0.10010.3.0.10010.5.1.10010.5.1.100 ↔↔ ↔↔ ↔ ↔ 10.3.1.10010.2.1.100 10.5.1.10010.5.1.100 10.2.1.10010.3.1.100 10.5.1.10010.5.1.100 10.2.1.10010.3.1.10010.5.1.10010.5.1.100 ↔↔ ↔↔ ↔ ↔ 2,000,000 total10.3.0.100 10.5.1.100 2,000,000 10.3.0.100total 10.5.1.100 2,000,000 10.3.0.100total 10.5.1.100 ↔ ↔ ↔ 0 10.3.1.100 10.5.1.100 0 10.3.1.100 10.5.1.100 0 10.3.1.100 10.5.1.100 0 50 100 150 200 ↔ 0 50 100 150 200 ↔ 0 50 100 150 200 ↔ total total total 0 time (seconds) 0 time (seconds) 0 time (seconds) 0 50 100 150 200 0 50 100 150 200 0 50 100 150 200 time (seconds) time (seconds) time (seconds) (a) HTML5 (b) DASH (c) HLS (a) HTML5 (b) DASH (c) HLS Figure 4: Streaming video from one server to eight clients over multiple video streaming protocols via HTTP/1.1 cleartext, 1080pFigure resolution, 4: Streaming 100 video Mbps from link speed. one server to eight clients over multiple video streaming protocols via HTTP/1.1 cleartext, 1080p resolution, 100 Mbps link speed.

12,976,781 14,459,021 16,000,000 14,000,000 12,000,000 14,000,000 12,000,000 10,000,000 12,000,000 10,000,000 8,000,000 10,000,000 8,000,000 8,000,000 6,000,000 data (bytes) data (bytes) 6,000,000 data (bytes) 6,000,000 4,000,000 4,000,000 4,000,000

2,000,000 protocol 2,000,000 2,000,000 protocol protocol TCP TCP TCP QUIC 0 0 0 0 10 20 30 40 50 60 70 80 0 10 20 30 40 50 60 70 80 0 10 20 30 40 50 60 70 80 time (seconds) time (seconds) time (seconds) (a) HTTP/1.1 + TLS (b) HTTP/2 + TLS (c) HTTP/3 (Q050) Figure 5: Streaming video from YouTube to one client over multiple transport protocols, 1440p resolution, 200 Mbps link speed. Figure 5: Streaming video from YouTube to one client over multiple transport protocols, 1440p resolution, 200 Mbps link speed. content (ads, telemetry, and CDNs), we graph the total throughput 4 FUTURE WORK AND CONCLUSION contentper underlying (ads, telemetry, protocol and (TCP CDNs),or QUIC we graph) for the simplicity. total throughput We start In4 this FUTURE paper we presented WORK toolsAND to CONCLUSION create representative and re- pervideo underlying playback at protocol around 10(TCP s intoor theQUIC capture) for to simplicity. allow enough We start time producibleIn this paper video we streaming presented traffic tools to to create evaluate representative next generation andQoS re- videoto download playback the at around website’s 10 ssupporting into the capture content. to allow We observe enough in time all algorithmsproducible videoand traffic streaming classification traffic to evaluate and engineering next generation systems.QoS We tothree download protocols the similar website’s playback supporting behavior content. in our priorWe observe experiments: in all discussedalgorithms 27 and different traffic combinations classification of streaming and engineering video traffic systems. types We threean initial protocols spike similar in throughput playback usage behavior to fill in ourthe prior buffer experiments: and compute indiscussed this preliminary 27 different work, combinations and illustrated of thestreaming diversity video of network- traffic types anavailable initial spike bandwidth, in throughput with steady-state usage to fill bursty the behavior buffer following. and compute levelin this dynamics preliminary in these work, protocols. and illustrated We will the continue diversity to of build network- our availableWhile bandwidth, HTTP/1.1 (Fig. with 5a) steady-state and HTTP/2 bursty (Fig. 5b) behavior have roughly following. the trafficlevel dynamics generators in these with protocols. the goal We willof scaling continueup to on build the our order ofe3 same throughput profile, we see that YouTube leverages HTTP/3, if While HTTP/1.1 (Fig. 5a) and HTTP/2 (Fig. 5b) have roughly the clientstraffic and generators servers. To makewith trafficthe goal moreof scaling representative, up on the orderof 10we3 plan available, with an overall throughput that is slightly higher. We see same throughput profile, we see that YouTube leverages HTTP/3, if toclients build and models servers. of human To make behavior traffic in video more streaming representative, consump- we plan in Fig. 5c an initial TCP connection, which negotiates and switches available, with an overall throughput that is slightly higher. We see tionto build (binge models watching of human vs. skipping behavior around) in video and streaming use these consump- models over to the use of HTTP/3 over QUIC for the rest of the experiment in Fig. 5c an initial TCP connection, which negotiates and switches totion drive (binge our watching clients. On vs. the skipping network- around) and server-side, and use these we plan models to duration—the lingering TCP connection remains open, but unused, over to the use of HTTP/3 over QUIC for the rest of the experiment emulateto drive ourrealistic clients. network On the architectures, network- and like server-side, CDNs and we anycast, plan to with keepalives. We also observe a larger maximum throughput duration—the lingering TCP connection remains open, but unused, andemulate implement realistic more network complex architectures, websites, incorporating like CDNs and supporting anycast, usage of 128 Mbps compared to earlier HTTP versions (104–116 with keepalives. We also observe a larger maximum throughput frameand implement content and moreABR complexstreaming websites, behavior. incorporating Our traffic supporting generators Mbps). We plan to continue adding support for more remote Internet usage of 128 Mbps compared to earlier HTTP versions (104–116 areframe available content at and https://mergetb.org/projects/searchlight/.ABR streaming behavior. Our traffic generators services and HTTP/3. Mbps). We plan to continue adding support for more remote Internet are available at https://mergetb.org/projects/searchlight/. services and HTTP/3. Building Reproducible Video Streaming Traffic Generators CSET ’21, August 9, 2021, Virtual, CA,USA

REFERENCES [8] DASH Industry Forum. 2021. dash.js JavaScript Reference Client v3.2.2. [1] Zahaib Akhtar, Yun Seong Nam, Jessica Chen, Ramesh Govindan, Ethan https://reference.dashif.org/dash.js/ Katz-Bassett, Sanjay Rao, Jibin Zhan, and Hui Zhang. 2018. Understanding Video [9] Mojgan Ghasemi, Partha Kanuparthy, Ahmed Mansy, Theophilus Benson, and Management Planes. In Proceedings of the Internet Measurement Conference 2018 Jennifer Rexford. 2016. Performance Characterization of a Commercial Video (Boston, MA, USA) (IMC ’18). Association for Computing Machinery, New York, Streaming Service. In Proceedings of the 2016 Internet Measurement Conference NY, USA, 238–251. https://doi.org/10.1145/3278532.3278554 (Santa Monica, California, USA) (IMC ’16). Association for Computing [2] Doreid Ammar, Thomas Begin, and Isabelle Guerin-Lassous. 2011. A New Tool Machinery, New York, NY, USA, 499–511. for Generating Realistic Internet Traffic in NS-3. In Proceedings of the 4th https://doi.org/10.1145/2987443.2987481 International ICST Conference on Simulation Tools and Techniques (Barcelona, [10] Jana Iyengar and Martin Thomson. 2021. QUIC: A UDP-Based Multiplexed and Spain) (SIMUTools ’11). ICST (Institute for Computer Sciences, Social-Informatics Secure Transport. RFC 9000. https://doi.org/10.17487/RFC9000 and Telecommunications Engineering), Brussels, BEL, 81–83. [11] Sina Keshvadi and Carey Williamson. 2021. An Empirical Measurement Study of [3] Abdelhak Bentaleb, Bayan Taani, Ali C. Begen, Christian Timmerer, and Roger Free Live Streaming Services. In Passive and Active Measurement, Oliver Zimmermann. 2019. A Survey on Bitrate Adaptation Schemes for Streaming Hohlfeld, Andra Lutu, and Dave Levin (Eds.). Springer International Publishing, Media Over HTTP. IEEE Communications Surveys Tutorials 21, 1 (2019), 562–585. Cham, 111–127. https://doi.org/10.1007/978-3-030-72582-2_7 https://doi.org/10.1109/COMST.2018.2862938 [12] ESnet / Lawrence Berkeley National Laboratory. 2020. iperf3 v3.9. [4] Mike Bishop. 2021. Hypertext Transfer Protocol Version 3 (HTTP/3). https://software.es.net/iperf/ Internet-Draft draft-ietf-quic-http-34. Internet Engineering Task Force. [13] Microsoft. 2021. Playwright v1.10.0. https://playwright.dev/ https://datatracker.ietf.org/doc/html/draft-ietf-quic-http-34 Work in Progress. [14] Sandvine. 2020. The Global Internet Phenomena Report COVID-19 Spotlight. (7 [5] Caddy. 2020. Caddy v2.1.1. https://caddyserver.com/ May 2020). https://www.sandvine.com/phenomena [6] Cisco. 2018. Cisco Visual Networking Index: Forecast and Trends, 2017–2022 [15] Joel Sommers, Hyungsuk Kim, and Paul Barford. 2004. Harpoon: A Flow-Level White Paper. (27 Nov. 2018). https://web.archive.org/web/20200215211855/https: Traffic Generator for Router and Network Tests. SIGMETRICS Perform. Eval. Rev. //www.cisco.com/c/en/us/solutions/collateral/service-provider/visual- 32, 1 (June 2004), 392. https://doi.org/10.1145/1012888.1005733 networking-index-vni/white-paper-c11-741490.html [16] Michele C. Weigle, Prashanth Adurthi, Félix Hernández-Campos, Kevin Jeffay, [7] Dailymotion. 2021. HLS.js v1.0.3. https://github.com/video-dev/hls.js and F. Donelson Smith. 2006. Tmix: A Tool for Generating Realistic TCP Application Workloads in Ns-2. SIGCOMM Comput. Commun. Rev. 36, 3 (July 2006), 65–76. https://doi.org/10.1145/1140086.1140094