Live Streaming Performance of the Zattoo Network Hyunseok Chang Sugih Jamin Wenjie Wang Alcatel-Lucent Bell Labs University of Michigan Zattoo Inc. Holmdel, NJ 07733 USA Ann Arbor, MI 48109 USA Ann Arbor, MI 48105 USA [email protected] [email protected] [email protected] ABSTRACT 1. INTRODUCTION A number of commercial peer-to-peer systems for live stream- We draw a distinction between three uses of peer-to-peer ing, such as PPLive, Joost, LiveStation, SOPCast, TVants, (P2P) networks: delay tolerant file download of archival ma- etc. have been introduced in recent years. The behavior of terial, delay sensitive progressive download (or streaming) these popular systems has been extensively studied in sev- of archival material, and real-time live streaming. In the eral measurement papers. Due to the proprietary nature first case, the completion of download is elastic, depending of these commercial systems, however, these studies have on available bandwidth in the P2P network. The applica- to rely on a “black-box” approach, where packet traces are tion buffer receives data as it trickles in and informs the collected from a single or a limited number of measurement user upon the completion of download. The user can then points, to infer various properties of traffic on the control and start playing back the file for viewing in the case of a video data planes. Although such studies are useful to compare file. In the second case, video playback starts as soon as different systems from end-user’s perspective, it is difficult the application assesses it has sufficient data buffered that, to intuitively understand the observed properties without given the estimated download rate and the playback rate, fully reverse-engineering the underlying systems. Our paper it will not deplete the buffer before the end of file. If this presents a large-scale measurement study of Zattoo, one of assessment is wrong, the application would have to either the largest production live streaming providers in Europe, pause playback and rebuffer, or slow down playback. While using data collected by the provider. To highlight, we found users would like playback to start as soon as possible, the that even when the Zattoo system was heavily loaded with application has some degree of freedom in trading off play- as high as 20,000 concurrent users on a single overlay, the back start time against estimated network capacity. The median channel join delay remained less than 2 to 5 seconds, third case, real-time live streaming, has the most stringent and that, for a majority of users, the streamed signal lags delay requirement. While progressive download may toler- over-the-air broadcast signal by no more than 3 seconds. To ate initial buffering of tens of seconds or even minutes, live motivate the measurement study, we also present a descrip- streaming generally cannot tolerate more than a few seconds tion of the Zattoo network architecture. of buffering. Taking into account the delay introduced by signal ingest and encoding, and network transmission and propagation, the live streaming system can introduce only Categories and Subject Descriptors a few seconds of buffering time end-to-end and still be con- C.2.1 [Computer-Communication Networks]: Network sidered “live” [1]. Architecture and Design; C.2.2 [Computer-Communication The Zattoo peer-to-peer live streaming system was a free- Networks]: Network Protocols; C.4 [Performance of Sys- to-use network serving over 3 million registered users in eight tems] European countries at the time of study, with a maximum of over 60,000 concurrent users on a single channel. The sys- tem delivers live streams using a receiver-based, peer-division General Terms multiplexing scheme as described in Section 2. After delv- measurement, performance, design ing into Zattoo’s architecture in detail, we study in Sections 3 and 4 large-scale measurements collected during the live broadcast of the UEFA European Football Championship, Keywords one of the most popular one-time events in Europe, in June, Peer-to-peer system, live streaming, network architecture 2008 [2]. During the course of the month of June 2008, Zattoo served more than 35 million sessions to more than one million distinct users. Drawing from these measure- ments, we report on the operational scalability of Zattoo’s Permission to make digital or hard copies of all or part of this work for live streaming system along several key issues: personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. IMC'09, November 4–6, 2009, Chicago, Illinois, USA. Copyright 2009 ACM 978-1-60558-770-7/09/11 ...$10.00. 417 1. How does the system scale in terms of overlay size and its effectiveness in utilizing peers’ uplink bandwidth? 2. How responsive is the system during channel switching, ¢ ¦ £ ¢ for example, when compared to the 3-second channel ¡ ¥ ¡ ¨ switch time of satellite TV? © ¡ § ¡ ¡ 3. How effective is the packet retransmission scheme in allowing a peer to recover from transient congestion? 4. How effective is the receiver-based peer-division multi- § ¡ ¢ ¦ ¢ © ¡ ¥ ¡ plexing scheme in delivering synchronized sub-streams? ¡ ¢ £ ¡ ¤ ¥ ¦ § ¨ © ¡ ¥ ¡ ¡ £ © ¡ ¥ ¡ 5. Would a peer further away from the stream source ex- ¡ ¡ £ ¢ ¨ ¡ ¥ ¡ ¨ perience adversely long lag compared to a peer closer ¦ to the stream source? 6. How effective is error-correcting code in isolating packet losses on the overlay? Figure 1: Zattoo delivery network architecture. We also discuss in Section 5 several challenges in increasing the bandwidth contribution of Zattoo peers. Finally, we describe related works in Section 6 and conclude in Section 7. network as n logical sub-streams. Thus the first packet gen- 2. SYSTEM ARCHITECTURE erated is considered part of the first sub-stream, the second The Zattoo system rebroadcasts live TV, captured from packet that of the second sub-stream, the n-th packet that satellites, onto the Internet. The system carries each TV of the n-th sub-stream. The n + 1-th packet cycles back to channel on a separate peer-to-peer delivery network and is the first sub-stream, etc. such that the i-th sub-stream car- not limited in the number of TV channels it can carry. Al- ries the mn + i-th packets, where m ≥ 0, 1 ≤ i ≤ n, and n though a peer can freely switch from one TV channel to a user-defined constant. We call a set of n packets with the another, and thereby departing and joining different peer- same index multiplier m a “segment.” Thus m serves as the to-peer networks, it can only join one peer-to-peer network segment index, while i serves as the packet index within a at any one time. We henceforth limit our description of the segment. Each segment is of size n packets. Being the packet Zattoo delivery network as it pertains to carrying one TV index, i also serves as the sub-stream index. The number channel. Fig. 1 shows a typical setup of a single TV chan- mn + i is carried in each packet as its sequence number. nel carried on the Zattoo network. TV signal captured from Zattoo uses the Reed-Solomon (RS) error correcting code satellite is encoded into H.264/AAC streams, encrypted, and (ECC) for forward error correction. The RS code is a sys- sent onto the Zattoo network. The encoding server may be tematic code: of the n packets sent per segment, k < n pack- physically separated from the server delivering the encoded ets carry the live stream data while the remainder carries the content onto the Zattoo network. For ease of exposition, redundant data [3, Section 7.3]. Due to the variable-bit rate we will consider the two as logically co-located on an En- nature of the data stream, the time period covered by a seg- coding Server. Users are required to register themselves at ment is variable, and a packet may be of size less than the the Zattoo website to download a free copy of the Zattoo maximum packet size. A packet smaller than the maximum player application. To receive the signal of a channel, the packet size is zero-padded to the maximum packet size for user first authenticates itself to the Zattoo Authentication the purposes of computing the (shortened) RS code, but is Server. Upon authentication, the user is granted a ticket transmitted in its original size. Once a peer has received k with limited lifetime. The user then presents this ticket, packets per segment, it can reconstruct the remaining n − k along with the identity of the TV channel of interest, to the packets. We do not differentiate between streaming data Zattoo Rendezvous Server. If the ticket specifies that the and redundant data in our discussion in the remainder of user is authorized to receive signal of the said TV channel, this paper. the Rendezvous Server returns to the user a list of peers When a new peer requests to join an existing peer, it spec- currently joined to the P2P network carrying the channel, ifies the sub-stream(s) it would like to receive from the exist- together with a signed channel ticket. If the user is the ing peer. These sub-streams do not have to be consecutive. first peer to join a channel, the list of peers it receives con- Contingent upon availability of bandwidth at existing peers, tain only the Encoding Server. The user joins the channel the receiving peer decides how to multiplex a stream onto by contacting the peers returned by the Rendezvous Server, its set of neighboring peers, giving rise to our description of presenting its channel ticket, and obtaining the live stream the Zattoo live streaming protocol as a receiver-based, peer- of the channel from them (see Section 2.1 for details).