Measuring and Improving the Quality of Experience of Adaptive Rate Video Hyunwoo Nam Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy in the Graduate School of Arts and Sciences COLUMBIA UNIVERSITY 2016 c 2016 Hyunwoo Nam All Rights Reserved ABSTRACT Measuring and Improving the Quality of Experience of Adaptive Rate Video Hyunwoo Nam Today’s popular over-the-top (OTT) video streaming services such as YouTube, Netflix and Hulu deliver video contents to viewers using adaptive bitrate (ABR) technologies. In ABR streaming, a video player running on a viewer’s device adaptively changes bitrates to match given network conditions. However, providing reliable streaming is challenging. First, an ABR player may select an inappropriate bitrate during playback due to the lack of direct knowledge of access networks, frequent user mobility and rapidly changing channel conditions. Second, OTT content is delivered to viewers without any cooperation with Internet service providers (ISPs). Last, there are no appropriate tools that evaluate the performance of ABR streaming along with video quality of experience (QoE). This thesis describes how to improve the video QoE of OTT video streaming services using ABR technologies. Our analysis starts from understanding ABR heuristics. How does ABR streaming work? What factors does an ABR player consider when switching bitrates during a download? Then, we propose our solutions to improve existing ABR streaming from the perspective of network operators who deliver video content through their networks and video service providers who build ABR players running on viewers’ devices. From the network operators’ point of view, we propose to find a better video content server based on round trip times (RTTs) between an edge node of a wireless network and available video content servers when a viewer requests a video. The edge node can be an Internet Service Provider (ISP) router in a Wi-Fi network and a packet data network gateway (P-GW) in a 4G network. During the experiments, our solution showed better TCP performance (e.g., higher TCP throughput during playback) 146 times out of 200 experiments (73%) over Wi-Fi networks and 162 times out of 200 experiments (81%) over 3G networks. In addition, we claim that the wireless edge nodes can assist an ABR video player in selecting the best available bitrate by controlling the available bandwidth in the radio access network between a base station and a viewer’s device. In our Wi-Fi testbed, the proposed solution saved up to 21% of radio bandwidth on mobile devices and enhanced the viewing experience by reducing rebu↵erings during playback. Last, we assert that software- defined networking (SDN) can improve video QoE by dynamically controlling routing paths of video streaming flows based on the provisioned networking information collected from SDN-enabled networking devices. Using an o↵-the-shelf SDN platform, we showed that our proposed solution can reduce rebu↵erings by 50% and provide higher bitrates during a download. From the perspective of video service providers, higher video QoE can be achieved by improving ABR heuristics implemented in an ABR player. To support this idea, we investigated the role of playout bu↵er size in ABR streaming and its impact on video QoE. Through our video QoE survey, we proved that a large bu↵er does not always outperform a small bu↵er, especially under rapidly varying network conditions. Based on this finding, we suggest to dynamically change the maximum bu↵er size in an ABR player depending on the current capacity of its playout bu↵er for improving the QoE of viewers. During the experiments, our proposed solution improved the viewing experience by o↵ering 15% higher average played bitrate, 70% fewer bitrate changes and 50% shorter rebu↵ering duration. Our experimental results show that even small changes of ABR heuristics and new fea- tures of network systems can greatly a↵ect video QoE. However, it is still difficult for video service providers or network operators to evaluate new ABR heuristics or network system changes due to lack of accurate QoE monitoring systems. In order to solve this issue, we have developed YouSlow (“YouTube Too Slow!? - YouSlow”) as a new approach to monitor- ing video QoE for the analysis of ABR performance. The lightweight web browser plug-in and mobile application are designed to monitor various playback events (e.g., rebu↵ering duration and frequency of bitrate changes) directly from within ABR video players and calculate statistics along with video QoE. Using YouSlow, we investigate the impact of the above playback events on video abandonment: about 10% of viewers abandoned the YouTube videos when the pre-roll ads lasted for 15 seconds. Even increasing the bitrate can annoy viewers; they prefer a high starting bitrate with no bitrate changes during playback. Our regression analysis shows that bitrate changes do not a↵ect video abandonment signif- icantly and the abandonment rate can be estimated accurately using the rebu↵ering ratio and the number of rebu↵erings (R2 = 0.94). The thesis includes four main contributions. First, we investigate today’s popular OTT video streaming services (e.g., YouTube and Netflix) that use ABR streaming technologies. Second, we propose to build QoS and QoE aware video streaming that can be implemented in existing wireless networks (e.g., Wi-Fi, 3G and 4G) and in SDN-enabled networks. Third, we propose to improve current ABR heuristics by dynamically changing the playout bu↵er size under varying network conditions. Last, we designed and implemented a new monitoring system for measuring video QoE. Table of Contents List of Figures vi List of Tables ix 1 Introduction 1 1.1 Challenges..................................... 3 1.2 Overview and main contributions . 4 I Prelude: Understanding OTT Video Streaming and ABR Streaming Technologies 7 2 An Empirical Study of OTT Video Streaming 8 2.1 Introduction.................................... 8 2.2 Onlinevideodeliverybackground. 9 2.2.1 Progressive download . 10 2.2.2 RTMP/RTSP chunk based delivery . 10 2.2.3 ABR streaming . 11 2.3 Understanding ABR streaming technologies . 11 2.3.1 Network traffic behavior in ABR streaming . 14 2.4 Understanding OTT video streaming platforms . 16 2.4.1 An analysis of YouTube video streaming . 16 2.4.2 An analysis of Netflix video streaming . 20 2.4.3 Summary of key observations . 21 i 2.5 Conclusions.................................... 22 II Intelligent Network Architecture for OTT Video Streaming 26 3 Towards Dynamic Network Condition-Aware Video Server Selection over Wireless Networks 27 3.1 Introduction.................................... 27 3.2 An analysis of YouTube video server selection algorithms . 29 3.2.1 Requesting videos on di↵erent devices over Wi-Fi networks . 30 3.2.2 Requesting videos on the same devices over Wi-Fi networks under varying network conditions . 30 3.2.3 Requesting videos on the same devices via di↵erent wireless network interfaces ................................. 31 3.2.4 Requesting videos on the same devices from the same place over Wi-Fi and 3G networks during a 24 hour period . 32 3.3 YouTube often assigns video content servers with long RTTs . 32 3.3.1 Finding locations of YouTube video content servers . 32 3.3.2 Measuring RTTs between video content servers and viewers . 33 3.3.3 Video content servers with long RTTs to viewers may degrade video QoE.................................... 35 3.4 RTT-based video server selection algorithms . 35 3.4.1 Caching addresses of video content servers . 37 3.4.2 Discoveringapreferredvideocontentserver . 37 3.5 Evaluation . 39 3.6 Relatedwork ................................... 41 3.7 Conclusions.................................... 42 4 Towards Dynamic QoS-aware OTT Video Streaming 43 4.1 Introduction.................................... 43 4.2 Poorly designed video players waste network bandwidth . 45 4.2.1 Calculating discard ratio . 47 ii 4.2.2 Summary of key observations . 48 4.3 ImprovingOTTvideocontentdeliveryin4Gnetworks . 48 4.3.1 QoS in 4G networks . 49 4.3.2 Dynamic QoS-aware video content delivery in 4G networks . 50 4.4 Performance evaluation of the dynamic QoS-aware video streaming platform 52 4.5 Relatedwork ................................... 57 4.6 Discussion..................................... 58 4.7 Conclusions.................................... 58 5 Towards QoE-aware Video Streaming using SDN 60 5.1 Introduction.................................... 60 5.2 ProblemsonexistingOTTvideodeliverysystem . 62 5.3 QoE-aware video streaming using SDN . 62 5.3.1 Application-level video QoE metrics . 64 5.3.2 Pinpointing a bottleneck using SDN . 64 5.3.3 Dynamic network condition-aware path optimization with SDN . 65 5.4 Implementation.................................. 67 5.5 Evaluation . 69 5.6 Relatedwork ................................... 71 5.7 Discussion..................................... 72 5.8 Conclusions.................................... 73 III ABR Streaming Heuristics 76 6 An Empirical Evaluation of Playout Bu↵er Dimensioning in ABR Stream- ing 77 6.1 Introduction.................................... 77 6.2 Motivation . 78 6.3 Analysis of the role of playout bu↵er size in ABR streaming . 80 6.3.1 Testbedsetups .............................. 81 6.3.2 Analysis of experimental results . 83 iii 6.4 The impact of playout bu↵er size on video QoE in ABR streaming . 92 6.4.1 Online crowdsourcing platform . 94 6.4.2 QoEsurveyresults............................ 97 6.4.3 Summary of key observations . 99 6.5 Adaptive playout bu↵ersize........................... 101 6.5.1 Evaluation . 103 6.6 Related work . 107 6.7 Conclusions . 107 IV Video QoE Monitoring Tool 110 7 QoE Matters More Than QoS: Why People Stop Watching Cat Videos 111 7.1 Introduction.................................... 111 7.2 YouSlowoverview................................. 113 7.2.1 Implementation . 113 7.2.2 What factors can YouSlow measure? . 115 7.3 YouTube measurements . 116 7.4 Video QoE analysis via YouSlow . 124 7.4.1 QoS and QoE methods for an analysis of video QoE . 124 7.4.2 QoE analysis report . 126 7.4.3 Regression analysis .