On Quality Aware Adaptation of Internet Video Dimitrios Miras A dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy of the University of London. Department of Computer Science University College London May 2004 ProQuest Number: U643878 All rights reserved INFORMATION TO ALL USERS The quality of this reproduction is dependent upon the quality of the copy submitted. In the unlikely event that the author did not send a complete manuscript and there are missing pages, these will be noted. Also, if material had to be removed, a note will indicate the deletion. uest. ProQuest U643878 Published by ProQuest LLC(2016). Copyright of the Dissertation is held by the Author. All rights reserved. This work is protected against unauthorized copying under Title 17, United States Code. Microform Edition © ProQuest LLC. ProQuest LLC 789 East Eisenhower Parkway P.O. Box 1346 Ann Arbor, Ml 48106-1346 To my parents, Fotios and Ioanna To Athena Abstract The main issue with the transmission of streamed video over the Internet is that of adaptation: due to the best-effort service model of the Internet, abundance of bandwidth to guarantee good quality is not always feasible. Instead, congestion control of the shared resources needs to be employed, resulting in variations in bandwidth availability. These variations of the transmis­ sion rate introduce a first level of quality degradation. In addition, the time-varying complexity of the underlying visual scenes requires a widely fluctuating transmission bandwidth in order to achieve good quality, otherwise quality oscillations occur. In order to tackle these prob­ lems, a multitude of rate-adaptation techniques have been proposed. However, these propos­ als either consider adaptation solely from the network point of view (e.g., TCP-friendliness, rate-adaptation, layering) and completely disregard the effect on quality, or employ simplistic metrics of quality (e.g., peak signal-to-noise ratio) that are not necessarily true representations of the quality as experienced by the user of the video service. This thesis advocates the integration of emerging objective video quality metrics within the adaptation cycle of Internet video. A result of recent research efforts, objective quality metrics are computational models that produce quality ratings which are highly correlated with human judgements of quality. By considering the time-changing relationship between properties of the video content, the available bit rate, and the effect of both on perceived quality, this disserta­ tion studies quality-aware rate adaptation techniques that improve end quality perception in the context of two different application scenarios of video streaming. Firstly, this dissertation examines applications that involve the transmission of multiple concurrent media streams to a receiver (e.g., the transmission and display of several video streams, relevant to the application scenario). We encounter the problem of efficiently appor­ tioning the bandwidth available to a multi-stream session among its constituent media flows, by allowing participating flows to jointly adapt their transmission rates in consideration of their re­ spective time-varying quality. Suitable adaptation timescales that coincide with changes in the video content (scene cuts) and an inter-stream adaptation mechanism that considers the time- varying objective quality of the participating streams are proposed. Experimental results show Abstract 4 the benefits of the proposed method, in terms of improved session quality and utilisation of the session bandwidth, in comparison to (i) a priority-based inter-stream adaptation and (ii) the case where the session flows are transmitted over independent congestion controlled connections. Secondly, the thesis deals with the problem of providing smooth quality rate adaptation for live, unicast, real-time video streaming. Since real-time performance is necessary, an objec­ tive quality metric cannot be applied in-line, as it is computationally intensive. For this reason, artificial neural networks are utilised to predict quality ratings in real-time. Predictions are sought based on descriptive features of the video content and the bandwidth that is available to the stream. The limitations of current approaches to provide stable or smooth quality are identified. A rate-quality controller, built on the principles of fuzzy logic, is then developed to alleviate annoying short-term quality variations that appear due to mismatches between the available bandwidth and the rate required for stable quality. Based on the neural network’s quality predictions, the controller continuously monitors the recent quality values, the nominal transmission rate and the occupancy levels of the participating buffers, to calculate appropriate encoding rates that eliminate short-term quality fluctuations. Experimental results show that the proposed solution offers significant stability of short-term quality and ‘smoothes out’ an­ noying oscillations of quality to extreme low and high values, while at the same time respects transmission rate constraints and preserves buffer stability. By presenting numerous experimental results with a wide variety of video sequences, this dissertation shows that video streaming systems can utilise objective measures of perceived quality to deliver improved presentation quality, by applying quality-aware adaptation tech­ niques which are tailored to the semantics of the specific streaming application. Contents 1 Introduction 16 1.1 Video adaptation - interaction between the codec and the n etw ork .................. 17 1.2 Video adaptation - interaction between the codec, network and metrics of per­ ceived quality 20 1.3 Contributions ...................................................................................................... 21 1.4 Structure of thesis ................................................................................................ 23 2 Internet video streaming - issues and challenges 25 2.1 Introduction ......................................................................................................... 25 2.2 Overview of video communications .................................................................. 26 2.3 Video compression ............................................................................................. 29 2.3.1 Current and emerging video compression solutions ................................ 30 2.4 Networked delivery of compressed video ............................................................ 32 2.4.1 Transport rate control ............................................................................... 34 2.4.2 Rate-adaptive video encoding ................................................................ 37 2.4.3 The role of buffering ............................................................................... 40 2.5 Streaming video content: the effect on quality ................................................... 41 2.5.1 Encoding artifacts .................................................................................. 42 2.5.2 Transmission artifacts ............................................................................ 43 2.6 Measuring video quality .................................................................................... 43 2.6.1 Subjective video assessment .................................................................... 46 2.6.2 Objective metrics of video quality ........................................................... 49 2.6.3 Weaknesses of video quality assessment techniques ............................... 56 2.7 A closer insight into an objective quality m etric ................................................ 57 2.8 IP video adaptation: from network friendly to mediafriendly ............................ 61 Contents 6 3 Quality aware adaptation in multi-stream sessions 64 3.1 Motivation .............................................................................................................. 65 3.2 Related w o r k ........................................................................................................ 68 3.2.1 Joint rate co n tro l ..................................................................................... 68 3.2.2 Integrated congestion control ................................................................... 70 3.3 Timescales of inter-stream adaptation ................................................................ 72 3.4 Content-aware quality adaptation m odel .............................................................. 75 3.5 Experimental results ........................................................................................... 78 3.5.1 Effect on quality smoothness ................................................................... 82 3.6 Chapter summary and discussion ......................................................................... 87 4 A Neural network predictor of quality 89 4.1 Motivation and problem description ................................................................... 90 4.1.1 Smooth quality video rate control- literature review ............................... 91 4.1.2 Constraints and requirements of smooth quality video streaming .... 94 4.2 Architecture of smooth quality live video streaming ........................................... 96 4.3 Artificial neural networks ...................................................................................... 98 4.3.1 Related w o rk .............................................................................................