Video Quality of Experience Score

A Sandvine Technology Showcase

Contents Executive Summary Third-party video applications that run over-the-top (OTT) of a Executive Summary ...... 1 communications service provider’s (CSP) transport layer are the dominant drivers of bandwidth on the Internet. Introduction to Internet Video Quality of Experience ...... 2 The popularity of such video is responsible for two fundamental Sandvine’s Video QoE Metric ...... 3 shifts in consumer behavior: higher peak bandwidth levels and heightened subscriber sensitivity to video quality. Foundational Capabilities ...... 3 Session Awareness and Sampling Frequency 3 Since 2010, Sandvine has provided CSPs with insight into video QoE that meets all of the requirements to measure quality from Overcoming Routing Asymmetry ...... 3 the perspective of the end viewer: Video QoE Measurements ...... 3 Our platform is completely session aware and resolves Progressive Video QoE ...... 4  routing asymmetry, so video QoE is based on complete Adaptive Video QoE ...... 5 visibility of the video content Conclusion ...... 6  The QoE metric takes into account both display quality and transport quality, the two necessary dimensions of Characteristics of Sandvine Video QoE ...... 6 video QoE  Different factors are used when calculating the QoE for progressive video or adaptive video, to account for the

different behavior of these streaming mechanisms Video Quality of Experience Score

Introduction to Internet Video Quality of Experience Third-party video applications that run over-the-top (OTT) of a communications service provider’s (CSP) transport layer are the dominant drivers of bandwidth on the Internet.1 The popularity of such video is responsible for two fundamental shifts in consumer behavior: higher peak bandwidth levels and heightened subscriber sensitivity to video quality.

During periods of peak utilization, resources on the network are more prone to congestion; from the viewers’ perspective, congestion can very visibly manifest as degradation in video streaming quality. Therefore, measuring the customer’s quality of experience (QoE) requires measurement of video QoE.

OTT video is delivered in two primary streaming mechanisms: progressive video, in which sections of a single file are delivered in bursts; and adaptive video, in which chunks of differing display quality are delivered based upon the network’s transport capabilities.

From a transport layer perspective, HTTP over TCP is the dominant transport mechanism, introducing two primary dimensions to video quality: display quality (fidelity) and transport quality (stalling).

For a representative assessment of Internet video QoE from the viewer’s perspective, the video QoE solution must use different methods to measure QoE of progressive video and QoE of adaptive video, and must take into account both display quality and transport quality.2

1 Information about worldwide traffic composition is available from several good sources, including Cisco’s Visual Networking Index (VNI), Akamai’s State of the Internet studies, and Sandvine’s own Global Internet Phenomena Reports 2 A detailed explanation of adaptive video streaming, progressive video streaming, and the considerations for measuring Internet video quality of experience are available in the Sandvine whitepaper Measuring Internet Video Quality of Experience from the Viewer’s Perspective

Page 2 Video Quality of Experience Score

Sandvine’s Video QoE Metric Since 2010, Sandvine has provided CSPs with insight into video quality of experience, from the subscriber perspective, with our video QoE score. This section explains how that score is created. Foundational Capabilities To properly measure video QoE, a solution must meet some basic criteria.

Session Awareness and Sampling Frequency Video sessions are typically long-lived3, and an accurate assessment of the viewer’s quality of experience is only possible if the necessary measurements are taken throughout the full duration. However, measurements taken too frequently will incur processing overhead with diminishing returns; measurements taken too infrequently will fail to accurately capture the quality of experience and may fall prey to a sampling error4.

Sandvine’s video QoE metric is based on measurements taken every 15 seconds, throughout the entire lifetime of a video session. As a result, there are multiple samples even for short videos, and an accurate assessment can be built of the viewer’s experience for the entire flow.

The measurements for each video are made by the Sandvine Policy Traffic Switch (PTS), and reported as they are made to the Service Delivery Engine (SDE) which performs the calculations to determine the QoE score for the entire flow.5

Since both the PTS and SDE are completely session aware, they perform state tracking for a video asset delivered across multiple HTTP GET transactions issued into the same, or different, TCP connections.

Overcoming Routing Asymmetry For the purposes of measuring video QoE, all routing asymmetry must be resolved – this is the only way in which the full video stream can be seen, which is the only way that quality of experience can be determined with any accuracy.

Sandvine’s solution (as a whole, not just for video quality of experience) completely resolves network asymmetry from the perspective of our packet-processing and policy control applications.6 Video QoE Measurements From a transport layer perspective, HTTP over TCP is the dominant transport mechanism for Internet video. As a consequence, there are two primary dimensions to video quality:

1. Display quality (fidelity): is the image quality sufficient for the device’s screen size? 2. Transport quality (stalling): how long does the video take to start, and does it play smoothly?

3 Even for videos where this statement is not necessary true in the strictly absolute sense (e.g., for a three-minute YouTube clip), it is still valid in the relative sense; that is, even a three-minute video is much longer than most Internet sessions and will likely be delivered in many bursts (progressive) or chunks (adaptive) 4 Intrepid readers can learn more at: http://en.wikipedia.org/wiki/Nyquist_rate#Nyquist_rate_relative_to_sampling 5 More information about the PTS and SDE, and their capabilities, is available at www.sandvine.com 6 To learn how we achieve this, regardless of network type, complexity, and scale, please read the technology showcase Policy Traffic Switch Clusters: Overcoming Routing Asymmetry and Achieving Scale

Page 3 Video Quality of Experience Score

To provide a representative assessment of QoE from the viewer’s perspective, Sandvine’s video QoE metric takes into account both of these dimensions, and accommodates the distinct behavior of both progressive and adaptive video.

Sandvine scores video QoE on a scale of 0 (minimum) to 5 (maximum). The QoE calculation itself uses equal weighting for the metrics being used, which differ for adaptive video and progressive video.

Progressive Video QoE To create a video QoE metric for progressive video streaming (and to provide additional business intelligence), Sandvine’s solution extracts information from each layer of the video stream:

 IP: Subscriber, CDN, BGP AS Path  Subscriber: physical location on network, service plan, device type  TCP: no certainty in relation to QoE  HTTP: asset (used to link multiple chunks together), duration, stall information (transport quality  Container: codec, resolution, bitrate (display quality), CDN  Elementary stream: bytes transferred This information is combined with on-the-wire measurements (e.g., number of buffer stall events, buffer stall duration, server response latency) to provide the elements of the video QoE metric.

The display quality can be extracted from the container. If desired, this information can be compared to characteristics including the device type in order to determine if the resolution is appropriate.

To measure both the duration and the transport quality of progressive video, the Sandvine solution dynamically monitors the seek actions for all HTTP video transactions (i.e., not just the initial GET on an HTTP video flow) that correspond to either the user moving a control (i.e., selecting a different time in the video) or the video client re-buffering due to a stall. By tracking these seek actions and measuring video flow on-the-wire, we are able to detect stalls that are caused by buffer under-runs and keep those distinct from re-buffering events that result from a user-initiated seek action. An illustration of this technique is shown in Figure 1.

Figure 1 - Measuring a buffer stall in a progressive video stream

Page 4 Video Quality of Experience Score

Adaptive Video QoE To create a video QoE metric for adaptive video streaming (and to provide additional business intelligence), Sandvine’s solution extracts information from each layer of the video stream:

 IP: Subscriber, CDN, BGP AS Path  Subscriber: physical location on network, service plan, device type  TCP: no certainty in relation to QoE  HTTP: asset (used to link multiple chunks together), ‘protocol’, CDN  Protocol: duration, stall information (transport quality)  Container: codec, resolution, bitrate (display quality)  Elementary stream: bytes transferred This information is combined with on-the-wire measurements (e.g., number of buffer stall events, buffer stall duration, server response latency, bitrate transition events) to provide the necessary building blocks of video quality of experience.

The key distinguishing measurement for adaptive video is counting the number of bitrate transition events (i.e., the number of upshifts and downshifts in display quality).

Page 5 Video Quality of Experience Score

Conclusion Video, particularly video from over-the-top content providers, is coming to dominate the global Internet. This video is primarily delivered in two ways: progressive streaming, in which sections of a single file are delivered in bursts; and adaptive video, in which chunks of differing display quality are delivered based upon the network’s transport capabilities.

These different delivery mechanisms necessitate different mechanisms to measure video quality of experience, but in both cases there are two dimensions that must be measured separately and then considered together: display quality and transport quality.

For any video QoE measurement to be possible, there are additional conditions that must be met. For instance, the solution must be completely session aware and must not be deployed in such a manner that routing asymmetry prevents it from seeing all the traffic associated with a single video session. Characteristics of Sandvine Video QoE Sandvine provides a video quality of experience solution that meets all of these conditions:

 Our platform is completely session aware and resolves routing asymmetry  The QoE metric takes into account both display quality and transport quality  Different factors are used when calculating the QoE for progressive video or adaptive video A further explanation of our capabilities is provided in the table below.

Consideration Characteristic of Sandvine Solution Explanation Measures the QoE from the perspective of the viewer, rather than transmission- The viewer’s experience is most focused measurements like signal-to-noise important ratio There are two dimensions to Measures both display quality and video quality that in transport quality, and integrates both into combination determine the QoE: the QoE metric display quality and transport QoE Metric quality These metrics by themselves Does not use any TCP jitter or packet loss offer a poor correlation to actual video QoE Progressive streaming and Is calculated using different factors for adaptive streaming are progressive streams and adaptive streams fundamentally and profoundly different in behavior Measurements are made throughout the Video sessions are generally full duration of a video session long-lived, with variable quality Sampling must be optimized to give accurate results without The measurement frequency is sufficient to onerous performance overhead; avoid sampling error practically, sampling at 15 Sessions second or 30 second intervals is appropriate Correlates delivery of the same asset What looks like a single video to across multiple HTTP GET transactions the end user is actually split issued into the same, or different, TCP within and across TCP connections connections that must all be

Page 6 Video Quality of Experience Score

considered as a whole Is deployed in such a manner that it sees each video flow in a perfectly symmetrical To compile an accurate metric Routing Asymmetry manner; that is, routing asymmetry is of video QoE, the entire video completely resolved from the perspective delivery must be considered of the measuring device Keeps track of seek actions (whether Required to both measure the initiated by the user or the video client) duration of a progressive video dynamically for all HTTP transactions flow and to count buffer events Progressive Video QoE The container holds the display Extracts display quality information from quality information, which is a the video container required dimension of video streaming QoE Required to measure the Is protocol aware transport quality The container holds the display Extracts display quality information from quality information, which is a the video container required dimension of video Adaptive Video QoE streaming QoE Since the display quality (and Is not based in any part on transferred bitrate) varies dynamically, the bytes number of bytes transferred is irrelevant

Page 7

Headquarters European Offices Copyright ©2015 Sandvine Sandvine Incorporated ULC Sandvine Limited Incorporated ULC. Sandvine and Waterloo, Ontario Canada Basingstoke, UK the Sandvine logo are registered Phone: +1 519 880 2600 Phone: +44 0 1256 698021 trademarks of Sandvine Incorporated Email: [email protected] Email: [email protected] ULC. All rights reserved.