Step Response Metric for Video Encoder Rate Control Characterisation NOSSDAV 2016 Ralf Globisch and Keith Ferguson (CSIR Meraka Institute) Yago Sanchez and Thomas Schierl (Fraunhofer HHI)

1 Contents

Outline • Motivation • Step Response Metric • Experimental Setup • Results • Conclusion & future work

2 Globisch et al Step Response Metric For Video Encoder Rate Control Characterisation NOSSDAV2016 Real-Time Streaming Application

Target rate

Sender Network

Cooperative Function

RC

Feedback Live Source Encoder Transmission

3 Globisch et al Step Response Metric For Video Encoder Rate Control Characterisation NOSSDAV2016 Motivation

• Real-Time Streaming Protocols – RTP/RTCP, RTSP, SIP – Non HTTP-based adaptive streaming • Video Conferencing: WebRTC, Skype, WeChat – Real-Time Communication in WEB-browsers RTCWeb1 • Video and audio in the browser – RTP Media Congestion Avoidance Techniques RMCAT2 • Receiver feedback results in rate configuration • GCC, NADA, Scream • CC dependent on Codec Rate Control and access network characteristics [1] https://tools.ietf.org/wg/rtcweb/ [2] https://tools.ietf.org/wg/rmcat/ 4 Globisch et al Step Response Metric For Video Encoder Rate Control Characterisation NOSSDAV2016 Goal

• Develop a metric that allows characterisation of encoder rate control algorithms • Metric should be resolution/framerate independent • Different configuration options of a single rate control implementation can also be characterised. • The goal is *NOT* to compare and rank different rate control implementations e.g. “ is better than OpenH264”

5 Globisch et al Step Response Metric For Video Encoder Rate Control Characterisation NOSSDAV2016 Rate Normalisation

• Bits per pixel (bpp) • bitrate_kbps = width * height * fps * bpp • Define that an encoder is under stress when using 0.035 bpp, not under stress when using 0.140 bpp. – For CIF at 30 fps = 106 – 426 kbps – For 720p HD at 25 fps = 806 – 3226 kbps – For 720p HD at 30 fps = 968 – 3870 kbps

6 Globisch et al Step Response Metric For Video Encoder Rate Control Characterisation NOSSDAV2016 Step Response Metric: Model

t0

R0

1 (� − � ) �

R1

�

()/ � = � + � − � �

7 Globisch et al Step Response Metric For Video Encoder Rate Control Characterisation NOSSDAV2016 Step Response Metric: Decay Function

()/ • � = � + � − � �

∑( ) • � = − | | ∑ .()

8 Globisch et al Step Response Metric For Video Encoder Rate Control Characterisation NOSSDAV2016 Step Response Metric: Rate Cost

• � = (∑ [� � − � ]) ( ∑ [ ]) • RCR=

9 Globisch et al Step Response Metric For Video Encoder Rate Control Characterisation NOSSDAV2016 Experimental Setup

• YUV420P sequences – CIF • Soccer (30 fps) • Foreman (30 fps) • Mobile (30 fps) • Four People (30 fps, cropped & scaled from original) – Base for testing rate normalisation – 720p • Four People (30fps) • Mobile Calendar (25fps)

10 Globisch et al Step Response Metric For Video Encoder Rate Control Characterisation NOSSDAV2016 Experimental Setup (Cntd.)

• Codecs – x264 (git clone http://git.videolan.org/git/x264.git) • preset=ultrafast, tune=zerolatency – OpenH264 v1.5 (https://github.com/cisco/openh264) • CAMERA_VIDEO_REAL_TIME – VPP H264v2 (https://sourceforge.net/projects/videoprocessing/ https://github.com/CSIR-RTVC/video-processing-project) • Log (mode=2) • Pow (mode=2) • Cbr (mode=1) – Frame-skipping disabled! – Automatic I-frame generation disabled • Various switching points tested

11 Globisch et al Step Response Metric For Video Encoder Rate Control Characterisation NOSSDAV2016 Rate Plots: Four People (720p)

0.3000 VPP-Cbr

0.2500 x264

Rate (bpp) VPP-Pow 0.2000 VPP-Log

0.1500 OpenH264

0.1000

0.0500

0.0000 0.000 1.000 2.000 3.000 4.000 5.000 6.000 7.000 8.000 9.000 10.000

Time (sec)

12 Globisch et al Step Response Metric For Video Encoder Rate Control Characterisation NOSSDAV2016 Rate Model: Four People (720p)

0.2000 VPP-Cbr 0.1800 x264 0.1600 VPP-Pow Rate (bpp) 0.1400 VPP-Log

0.1200 OpenH264

0.1000

0.0800

0.0600

0.0400

0.0200

0.0000 0.000 1.000 2.000 3.000 4.000 5.000 6.000 7.000 8.000 9.000 10.000

Time (sec)

13 Globisch et al Step Response Metric For Video Encoder Rate Control Characterisation NOSSDAV2016 Rate Plot: Mobile (CIF)

0.3000 VPP-Cbr x264 0.2500 VPP-Pow

Rate (bpp) VPP-Log OpenH264 0.2000

0.1500

0.1000

0.0500

0.0000 0.000 1.000 2.000 3.000 4.000 5.000 6.000 7.000 8.000 9.000 10.000

Time (sec)

14 Globisch et al Step Response Metric For Video Encoder Rate Control Characterisation NOSSDAV2016 Rate Model: Mobile (CIF)

0.2000 VPP-Cbr 0.1800 x264

0.1600 VPP-Pow Rate (bpp) 0.1400 VPP-Log OpenH264 0.1200

0.1000

0.0800

0.0600

0.0400

0.0200

0.0000 0.000 1.000 2.000 3.000 4.000 5.000 6.000 7.000 8.000 9.000 10.000

Time (sec)

15 Globisch et al Step Response Metric For Video Encoder Rate Control Characterisation NOSSDAV2016 Results: Summary of Encoder Time Constants and Total Rate Cost Ratio

• Mid point switch

Soccer Foreman Mobile Four People CIF Four People HD Calendar HD Encoder Tau (sec) Rate Cost Tau (sec) Rate Cost Tau (sec) Rate Cost Tau (sec) Rate Cost Tau (sec) Rate Cost Tau (sec) Rate Cost Ratio Ratio Ratio Ratio Ratio Ratio

x264 0.29 3.21 0.26 3.14 0.16 2.95 0.32 2.71 0.28 2.68 0.27 2.41 OpenH264 11.07 2.52 0.35 2.21 -35.89 3.55 0.33 0.42 0.04 0.08 0.04 0.18 VPP-Log 0.60 1.59 0.27 2.15 0.23 0.72 0.17 1.13 0.19 1.04 0.25 0.95 VPP-Pow 0.58 1.62 0.22 0.85 0.12 0.00 0.09 -0.70 0.08 -0.57 0.04 -0.23 VPP-Cbr 0.01 -0.06 0.02 -0.02 0.01 -0.01 0.02 -0.02 0.01 -0.01 0.01 -0.01 • Third point switch

Soccer Foreman Mobile Four People HD Encoder Tau (sec) Rate Cost Tau (sec) Rate Cost Tau (sec) Rate Cost Tau (sec) Rate Cost Ratio Ratio Ratio Ratio

x264 0.29 2.48 0.26 2.23 0.26 1.59 0.41 1.53 OpenH264 15.21 2.41 0.24 0.17 -33.76 3.78 0.07 -0.06 VPP-Log 1.96 2.29 0.27 0.41 0.31 0.70 0.42 0.57 VPP-Pow 0.78 1.30 0.19 0.02 0.26 0.12 0.19 -0.53 VPP-Cbr 0.02 -0.02 0.01 -0.01 0.01 -0.02 0.01 -0.01

16 Globisch et al Step Response Metric For Video Encoder Rate Control Characterisation NOSSDAV2016 Results: Transition Encoder Response

All Encoder Tau (sec) Rate Cost Ratio

x264 0.27 2.47 OpenH264 0.15 0.51 VPP-Log 0.50 1.16 VPP-Pow 0.27 0.29 VPP-Cbr 0.01 -0.02

17 Globisch et al Step Response Metric For Video Encoder Rate Control Characterisation NOSSDAV2016 Steady State Encoder Rate-Distortion

• High Rate • A Soccer Foreman Mobile Four People HD Calendar HD Encoder Average Encoder Rate PSNR (dB) Rate PSNR (dB) Rate PSNR (dB) Rate PSNR (dB) Rate PSNR (dB) Rate PSNR (dB) (bpp) (bpp) (bpp) (bpp) (bpp) (bpp) •x264 A 0.136 31.27 0.144 32.43 0.140 21.58 0.140 40.40 0.138 29.12 0.139 30.96 OpenH264 0.140 33.32 0.141 34.82 0.190 27.32 0.139 40.21 0.138 32.67 0.150 33.67 •VPP-LogA 0.149 30.26 0.145 34.27 0.141 25.34 0.141 40.82 0.141 32.00 0.143 32.54 VPP-Pow 0.146 30.21 0.148 34.38 0.142 25.26 0.142 40.98 0.145 32.29 0.144 32.63 •VPP-CbrA 0.139 30.02 0.139 34.21 0.139 25.24 0.139 40.85 0.139 32.20 0.139 32.50 • Low Rate Soccer Foreman Mobile Four People HD Calendar HD Encoder Average Encoder Rate PSNR (dB) Rate PSNR (dB) Rate PSNR (dB) Rate PSNR (dB) Rate PSNR (dB) Rate PSNR (dB) (bpp) (bpp) (bpp) (bpp) (bpp) (bpp) x264 0.035 27.03 0.033 25.79 0.034 18.84 0.034 37.11 0.034 28.35 0.034 27.43 OpenH264 0.114 31.83 0.048 29.51 0.156 27.01 0.035 38.28 0.050 32.05 0.081 31.74 VPP-Log 0.073 26.59 0.043 23.78 0.045 19.90 0.033 37.47 0.065 25.71 0.052 26.69 VPP-Pow 0.071 26.59 0.043 23.77 0.045 19.82 0.035 37.69 0.065 25.68 0.052 26.71 VPP-Cbr 0.033 22.51 0.034 22.73 0.035 19.67 0.035 37.86 0.034 22.66 0.034 25.08

18 Globisch et al Step Response Metric For Video Encoder Rate Control Characterisation NOSSDAV2016 Results: Steady-state RD considerations

High Rate Low Rate Encoder Rate PSNR (dB) Rate PSNR (dB) (bpp) (bpp) x264 0.139 30.96 0.034 27.43 OpenH264 0.150 33.67 0.081 31.74 VPP-Log 0.143 32.54 0.052 26.69 VPP-Pow 0.144 32.63 0.052 26.71 VPP-Cbr 0.139 32.50 0.034 25.08

Table 6: Averaged Steady State Rate-Distortion

19 Globisch et al Step Response Metric For Video Encoder Rate Control Characterisation NOSSDAV2016 Conclusion: Summary

• Binary composite metric • Allows characterisation and evaluation of encoder rate control algorithms and/or configurations in terms of duration and rate cost. • Trade-off between quality and meeting rate constraints.

20 Globisch et al Step Response Metric For Video Encoder Rate Control Characterisation NOSSDAV2016 Conclusion: Why is this useful

• Assists system designers/implementors to make informed decisions on codec selection and configuration as well as access network • Allows determination of usefulness of switch in real-time communication/stream: when/how often • Helpful in determining bugs or improving rate control

21 Globisch et al Step Response Metric For Video Encoder Rate Control Characterisation NOSSDAV2016 Conclusion & Future Work

• Code & Scripts – https://github.com/CSIR-RTVC/CodecStepResponse – Latest VPP H264v2 to be released • Evaluate rate adaptation • Add VP8 & 9 codec support

22 Globisch et al Step Response Metric For Video Encoder Rate Control Characterisation NOSSDAV2016 Thank you

23 Name ([email protected]) MEME-B Mobile IPTV Platform

rtsp://news.mm1

Media Stripped social App messaging media Server

User interaction Analysis Media Activity viewing Web INTERNET Server Remote database stats Transcoder Server Client 1

Remote channel Client 2

Channel Studio channel viewing Client 3

Satellite channel Client 4

Globisch et al Step Response Metric For Video Encoder Rate Control Characterisation NOSSDAV2016 24 Playlist channel Transcoder Servers Media Switching Servers