Versatile Video Coding (VVC) – the New Standard and Its Capabilities

19/11/2020

Versatile Video Coding (VVC) – The New Standard and its Capabilities

Gary Sullivan Co-chair, ITU-T/ISO/IEC Joint Video Experts Team (JVET) Microsoft Research based primarily on slides prepared by Benjamin Bross, Fraunhofer Heinrich Hertz Institute, Berlin

Versatile Video Coding (VVC) Joint ITU-T (VCEG) and ISO/IEC (MPEG) project, completed July 2020 Rec. ITU-T H.266 & ISO/IEC 23090-3

Coding Efficiency Versatility

Rendered ”screen“ content coding ~50% saving over H.265/HEVC Adaptive resolution changes Independent sub-pictures emph. HD / UHD / 8K resolutions Tiles, slices and wavefronts emph. HDR / WCG Layered multistream & scalability emph. 10 bit Bitstream extraction and merging 360° video projection handling Random access & splicing features Gradual decoder refresh 2

1 19/11/2020

History of Video Coding Standards

Quality VVC HEVC AVC MPEG-2 / PSNR H.266 / MPEG-I H.265 / MPEG-H H.264 / MPEG-4 H.262 (dB) 41 (2020) (2013) (2003) (1995)

BasketballDrive (1920 x 1080, 50Hz) 33

31 bit rate 0 5 10 15 20 25 30 35 40 (Mbit/s)

VVC – Coding Efficiency Jevons Paradox

"The efficiency with which a resource is used tends to increase (rather than decrease) the rate of consumption of that resource."

Video is >80% of internet traffic (and rising)

2 19/11/2020

Video Coding Consumption 2018–2020

82% 12X 7X

of all Internet increase in AR/VR increase in video

traffic is now video traffic monitoring traffic

* Cisco Visual Networking Index: Forecast and Trends, 2018–2022 5

VVC – Timeline

2015 Oct. – Exploration Phase • Joint Video Exploration Team (JVET) of ITU-T VCEG and ISO/IEC MPEG established October ‘15 in Geneva • Joint Video Exploration Model (JEM) as software playground to explore new coding tools • 34% bitrate savings for JEM relative to HEVC provided evidence to start a new joint standardization activity with a…

2017 Oct. – Joint Call for Proposals (CfP) • Submit bitstreams and decoded video for proposed video coding technology • Compare submission with HEVC anchor for given sequences, bitrates and coding conditions

2018 Apr. – Development Phase • Subjective evaluation results of submitted CfP responses and HEVC anchor • Initial starting point of standard development

2020 Jul. – Final Standard

3 19/11/2020

VVC – Call for Proposals Results

• JVET received submissions from 32 organizations.

• 40% or more bitrate savings in terms of PSNR over HEVC were shown.

• All submissions were superior in terms of subjective quality than… • HEVC (in most test cases). • JEM (in a relevant number of test cases).

VVC – Call for Proposals Subjective testing result example Best performing HEVC anchor JEM (for this sequence) SunsetBeach (UHD, HLG) 10 9 8 7 6 ◇ 10 Mbit/s 5 △ 6 Mbit/s 4 3 ☐ 3 Mbit/s

2 – 1.5 Mbit/s Mean Opinion Score Mean (MOS) Opinion Score 1 0 JVET-J0080: “Results of Subjective Testing of Responses to the Joint CfP on Video Compression th Technology with Capability beyond HEVC”, 10 JVET Meeting,8 San Diego, April 2018

4 19/11/2020

VVC – Development Draft 1 and First Test Model (VTM-1.0)

• Start off with a clean slate

• Add quadtree plus multi-type tree block partitioning (QT+MTT) • Fundamental impact on all coding tools to be added • Most common partitioning scheme among all CfP submissions

• VVC Test Model (VTM) as reference implementation of VVC specification draft

• Test promising coding tools from CfP on that lean basis (efficiency / complexity aspects)

• Agree on adding tested coding tools until sufficient bitrate reduction is achieved

VVC – Development Draft 10 and VTM-10 - New coding tools for coding efficiency

• Quadtree with Multi-type Tree (QT+MTT) • Bi-prediction with CU weights (BCW)

• Separate Tree for Luma and Chroma (CST) • Decoder-side motion vector refinement (DMVR)

• Dependent Quantization (DQ) • Symmetric motion vector difference (SMVD)

• Joint coding of chrominance residuals (JCCR) • Sub-block transform (SBT)

• Multiple Transform Set (MTS) • Combined intra/inter prediction (CIIP)

• Low frequency non-separable transform (LFNST) • Multi-reference line intra prediction (MRL)

• Adaptive Loop Filter (ALF) • Intra block copy (IBC)

• Affine Motion Compensation • Intra sub-partitioning (ISP)

• Subblock-based Temporal Merging Candidates • Matrix based intra prediction (MIP)

• Adaptive motion vector resolution (AMVR) • Cross-component Linear Model (CCLM)

• Geometric partition mode (GPM) • Luma mapping with chroma scaling (LMCS)

• Bi-directional optical flow (BDOF) • Transform Skip Residual Coding (TSRC)

• Merge with MVD (MMVD) • Quantized residual DPCM …

• Adaptive color transform (ACT) 10

5 19/11/2020

VVC – Coding Efficiency VVC reference software (VTM) vs. HEVC reference software (HM)

40 15 38 40 40 14 Even higher than 35 38 39 13 36 12 34 HEVC vs. AVC 30 28 32 11 10 25 10.0 9 9.7 9.4 9.1 8.8 8 20 8.0 7 Subjective gains seem a 7.4 YUV BD-Rate savings 6 15 Enc. Speed 5 bit higher than this 5.2 4 10 11 Dec. Speed

Bit rate reduction [%] rate reduction Bit 3.6 3 5 2.2 2 0.8 1.3 1.8 1.9 1.8 2.0 1.9 1 / increase runtime Complexity 0 1.3 1.5 1.6 0 11

VVC – Coding Efficiency VVC reference software (VTM) vs. HEVC reference software (HM)

UHD SDR subjective verification testing Oct 2020

Subjective gain • ~43 % for VTM enc reference software • ~49 % for VVenC (>100x faster,

12 subjective opt.)

6 19/11/2020

VVC – Versatility

Screen content coding (SCC)

Application Problem Solution

Gaming, screen sharing / • Video codecs typically • Special screen content remote desktop,… optimized for camera coding tools

captured video (different • HEVC v4 SCC extensions -> signal characteristics) not in main profile!

• Reduces coding efficiency • VVC supports SCC already for screen content in v1 Main profile Arena of Valor (1080p60)

VVC – Versatility

Adaptive resolution change

Application Problem Solution

Adaptive streaming with • Different resolution • Resample different resolution resolution switching reference pictures cannot reference picture

be used in inter-picture • VVC supports reference prediction picture resampling (RPR)

• Reduces coding efficiency • RPR as enabler for spatial at switching points scalability in VVC v1

7 19/11/2020

VVC – Versatility

Tile-based streaming

Application Problem Solution

Tiled streaming of 360-degree • Managing a decoder • More efficient coding of videos sample budget dynamically independent sub-pictures

post-encoding (in-picture padding) degree ERP ERP degree

Full360 • •

video Throwing 24K video (tiles) Flexible block addressing for Preferred at a 4K decoder easier extraction and viewing direction merging of sub-pictures 1

VVC – Profiles Multi-layer Main 10 4:4:4 (Six, logically related) Multi-layer Main 10 Main 10 4:4:4 Supports layers for scalability, multiview, Supports additional chroma formats and alpha planes, etc. coding tools 1) Main 10 profile: monochrome and 4:2:0, 8 to 10 bits, 1 layer Main 10 Supports 4:2:0 & monochrome, up to 10 bits 2) Main 10 Still Picture profile: Main 10 Still Picture based on the Main 10 profile, 1 picture only Main 10 4:4:4 Still Picture 3) Main 10 4:4:4 profile: based on Main 10 profile, also supports 4:2:2 and 4:4:4 Figure originally by Virginie Drugeon of Panasonic 4) Main 10 4:4:4 Still Picture profile: based on the Main 10 4:4:4 profile, 1 picture only 5) Multilayer Main 10 profile, based on the Main 10 profile, >= 1 layer 6) Multilayer Main 10 4:4:4 profile, based on the Main 10 4:4:4 profile, >= 1 layer

8 19/11/2020

Versatile Video Coding (VVC) Summary

• Coding Efficiency – VVC Test Model 6.1 over HEVC (HM) H.264 / AVC • ~50% lower bit rate for equal subjective quality HD and UHD (40% for PSNR) • ~9x encoder and 1.6x decoder runtime • Versatility – enabled by: H.265 / HEVC • Screen content coding tools (gaming, screen sharing,…) • Reference picture resampling (adaptive streaming) • Spatial scalability using RPR filters, also temporal, view, quality scalability • Independent sub-pictures (360 video, ROI) H.266 / VVC • Boundary handling for 360°video & gradual decoder refresh • Bitstream extraction & merging • Random access & splicing features • Final Standard established July 2020 17

Thank you very much!

Further Information:

[email protected] [email protected] jvet.hhi.fraunhofer.de