Versatile Video Coding (VVC) Delivers: Coding Efficiency and Beyond
Total Page:16
File Type:pdf, Size:1020Kb
25/03/2021 Versatile Video Coding (VVC) Delivers: Coding Efficiency and Beyond Gary Sullivan Co-chair, ITU-T/ISO/IEC Joint Video Experts Team (JVET) Microsoft Research based primarily on slides prepared by Benjamin Bross, Fraunhofer Heinrich Hertz Institute, Berlin 2021 Data Compression Conference (DCC) 25 March 2021 1 Versatile Video Coding (VVC) Joint ITU-T (VCEG) and ISO/IEC (MPEG) project, completed July 2020 Rec. ITU-T H.266 & ISO/IEC 23090-3 Coding Efficiency Versatility Rendered ”screen“ content coding ~50% saving over H.265/HEVC Adaptive resolution changes Independent sub-pictures emph. HD / UHD / 8K resolutions Tiles, slices and wavefronts emph. HDR / WCG Layered multistream & scalability emph. 10 bit Bitstream extraction and merging 360° video projection handling Random access & splicing features Gradual decoder refresh 2 1 2 25/03/2021 History of Video Coding Standards Quality VVC HEVC AVC MPEG-2 / PSNR H.266 / MPEG-I H.265 / MPEG-H H.264 / MPEG-4 H.262 (dB) 41 (2020) (2013) (2003) (1995) 39 37 35 BasketballDrive (1920 x 1080, 50Hz) 33 31 bit rate 0 5 10 15 20 25 30 35 40 (Mbit/s) 3 3 VVC – Coding Efficiency Jevons Paradox "The efficiency with which a resource is used tends to increase (rather than decrease) the rate of consumption of that resource." Video is 80% of internet traffic (and rising) 4 2 4 25/03/2021 Video Coding Consumption 2018–2020 80% 12X 7X of all Internet increase in AR/VR increase in video traffic is now video traffic monitoring traffic * Cisco Visual Networking Index: Forecast and Trends, 2017–2022 (2018) 5 5 VVC – Timeline 2015 Oct. – Exploration Phase • Joint Video Exploration Team (JVET) of ITU-T VCEG and ISO/IEC MPEG established October ‘15 in Geneva • Joint Video Exploration Model (JEM) as software playground to explore new coding tools • 34% bitrate savings for JEM relative to HEVC provided evidence to start a new joint standardization activity with a… 2017 Oct. – Joint Call for Proposals (CfP) • Submit bitstreams and decoded video for proposed video coding technology • Compare submission with HEVC anchor for given sequences, bit rates and coding conditions 2018 Apr. – Development Phase • Subjective evaluation results of submitted CfP responses and HEVC anchor • Initial starting point of standard development 2020 Jul. – Final Standard (VVC, also Versatile 6Supplemental Enhancement Information – VSEI) 3 6 25/03/2021 VVC – Call for Proposals Results • JVET received submissions from 32 organizations. • 40% or more bit rate savings in terms of PSNR over HEVC were shown. • All submissions were superior in terms of subjective quality than… • HEVC (in most test cases) • JEM (in a relevant number of test cases) 7 7 VVC – Call for Proposals Subjective testing result example Best performing HEVC anchor JEM (for this sequence) SunsetBeach (UHD, HLG) 10 9 8 7 6 ◇ 10 Mbit/s 5 △ 6 Mbit/s 4 3 ☐ 3 Mbit/s 2 – 1.5 Mbit/s Mean Opinion (MOS) Score 1 0 JVET-J0080: “Results of Subjective Testing of Responses to the Joint CfP on Video Compression th Technology with Capability beyond HEVC”, 10 JVET Meeting,8 San Diego, April 2018 4 8 25/03/2021 VVC – Development Draft 1 and First Test Model (VTM-1.0) • Started off with a clean slate • Based on quadtree plus multi-type tree block partitioning (QT+MTT) • Fundamental impact on all coding tools to be added • Most common partitioning scheme among all CfP submissions • VVC Test Model (VTM) as reference implementation • Test promising coding tools from CfP on that lean basis (efficiency / complexity aspects) • Agree on adding tested coding tools until sufficient bitrate reduction is achieved 9 9 VVC – New Coding Tools (beyond HEVC v1) Block Segmentation Intra Prediction MV Pred & Deltas • Quadtree with Multi-Type • 93 prediction angles! • History-based MVP Tree (65 for each block size) • Pair-wise Average MVP • Larger max CU size • 4-tap interpolation filters Candidate • Sub-Picture Segmentation • Position-Dependent • Subblock-Based Temporal • Chroma Separate Tree Prediction Combination MVP • Local Dual Tree • Multiple Reference Lines • Adaptive MV Resolution • Virtual Pipeline Data Units • Matrix-based Intra Pred. • Merge with MVD • Cross-Component Linear • Symmetric MVD Model • Intra Sub-Partitions 10 5 10 25/03/2021 VVC – New Coding Tools (beyond HEVC v1) Inter Prediction Transforms & Quant Entropy Coding • Geometric Partition Mode • Larger & Non-square • Multi-Hypothesis Probability • Reference Picture Transforms Estimation Resampling • Multiple Transform • Additional Coefficient • Bipred with CU Weights Selection Group Sizes • Combined Intra/Inter Pred • Low-Frequency Non- • Single Reverse Diagonal Separable Transform Scan Order • Decoder MV Refinement • Sub-Block Transform • Improved Probability Model • Affine Motion • Dependent Quantization Selection for Absolute Levels • Prediction Refinement with Optical Flow • Joint Coding of Chroma Residuals • Bi-Directional Optical Flow • Local Chroma11 QP Offset 11 VVC – New Coding Tools (beyond HEVC v1) In-Loop Filtering Screen Content Tools 360°Coding Tools • Luma Mapping with • Block-Level DPCM • MV Wrap-Around Chroma Scaling • Transform-Skip Residual • Virtual Boundaries • Deblocking Boundary Coding Handling Modifications • Intra Block Copy • Deblocking Long Filter • Palette Mode • Deblocking Strong • Adaptive Color Transform Chroma Filter • Luma-Adaptive Deblocking • Adaptive Loop Filter • Cross-Component ALF 12 6 12 25/03/2021 VVC – Coding Efficiency VVC reference software (VTM) vs. HEVC reference software (HM) 40 41 15 38 40 40 14 Even higher than 38 39 13 35 36 34 12 HEVC vs. AVC 30 28 32 11 10 25 10.0 9 9.7 9.4 9.1 8.8 8 20 7 Subjective gains YUV BD-Rate savings 8.0 7.4 7.4 6 15 Enc. runtime 5 seem a bit higher 11 5.2 4 10 Dec. runtime Bitrate reduction [%] reduction Bitrate 3.6 3 5 2.2 2 0.8 1.3 1.8 1.9 1.8 2.0 1.9 1 Complexity / increase runtime * Additional gain in VTM-11 is due to 0 1.3 1.5 1.6 1.6 0 GOP config. GOP32 is not supported 13 in HEVC when operating at high resolution. MCTF is used in both configs for VTM 11 comparison. 13 VVC – Coding Efficiency VVC reference software (VTM) vs. HEVC reference software (HM) UHD SDR subjective verification testing Oct 2020 Subjective gain • ~43 % for VTM enc reference software • ~49 % for VVenC (>100x faster, 14 subjective opt.) 7 14 25/03/2021 VVC – Versatility Screen content coding (SCC) Application Problem Solution Gaming, screen sharing / • Video codecs typically • Special screen content remote desktop,… optimized for natural video coding tools (different signal • HEVC v4 SCC extensions -> characteristic) not in main profile! • Reduces coding efficiency • VVC supports SCC already for screen content in v1 Main profile Arena of Valor (1080p60) 15 15 VVC – Versatility Adaptive resolution change Application Problem Solution Adaptive streaming with • Different resolution • Resample different resolution resolution switching reference pictures cannot reference picture be used in inter-picture • VVC supports reference prediction picture resampling (RPR) • Reduces coding efficiency • RPR as enabler for spatial at switching points scalability in VVC v1 16 8 16 25/03/2021 VVC – Versatility Tile-based streaming Application Problem Solution Tiled streaming of 360-degree • Managing a decoder • More efficient coding of videos sample budget dynamically independent sub-pictures post-encoding (in-picture padding) ereERP degree Full360 • • video Throwing 24K video (tiles) Flexible block addressing for Preferred at a 4K decoder easier extraction and viewing direction merging of sub-pictures 1 17 17 VVC – Profiles Multi-layer Main 10 4:4:4 (Six, logically related) Multi-layer Main 10 Main 10 4:4:4 Supports layers for scalability, multiview, Supports additional chroma formats and alpha planes, etc. coding tools 1) Main 10 profile: monochrome and 4:2:0, 8 to 10 bits, 1 layer Main 10 Supports 4:2:0 & monochrome, up to 10 bits 2) Main 10 Still Picture profile: Main 10 Still Picture based on the Main 10 profile, 1 picture only Main 10 4:4:4 Still Picture 3) Main 10 4:4:4 profile: based on Main 10 profile, also supports 4:2:2 and 4:4:4 Figure originally by Virginie Drugeon of Panasonic 4) Main 10 4:4:4 Still Picture profile: based on the Main 10 4:4:4 profile, 1 picture only 5) Multilayer Main 10 profile, based on the Main 10 profile, >= 1 layer 6) Multilayer Main 10 4:4:4 profile, based on the Main 10 4:4:4 profile, >= 1 layer 18 9 18 25/03/2021 Early Deployments (cf. JVET-V0021) Public Source Code Software Decoders Encoder Prod & Svcs • JVET VTM reference • Sharp JVET-S0224 (4K • KDDI Research (4K@60 fps) software decode more than 60 fps) • Ateme Titan (with initial • InterDigital JVET-T0061 • Tencent O266dec (with OTT channel Nov 2020) multithreaded decoder mobile platform support) • Fraunhofer HHI VVenC • Fraunhofer HHI VVenC • Alibaba Ali266 (for mobile (prev. column) encoder platforms) • Bitmovin, based on VVenC • Fraunhofer HHI VVdeC • VTM, Interdigital, and decoder Fraunhofer HHI • FAU bitstream analyser (prev. column) 19 19 Technical Support (cf. JVET-V0021) Bitstream Analysers Conformance Testing • Elecard (April 2020) • JVET Conformance Testing • ViCueSoft (late 2020) standard • FAU software (prev. slide) • AllegroDVT (Jan 2020) 20 10 20 25/03/2021 Versatile Video Coding (VVC) Summary • Coding Efficiency – VVC Test Model 11 over HEVC (HM) H.264 / AVC • ~41% PSNR-based bit rate reduction for HD and UHD • ~7.4x encoder and 1.6x decoder runtime • Versatility – enabled by: • Screen content coding tools (gaming, screen sharing,…) H.265 / HEVC • Reference picture resampling (adaptive