Presentation

Introducing MPEG-5 Part 2 LCEVC A codec to improve other codecs

Guido Meardi Geneva, ITU Workshop on the Future of Media 8/10/2019

Private and confidential Why a Low Complexity Enhancement Video Coding Standard?

(…) “ Encoder complexity is outpacing Moore’s law. (…) The longer-term answer to video encoding can not be to simply add more CPU capacity. This is an unsustainable model; both financially, and environmentally. (… Codec research must emphasize compression David Ronca efficiency AND computational efficiency. Here's a thought exercise: If the next video encoder Director of Video Encoding maintained the efficiency of VVC/AV1 but reduced Facebook computational complexity by 50%, would we consider (previously Netflix) that a successful new codec?

(from “Encoder complexity hits the wall”, 7/10/2019) “

Legacy (2003), but still large majority of video

File size (compression efficiency)

Time to encode (computational complexity)

MPEG-2 AVC/H.264 HEVC AV1 VP9 VVC

File size (compression efficiency)

Time to encode (computational complexity)

MPEG-2 AVC/H.264 HEVC VVC VP9 AV1

. Codec-agnostic codec of enhancement data, where enhancing n-th generation codec results in  Compression performance as close as possible to (n+1)-th generation codec  Encoding and decoding complexity in line with n-th generation codec

Standard Ancillary Compliant data stream Stream Enhanced Output video Demux decoder (full resolution)

Standard Capability Extension Decoder* of Existing Video Video Codec stream (codec agnostic)

* Any standard format: AVC, HEVC, VVC, … Copyright © 2019 V-Nova Ltd. All rights reserved 6 Copyright ©Copyright 2016 V- Nova© 2016 Ltd. V -AllNova rights Ltd. reserved All rights reserved What is a sub-layer of enhancement residual data?

Sparse highly detailed information

Copyright © 2019 V-Nova Ltd. All rights reserved 7 Copyright ©Copyright 2016 V- Nova© 2016 Ltd. V -AllNova rights Ltd. reserved All rights reserved Beware of super-resolution, fidelity requires the actual data

Sparse highly detailed information

Second layer of Residuals Combined Preliminary (Enhancement Output Picture Output Picture Sub-Layer 2)

First layer of = Residuals + Preliminary (Enhancement Intermediate Sub-Layer 1) Picture + = + Combined Intermediate Picture Decoded Base Picture (Layer 0) Temporal Buffer Copyright © 2019 V-Nova Ltd. All rights reserved 9 Copyright ©Copyright 2016 V- Nova© 2016 Ltd. V -AllNova rights Ltd. reserved All rights reserved Signal flow

sub-layer 1 enhancement layer

sub-layer 0

½ Resolution image stream base layer

Copyright © 2019 V-Nova Ltd. All rights reserved 10 Copyright ©Copyright 2016 V- Nova© 2016 Ltd. V -AllNova rights Ltd. reserved All rights reserved A standard that it helpful for the old, the current, and the future Compression benefits

Improving the quality of AVC … … as well as HEVC … … as well as VVC

ParkRunning UHDp50 - LCEVC VTL vs. VTL - VMAF

90.00 -45% -32% 88.00 -19%

86.00 -37% -45% 84.00 -14%

82.00 VMAF -38% 80.00

-43% BD-rate-VMAF BD-rate-VMAF 78.00 BD-rate-VMAF LCEVC AVC vs. AVC: LCEVC HEVC vs. HEVC: LCEVC VVC vs. VVC: -44.96% -34.20% 76.00 (coming soon) 74.00 4000 5500 7000 8500 10000 11500 13000 14500 Bitrate (Kbps)

LCEVC+VVC VVC

Copyright © 2019 V-Nova Ltd. All rights reserved 11 Copyright ©Copyright 2016 V- Nova© 2016 Ltd. V -AllNova rights Ltd. reserved All rights reserved What it means in practice – Example CampFire sequence

Copyright © 2019 V-Nova Ltd. All rights reserved 12 HEVC UHDp30 @ 8 Mbps Copyright ©Copyright 2016 V- Nova© 2016 Ltd. V -AllNova rights Ltd.LCEVC reserved All rights reserved HEVC UHDp30 @ 8 Mbps A standard that it helpful for the old, the current, and the future Encoding complexity benefits

Reducing the complexity of AVC … … as well as HEVC … … as well as VVC

LCEVC AVC Encoding Time, LCEVC HEVC Encoding Time, LCEVC VVC Encoding Time, as % of AVC Encoding Time as % of HEVC Encoding Time as % of VVC Encoding Time 100% = 74 sec 100% = 736 sec 100% = 1,412,882 sec 100 100 2.4x 2.7x 100 4.8x

41.3 37.6

21.3

x264 LCEVC x264 x265 LCEVC x265 VTL LCEVC VTL Copyright © 2019 V-Nova Ltd. All rights reserved 13 Copyright ©Copyright 2016 V- Nova© 2016 Ltd. V -AllNova rights Ltd. reserved All rights reserved The collaborative work performed in the past months produced notable improvements

V-Nova CfP Submission 1 (P11 ) V-Nova CfP Submission 1 (P11 ) P11 JM vs. JM P11 JM vs. JM BD-rate-PSNR BD-rate-MOS • The good results of the CfP ParkRunning3 -32.68% -51.70% improved materially! Campfire -18.32% -39.30% Average Class A -25.50% -45.50% • Specification, definitions and overall clarity also LCEVC improved significantly LCEVC JM vs. JM BD-rate-PSNR ParkRunning3 -43.52% Campfire -28.54% (MOS still to be done) Average Class A -36.03%

LCEVC LCEVC JM vs.P11 JM BD-rate-PSNR ParkRunning3 -17.28% (MOS still to be done) Campfire -13.80% Average Class A -15.54%

[email protected]