1

High Efficiency Video Coding (HEVC) The MPEG Vision 2

Three years ago in 2009, it was expected -- Ultra-HD (e.g., 4kx2k) video will emerge -- Mobile HD applications will become popular -- Video bitrate using current technology will go up faster than the network infrastructure

Now, we see … London 2012 Olympic Games 3

Immersive experience in Super Hi-Vision 16 times the resolution of Full HD ()

Pioneered by NHK & BBC 7680x4320 Go beyond Full HD 4

DVD (720x480)

SDTV (1280x720) VIDEO RESOLUTION HDTV (1920x1080) 2K (2048x1080) COMPARISON FROM 480P TO 4320P

Digital Cinema 4K (4096x2160)

Super Hi-Vision / Ultra HD (7680x4320) High Efficiency Video Coding (HEVC) 5

The latest draft video coding standard developed by a joint team of experts from ISO/IEC MPEG and ITU-T VCEG

Goal: substantially better performance than the H.264/AVC standard, especially in coding HD and Ultra- HD video

International Standard (IS) in April 2013 Timeline 6

 2005-: MPEG/VCEG exploration activities

 2010/01: Joint Collaborative Team (JCT-VC)  Final Call-for-Proposals (CfP)

 2010/04: 1st JCT-VC Meeting (Dresden, Germany)  27 proposals received

 2010/10: Working Draft 1.0 (WD1.0) and HM-1.0

 2013/01: Final Draft International Standard (FDIS)

 2013/04: International Standard (IS) HEVC Version 1 7

Press letter of 103rd Geneva Meeting (N13253) – The next major milestone in MPEG video history is achieved “ISO/IEC JTC1/SC29/WG11 MPEG is proud to announce the completion of the new High Efficiency Video Coding (HEVC) standard which has been promoted to Final Draft International Standard (FDIS) status at the 103rd MPEG meeting.” Subjective Assessment 8

HM 5.0 JM 18.2 Over -50% MOS

 Class B (HD): -67%  Class C (SD): -49%

Bit Rate [Kbps] J. R. Ohm, G. J. Sullivan, F. Bossen, T. Wiegand, V. Baroncini, M. Wien, and J. Xu,“JCT-VC AHG report: HM subjective quality investigation (AHG22)”, JCTVC-H0022, San José, CA, Feb., 2012. New milestone since year 2003 9

Park Scene, 1920x1080, 24Hz 42 H.264/AVC 41 35.4% 40 39 HEVC 38 37 MPEG-4

PSNR (dB) 36 - H.264/MPEG-2 (MP) 35 MPEG-4 (ASP) YUV H.263 34 H.263 (HLP) 33 H.264/MPEG-4 AVC (HP) 32 MPEG-2 HEVC (MP) 31 0 2 4 6 8 10 12 14 Bitrate (Mbps) J.-R. Ohm, G. J. Sullivan, H. Schwarz, T. K. Tan, and T. Wiegand, “Comparison of the Coding Efficiency of Video Coding Standards—Including High Efficiency Video Coding (HEVC)”, IEEE Trans. CSVT, Dec., 2012 HEVC vs. AVC (1/2) 10

HM10.0 (Main) vs. JM18.4 (High)

23% ALL INTRA 33.5% RANDOM ACCESS 36.4% Test Condition Test LOW DELAY

0 10 20 30 40 50 BD-rate Saving (%)

B. Li, G. J. Sullivan, and J. Xu,“Comparison of Compression Performance of HEVC Draft 10 with AVC High Profile,” JCTVC-M0329, Incheon, April, 2013. HEVC vs. AVC (2/2) 11

Y BD-Rate (%) Class Resolution All Intra Radom Access Low Delay A 2500x1600 -23.6 -36.6 B 1080p -22.7 -39.8 -42.1 C 480p -19.7 -30.3 -32.7 D 240p -16.4 -28.0 -29.9 E -28.8 -44.1 F 480p, 720p -28.6 -31.2 -33.8 (Minus sign means coding gain) Encoding Time 12

HM10.0 (Main) vs. JM18.4 (High)

97% ALL INTRA 109% RANDOM ACCESS 71% LOW DELAY Test Condition Test

20% 40% 60% 80% 100% 120% Encoding Time Ratio (%)

B. Li, G. J. Sullivan, and J. Xu,“Comparison of Compression Performance of HEVC Draft 10 with AVC High Profile,” JCTVC-M0329, Incheon, April, 2013. Decoding Time 13

HM10.0 (Main) vs. JM18.4 (High) 107% ALL INTRA 42% RANDOM ACCESS 56% LOW DELAY Test Condition Test

20% 40% 60% 80% 100% 120% Decoding Time Ratio (%)

B. Li, G. J. Sullivan, and J. Xu,“Comparison of Compression Performance of HEVC Draft 10 with AVC High Profile,” JCTVC-M0329, Incheon, April, 2013. Multi-thread Decoding 14

HM-9.0 (RA-Main) 3840x2160(12Mbps) 90 80 1080p 70 720p 60fps 60 50 40 30fps 30

Frame Rate (fps) Rate Frame 20 10 0 0 0.5K 1K 1.5K 2K 2.5K Vertical Resolution T. Tan, Y. Suzuki, and F. Bossen, “On software complexity: decoding 4K60p content on a laptop,” JCTVC- L0098, Geneva, CH, Jan., 2013. Few weeks later … 15

Display 4.99 inch, 1920x1080 Camera 13 Mega Pixels Video 1080p@30fps MPEG4, H.264, H.263, Codec DivX, VC-1, VP8, WMV7/8, Sorenson Spark, HEVC

Samsung UNPACKED 2013 - http://www.youtube.com/watch?v=Yaw6CSaPnfk 1616

HEVC TOOL FEATURES HEVC Tool Features 17 • CABAC • Tiles • Wavefront

Current + DCT Entropy Bitstream Frame• Asymmetric Motion Partitioning - Q Coding • Merged Skip / Motion Merging • Advanced MV Prediction • Residual Quad-tree Trans. • DCT-based Interpolation Filter • Transform Skipping • Adaptive Coeff. Scanning Frame Inter Inter Buffer Prediction

• Deblocking Filter • More Directions Intra • Sample Adaptive Offset • Pre-/Post-filtering Prediction Intra• Direct Chroma In-loop + IQ filter + IDCT Coding Unit (CU) 18

Basic unit for coding, conceptually similar to macroblock but now can be of variable size

H.264/AVC HEVC 16 64

CU CU 16 MB MB CU  CU CU 64 CU CU CU

CTU0 CTU1 Prediction Unit (PU) 19

PU PU PU PU 16x16 PU PU MB PU PU PU PU PU PU PU PU PU PU PU

CU (Only for SCU) Asymmetric

Sub-MB Partition Transform Unit (TU) 20

Residual Quad-tree Transform (RQT) Transform can cross PU boundaries 8 4 TU TU TU TU 4 TU TU 8 16x16 TU TU TU TU PU MB TU TU TU TU Aligned TU TU TU TU TU TU May be skipped if 4x4

TU TU TU TU TU RQT TU TU CU TU TU TU TU TU Intra Prediction 21

More directions (up to 33) Adaptive pre-filtering of reference pixels Boundary smoothing for DC/Ver./Hor. modes 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 Vertical Direct mode for Chroma 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2

TR

Horizontal Reference Sample TR Mean

LB LB 0: DC 1: Planar DCT-based Interpolation Filter 22

P-3 P-2 P-1 P0 Pa P1 P2 P3 P4

M Integer-pels M Integer-pels Fraction-pel Forward DCT Spatial Domain Samples DCT Domain Coefficients C , C , …, C {P-3, P-2, …, P4} Inverse DCT { -3 -2 4}

21M − π −+ C0 kM(2 12a) pCak= + ∑ cos 24k =1 M DCT Coefficients Motion Merging (1/2) 23

Optional

5 Candidates at most Motion Merging (2/2) 24

Before Merging After Merging

mv0 mv1 mv0 mv1 mv4 mv5 mv4 mv5

mv2 mv3 mv2 mv3

Current PU 0 mv5 mv6 mv7 mv6 mv7

Current PU1 mv7

Merge w/ Nearby Motion Irregular Motion Partitions Temporal Merge Candidate Derivation 25

2

Co-located Current 3 1

L0 L0 Current L1 L1 Ref0 Ref0 Advanced Motion Vector Prediction 26

2 Find First Available

B2 B1 B0

Optional 2 Candidates at most 3 T1 1

A1 Current PU A0 T0 If LCU boundary, Available irst Find F irst exclude A0 & T0 Transform Skipping 27

Operable in 4x4 TUs Switchable with DST (intra) / DCT (inter)

TS Disabled TS Enabled

(QP37, 608.4Kbps, 34.8dB) (QP36, 600.8Kbps, 36.1dB) Significance Map Scanning 28

TU Size Prediction Type Scanning Order 4x4, 8x8 Intra (Vertical, Horizontal) Vertical, Horizontal All Intra, Inter 4x4 Sub-diagonal

Vertical Horizontal 4x4 Sub-diagonal

4

4 Sample Adaptive Offset (1/3) 29

Band Offset (BO): intensity-based sample classification Edge Offset (EO): edge-based sample classification

0 4 Bands MAX [BO]

[EO] EO Off Off EO C C BO BO BO EO

BO BO BO EO

C C Off EO EO EO

LCUs in a Frame Sample Adaptive Offset (2/3) 30

I. Positive Edge Offset evel L ixel P ixel x-1 x x+1 x-1 x x+1 x-1 x x+1 Index

II. Negative Edge Offset PixelLevel x-1 x x+1 x-1 x x+1 x-1 x x+1 Pixel Index Sample Adaptive Offset (3/3) 31

SAO Disabled SAO Enabled

(Frame15, 32.6dB, 3.0Mbps) (Frame15, 32.7dB, 3.1Mbps) Parallel-friendly Design 32

Transcoding to WPP

Sequential Bitstream Chunks re-ordering

Chunk encoding using 16 cores

Live Chunk re-ordering Content 2 entry points

G. Clare, F. Henry, and S. Pateux,“Wavefront and Cabac Flush: Different Degrees of Parallelism Without Transcoding”, JCTVC-F275, Torino, IT, July, 2011. Wavefront Parallel Processing (WPP) 33

Syntax Decoded Pass CABAC states Elements Pixels to next Wave Parsing Decompressing

Dependent Dependent Syntax Decoded Elements Pixels Parsing Decompressing

 Dependent Dependent [n Waves in a Slice]

Parallel Parallel Flush CABAC States Tiles 34

Syntax Decoded Elements Pixels Parsing Decompressing

Syntax Decoded Elements Pixels Parsing Decompressing

[4 Tiles in a Slice/Frame]

Parallel Parallel

Prediction can’t cross Tiles Parallel Merge Group 35

Groupm-1,n-1 Groupm,n-1 Groupm+1,n-1

Above Inferable

CUs/PUs within Groupm-1,n Groupm,n

Left Inferable

PUs/CUs within a group run in parallel Profiles & Tool Summary 36

Tool \ Profile Main Still Picture Main Main 10 Bit Depth 8 8 8, 9, 10 CU Size 16x16~64x64 PU Partition Symmetric Symmetric, Asymmetric TU Partition Residual Quad-tree Transform MV Prediction - AMVP, MRG, MRG-Skip Interpolation Filter - DCT-IF Intra Prediction DC, Planar, 33 Directions, DM Transform DCT 4x4~32x32, Skip 4x4, DST 4x4 (Intra) In-loop Filter De-blocking, SAO Entropy Coding CABAC (Tiles, Wavefront)

* Support picture resolution ranging from 128x96 to 8192x4320. Information 37

http://phenix.int-evry.fr/jct/ (Website)

http://mailman.rwth-aachen.de/mailman/listinfo/jct-vc (Subscribe)

https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/ (SVN, Software Manual JCTVC-J0470)

Text Specification Draft 9 (JCTVC-K1003) 3838

RANGE EXTENSIONS TO HEVC Range Extensions 39

Goal: minimum changes to HEVC V1 to support non- 4:2:0 chroma formats and bit depths beyond 8 bits and to improve lossless coding

Finalizes in July 2014 4040

RANGE EXTENSIONS TOOL FEATURES RExt Tool Features • One separate CABAC for41 significant map coding for TS block

Current + DCT Entropy Bitstream Frame - Q Coding

• Residual DPCM • Large TS block • Residual rotation Frame Inter Inter • Cross component prediction Buffer Prediction

Intra Prediction Intra In-loop + IQ filter + IDCT Residual DPCM 42

Second-order residual prediction Vertical/horizontal prediction Implicit for Intra, explicit for Inter Short-distance Processed at residual prediction parsing stage

sub. sub. sub. add add add Parallel p p a-p b-a c-b a-p b-p c-p d-p Encoding d-c

Parallel Decoding

Residual Block Reference Residual DPCM Samples Intra Residual Rotation 43

Reverse scan order for intra residual blocks when transform skip is in use

Much similar to the pattern of coefficients

tend to tend to be Rotate be zero 180 degree non-zero

tend to tend to be be non-zero zero

Residual Block Rotated Block Cross Component Prediction 44

Predict chroma residuals from luma residuals Apply to intra DM and inter (if luma residual exists) Adapt α {-8,-4,…,8} at TU level for U/V separately

∈

2nd-order Chroma Luma Chroma 𝜶𝜶 ∗ Recon. Chroma 𝟖𝟖 Recon. Luma Decoded 2nd-order Residual Residual Chroma Residual Common Test Conditions (AHG5) 45

Coding Structure QP Range (Lossy) • All Intra • MT (22, 27, 32, 37) • Random access • HT (17, 22, 27, 32) • Low delay B • SHT (12,17, 22, 27) Test Sequence Color Format Bit Depth Resolution (# of Seqs) RGB 4:4:4 8, 10,12 2560x1600 (1), 1920x1080 (7)

YUV 4:4:4 10 2560x1600 (1), 1920x1080 (6) YUV 4:2:2 10 2560x1600 (1), 1920x1080 (6) HM-14.0+RExt-7.0 vs. JM-18.6 46

Y BD-Rate (%) Radom All Intra Low Delay B Format Access MT HT SHT MT HT MT HT RGB 4:4:4 -32.6 -25.1 -19.6 -36.0 -25.1 -36.2 -25.0 YUV 4:4:4 -22.4 -19.0 -14.8 -35.1 -29.8 -39.8 -32.9 YUV 4:2:2 -19.7 -15.8 -11.7 -30.2 -27.8 -35.9 -31.0

B. Li, J. Xu, and Gary. J. Sullivan, “Comparison of Compression Performance of HEVC 4:4:4 Range Extensions Test Model 7 and HEVC Screen Content Coding Extensions Test Model 1 with AVC High 4:4:4 Predictive profile,” JCTVC-R0101, Sapporo, June, 2014. Information 47

http://phenix.int-evry.fr/jct/ (Website)

http://mailman.rwth-aachen.de/mailman/listinfo/jct-vc (Subscribe)

https://hevc.hhi.fraunhofer.de/svn/svn_SHVCSoftware/ (SVN)

Edition 2 Draft Text of High Efficiency Video Coding (HEVC), Including Format Range (RExt), Scalability (SHVC), and Multi-View (MV-HEVC) Extensions (JCTVC-R1013) 4848

SCREEN CONTENT CODING (SCC) Screen Content Coding (SCC) 49

Screen content video, usually a mixture of text, graphics and nature scene images, exhibits very different characteristics from camera-captured video Screen Sharing 50

Share & Play Together by TM Cloud Gaming 51 Camera-captured vs. Screen Contents 52

Camera-capture video – Continuous-tone with camera noise – Smooth edges, complicated texture, thick lines with rich colors Screen content video – Discontinuous-tone with less/no noise – Sharp edges, simple shapes, thin lines (e.g. 1-pixel wide) with few colors Why SCC is Challenging? 53

Missing details and annoying artifacts

Original SCM-1.0

(All Intra, QP27, 2.9Mbps, 42.5dB) Requirements 54

From the discussions on the reflector, SCC should address – 4:4:4 chroma sampling format (RGB, YUV) – Up to 10-bit for each color component – Low latency/complexity on encoder & decoder – Temporal stability – Subjectively lossless – Mathematically lossless for some application – Low bitrate for discontinuous-tone contents Screen Content Coding 55

Future HEVC extensions in coding screen content targeting at coding of 4:4:4 8-bit sequences

 2014/01: Final Call-for-Proposals

 2014/03: Evaluation of Proposals

 2014/04: SCC Test Model 1.0

 2014/07: SCC Test Model 2.0

 2015/02: Proposed Draft Amendment (PDAM)

 2015/10: Final Draft Amendment (FDAM) SCC Tool Features 56

• Adaptive color transform

Current + DCT Entropy Bitstream Frame - Q Coding • IBC mode • Palette mode

Frame Inter Inter Buffer Prediction

Intra Prediction Intra In-loop + IQ filter + IDCT Intra Block Copy (IBC) 57

Similar to inter motion compensation, except – using the current frame as reference – referring to un-deblocked samples Palette Mode 58

Adaptive color quantization for PCM pixels Represent a block by major colors & an index map Specify the major color a pixel mapped to by index Analysis Quantization Index [Block Samples] p(x) [Index Map] 0 1 Major 0 0 1 1 Color 0 0 1 1  0 0 1 1   1 Probability 0 0 1

 x   0 1 2 3 4 5 6 7 8 9   Intensity Value Other tools 59

Palette mode – Major color coding (e.g. stuffing, merging,…) – Index map coding (e.g. transition mode, dictionary,…) Intra string matching – PCM sample coding – Index coding Intra line copy Major Color Coding (1/2) 60

Major color table (MCT) propagation – Infer major colors from left or above CU (or previously decode ones when unavailable) Major color merge (left or above CU) Above Long-term palette prediction Reference – Signal a long-term palette MCT propagation CU at the slice header MCT propagation Left Current Triplet palette coding Reference CU CU – Three component-wise palette Major Color Coding (2/2) 61

Palette stuffing (predictor propagation) Index Map Coding (1/2) 62

Find a match inside current CU – Run mode (copy-left, copy-above) – Transition copy Index Map Coding (2/2) 63

Search a match outside current CU – 1-D/2-D String matching – Re-quantization required on reconstructed pixels

Search Range

Search Range Dictionary-based String Matching 64

Patterns repeat frequently within text and graphics regions (e.g. text, icons, lines etc.) Reconstruction-based String Matching 65

Preserve block structure Hybrid 1-D/2-D string matching for index coding More flexible than IBC, higher data dependency Intra Line Copy 66

Split a PU equally into 1xN/Nx1 lines Lower data dependency than string matching; more flexible than PU-based IBC

Search BV0, PU1 Range

BV0, PU0 ...... 2N BV1, PU0 Line (size=1) PU0 PU1 Search Range Common Test Conditions (SCC) 67

Coding Structure QP Range • All Intra (AI) • Lossy: 22, 27, 32, 37 • Random access (RA) • Lossless • Low delay B (LB) Test Sequence Category Resolution (# of Seqs) Text and graphics with motion 1920x1080 (6), 1280x720 (8)

Mixed content 2560x1440 (4), 1920x1080 (2)

Animation 1280x720 (2)

Camera-captured content 1920x1080 (4) SCM-1.0 vs. JM-18.6 68 Y BD-Rate (%) Coding Text, Graphics Mixed Content Anima. Camera Structure 1080p 720p 1440p 1080p 720p 1080p RGB Sequences AI -85.8 -71.6 -69.5 -74.9 -36.8 -45.1 RA -78.8 -65.3 -60.5 -70.1 -39.6 -49.8 LB -78.4 -61.3 -55.3 -62.0 -42.9 -46.6 YUV Sequences AI -77.2 -56.6 -54.0 -63.8 -23.6 -26.9 RA -69.4 -54.0 -48.3 -62.0 -32.6 -40.1 LB -68.8 -52.8 -45.9 -56.1 -39.1 -40.0 B. Li, J. Xu, and Gary. J. Sullivan, “Comparison of Compression Performance of HEVC 4:4:4 Range Extensions Test Model 7 and HEVC Screen Content Coding Extensions Test Model 1 with AVC High 4:4:4 Predictive profile,” JCTVC-R0101, Sapporo, June, 2014. Information 69

http://phenix.int-evry.fr/jct/ (Website)

http://mailman.rwth-aachen.de/mailman/listinfo/jct-vc (Subscribe - AHG7: Screen Content Coding)

https://hevc.hhi.fraunhofer.de/svn/svn_SHVCSoftware/ (SVN)

HEVC Screen Content Coding Draft Text 1(JCTVC-R1005) TCSVT Special Issue on HEVC 70

IEEE Transaction on Circuits and Systems for Video Technology (TCSVT), vol. 22, no. 12, Dec., 2012 Special Section: HEVC Standard – G. J. Sullivan, J.-R. Ohm, W.-J. Han, and T. Wiegand, “Overview of the High Efficiency Video Coding (HEVC) Standard” – J.-R. Ohm, G. J. Sullivan, H.Schwarz, T. K. Tan, and T. Wiegand, “Comparison of the Coding Efficiency of Video Coding Standards— Including High Efficiency Video Coding (HEVC)” Special Issue: Emerging Research and Standards in Next Generation Video Coding (HEVC) 71

Thank You ~