DEGREE PROJECT IN ELECTRICAL ENGINEERING, SECOND CYCLE, 30 CREDITS STOCKHOLM, SWEDEN 2019 Efficient Intra Prediction for the Next-Generation Video Compression Standard HENGYU MAN KTH ROYAL INSTITUTE OF TECHNOLOGY SCHOOL OF ELECTRICAL ENGINEERING AND COMPUTER SCIENCE Efficient Intra Prediction for the Next-Generation Video Compression Standard HENGYU MAN Master in Science and Engineering Date: August 8, 2019 Supervisor: Zhi Zhang Examiner: Markus Flierl School of Electrical Engineering and Computer Science Host company: Ericsson AB Swedish title: Effektiv Intra Prediction för Next Generation Video Compression Standard iii List of Abbreviations HEV C High Efficiency Video Coding AV C Advanced Video Coding VVC Versatile Video Coding DCT Discrete Cosine Transform DST Discrete Sine Transform DP B Decoded Picture Buffer P SNR Peak Signal to Noise Ratio MSE Mean Square Error BD − rate Bjøntegaard delta-rate QP Quantization Parameter CTU Coding Tree Units CTB Coding Tree Blocks CU Coding Units CB Coding Blocks PU Prediction Units PB Prediction Blocks TU Transform Units TB Transform Blocks SP S Sequence Parameter Set W AIP Wide Angular Intra Prediction MPM Most Probable Modes SAT D Sum of Absolute Transformed Difference R − D Rate-Distortion iv AAM Adaptive Angular Mode DIA_IDX Diagonal_index EncT Encoder Time DecT Decoder Time v Abstract VVC (Versatile Video Coding) is a next-generation video compression stan- dard. It is a potential successor to the MPEG-4/H.264 and High Efficiency Video Coding (HEVC)/H.265. Its first working draft was released in April 2018 by MPEG and VCEG joint video exploration team. Its final standard is to be approved before the end of 2020. Video compression enables more ef- ficient use of transmission and storage raw video data to meet the increase of demand for high quality video over the internet. The objective of this thesis work is to research and implement algorithms developed at Ericsson Research to improve video coding efficiency. The al- gorithm is about intra angular prediction. The intra angular prediction uses previous decoded samples of the current picture to predict the current block. In HEVC there are 35 directional intra prediction modes designed to be able to efficiently model different directional structures typically present in video and image contents. In VVC, the total number of directional intra prediction modes increases to 67. This thesis work is to research a dynamic granularity intra angular prediction algorithm. The thesis work has investigated several variants of dynamic granularity intra angular prediction and focused on im- proving video coding efficiency. vi Sammanfattning VVC (Versatile Video Coding) är en nästa generations videokomprimerings- standard. Det är en potentiell efterföljare för MPEG-4 / H.264 och High Effi- ciency Video Coding (HEVC) /H.265. Det första arbetsutkastet släpptes i april 2018 av MPEG och VCEG: s gemensamma videoutforskningsteam. Dess slut- liga standard ska godkännas före utgången av 2020. Videokomprimering möj- liggör en effektivare användning av överföring och lagring av rå videodata för att möta ökningen av efterfrågan på högkvalitativ video över internet. Målet med detta avhandlingsarbete är att forska och implementera algoritmer utvecklade vid Ericsson Research för att förbättra videokodningseffektiviteten. Algoritmen handlar om intravinkell förutsägelse. Den intravinkliga förutsägel- sen använder tidigare avkodade prover av den aktuella bilden för att förutsäga det aktuella blocket. I HEVC finns det 35 riktningsintredikationslägen utfor- made för att effektivt kunna modellera olika riktningsstrukturer som vanligtvis finns i video- och bildinnehåll. I VVC ökar det totala antalet riktade intrapre- diktionslägen till 67. Detta avhandlingsarbete är att undersöka en dynamisk granularitet intravinkell förutsägelsealgoritm. Examensarbetet har undersökt flera varianter av dynamisk granularitet inom vinkel förutsägelse och fokuserat på att förbättra videokodningseffektiviteten. List of Figures 2.1 Typical HEVC Video Encoder, with decoder module shaded ingray..............................4 2.2 Example for partitioning of a 64*64 CTU into CUs. This pro- cess can be described using a coding tree as shown on the right.9 2.3 Example for partitioning of a 64*64 CTU into CUs (blue) and TUs (red). This process can be illustrated using a coding tree as shown on the right, blue line represents CU partition and red line represents TU partition. 10 2.4 Reference samples Rx;y used for the prediction of block sam- ples Px;y ............................ 12 2.5 An example of reference sample substitution. (A) indicate the reference samples before substitution (gray parts is unavail- able reference samples). (B) indicate reference samples after substitution. 13 2.6 An example of reference sample smoothing . 15 2.7 Angular prediction modes from 2 to 34 with associated dis- placements in HEVC. H and V represent horizontal likely and vertical likely directions, respectively. The numerical part near the H and V indicates the sample displacement with 1/32 frac- tion of sample grid positions . 16 2.8 Extend the above main reference samples by projecting the left reference column when the prediction mode is 23. 18 2.9 An example of planar prediction mode. (a) indicates the ref- erence samples used for the prediction of pixel P[1][2] when the block size is 4*4. (b) shows the calculation result. 20 2.10 The derivation process of Most probable mode. A and B rep- resent the prediction modes of left and above PU, respectively. 23 vii viii LIST OF FIGURES 3.1 Number of intra angular prediction modes increases from 33 to65............................... 28 3.2 Clockwise shift for odd-horizontal modes . 31 3.3 Inserting new modes on vertical side . 35 3.4 The schematic diagram of block shape-number of modes de- pendency . 37 3.5 Vertical case . 38 3.6 Horizontal case . 38 List of Tables 3.1 Interpolation from 33 modes to 65 modes . 27 3.2 129 modes’ distribution . 27 3.3 Relationship between Inserting and Block’s Size . 39 3.4 Relationship between number of modes and Block’s Size . 42 4.1 Performance of video coding system with 131 prediction modes 45 4.2 Performance of video coding system with AAM by shifting . 47 4.3 Performance of video coding system with adaptive angular mode algorithm with block-shape restriction . 48 4.4 Performance of video coding system with adaptive angular mode algorithm with Block-shape restriction and mode-decrease scheme . 49 4.5 Performance of AAM video coding system without block-shape restriction . 49 4.6 Performance of video coding system with efficiency mode- selection algorithm . 50 4.7 Performance of original anchor after adding 3-round search algorithm . 51 ix Contents List of Figures vii List of Tables ix 1 Introduction 1 1.1 Video Coding Standard . .1 1.2 Video Coding Technology . .1 1.3 Intra Prediction and motivation . .2 2 Background 3 2.1 High Efficiency Video Coding . .3 2.1.1 History . .3 2.1.2 Significance of Development . .4 2.1.3 HEVC Video Coding Layers . .4 2.1.4 Evaluation Methods in HEVC . .5 2.1.5 Block Structures in HEVC . .7 2.2 Intra prediction in HEVC . 11 2.2.1 Overview of Intra Prediction . 11 2.2.2 Generation of Reference Samples . 12 2.2.3 Angular Prediction Modes . 15 2.2.4 DC Prediction and Planar Prediction . 19 2.2.5 Post-Processing Technology . 20 2.2.6 Prediction of Luma Intra Mode . 21 2.3 Software Tools . 25 2.3.1 Linux Cluster . 25 2.3.2 Python Script . 25 2.3.3 Base Code Version . 25 x CONTENTS xi 3 Adaptive Angular Modes for Intra Prediction 26 3.1 Research Direction . 26 3.2 Hard Interpolation . 27 3.3 Adaptive Angular Mode by Shifting . 30 3.4 New Adaptive Angular Mode Algorithm with Block-shape Re- striction . 34 3.4.1 Replace Rotation by Interpolation . 34 3.4.2 Block Shape Restriction . 35 3.4.3 Encoder Search Process . 39 3.5 Efficiency Mode-Selection Algorithm according to Block shape 41 4 Experimental Results 44 4.1 Result of Hard Interpolation Algorithm . 44 4.2 Result of Adaptive Angular Mode by shifting . 46 4.3 Result of Adaptive Angular Mode Algorithm with Block-shape Restriction . 47 4.4 Result of Efficiency Mode-Selection Algorithm according to Block shape . 50 5 Conclusions and Future Work 53 Bibliography 55 Chapter 1 Introduction 1.1 Video Coding Standard There were two eminent international organizations laying down the video coding standard, named ITU-T Video Coding Experts Group (VCEG) and ISO/IEC Moving Picture Experts Group (MPEG). In April 2010, this two organizations formed the Joint Collaborative Team on Video Coding (JCT- VC). In 2012, the first version of H.265/HEVC was drafted. Compared to H.264/AVC, H.265/HEVC increased coding efficiency and decreased the com- plexity, especially enabled high-resolution and high-quality video application. Since then, the research on next video coding standard had already started. April 2018, the joint collaborative team began drafting the first version of Ver- satile Video Coding (VVC). 1.2 Video Coding Technology Video encoding is a process of compressing and possibly converting the for- mat of video content, sometimes even converting an analog source content to a digital content. In terms of compression, the goal is to reduce the footprint. Video compression is a lossy process. The reconstructed video data for play- back is created by video decompression process. Video coding technology can be treated as one of the most important Inter- net technology. This is because compression reduces the bandwidth required while providing a high-quality experience and enable high-efficiency transmis- sion. Without compression, due to the limit of insufficient normal connection speed, raw video content may not be transformed into streaming content and 1 2 CHAPTER 1. INTRODUCTION transmitted through the Internet. The important aspect is the bit rate, or the amount of data per second in the video.