ECE 5578 Multimedia Communication

Lec 11 - Coding Standard & System

Zhu Li Dept of CSEE, UMKC Office: FH560E, Email: [email protected], Ph: x 2346. http://l.web.umkc.edu/lizhu

slides created with WPS Office Linux and EqualX LaTex equation editor

Z. Li Multimedia Communciation, 2018 p.1 Outline  Lecture 09 Video Signal Processing ReCap  Course Project Topic: Light Field Compression  Video Coding Standard and Systems . HEVC History . HEVC Performance . HEVC Key Coding Features

Z. Li, ECE 5578 Multimedia Communciation, 2018 p.2 Video Signal Processing  Coding Structure

……

I P P P … P P P 1 2 3 4 5 6 7 I B B P B B P GoP . Fast Block : GoP

Z. Li, ECE 5578 Multimedia Communciation, 2018 p.3 MV coding modes in HEVC  3 MV coding modes  key ideas, predicting from current block’s spatio-temporal neighbors  new merge mode: saving MV coding bits by combine small blocks into larger blocks.

Z. Li, ECE 5578 Multimedia Communciation, 2018 p.4 Intra Prediction in HEVC  Much more modes . DC mode: copy DC values from neighbor Like a sparse transform basis! . Planar mode: top row or left col average . Angular: pixels on certain line . Ref: Jani Lainema, Frank Bossen, Woojin Han, Junghye Min, Kemal Ugur, Intra Coding of the HEVC Standard. IEEE Trans. Circuits Syst. Video Tech. 22(12): 1792- 1801 (2012)

Z. Li, ECE 5578 Multimedia Communciation, 2018 p.5

 Reduce blocking artifact in the reconstructed frames  Can improve both subjective and objective quality  Filter in H.261:  [1/4, 1/2, 1/4]: Applied to non-block-boundary pixels in each block.  A low-pass smoothing filter.  In H.264 (and H.263v2), this is used in the prediction loop to improve motion estimation accuracy. Decoder needs to do the same. Also called loop filter.

H.264: 4x4 block level

H.265: 8x4 block level Before…. and After

Z. Li, ECE 5578 Multimedia Communciation, 2018 p.6 Sample Adaptive Offset (SAO) Filter  Classify pixels on block edge as one of the four categories

 Offset its pixel value accordingly

Z. Li, ECE 5578 Multimedia Communciation, 2018 p.7 Scalable Video Coding

 Scalable video coding is desired in heterogeneous networks:

 Types of scalability:

Z. Li, ECE 5578 Multimedia Communciation, 2018 p.8 HW-3: Motion Estimation  Fractional Pel . Use bi-linear interpolation to scale up the frames 2x2 . Do ½ pel motion estimation and compensation . Compare block size and pel accuracy effects on the residual MSE  B-Frames . Implement a special version of B-frames, have forward and backward motion vectors . Compare the results with P frame motion estimation  Fast Motion Estimation . Implement Diamond search pattern, compare with the exhaustive search  Data set: . “foreman” sequence frames 150, 151, and 152.

Z. Li, ECE 5578 Multimedia Communciation, 2018 p.9 Outline  Lecture 09 Video Signal Processing ReCap  Course Project Topic: Light Field Compression  Video Coding Standard and Systems . HEVC History . HEVC Performance . HEVC Key Coding Features

Z. Li, ECE 5578 Multimedia Communciation, 2018 p.10 What is Light Field ?  Capturing all lights in an observing volume

Z. Li, ECE 5578 Multimedia Communciation, 2018 p.11 Light Field Acquisition and Format  Lytro System

• Images were acquired using a Lytro Illum camera • LFR (light field raw) file format as defined by Lytro contains the raw image data:

 Output of camera sensor, overlaid with ‘GRBG’ Bayer filter;

 7728×5368 samples - 10-bit little endian integer values;

• A demosaicing process is required to obtain the color information from the sensor data;

 Demosaicing: raw data → RGB image;

From Prof. Fernando Pereira’s San Diego JPEG Presentation

Z. Li, ECE 5578 Multimedia Communciation, 2018 p.12 View Extraction Processes  Compression Code Point: • Light fields were extracted from the LFR files using the Light Field Toolbox v0.4 made available by Donald Dansereau, QUT, AU;

• http://lightfield-forum.com/tag/lightfield-toolbox/ • The process of obtaining a specific view from the full light field involves two main stages:

Coding

Multi-perspectives

Multi-focus

From Prof. Fernando Pereira’s San Diego JPEG Presentation

Z. Li, ECE 5578 Multimedia Communciation, 2018 p.13 Lenslet (sensor) vs Sub-aperture (synthesis) Images

Lenslet sensor capture Sub-aperture image

Z. Li, ECE 5578 Multimedia Communciation, 2018 p.14 Light Field Data Set  EPFL data set (6.6GB) : http://mmspg.epfl.ch/EPFL-light-field- image-dataset

a) Bikes b) Desktop c) Flowers

d) Friends 1 e) Magnets 1 f) Stone Pillars Outside

Z. Li, ECE 5578 Multimedia Communciation, 2018 p.15 Light Field Compression Test  View Point Compression . Pre-selected 9 perspective view point in the light field

Z. Li, ECE 5578 Multimedia Communciation, 2018 p.16 Naïve Solution – Direct Coding of Lenslet  R-D curves: Frenando Pereira/IST results:

JPEG JPEG 2000

H.264/AVC HEVC Intra Intra

Z. Li, ECE 5578 Multimedia Communciation, 2018 p.17 Naïve Solution – Performances  JPEG, JPEG2K, AVC Intra, HEVC Intra . Fernando Pereira, at JPEG meeting in San Diego, 2016.02 Bikes Desktop Flowers

Z. Li, ECE 5578 Multimedia Communciation, 2018 p.18 Re-Focusing Point Performance  Re-Focusing Operating Points

Z. Li, ECE 5578 Multimedia Communciation, 2018 p.19 Re-Focusing Point R-D curves  Also across different codec, naïve solution (Fernando Pereira, IST)

JPEG JPEG 2000

H.264/AVC HEVC Intra Intra

Z. Li, ECE 5578 Multimedia Communciation, 2018 p.20 LF Coding Discussions  Redundancy in the lenslet images . Across lenslet prediction ? o Optical correction ? o Affine block matching ?

. R-D optimization for certain view perspective ?

. Across focal lens R-D optimization ?

Z. Li, ECE 5578 Multimedia Communciation, 2018 p.21 Outline  Lecture 09 Video Signal Processing ReCap  Course Project Topic: Light Field Compression  Video Coding Standard and Systems . HEVC History . HEVC Performance . HEVC Key Coding Features

Z. Li, ECE 5578 Multimedia Communciation, 2018 p.22 HEVC/H.265 History  HEVC Objective:

Credit: Vivienne Sze & Madhukar Budagavi, ISCAS 2014 Tutorial Z. Li, ECE 5578 Multimedia Communciation, 2018 p.23 New Design Features  HEVC (H.265) vs AVC (H.264)

Credit: Vivienne Sze & Madhukar Budagavi, ISCAS 2014 Tutorial Z. Li, ECE 5578 Multimedia Communciation, 2018 p.24 HEVC Development History  Long hard fight, many casualties…

Z. Li, ECE 5578 Multimedia Communciation, 2018 p.25 HEVC Performance – PSNR  HEVC vs AVC

Z. Li, ECE 5578 Multimedia Communciation, 2018 p.26 HEVC Performance – Subjective Score  HEVC vs AVC

Z. Li, ECE 5578 Multimedia Communciation, 2018 p.27 HEVC Coding Pipeline  HEVC Diagram

Credit: Prof. Oscar Au, Tutorial on HEVC Z. Li, ECE 5578 Multimedia Communciation, 2018 p.28 HEVC Coding Structure Slide Credit: Vivienne Sze & Madhukar Budagavi, ISCAS 2014 Tutorial  Quad Tree Decomposition:

Ref: G. Schuster, PhD Thesis, 1996: Optimal Allocation of Bits Among Motion, Segmentation and Residual

Z. Li, ECE 5578 Multimedia Communciation, 2018 p.29 HEVC Coding Structure  CTU- Coding Tree Unit . Upto 64x64 pel block . Macro Block equivalent . Aka LCU: Largest Coding Unit  CU – Coding Unit . Square NxN pel . Like Block  PU – Prediction Unit . Inter Prediction . Intra Prediction

Z. Li, ECE 5578 Multimedia Communciation, 2018 p.30 Prediction Unit  PU: . Basic Unit for Transform & Quantization

Z. Li, ECE 5578 Multimedia Communciation, 2018 p.31 Transform Unit (TU)  TU . Square blocks for transform & quantization/coding, size: 4x4, 8x8, 16x16, 32x32 . MaxTU size: 32 for luma, 16 for chroma . Min TU size: 4x4 for both luma and chroma . TU size can be larger than PU for inter PU case (MV merge)

Z. Li, ECE 5578 Multimedia Communciation, 2018 p.32 HEVC Transforms  Transform + Quant:

Z. Li, ECE 5578 Multimedia Communciation, 2018 p.33 HEVC Intra-Prediction  Intra-Prediction Modes

Z. Li, ECE 5578 Multimedia Communciation, 2018 p.34 Intra-Predicted Basis  As if it is a 1-non zero coefficient transform…

Ref: J. Laniema and W.-J. Han, “Intra Picture Prediction in HEVC”, Chapter in, Springer- Velag Book on High Efficiency Video Coding (HEVC): Algorithms and Architectures, Springer, 2014. Ed. V. Sze et. Al.

Z. Li, ECE 5578 Multimedia Communciation, 2018 p.35 Pre-Processing for Intra Prediction  Smoothing filtering before intra prediction

Z. Li, ECE 5578 Multimedia Communciation, 2018 p.36 Post-Processing for Intra Prediction  Remove discontinuity:

Z. Li, ECE 5578 Multimedia Communciation, 2018 p.37 HEVC Intra as Still Image Coding Standard  Advancing the state or art by quite a bit

Z. Li, ECE 5578 Multimedia Communciation, 2018 p.38 HEVC Inter Prediction  Supports 4x4 block ¼ pel accuracy Motion Compensation

Z. Li, ECE 5578 Multimedia Communciation, 2018 p.39 Interpolation Filters  Quite a bit of memory and computing cost:

Z. Li, ECE 5578 Multimedia Communciation, 2018 p.40 Fast Motion Estimation in HEVC HM  Diamond Search

Z. Li, ECE 5578 Multimedia Communciation, 2018 p.41 HEVC Entropy Coding  Binary :

Z. Li, ECE 5578 Multimedia Communciation, 2018 p.42 HEVC CABAC Throughput Improvement  Context Bypassing Mode:

Z. Li, ECE 5578 Multimedia Communciation, 2018 p.43 Parallel Processing Tools: Slice/Tile

Credit: Vivienne Sze & Madhukar Budagavi, ISCAS 2014 Tutorial Z. Li, ECE 5578 Multimedia Communciation, 2018 p.44 Profiles, Levels and Tiers  Operating HEVC in Profiles/Levels/Tiers

Z. Li, ECE 5578 Multimedia Communciation, 2018 p.45 HEVC Resources

 Main Spec: . http://www.itu.int/ITU-T/recommendaBons/rec.aspx?rec=11885  T-CSVT Special Issue: . 2012: Combined Issue on HEVC Standard and Research: http://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=6403920 . 2016: Special Issue on HEVC Extensions and Efficient HEVC Implementations: http://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=7372356  Springer Book: . V. Sze, M. Budagavi, G. J. Sullivan (Editors), “High Efficiency Video Coding (HEVC): Algorithms and Architectures,” Springer, 2014, http://www.springer.com/engineering/signals/book/ 978-3-319-06894-7  HM (open source software): . https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSovware/  HARP: A python HM access tool: . http://www.lms.lnt.de/HARP/

Z. Li, ECE 5578 Multimedia Communciation, 2018 p.46 Summary  Light Field Compression . Sensor data at lenslet . Support new perspective and re-focusing . Interesting R-D optimization issue at new perspective and focusing point in the light field . Calls for new coding tools, based on HEVC intra, e.g,  HEVC . Crown Jewel of 20+ years of research and engineering . Significant performance gains over the previous state of art . Worthwhile to master this tool for a variety of research and biz opportunities.

Z. Li, ECE 5578 Multimedia Communciation, 2018 p.47