ECE 5578 Multimedia Communication
Lec 11 - Video Coding Standard & System
Zhu Li Dept of CSEE, UMKC Office: FH560E, Email: [email protected], Ph: x 2346. http://l.web.umkc.edu/lizhu
slides created with WPS Office Linux and EqualX LaTex equation editor
Z. Li Multimedia Communciation, 2018 p.1 Outline Lecture 09 Video Signal Processing ReCap Course Project Topic: Light Field Compression Video Coding Standard and Systems . HEVC History . HEVC Performance . HEVC Key Coding Features
Z. Li, ECE 5578 Multimedia Communciation, 2018 p.2 Video Signal Processing Coding Structure
……
I P P P … P P P 1 2 3 4 5 6 7 I B B P B B P GoP . Fast Block Motion Estimation: GoP
Z. Li, ECE 5578 Multimedia Communciation, 2018 p.3 MV coding modes in HEVC 3 MV coding modes key ideas, predicting from current block’s spatio-temporal neighbors new merge mode: saving MV coding bits by combine small blocks into larger blocks.
Z. Li, ECE 5578 Multimedia Communciation, 2018 p.4 Intra Prediction in HEVC Much more modes . DC mode: copy DC values from neighbor Like a sparse transform basis! . Planar mode: top row or left col average . Angular: pixels on certain line . Ref: Jani Lainema, Frank Bossen, Woojin Han, Junghye Min, Kemal Ugur, Intra Coding of the HEVC Standard. IEEE Trans. Circuits Syst. Video Tech. 22(12): 1792- 1801 (2012)
Z. Li, ECE 5578 Multimedia Communciation, 2018 p.5 Deblocking Filter
Reduce blocking artifact in the reconstructed frames Can improve both subjective and objective quality Filter in H.261: [1/4, 1/2, 1/4]: Applied to non-block-boundary pixels in each block. A low-pass smoothing filter. In H.264 (and H.263v2), this is used in the prediction loop to improve motion estimation accuracy. Decoder needs to do the same. Also called loop filter.
H.264: 4x4 block level
H.265: 8x4 block level Before…. and After
Z. Li, ECE 5578 Multimedia Communciation, 2018 p.6 Sample Adaptive Offset (SAO) Filter Classify pixels on block edge as one of the four categories
Offset its pixel value accordingly
Z. Li, ECE 5578 Multimedia Communciation, 2018 p.7 Scalable Video Coding
Scalable video coding is desired in heterogeneous networks:
Types of scalability:
Z. Li, ECE 5578 Multimedia Communciation, 2018 p.8 HW-3: Motion Estimation Fractional Pel Motion Compensation . Use bi-linear interpolation to scale up the frames 2x2 . Do ½ pel motion estimation and compensation . Compare block size and pel accuracy effects on the residual MSE B-Frames . Implement a special version of B-frames, have forward and backward motion vectors . Compare the results with P frame motion estimation Fast Motion Estimation . Implement Diamond search pattern, compare with the exhaustive search Data set: . “foreman” sequence frames 150, 151, and 152.
Z. Li, ECE 5578 Multimedia Communciation, 2018 p.9 Outline Lecture 09 Video Signal Processing ReCap Course Project Topic: Light Field Compression Video Coding Standard and Systems . HEVC History . HEVC Performance . HEVC Key Coding Features
Z. Li, ECE 5578 Multimedia Communciation, 2018 p.10 What is Light Field ? Capturing all lights in an observing volume
Z. Li, ECE 5578 Multimedia Communciation, 2018 p.11 Light Field Acquisition and Format Lytro System
• Images were acquired using a Lytro Illum camera • LFR (light field raw) file format as defined by Lytro contains the raw image data:
Output of camera sensor, overlaid with ‘GRBG’ Bayer filter;
7728×5368 samples - 10-bit little endian integer values;
• A demosaicing process is required to obtain the color information from the sensor data;
Demosaicing: raw data → RGB image;
From Prof. Fernando Pereira’s San Diego JPEG Presentation
Z. Li, ECE 5578 Multimedia Communciation, 2018 p.12 View Extraction Processes Compression Code Point: • Light fields were extracted from the LFR files using the Light Field Toolbox v0.4 made available by Donald Dansereau, QUT, AU;
• http://lightfield-forum.com/tag/lightfield-toolbox/ • The process of obtaining a specific view from the full light field involves two main stages:
Coding
Multi-perspectives
Multi-focus
From Prof. Fernando Pereira’s San Diego JPEG Presentation
Z. Li, ECE 5578 Multimedia Communciation, 2018 p.13 Lenslet (sensor) vs Sub-aperture (synthesis) Images
Lenslet sensor capture Sub-aperture image
Z. Li, ECE 5578 Multimedia Communciation, 2018 p.14 Light Field Data Set EPFL data set (6.6GB) : http://mmspg.epfl.ch/EPFL-light-field- image-dataset
a) Bikes b) Desktop c) Flowers
d) Friends 1 e) Magnets 1 f) Stone Pillars Outside
Z. Li, ECE 5578 Multimedia Communciation, 2018 p.15 Light Field Compression Test View Point Compression . Pre-selected 9 perspective view point in the light field
Z. Li, ECE 5578 Multimedia Communciation, 2018 p.16 Naïve Solution – Direct Coding of Lenslet R-D curves: Frenando Pereira/IST results:
JPEG JPEG 2000
H.264/AVC HEVC Intra Intra
Z. Li, ECE 5578 Multimedia Communciation, 2018 p.17 Naïve Solution – Codec Performances JPEG, JPEG2K, AVC Intra, HEVC Intra . Fernando Pereira, at JPEG meeting in San Diego, 2016.02 Bikes Desktop Flowers
Z. Li, ECE 5578 Multimedia Communciation, 2018 p.18 Re-Focusing Point Performance Re-Focusing Operating Points
Z. Li, ECE 5578 Multimedia Communciation, 2018 p.19 Re-Focusing Point R-D curves Also across different codec, naïve solution (Fernando Pereira, IST)
JPEG JPEG 2000
H.264/AVC HEVC Intra Intra
Z. Li, ECE 5578 Multimedia Communciation, 2018 p.20 LF Coding Discussions Redundancy in the lenslet images . Across lenslet prediction ? o Optical correction ? o Affine block matching ?
. R-D optimization for certain view perspective ?
. Across focal lens R-D optimization ?
Z. Li, ECE 5578 Multimedia Communciation, 2018 p.21 Outline Lecture 09 Video Signal Processing ReCap Course Project Topic: Light Field Compression Video Coding Standard and Systems . HEVC History . HEVC Performance . HEVC Key Coding Features
Z. Li, ECE 5578 Multimedia Communciation, 2018 p.22 HEVC/H.265 History HEVC Objective:
Credit: Vivienne Sze & Madhukar Budagavi, ISCAS 2014 Tutorial Z. Li, ECE 5578 Multimedia Communciation, 2018 p.23 New Design Features HEVC (H.265) vs AVC (H.264)
Credit: Vivienne Sze & Madhukar Budagavi, ISCAS 2014 Tutorial Z. Li, ECE 5578 Multimedia Communciation, 2018 p.24 HEVC Development History Long hard fight, many casualties…
Z. Li, ECE 5578 Multimedia Communciation, 2018 p.25 HEVC Performance – PSNR HEVC vs AVC
Z. Li, ECE 5578 Multimedia Communciation, 2018 p.26 HEVC Performance – Subjective Score HEVC vs AVC
Z. Li, ECE 5578 Multimedia Communciation, 2018 p.27 HEVC Coding Pipeline HEVC Diagram
Credit: Prof. Oscar Au, Tutorial on HEVC Z. Li, ECE 5578 Multimedia Communciation, 2018 p.28 HEVC Coding Structure Slide Credit: Vivienne Sze & Madhukar Budagavi, ISCAS 2014 Tutorial Quad Tree Decomposition:
Ref: G. Schuster, PhD Thesis, 1996: Optimal Allocation of Bits Among Motion, Segmentation and Residual
Z. Li, ECE 5578 Multimedia Communciation, 2018 p.29 HEVC Coding Structure CTU- Coding Tree Unit . Upto 64x64 pel block . Macro Block equivalent . Aka LCU: Largest Coding Unit CU – Coding Unit . Square NxN pel . Like Block PU – Prediction Unit . Inter Prediction . Intra Prediction
Z. Li, ECE 5578 Multimedia Communciation, 2018 p.30 Prediction Unit PU: . Basic Unit for Transform & Quantization
Z. Li, ECE 5578 Multimedia Communciation, 2018 p.31 Transform Unit (TU) TU . Square blocks for transform & quantization/coding, size: 4x4, 8x8, 16x16, 32x32 . MaxTU size: 32 for luma, 16 for chroma . Min TU size: 4x4 for both luma and chroma . TU size can be larger than PU for inter PU case (MV merge)
Z. Li, ECE 5578 Multimedia Communciation, 2018 p.32 HEVC Transforms Transform + Quant:
Z. Li, ECE 5578 Multimedia Communciation, 2018 p.33 HEVC Intra-Prediction Intra-Prediction Modes
Z. Li, ECE 5578 Multimedia Communciation, 2018 p.34 Intra-Predicted Basis As if it is a 1-non zero coefficient transform…
Ref: J. Laniema and W.-J. Han, “Intra Picture Prediction in HEVC”, Chapter in, Springer- Velag Book on High Efficiency Video Coding (HEVC): Algorithms and Architectures, Springer, 2014. Ed. V. Sze et. Al.
Z. Li, ECE 5578 Multimedia Communciation, 2018 p.35 Pre-Processing for Intra Prediction Smoothing filtering before intra prediction
Z. Li, ECE 5578 Multimedia Communciation, 2018 p.36 Post-Processing for Intra Prediction Remove discontinuity:
Z. Li, ECE 5578 Multimedia Communciation, 2018 p.37 HEVC Intra as Still Image Coding Standard Advancing the state or art by quite a bit
Z. Li, ECE 5578 Multimedia Communciation, 2018 p.38 HEVC Inter Prediction Supports 4x4 block ¼ pel accuracy Motion Compensation
Z. Li, ECE 5578 Multimedia Communciation, 2018 p.39 Interpolation Filters Quite a bit of memory and computing cost:
Z. Li, ECE 5578 Multimedia Communciation, 2018 p.40 Fast Motion Estimation in HEVC HM Diamond Search
Z. Li, ECE 5578 Multimedia Communciation, 2018 p.41 HEVC Entropy Coding Binary Arithmetic Coding:
Z. Li, ECE 5578 Multimedia Communciation, 2018 p.42 HEVC CABAC Throughput Improvement Context Bypassing Mode:
Z. Li, ECE 5578 Multimedia Communciation, 2018 p.43 Parallel Processing Tools: Slice/Tile
Credit: Vivienne Sze & Madhukar Budagavi, ISCAS 2014 Tutorial Z. Li, ECE 5578 Multimedia Communciation, 2018 p.44 Profiles, Levels and Tiers Operating HEVC in Profiles/Levels/Tiers
Z. Li, ECE 5578 Multimedia Communciation, 2018 p.45 HEVC Resources
Main Spec: . http://www.itu.int/ITU-T/recommendaBons/rec.aspx?rec=11885 T-CSVT Special Issue: . 2012: Combined Issue on HEVC Standard and Research: http://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=6403920 . 2016: Special Issue on HEVC Extensions and Efficient HEVC Implementations: http://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=7372356 Springer Book: . V. Sze, M. Budagavi, G. J. Sullivan (Editors), “High Efficiency Video Coding (HEVC): Algorithms and Architectures,” Springer, 2014, http://www.springer.com/engineering/signals/book/ 978-3-319-06894-7 HM (open source software): . https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSovware/ HARP: A python HM access tool: . http://www.lms.lnt.de/HARP/
Z. Li, ECE 5578 Multimedia Communciation, 2018 p.46 Summary Light Field Compression . Sensor data at lenslet . Support new perspective and re-focusing . Interesting R-D optimization issue at new perspective and focusing point in the light field . Calls for new coding tools, based on HEVC intra, e.g, HEVC . Crown Jewel of 20+ years of research and engineering . Significant performance gains over the previous state of art . Worthwhile to master this tool for a variety of research and biz opportunities.
Z. Li, ECE 5578 Multimedia Communciation, 2018 p.47