Versatile Video Coding RIVER PUBLISHERS SERIES in SIGNAL, IMAGE and SPEECH PROCESSING

Versatile Video Coding RIVER PUBLISHERS SERIES IN SIGNAL, IMAGE AND SPEECH PROCESSING

Series Editors: MONCEF GABBOUJ Tampere University of Technology Finland

THANOS STOURAITIS University of Patras, Greece and Khalifa University, UAE

Indexing: All books published in this series are submitted to Thomson Reuters Book Citation Index (BkCI), CrossRef and to Google Scholar

The “River Publishers Series in Signal, Image and Speech Processing” is a series of com- prehensive academic and professional books which focus on all aspects of the theory and practice of signal processing. Books published in the series include research monographs, edited volumes, handbooks and textbooks. The books provide professionals, researchers, educators, and advanced students in the ﬁeld with an invaluable insight into the latest research and developments. Topics covered in the series include, but are by no means restricted to the following: • Signal Processing Systems • Digital Signal Processing • Image Processing • Signal Theory • Stochastic Processes • Detection and Estimation • Pattern Recognition • Optical Signal Processing • Multi-dimensional Signal Processing • Communication Signal Processing • Biomedical Signal Processing • Acoustic and Vibration Signal Processing • Data Processing • Remote Sensing • Signal Processing Technology • Speech Processing • Radar Signal Processing

For a list of other books in this series, visit www.riverpublishers.com Versatile Video Coding

Humberto Ochoa-Dom´ınguez IIT-UACJ Mexico

K. R. Rao University of Texas at Arlington USA

River Publishers Published, sold and distributed by: River Publishers Alsbjergvej 10 9260 Gistrup Denmark

River Publishers Lange Geer 44 2611 PW Delft The Netherlands

Tel.: +45369953197 www.riverpublishers.com

ISBN: 978-87-7022-047-7 (Hardback) 978-87-7022-046-0 (Ebook)

c 2018 River Publishers

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, mechanical, photocopying, recording or otherwise, without prior written permission of the publishers. Contents

Preface ix

Acknowledgements xi

List of Figures xiii

List of Tables xv

List of Abbreviations xvii

5 High Efficiency Video Coding (HEVC) 1 5.1 Introduction ...... 2 5.2 Joint Collaborative Team on Video Coding (JCT-VC) . . . . 2 5.3 Analysis of Coding Tools in HEVC Test Model, HM 1.0 – Intra Prediction ...... 13 5.4 HEVC Encoder ...... 15 5.4.1 Intra prediction ...... 18 5.4.2 Transform Coefficient Scanning ...... 22 5.4.3 Luma and Chroma Fractional Pixel Interpolation . . 23 5.4.4 Comparison of Coding Tools of HM1 and HEVC Draft 9 ...... 24 5.5 Extensions to HEVC ...... 25 5.6 Profiles and Levels ...... 26 5.7 Performance and Computational Complexity of HEVC Encoders ...... 26 5.8 System Layer Integration of HEVC ...... 28 5.9 HEVC Lossless Coding and Improvements ...... 28 5.10 Summary ...... 30 5.11 Projects ...... 42

8 Screen Content Coding for HEVC 121 8.1 Introduction to SCC ...... 121 8.2 Screen Content Coding Tools ...... 124

v vi Contents

8.2.1 Intra Block Copy ...... 124 8.2.2 Palette Mode ...... 127 8.2.2.1 Palette derivation ...... 127 8.2.2.2 Coding the palette entries ...... 129 8.2.2.3 Coding the palette indices ...... 130 8.2.3 Adaptive Colour Transform (ACT) ...... 130 8.2.3.1 Colour space conversion ...... 131 8.2.3.2 Encoder optimization ...... 132 8.2.4 Adaptive Motion Vector Resolution ...... 132 8.3 Lossless and Visually Lossless Coding Algorithms . . . . . 133 8.3.1 Residual DPCM ...... 133 8.3.2 Sample-based Weighted Prediction with Directional Template Matching ...... 135 8.3.3 Sample-based Angular Intra-prediction ...... 137 8.3.4 Sample-Based Angular Intra-prediction with Edge Prediction ...... 137 8.4 Fast Coding Algorithms ...... 138 8.4.1 Adaptive Motion Compensation Precision ...... 139 8.4.2 Fast Intra Coding ...... 140 8.5 Visual Quality Assessment ...... 142 8.5.1 Screen Image Quality Assessment ...... 142 8.5.2 Objective Quality Assessment ...... 144 8.5.3 Subjective Quality Assessment ...... 145 8.6 Other SCC Algorithms ...... 146 8.6.1 Segmentation ...... 146 8.6.2 Rate Control ...... 147 8.7 Summary ...... 148 8.8 Projects ...... 149 H.264 Advance Video Coding (AVC)/MPEG-4 Part 10 References ...... 161 Books on H.264 ...... 169 H.264 Standard, JM SOFTWARE ...... 169 DCT References ...... 170 HEVC (High Efﬁciency Video Coding) References ...... 187 Software Repository; Scalable Extensions of HEVC ...... 204 Build System ...... 204 Software Structure ...... 204 ATSC Advanced television systems committee www.atsc.org . . . 205 Video Coding: Recent Developments for HEVC and Future Trends ...... 222 Contents vii

MPEG Internet Video Coding ...... 228 Conclusion ...... 229 MPEG-DASH ...... 252 HEVC Overview Online ...... 266 Test Sequences Data bases ...... 267 IEEE Journal on Emerging and Selected Topics in Circuits and Systems (JETCAS) ...... 271 How to Access JCT-VC Documents ...... 273 Performance Comparison of Video Standards (PCVS) ...... 274 Subjective Evaluaton of Compression Algorithms and Standards . 275 Books on HEVC and related areas ...... 283 Overview Papers ...... 284 Tutorials ...... 287 Special Sessions ...... 293 Transcoders ...... 295 Encryption of HEVC Bit Streams ...... 299 File Format ...... 299 Online Courses (OLC) ...... 299 Open Source Software ...... 300 X265 Source code ...... 303 vTune ampliﬁer by Intel ...... 304 General ...... 304 JVT REFLECTOR ...... 306 HEVC Quality Evaluation ...... 306 JCT-VC Documents ...... 306 References on SSIM ...... 307 SSIMPLUS Index for Video Quality-of-Experience Assessment ...... 310 Bjontegaard Metric ...... 311 VP8, VP9 ...... 312 JPEG 2000 ...... 316 Digital Cinema ...... 320 JPEG 2000 Related Work ...... 320 JPSearch ...... 321 Sweden Records Missile Tests Using JPEG2000 Technology April 2009 ...... 322 JPEG XR ...... 324 JPEG-LS ...... 326 JPEG ...... 327 viii Contents

JPEG XT ...... 338 JPEG XT Projects ...... 341 JPEG PLENO ...... 343 JPEG XS ...... 347 JPEG AIC ...... 351 JBIG ...... 351 LAR-LLC ...... 351 PNG ...... 352 WebP ...... 352 WebM ...... 352 DIRAC (BBC) ...... 353 DAALA ...... 354 Bjontegaard metric ...... 357 AVS China ...... 357 Thor Video Codec ...... 360 References on Screen Content Coding ...... 361 Beyond HEVC ...... 367 Projects on BEYOND HEVC ...... 368 Post – HEVC activity ...... 372 AV1 codec (Alliance for open media – AOM) ...... 372 Alliance for Open Media ...... 374 Real Media HD (RMHD) ...... 381 SMPTE ...... 381 WMV-9 (VC-1) ...... 383 VC-2 Standard ...... 386 Legacy Codec ...... 389 DSC by VESA ...... 389 Joint Video Experts Team (JVET) ...... 390 360 Degree Video Coding ...... 395 RGB <======> YCbCr Conversion ...... 395 PSNRAVG ...... 395 Weighted BDBR ...... 396 Colour-sensitivity-based Combined PSNR (CSPSNR) ...... 396 Cross-Media Retrieval ...... 396 Multiple Description Coding ...... 397

Index 399

About the Authors 405 Preface

The primary focus is on the developments beyond High-Efficiency Video Coding (HEVC). The new standard called versatile video coding (VVC) is projected to be finalized by 2020. A detailed description of the tools and techniques that govern the encoder indirectly and the decoder directly is intentionally avoided as a number of books (specially by the specialists who are directly involved in proposing/contributing/evaluating/finalizing the detailed processes that constitute the standards) are already available in this field, besides the overview papers, standards documents, reference software, software manuals, test sequences, source codes, tutorials, keynote speakers, panel discussions, reflector, and ftp/web sites – all in the public domain. Access to these categories is also provided. Since 2014, in the standards arena in HEVC, range extensions and new profiles (3D, multi-view, scala- bility, screen content coding – SCC) have been finalized. Also others such as MPEG-4 Internet Video Coding (ISO/IEC 14496-33) and AVS2 (IEEE 1857-4) have been standardized. Similarly, the industry has also finalized codecs such as, DAALA, THOR, VP9 (Google), VC1 (SMPTE), real media HD (Real Networks), and DSC (display stream compression) by VESA (Video Electronics Standards Association). AV1 (Alliance for Open Media, AOM) has been finalized. A brief introduction to the joint exploration model (JEM) by joint video experts team (JVET), versatile video coding (VVC), and future video coding (FVC) is provided. References related to 360-degree video coding are also listed. Other interesting and relevant topics such as cross-media retrieval and multiple description coding are also described. This book provides access to all these developments. International, regional, and national standards organizations, industry groups (alliances), professional societies, etc. are continuously exploring more efficient video coding techniques. A brief description of FVC beyond HEVC/H.265 is provided. Nearly 500 references related to HEVC and other emerging video coding standards are added along with 405 projects/problems. The latter are self-explanatory and govern the spectrum from a 3-hour graduate credit to research at the

ix x Preface masters and doctoral levels. Some require a dedicated group of researchers with extensive computational (software) and testing facilities. Additional projects/problems based on image coding standards (both developed and some in final stages) such as JPEG2000, JPEG LS, JPEG XR, JPEG-XT, JPEG XS (call for proposals for a low-latency lightweight image coding system issued in March 2016 by JPEG and almost finalized in April 2018), JPEG-HDR, and JPEG-PLENO are added. References, overview papers, panel discussions, tutorials, open source software, test sequences, confor- mance bit streams, etc. emphasizing these topics are also listed. A brief description related to JVET, also known as next-generation video coding (NGVC), formed by both MPEG and VCEG, is targeted for a potential new standard by the end of 2020. This is also called VVC. Also, VESA issued a “call for technology” with the objective to standardize a significantly more complex codec called ADSC (advanced DSC) that is visually lossless at a bit rate lower than DSC. The AVS workgroup of China is on a fast forward track in adding SCC capability to AVS2. All these developments can immensely help the researchers, academia, and the graduate students and provide food for thought to delve deeply into the fascinating world of multimedia compression. The reader is now well aware that this book is mainly at the research/reference level rather than as a textbook. It challenges the academic/research/industrial community regarding not only the present state-of- the-art but also, more specifically, the future trends and projections. Hence, it is an invaluable resource to this community. Acknowledgements

This book is the result of a long-term association of Dr. Humberto Ochoa- Dom´ınguez with Dr. K. R. Rao. Special thanks to their respective families for their support, perseverance and understanding. Dr. Rao likes to acknowledge the support provided in various forms by Dr. Jean-Pierre Bardet, Former Dean, College of Engineering (COE), Dr. J. W. Bredow, Chair, Department of Electrical Engineering, and colleagues all in UTA. Dr. G. J Sullivan, Microsoft, Dr. Nam Ling, University of Santa Clara, Dr. Ankur Saxena and Dr. Zhan Ma (both from Samsung Research Labs), Dr. Wen Gao, Peking University, Dr. M. Budagavi, Samsung Research America (SRA), Dr. M. T. Sun, University of Washington, Dr. H. Lakshman, Fr. D. Grois of Fraunhofer HHI, Dr. T. Borer, BBC, Dr. Deshpande, Sharp Labs, Dr. Bankoski and Dr. D. Mukherjee (both from Google) Dr. Y. Reznik, Brightcove, Dr. H. Kalva, Florida Atlantic University, Dr. E. Izquierdo, Queen Mary University of London, Dr. E. Magli, Dept. of Electronics and Telecommu- nications, Politecnico di Torino, Italy, Dr. W.-K. Cham, Chinese University of Hong Kong, Hong Kong and P. Topiwala, FastVDO for providing various resources in this regard. Constructive review by Dr. Ashraf Kassim, National University of Singapore, is highly valuable. Shiba Kuanar, Harsha Nagathihalli Jagadish, Anusha Vasireddy and Swaroop Krishna Rao at UTA contributed countless hours in tying up all loose ends (references/copy right releases, proof reading and million other details). The graduate students and alumnae in multimedia processing lab (MPL) at UTA in various ways have made constructive comments.