A Review of Emerging Video Codecs: Challenges and Opportunities Amal Punchihewa, Donald Bailey Centre for Research in Image and Signal Processing Massey University Palmerston North, New Zealand [email protected], [email protected] Abstract—This paper presents a review of video codecs that high-efficiency video coding HEVC or H265 and Versatile are in use and currently being developed, the codec development Video Coding (VVC) or H266. Online or streaming service process, current trends, challenges and opportunities for theproviders are also contributing to compression research in the research community. There is a paradigm shift in video coding areas of efficiency and performance evaluation. This paper standards. Concurrently, multiple video standards arebriefly discusses and presents some of the research in standardised by standardising organisations. At the same time,developing next-generation codecs (VVC-Versatile Video royalty free video compression standards are being developed Coding, EVC-Enhancement Video Coding, LCEVC-Low and standardised. Introduction of enhancement-layer-based Complexity Enhancement Video Coding) and needs of the coding standards will extend the lifetime of legacy video codecs industry. However, those multiple codecs where some are finding middle ground in improved coding efficiency, royalty-free have disturbed the ecosystem. This is another computational complexity and power requirements. The video coding landscape is changing that is challenged by emergence disruption of that broadcasters and media industry need to multiple video coding standards for different use cases. Thesemanage in business and digital transformation. may offer some opportunities for coding industry, especially for Compression technologies have been an enabler for the New Zealand researchers serving niche markets in video games, broadcast industry starting with analogue television. Over the computer generated videos and animations. last four decades, MPEG - Motion Pictures Expert Group - developed MPEG-1, MPEG-2, MPEG-4 and MPEG-H video Keywords—'video coding, compression, MPEG, ITU, computational complexity, coding efficiency compression standards. A given codec standard is developed with contribution from many researchers over a considerable I. Introduction time. International Standard Organisation/Motion Picture Expert Group - ISO/MPEG and International Video applications require acquisition, storage and Telecommunication Union - ITU have harmonised and transportation of picture data that have different standardised them. The following sections discuss briefly the characteristics compared to audio and text. Video data are evolving video coding standards, standardisation process, generally captured at a high rate. Resource constraints in advancements, trends and challenges. storage and transportation led to the development of methods to reduce the large quantity of video data per second for For video coding standards, the core problem has different applications. Television broadcasting has beenremained the same: i.e. to reduce the size of stored or transporting compressed video using radio frequencies overtransmitted video data as much as possible while keeping the the air extensively. In recent times the applications of videovisual quality as close as possible to the original video. The have expanded enormously to include applications such conventionas when it comes to video coding standards has been online video, surveillance, video calling, Internet of Thingsto only define the bitstream format (syntax) and the decoder (IoT), augmented and virtual reality and video games. Thesebut not the encoder. This allows for cross-industry applications require new structures and tools to compress compatibility of the most critical component, i.e. the decoder, video data. Current tools used in video coding are not adequate and at the same time it allows for innovation and flexibility in to reduce the ever-increasing raw video data rates from video the design of the encoding process, for example, meeting capture with increasing spatial and temporal resolutions asrequirements of latency and exploiting the availability of well as wide colour gamut and high dynamic range. computational resources. Increasing popularity of video games and video conferencing have presented new requirements to have new tools and II. Ev o l u t io n o f Vid e o Co d e c s structures. Video compression techniques and tools aim to reduce the Compression technologies are advancing at a rapid pace. size of a video by eliminating redundancies. The reduction in These advancements are partly fuelled by advances in othersize results in lower bandwidth requirements and smaller areas such as artificial intelligence. Since MPEG-1, the storage needs. The tools that compress the video files or adoption of a new generation of codecs was observed at streams are called video codecs. Various codecs are being regular time intervals. The standardisation process, used in the broadcast and media industries. This section harmonisation and management of intellectual propertybriefly introduces some of them. among competitive contributors are key activities for the Codec, which stands for coder-decoder is hardware, success of a standard beyond its technical superiority. The software, or combination that applies compression algorithms collaborative work between International Telecommunicationto the video. The algorithms search for and remove Union-ITU and ISO/MPEG led to the most common and redundancies, thus reducing the amount of data; this can be affordable compression standard - Advanced Video Coding - the size of the file or bit rate if the video is streamed. There AVC H264 or MPEG part 10. A similar collaboration led to are four types of redundancy - spatial, temporal, statistical and 978-1-7281-8579-8/20/$31.00 ©2020 IEEE Authorized licensed use limited to: Universitaetsbibliothek der RWTH Aachen. Downloaded on April 06,2021 at 07:54:13 UTC from IEEE Xplore. Restrictions apply. colour space. For example, repetitive images or a background High-Efficiency Video Coding (HEVC), or H.265, which is that repeats throughout a video are temporal and spatial overtaking the previous version with the explosion of 4K redundant data. video content. The H.265 has much higher data reduction efficiency in compressing videos and makes streaming of Compression changes the original format of the video into ultra-high-definition 4K or 8K videos easier. a format supported by the video player or can be decoded by a standard decoder. The video codec determines the format of A. Video Coding Standards the video. There are two major classes of video compression, A video coding standard is a document describing a namely - lossy or lossless. Lossless compression, reduces bitstream structure and associated decoding method for video redundant data without affecting quality, where the process compression. The standard defines the output structure that an maintains data integrity and is fully reversible. Unfortunately,encoder should produce rather than defining the encoder itself. lossless compression gives limited reduction in bit rate or file This enables innovation in encoders. Generally, video coding size of an image or video. Lossy compression, on the other standards define a set of tools for compression. Some of the hand, is able to give further reductions in file size or bit rate, tools are optional to be implemented to create a standard at the expense of only being able to recover an approximationcompliant bitstream. However, decoders conforming to the of the original source video. This commonly exploits standard must implement a common minimal subset of perceptual coding (based on human perception) to make compression tools. approximations such that they are less noticeable. However, the compression loses some data permanently, which affects Since the MPEG-1 standard, a wide range of video codecs the quality of the reconstructed image or video. If heavily have been developed. The majority of these have been compressed, the losses become noticeable resulting in developed and standardised by the International annoying visible artefacts such as blockiness, blur, colourTelecommunication Union (ITU) Telecommunication sector bleeding, and banding in the reconstructed image or video. and International Standardization Organization/ International Electrotechnical Commission (ISO/IEC). Those standards Video can be compressed using two stages: intra-frame have been given an ISO/IEC identification under the Working and inter-frame. Intra-frame means that the codec compresses Group 29 Motion Picture Expert Group (MPEG) such as an individual frame. Inter-frame compression exploitsMPEG-1, MPEG-2, MPEG-4, MPEG-H, MPEG-I, MPEG-5. redundant data between successive video frames. These are Most of them have been developed jointly with ITU-T with deployed in various codecs such as AVC or H264, VP9, identification H26X, where X currently goes from 1 to 6. HEVC and VVC. Table 1 summarises some video coding standards. MPEG-1 Video was the first MPEG standard that was completed in 1993 [1]. MPEG-1 was extensively deployed TABLE I. MPEG v id eo c o d in g sta n d a r d s within Video CDs. MPEG-2 Video was completed in 1995 MPEG video coding standardYear Known as ITU-T and became popular with the introduction of DVDs and digital MPEG-2 part 2 1998 MPEG-2 H262 TV broadcasting [2]. The MPEG-4 Advanced Video Coding (AVC/H.264) was completed in 2003. The primary MPEG-4 part 10 2003 AVC H264 applications of AVC were