AVC/H.264 Advanced Video Coding Codec, Profiles & System

Ali Tabatabai ali.tabatabai@am.sony.com Media Processing Division Sony Electronics Inc.

October 25, 2005 Sony Electronics Inc Content

• Overview • Rate Distortion Performance • Codec Complexity • Profiles & Applications • Carriage over the Network • Current MPEG & JVT Activities • Conclusion

October 25, 2005 Sony Electronics Inc Video Coding Standards: A Brief History

October 25, 2005 Sony Electronics Inc Video Standards and JVT Organization • Two organizations dominate the video compression standardization activities: – ISO/IEC Moving Picture Experts Group (MPEG) • International Standardization Organization and International Electrotechnical Commission, Joint Technical Committee Number 1, Subcommittee 29, Working Group 11 – ITU-T Video Coding Experts Group (VCEG) • International Telecommunications Union – Telecommunications Standardization Sector (ITU-T, a United Nations Organization, formerly CCITT), Study Group 16, Question 6

October 25, 2005 Sony Electronics Inc Evolution of Video Compression Standards

ITU-T MPEG

Video Consumer Telephony Video CD . 1990 H.261

Digital TV/DVD MPEG-1 Video Conferencing

1995 H.262/MPEG-2 H.263 Object-based coding 1998 MPEG-4 H.264/MPEG-4 AVC

October 25, 2005 Sony Electronics Inc Joint Video Team ITU-T VCEG and ISO/IEC MPEG

• Design Goals – Simplified and Clean Design • No backward compatibility requirements. – Compression Efficiency • Average bit rate reduction of 50% compared to existing video coding standards (MPEG-2, MPEG-4, H.263). – Improved Network Friendliness • Clean and flexible interface to network protocols • Improved error resilience for Internet and mobile (3GPP) applications.

October 25, 2005 Sony Electronics Inc AVC/H.264 Standards Development Schedule

Defined Baseline & Main profiles.

Committee Final Draft International Draft Standard Standard

Dec01 May02 Jul02 Mar03 Aug03 Sep/Oct 04 Working Final Draft 1 AVC 3rd Edition Committee - New Profile Draft Definitions (FRExt) MPEG-4 Part 10 JVT starts ITU-T H.264 technical work Added streaming together. profile and redefined Main & Baseline profiles.

October 25, 2005 Sony Electronics Inc AVC/H.264 Coding Technologies

October 25, 2005 Sony Electronics Inc Architecture

AVC/H.264 CODEC Architecture Control Information Contains information needed SEI Messages Responsible for to control operation Video Coding Layer efficient coding of decoding of video Parameter Sets (VCL) of video data. coding layer data. Defines an generic format Slices for sending JVT video data over a network. Network Abstraction Layer (NAL) The layer that encapsulates the NAL Units JVT CODEC data for transport, Systems Layer controls timing, etc. October 25, 2005 Sony Electronics Inc AVC/H.264 Block Diagram

Integer - 4x4 Transform

Quantization Deblocking Filter Entropy Coding Inter/Intra Frame (CAVLC/CABAC) Encoder Prediction Buffer

I, P, B Frames Inter/Intra Frame Prediction Buffer Entropy Decoding (CAVLC/CABAC)

Deblocking Inverse Inverse Filter + 4x4 Transform Quantization Decoder October 25, 2005 Sony Electronics Inc Some AVC/H.264 Key Features

• Variable Block Sizes for Motion Compensation – 4X4, 8x8, 8x16… to 16X16 • Quarter-Pel Motion • Multiple Reference Pictures • Intra Prediction • Block Transform – Integer 4x4 Transform – No drift – 8x8 Transform for FRExt-Only • Adaptive Entropy Coding • De-Blocking Filter

October 25, 2005 Sony Electronics Inc Deblocking Filter

• Deblocked filter reduces visible blocking artifacts block edges. • Deblocking is an integral part of the coding loop, not a post-processing • Typically reduces bit rate by 5-10% compared to no filtering.

Before Deblocking After Deblocking October 25, 2005 Sony Electronics Inc AVC/H.264 Performance

Mobile & Calendar Sequence at CIF resolution Original Version of AVC (without FRExt) Mobile & Calendar Sequence at BT.601 (720x480) resolution

*G. Sullivan, P. Topiwala and A. Luthra, "The H.264/AVC Advanced Video Coding Standard: Overview and Introduction to the Fidelity Range Extensions," SPIE Conference on Applications of Digital Image Processing XXVII, vol. 5558, pp. 53-74, Aug. 2004.

October 25, 2005 Sony Electronics Inc Complexity of the Codec

• Smaller block sizes for motion compensation. • Longer filters for motion compensation • Multi-frame motion compensation • More complicated mode selection • Intra prediction • Deblocking filter • Arithmetic coding

October 25, 2005 Sony Electronics Inc AVC/H.264 Profiles

October 25, 2005 Sony Electronics Inc Application Areas of AVC/H.264

Entertainment Video (1 - 8+ Mbps, higher latency) Conversational H.32X Services (usually < 1Mbps, low latency) Streaming Services (usually lower bit rate, higher latency)

October 25, 2005 Sony Electronics Inc AVC/H.264 Profiles

• Three Profiles: Baseline, Main and Extended – Baseline: Low-latency, real-time video coding applications such as video conferencing. – Main: Broadcasting, packaged media, and high-end (e.g. digital cinema) applications. – Extended: IP-based video streaming applications over both wired and wireless networks.

Main

BaselineBaseline Extended

October 25, 2005 Sony Electronics Inc Current Profile Definition s n io it n fi High 444 e D le fi High 422 ro P t x e E r High 10 tu R c F u tr S g High in H R ig - Main h n io n O e ru T BaselineBaseline Extended

October 25, 2005 Sony Electronics Inc New Profiles in the FRExt

• The FRExt Amendment defines four new profiles: –High (HP) – High 10 (Hi10P) – High 4:2:2 (Hi422P) – High 4:4:4 (Hi444P) • All these profiles build upon the Main Profile (MP)

October 25, 2005 Sony Electronics Inc ISO/IEC: AVC Current Status

• AVC Advanced Video Codec (MPEG-4: Part10) – Current Status: • AVC/H.264 Specification: – AVC 3rd edition was released in July. • AVC Reference Software – The latest reference software version is JM 10.1 – Ongoing Work: • Work on FR-Ext, to provide support for 14-bit depth and improve the lossless coding performance. • Work on investigating a new “Low Complexity Profile”

October 25, 2005 Sony Electronics Inc AVC/H.264 Systems/Carriage

October 25, 2005 Sony Electronics Inc Carriage of AVC Content

• Overview – AVC Conceptual layers –NAL Units – Access Units • Carriage: – AVC over MPEG-2 Systems – AVC File Format – AVC over RTP

October 25, 2005 Sony Electronics Inc AVC Conceptual Layers

AVC Conceptual Layers

Video Coding Layer Video Coding Layer Encoder Decoder Control Data Control SEI Messages)

(Parameter Sets, (Parameter Sets, VCL-NAL Interface

Network Abstraction Layer Network Abstraction Layer Encoder Decoder

NAL Encoder Interface NAL Units NAL Units NAL Decoder Interface

Transport Layer AVC to AVC AVC to AVC to MPEG-2 to File Format RTP/IP Systems H.32x

Wired/Wireless Networks

October 25, 2005 Sony Electronics Inc Concept of a NAL (Network Abstraction Layer) unit

• The Network Abstraction Layer (NAL) defines a network friendly interface between the Video Codec a variety of transport layers (e.g. RTP/IP, MPEG-2 Systems, File Format). NAL Unit

NAL NAL Unit Header Unit Payload

forbidden_zero_bit nal_ref_idc nal_unit_type 1 bit 2 bits 5 bits Indicates the NAL unit type (e.g. SEI Messages, Parameter Sets, VCL data, etc.)

Indicates if the NAL unit is disposable.

If set, it indicates that the NAL unit is in error. • Packet oriented systems can employ NAL units directly • Bit and Byte stream oriented systems can employ the byte-stream version of NAL units, which are NAL units encapsulated by start codes.

October 25, 2005 Sony Electronics Inc Categorization of NAL units

• NAL units can be categorized in two types: – VCL NAL units: These NAL units contain data that represents the values of the samples in the video pictures. – Non-VCL NAL units: These NAL units contain any associated additional information such as parameter sets and supplemental enhancement information.

October 25, 2005 Sony Electronics Inc Definition of an Access Unit in AVC

• Access Unit: “ An access unit consists of one primary coded picture, zero or more corresponding redundant coded pictures, and zero or more non-VCL NAL units.”

Access Unit Parameter Sets SEI Primary Redundant End of End of Delimiter (SPS & PPS) Messages Coded Coded Picture Sequence Stream (if present) (if present) (if present) Picture (if present) (if present) (if present)

Access Unit

Video Slice Slice ES NALU NALU

Parameter Param Set ES NALU

October 25, 2005 Sony Electronics Inc Carriage of AVC on MPEG-2 Systems

An Amendment to MPEG-2 Systems is being prepared so that AVC video can be carried over the existing MPEG-2 infrastructure.

Mapping AVC Access Units into MPEG-2 PES Packets

AU Delimiter SPS PES Packet (with VUI if needed) Start Code Prefix PPS stream_id MPEG-2 SEI’s Map to Transport or PES Program Stream VCL NAL unit packet length VCL NAL unit PES Packet Data End of Sequence (if present) End of Stream (if present) AVC Byte Stream NAL units MPEG-2 PES Packets October 25, 2005 Sony Electronics Inc Storage using the AVC File Format

• This AVC File Format defines extensions to the ISO Base Media File format to provide support for the storage of AVC video data.

ISOISO Base Base Media Media FileFile Format Format (14496-12) (14496-12)

MPEG-4MPEG-4 MotionMotion JPEG2000 JPEG2000 FileFile Format Format (14496-14) (14496-14) FileFile Format Format (15444-3) (15444-3)

• They use the same file AVCAVC extension, i.e. .mp4 FileFile Format Format (14496-15) (14496-15) • The presence of ‘avc1’ as a major brand, implies usage of AVC specific extensions

October 25, 2005 Sony Electronics Inc Mapping AVC Bitstream -> AVC File Format

Metadata AU Delimiter Track 1 SPS (with VUI if needed) Track 2 PPS SEI’s VCL NAL unit AVC VCL NAL unit File Media Data

End of Sequence Sample Sample (if present) Sample Sample End of Stream (if present) Sample

Video ES Parameter ES Series of AVC Access Units

October 25, 2005 Sony Electronics Inc RTP Payload Format for H.264 Video (RFC 3984)

AVC Various Possible Mappings defined for Access Unit’s AVC over RTP

AU Delimiter RTP RTP Header NAL UnitA Header NAL UnitB SPS (with VUI if needed) Single NAL Unit RTP Packet: One NAL unit per RTP packet PPS

SEI’s RTP RTP VCL NAL unit Header NAL UnitA Header NAL UnitA VCL NAL unit Fragmentation Unit RTP Packet: One NAL unit over multiple RTP packets.

End of Sequence ¾ STAP-A: (if present) NAL units share same timestamp and are in valid decoding order. End of Stream ¾ STAP-B: (if present) NAL units share same timestamp and MAY be out of decoding order ¾ MTAP16: NAL units share different timestamps with a max offset of 16 ¾ MTAP24: NAL units share different timestamps with a max offset of 24

Aggregation Unit RTP Packets: October 25, 2005 Sony Electronics Inc Current MPEG & JVT Activities

October 25, 2005 Sony Electronics Inc ISO/IEC: MPEG

• Scalable Video Coding (SVC): – Goal: • This is being developed by the Joint Video Team (JVT) as an extension of AVC. • It intends to address the need for the reliable delivery of video to diverse clients over heterogeneous networks, by providing scalability (such as spatial, temporal and SNR/fidelity) with good compression efficiency. – Development Status: • Working Draft 3 as AVC/AMD1 & JSVM 3 Software released in July – Schedule: • Working Draft: WD 3 stage in July • CD: 2006 March • FCD: 2006 - July/Oct • FDIS: 2007 (expected)

October 25, 2005 Sony Electronics Inc ISO/IEC: MPEG • Wavelet Coding Exploration: – Background: • Initially this activity was part of the SVC development effort in MPEG. • Later on it forked off as a separate “Exploration Activity” within MPEG – Status: • A report based on the results of the Exploration Activity will be released during early 2006.

• Multi view Video Coding (MVC) – Background: • Multi-view Video Coding (MVC) is a key technology that has been in exploration status in MPEG since the past 1 year. • It serves a variety of applications such as FTV (free-viewpoint television), 3DTV (3D television) and surveillance. • It allows the viewpoint and view direction to be interactively changed (as in FTV) or multiple viewers can see different stereoscopic views consistent with their relative locations (as in 3DTV) – Status: • Call for Proposals was issued in July ‘05.

October 25, 2005 Sony Electronics Inc Conclusion

• New International Video Coding Standards – Joint Development by MPEG Group of ISO/IEC and VCEG Group of ITU-T • Improved Coding Efficiency & Flexibility – Introduction of New Coding Tools. – About 50% Compared to MPEG-2/4 – Use Over a Wide Range of Networks • Increased Complexity • Two Patent Pools – MPEG LA & Via Licensing • New Work Items – MPEG: Scalable Video Coding – Multiview Coding

October 25, 2005 Sony Electronics Inc Thank You for Your Attention......

October 25, 2005 Sony Electronics Inc