Fundamentals of Digital Video Compression

FUNDAMENTALS OF DIGITAL VIDEO COMPRESSION

MPEG Compression Streaming Video and Progressive Download

• Basic concepts of MPEG compression

• Types of MPEG

• Applications of MPEG

• What GOP (group of pictures) in MPEG is

• How GOP settings affect MPEG video file size

• How GOP settings affect MPEG picture quality

• What streaming video and progressive download are

• Moving Pictures Experts Group Committee who derives standards for encoding video

• Allow high compression

• MPEG-1, MPEG-2, MPEG-4

• NOT MP3 (which is audio format)

• Intended for HDTV

• HDTV specifications was merged into MPEG-2

• Video quality comparable to VHS

• Originally intended for Web and CD-ROM playback

• Frame sizes up to 352 × 240 pixels

• Video format for VCD (VideoCD) before DVD became widespread

• Supports DVD-video, HDTV, HDV standards

• For DVD video production: Export video into DVD MPEG-2 format

• For HDV video production: Export video into HDV's MPEG-2 format

• Newer standard of MPEG family

• Different encoding approach from MPEG-1 and MPEG-2 (will discuss after MPEG-1 and MPEG-2 compression in this lecture)

• Neighboring frames are very similar • This property is called Temporal redundancy .

• MPEG compression exploits temporal redundancy to reduce video file size • by looking for motion difference from one frame to the next • This technique is called motion compensation .

1. An video frame image is read in as a reference frame 2. Read the next video frame image 3. Compare the image content between this current frame ( target frame ) with the reference frame • one block of pixels at a time

Case 1: If a pixel block is identical at the same location in both frames: • No need to encode the target frame's • Just save an instruction to refer to the block in the reference image— requires less space than encoding the whole pixel block

Case 2: If a pixel block at the same location in both frames is not identical: • Search for the reference image for a match (because the content may be moved to another location) • Case 2a: No match is found • Case 2b: A match is found

• The whole pixel block is encoded

• Thus, no saving in file size

• The displacement information of the block is saved.

• Displacement information: – A 2-dimensional value, indicating how much the block moves horizontally and vertically from the reference frame to the target frame – Called motion vector – Much smaller size than encoding the whole pixel blocks (thus the saving in file size)

Let's consider the first 4 frames of a video

Frame 1 Frame 2 Frame 3 Frame 4

The square grid cell in a frame represents a block of pixels.

Frame 1 is read in.

Frame 1 Frame 2 Frame 3 Frame 4

Frame 1 is used as a reference frame.

Frame 1 Frame 2 Frame 3 Frame 4

The red color highlights the pixel blocks that are encoded. The whole frame of a reference frame is encoded. No saving in file size.

Frame 2 is read in.

Frame 1 Frame 2 Frame 3 Frame 4

Frame 1Frame Frame 2 Frame 3 Frame 4

The blue color highlights some pixel blocks that are found in Frame 1.

The yellow color highlights some pixel blocks that are found in Frame 1 (but later in the process they are© 2016found Pearson in Education,the next Inc., refere Hoboken,nce NJ. All frame.) 20 rights reserved. Illustration of the Ideas

Frame 3 is read in.

Frame 1Frame 2 Frame 3 Frame 4

The process is repeated similar to that for Frame 2.

Frame 1 Frame 2

Frame 1

Frame 2

Frame 1

Frame 2

Frame 1

Frame 2

Frame 1

Frame 2

The red line indicates the displacement of the pixel block. © 2016 Pearson Education, Inc., Hoboken, NJ. All 28 rights reserved. SO HOW ARE REFERENCE FRAMES CHOSEN? We will need to understand an important concept of group of pictures (GOP ) first.

To understand:

• the terminology related to GOP because

– it is used by MPEG which in turns used by DVD video and HDTV

– it comes up in video production application dialog boxes when you export video into MPEG format

• the impact of the GOP parameters on video file size and quality

• why some video editing programs may not support frame accurate editing of MPEG

• Specifies the grouping structure of frames

• Frame types in a structure: – I-frames – P-frames – B-frames

• An MPEG video contains 1 or more repeating GOPs

• Each GOP in a video has a fixed number of frames

• The number is the N parameter

• Example: • DVD-compliant MPEG-3: N = 15 • HDV: N = 15

• Stands for Intraframes

• Encoded only using the information within the frame-- intracoding . – Use spatial compression • Similar to JPEG compression – No temporal compression

• Least compressed of the three types

• Reference frame in the previous 4-frame example

• Each GOP starts with an I-frame

• Each GOP has only one I-frame

• Stands for predicted frames

• Encoded using the information from the previous I- or P-frames as the reference frame if a match of a pixel block is found

• The 2 nd and 3 rd frames in the previous 4-video example

• Stands for bidirectional frames

• Between I- and P-frames

• Encoded using information from the previous and subsequent I- and/or P-frame as the reference frames

DVD-compliant MPEG-2:

I B B P B B P B B P B B P B B

N = 15 i.e., number of frames in a GOP = 15

DVD-compliant MPEG-2:

I B B P B B P B B P B B P B B

M = 3 i.e., number of frames between non-B-frames plus one = 2+1 = 3

Frame 1 Frame 2 Frame 3 Frame 4

Frame 1 Frame 2 Frame 3 Frame 4 B-frame: Most compressed. © 2016 Pearson Education, Inc., Hoboken, NJ. All Based on changes from 45 Frames 1 and rights4 reserved. Example GOP: N = 15, M = 3 I B B P B B P B B P B B P B B

• Shorter GOP (i.e., lower N): – larger file size

• Rationales: – A MPEG-2 consists of a repeating GOP structure – Each GOP contains one I-frame – I-frames are the least compressed among the three types of frames, and thus take up more storage space – Shorter GOP means more GOPs in a MPEG-2 – More GOPs means more I-frames in a MPEG-2

• Shorter GOP (i.e., lower N): • better picture quality

• Rationales: • I-frames are compressed based on the frame information, rather than predicted from other frames

• Some video editing programs may not support frame-accurate video editing of MPEG-2 because it is more complex than other video format

• Rationales: – The information for a P-frame depends on the information of its previous I-frame. – A B-frame depends on the information of its previous and subsequent I- or P-frames. – Thus, it is more complex to trim out frames from a MPEG-2 than other video format.

• Cover a wide range of data rate

• Low end of the data rate: Video playback on mobile devices

• High end of the data rate: – HDTV – Handheld and portable game devices (e.g., Sony PSP)

• Uses media objects – A scene may contain separate media objects

• Not the frame-based coding used in MPEG-1 and MPEG-2

• Conventional frame-based video can be converted to MPEG-4 because a frame can be treated as a media object--a degenearted case.

• Also known as MPEG-4 Part 10 • Based on MPEG-4 codecs • Improved compression to have smaller file size, better visual quality

• Play from disk • Play over a network

• An entire clip needs to be on disk before it can be played

• Played from hard drive, CD, or DVD

• The video can be played while it is being downloaded

• Can be played from disk

• Streaming video

• Progressive download

• Play video as soon as enough data has arrived

• Examples: • Streaming QuickTime • Real Video • Window Media Video (WMV)

• Require a streaming server to stream video

• Allow saving several different compression levels of a video in a single file

• The server chooses the compression level to match the speed of network connection

• Buffering: Wait time depends on network speed

• Play video as soon as enough data has arrived

• Does not require special servers

• Example: • QuickTime fast-start • Created by saving the QuickTime movie as self-contained using QuickTime Pro