Overview: image and video coding standards n JPEG n MPEG-1 and MPEG-2 n H.261 and H.263 n Progress in video compression
Bernd Girod: EE368b Image and Video Compression Coding Standards no. 1
JPEG: background
n JPEG = Joint Photographic Experts Group: Joint standards committee of ITU-T and ISO n Flexible standard for monochrome and color image compression: “Digital Compression and Coding of Continuous-tone Still Images” n Intraframe coding scheme, optimized for still images n Flexible picture size n Coding of color components separately, arbitrary color spaces possible, best compression for Y/R-Y/B-Y n Variable compression ratio n Compression 24:1 for ITU-R 601 images without loss of quality
Bernd Girod: EE368b Image and Video Compression Coding Standards no. 2
1 JPEG derived industry standards
n JFIF (JPEG File Interchange Format,
Bernd Girod: EE368b Image and Video Compression Coding Standards no. 3
JPEG: baseline algorithm
weighting / Image Blockwise uniform Huffman in 8x8 DCT quantization Coding
Blockwise Image Inverse inverse Huffman out weighting 8x8 DCT Decoding
Bernd Girod: EE368b Image and Video Compression Coding Standards no. 4
2 JPEG: quantizer stepsize
n Different weighting matrices are standardized, adapted to human visual contrast sensitivity n Example: Inverse weighting for ITU-R 601 images
Luminance Chrominance
16 11 10 16 24 40 51 61 17 18 24 47 99 99 99 99 12 12 14 19 26 58 60 55 18 21 26 66 99 99 99 99 14 13 16 24 40 57 69 56 24 26 56 99 99 99 99 99 14 17 22 29 51 87 80 62 47 66 99 99 99 99 99 99 18 22 37 56 68 109 103 77 99 99 99 99 99 99 99 99 24 35 55 64 81 104 113 92 99 99 99 99 99 99 99 99 49 64 78 87 103 121 120 101 99 99 99 99 99 99 99 99 72 92 95 98 112 100 103 99 99 99 99 99 99 99 99 99
Bernd Girod: EE368b Image and Video Compression Coding Standards no. 5
JPEG: coding of DCT coefficients
n Differential coding of DC coefficient (0,0) n Zig-zag-scan and run-level-coding of AC coefficients n Two options for Huffman coding:
l with predefined code tables (one-pass system)
l code tables optimized for individual image (two-pass system)
Bernd Girod: EE368b Image and Video Compression Coding Standards no. 6
3 JPEG-2000
n New generation JPEG standard, recently finalized. n Discrete wavelet transform n Embedded scalar quantization with deadzone n Entropy-coding: bit-plane arithmetic coding with „context modeling“ n Typically 30% lower bit-rate than old JPEG at the same image quality
Bernd Girod: EE368b Image and Video Compression Coding Standards no. 7
Video coding standards
MPEG-2 digital TV 2-6 Mbps
ITU-R 601 166 Mbit/s H.261 ISDN picture 64 kbps phone
H.263 PSTN < 28.8 kbps picture phone
Bernd Girod: EE368b Image and Video Compression Coding Standards no. 8
4 Generic encoder block diagram for H.261, H.263, and MPEG
Control
Video Channel signal Intraframe S e DCT coder - Intraframe Decoder
Motion compensated predictor Decoder
Bernd Girod: EE368b Image and Video Compression Coding Standards no. 9
Image formats
sub QCIF 176 x 144
QCIF
352 x 288 ITU-R 601 CIF
Bernd Girod: EE368b Image and Video Compression Coding Standards no. 10
5 ISO MPEG
n MPEG-1 Standard (1991)
l Target bit-rate about 1.5 Mbit/s
l Typical image format CIF, no interlace
l Frame rate 24 ... 30 fps
l Main application: video storage for multimedia (e.g., on CD-ROM) n MPEG-2 Standard (1994)
l Extension for interlace, optimized for TV resolution (NTSC: 704 x 480 Pixel)
l Image quality similar to NTSC, PAL, SECAM at 4 - 8 Mbit/s
l HDTV at 20 Mbit/s
Bernd Girod: EE368b Image and Video Compression Coding Standards no. 11
MPEG: hierarchical syntax I
n "Video Sequence" n "Group of Pictures" = “GOP“, GOP structure is very flexible
I-Picture P-Picture P-Picture
B-Pictures
time
Bernd Girod: EE368b Image and Video Compression Coding Standards no. 12
6 MPEG: hierarchical syntax II n "Picture": N x M luminance samples, N/2 x M/2 Cb- and Cr- samples n "Slice": Sequence of Macroblocks n "Macroblock” 16x16 luminance 8x8 Cb- 8x8 Cr- samples samples samples
0 1 4 5
2 3
n "Block": 8x8 samples
Bernd Girod: EE368b Image and Video Compression Coding Standards no. 13
MPEG: coding of I-pictures
n I-pictures: intraframe coded n 8x8 DCT n Arbitrary weighting matrix for coefficients n Differential coding of DC-coefficients n Uniform quantization n Zig-zag-scan, run-level-coding n Entropy coding n Unfortunately, not quite JPEG
Bernd Girod: EE368b Image and Video Compression Coding Standards no. 14
7 MPEG: coding of P-pictures
n Motion-compensated prediction from an encoded I-picture or P-picture (DPCM) n Half-pel accuracy of motion compensation, bilinear interpolation n One displacement vector per macroblock n Differential coding of displacement vectors n Coding of prediction error with 8x8-DCT, uniform threshold quantization, zig-zag-scan as in I-pictures
Bernd Girod: EE368b Image and Video Compression Coding Standards no. 15
MPEG: coding of B-pictures
n Motion-compensated prediction from two consecutive P- or I-pictures
l either • only forward prediction (1 vector/macroblock)
l or • only backward prediction (1 vector/macroblock)
l or • Average of forward and backward prediction = interpolation (2 vectors/macroblock) n Half-pel accuracy of motion compensation, bilinear interpolation n Coding of prediction error with 8x8-DCT, uniform quantization, zig-zag-scan as in I-pictures
Bernd Girod: EE368b Image and Video Compression Coding Standards no. 16
8 ITU-T Rec. H.261
n International standard for ISDN picture phones and for video conferencing systems (1990)
n Image format: CIF (352 x 288 Y samples) or QCIF (176 * 144 Y samples), frame rate 7.5 ... 30 fps
n Bit-rate: multiple of 64 kbps (= ISDN-channel), typically 128 kbps including audio.
n Picture quality: for 128 kbps acceptable with limited motion in the scene
n Stand-alone videoconferencing system or desk-top videoconferencing system, integrated with PC
Bernd Girod: EE368b Image and Video Compression Coding Standards no. 17
ITU-T Rec. H.263 n International standard for picture phones over analog subscriber lines (1995) n Image format usually CIF, QCIF or Sub-QCIF, frame rate usually below 10 fps n Bit-rate: arbitrary, typically 20 kbps for PSTN n Picture quality: with new options as good as H.261 (at half rate) n Software-only PC video phone
or TV set-top box Example: 8x8 ViaTV phone VC 105 n Widely used as compression engine for Internet video streaming n H.263 is also the compression core of the MPEG-4 standard
Bernd Girod: EE368b Image and Video Compression Coding Standards no. 18
9 Motion compensation H.261 vs H.263
n H.261 (1990): integer-pel accuracy, loop filter, 1 motion vector per MB n H.263 (1995): half-pel accuracy, no loop filter, 1 motion vector per MB n H.263 option „Advanced Prediction Mode“
l overlapped block motion compensation (OBMC),
l switch between 1 or 4 motion vectors per MB n H.263 option „PB frames“
Bernd Girod: EE368b Image and Video Compression Coding Standards no. 19
Performance of H.263 and H.261
1) 2) 3) 34 4) 5) 32
30
28 1) H.263 2) H.263 w/o options PSNR [dB] 26 3) H.261 4) H.263 w/o options, integer-pel ME 24 5) H.261 w/o loop filter
32 64 128 rate [kbps]
Bernd Girod: EE368b Image and Video Compression Coding Standards no. 20
10 Performance of H.263 SAC mode
36 1) 35 2) 34 33 32 31 30 29 28 1) SAC-mode 2) w/o options 27 26 0 32 64 128 rate [kbps]
Bernd Girod: EE368b Image and Video Compression Coding Standards no. 21
H.263: overlapped block motion compensation (OBMC)
remote luminance Current block luminance block (8x8)
M A C
remote luminance remote luminance R block block O B L O remote luminance C block K
Bernd Girod: EE368b Image and Video Compression Coding Standards no. 22
11 H.263: OBMC weights
4 5 5 5 5 5 5 4 for MV of current luminance block 5 5 5 5 5 5 5 5 5 5 6 6 6 6 5 5 5 5 6 6 6 6 5 5 5 5 6 6 6 6 5 5 5 5 6 6 6 6 5 5 2 2 2 2 2 2 2 2 for remote MV 5 5 5 5 5 5 5 5 1 1 2 2 2 2 1 1 4 5 5 5 5 5 5 4 1 1 1 1 1 1 1 1 of top/bottom 1 1 1 1 1 1 1 1 luminance block 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 2 2 1 1 1 1 2 2 1 1 2 2 2 2 2 2 2 2 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 1 1 1 1 2 2 2 2 1 1 1 1 2 2 2 2 1 1 1 1 2 2 for remote MV of left/right 2 2 1 1 1 1 2 2 2 1 1 1 1 1 1 2 luminance block
Bernd Girod: EE368b Image and Video Compression Coding Standards no. 23
Performance of H.263 AP mode
36 1) 35 2) 34 33 32 31 30 29 1) AP-mode 28 2) w/o options 27 26 0 32 64 128 rate [kbps]
Bernd Girod: EE368b Image and Video Compression Coding Standards no. 24
12 H.263: PB-frames
forward prediction
P B P
bidirectional bidirectional prediction prediction
PB-Frame
Bernd Girod: EE368b Image and Video Compression Coding Standards no. 25
Performance of H.263 PB-mode
36 1) 35 3a) 2) 34
33 3b) 32 31 1) w/o options (6.25 fps) 30 2) w/o options (12.5 fps) PSNR [dB] 29 3) PB-mode (12.5 fps) 28 a) P-frames b) B-frames 27 26 0 32 64 128 rate [kbps]
Bernd Girod: EE368b Image and Video Compression Coding Standards no. 26
13 Video compression progress
• Intraframe coding: only spatial correlation exploited Complexity è DCT [Ahmed, Natarajan, Rao 1974], JPEG [1992] increases
• Conditional replenishment, DPCM, scalar quantization è H.120 [1984]
• Frame difference coding è H.120 Version 2 [1988]
• Motion compensation: integer-pel accurate displacements è H.261 [1991]
• Half-pel accurate motion compensation è MPEG-1 [1993], MPEG-2/H.262 [1994]
• Variable block-size motion compensation è H.263 [1996], MPEG-4 [1999]
Bernd Girod: EE368b Image and Video Compression Coding Standards no. 27
Video compression progress
Half-pel Frame Variable Integer-pel motion difference block size motion compensation coding motion compensation 40 (MPEG-1 1993) (H.120 1988) compensation (H.261 1991) 38 (H.263 1996) Foreman 36 Conditional 10 Hz, QCIF PSNR 34 ~40 % Replenishment 100 frames encoded [dB] (H.120) 32 Intraframe DCT coding 30 (JPEG)
28 ~20 %
26 ~30 %
24 Bit Rate [kbps] 0 100 200 300 400 500 600
Bernd Girod: EE368b Image and Video Compression Coding Standards no. 28
14 Video compression progress
Variable block size 42 motion compensation Conditional (H.263 1996) 40 Replenishment (H.120) Mother & Daughter 38 PSNR 10 Hz, QCIF [dB] 100 frames encoded 36
Intraframe 34 ~60 % DCT coding ~70 % (JPEG) 32
30
Bit Rate [kbps] 28 0 50 100 150 200 250 300 350
Bernd Girod: EE368b Image and Video Compression Coding Standards no. 29
Video compression progress
Variable Integer-pel block size 40 motion motion compensation compensation 38 (H.261 1991) (H.263 1996) 36 Mobile &Calendar
PSNR 34 10 Hz, QCIF [dB] ~ 35 % 100 frames encoded 32 ~ 40 %
30 Intraframe DCT coding 28 (JPEG) 26
24
22 Bit Rate [kbps] 0 200 400 600 800 1000 1200 1400
Bernd Girod: EE368b Image and Video Compression Coding Standards no. 30
15