Lab Session 1 ( I H S L L M I L (With Supplemental Materials to Lecture

Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding

Outline Lab Session 1 z Review (ihS(with Supp lementa lMl Materi ilals to z Color Spaces in General Lecture 1) z Color Spaces for Image/Video Coding z Image/Video Formats Review z Perceptual Quality z MATLAB Exercises April 27 , 2009 • Reading and showing images and video sequences • Getting familiar with color spaces

Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding

Image and video encoding: A big picture Image and video encoding: A big picture Image and video decoding: A big picture Differential Coding Motion Estimation and Compensation A/D Conversion Context-Based Coding Color Space Conversion Predictive Predictive … Pre-Filtering Predictive Encoded Coding Coding Partitioning Coding Image/Video … Input Input Image/Video Post- Image/Video Post- Decoded Pre- Lossy Lossless Pre- Lossy Lossless Processing Processing Image/Video Post- Lossy Lossless Processing Coding Coding Processing Coding Coding Pre-Processing (Post-Filteri ng) (Post-filteri ng) PiProcessing CdiCoding CdiCoding

Quantization Entropy Coding Transform Coding Dictionary-Based Coding Encoded Model-Based Coding Run-Length Coding Encoded Image/Video … … Image/Video

3 4 5

1 Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding

A zoo of standards An example: MPEG-video codec MPEG-video encoding process

z Image coding standards • JPEG • JPEG2000 • JBIG/JBIG2 • JPEG XR (Microsoft HD Photo) • … z Video coding standards • MPEG-1/2 • MPEG-4Part24 Part 2 • H.264/MPEG-4 Part 10 AVC • SMPTE VC-1 (Microsoft WMV9) • … z International standardization bodies • ISO, IEC, ITU, SMPTE, ANSI, … • MPEG, JPEG, JVT, VCEG, … 6 7 8

Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding

Some terms Human visual system: Color perception Cone z Luminance = Brightness = Lightness = Intensity = Value z Hue: z Saturation = Colorfulness = Chroma = Vividness = Purity z Chromaticity = Hue + Saturation ⇒ It is a 2-tuple. Light Color Spaces z Chromaticity Diagram: a 2-D diagram showing chromaticities z Color = Luminance + Chromaticity ⇒ So, it is a 3-tuple. z Chrominance (Chroma) = Color Difference z Color Space: a 3-D space of colors z Color Mixing Systems (light) vs. Color Appearance Systems (perception) z Gamut: range of colors (in a color space) Anatomy of human retina z JND: Just-Noticeable Difference (50% accuracy) Retinal neurons Rod

10 11

2 Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding

CIE 1931 XYZ color space CIE 1976 uniform color spaces CIE 1931 RGB color space

CIE = Commission internationale de l'éclairage = International Commission on Illumination z Problem: CIE XYZ color space is nonlinear in term of perceptual color differences. b*

CIE 1931 RGB Color matching Color matching functions of the functions CIE standard observer CIE 1931 xyY color space CIE 1931 RGB color space chromaticity diagram CIELUV (CIE 1976 L*u*v*) color space CIELAB (CIE 1976 L*a*b*) color space chromaticity diagram 12 13 14

Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding

sRGB color space (IEC 61966-2-1:1999) More RGB color spaces… HSV/HSL (HSI/HSB) color spaces

HSV color space HSL color space Adobe 1998 RGB color space ProPhoto RGB color space (by Kodak) 15 16 17

3 Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding

Munsell color system Gamma correction

Color Spaces for Image/Video Coding

y=xr 18 20

Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding

Luma (Y’) vs. Luminance (Y) Y’UV/Y’CbCr color space Color conversion in image/video encoding

z Roughly, U=C =B-Y, z Luma = Luminance after gamma correction b Pre-Processing V=Cr=R-Y Input Image/Video z RGB ⇒ R’G’B’ z But, Y ≠ Y’ and Y’UV Color Space A/D ≠ Y’C C RGB R’G’B’Conversion Y’UV/ Conversion z Y=kr*R+kg*G+kb*B ⇒ b r Y’PbPr z Practically, YUV = Y’CbCr Y’=kr’*R’+kg’*G’+kb’*B’ YCbCr = Y’UV = Chroma Subsampling Y’CbCr z In practice, many people (even some z Y’PbPr = Y’CbCr for analog signal …001110101001… experts!) don’t know the difference between Other Parts of Encoding Process z YCC = Y’CbCr Encoded Y and Y’. Image/Video U-V plan (Y’=0.5) 21 22 23

4 Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding

Color conversion: R’G’B’ => Y’CbCr Why Y’CbCr? Chroma subsampling formats

z R’,G’,B’∈[[,],0,1], kr, kb∈((,),0,1), kr+kb<1 z Human eyes are less sensitive to A 4X2 Image z Y’=kr*R’+(1-kr-kb)*G’+kb*B’∈[0,1] chrominance. P(1,1) P(1,2) P(1,3) P(1,4) Pb=0.5*(B’-Y’)/(1-kb)∈[-0.5,0.5] P(2,1) P(2,2) P(2,3) P(2,4) Pr=0.5*(R’-Y’)/(1-kr)∈[-0.5,0.5] Ø z Values of k and k can be different! Y Sampling C Sampling C Sampling b r z We can sample chroma (CbCr) less YC C b r b r Locations Locations Locations • ITU-T BT.601 (SDTV): kr=0.229, kb=0.114 frequently than the luma (Y’). 4:4:4 4 4 4 ALL ALL ALL • ITU-T BT .709 (HDTV): kr=0.2126 , kb=007220.0722 P(1,1) P(1,3) P(1,1) P(1,3) • ANSI/SMPTE 240M-1995 (HDTV): kr=0.212, kb=0.087 Ø 4:2:2 4 2 2 ALL P(2,1) P(2,3) P(2,1) P(2,3) z Y’P P ⇒ Y’C C (taking MPEG-2 as an example) b r b r z We achieve the goal of data compression. 4:2:0 4 1 1 ALL P(1,1) P(1,3) P(1,1) P(1,3) • Y’=219*Y’+16∈[16,235] 4:1:1 4 1 1 ALL P(1,1) P(2,1) P(1,1) P(2,1) Cb=224*Pb+128∈[16,240] Cr=224*Pr+128∈[16,240] 24 25 26

Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding

Spatial display resolutions (W×H) Spatial resolutions: Images

z No standards z For research purpose: 2n×2n, such as Image/Video Formats 128×128, 256×256, 512×512, 1024×1024

28 29

5 Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding

Some test images for research Digital video formats YUV video file format (.yuv/.cif/.qcif/.sif/…)

z CIF = Common Intermediate Format (()since H.261) z Planar formats • CIF (Full CIF = FCIF) = 352×288 • YUV = I420 = IYUV (4:2:0) • QCIF (Quarter CIF) = 176×144 • YV12 (4:2:0) • SQCIF (Sub Quarter CIF) = 128×96 • 4CIF = 4×CIF = 704×576 • YV16 (4:2:2) • 16CIF = 4×4CIF = 1408×1152 • … z SIF = Source Inpp(ut Format (since MPEG-1) z Packed formats • 625/50 (TV: PAL/SECAM) = 352×288/360×288 • UYVY = UYNV = Y422 (4:2:2) • 525/59.94 (TV: NTSC) = 352×240/ 360×240 • YUY2 = YUNV = V422 = YUYV (4:2:2) • Sub-SIF (Computers) = 320×288 or 384×288 • … z More info available at http://www.fourcc.org/yuv.php 30 31 32

Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding

YUV4MPEG2 format (.y4m) Image and video coding: Quality issue

Quality z File Header MtMeter ics • ‘YUV4MPEG2’ + a number of parameters + 0x0A • Parameters Input Decoded • (Mandatory) Width, height and frame rate: ‘ Wxxx Hyyy Fa:b’ image/video image/video • (Optional) Interlacing: ‘ Ip’ (progressive), ‘ It’ (top field first), ‘ Ib’ (bottom Perceptual Quality field first), ‘ Im’ (mixed mode, detailed in frame headers) Visual Quality • (Optional) Aspect ratio: ‘ Aa:b’ Measurement • (Optional) Color space (Chroma format): ‘ C4xx...’ • (il)(Optional) Comment: ‘ X….’ Encoder Decoder z Frames • Frame Header Encoded • ‘FRAME’ + a number of parameters (optional) + 0x0A image/video • Frame (YUV planar format)

33 35

6 Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding

Visual quality measurement: Subjective Visual quality measurement: Objective Visual quality measurement: Objective

z DSIS (Double Stimulus Impairment Scale) z Two imaggges: the original one f(x,y) and the z DSCQS (Double Stimulus Continuous decoded one f’(x,y) Quality Scale) z MSE = Mean Squared Error 1 2 (f0(x, y) f(x, y)) z SSCQE (Single Stimulus Continuous Quality MN − x,y X Evaluation) z SNR = Signal-to-Noise Ratio The original image PSNR=32.7 dB PSNR=37.5 dB (f (x, y))2 z StthdhbSome measurement methods have been x,y 0 10 log10 2 Ã x,yP(f0(x, y) f(x, y)) ! standardized: ITU-R BT.500, ITU-R BT.710, − Research Question: PSNR is not good enough to perfectly z PSNR =P Peak Signal-to-Noise Ratio ITU-T P.910 MNL2 reflect visual quality, more advanced metrics considering 10 log10 2 HVS are wanted. Some objective metrics are being Ã x,y(f 0(x, y) f(x, y)) ! − standardized by the ITU-T. P 36 37 38

Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding Shujun LI (李树钧): INF-10845-20091 Multimedia Coding

Reading and Showing Image/Video Getting familiar with color spaces

z Readinggg an image z Showing YUV planes of a video • f=imread(‘test.bmp’);

z Showing an image f=yuvread(‘test.cif',1); • imshow(f); subplot(2,2,1:2); MATLAB Exercises imshow(f.Y); z Reading a frame from a YUV video subplot(2,2,3); • f=yuvread(‘test.cif’,1); imshow(f.U); subplot(2,2,4); z Showing a video frame imshow(f.V); • imshow(ycbcr2rgb(YUV4XX2YUV444(f))); z Playing a YUV video z Trying to read an image and convert it to • yuvplay(‘test.cif’); YUV color space.

40 41