Video Digitization and Format Conversion

Yao Wang Polytechnic University, Brooklyn, NY11201 Outline

• How to capture digital video – Direct acquisition: digital video camcorder – Digitizing analog raster video – Digitizing analog film video (frame-based) • Chrominance subsampling • Color coordinate • Digital video formats and applications • Video format conversion – Deinterlacing – NTSC <-> PAL/SECAM – SDTV <-> HDTV

• Digitization = Sampling + Quantization

Sampling Quanti- zation xc(t) x[n] = xc(nT) xn[]

Sampling Quantization Period Interval T Q

1 T=0.1 Q=0.25

0.5

-0.5

-1

0 0.2 0.4 0.6 0.8 1

Progressive Frame Interlaced Frame Horizontal retrace Field 1 Field 2

Vertical retrace How to convert a raster video to digital signal?

• Digitization = Sampling + Quantization • Sample the raster waveform = Sample along the horizontal direction • Apply the above sampling on Y,I,Q rasters separately • Quantize Y,I,Q samples to 8 bits each • How should we select the sampling rate? – Must be faster than the Nyquist rate (twice the highest frequency) – For the samples to be aligned vertically, the sampling rate should be multiples of the line rate – Horizontal sampling interval = vertical sampling interval (square pixel) – Total sampling rate equal among different systems (525/30 vs 625/25 = = = fs 858 fl (NTSC) 864 fl (PAL/SECAM) 13.5 MHz

858 pels 864 pels 720 pels 720 pels

Ac tive Ac tive 480 lines 525 525 lines Area Area 625 lines 576 lines

122 16 132 12 pel pel pel pel

525/60: 60 field/s 625/50: 50 field/s

Pixels in non-shaded areas correspond to samples taken during horizontal/vertical retrace * BT.601 is formerly known as CCIR601

• YCbCr = Digital equivalent of YUV – Cb = U = B-Y, Cr = V = R- Y – Each represented in 8 bits

Y_d = 0.257 R_d + 0.504 G_d + 0.098 B_d + 16, C_b = -0.148 R_d - 0.291 G_d + 0.439 B_d + 128, C_r = 0.439 R_d -0.368 G_d - 0.071 B_d + 128,

R_d = 1.164 Y_d’ + 0.0 C_b’+ 1.596 C_r’, G_d = 1.164 Y_d’ - 0.392 C_b’ -0.813 C_r’, B_d = 1.164 Y_d’ + 2.017 C_b’ + 0.0 C_r’,

Y_d’=Y_d -16, C_b’=C_b-128, C_r’=C_r-128

• Human eye is less sensitive to chrominance • Chrominance signal has lower bandwidth – Can be sampled at lower rate than Y

4:4:4 4:2:2 4:1:1 4:2:0 For every 2x2 Y Pixels For every 2x2 Y Pixels Fo r ev er y 4x 1 Y Pi x els For every 2x2 Y Pixels 4 Cb & 4 Cr Pixel 2 Cb & 2 Cr Pixel 1Cb& 1 CrPixel 1Cb&1CrPixel (No subsampling) (Subsampling by 2:1 (Subsampling by 4:1 (Subsampling by 2:1 both horizontally only) horizontally only) horizontally and vertically)

Y Pixel Cb and Cr Pixel

I,Q raster are I,Q raster are I,Q raster are Vertical down- sampled at the sampled at ½ sampled at ¼ sampling from same rate as Y rate rate 4:2:2

Video Format Y Size Color Frame Rate Raw Data Rate Sampling (Hz) (Mbps)

HDTV Over air. cable, satellite, MPEG2 video, 20-45 Mbps SMPTE296M 1280x720 4:2:0 24P/30P/60P 265/332/664 SMPTE295M 1920x1080 4:2:0 24P/30P/60I 597/746/746

Video production, MPEG2, 15-50 Mbps BT.601 720x480/576 4:4:4 60I/50I 249 BT.601 720x480/576 4:2:2 60I/50I 166

High quality video distribution (DVD, SDTV), MPEG2, 4-10 Mbps BT.601 720x480/576 4:2:0 60I/50I 124

Intermediate quality video distribution (VCD, WWW), MPEG1, 1.5 Mbps SIF 352x240/288 4:2:0 30P/25P 30

Video conferencing over ISDN/Internet, H.261/H.263/MPEG4, 128-384 Kbps CIF 352x288 4:2:0 30P 37

Video telephony over wired/wireless modem, H.263/MPEG4, 20-64 Kbps QCIF 176x144 4:2:0 30P 9.1

• Needed for various applications – Deinterlacing (interlaced -> progressive) – NTSC (525/50 ) <-> PAL (625/60) –SDTV <-> HDTV • Using digital interpolation filters! – Filtering within one frame (spatial interpolation) – Filtering across frames for the same pixel (temporal interpolation) – Spatial-temporal filtering

• Needed for – Display interlaced sequence in progressive monitor with twice frame rate – Convert old TV material to SDTV/HDTV with good quality – Convert between NTSC and PAL with good quality –…

From [Wang02] © Yao Wang, 2003 Video digitization and format conversion 15 Deinterlacing by Temporal Interpolation

• Field merging: – Merge two fields into one frame, repeats the frame twice – Equivalent to temporal interpolating using replication filter • Field averaging – Averaging corresponding lines in preceding and following fields – Equivalent to temporal interpolation using linear averaging filter – Require two frame store ! • Both methods have problem when there are significant motion between two fields, especially moving vertical lines • But can recover spatial details on stationary objects very well

Frame 1 Frame 2

100 50 50 100 100 50

100 100 50 100 100 100

100 50 50 100 100 100

100 100 50 100 100 100

100 50 50 100 100 50

100 100 50 100 100 100

100 50 50 100 100 100

100 100 50 100 100 100

100 50 50 100 100 50

Field 1 Field 2 Field 1 Field 2

Frame 1 Frame 2

100 50 50 100 100 50

100 100 50 100 100 100

100 50 50 100 100 100

100 100 50 100 100 100

100 50 50 100 100 50

100 50 50 100 75 50 100 100 50

100 100 50 100 100 100

100 50 50 100 75 75 100 100 100

100 100 50 100 100 100

100 50 50 100 75 50 100 100 50

Field 1 Field 2 Field 1 Field 2 Missing line3 = (field1 of frame1+field1 of frame2)/2 © Yao Wang, 2003 Video digitization and format conversion 18 Deinterlacing by Spatial Interpolation

• Line averaging within same field –D=(C+E)/2 – Equivalent to vertical interpolation using linear averaging filter – Will not be able to recover fine details (e.g. alternating horizontal lines) – Can recover moving vertical lines very well • Higher order vertical filter can be used to improve quality • Line/field averaging – D=(C+E+K+R)/4 – The motion and spatial blur are both introduced but not as obvious, on average may be better

Frame 1 Frame 2

100 50 50 100 100 50

100 100 50 100 100 100

100 50 50 100 100 100

100 100 50 100 100 100

100 50 50 100 100 50

100 50 50 100 100 50 100 100 50

100 100 50 100 100 100

100 50 50 100 100 50 100 100 100

100 100 50 100 100 100

100 50 50 100 100 50 100 100 50

Field 1 Field 2 Field 1 Field 2 Missing line 3= (line2+line4)/2 © Yao Wang, 2003 Video digitization and format conversion 20 Deinterlacing by field and line averaging

Frame 1 Frame 2

100 50 50 100 100 50

100 100 50 100 100 100

100 50 50 100 100 100

100 100 50 100 100 100

100 50 50 100 100 50

100 50 50 100 88 50 100 100 50

100 100 50 100 100 100

100 50 50 100 88 62.5 100 100 100

100 100 50 100 100 100

100 50 50 100 88 50 100 100 50

Field 1 Field 2 Field 1 Field 2 Missing line 3= (line2+line4+field1,frame1+field1,frame2)/4 © Yao Wang, 2003 Video digitization and format conversion 21 Field 1 of Frame1

Field 2 of Frame1

• Go through matlab script at class – deinterlacing.m

• Motion adaptive: – Switch between temporal/spatial interpolation depending on motion detection results • Edge adaptive – For spatial interpolation: interpolate along edge direction

13125=625*21=525*25 300=50*6=60*5

From [Wang02]

• What are the factors considered when choosing the sampling rate for digitizing a raster video? • Why should we sample chrominance components at lower sampling rate than the luminance component? • What are the difference between 4:4:4, 4:2:2, 4:1:1, 4:2:0 color formats? • What are some of the common video formats and their intended applications? • What is deinterlacing? What are some simple deinterlacing method (should be able to do calculation on example images and understand matlab scripts) • What are some other format conversion problems? How are they accomplished?

• Y. Wang, J. Ostermann, Y. Q. Zhang, Video Processing and Communications, Prentice Hall, 2002. chapter 1, 3, 4. Mainly Sec. 1.5.