ECE 634 – Digital Video Systems Spring 2021 Fengqing Maggie Zhu Assistant Professor of ECE MSEE 334 [email protected] Video Basics ECE634 – Spring 2021 Jan 19, 2021 1 Outline • Color Perception • Video Capture and Display • Analog and Digital Video ECE634 – Spring 2021 Jan 19, 2021 2 Color Perception ECE634 – Spring 2021 Jan 19, 2021 3 Light Sources • Illuminating sources: – Emit light (e.g. the sun, light bulb, TV monitors) – Perceived color depends on the emitted freq. – Follows additive rule • R+G+B=White • Reflecting sources: – Reflect an incoming light (e.g. the color dye, matte surface, cloth) – Perceived color depends on reflected freq. (= emitted freq. - absorbed freq.) – Follows subtractive rule • R+G+B=Black ECE634 – Spring 2021 Jan 19, 2021 4 Eyes – Human Visual System • Most intelligent and high-end camera Lens: Cornea, Lens Lens Control: Zonula (muscle group) Aperture Control: Iris, Pupil Photo Sensor: Retina, Fovea ECE634 – Spring 2021 Jan 19, 2021 5 Human Perception of Color • Photo receptors in the retina (the surface of the rear of the eye ball) – Cones: function under bright light, can perceive color tone • Red (~570 nm), green (~535 nm), blue (~445 nm) cones • Passed via optic nerve fibers to the visual cortex for processing – Rods: work under low light, can only perceive luminance information • Color sensation of human: – Luminance: brightness – Chrominance: hue (color tone), saturation (color purity) ECE634 – Spring 2021 Jan 19, 2021 6 Spectral Sensitivity Curves ECE634 – Spring 2021 Jan 19, 2021 7 Trichromatic Color Mixing • Trichromatic color mixing theory – Any color can be obtained by mixing three properly chosen primary colors with a right proportion C = åTkCk , Tk : Tristimulus values k=1,2,3 • Primary colors for illuminating sources – Red, Green, Blue (RGB) – Color monitor works by exciting red, green, blue phosphors using separate electronic guns • Primary colors for reflecting sources – Cyan, Magenta, Yellow (CMY) – Color printer works by using cyan, magenta, yellow and black (CMYK) dyes ECE634 – Spring 2021 Jan 19, 2021 8 Color Representation Models • Specify the tristimulus values associated with the three primary colors – RGB – CMY • Specify the luminance and chrominance – HSI (Hue, saturation, intensity) – YIQ (used in NTSC color TV) – YCbCr (used in digital color TV) • Amplitude specification: – 8 bits for each color component, or 24 bits total for each pixel – Total of 16 million colors – A true RGB color display of size 1Kx1K requires a display bufFer memory size of 3 MB ECE634 – Spring 2021 Jan 19, 2021 9 Color Space Conversion • Conversion between different primary sets is linear (3x3 matrix) • Conversion between primary and XYZ/YIQ/YUV are also linear – XYZ colors are not realizable by actual stimuli – YIQ and YUV are derived from the XYZ coordinate • Conversion to LSI/Lab are nonlinear – Coordinate Euclidean distance proportional to actual color difference ECE634 – Spring 2021 Jan 19, 2021 10 Choosing Color Coordinates • For display or printing: RGB or CMY, to produce more colors • For analyzing color differences: HSI, for linear relationship • For processing perceptually meaningful color: L*a*b* • For transmission or storage: YIQ or YUV, for a less redundant representation ECE634 – Spring 2021 Jan 19, 2021 11 Color in Images and Videos • Images are commonly RGB, and each pixel location has 3 colors – (this is ignoring Bayer color sampling) – BE CAREFUL!!! OpenCV loads images as BGR • Videos are commonly YUV or YCbCr, and there are fewer color pixels than luminance pixels – OpenCV will automatically convert videos in YUV into consecutive images of RGB, upsampling the color information ECE634 – Spring 2021 Jan 19, 2021 12 Video Capture and Display ECE634 – Spring 2021 Jan 19, 2021 13 Plenoptic Function (Light Field) • Measures the intensity of light that passes through a particular point in space • Every possible viewing position, with any viewing angle, at every moment in time – 3 location coordinates – 2 angular directions – Time – Wavelength • Light field (plenoptic) camera Adelson and Bergen ’91 ECE634 – Spring 2021 Jan 19, 2021 14 Image Formation (Pinhole Camera) A video records the emitted and/or reflected light intensity from the objects in the scene that is observed by a viewing system (a human eye or a camera) 3-D point X Y x = F , y = F Z Z Camera center The image of an object is reversed from its 3-D position. The object appears 2-D smaller when it is farther away. Image image plane ECE634 – Spring 2021 Jan 19, 2021 15 Video Signal • Real-world scene is a continuous 3-D signal (temporal, horizontal, vertical) • Film records samples in time but continuous in space (typically 24 frames/sec) • Analog video samples in time and samples vertically; continuous horizontally (about 30 frames/sec or higher) – Number of lines controls the maximum vertical frequency that can be displayed for a given viewing distance – Video-raster = 1-D signal consisting of scan lines from successive frames • Digital video: samples in time, vertically, and horizontally ECE634 – Spring 2021 Jan 19, 2021 16 Progressive Scanning Scan lines Horizontal retrace Vertical retrace • Progressive scan: – Captures consecutive lines – Captures a complete frame every D t sec – Also referred to as sequential or non-interlaced – Used by TVs, monitors, video projectors ECE634 – Spring 2021 Jan 19, 2021 17 Interlaced Scanning E A C B Even field Odd field (Horizontal & vertical retrace not shown) • Interlaced scan: D F – Captures alternate lines (each frame split into two fields) • Odd lines are captured (odd field), then even lines (even field) – Captures a complete frame every D t sec – Used in analog television ECE634 – Spring 2021 Jan 19, 2021 18 Why Interlace? • To provide a trade-off between temporal and vertical resolution, for a given, fixed data rate (number of line/sec) • Interlace Artifact field 0 field 1 field 2 field 3 frame 0 frame 1 ECE634 – Spring 2021 Jan 19, 2021 19 Capture Color • Sensors – CCD: Charge-Coupled Devices – CMOS: complementary Metal-Oxide-Semiconductor • Bayer Grid • Demosaicing • Dynamic Range ECE634 – Spring 2021 Jan 19, 2021 20 Video Display • CRT (cathode ray tube) vs LCD (liquid crystal display) vs LED (light emitting diode) • Gamma correction: non-linear relation between camera output signal and actual color values ECE634 – Spring 2021 Jan 19, 2021 21 Analog Video ECE634 – Spring 2021 Jan 19, 2021 22 History of TV in US • 1941: First NTSC broadcast, monochrome – 4:3 aspect ratio; Interlacing – 60 Hz (60 fields per second) – 525 lines but only 480 active lines • 1953: Color NTSC – Backwards compatible with black and white TVs • 1993: Grand Alliance forms to design HDTV • 1996: First public broadcast of HDTV • 2000: First HDTV Superbowl transmission • 2009: Last analog transmission ECE634 – Spring 2021 Jan 19, 2021 23 TV at Purdue • Roscoe George develops the first electronic television receiver in 1929 https://www.earlytelevision.org/pdf/roscoe_george_and_television.pdf ECE634 – Spring 2021 Jan 19, 2021 24 Video Terminology • Component video – Three color components stored/transmitted separately – Use either RGB or YIQ (YUV) or YCrCb coordinate – Betacam (professional tape recorder) use this format • Composite video – Convert RGB to YIQ (YUV) – Multiplexing YIQ into a single signal – Used in most consumer analog video devices • S-video – Y and C (I and Q) are stored separately – Used in consumer video devices ECE634 – Spring 2021 Jan 19, 2021 25 25 TV Broadcasting and Receiving Lu m i n a n c e , RG B Chrominance, ---> Aud io Modulation YC 1 C 2 Multiplexing YC 1 C 2 De- De- ---> Multiplexing Modulation RG B ECE634 – Spring 2021 Jan 19, 2021 26 Why not using RGB directly? • R,G,B components are correlated – Transmitting R,G,B components separately is redundant – More efficient use of bandwidth is desired • RGB->YC1C2 transformation – Decorrelating: Y,C1,C2 are uncorrelated – C1 and C2 require lower bandwidth – Y (luminance) component can be received by B/W TV sets • YIQ in NTSC – I: orange-to-cyan – Q: green-to-purple (human eye is less sensitive) • Q can be further bandlimited than I – Phase=Arctan(Q/I) = hue, Magnitude=sqrt (I^2+Q^2) = saturation – Hue is better retained than saturation ECE634 – Spring 2021 Jan 19, 2021 27 Different Color TV Systems Parameters NTSC PAL SECAM Field Rate (Hz) 59.95 (60) 50 50 Line Number/Frame 525 625 625 Line Rate (Line/s) 15,750 15,625 15,625 Color Coordinate YIQ YUV YDbDr Luminance Bandwidth (MHz) 4.2 5.0/5.5 6.0 Chrominance Bandwidth (MHz) 1.5(I)/0.5(Q) 1.3(U,V) 1.0 (U,V) Color Subcarrier (MHz) 3.58 4.43 4.25(Db),4.41(Dr) Color Modulation QAM QAM FM Audio Subcarrier 4.5 5.5/6.0 6.5 Total Bandwidth (MHz) 6.0 7.0/8.0 8.0 ECE634 – Spring 2021 Jan 19, 2021 28 Digital Video ECE634 – Spring 2021 Jan 19, 2021 29 ECE634 – Spring 2021 pel 122 525 lines 480 lines video of analog encoding Digital ITU 525/60: 60 field/s NTSC 858 pels 720 pels Ac tive Area - R BT.601 VideoFormatR BT.601 Jan 19, pel 16 2021 pel 132 625 lines 576 lines 625/50: 50 field/s PAL/SECAM 864 pels 720 pels Ac tive Area pel 12 30 Color Coordinate - YCbCr • Scaled and shifted versions of analog YUV so the values are in the range of (0, 255) ECE634 – Spring 2021 Jan 19, 2021 31 Chrominance Subsampling Formats 4:4:4 4:2:2 4:1:1 4:2:0 For every 2x2 Y Pixels For every 2x2 Y Pixels For every 4x1 Y Pixels For every 2x2 Y Pixels 4 Cb & 4 Cr Pixel 2 Cb & 2 Cr Pixel 1 Cb & 1 Cr Pixel 1 Cb & 1 Cr Pixel (No subsampling) (Subsampling by 2:1 (Subsampling by 4:1 (Subsampling by 2:1 both horizontally only) horizontally only) horizontally and vertically) Y Pixel Cb and Cr Pixel ECE634 – Spring 2021 Jan 19, 2021 32 Digital Formats – Pixel === “picture element”, a point sample • Digital video is a sequence of frames (x,y,t) • Often denoted {lines}{i,p} or {lines}{i,p}{fps} – 1080i, 720p, 1080p60 • Temporal resolutions – Video • 25, 30, 60 frames per second (Fps) • 50, 60, 120 fields per second – Film: 24, 48 fps – Animation: often lower • Why use more FPS? ECE634 – Spring 2021 Jan 19, 2021 33 Spatial Resolutions • 2K, 4K, 8K, etc.