IEE 5037 Multimedia Communications Lecture 2: Basics of Video
Total Page:16
File Type:pdf, Size:1020Kb
Dept. Electronics Engineering, Dept. Electronics IEE 5037 Multimedia Communications National Chiao Tung University Lecture 2: Basics of Video Reading: ch.1. video processing and communications Adapted from Prof. Yao Wang’s slides And Prof. Hang’s slides 1 Outline • Color perception and specification • Video capture and display • Analog raster video • Analog TV systems • Digital video Video Basics 1. Color Perception and Specification • Light -> color perception • Human perception of color • Type of light sources • Trichromatic color mixing theory • Specification of color – Tristimulus representation – Luminance/Chrominance representation • Color coordinate conversion Video Basics 3 Light is part of the EM wave from [Gonzalez02] Video Basics 4 Illuminating and Reflecting Light • Illuminating sources: – emit light (e.g. the sun, light bulb, TV monitors) – perceived color depends on the emitted freq. – follows additive rule • R+G+B=White • Reflecting sources: – reflect an incoming light (e.g. the color dye, matte surface, cloth) – perceived color depends on reflected freq (=emitted freq- absorbed freq.) – follows subtractive rule • R+G+B=Black Video Basics 5 Color Representation (N&H, Sec. 1.8) • Colors (human perception) are related to the wavelength of light, but they are subjective, depending upon the physical structure of human eye. • Human retina contains 3 different color receptor (cones): red, green, and blue Æ Basis of trichromatic theory. • Tristimulus: Any color appeared to human eye can be specified by a weighted combination of 3 primary colors. Video Basics 6 Eye Anatomy From http://www.stlukeseye.com/Anatomy.asp Video Basics 7 Eye vs. Camera Camera components Eye components Lens Lens, cornea Shutter Iris, pupil Film Retina Cable to transfer images Optic nerve send the info to the brain Video Basics 8 Human Perception of Color • Retina contains photo receptors – Cones: day vision, can perceive color tone • Red, green, and blue cones • Different cones have different frequency responses • Tri-receptor theory of color vision [Young1802] – Rods: night vision, perceive brightness From only http://www.macula.org/anatomy /retinaframe.html • Color sensation is characterized by – Luminance (brightness) – Chrominance • Hue (color tone) • Saturation (color purity) Video Basics 9 Human Vision • Cross-section of the human eye (N&H, p.263) Cone: fat, clustered around fovea; color, daylight Rod: slender, dense; intensity, low light Video Basics 10 The Retina • Schematic diagram of the retina (N&H, p. 263) Video Basics 11 Retina Circuitry • Photoreceptors first respond to the light. The produced signals are “processed” by the bipolar and ganglion cells. Then, visual information (pulses) are passed through optical nerves into the brain. (Carlson, p.145~) Video Basics 12 Primary Visual Path • The left half of the visual fields of both eyes are sent to the right hemisphere. (Carlson, p.149) Video Basics 13 Color Receptors • Relative absorbance of light of various wavelengths by rods and cones (Carlson, p.155) Video Basics 14 Frequency Responses of Cones from [Gonzalez02] Ci = ∫C(λ)ai (λ)dλ, i = r, g,b, y Video Basics 15 Cones Sensitivity: R, G, B and Luminous 100 Blueϫ20 Luminous Luminosity function 80 Red Green 60 G R 40 Relative sensitivity B 20 0 400 500 600 700 Wavelength Ci = ∫C(λ)ai (λ)dλ, i = r, g,b, y Video Basics 16 Color Hue Specification Video Basics 17 Trichromatic Theory • Select 3 primary stimuli, for example, R (700.0), G (546.1) ,and B (435.8) Any color S can be represented as combination of these 3 primaries R, G, and B. S = Rs ⋅R + Gs ⋅G + Bs ⋅B • Any 3 independent colors can be selected as primaries as long as one is not a mix of the other two. • Different sets of primaries are related by linear transformations. Video Basics 18 Trichromatic Color Mixing • Trichromatic color mixing theory – Any color can be obtained by mixing three primary colors with a right proportion C = ∑TkCk , Tk : Tristimulus values k=1,2,3 • Primary colors for illuminating sources: – Red, Green, Blue (RGB) – Color monitor works by exciting red, green, blue phosphors using separate electronic guns • Primary colors for reflecting sources (also known as secondary colors): – Cyan, Magenta, Yellow (CMY) – Color printer works by using cyan, magenta, yellow and black (CMYK) dyes Video Basics 19 RGB vs CMY Video Basics 20 red Green Blue Video Basics 21 Color Representation Models • Specify the tristimulus values associated with the three primary colors –RGB –CMY • Specify the luminance and chrominance – HSI (Hue, saturation, intensity) – YIQ (used in NTSC color TV) – YCbCr (used in digital color TV) • Amplitude specification: – 8 bits for each color component, or 24 bits total for each pixel – Total of 16 million colors – A true RGB color display of size 1Kx1K requires a display buffer memory size of 3 MB Video Basics 22 Color Coordinate Conversion • Conversion between different primary sets are linear (3x3 matrix) • Conversion between primary and XYZ/YIQ/YUV are also linear • Conversion to LSI/Lab are nonlinear – LSI and Lab coordinates • coordinate Euclidean distance proportional to actual color difference • Conversion formulae between many color coordinates can be found in [Gonzalez92] Video Basics 23 CIE Color Specification ─ CIE (Commission Internationale de L'eclairage─ International Commission on Illumination) • R (700.0), G (546.1), and B (435.8) ─1931 • X, Y, Z ─ 1931; (1) all color matching functions are positive, and (2) Y = Luminance. (N&H, p. 51) ⎡X ⎤ ⎡2.365 −.515 0.005 ⎤⎡R⎤ ⎡R⎤ ⎢Y ⎥ = ⎢−.897 1.426 −.014⎥⎢G⎥ = M −1 ⎢G⎥ ⎢ ⎥ ⎢ ⎥⎢ ⎥ ⎢ ⎥ ⎣⎢Z ⎦⎥ ⎣⎢−.468 0.089 1.009 ⎦⎥⎣⎢B⎦⎥ ⎣⎢B⎦⎥ Video Basics 24 CIE Color Specification (cont.) (Basis vectors transformation) ⎡R⎤ ⎡0.490 0.177 0.000⎤⎡X ⎤ ⎡X ⎤ ⎢G⎥ = ⎢0.310 0.813 0.010⎥⎢Y ⎥ = M ⎢Y ⎥ ⎢ ⎥ ⎢ ⎥⎢ ⎥ ⎢ ⎥ ⎣⎢B⎦⎥ ⎣⎢0.200 0.010 0.990⎦⎥⎣⎢Z ⎦⎥ ⎣⎢Z ⎦⎥ Tristimulus values transformation: (Coordinates transformation) ⎡X ⎤ ⎡R⎤ ⎡R⎤ ⎡X ⎤ ⎢Y ⎥ = M −1 ⎢G⎥; ⎢G⎥ = M ⎢Y ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎣⎢Z ⎦⎥ ⎣⎢B⎦⎥ ⎣⎢B⎦⎥ ⎣⎢Z ⎦⎥ Video Basics 25 Chromaticity Coordinates X x = X +Y + Z Y y = X +Y + Z Z z = X +Y + Z Note: Hence, (x,y,Y) completely specifies a color. 1931 CIE-xy chromaticity diagram (N&H, p. 51) Video Basics 26 CIE Chromaticity Diagram CIE XYZ diagram CIE L*a*b diagram Concept chart of L*a*b* L: luminance a: from green to red b: from blue to yellow Source: http://www.experience.epson.com.au/help/understandingcolour/COL_G/0503_3.htm Video Basics 27 Just Noticeable Color Difference • MacAdam's ellipses (just noticeable regions, ten times enlarged) on CIE-xy chromaticity diagram (N&H, p.293) Video Basics 28 2. Video Capture and Display • Light reflection physics • Imaging operator • Color capture • Color display • Component vs. composite video Video Basics 29 Video Capture ? • For natural images we need a light source (λ: wavelength of the source) . – E(x, y, z, λ): incident light on a point (x, y, z world coordinates of the point) • Each point in the scene has a reflectivity function. – r(x, y, z, λ): reflectivity function • Light reflects from a point and the reflected light is captured by an imaging device. – c(x, y, z, λ)=E(x, y, z, λ) × r(x, y, z, λ): reflected light. Courtesy of Onur Guleryuz Video Basics 30 More on Video Capture • Reflected light to camera – Camera absorption function ψ (X,t) = ∫C(X,t,λ)ac (λ)dλ – Projection from 3-D to 2-D X→x P ψ (P(X),t) =ψ (X,t) or ψ (x,t) =ψ (P−1(x),t) • The projection operator is non-linear – Perspective projection – Othographic projection Video Basics 31 Perspective Projection Model Y X 3-D point Y Z X X Z C y F x Camera center X Y x = F , y = F Z Z x y x The image of an object is reversed from its 2-D 3-D position. The object appears smaller image Image when it is farther away. plane Video Basics 32 How to Capture Color • Need three types of sensors • Complicated digital processing is incorporated in advanced cameras 2fs,1/fs,2 f f Rate ( fs,1) s,1 s,1 Image 2fs,1 Digital CN conv. CCDs enhancer output 2fs,1 13.5 MHz Lens R G D/A Analog CN & A/D CS output Nonlinear processing Pre-process Interpolation Color corrector Analog process B Matrix & encoder D/A Viewfinder output Figure 1.2 Schematic block diagram of a professional color video camera. Reprinted from Y. Hashimoto, M. Yamamoto, and T. Asaida, Cameras and display systems, IEEE (July 1995), 83(7):1032–43. Copyright 1995 IEEE. Video Basics 33 Video Display • CRT vs LCD • Need three light sources projecting red, green, blue components respectively Video Basics 34 3. Analog Video • Video raster • Progressive vs. interlaced raster • Analog TV systems Video Basics 35 Raster Scan • Real-world scene is a continuous 3-D signal (temporal, horizontal, vertical) • Analog video is stored in the raster format – Sampling in time: consecutive sets of frames • To render motion properly, >=30 frame/s is needed – Sampling in vertical direction: a frame is represented by a set of scan lines • Number of lines depends on maximum vertical frequency and viewing distance, 525 lines in the NTSC system – Video-raster = 1-D signal consisting of scan lines from successive frames . Video Basics 36 Two Scan Methods: Progressive and Interlaced Scans Progressive Frame Interlaced Frame Horizontal retrace Field 1 Field 2 Vertical retrace Interlaced scan is developed to provide a trade-off between temporal and vertical resolution, for a given, fixed data rate (number of line/sec). Video Basics 37 Two Scan Methods: Progressive and Interlaced Scans Video Basics 38 Color TV Broadcasting and Receiving Lu m i n a n c e , RG B Chrominance, ---> Aud io Modulation YC