Image Processing System Applications: from Barbie Cams to the Space Telescope Robert Kremens, Ph.D Rochester Institute for Technology Center for Imaging Science Digital Imaging and Remote Sensing Group and Pixelphysics, Inc. May 2001 Outline • Fundamentals of Image Processing for Digital Cameras • Solid State Image Sensors - CCDs, CMOS, etc. • System Requirements for Several Applications – Break • Hardware Analysis: The Jam Cam • Hardware Analysis: The Kodak DC210 • Hardware Analysis: The Chandra Orbital X-ray Telescope Fundamentals of Camera Image Processing Robert Kremens May 2001 Digital Camera Image Processing Pipeline Analog White Scene Processing & A/D Conv. Balance Balance Color Filter Array (CFA) RGB to Gamma CFA Blurring* YCC Conv. Correction Interpolation Unsharp Chroma JPEG Finished Masking Compress File Format Subsample (Edge Enh.) White Balance • Usually performed on raw CFA pixels • White balance attempts to adjust for variations in the illuminant (D65, Tungsten, etc.) by adjusting analog amplifier gain in the R,G,B channels • What is white? – R=G=B= ~255 Implementing White Balance • Method A - Predetermine the illuminant. – Acquire image. • Camcorder method - white lens cap pointed at light source. • Can be a problem if scene has no white. – R, G and B adjustment values calculated to make them scale to ~255. – Subsequent images have incoming raw pixels multiplied by adjustment value. • Can be done on the fly with hardware, or in analog stages (preferable). Count 0 Pixel Level 255 Implementing White Balance (cont’d) • Method B - Adjust each image after acquisition – Find area with R~G~B at highest intensity. • Examine Full Image - time consuming. • Predetermined small image area - what if no white? • Subsampled image - faster. – Determine adjustment values for R, G and B. – If G isn’t high enough, there is no white. • Still possible to scale to gray by adjusting R and B? • Or just leave it alone. • Flash illumination removes much of the need for determining the adjustment parameters, since the color spectrum of the illumination source is known. Digital Camera Image Processing Analog White Scene Processing & A/D Conv. Balance Balance Color Filter Array (CFA) RGB to Gamma CFA Blurring* YCC Conv. Correction Interpolation Unsharp Chroma JPEG Finished Masking Compress File Format Subsample (Edge Enh.) Scene Balance • Adjusts the color balance of an image so that neutral images are seen as neutral. • Adjust color planes throughout their range; can use adjustment that is a function of pixel value. • Unless subject is holding a neutral density chart, this is much more of an art than a science. Implementing Scene Balance • Look for areas of image with approximately equal R, G and B values. – Look at entire image - time consuming. – Look at sub-sampled image - can miss data. – Look at blocks of image. • Create areas of image by averaging over 20x20 pixels. • Histogram and scene classification are basic methods - force color histogram to be ‘correct’ • Adjustment (multiplication) values must not affect overall brightness of image. – If R an B need to be increased, G should be decreased also. • Be careful of interaction w/ White Balance. Digital Camera Image Processing Analog White Scene Processing & A/D Conv. Balance Balance Color Filter Array (CFA) RGB to Gamma CFA Blurring* YCC Conv. Correction Interpolation Unsharp Chroma JPEG Finished Masking Compress File Format Subsample (Edge Enh.) CFA Interpolation • CFA interpolation creates 3 separate bit planes for each pixel location of the sensor. • Weighted averages are typical, but algorithms vary depending on filter array patterns. • Can be clever and use adaptive algorithms to reduce sub- sampling effects and undesirable artifacts (‘zippers’) Implementing CFA Interpolation (Median method) • Mean Interpolation (Green) 00 01 02 03 – G11 = (G01 + G10 + G12 + G21)/4 • Median Interpolation (Green) 10 11 12 13 – G11 = [(G01 + G10 + G12 + G21) - MAX(G01 + G10 + G12 + G21) - MIN(G01 + G10 + G12 + G21)]/2 20 21 22 23 30 31 32 33 • Red and Blue may interpolated differently than green. – R11 = (R00 + R02 + R20 + R22) / 4 – B11 = B11 – R12 = (R02 + R22) / 2 – B12 = (B11 + B13) / 2 – B22 = (B11 + B13 + B31 + B33) / 4 Some Observations on CFA Interpolation • The math is not complicated (adds, compares and shifts) – Data re-organization is key to speed. • Barrel shifters and/or byte extraction instructions. • MMX style pack and unpack. – SIMD instructions (such as in MMX) can greatly accelerate math. – A good candidate for hardware acceleration. • Arithmetic compares, adders and shifters are easy to implement. • Interpolation schemes can cause image artifacts. – Edges, corners and stripes present a problem. Color Spaces and Standards • How is the image represented in R,G,B space? – Attempt to maximize color gamut and psycho-visual quality while minimizing non-linear effects. • CCIR 601 - now ITU-R BT.601 – Digital Video Standard – Y’CrCb (Y’Cr’Cb’) Color Space, 4:2:2 Subsampling • Y excursion 0 - 219, offset = 16 (Y = 16 to 235) • Cx excursion +/- 112, offset = 128 (Cx = 16 to 240) – No assumptions about white point • CCIR 709 - now ITU-R BT.709 – HDTV Studio Standard – Y’CrCb Color Space • Y excursion 0 - 219, offset = 16 (Y = 16 to 235) • Cx excursion +/- 112, offset = 128 (Cx = 16 to 240) – Specifies White Point (x = .3127, y = .3290, z = .3582) (D65) – Specifies dark viewing conditions. Color Spaces and Standards (cont’d) • sRGB (called NIFRGB by Kodak) – Default color space for HP and Microsoft – Same as CCIR 709 except • Specifies DIM viewing environment • Full 0 - 255 encoding of YCrCb values • Photo YCC – Also uses CCIR 709 • White is 189 instead of 219 • Results in RGB values from 0 - 346 when reconverted • Chroma channels are unbalanced (supposedly follows distribution of colors in a real scene) Digital Camera Image Processing Analog White Scene Processing & A/D Conv. Balance Balance Color Filter Array (CFA) RGB to Gamma CFA Blurring* YCC Conv. Correction Interpolation Unsharp Chroma JPEG Finished Masking Compress File Format Subsample (Edge Enh.) Gamma Correction • Gamma describes the nonlinear response of a display device (CRT) to an applied signal. 4.5L, L<=0.018 L’ = 0.45 709 {1.099L -0.99, 0.018<L Light Intensity Video Signal • RGB values must be corrected for Gamma before they are transformed into a video space. Digital Camera Image Processing Analog White Scene Processing & A/D Conv. Balance Balance Color Filter Array (CFA) RGB to Gamma CFA Blurring* YCC Conv. Correction Interpolation Unsharp Chroma JPEG Finished Masking Compress File Format Subsample (Edge Enh.) R’G’B’ to Y’CrCb Conversion • Conversion to YCrCb occurs for 2 reasons: – Chroma can be subsampled. • Eye is more responsive to intensity changes (G) than color changes (R,B) • Can compress color channels (R,B) for smaller stored image – Video output potential. • Well known conversion matrices convert Gamma Corrected RGB to Y’CrCb. Y’ 0.257 0.504 0.098 R’ 0* Cr = 0.439 -0.368 -0.071 G’ + 128 Cb -0.148 -0.291 0.439 B’ 128 * 0 for UPF format, 16 for CCIR 601 Implementing RGB - YCrCb Conversion • Color space conversion can be implemented in several ways. – Straight software implementation • Flexible, but inefficient – Hardware assist • Single cycle MAC - 9 clock cycles • SIMD instructions - 3 clock cycles Red – Straight Hardware • Fast - 3 clock cycles • Costly – 3 Dimensional Lookup Tables Green Blue Implementing Nonlinear Color Space Conversion - 3D Lookup Tables • CMY Space is very device dependent and the conversions from RGB, L*ab or YCrCb are not linear. 0,0,255 255,255,255 0,255,0 255,0,0 Implementing Nonlinear Color Space Conversion - 3D Lookup Tables • A 3-D Lookup Table defines the conversion for specific colors 0,0,255 • A fully populated table would have over 16M 255,255,255 entries (256x256x256) • A subset of entries is chosen to populate the table. 0,255,0 255,0,0 Implementing Nonlinear Color Space Conversion - 3D Lookup Tables • The actual values for the conversion are interpolated using various mechanisms. – Tri-linear interpolation (10 */ 7+-) – Prism Interpolation (8 */ 5+-) – Tetrahedral Interpolation (6 */ 3+-) – Pyramid Interpolation (7 */ 4+-) – Fuzzy Logic methods • Simple Table Example C,M,Y Y,Cr,Cb 0,0,0 219,120.2,128.0 0,0,128 192,120.5,0.8 0,0,255 . 0,128,0 . 0,128,128 . … . 255,255,255 0.2, 120.5,127 Digital Camera Image Processing Analog White Scene Processing & A/D Conv. Balance Balance Color Filter Array (CFA) RGB to Gamma CFA Blurring* YCC Conv. Correction Interpolation Unsharp Chroma JPEG Finished Masking Compress File Format Subsample (Edge Enh.) Chroma Subsampling • The human eye is much more sensitive to intensity variations than color variations. • Some color information can be discarded without loss of image quality. (Y) Luminance (RS-170) + = Chrominance (I & Q) Chroma Subsampling • 4:2:2 and 4:2:0 are typical subsampling ratios – 4:2:2 is typically used in video. – 4:2:0 is prevalent in still photography. 4:4:4 (No subsampling) Cb Cb Cb Cb 4:2:2 4:2:0 Cb Cb Cb Cb Cb CbCb Cb Cr Cr Cr Cr Cb Cb Cb CbCb Cb Cb Cb Cr Cr Cr Cr CrCr CrCr Cr Y Y Y Y Cr Cr CrCr CrCr Cr Cr Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y What about other image processing? • In consumer cameras, a large amount of post-capture processing takes place to enhance the quality of the image. • Blurring and subsequent unsharp-masking are common image improvements in processing chains • Blurring with a 3 X 3 convolution kernal after CFA interpolation reduces artifacts (Moire patterns, ‘zippers’, color banding) •