XCELLENCE IN VIDEO Image Sensor Color Calibration Using the Zynq-7000 All Programmable SoC by Gabor Szedo Staff Video Design Engineer Xilinx Inc. [email protected] Steve Elzinga Video IP Design Engineer Xilinx Inc. [email protected] Greg Jewett Video Marketing Manager Xilinx Inc. [email protected] 14 Xcell Journal Fourth Quarter 2012 XCELLENCE IN VIDEO Xilinx image- and video-processing cores and kits provide the perfect prototyping platform for camera developers. mage sensors are used in a wide The signal power measured by a tristimulus values that a camera or range of applications, from cell detector can be expressed as: image sensor measures, such that the I phones and video surveillance spectral responses match those of the ∞ λ λ λ λ products to automobiles and missile P=∫ 0 I( )R( )S( )d CIE standard observer. systems. Almost all of these applica- tions require white-balance correction In order to get a color image, the WHITE BALANCE (also referred to as color correction) human eye, as well as photographic You may view any object under various in order to produce images with colors and video equipment, uses multiple lighting conditions—for example, illu- that appear correct to the human eye adjacent sensors with different spec- minated by natural sunlight, the light of regardless of the type of illumina- tral responses. Human vision relies on a fire, fluorescent or incandescent tion—daylight, incandescent, fluores- three types of light-sensitive cone cells bulbs. In all of these situations, human cent and so on. to formulate color perception. In vision perceives the object as having Implementing automatic white-bal- developing a color model based on the same color, a phenomenon called ance correction in a programmable human perception, the International “chromatic adaptation” or “color con- logic device such as a Xilinx® FPGA or Commission on Illumination (CIE) has stancy.” However, a camera with no Zynq™-7000 All Programmable SoC is defined a set of three color-matching adjustment or automatic compensa- likely to be a new challenge for many functions, x¯ (λ), y¯ (λ) and z¯ (λ). These tion for illuminants may register the developers who have used ASIC or can be thought of as the spectral sensi- color as varying. When a camera cor- ASSP devices previously. Let’s look at tivity curves of three linear light detec- rects for this situation, it is referred to how software running on an embed- tors that yield the CIE XYZ tristimulus as white-balance correction. ded processor, such as an ARM9 pro- values Px, Py, and Pz, known collec- According to the top equation at the cessing system on the Zynq-7000 All tively as the “CIE standard observer.” right of Figure 1, describing spectra of Programmable SoC, can control cus- Digital image sensors predominant- the illuminants, the reflective proper- tom image- and video-processing logic ly use two methods to measure tristim- ties of objects in a scene and the spec- to perform real-time pixel-level ulus values: a color filter array overlay tral sensitivity of the detector all con- color/white-balance correction. above inherently monochromatic pho- tribute to the resulting color measure- To set the stage for how this is todiodes; and stacked photodiodes ment. Therefore, even with the same done, it’s helpful to first examine some that measure the absorption depth of detectors, measurement results will basic concepts of color perception and photons, which is proportional to mix information from innate object camera calibration. wavelength λ. colors and the spectrum of the illumi- However, neither of these methods nant. White balancing, or the separa- CAMERA CALIBRATION creates spectral responses similar to tion of innate reflective properties R(λ) The measured color and intensity of those of the human eye. As a result, from the spectrum of the illuminant reflections from a small, uniform sur- color measurements between different I(λ), is possible only if: face element with no inherent light photo detection and reproduction equip- emission or opacity depend on three ment will differ, as will measurements • Some heuristics, e.g. the spatial functions: the spectral power distribu- between image sensors and human frequency limits on the illuminant, tion of the illuminant, I(λ); the spec- observers when photographing the or object colors are known a pri- tral reflective properties of the surface same scene—the same (Iλ) and (Rλ). ori. For example, when photo- material, R(λ); and the spectral sensi- Thus, the purpose of camera cali- graphing a scene with natural sun- tivities of the imager, S(λ). bration is to transform and correct the light, it is expected that the spec- Fourth Quarter 2012 Xcell Journal 15 XCELLENCE IN VIDEO Figure 1 – Spectral responses of the “standard observer” tral properties of the illuminant ly 8-, 10- or 12-bit) vector of R,G,B prin- that D1 S f1 =D2 S f2, where S is the will remain constant over the cipal color components. Based on cone sensitivity matrix. In the LMS entire image. Conversely, when an whether you are going to perform map- (long-, medium-, short-wave sensitive image is projected onto a white ping linearly and whether color com- cone-response space), screen, spectral properties of the ponents are corrected independently, L /L 00 illuminant change dramatically the mapping function can be catego- 2 1 D = D –1D = 0 M /M 0 from pixel to pixel, while the rized as shown in Table 1. 1 2 2 1 00S /S reflective properties of the scene 2 1 (the canvas) remain constant. THE VON KRIES HYPOTHESIS The advantage of this method is its rela- When both illuminant and reflec- The simplest, and most widely used tive simplicity and easy implementation tive properties change abruptly, it method for camera calibration is with three parallel multipliers as part of is very difficult to isolate the based on the von Kries Hypothesis either a digital image sensor or the scene’s objects and illuminants. [1], which aims to transform colors to image sensor pipeline (ISP): the LMS color space, then performs • Detector sensitivity S(λ) and the correction using only three multipli- k L' L 00L illuminant spectrum I(λ) do not ers on a per-channel basis. The k M' 0 M 0 M have zeros in the range of spec- hypothesis rests on the assumption 00k S' S S trum observed. You cannot gain that color constancy in the human any information about the reflec- visual system can be achieved by indi- In a practical implementation, instead tive properties of objects outside vidually adapting the gains of the of using the LMS space, the RGB color the illuminant spectrum. For three cone responses; the gains will space is used to adjust channel gains example, when a scene is illumi- depend on the sensory context, that such that one color, typically white, is nated by a monochromatic red is, the color history and surround. represented by equal R,G,B values. source, a blue object will look just Cone responses from two radiant However, adjusting the perceived cone as black as a green one. spectra, f1 and f2, can be matched by responses or R,G,B values for one an appropriate choice of diagonal color does not guarantee that other PRIOR METHODS adaptation matrices D1 and D2 such colors are represented faithfully. In digital imaging systems, the prob- lem of camera calibration for a known illuminant can be represented Linear Nonlinear as a discrete, three-dimensional vec- tor function: Independent von Kries Component correction _x'=F(x) Dependent Color-correction matrix Full lookup table where F(x) is the mapping vector Table 1 – Camera calibration methods function and x_ is the discrete (typical- 16 Xcell Journal Fourth Quarter 2012 XCELLENCE IN VIDEO COMPONENT CORRECTION table is 230 word (4 Gbytes) deep and COLOR-CORRECTION MATRIX For any particular color component, 30 bits wide. The second problem is The calibration method we describe in the von Kries Hypothesis can only rep- initialization values. Typically only a this article demonstrates how you can resent linear relationships between few dozen to a few hundred camera use a 3x3-matrix multiplier to perform input and output. Assuming similar input/expected-value pairs are estab- a coordinate transformation aiming to data representation (e.g. 8, 10 or 12 lished via calibration measurements. orthogonalize measured red, green bits per component), unless k is 1.0, The rest of the sparse lookup-table and blue components. The advantage some of the output dynamic range is values have to be interpolated. This of this method over the von Kries unused or some of the input values interpolation task is not trivial, as the approach is that all three color chan- correspond to values that need to be heterogeneous component input-to- nels are involved in the calibration clipped/clamped. Instead of multipli- output functions are neither mono- process. For example, you can incor- ers, you can represent any function tone nor smooth. Figure 2a presents porate information from the red and defining input/output mapping using the measured vs. expected-value blue channels when adjusting green- small, component-based lookup pairs for R,G,B input (rows) and out- channel gains. Also, this solution lends tables. This way you can address put (columns) values. itself well for camera calibration and sensor/display nonlinearity and A visual evaluation of empirical white-balance correction to be per- gamma correction in one block. In an results interpolated (Figure 2b) did formed simultaneously using the same FPGA image-processing pipeline not show significant quality improve- module, updating matrix coefficients implementation, you can use the ment over a gamma-corrected, color- to match changing illuminants smooth- Xilinx Gamma Correction IP block to correction matrix-based solution.