[Rajeev Ramanath, Wesley E. Snyder, Youngjun Yoo, and Mark S. Drew ] FLOWER PHOTO © PHOTO FLOWER MEDIA, 1991 21ST CENTURY PHOTO:CAMERA AND BACKGROUND ©VISION LTD. DIGITAL Color Image Processing Pipeline [A general survey of digital still camera processing] igital still color cameras (DSCs) have gained significant popularity in recent years, with projected sales in the order of 44 million units by the year 2005. Such an explosive demand calls for an understanding of the processing involved and the implementation issues, bearing in mind the otherwise difficult problems these cameras solve. This article presents an overview of the image processing pipeline, Dfirst from a signal processing perspective and later from an implementation perspective, along with the tradeoffs involved. IMAGE FORMATION A good starting point to fully comprehend the signal processing performed in a DSC is to con- sider the steps by which images are formed and how each stage affects the final rendered image. There are two distinct aspects of image formation: one that has a colorimetric perspective and another that has a generic imaging perspective, and we treat these separately. In a vector space model for color systems, a reflectance spectrum r(λ) sampled uniformly in a spectral range [λmin,λmax] interacts with the illuminant spectrum L(λ) to form a projection onto the color space of the camera RGBc as follows: c = N STLMr + n ,(1) where S is a matrix formed by stacking the spectral sensitivities of the K color filters used in the imaging system column-wise, L is a diagonal matrix with samples of the illuminant spec- trum along its diagonal, M is another diagonal matrix with samples of the relative spectral sensitivity of the charge-coupled device (CCD) sensor, r is a vector corresponding to the rela- IEEE SIGNAL PROCESSING MAGAZINE [34] JANUARY 2005 1053-5888/05/$20.00©2005IEEE tive surface spectral reflectance of the object, Sensor, Aperture, and n is an additive noise and Lens term (in fact, noise may Focus even be multiplicative Control and signal dependent, making matters mathe- matically much more complex) [1], [2]. N cor- Preprocessing White Balance Demosaic responds to the nonlin- earity introduced in the system. A similar algebra- ic setting exists for color Exposure formation in the human Control eye, producing tristimu- lus values where A is used to denote the equivalent of matrix S, and t denotes Display on Preferred Color Transform Color Transform the tristimulus values in Device Post Processing RGB RGB RGB RGB the CIEXYZ space [3]. Let Compress and Store u s c u us denote by f an image with M rows, N columns, [FIG1] Image processing involved in a DSC. and K spectral bands. DSCs typically use K = 3, although one may conceive of a sen- Spaulding details some of these steps [5]. An excellent resource sor with a more spectral bands. In some recent cameras, four for DSC processing is a book chapter by Holm et al. [6]. The sensors are used: red, green, blue, and emerald or cyan, multitude of problems that still remain unsolved is a fascinating magenta, yellow, and green. (Depending on the choice of the source of research in the community. spectral sensitivities, the colors captured form color gamut of different sizes; for DSCs, it is typically important to capture PIPELINE skin tones with reasonable accuracy.) The image may also be The signal flowchart shown in Figure 1 briefly summarizes DSC considered as a two-dimensional (2-D) array with vector-val- processing. It should be noted that the sequence of operations ued pixels. Each vector-valued pixel is formed according to the differs from manufacturer to manufacturer. model in (1), with values determined by the reflectance and Each of these blocks may be fine-tuned to achieve better sys- illumination at the three-dimensional (3-D) world point temic performance [7], e.g., introducing a small amount of blur indexed by 2-D camera pixel position. The image formed is using the lens system increases the correlation between neigh- then further modeled as follows: boring pixels, which in turn may be used in the demosaicking step. Let us now consider each block in Figure 1. g = B{Hf},(2) SENSOR, APERTURE AND LENS where B is a color filter array (CFA) sampling operator, H is the Although there is a need to measure three (or more) bands at point spread function (a blur) corresponding to the optical sys- each pixel location, this requires the use of more than one sen- tem, and f is a lexical representation of the full-color image in sor and, consequently, drives up the cost of the camera. As a which each pixel is formed according to (1). cheaper and more robust solution, manufacturers place a CFA Although there is an overlap between color processing prob- on top of the sensor element. Of the many CFA patterns avail- lems for other devices, such as scanners and printers, working able, the Bayer array is by far the most popular [8]. Control with DSCs is complicated by problems stemming from the man- mechanisms interact with the sensor (shown in Figure 1 as a ner in which the input image is captured; the spatial variation in red-green-blue checkered pattern, the CFA), to determine the the scene lighting, the nonfixed scene geometry (location and exposure (aperture size, shutter speed, and automatic gain con- orientation of light source, camera, and surfaces in the scene), trol) and focal position of the lens. These parameters need to be varying scene illuminants (including combination of different determined dynamically based on scene content. It is conven- light sources in the same scene), or the use of a CFA to obtain tional to include an infrared blocking filter called a “hot mirror” one color sample at a sensor location, to name a few. (as it reflects infrared energy) along with the lens system, as Further resources on DSC issues include an illustrative most of the filters that are used in CFAs are sensitive in the overview of some of the processing steps involved in a DSC by near-infrared part of the spectrum, as is the silicon substrate Adams et al. [4]. A recent book chapter by Parulski and used in the sensor. IEEE SIGNAL PROCESSING MAGAZINE [35] JANUARY 2005 contrast ratio between the brightest pixel and the darkest pixel B in an image. The human visual system (HVS) can adapt to about four orders of magnitude in contrast ratio, while the sRGB sys- A tem and typical computer monitors and television sets have a dynamic range of about two orders of magnitude. This leads to spatial detail in darker areas becoming indistinguishable from black and spatial detail in bright areas become indistinguishable (a) (b) from white. To address this problem, researchers have used the approach of capturing multiple images of the same scene at [FIG2] Illustration of the division of the image into various blocks varying exposure levels and combining them to obtain a fused over which the luminance signal is measured for exposure control. (a) The image is divided into 24 blocks. (b) Sample image that represents the highlight (bright) and shadow (dark) partitioning of the scene for contrast measurement. regions of an image in reasonable detail [11]. A detailed discus- sion of this topic is beyond the scope of this article, however, we refer the interested reader to the appropriate references Exposure control usually requires characterization of the [12]–[14]. Another interesting approach to this problem is to brightness (or intensity) of the image: an over- or underexposed squeeze in two sensors in the location of what was formerly one image will greatly affect output colors. Depending on the meas- sensor element; each with a different sensitivity to light, captur- ured energy in the sensor, the exposure control system changes ing two images of different dynamic ranges, hence effectively the aperture size and/or the shutter speed along with a carefully increasing the net dynamic range of the camera. Commercial calibrated automatic gain controller to capture well-exposed cameras are beginning to incorporate high dynamic range images. Both the exposure and focus controls may be based on (HDR) imaging solutions into their systems, either in software either the actual luminance component through image processing routines or in derived from the complete RGB image or hardware by modifying the actual sensor, to simply the green channel data, which is a facilitate the capture of excessively front or good estimate of the luminance signal. backlit scenes. The image is divided into blocks [9], Focus control may be performed by [10] as shown in Figure 2(a). The average using one of two types of approaches, luminance signal is measured in each one active approaches that typically use a of these blocks that later is combined to pulsed beam of infrared light from a small form a measure of exposure based on the source placed near the lens system (called type of scene being imaged: backlit or (a) an auto-focus assist lamp) to obtain an frontlit scene or a nature shot. In a typical estimate of the distance to the object of image, the average luminance signal is interest or passive approaches that make measured and is compared to a reference use of the image formed in the camera to level, and the amount of exposure is con- determine the best focus. Passive trolled to maintain a constant scene lumi- approaches may be further divided into nance. Backlit or frontlit scenes may be two types, ones that analyze the spatial distinguished by measuring the difference frequency content of the image and ones between the average luminance signal in that use a phase detection technique to the blocks, as shown in Figure 2(b).