Thesis Prop osal Demosaicing Solutions for Digital Camera Oren Kapah under the sup ervision of Dr. Hagit Hel-Or Bar Ilan University, Octob er 12, 1997 1 1 Intro duction Color images are typically sp eci ed by triplets of red, green and blue values at each pixel on a rectangular sampling grid. However, this representation is not available directly from the output of many color CCD cameras. Rather, these cameras provide only a single photo- sensor resp onse at each pixel. The camera contains several classes of photo-sensor, each class has a distinct sp ectral resp onse. Usually there are three classes, referred to as red, green and blue. There exist several designs of Color Filter Array (CFA) patterns, but the most successful is Bayer[1 ] CFA. Figure 1 provides aschematic illustration of this design. Figure 1: Bayer Color Filter Array In order to obtain a full color image, we must reconstruct the values of the two missing classed at each point. This pro cess of reconstruction is known as demosaicing. The most straightforward approach to demosaicing is to apply one of the standard reconstruction metho ds to each color band separately. The disadvantage of this approach is that the resolution of each band is limited by the Nyquist sampling rate of that band even though the image is sampled at a higher rate, this means that we don't use the information from the other bands and the correlation b etween them to obtain a b etter reconstruction of the image. Another and more critical disadvantage is that the phase di erence between the sampling of di erent color bands may cause the app earance of new colors in the image, particularly around the edges, and thus, distorting the image. This kind of distortion is illustrated in Figure 2 for a 1D example. A black-white edge is captured by the CCD array as shown in Figure 2a. Assuming a simple linear interp olation on each band separately (Figure 2b and Figure 2c) and a p erfect reconstructed blue channel, a distortion is intro duced in the reconstruction (Figure 2d). This typ e of distortion pro duces a much stronger distortion in the reconstructed image when there are many edges in the same area. This e ect is describ ed in the example b elow. Supp ose we have an image of a fence consisting of white and black lines at the Example: sampling rate of the digital camera. This image would be captured by the camera CCD as 2 a. b. c. d. Figure 2: a) The original image as sensed by the CCD. b) The reconstructed red channel. c) The reconsturcted green channel. d) The reconstructed image. (a) (b) Figure 3: a) The original image as sensed by the CCD. b) The reconstructed image. 3 illustrated in Figure 3a. As we can see from Figure 3a, all the blue sensors capture dark values (represented as 0), and thus, in the reconstruction of the full color image, the value of the blue channel in all the pixels would b e 0. For the same reasons the value of the red channel in all the pixels would b e 255. In the green channel half of the sensors capture dark values and half of them capture bright values, thus, even if we reconstruct this channel as it should be in the full color image there would still b e distortion (Figure 3b). We can obtain b etter results if wewould takeinto consideration the cross-channel corre- lation b etween the di erent color bands. Recentwork demonstrated the imp ortance of using the cross-channel correlation information in demosaicing metho ds. However, the algorithms suggested still pro duce artifacts in the reconstructed images. 2 Previous Work Several metho ds have recently b een suggested to solve the demosaicing problem. In [3] Cok estimates the green channel values in the missing green pixels indep endently from the other channels by simply calculating the average of the neighb oring green pixels, thus, in order to obtain b etter results he raised the prop ortion of the green pixels. After all the green values have been reconstructed, the missing red and blue values are reconstructed using the red to green and blue to green prop ortions extracted from the known red pixels and the known blue pixels. Cok suggested for this algorithm a CFA that consists of 3/4 green pixels, 1/8 red and 1/8 blue pixels. Kimmel [5] has improved Cok's metho d by using the computed gradients of the green channel for estimating the green value at the missing green p oints. Then rep eatedly calculates the red and blue values at the missing red and blue lo cations using the red to green and blue to green ratios, and corrects the green values to satisfy the ratio rule. Kimmel improves the reconstruction by following this pro cedure with an enhancement stage which is based on reaction-di usion techniques. Brainard and Sherman [2] used the Baysian metho d for reconstruction of the full color image and develop ed a statistical mo del for evaluating the likeliho o d of an image to pro duce the sampled image. Several rules were formulated for this mo del: The average power sp ectrum of natural images falls o fairly rapidly as a function of spatial frequency. The signals in di erent color bands are p ositively correlated. Based on these rules, the image that is reconstructed is the image most likely to be the original image. Gamer [4] prop osed to rst estimate the red, green and blue in the missing pixels, and then calculate the red - green and blue - green value di erences for each pixel. At the edges where artifacts app ear, there will be a spike in the calculated color di erence. To overcome the artifacts it was suggested to p erform median ltering on the color di erences, to smo oth the spikes, and then re-estimate the RGB values according to the new calculated color di erences. 4 3 Research Plan For our research we have develop ed a formal mo del of the acquisition pro cess of an image by a digital camera. This mo del will formally assist in de ning our problem and the to ols that we can use to solve this problem. 3.1 Image Acquisition Mo del In our mo del there are two stages for the degradation of the image: The degradation that the image undergo es in the camera lens. The degradation that the image undergo es in the camera CFA. The degradation of the image in the lens can be formulated as follows: I = I f (1) f o Where: I is the original image which, for reasons of convenience, is represented as a 2D grid of discrete sample lo cations. Each sample contains a vector which represents the dis- cretized sp ectrum of the image at this lo cation. This gives us a 3D matrix of size M N Z . f is the spatial lter represented by a matrix of co ecients which de nes the spatial o blur of the lens. I is the image obtained b ehind the lens. f is the op eration of convoluting the M N matrix in each entry of Z with f . o The degradation of the image in the CFA can be formulated as follows: At each pixel lo cation (x; y ): t ~ ~ I (x; y )= I (x; y )f (x; y ) (2) s c f Where: I (x; y ) is the sensor value at pixel (x; y ). s ~ I (x; y ) is the vector representing the sp ectrum of the image obtained b ehind the lens f at pixel (x; y ). 5 ~ f (x; y )is the vector of co ecients representing the sp ectral resp onse of the sensor at c pixel (x; y ). Equation 1 and 2can be rewritten as a matrix equation as follows: At each pixel lo cation (x; y ): t 0 0 I (x; y )= f I f (3) s o c t Where: I (x; y ) and f is as de ned in Equation 2 representing the image sp ectrum in s c 0 the pixel of the original image and f is a vector representation of the lens spatial lter f o o of Equation 1. As can b e seen from this equation, we can reverse the order of the two ltering steps and apply the sp ectral lter rst and then apply the spatial lter. The standard digital camera with the Bayer CFA describ ed in Section 1 can be viewed under this acquisition mo del as follows: f is a 1 1 unit matrix and there are three classes o of f representing the sp ectral sensitivities of the R,G and B sensors organized according to c the Bayer CFA. 3.2 Research Goal The research goal is to nd a spatial lter f and develop a grid of sp ectral lters f that , o c together with an appropriate demosaicing algorithm, will pro duce a b etter reconstruction of the full color image. Each p oint in the reconstructed full color image will be presented as a vector of three values. These values should b e as close as p ossible to the values that we would obtain from the original image if we pass it through a set of three sp ectral lters representing the red (R), green (G) and blue (B) sensor sp ectral sensitivities. 4 Preliminary Work As a preliminary study we assumed the spatial lter is the basic 1 1 unit matrix and we tested a new grid of sp ectral lters consisting of the three standard R,G and B sp ectral lters, and an additional sp ectral lter referred to as the X-sensor.