PRIMARY-CONSISTENT SOFT-DECISION COLOR DEMOSAIC for DIGITAL CAMERAS (Patent Pending)
Total Page:16
File Type:pdf, Size:1020Kb
PRIMARY-CONSISTENT SOFT-DECISION COLOR DEMOSAIC FOR DIGITAL CAMERAS (Patent Pending) Xiaolin Wu and Ning Zhang Department of Electrical and Computer Engineering McMaster University Hamilton, Ontario L8S 4M2 [email protected] ABSTRACT Another drawback of the existing color demosaic algorithms is that they interpolate missing color components at a Bayer color mosaic sampling scheme is widely used in digital pixel independently of the color interpolations at neighboring cameras. Given the resolution of CCD sensor arrays, the image pixels. The interpolation decision is made on a hypothesis on quality of digital cameras using Bayer sampling mosaic largely the local gradient direction. But these algorithms do not validate depends on the performance of the color demosaic process. A the underlying hypothesis after the color interpolation is done. common and serious weakness shared by all existing color The verification of the hypothesis is difficult if the pixels are demosaic algorithms is an inconsistency of sample treated individually. To overcome this drawback we introduce a interpolations in different primary color components, which is new notion of soft-decision color demosaic. At each pixel, the culprit for the most objectionable color artifacts. To cure the instead of forcing a decision on the gradient with insufficient problem we propose a primary-consistent color demosaic information to guide the interpolation, we make multiple algorithm. The performance of this algorithm is further hypotheses and interpolate missing color components for each enhanced by a soft-decision sample interpolation scheme. of the hypotheses. Then we examine the interpolation results Experiments demonstrate that the proposed framework of under different hypotheses in a window of the pixel in question, primary-consistent soft-decision color demosaic can and choose the one whose underlying hypothesis agrees with the significantly improve the image quality of digital cameras. reconstructed color image the best. In other words, the decision to choose the optimal interpolation is delayed until the final 1. INTRODUCTION color reconstructions under different hypotheses can be compared. In practice, only two hypotheses, one for horizontal The Bayer color mosaic CCD sensor arrays [2] are widely used structure and the other for vertical structure, suffice to eliminate in digital cameras because of their simplicity and low cost. most of color artifacts of the existing color demosaic algorithms. However, an inherent drawback of the Bayer pattern is the subsampling of primary colors, particularly in red and blue bands. Color demosaic via intelligent and robust sample 2. PRIMARY-CONSISTENT COLOR INTERPOLATION interpolation thus holds the key to the visual quality of digital cameras. Many color demosaic algorithms were proposed (see For each pixel, we interpolate each of the missing primary color [5,6] for a comprehensive survey). A common characteristic of components twice, in horizontal and vertical directions these algorithms is gradient-guided directional interpolation. separately. Unlike existing color demosaic algorithms, we do The main idea is to reconstruct the missing samples via not decide, due to lack of data, on the gradient direction at the interpolation along rather than perpendicular to the edges. time of color interpolation. The decision between the two However, much to our surprise, all existing color demosaic interpolations (or weighting the two interpolations) for the pixel algorithms suffer from an oversight of not imposing the is delayed till the results of both interpolations are available for consistency of sample interpolations in three primary colors red, all the missing primary components in a local window. Since the green, and blue. For instance in [4] which is considered one of decision will be eventually validated as described in Section 3, the best color demosaic algorithms, at some positions, the green at the moment we are free to make even conflicting hypotheses interpolation is an axial direction, whereas the blue or red on sample structures. In the sequel we denote by (hr,hg,hb) interpolation is along a diagonal direction. Fine objects in a v v v scene such as hairs and cloth threads, which the existing color and ( r, g, b) the interpolated colors in horizontal and vertical demosaic algorithms often fail to reconstruct, typically have a directions. constant hue and subtle changes, if any, in intensity and/or saturation due to illumination conditions. Consequently, there is 2.1. Interpolation of missing G values a high degree of agreement in the gradients of the three primary colors in natural images, which we call primary consistency. Consider the case of interpolating the missing G value at the This is an important physical constraint. The violation of this location of an R sampling position in the Bayer pattern as consistency causes objectionable visual artifacts. A main illustrated in Fig. 1. contribution of this paper is the new concept of primary- consistent color interpolation. 0-7803-7750-8/03/$17.00 ©2003 IEEE. ICIP 2003 h v Note that by now all four neighboring green values gw , gw , NN h v ge , ge have been tentatively computed. Using these N reconstructed green values and the original sample values, we compute WW W C E EE h = + 1 −h + −h rc Gc (Rw gw Re ge ) S 2 v = + 1 −v + −v rc Gc (Rw gw Re ge ) . SS 2 Figure 1 Bayer pattern 1 hb = G + (B −hg + B −hg ) c c 2 n n s s The missing G value will be interpolated in both horizontal 1 and vertical directions: vb = G + (B −vg + B −vg ) c c 2 n n s s h = 1 + + 1 − + − gc (Gw Ge) (Rc Rww Rc Ree) (1) Since blue color is not sampled at all in the current row and red 2 4 samples are completely missing in the current column, the v = 1 + + 1 − + − inherently difficult task in the terms of maintaining primary gc (Gn Gs ) (Rc Rnn Rc Rss ) (2) 2 4 v h consistency is to estimate rc and bc . The vertical R As a convention of this paper, capital letters refer to original v primary color values in the Bayer sampling scheme, and small interpolation rc has to use red samples of the horizontal letters to reconstructed primary color values obtained by color neighbors Rw and Re , which is in conflict of the underlying demosaic, with subscripts denoting the relative geographic assumption of vertical structure. The best one can do here is to locations of the pixels as in Fig. 1. An interpretation of (1) is fully utilize available vertical information of the neighboring that g − G ≈ (R − R ) / 2 , g − G ≈ (R − R ) / 2 , l w c ww r e c ee v v columns. The green estimates gw and ge associated with where gl and gr are the left and right estimates of G, and we let v h = + Rw and Re are used to estimate rc . It is important to realize gc (gl gr )/ 2 . Clearly, the assumption used in (1) is that v v the horizontal gradients in R and G are approximately the same. that gw and ge are estimates under the hypothesis of vertical The same applies to (2) with symmetry between horizontal and structure. The influence of the vertical structure to the missing vertical directions. The advantage of using the adjacent red red value in the current column is factored in by assuming that samples in the interpolation direction to interpolate the missing the difference between red and green values remains a constant green value is that they can factor in the second order term in in the small locality. Namely, the green waveform. 1 vr − G = (R −vg + R −vg ) c c 2 w w e e The interpolation of missing green values at the blue sampling v v v positions of the Bayer pattern is symmetric to the case The use of gw and ge in our estimate of rc is to ensure considered above. One only needs to replace R values of (1) primary consistency. This also explains why we need to first and (2) by corresponding B values. estimate all the missing green values under the hypotheses of horizontal and vertical sample structures separately. After all the missing green values are tentatively The estimates for G sampling position with horizontal B and interpolated in both horizontal and vertical directions, we vertical R neighbors are symmetric with the same rationale. proceed to interpolate the missing red and blue values. The reason for interpolating the missing green values first is two 2.3. Interpolation of missing R/B values at B/R sampling folds. First, green interpolation tends to be more robust because positions the sampling frequency of the green band is twice as high as that of the other two primary color bands. Second, once the missing The most difficult task for color demosaic of the Bayer pattern is green values are reconstructed, we have a full-resolution to estimate the missing blue (red) value at a red (blue) sampling complete green image that can greatly aid the interpolation of position. This is because the blue (red) color is not sampled at missing red and blue values. all in both the current row and current column. The missing blue (red) value has to be inferred from the samples at the four 2.2. Interpolation of missing R/B values at G sampling corners or beyond. Specifically, consider the following case of positions missing blue value at the red sampling position: Case 1: G sampling position with horizontal R and vertical B h v h v Bnw , gnw, gnw Bne, gne, gne neighbors, as illustrated below. h v Rc, gc, gc B ,hg ,vg n n n B ,hg ,vg B ,hg ,vg h v h v sw sw sw se se se Rw, gw, gw Gc Re, ge, ge h v Bs , gs , gs In this case, on the same line of reasoning as in Case 1, the primary-consistent estimates are 1 e.g., estimated from a large suitable training set.