PRIMARY-CONSISTENT SOFT-DECISION COLOR DEMOSAIC FOR DIGITAL CAMERAS (Patent Pending)
Xiaolin Wu and Ning Zhang Department of Electrical and Computer Engineering McMaster University Hamilton, Ontario L8S 4M2 [email protected]
ABSTRACT Another drawback of the existing color demosaic algorithms is that they interpolate missing color components at a Bayer color mosaic sampling scheme is widely used in digital pixel independently of the color interpolations at neighboring cameras. Given the resolution of CCD sensor arrays, the image pixels. The interpolation decision is made on a hypothesis on quality of digital cameras using Bayer sampling mosaic largely the local gradient direction. But these algorithms do not validate depends on the performance of the color demosaic process. A the underlying hypothesis after the color interpolation is done. common and serious weakness shared by all existing color The verification of the hypothesis is difficult if the pixels are demosaic algorithms is an inconsistency of sample treated individually. To overcome this drawback we introduce a interpolations in different primary color components, which is new notion of soft-decision color demosaic. At each pixel, the culprit for the most objectionable color artifacts. To cure the instead of forcing a decision on the gradient with insufficient problem we propose a primary-consistent color demosaic information to guide the interpolation, we make multiple algorithm. The performance of this algorithm is further hypotheses and interpolate missing color components for each enhanced by a soft-decision sample interpolation scheme. of the hypotheses. Then we examine the interpolation results Experiments demonstrate that the proposed framework of under different hypotheses in a window of the pixel in question, primary-consistent soft-decision color demosaic can and choose the one whose underlying hypothesis agrees with the significantly improve the image quality of digital cameras. reconstructed color image the best. In other words, the decision to choose the optimal interpolation is delayed until the final 1. INTRODUCTION color reconstructions under different hypotheses can be compared. In practice, only two hypotheses, one for horizontal The Bayer color mosaic CCD sensor arrays [2] are widely used structure and the other for vertical structure, suffice to eliminate in digital cameras because of their simplicity and low cost. most of color artifacts of the existing color demosaic algorithms. However, an inherent drawback of the Bayer pattern is the subsampling of primary colors, particularly in red and blue bands. Color demosaic via intelligent and robust sample 2. PRIMARY-CONSISTENT COLOR INTERPOLATION interpolation thus holds the key to the visual quality of digital cameras. Many color demosaic algorithms were proposed (see For each pixel, we interpolate each of the missing primary color [5,6] for a comprehensive survey). A common characteristic of components twice, in horizontal and vertical directions these algorithms is gradient-guided directional interpolation. separately. Unlike existing color demosaic algorithms, we do The main idea is to reconstruct the missing samples via not decide, due to lack of data, on the gradient direction at the interpolation along rather than perpendicular to the edges. time of color interpolation. The decision between the two However, much to our surprise, all existing color demosaic interpolations (or weighting the two interpolations) for the pixel algorithms suffer from an oversight of not imposing the is delayed till the results of both interpolations are available for consistency of sample interpolations in three primary colors red, all the missing primary components in a local window. Since the green, and blue. For instance in [4] which is considered one of decision will be eventually validated as described in Section 3, the best color demosaic algorithms, at some positions, the green at the moment we are free to make even conflicting hypotheses interpolation is an axial direction, whereas the blue or red on sample structures. In the sequel we denote by (hr,hg,hb ) interpolation is along a diagonal direction. Fine objects in a v v v scene such as hairs and cloth threads, which the existing color and ( r, g, b) the interpolated colors in horizontal and vertical demosaic algorithms often fail to reconstruct, typically have a directions. constant hue and subtle changes, if any, in intensity and/or saturation due to illumination conditions. Consequently, there is 2.1. Interpolation of missing G values a high degree of agreement in the gradients of the three primary colors in natural images, which we call primary consistency. Consider the case of interpolating the missing G value at the This is an important physical constraint. The violation of this location of an R sampling position in the Bayer pattern as consistency causes objectionable visual artifacts. A main illustrated in Fig. 1. contribution of this paper is the new concept of primary- consistent color interpolation.
0-7803-7750-8/03/$17.00 ©2003 IEEE. ICIP 2003 h v Note that by now all four neighboring green values gw , gw , NN h v ge , ge have been tentatively computed. Using these N reconstructed green values and the original sample values, we compute WW W C E EE h = + 1 −h + −h rc Gc (Rw gw Re ge ) S 2 v = + 1 −v + −v rc Gc (Rw gw Re ge ) . SS 2 Figure 1 Bayer pattern 1 hb = G + (B −hg + B −hg ) c c 2 n n s s The missing G value will be interpolated in both horizontal 1 and vertical directions: vb = G + (B −vg + B −vg ) c c 2 n n s s h = 1 + + 1 − + − gc (Gw Ge) (Rc Rww Rc Ree) (1) Since blue color is not sampled at all in the current row and red 2 4 samples are completely missing in the current column, the v = 1 + + 1 − + − inherently difficult task in the terms of maintaining primary gc (Gn Gs ) (Rc Rnn Rc Rss ) (2) 2 4 v h consistency is to estimate rc and bc . The vertical R As a convention of this paper, capital letters refer to original v primary color values in the Bayer sampling scheme, and small interpolation rc has to use red samples of the horizontal letters to reconstructed primary color values obtained by color neighbors Rw and Re , which is in conflict of the underlying demosaic, with subscripts denoting the relative geographic assumption of vertical structure. The best one can do here is to locations of the pixels as in Fig. 1. An interpretation of (1) is fully utilize available vertical information of the neighboring that g − G ≈ (R − R ) / 2 , g − G ≈ (R − R ) / 2 , l w c ww r e c ee v v columns. The green estimates gw and ge associated with where gl and gr are the left and right estimates of G, and we let v h = + Rw and Re are used to estimate rc . It is important to realize gc (gl gr )/ 2 . Clearly, the assumption used in (1) is that v v the horizontal gradients in R and G are approximately the same. that gw and ge are estimates under the hypothesis of vertical The same applies to (2) with symmetry between horizontal and structure. The influence of the vertical structure to the missing vertical directions. The advantage of using the adjacent red red value in the current column is factored in by assuming that samples in the interpolation direction to interpolate the missing the difference between red and green values remains a constant green value is that they can factor in the second order term in in the small locality. Namely, the green waveform. 1 vr − G = (R −vg + R −vg ) c c 2 w w e e The interpolation of missing green values at the blue sampling v v v positions of the Bayer pattern is symmetric to the case The use of gw and ge in our estimate of rc is to ensure considered above. One only needs to replace R values of (1) primary consistency. This also explains why we need to first and (2) by corresponding B values. estimate all the missing green values under the hypotheses of horizontal and vertical sample structures separately. After all the missing green values are tentatively The estimates for G sampling position with horizontal B and interpolated in both horizontal and vertical directions, we vertical R neighbors are symmetric with the same rationale. proceed to interpolate the missing red and blue values. The reason for interpolating the missing green values first is two 2.3. Interpolation of missing R/B values at B/R sampling folds. First, green interpolation tends to be more robust because positions the sampling frequency of the green band is twice as high as that of the other two primary color bands. Second, once the missing The most difficult task for color demosaic of the Bayer pattern is green values are reconstructed, we have a full-resolution to estimate the missing blue (red) value at a red (blue) sampling complete green image that can greatly aid the interpolation of position. This is because the blue (red) color is not sampled at missing red and blue values. all in both the current row and current column. The missing blue (red) value has to be inferred from the samples at the four 2.2. Interpolation of missing R/B values at G sampling corners or beyond. Specifically, consider the following case of positions missing blue value at the red sampling position:
Case 1: G sampling position with horizontal R and vertical B h v h v Bnw , gnw, gnw Bne, gne, gne neighbors, as illustrated below. h v Rc, gc, gc B ,hg ,vg n n n B ,hg ,vg B ,hg ,vg h v h v sw sw sw se se se Rw, gw, gw Gc Re, ge, ge h v Bs , gs , gs In this case, on the same line of reasoning as in Case 1, the primary-consistent estimates are 1 e.g., estimated from a large suitable training set. The hypothesis hb =hg + ∑(B −hg ) c c 4 p p testing becomes a classification problem. Conventional p∈(nw,ne,sw,se) minimum-risk Bayesian classifier can be used. Fisher’s 1 discriminant also proved to be effective in our experiments. Due vb =vg + (B −vg ) c c ∑ p p to space limit of this extended abstract, we omit the details of 4 ∈ p (nw,ne,sw,se) these techniques, which will be offered in the full version of the Again note the association of v g with vb and the association of paper. The key contribution of this paper is the general h h framework of primary-consistent soft-decision color demosaic, g with b for primary consistency. which is independent of whatever chosen statistical inference Estimation of missing red value at the blue sampling techniques. position is symmetric. 4. EXPERIMENTAL RESULTS 3. DECISION ON ESTIMATES We have tested the proposed primary-consistent soft-decision After two sets of primary-consistent color estimates color demosaic algorithm on a wide range of natural color (hr,hg,hb) and (vr,vg,vb) are made for each pixel based on two images and compared it with some of the best existing different hypotheses of horizontal and vertical image structures, algorithms in the literature [1],[3],[4]. The following three we will select one of them as the final estimate based on which tables tabulate the PSNR results (dB) of different color hypothesis is better satisfied in a local window centered at the demosaic algorithms, one for each of the primary colors green, pixel. Define the primary differences at position i red, and blue. As one can see from the tables our algorithm can outperform existing ones by significant margins. The difference hγ =h −h hβ =h −h i ri gi , i bi gi ranges from 1.6 dB to 4.6 dB on average of six test images. In vγ =v −v vβ =v −v some instances the gap can be as large as 7.9 dB. i ri gi , i bi gi In order to select between the horizontal and vertical estimates, Green the primary differences of horizontally estimated colors in a 3x3 window: Image [4] [1] [3] OURS Bikes 35.92 34.17 35.62 37.50 (hγ ,hβ ) (hγ ,hβ ) (hγ ,hβ ) nw nw n n ne ne Ribbon 39.36 38.87 40.64 41.07 hγ hβ hγ hβ hγ hβ ( w, w ) ( c , c ) ( e , e ) Water 39.67 38.8 40.68 42.71 hγ hβ hγ hβ hγ hβ ( sw , sw ) ( s , s ) ( se , se ) Fence 38.24 37.26 38.54 39.63 are used to verify the horizontal hypothesis in the window. We Sail 37.05 36.27 37.63 38.35 define the horizontal and vertical gradients of the horizontally Barb 33.24 33.22 34.56 38.07 estimated colors by Ave. 37.25 36.43 37.95 39.56 ∆ = hγ hβ hγ hβ h h ∑ ( p , p ),( q , q ) 1 Red (nw,n),(ne,n),(nw,ne), ( p,q)∈(w,c),(e,c),(w,e), Image [4] [1] [3] OURS (sw,s),(se,s),(sw,se) Bikes 31.80 33.01 32.40 35.63 ∆ = hγ hβ hγ hβ Ribbon 34.83 35.33 34.77 36.27 v h ∑ ( p , p ),( q , q ) 1 Water 35.87 37.38 34.63 41.17 ( p,q)∈(nw,w),(n,c),(ne,e), (sw,w),(s,c),(se,e) Fence 33.31 35.79 29.63 37.24 Sail 33.20 34.63 33.91 36.01 Similarly, based on the primary differences of vertically Barb 29.68 30.63 29.25 33.89 estimated colors in the 3x3 window, we define Ave. 33.12 34.46 32.43 36.70 ∆ = vγ vβ vγ vβ v v ∑ ( p , p ),( q , q ) 1 (nw,w),(n,c),(ne,e), Blue ( p,q)∈(sw,w),(s,c),(se,e), (nw,sw),(n,s),(ne,se) Image [4] [1] [3] OURS Bikes 31.78 32.75 32.01 35.04 Ribbon 35.65 37.15 34.86 38.88 ∆ = vγ vβ vγ vβ h v ∑ ( p , p ),( q , q ) 1 Water 35.91 37.14 34.57 40.61 (nw,n),(ne,n), Fence 33.55 35.92 29.90 37.81 ( p,q)∈(w,c),(e,c), (sw,s),(se,s) Sail 33.41 34.65 33.97 35.97 Barb 29.63 30.51 29.11 33.70 Finally, we reach the point of choosing one of the two estimates Ave. 33.32 34.69 32.40 37.00 (hr,hg,hb) and(vr,vg,vb ) as the demosaic output for each pixel, Due to space limit we cannot display all the six test images depending on which of the two hypotheses on horizontal and listed above. In Fig. 2 we present the original and reconstructed vertical pixel structures fits the image reality better. Consider color images by different algorithms for three test images Fence, the optimal decision that minimizes the estimate error for the Barb, and Water. The superior image quality of the new concerned pixel as a binary random variable X , and suppose algorithm over others is clearly demonstrated by Fig. 2. ∆ ∆ ∆ ∆ the posteriori probability P(X|h h,v h,v v,h v ) can be learnt, 5. REFERENCES
[1] Adams, James E. et al., "Adaptive color plane interpolation in single color electronic camera", U.S. Patent 5,506,619. (a) [2] Bayer, Bryce E. "Color imaging array", U.S. Patent 3,971,065. [3] Chang, E. et al., "Color filter array recovery using a threshold-based variable number of gradients", in Proceedings of SPIE, January, 1999. [4] Hamilton, Jr. et al., "Adaptive color plane interpolation in single sensor color electronic camera", U.S. Patent 5,629,734. [5] Wu, X. et al., "Color restoration from digital Camera data by (b) pattern matching", Proceedings of SPIE, Vol.3018, p.12-17. [6] Website http://www-se.stanford.edu/~tingchen/
(c)
(a)
(d)
(b)
(a)
(c)
(b)
(d)
Figure 2: Comparison of different color demosaic algorithms in image quality. (a) the original; (b) output of algorithm [3]; (c) (c) output of algorithm [4]; (d) output of our algorithm.
(d)