Bayer Demosaicking Using Optimized intensity distributions along the contours while minimizing absolute value of Mean Curvature over RGB channels in piecewise smooth regions. These properties suggest that the artifact parts of images can be removed while edges and textures can be preserved. Accordingly, the proposed variational MC model is written as follows: Rui Chen, Huizhu Jia, Xiange Wen and Xiaodong Xie 1 GGGG arg min ()() 2 (3) Color artifacts of demosaicked images are often found at contours due 2 to interpolation across edges and cross-channel aliasing. To tackle this To reconstruct final G-channel G BV(), we use augmented Lagrangian problem, we propose a novel demosaicking method to reliably reconstruct color channels of a Bayer image based on two different method to solve optimization problem of the Eq. (3). Let's first introduce the optimized mean-curvature (MC) models. The missing pixel values in auxiliary variables: q divn , n p p R 3 , p[ G , 1] R 3 , mR 3 . The green (G) channel are first estimated by minimizing a variational MC associated augmented Lagrangian functional reads as follows: model. The curvatures of restored G-image surface are approximated as a linear MC model which guides the initial reconstruction of red (R) and L(,,,,;,,,)G q p n m 1 2 3 4 1 blue (B) channels. Then a refinement process is performed to interpolate 2 ()()()GG q r11 p p m p p m accurate full-resolution R and B images. Experiments on benchmark 2 22 images have testified to the superiority of the proposed method in terms rr23 (4) of both the objective and subjective quality. p [GG ,1] 2 ( p [ ,1]) ( q div n ) 22 2 r4 Introduction: Demosaicking is a technique for reconstructing a full-color image 34()()()q div n n m n m m 2 from the raw image captured by a digital color camera that utilizes a single image where is the regularization parameter. , R and , R3 are Lagrange sensor with a color filter array (CFA). For the popular Bayer CFA pattern, only 13 24 multipliers. r,,, r r r R are step parameters. The variable m is required to lie one color component is captured at each pixel and the other missing components 1 2 3 4 in the set . () is the pulse function on The subproblems associated with must be estimated to produce high-quality RGB images. A number of Bayer each variable are derived to find the optimal solution of Eq. (4). For G demosaicking approaches have been developed by mainly exploiting the spatial and spectral characteristics of raw images [1-6]. The high interchannel and subproblem, the minimizer is determined by associate Euler-Lagrange equation. intrachannel correlations are often assumed to restore the unknown color We here employ Gauss-Seidel method to solve this equation and get 4r r components. However, for images with sharp color transition and high color 22k1 k 1 k k 1 k (122 )G(,)ij g (,) ij [ G (1,) ij G (1,) ij G (,1) ij G (,1) ij ] saturation, the demosaicking performance is subject to the unreliable estimate of hh (5) local image structures and the extent of spectral correlations. This Letter proposes gG(,)i j x()()r 2121 p y r 2222 p a novel demosaicking algorithm by transforming into a curvature domain to k1 where p [,,] p p p and [,,] . G(,)ij denotes a updated reconstruct full-resolution RGB channels, which can improve the structure- 1 2 3 2 21 22 23 value of k+1-th iteration at the pixel point (,)ij. and y denote differential preserving capability, enhance the weak correlations and reduce the color artifacts. x operators. h is the discretized spatial size. In a similar way, the variable n is The whole demosaicking process is shown in Fig. 1. updated. Other subproblems can be solved using Newton method [8].
Red and Blue Channel Reconstruction: The obtained G is used to guide the reconstruction of the missing R and B channels. The high accuracy of G values can facilitate the color interpolation in R and B spaces. Based on the observation that the inter-channel curvature profiles have higher correlation than the usual differences or ratios of color components, we use the mean curvature measured from full-resolution image as a guided mask, which can be further approximated into a linear MC model. For the (,)ijth pixel of the image G, we first compute second-order finite difference approximations of G : Fig. 1 Flowchart of the proposed demosaicking method GGGGGG 2 216 1/2 dijW,[( (,) ij (1,) ij ) ( (-1,1) ij (,1) ij (1,1) ij (,1) ij ) ] 2 2 1/2 Green Channel Reconstruction: Because original G channel is sampled twice dijS,[(GGGGGG (,) ij (,1) ij ) ( (1,) ij (1,1) ij (1,) ij (1,1) ij )16 ] (6) more than the other two channels and preserves much more image structural d d; d d ij,,,, E ij W ij N ij S information, the subsampled image G is first interpolated to estimate an initial So far now we approximate the curvature in Equation (2) by a combination full-resolution image G by utilizing an arbitrary color difference interpolation algorithm. We here adopt a current state-of-the-art algorithm, the gradient based of orthogonal difference projections in x- and y-coordinate directions. To smooth threshold free (GBTF) [3], to obtain the missing G pixel values as local extrema, the gradient magnitude G is weighted the directional curvature terms. Then it follows from Eq. (6) and a linear MC model can get Gr R(,)(,) i jgr, i j (1) xGG(,)(,) i j y i j ()GGG(,)(,)(,)i j i j i j Gb B(,)(,) i jgb, i j GG(,)(,)i j i j (7) xy where the pixel values, G r and G b , at the R and B pixel locations are obtained uGGGG u u u 4 by adding estimated color differences gr, and gb, to R and B pixel values, ijW, (1,) i j ijE , (1,) i j ijS , (,1) ij ijN , (,) ij respectively. The inaccurate differences often cause the artifacts. where x and y denote the forward differences. uij,,, W,, u ij E u ij S and uij, N have To prevent possible artifacts and preserve detailed image structures, we use 2dij,,,, E 2 d ij W 2 d ij N 2 d ij S mean curvature of the associated image surface as a good geometric metrics to uij,,,, W;;; u ij E u ij S u ij N (8) characterize edges and textures. The mean curvature can be simplified along dij,,,,,,,, W d ij E d ij W d ij E d ij S d ij N d ij S d ij N level lines passing through smooth surface [7]. Then we define it as: The weighted coefficients in model (7) can be computed from image G based I on Eq. (8). Since these curvature-related coefficients characterize local image ()I div (2) 2 structures in RGB channels, the missing R components are reliably estimated I 1 using a set of same coefficients, which measure the importance of neighboring Here denotes the gradient operator of the image I and div is the divergence pixels of whose larger values are assigned to pixels at edges than those in flat operator. This realization form of mean curvature can effectively hold geometric areas. We consider that R channel is interpolated in color difference domain features of image surface and differentiate with non-regularity structures. By because the interpolation performance is generally affected by the smoothness treating the interpolation error or artifacts of Eq. (1) as additive noise, we degree of local structures. The missing component R(,)ij at B sampling position model the expected G-channel image as GG . The norm of curvature is initially interpolated using four color differences {ij,,,, W , ij E , ij S , ij N }. Then 1 as the regularizer allows jumps of image structures as well as continuous the estimated R component R(,)ij is interpolated as 1
From demosaicked images shown in Fig. 2, we can observe that compared uij,,,,,,,, W ij W u ij E ij E u ij S ij S u ij N ij N RG(,)(,)i j i j with other demosaicking algorithms, the Bayer images reconstructed by the uij,,,, W u ij E u ij S u ij N proposed method have much higher subjective quality and less color artifacts, (9) ijW,RGRG ( ij -1, -1) ( ij 1, -1); ijE , ( ij +1, +1) ( ij +1, +1) which involve blocking, blurring and zipper effect at edges. To measure the RGRG ; demosaicking quality, the average S-CIELAB and CPSNR values [6] are ijS, ( ij +1, -1) ( ij +1, -1) ijN , ( ij -1, +1) ( ij 1, +1) computed on two image datasets. It can be seen from Table 1 that our method has Since the four nearest neighboring pixels along the diagonal directions within totally achieved higher reconstruction performance. This is mainly because our the R color plane have stable correlation with the expected value R , the (,)ij MC models are very effective to reduce color artifacts and much less sensitive to obtained R(,)ij is further refined by the following equation: the change of spectral correlation. RR(,)(,)i j i j (1 ) k k k (10) Table 1: Average S-CIELAB and CPSNR (in dB) results kRR k (,) i j, k 1 1 k IMAX Kodak Datasets where k represents four neighboring pixels. The factor is used to adjust the S-CIELAB CPSNR S-CIELAB CPSNR refinement performance. We set it experimentally as a constant value between DLMMSE 1.436 34.46 1.309 39.58 0.5 and 0.7 to obtain the optimal reconstruction. Using the nearest neighboring MSG 1.368 34.59 1.162 41.00 pixels R and known R components along the four directions, the interpolation process for the missing R components at G sampling positions is identical to MDWI 1.081 36.42 0.717 41.96 restoration of the missing R components at B sampling positions. The B channel can be demosaicked by means of the same procedure as R channel. MLRI+wei 1.035 36.91 0.624 42.74 Proposed 1.020 37.23 0.586 43.15 Experimental results: To evaluate the proposed method, we have conducted testing experiments and compared its performance against four state-of-the-art Conclusion: In this Letter, we propose an efficient demosaicking algorithms: DLMMSE [2], MSG [4], MDWI [5] and MLRI+wei [6]. The algorithm by developing two optimized MC models to enhance the representative images are selected from the Kodak and IMAX datasets. These interpolation accuracy and suppress color artifacts. By exploiting the full color images are first decomposed into monochrome images with the Bayer mean curvature of G image surface as a regularizer, the variational MC pattern. The regularization parameter is set as 2 103 . The spatial step h is model is iteratively solved to reconstruct full-resolution G image with set to 5. r1,, r 2 r 3 and r4 are set as 40, 40, 100 and 100, respectively. The high accuracy and powerful structure-preserving capability. Moreover, k k1 k 4 stopping condition for iteratively solving Eq. (5) is GGG 2 2 10 . this model can be incorporated into other demosaicking methods. Based The factor is selected as 0.6. on high correlation in curvature domain, the R and B full-resolution images can be well restored using the linear MC model and a postprocessing refinement. Extensive experiments have fully showed the superior performance of our method.
Acknowledgments: This work is supported by China Postdoctoral Science Foundation (2014M560852) and Major National Scientific Instrument and Equipment Development Project of China (2013YQ030967).
Rui Chen, Huizhu Jia, Xiange Wen and Xiaodong Xie (National Engineering Laboratory for Video Technology, Peking University, Beijing 100871, People's Republic of China) E-mail: [email protected]
References 1 Yu T.Z., Hu W.R., Xue W., and Zhang W.S.: ‘Colour image a b c demosaicing via joint intra and inter channel information’, Electron. Lett., 2016, 52, (8), pp. 605–607 2 Zhang L., and Wu X.: ‘Color demosaicking via directional linear minimum square-error estimation’, IEEE Trans. Image Process., 2005, 14, (12), pp. 2167–2178 3 Pekkucuksen I., and Altunbasak Y.: ‘Gradient based threshold free color filter array interpolation’, in Proc. IEEE Int. Conf. Image Process. (ICIP), 2010, pp. 137–140 4 Pekkucuksen I., and Altunbasak Y.: ‘Multiscale gradients-based color filter array interpolation’, IEEE Trans. Image Process., 2013, 22, (1), pp. 157–165 5 Chen X.D., He L.W., Jeon G., and Jeong J.: ‘Multidirectional weighted interpolation and refinement method for bayer pattern CFA demosaicking’, IEEE Trans. Circuits Syst. Video Technol., 2015, 25, (8), pp. 1271–1282 6 Kiku D., Monno Y., Tanaka M., and Okutomi M.: ‘Beyond color d e f difference: residual interpolation for color image demosaicking’, Fig. 2 Visual comparison of different demoaicking methods IEEE Trans. Image Process., 2016, 25, (3), pp. 1288–1300 a Cropped regions of the original images 7 Kim H., Cha Y., and Kim S.: ‘Curvature interpolation method for b DLMMSE image zooming’, IEEE Trans. Image Process., 2011, 20, (7), pp. c MSG 1895–1903 d MDWI 8 Myllykoski M., Glowinski R. Karkkainen T., and Ross T.: ‘A new e MLRI+wei augmented Lagrangian approach L1-mean curvature image f Proposed denoising’, SIAM J. Imaging Sciences, 2012, 8, (1), pp. 95–125 2