An Efficient Combined Demosaicing and Zooming Algorithm for Digital Camera
Total Page:16
File Type:pdf, Size:1020Kb
An Efficient Combined Demosaicing and Zooming Algorithm for Digital Camera King-Hong Chung, Yuk-Hee Chan, Chang-Hong Fu and Yui-Lam Chan Department of Electronic and Information Engineering The Hong Kong Polytechnic University, Hong Kong Abstract—Color demosaicing and digital zooming are model [6]. Simulation results show that the proposed common processes in digital cameras and they often employ algorithm is superior to conventional approaches, which are similar interpolation concepts based on the information generally combinations of different demosaicing and zooming extracted from the raw sensor data. Realizing them algorithms, in producing zoomed full-color images in terms independently is not efficient as separate extraction processes of output image quality and complexity. are required. It may also cause inconsistent utilization of the raw sensor data in different stages. This paper presents a This paper is organized as follows. In Section II, a green low-complexity combined algorithm which directly extracts edge plane demosaicing scheme, which is used in the proposed information from raw sensor data and exploits it consistently combined demosaicing and zooming algorithm, is presented. and efficiently in both demosaicing and zooming. The proposed The details of our combined demosaicing and zooming algorithm can produce zoomed full-color images and zoomed algorithm is then described in Section III. Experimental CFA images with outstanding performance as compared with results and a complexity analysis are respectively provided in conventional approaches. Section IV and V. Index Terms—Interpolation, Color Filter Array, Cameras j-2 j-1 j j+1 j+2 j-2 j-1 j j+1 j+2 I. INTRODUCTION i-2 R G R G R B G B G B i-1 G B G B G G R G R G Image sensors covered with Bayer color filter array (CFA) i R G R G R B G B G B [1] are widely used nowadays in many portable electronic i+1 G B G B G G R G R G devices like digital cameras and mobile phones to acquire i+2 R G R G R B G B G B scene. Their output images (CFA images) contain only one (a) (b) color component at each pixel location as shown in Fig. 1. To Fig. 1 – Two 5×5 regions of Bayer pattern having centers at (a) red convert them to full-color images, the two missing color and (b) blue samples components of each pixel have to be estimated by color demosaicing [2-7]. CFA Color Full Color Color Image Zoomed Full Traditionally, to produce a zoomed full-color image, Image s Demosaicing Image Zooming Color Image S (a) digital zooming is performed after demosaicing as shown in CFA CFA Image Zoomed Color Zoomed Full Fig. 2a. The zooming can be achieved by applying either Image s Zooming CFA Image Demosaicing Color Image S plane-wise interpolation [8] or vector color interpolation [9] (b) on the demosaiced image. However, demosaicing always CFA Combined Color Demosaicking and Zoomed Full introduces visual artifacts such as blurred edges and false Image s Image Zooming Color Image S colors [10]. These artifacts may be amplified during the (c) zooming process and the quality of the resultant zoomed Fig. 2 – Image zooming methods: (a) zooming after demosaicing, (b) image can be very poor in regions of complicated details. demosaicing after zooming, and (c) the proposed combined Recently, an alternative approach [11,12] to produce a demosaicing and zooming zoomed full-color image has been proposed as shown in Fig. 2b to improve the zoom performance and to reduce the II. GREEN PLANE DEMOSAICING SCHEME realization complexity. It zooms the CFA image first so that The proposed scheme is designed based on the conventional demosaicing technique can be applied to the demosaicing method proposed in [7]. It interpolates the enlarged CFA image to produce a zoomed full-color image. missing green samples in a CFA image by selecting one of the Both approaches carry out demosaicing and zooming three directional second-order Laplacian interpolators [2]. As individually. As demosaicing and zooming often employs an example, the missing green sample of the center pixel (i,j) similar interpolation concepts, performing these two in Fig.1a is estimated with one of the following directional processes independently may cause inconsistent and interpolators. inefficient utilization of the raw sensor data. H Gi, j 1 Gi, j 1 2Ri, j Ri, j 2 Ri, j 2 In this paper, a simple combined demosaicing and Horizontal (H ) : gi, j (1) zooming method is proposed as shown in Fig. 2c to solve this 2 4 G G 2R R R problem. By sharing the edge information extracted directly Vertical (V ) : g V i1, j i1, j i, j i2, j i2, j (2) form the raw sensor data, the green plane in the CFA image is i, j 2 4 first enlarged effectively and efficiently. The red and the blue g H g V Diagonal (D) : g D i, j i, j (3) missing samples in the enlarged image are then estimated i, j 2 with the interpolated green plane and the color difference 1-4244-1437-7/07/$20.00 ©2007 IEEE IV - 197 ICIP 2007 1st pass 2nd pass For each missing For each remaining Ri2k, j gi2k, j if gi2k, j was estimated and d (7) green sample position missing green sample i2k, j ® V ¯Ri2k, j gi2k, j otherwise H V Find L , L and E Find Hĭi,j, Vĭi,j and Dĭi,j As for the diagonal absolute moment Dĭi,j, since there is No Leave it to Yes no green sample available along the diagonal axes of the E>T? nd Is ĭ minimum? Dir =H the 2 pass H i,j i,j window, it is evaluated as Yes No H V No Yes Ȱ 2 2 2 2 L >L ? Diri,j=H Is Vĭi,j minimum? Diri,j=V 1 1 1 1 D i, j ¦¦di, j2k di, j2n ¦¦di2k, j di2n, j . (8) Yes No 2 kn 2 5 2 2 kn 2 5 2 Diri,j=V Diri,j=D where the color difference d is found in a way similar to Fig. 3 – Procedures for determining the direction to interpolate a p,q missing green component in the proposed green plane demosaicing D that in eqn.(7) but the preliminary green estimates gi, j2k and scheme g D are utilized instead of g H and g V respectively. The selection of the interpolator is critical to the i2k, j i, j2k i2k, j demosaicing performance [7]. The proposed scheme exploits a two-pass decision scheme. Fig. 3 summarizes the workflow As color differences are more or less the same within a of selecting an interpolator for a particular pixel in different small region [6], the interpolation direction for pixel (i,j) passes of the proposed scheme. The details are as follows. should be the one which provides the minimum absolute moment of color differences. Hence, the missing green Pass 1 component of pixel, gi,j, is determined as follows. In this pass, all the pixels in sharp edge regions are H Ȱ Ȱ Ȱ Ȱ Diri, j H and gi, j gi, j if H i, j min(H i, j ,V i, j ,D i, j ) processed. Similar to the algorithm proposed in [7], the ° Ȱ Ȱ Ȱ Ȱ Dir V and g gV if min( , , ) (9) scheme starts with determining the horizontal and vertical ® i, j i, j i, j V iȰ, j H iȰ, j V iȰ, j D iȰ, j H V ° D edge levels, L and L , of a pixel by using the local intensity ¯Diri, j D and gi, j gi, j if D i, j min(H i, j ,V i, j ,D i, j ) gradient and the color difference information in the pixel’s For the case in which the concerned pixel is the center 5×5 local region. As an example, for pixel (i,j) in Fig. 1a, we pixel of a region shown in Fig. 1b, one can treat blue samples have as red samples and follow the above procedures to estimate 2 2 LH X X and LV X X the missing green component. A complete demosaiced green ¦¦ im, jk im, j ¦¦ ik, jm i, jm km r1,r2 2 km r1,r2 2 plane is obtained after Pass 2. (4) III. COMBINED DEMOSAICING AND ZOOMING ALGORITHM where Xp,q denotes the known sample at pixel position (p,q). The edge level ratio E=max(LH/LV,LV/LH) is then As the green channel provides double samples than the computed and compared with a predefined threshold T. If it is others in a CFA image, the proposed combined algorithm larger than T, the pixel is said to be in a sharp edge region. Its starts with green-plane interpolation. The interpolation of the red plane and the blue plane then follows. Assume that a CFA interpolation direction Diri,j {H,V,D} and the missing green image s of size M×N has to be enlarged to a zoomed full-color component gi,j are then determined as image S of size ȜM×ȜN, where Ȝ=2z is a zooming factor and z H V H °Diri, j H and gi, j gi, j if L L ! T is a positive integer. In this paper, Ȝ=2 is selected for . (5) ® V H V ¯°Diri, j V and gi, j gi, j if L L ! T simplicity to facilitate the following discussion. For the sake of reference, hereafter, pixels at location (p,q) Otherwise, it is said to be in a flat or texture region and left in image s and image S are respectively represented by behind for being processed in Pass 2.