Color demosaicing in YUV color space Jayanta Mukhopadhyay Manfred K. Lang Department of Computer Science & Engineering Institute for Human-Machine Communication, Indian Institute of Technology, Kharagpur Technical University of Munich, India 721302. D-80290 Munich, Germany Email: [email protected] Email: [email protected] Sanjit K. Mitra Department of Electrical and Computer Engineering, University of California, Santa Barbara, CA, 93106, USA. Email: [email protected] ABSTRACT Ê Ê In a single-chip digital color imaging sensor, a color fil- ter array (CFA) is used to obtain sampled spectral com- ponents (red, green and blue) in an interleaved fashion. Ê Ê Color demosaicing is the process of interpolating these regularly spaced sampled values into the dense pixel maps for each spectral components. In this paper we present techniques for interpolating the color images in Figure 1. The Bayer pattern. the YUV color space. The resulting interpolated images could be directly used in the DCT based JPEG compres- sion scheme. As the final results are desired in the DCT space, we have also used the concept of subband DCT eras provide the output interpolated image in the DCT computation for interpolating the individual components based JPEG compression standard. Most of the exist- of color images in the DCT domain. Based on a simple ing interpolation algorithms are based on the RGB color Í Î strategy for computing , and components, several space. This implies that further computations are re- modifications are also proposed for improving the quality quired for converting these interpolated images into the of the reconstructed images. We have observed that me- YUV space, which are subsequently transformed into dian filtering of the chrominance components improves DCTs and subjected to different encoding stages of the the end results remarkably. JPEG compression scheme [6]. In our work we propose to reduce this overhead by directly interpolating in the KEY WORDS YUV space. Moreover, we have used the concept of sub- Digital Color Imaging, Color Interpolation, Color De- band DCT computation [7] for image interpolation in the mosaicing, Color Filter Array (CFA), RGB color space, DCT-domain. In [8], algorithms for image resizing using YUV color space, Discrete Cosine Transform (DCT), subband DCT computations are presented. We have used Subband DCT. the image-upsampling (or image-doubling operation as called in [8]) algorithm for the purpose of interpolation 1. Introduction in this work. In this algorithm, DCT coefficients for ev- ¢ ery block are converted to DCT coefficients of a ¢ ½ ½ block using subband DCT theory [7]. Later, in- Single-sensor cameras [1], [2], [3], [4], for color image ¢ ½ verse DCTs of ½ blocks provide the upsampled acquisition use color filter arrays (CFAs) to obtain sam- images. The details are discussed in [8]. pled red, green and blue pixel data (or luminance and In the next section, we present our algorithms of chrominance signals) in an interleaved fashion. To this color interpolation. Finally, we have discussed the mer- end, different checker-board patterns are used as color its and demerits of the proposed techniques against other filter arrays (CFAs) [3], of which the Bayer pattern CFA known methods. [5] (shown in Figure 1) is more commonly employed and is considered in this paper. In Figure 1 the sampled color components are denoted by R (for red), G (for green) and 2. Color Interpolation in YUV space B (for blue). ¢ ¾ From the sampled color pixel data, the missing Let us partition the CFA in smaller blocks of sizes ¾ , color pixel values are interpolated to obtain dense pixel where each block consists of the color masks as depicted maps in all three spectral components. The process of in Figure 2. interpolating these sparse data into dense pixel maps Let us also denote the pixels in the corresponding ¢ ¾ is commonly known as color interpolation or color ¾ blocks in the mosaiced image as shown in Figure 3. demosaicing. Interestingly many of the digital cam- Let the functions for converting a pixel having Ö , and Table 1. Recovery of individual components using G R SYUV interpolation PSNR images Y U V B G (dB) (dB) (dB) Statue 28.49 31.16 31.88 Lighthouse 24.92 28.68 29.40 Window 27.71 30.25 30.88 Sail 27.67 31.19 31.75 ¢ ¾ Figure 2. A ¾ block in the CFA. Pepper 26.83 29.95 29.90 End Simple YUV interpolation (SYUV) Ö ½½ ½¾ It may be noted that as in the JPEG compression Í Î ¾¾ ¾½ standard, downsampled and components are used, it is not necessary to upsample them. However, for the component it is necessary to upsample. For experi- mentations, we created the Bayer pattern array from a Figure 3. Pixels in the mosaiced image for the corre- number of original color images. Next, full color images ¢ ¾ sponding ¾ block. were generated from each of the Bayer pattern arrays us- ing the SYUV algorithm described above. In Figure 4 we present a typical example of image-reconstruction. It values (for R, G and B spectral components respectively) may be noted that in presenting our results (also in sub- to a pixel in YUV space be denoted as: sequent sections) we do not perform any kind of quan- tization or compression on the DCT coefficients. One ´Ö µ Ý can observe that the quality of the reconstructed image is Í ´Ö µ Ù (1) poor and many of the details are blurred. In some cases, Î ´Ö µ Ú false colors also appear near the edges (specially near ‘achromatic’ edges). The performance of the algorithm The transformation is linear and given by the following Í Î in recovering individual , and components for var- equation: ious images are shown in Table 1. ¿ ¿ ¾ ¿ ¾ ¾ Ö ¼¾ ¼ ¼½½ Ý ¼½ ¿ ¼¿¿ ¼ ½½ Ù ¼ ½½ ¼ ¾ ¼¼¿ Ú (2) Now, the algorithm is described below: Algorithm Simple YUV interpolation (SYUV) Input: Mosaiced image obtained through the Bayer CFA. Output: Interpolated image in the compressed domain. Begin ¢ ¾ 1. For each ¾ block of the input image do the fol- lowing: (a) Statue (original) (b) by SYUV Ö Ú (a) Compute y, u and v values from ½¾ , and ¾½ as follows, Figure 4. Reconstructed images by SYUV ´ · µ¾ Ú ½½ ¾¾ Ý ´Ö µ Ú ¾½ ½¾ (3) Interestingly, one could observe from Table 1 that Ù Í ´Ö µ ½¾ Ú ¾½ Î Í and components are reconstructed more reliably Ú Î ´Ö µ ½¾ Ú ¾½ than the component. The justifications for this prop- Ù Ú Í (b) Store Ý , and values as downsampled , erty could be given from the fact that the transformations and Î components respectively. of the image in YUV from the RGB space are linear. For Î a downsampled Í and components, values are formed by the averages over red, green and blue components in ¢ ¾ a ¾ block. Hence, the expected deviations of the rep- 2. Compute DCT of and upsample it using SBDCT computation as described in [8]. resentative (true) sample values in the mosaiced image from the average values are less (following the central Î 3. Compute DCT’s of downsampled Í and compo- limit theorem and assuming that the probability distribu- nents. tion of a spectral component in a small neighborhood is Table 2. Recovery of individual components using YUVG interpolation PSNR images Y U V (dB) (dB) (dB) Statue 32.85 31.50 32.13 Lighthouse 30.68 28.90 29.67 Window 32.50 30.41 31.25 Sail 33.16 31.46 32.14 Pepper 29.35 30.22 30.50 (a) (b) Gaussian). But this is not true for . Here, we have to in- terpolate them in full resolution. Hence we have consid- ered following strategies for improving the performance of our proposed technique. 2.1 Computations through ‘green’ inter- polation For improving the performance of SYUV algorithm, we have to improve the quality of reconstruction of the component. As the green component plays the domi- (c) nant role in determining the luminance ( ) component (see Eq. (2)), we propose to carry out interpolation of the green component in the first step using any conven- Figure 5. Reconstructed images by : (a) YUVG (b) YU- tional technique and then use these values to determine VGM and (c) YUVGMSB Í Î , and subsequently. In our work we have used an edge correlated interpolation technique for interpolat- ing the green component [9]. In these techniques [10], [11], horizontal and vertical gradient values are used in the interpolation computation. The algorithm makes use of only the pixel values in the sampled array in the in- terpolation which are lying along the least gradient path Table 3. Recovery of individual components using YU- (either horizontal or vertical). We have implemented the VGM interpolation method proposed by Hamilton et al. [9]. In this case PSNR the interpolated values are corrected from the second or- images Y U V der derivatives of the spectral components. For a missing (dB) (dB) (dB) Statue 32.91 40.38 41.82 green value at a blue (red) pixel in the sampled array, Lighthouse 30.77 35.23 35.97 the second order derivatives of the blue (red) pixel val- Window 32.53 36.52 37.96 Sail 33.25 39.60 40.90 ues along the same direction are added with the average Pepper 29.46 34.25 33.65 green values. In Figure 5(a), the reconstructed image obtained by this algorithm (to be called the YUVG algorithm) is shown. One could observe that there is a considerable improvement in the quality of the reconstruction. But in some cases one could observe false colors near the matic edges of the color image. We have adopted a very edges. The improvement in the recovery of compo- simple strategy for suppressing false colors.