A New Color Filter Array with Optimal Sensing Properties Laurent Condat
Total Page:16
File Type:pdf, Size:1020Kb
A new color filter array with optimal sensing properties Laurent Condat To cite this version: Laurent Condat. A new color filter array with optimal sensing properties. Actes de IEEE ICIP,Nov 2009, Le Caire, Egypt. pp.457 - 460, 10.1109/ICIP.2009.5414383. hal-00814058 HAL Id: hal-00814058 https://hal.archives-ouvertes.fr/hal-00814058 Submitted on 16 Apr 2013 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. A NEW COLOR FILTER ARRAY WITH OPTIMAL SENSING PROPERTIES Laurent Condat∗ GREYC 6BdduMar´echal Juin 14050 CAEN Cedex, France ABSTRACT 2. SPECTRAL CHARACTERIZATION OF CFAS We propose a new color filter array (CFA) with optimal charac- k = teristics for the acquisition of color scenes, since the luminance and In this article, boldface quantities denote vectors, e.g. [k ,k ]T ∈ Z2 0 =[0, 0]T chrominance information is encoded in the mosaicked image with 1 2 and . WedefineaCFA cfa =(cfa[k]) cfa[k]= the best achievable robustness to aliasing and noise. Our 2 × 3 pat- ˆas a color image ˜ k∈Z2 ,where R G B T 3 tern is based on the paradigm recently introduced by Hirakawa et cfa [k], cfa [k], cfa [k] ∈ [0, 1] is the color value in the al. [1], which focuses on the spectral properties of CFAs. Moreover, R,G,B basis of the filter centered at the location k. The color val- these superior properties are fully exploited by a simple, linear and ues, constrained to lie in [0, 1] for physical realizability, correspond T efficient demosaicking method. to opacity rates: the white color [1, 1, 1] stands for a transparent filter. We only consider periodic CFAs defined on the square lattice, Index Terms— Color filter array (CFA), demosaicking, spatio- and we denote by N1 × N2 the size of its smallest repeating pattern. spectral sampling, noise sensitivity. We define the color image im =(im[k])k∈Z2 as the ground truth to be estimated by the demosaicking process. The mosaicked T 2 1. INTRODUCTION image v =(v[k])k∈Z2 is such that v[k]=im[k] cfa[k], ∀k ∈ Z . In practice, a random term modeling the effect of noise has to be In digital still and video cameras, a color filter array (CFA) is over- added to this model [5]. laid on the sensor, in order to encode the color information as high It is well known that in natural images, the R,G,B compo- frequency content in the mosaicked image [2, 1]. Hence, the demo- nents are not independent [6, 7]. Thus, we define the orthonor- saicking task, which consists in reconstructing a color image with mal basis corresponding to luminance, red/blue and green/magenta 1 T 1 T its three R,G,B channels, essentially amounts to appropriately as- chrominances, as L = √ [1, 1, 1] , C1 = √ [1, 0, −1] , C2 = signing the frequency content of the mosaicked image to these three 3 2 √1 [−1, 2, −1]T. uL uC1 uC2 channels [2, 3]. Based on this observation, Hirakawa et al. re- 6 We denote by , ,and the components u cently reformulated the problem of CFA design as the minimization of a color signal in this basis. They can be considered statistically of aliasing between the baseband luminance and the chrominance independent for natural images [6]. modulated at high frequencies in the Fourier domain [1]. A similar In order to analyse the properties of the Bayer CFA, Alleysson methodology was proposed in [4]. In fact, most of the demosaick- et al. showed that the mosaicked image v can be interpreted, in the ing artifacts which occur with the popular Bayer CFA originate from Fourier domain, as the sum of the luminance and chrominance com- aliasing between the luminance and the R/B chrominance bands of ponents of the color image im, moved at different locations of the the color scene, since the latter is modulated on the vertical and hor- frequency plane [2]. This characterization can be extended to every izontal axes of the frequency plane, where most luminance energy is CFA, by simply writing cfa as the sum of its Fourier components: present [2]. Therefore, the CFA has to be designed so that its chromi- “ ” P N1 P N2 nance is modulated farther away from the origin of the frequency X 2 2 X 2πn1 2πn2 cfa [k]= N −1 n =0 αn cos k1 + k2 + n = 1 2 N1 N2 plane, to maximize the robustness to aliasing. However, many pa- 1 “ 2 ” rameters are left free with this framework and the problem of noise X 2πn1 2πn2 βn sin k1 + k2 , (1) sensitivity is not adressed. In this article, we exhibit the admissible N1 N2 pattern having maximum chrominance energy for a given luminance 2 sensitivity. This provides the best achievable signal-to-noise ratio for every X ∈{L, C1,C2} and k ∈ Z . So, designing cfa X X when the mosaicked image is contaminated by sensor noise. amounts to choosing its 3N1N2 Fourier coefficients αn and βn The paper is organized as follows. In Section 2, we express the appropriately. For this, we express the Fourier transform vˆ(ω)= P −jω Tk design problem in the Fourier domain. In Section 3, we enforce some k∈Z2 v[k]e in function of the Fourier transforms of the constraints to ensure optimal sensitivity properties for the CFA. A components of im: new 2 × 3 pattern shows up as the unique solution. In Section 4, we present a simple and efficient linear demosaicking method, which P P N1 P N2 vˆ(ω)= 2 2 , X∈{L,C1,C2} n = N1−1 n2=0 fully exploits the spectral characteristics of our CFA. 1 2 „ “ h i ” “ h i ”« ∗ αX T T Part of this work was performed during the stay of the au- n imdX ω+ 2πn1 , 2πn2 + imdX ω− 2πn1 , 2πn2 + thor in the Helmholtz Zentrum M¨unchen, supported by the Marie 2 N1 N2 N1 N2 Curie Excellence Team Grant MEXT-CT-2004-013477, Acronym „ “ h i ” “ h i ”« βX T T MAMEBIA, funded by the European Commission. Contact: n imdX ω+ 2πn1 , 2πn2 − imdX ω− 2πn1 , 2πn2 , [email protected]. 2j N1 N2 N1 N2 978-1-4244-5654-3/09/$26.00 ©2009 IEEE 457 ICIP 2009 for every ω ∈ R2. Hence, whatever the CFA, vˆ is the sum of the lu- d minance and chrominance components imX , replicated at the sites of the dual lattice induced by the periodicity of the pattern. This is essentially the task of demosaicking to separate these components back. Given this observation, Hirakawa et al. had the ingenious idea to directly design the CFA in the Fourier domain, so that the com- d ponents imX are well separated in vˆ [1]. In their framework, the L L baseband luminance is at the origin (cfa [k]=α0 , in (1) ) and the chrominance is modulated far away from it, on the boundaries of the Nyquist band (for this, we assume N1 =2)andfarawayfrom the axes of the frequency plane [1]. This constrains the coefficients X X αn ,βn to some extent, but the question of further defining the re- maining degrees of freedom is left open. This is the aim of this work to show how to tune these parameters to obtain a CFA with optimal sensitivity characteristics. Bayer Hirakawa et al. [1] Proposed Fig. 1. The thee CFAs considered in this work (top) and the respec- 3. A NEW CFA WITH OPTIMAL SENSITIVITY tive results of our demosaicking method on a grayscale synthetic CHARACTERISTICS zoneplate pattern (bottom), to illustrate their spectral properties. We now construct our new CFA step by step, by enforcing several design criteria in the frequency domain. that, for fixed N2 and n2, the desired values in (3)-(4) are √ First, in order to maximally reduce the overlaps between the C 3 π channels, we impose the chrominance to be shifted at only one fre- γ = ,ϕ= , (5) T 4cos(ϕ) lcm(6,N2) quency ±ω0, with ω0 =[π,ω0] and ω0 =2πn2/N2 for some integer n2 ∈ (1, (N2 − 1)/2) relatively prime with N2. Hence, where lcm denotes the least common multiple. Consequently, the the luminance information is concentrated around 0 in vˆ, while value N2 =3(therefore, n2 =1) maximizes the chrominance gain C the chrominance information is around ±ω0. Moreover, the two γ (the other solution N2 =6, n2 =1is not interesting, since its chrominance bands should be orthogonal, so that they can be op- modulating frequency [π,π/3]T is closer to 0). timally separated during demosaicking. In comparison with the de- The obtained 2 × 3 pattern, depicted in Fig. 1, has a small [1, 0, 1 ] [1, 1 , 0] signs in [1], whith the chrominance spread at several frequencies, the number (six) of distinct filters, with the colors 2 , 2 , risk of inter-chrominance aliasing is drastically reduced. We also im- [0, 1, 1 ] [0, 1 , 1] [ 1 , 0, 1] [ 1 , 1, 0] 2 , 2 , 2 and 2 . Its modulation frequency pose that the carrier waves of the two chrominance channels have the T ω0 =[π,2π/3] is ideally located. It is far away from the origin same magnitude, so that the color discrimination of the CFA is the and the axes of the frequency plane, but not too close to [π,π]T, same for every color, without privileged chrominance axis.