Color Filter Arrays

New Color Filter Arrays of High Light Sensitivity and High Demosaicking Performance Jue Wang1, Chao Zhang1, Pengwei Hao1,2 1 Peking University, 2 Queen Mary, University of London Outline • Introduction: What is CFA • How CFA works: demosaicking • CFA representation: frequency structure • CFA design: what should be optimized • CFA design: what constraints • New CFA designs & Evaluation • Conclusions Single-chip Color Cameras

• Single-Chip Camera is based on a color filter mosaic fabricated on top of the light sensors, and the mosaic is generally an array (Color Filter Array, CFA).

Lens On-chip color filter array

Bayer Kodak CFA2.0 Image sensor Color filter arrays Demosaicking • As each individual sensor only records one color, at each pixel all the three primary colors, red, green and blue, of a color image must be reconstructed using a computational interpolation method – demosaicking.

RGB pixels of CFA

The CFA pattern

CFA-filtered image Demosaicking

Demosaicked image A few Commercialized CFAs

• Bayer CFA pattern (Kodak, red, green, blue)

• CMY CFA (Kodak, cyan, yellow, magenta)

• RGBE CFA (Sony, red, green, blue, emerald)

• CYGM CFA (a few, cyan, yellow, green, magenta)

• CYGW CFA (JVC, yellow, cyan, green, unfiltered) CFAs proposed by researchers

• Gindele & Gallagher (with white, 2002)

• Parmar & Reeves (random, 2004)

• Hirakawa & Wolfe (better recovery, 2008)

• Condat (robust to noise, 2009)

• Hao, Li, Lin & Dubois (better demosaicking, 2011) Kodak’s CFA2.0

• Second generation CFA for high-light sensitivity

with 50% unfiltered pixels (2007) (aka panchromatic or white pixels) Periodical CFAs: Representation • Matrix representation in the spatial domain

just use one period Bayer CFA

G R    B G The Frequency Structure • Apply symbolic DFT to the one period matrix representation in the spatial domain

G R 1 R  2G  B B  R   FL FC 2  DFT         B G 4  R  B 2G  R  B  FC 2 FC1 

The spectrum (R+2G+B)/4 (B-R)/4 with Bayer CFA

(2G-R-B)/4 (R-B)/4 The Frequency Structure • Apply symbolic DFT to the one period matrix representation in the spatial domain

G R 1 R  2G  B B  R   FL FC 2  DFT         B G 4  R  B 2G  R  B  FC 2 FC1 

The spectrum (R+2G+B)/4 (B-R)/4 with Bayer CFA (periodical) (2G-R-B)/4 (R-B)/4 Relations: luma&chromas and RGB • It is a linear transform:

G R 1 R  2G  B B  R   FL FC 2  DFT         B G 4  R  B 2G  R  B  FC 2 FC1 

 FL   1 2 1  R R 1 F   1 2 1  G  T  G  C1  4       FC2  1 0 1  B B where T is a multiplexing matrix Spectra and Frequency Structures • DFT of the CFA-filtered image and CFA patterns

F 0 0   F  F   F F   F F  L F 0 F 0  L C ? L C 2 L C1  0 0 F  L C1           C1       FC 2 FC1  FC 2 0   0 FC 2 0  FC 2   0 FC 2 0  FC ?  FC ?   1 2 1  1 2 1 2 2 2   1 2 1     1 1 1 1 T  1 2 1 0 1 1 2 1 i 3 1 i 3 1 2 1      4   4   6   4     1 0 1  1 1 0 2 1 i 3 1 i 3 1 0 1     Demosaicking • First: to find the luma and the chromas by filtering/interpolation: FL, FC1 and FC2 . • Second: to find the RGB values by the inverse :  FL  R R  FL   FL  F   T  G  G  T 1  F   D  F   C1       C1   C1  FC 2  B B FC 2  FC 2  or  F   F   F  L R R L L F  F  F   C1        C1   C1   T  G  G  T   D  FC 2  FC 2  FC 2    B B              where D is a demosaicking matrix Demosaicking Optimization • First: to find more accurate luma and chromas: FL, FC1 and FC2 → further distance (less cross- talk) → to choose a good frequency structure. • Second: to find accurate RGB values from the inverse :  F  R L F     C1  G  D   min D FC 2  B      ǁ D ǁ is an objective function to be minimized. Constraints for High Light Sensitivity

• Some pixels are prescribed as unfiltered pixels, aka panchromatic or white pixels (W). • Pixels are only of primary colors, red, green and blue, and white: (i, j),CFA(i, j){R,G, B,W} • Then to optimize for high demosaicking performance.

• Not all frequency structures satisfy all the constraints – this makes the problem hard. A Frequency Structure and Optimization • The frequency structure:

FL 0 0 0 0   0 0 F 0 0   C 2  *  0 0 0 0 FC1     0 FC1 0 0 0   *   0 0 0 FC 2 0  • By optimization : min ǁ D ǁ New High Light Sensitivity CFA

• The corresponding CFA pattern (40% white, 5x5): F 0 0 0 0 W R B W G  L     0 0 F 0 0  W G W R B   C 2    * Symbolic IDFT  0 0 0 0 FC1   R B W G W       0 FC1 0 0 0  G W R B W   0 0 0 F * 0   B W G W R   C 2    • The multiplexing matrix T = Demosaicking Performance

Demosaicking Performance

Dubois’s demosaicing method

Condat’s demosaicing method More Challenges

• High light sensitivity • High noise tolerance • High dynamic range • Good color matching • Low manufacture costs • Easy and fast demosaicking • …… Thank you