New Color Filter Arrays of High Light Sensitivity and High Demosaicking Performance Jue Wang1, Chao Zhang1, Pengwei Hao1,2 1 Peking University, 2 Queen Mary, University of London Outline • Introduction: What is CFA • How CFA works: demosaicking • CFA representation: frequency structure • CFA design: what should be optimized • CFA design: what constraints • New CFA designs & Evaluation • Conclusions Single-chip Color Cameras • Single-Chip Camera is based on a color filter mosaic fabricated on top of the light sensors, and the mosaic is generally an array (Color Filter Array, CFA). Lens On-chip color filter array Bayer Kodak CFA2.0 Image sensor Color filter arrays Demosaicking • As each individual sensor only records one color, at each pixel all the three primary colors, red, green and blue, of a color image must be reconstructed using a computational interpolation method – demosaicking. RGB pixels of CFA The CFA pattern CFA-filtered image Demosaicking Demosaicked image A few Commercialized CFAs • Bayer CFA pattern (Kodak, red, green, blue) • CMY CFA (Kodak, cyan, yellow, magenta) • RGBE CFA (Sony, red, green, blue, emerald) • CYGM CFA (a few, cyan, yellow, green, magenta) • CYGW CFA (JVC, yellow, cyan, green, unfiltered) CFAs proposed by researchers • Gindele & Gallagher (with white, 2002) • Parmar & Reeves (random, 2004) • Hirakawa & Wolfe (better recovery, 2008) • Condat (robust to noise, 2009) • Hao, Li, Lin & Dubois (better demosaicking, 2011) Kodak’s CFA2.0 • Second generation CFA for high-light sensitivity with 50% unfiltered pixels (2007) (aka panchromatic or white pixels) Periodical CFAs: Representation • Matrix representation in the spatial domain just use one period Bayer CFA G R B G The Frequency Structure • Apply symbolic DFT to the one period matrix representation in the spatial domain G R 1 R 2G B B R FL FC 2 DFT B G 4 R B 2G R B FC 2 FC1 The spectrum (R+2G+B)/4 (B-R)/4 with Bayer CFA (2G-R-B)/4 (R-B)/4 The Frequency Structure • Apply symbolic DFT to the one period matrix representation in the spatial domain G R 1 R 2G B B R FL FC 2 DFT B G 4 R B 2G R B FC 2 FC1 The spectrum (R+2G+B)/4 (B-R)/4 with Bayer CFA (periodical) (2G-R-B)/4 (R-B)/4 Relations: luma&chromas and RGB • It is a linear transform: G R 1 R 2G B B R FL FC 2 DFT B G 4 R B 2G R B FC 2 FC1 FL 1 2 1 R R 1 F 1 2 1 G T G C1 4 FC2 1 0 1 B B where T is a multiplexing matrix Spectra and Frequency Structures • DFT of the CFA-filtered image and CFA patterns F 0 0 F F F F F F L F 0 F 0 L C ? L C 2 L C1 0 0 F L C1 C1 FC 2 FC1 FC 2 0 0 FC 2 0 FC 2 0 FC 2 0 FC ? FC ? 1 2 1 1 2 1 2 2 2 1 2 1 1 1 1 1 T 1 2 1 0 1 1 2 1 i 3 1 i 3 1 2 1 4 4 6 4 1 0 1 1 1 0 2 1 i 3 1 i 3 1 0 1 Demosaicking • First: to find the luma and the chromas by filtering/interpolation: FL, FC1 and FC2 . • Second: to find the RGB values by the inverse : FL R R FL FL F T G G T 1 F D F C1 C1 C1 FC 2 B B FC 2 FC 2 or F F F L R R L L F F F C1 C1 C1 T G G T D FC 2 FC 2 FC 2 B B where D is a demosaicking matrix Demosaicking Optimization • First: to find more accurate luma and chromas: FL, FC1 and FC2 → further distance (less cross- talk) → to choose a good frequency structure. • Second: to find accurate RGB values from the inverse : F R L F C1 G D min D FC 2 B ǁ D ǁ is an objective function to be minimized. Constraints for High Light Sensitivity • Some pixels are prescribed as unfiltered pixels, aka panchromatic or white pixels (W). • Pixels are only of primary colors, red, green and blue, and white: (i, j),CFA(i, j){R,G, B,W} • Then to optimize for high demosaicking performance. • Not all frequency structures satisfy all the constraints – this makes the problem hard. A Frequency Structure and Optimization • The frequency structure: FL 0 0 0 0 0 0 F 0 0 C 2 * 0 0 0 0 FC1 0 FC1 0 0 0 * 0 0 0 FC 2 0 • By optimization : min ǁ D ǁ New High Light Sensitivity CFA • The corresponding CFA pattern (40% white, 5x5): F 0 0 0 0 W R B W G L 0 0 F 0 0 W G W R B C 2 * Symbolic IDFT 0 0 0 0 FC1 R B W G W 0 FC1 0 0 0 G W R B W 0 0 0 F * 0 B W G W R C 2 • The multiplexing matrix T = Demosaicking Performance Demosaicking Performance Dubois’s demosaicing method Condat’s demosaicing method More Challenges • High light sensitivity • High noise tolerance • High dynamic range • Good color matching • Low manufacture costs • Easy and fast demosaicking • …… Thank you .