HybridHybrid ColorColor FilterFilter ArrayArray DemosaickingDemosaicking forfor EffectiveEffective ArtifactArtifact SuppressionSuppression

Journal of Electronic Imaging, vol. 15, no 1, Jan.-Mar. 2006 Lanlan Chang and Yap-Peng Tan

School of Electrical Engineering and Computer Science Kyungpook National Univ.

1 / 40 AbstractAbstract

‹ Simulation and analysis of past algorithms – Analysis of merits and artifacts according to each methods ‹ New CRA demosaicking algorithm – Combination and extension of before algorithms • Adoption of the each algorithm depending on the artifacts – Adaptive color-plane interpolation – Pattern recognition interpolation – Edge-adaptive median filtering

2 / 40 IntroductionIntroduction

‹ Demosaicking – Estimation of the RGB value from neighboring CFA samples • CCD or CMOS (single electronic sensor) • CFA (color filter array) – Bayer pattern

Fig 1. Bayer color filter array pattern. 3 / 40 ‹ Dominant artifacts from simple demosaicking – Zipper effects • “on-off” pattern around edges • Improper averaging of neighboring color values – False colors • spurious colors which is not in the original image around fine image details and edges • Inconsistency among the three color plane

Fig 2. Zipper effects and false colors 4 / 40 ‹ Four type of demosaicking – Consideration of only spatial correlation • Bilinear interpolation and cubic spline interpolation in each color plane • Artifacts in regions with ample edges and details – Consideration of litter or no spectral correlation • Adaptive interpolation in each color plane – Direction and local intensity patterns • Pattern recognition interpolation (PRI) and Edge-sensing (ES) • Suppression of the zipper effects but False colors

5 / 40 – Consideration of spectral correlation • Interpolation with the dependency among different color planes • Median interpolation, and Alternating projections (AP) • Alleviation of false color artifacts but visible zipper effects – Consideration of spatial and spectral correlation • Adaptive interpolation schemes and intercolor plane information • Interpolation in the color difference planes • Gradient-based interpolation, new edge-directed interpolation, and adaptive color-plane interpolation

6 / 40 ‹ Result of above algorithms

(a) False color (PRI) (b) Zipper effect (AP) (c) Zipper effect (ECI)

(d) False color (Laroche)(e) False color (Hamilton) (f) False color (PCSD)

Fig 3. Artifacts in the results obtained by different CFA demosaicking methods.

7 / 40 ‹ Adopted algorithm for hybrid demosaicking method – Reduction of zipper effects on the horizontal and vertical lines • Hamilton’s adaptive color-plane interpolation • Cok’s pattern recognition interpolation – Free man’s median interpolation • Remove of false colors

8 / 40 DemosaickingDemosaicking artifactsartifacts

‹ Zipper effects – Dominant at the green plane in bilinear interpolation

(a) Bilinear (c) G plane

(b) R plane (d) B plane

Fig 4-1. Zipper effects

9 / 40 – Other results to remove zipper effects

(e) ECI (f) R plane (g) G plane (h) B plane

(i) Hamilton (j) R plane (k) G plane (l) B plane

(m) Weighted ECI (n) R plane (o) G plane (p) B plane

Fig 4-1. Zipper effects 10 / 40 – Zipper artifacts on horizontal and vertical lines • Results from the bilinear interpolation – Zipper effect only on the vertical line

(a) Vertical line (c) Result of demosiacking

(b) Diagonal line (d) Result of demosiacking Fig 5. Synthetic images for test 11 / 40 – Analysis of the cause of zipper artifacts • Different errors due to different sampling patterns

• Zipper shape at the green vertical edge (L + 3H ) / 4 zipper effect (L + H ) / 2

(a) Vertical Edge (b) Interp. R plane (c) Interp. G plane (d) Interp. B plane (3L + H ) / 4 (3L + H ) / 4

(e) Horizontal Edge (f) Interp. R plane (g) Interp. G plane (h) Interp. B plane Fig 6. CFA samples 12 / 40 – ECI (effective color interpolation) method • Exploitation of high spectral correlation • Smooth color difference planes in local image regions – bilinear interpolation in color different space

(a) Result of vertical Edge on the G plane (b) Result of diagonal Edge on the G plane

Fig 7-1. Result of ECI method

13 / 40 • Procedure of ECI

– Definition K R = G − R

K B = G − B – First calculation (bilinear interpolation to find around K ) ' K R 3 = G3− R'3 = G3− (R1+ R7) / 2 ' K R 6 = G6 − R'6 = G3− (R5 + R7) / 2 – Interpolation of G value ' ' ' ' G'7 = R7 + (K R 3+ K R 6 + K R 8 + K R11) / 4 – Interpolation of R and B value

' ' R'3 = G3− (K R1+ K R 7) / 2 Bayer pattern ' ' ' ' B'7 = G'7 − (K B 2 + K B 4 + K B10 + K B12) / 4

14 / 40 ‹ Method for reducing zipper effects – Adaptive color-plane interpolation from Hamilton • Reduction of zipper effects in the G plane ˆ • Interpolation of G i , j along edges

Fig 8. Reference neighboring samples. G denotes green samples and C denotes red or blue samples

15 / 40 ˆ • Formula to calculation the Gi, j

⎧ G + G − C + 2C − C i, j−1 i, j+1 + i, j−2 i, j i, j+2 DH < DV ⎪ 2 2 ⎪ G + G − C + 2C − C ⎪ i−1, j i+1, j + i−2, j i, j i+2, j DH > DV ⎪ (1) Gˆ = 2 2 i, j ⎨ G + G + G + G ⎪ i−1, j i, j−1 i, j+1 i+1, j ⎪ 4 ⎪ − Ci−2, j − Ci, j−2 + 4Ci, j − Ci, j+2 − Ci+2, j ⎪ + DH = DV ⎩ 8 Where DH and DV are the edge indicators of horizontal and vertical directions. (5) DH = − Ci, j−2 + 2Ci, j − Ci, j+2 + Gi, j−1 − Gi, j+1 (6) DV = − Ci−2, j + 2Ci, j − Ci+2, j + Gi−1, j − Gi+1, j

16 / 40 ˆ • Formula to calculation the G i, j when DH = DV C + C ˆ ⎡ i, j−2 i, j Gi, j = Ci, j + ⎢Gi, j−1 − ⎣ 2

Ci, j + Ci, j+2 Ci−2, j + Ci, j + Gi, j+1 − + Gi−1, j − 2 2 (2)

Ci, j + Ci+2, j ⎤ + Gi+1, j − ⎥ / 4 2 ⎦

= Ci, j +[gˆci, j−1 + gˆci, j+1 + gˆci−1, j + gˆci+1, j ]/ 4

Where gˆ c i , j − 1 = G i , j− 1 − ( C i , j − 2 + C i , j ) / 2 is the estimate of the color difference value

at location ( i , j − 1 ) , while gˆ c i , j + 1 , g ˆ c i + 1 , j and g ˆ c i − 1 , j are the corresponding estimates at locations ( i , j + 1 ), ( i + , j ), and (i −1, j) gr = G − R gb = G − B

17 / 40 ˆ • Formula to calculation the G i, j when DH < DV

G − C + G − C Gˆ = C + − i, j−1 i, j−2 i, j+1 i, j+2 i, j i, j 2 gˆc + gˆc (3) = C + i, j−1 i, j+1 i, j 2

Where g ˆ c i, j − 1 = G i, j − 1 − C i , j− 2 and gˆ c i , j + 1 = G i , j+ 1 − Ci, j+2 are is the estimate of the color difference values in the two neighboring location (i, j −1) and (i, j +1)

Fig 7. Result of Hamelt’s method 18 / 40 – Adaptive weighted method proposed by Kimmel • Suppression of zipper effects by weighted interpolation

w ×Gˆ ˆ ∑k (k ) (k ) (4) Gi, j = ∑ w(k )

ˆ Where G ( k ) is the estimate of the missing green value, and w(k ) is the weight along the direction k

• Weights setting – Proportional values according to the respective direction from spatial correlation

ECI Weighted ECI

19 / 40 – Pattern recognition interpolation propose by Cok • Usage of median filters –3х3 median filter for edge І and edge ІІ which means horizontal direction –5х5 median filter for stripe and corner which means horizontal direction • Unsatisfied results – Some errors in high frequency area due to the direction – Computational complexity with the 5х5 filter

(a) edge І (b) edge ІІ (c) stripe (d) coner

Fig 9. Four neighborhood patterns unique up to a rotation, where H/L denotes that the green value of the pixel is larger/smaller than the average of all four green values. 20 / 40 ‹ False colors – Due to inconsistency among the three color planes • Inconsistency from spurious intensity changes in the color difference plane • False colors from the spurious edges in fig 10-(e) by bilinear interpolation

(d) (e) (f)

Fig 10. Color difference space (green minus red) (d) original , (b) result by bilinear interpolation, and (c) Freeman’s method 21 / 40 ‹ Method for eliminating false colors – Freeman’s median filtering • Smoothness in the color difference planes after bilinear interpolation • Reduction of false color and preservation of the sharp edged

(a) (b) (c)

Fig 10. (a) original image, (b) result by bilinear interpolation, and (c) Freeman’s method 22 / 40 – Deficit by using median filter on whole images • Zipper effect around the border – Due to smearing of the texture besides edge region in color difference planes – Need to avoid the median filter on the edges » Edge-adaptive median filtering we proposed

(a) Original (b) bilinear (c) median filter to (e) median filter to the entire image the smooth image region Fig 11. Result of median filter 23 / 40 ProposedProposed hybridhybrid CFACFA demosaickingdemosaicking methodmethod

‹ Whole procedure

Hemilton’s method Cok’s method

Kimmel’s method

Proposed method

Fig 12. Procedure of the proposed hybrid CFA demosaicking method.

24 / 40 ‹ Reconstruction process – A. Initialization • 1) Green planes – Adaptive color-plane interpolation with fewer zipper effect on the horizontal and vertical directions – Production of jagged edges in the other directions • 2) Red and blue planes – Pattern adaptive interpolation for reducing diagonal jagged effect – Similar to fig 9 proposed by Cok – Consideration both spatial and spectral correlations

25 / 40 – First step for the pattern adaptive interpolation » Interpolation of the missing red value at blue pixels » Four neighboring color difference at only blue point » Obtainment of the edge information along diagonal gr = G − R at (i −1, j −1),(i −1, j +1),(i +1, j −1),(i +1, j +1) 1. Obtainment of gr value at the blue pixel R,G(gr) G R,G(gr) for edge pattern and corner pattern

G G,B G gˆri, j = median{gˆri−1, j−1, gˆri−1, j+1, gˆri+1, j−1, gˆri+1, j+1} (7)

R,G(gr) G R,G(gr) for strip pattern

⎧(gri−1, j−1 + gri+1, j+1) / 2 D1< D2 (8) gˆri, j = ⎨ ⎩(gri−1, j+1 + gri+1, j−1) / 2 D1≥ D2

Where D1 =| Ri−1, j−1 − Ri+1, j+1 |

D2 =| Ri−1, j+1 − Ri+1, j−1 | 2. Estimation of R ˆ ˆ Ri, j = Gi, j − gˆri, j 26 / 40 » Fill of the missing red at green pixels gr = G − R at (i −1, j),(i, j +1),(i +1, j),(i, j −1)

Selection of g ˆ r i , j with median filter G R,G(gr) G Calculation of R ˆ ˆ R,G,B(gr) G R,G,B(gr) Ri, j = Gi, j − gˆri, j

G R,G(gr) G

» Calculation of the blue value with the same method above

» Obtainment of whole RGB ,and g ˆ r i , j and gˆ b i , j values

27 / 40 – B. Enhancement • Reduction of the remaining artifacts in the diagonal edges and high-detail regions with edge-weighted interpolation • Update of green plane (edge-weighted interpolation) ˆ – Estimation of Gi, j 4 Gˆα 4 ˆ (k ) 1 Gi, j = ∑ ∑ (11) k =1 1+ (k ) i=k 1+α(k ) α Where ˆ G(1) = Ci, j + gˆci−1, j (1) = Ci−2, j − Ci, j + Gi−1, j − Gi+1, j + Ci−1, j − Ci, j Gˆ = C + gˆc α = C − C + G − G + C − C (2) i, j i+1, j (2) i+2, j i, j i+1, j i−1, j i+1, j i, j (9),(10) ˆ α G(3) = Ci, j + gˆci, j−1 (3) = Ci, j−2 − Ci, j + Gi, j−1 − Gi, j+1 + Ci, j+1 − Ci, j ˆ ˆ G(4) = Ci, j + gci, j+1, α(4) = Ci, j+2 − Ci, j + Gi, j+1 − Gi, j−1 + Ci, j+1 − Ci, j gˆc = gr or gb

• Update of red and blue plane (pattern adaptive interpolation) – Good work with sharp edges and fine details

28 / 40 – Result of reconstruction process

(a) Initialization (b) Enhancement

(c) Initialization (d) Enhancement Fig 13. Improvement of the enhancement step.

29 / 40 ‹ Refinement process to reduce of the false colors – A. Edge-adaptive median filtering •5х5 median filter to suppress spurious edges on the color difference planes • Convolution of the median-filtered clor difference planes ( Gc ) image with Laplacian filter to detect the edge region E = Gc* H ⎡0.0909 0.8182 0.0909⎤ (12) ⎢ ⎥ H = ⎢0.8182 − 3.6364 0.8182⎥ ⎣⎢0.0909 0.8182 0.0909⎦⎥ • Obtainment of the binary image ⎧1 if E(i, j) > T B(i, j) = ⎨ (13) ⎩0 otherwise, • Performance of the edge-adaptive median filtering – Filtering except the edge region

30 / 40 – B. Updating • Update the three color planes with the color difference values in the smooth regions – Red and blue planes update in smooth regions » Obtainment of the red/blue values at each green pixel (gˆc + gˆc ) Cˆ = G − i−2, j i, j (14) i−1, j i−1, j 2 » Update of the red values at blue pixels and vice versa (pattern adaptive interpolation) – Green plane update in smooth regions » Usage of pattern adaptive interpolation

31 / 40 – Result of refinement process and efficiency Table 1. Performance comparison: PSNR values in dB of the red, green, and blue planes are listed in the 1st, 2nd, and 3rd rows after each method.

(a) Reconstruction (b) Refinement

(c) Reconstruction (d) Refinement Fig 14. Improvement of the enhancement step. 32 / 40 ExperimentalExperimental resultsresults

‹ Visual Evaluation

(a) Original (b) Bilinear (c) PRI

(d) Freeman (e) Hamilton (f) ECI

(g) NECI (h) AP (i) Proposed Fig 16. Demosaicking on the vertical edges 33 / 40 (a) Original (b) Bilinear (c) PRI

(d) Freeman (e) Hamilton (f) ECI

(g) NECI (h) AP (i) Proposed Fig 17. Sharpness and false colors 34 / 40 (a) Original (b) Bilinear (c) PRI

(d) Freeman (e) Hamilton (f) ECI

(g) NECI (h) AP (i) Proposed

Fig 18. Fine details and various directions 35 / 40 (a) Original (b) AHD (c) Proposed

(d) Original (e) AHD (f) Proposed

Fig 21. Comparison with AHD which exploits both the spatial and spectral correlations

36 / 40 Red (a) step1 (4.08%) (b) step2 (2.85%) (c) step3 (2.10%)

Green (a) step1 (3.16%) (b) step2 (2.01%) (c) step3 (0.74%)

Fig 22. Differences with respective to the originals in the red, green, and blue planes after each main step. (a) Initialization (b) Enhancement (c) Refinement

Blue (a) step1 (3.56%) (b) step2 (2.40%) (c) step3 (1.32%) 37 / 40 ‹ Objective comparison – Table 2. PSNR performances

Table 3. Average PSNR results in dB when one of the three main steps is not included in the proposed method.

Fig 23. PSNR results of the red, green, 38 / 40 and blue planes obtained after each main step. ‹ Computational complexity – Proposed method • 80 additions, 15 multiplications, 14 bit-shifts, 33 absolute operations, 8 comparisons, and 4 median operations –AHD • 97 additions, 48 multiplications (including bit-shift), 12 absolute operations, and 8 median operations –AP • 384 additions and multiplications for 3 iterations – Close to AHD and lower than AP

39 / 40 ConclusionConclusion

‹ Hybrid CFA demosaicking method – Elimination of two main artifacts of CFA • Zipper effects and false colors – Incorporation of both spatial and spectral colrrelations –Result • Great suppression of the demosaicking artifacts • Best PSNR

40 / 40