Image Noise: Detection, Measurement and Removal Techniques

Zhifei Zhang Outline

 Noise measurement  Noise removal • Filter-based • Spatial domain • Block-based • Transform domain • Wavelet-based • Non-local methods • TV, SC, DL

2 Noise Measurement --- Noise Model

Additive White Gaussian Noise (AWGN)

풗풙풚 = 풖풙풚 + 풏풙풚

White noise True value Observation Without noise With Gaussian noise ퟐ 풏풙풚~퓝 ퟎ, 흈

3 Noise Measurement --- Filter-based

 Low-pass filter:

풏풙풚 = 풗풙풚 − 풌풍 ∗ 풗풙풚

풌풍 can be average, median, Gaussian, etc.

 High-pass filter:

풏풙풚 = 풌풉 ∗ 풗풙풚 (Faster)

풌풉 can be Laplacian, gradient, etc.

4 Noise Measurement --- Filter-based Limitation: See no difference between details and noise. Remedy: Edge detector + Laplacian filter [Tai, et al. 2008]

Edge Laplacian Averaging detection Convolution

−ퟏ −ퟐ −ퟏ 흅 ퟏ 푮 = ퟎ ퟎ ퟎ 흈 = 푰(풙, 풚) ∗ 푵 풙 ퟐ ퟔ(푾 − ퟐ)(푯 − ퟐ) ퟏ ퟐ ퟏ 풊풎풂품풆 푰 −ퟏ ퟎ ퟏ ퟏ −ퟐ ퟏ 푵 = 푮풚 = −ퟐ ퟎ ퟐ −ퟐ ퟒ −ퟐ −ퟏ ퟎ ퟏ ퟏ −ퟐ ퟏ 5 Noise Measurement --- Filter-based Another limitation: The filtered result is assumed to be the noise, which is not always true especially for images with complex structure or fine details. Remedy: Measure the noise only on those smooth blocks.

Gaussian Noise filter estimation

[D.-H. Shin, et al. 2005]

6 Noise Measurement --- Block-based How to determine the homogeneous blocks ? A homogeneous block is a block with uniform intensities.

 Compute local variation  Analyze local structure [Amer, et al. 2005]

7 Noise Measurement --- Block-based Limitation: Assume the existence of homogeneous blocks. Remedy: Principle component analysis [Pyatykh, et al. 2013] The noise variance can be estimated as the smallest eigenvalue of the image block covariance matrix.

Toy example:

1D signal {xk} = {1, 3, 1, 3, …} 2 Noise nk ~ N(0, 0.5 )

8 Noise Measurement --- Wavelet-based

풎풆풅풊풂풏 풄 Median Absolute Deviation (MAD) = 풊 풄풐풏풔풕풂풏풕 [D. L. Donoho, 1995]

9 Noise Measurement --- Wavelet-based Limitation: MAD assumes HH1 associates only to the noise, and it tends to overestimate the noise standard deviation under high SNR. Remedy: Adaptive thresholding • SureShrink [Donoho, et al. 1995] • BayesShrink [Simoncelli, et al. 1996] Non-local (BM3D) [Danielyan, et al. 2009]

10 Noise Measurement --- Wavelet-based

 Statistical model [Zoran, et al. 2009]

kurtosis of marginal distributions in clean natural images is constant throughout scales

11 Outline

 Noise measurement  Noise removal • Filter-based • Spatial domain • Block-based • Transform domain • Wavelet-based • Non-local methods • TV, SC, DL

12 Noise Removal --- Evolution

Spatial domain Non-local

Wiener Gaussian Steerable NL-means Average Anisotropic Adaptive neighbor BM3D Median Bilateral Kernel regression BM3D-SAPCA

1990 2000 2010 year

FFT DWT SB-TS GSM Min TV Spatial- UDWT SIWPD Curvelet Sparse coding frequency Thresholding MRF SA-DCT Deep learning

Transform domain Recent

13 Noise Removal --- Spatial Domain

Design of the kernel (sliding window)

Non-linear

Isotropic Anisotropic (Blur edges) (Preserve edges)

Kernel Median Average Gaussian Steerable Bilateral regression

14 Noise Removal --- Spatial Domain

Bilateral filter [Tomasi, et al. 1998]

A 100-gray-level step Similarity weights After bilateral filter perturbed by Gaussian noise

15 Noise Removal --- Spatial Domain

Kernel regression / Steerable filter [Takeda, et al. 2007]

16 Noise Removal --- Transform Domain

UDWT SB-TS DWT SIWPD Curvelet

Thresholding Statistics

Visu- Sure- Bayes- MRF HMM GSM Shrink Shrink Shrink

17 Noise Removal --- Transform Domain

 DWT thresholding Thresholded

Coefficient

Hard thresholding Soft thresholding Bayes thresholding

18 Noise Removal --- Transform Domain

 DWT statistical modeling --- HMMs [Crouse, et al. 1998]

19 Noise Removal --- Transform Domain

 DWT statistical modeling --- GSM [Portilla, et al. 2003]

풚 = 풙 + 휺 = 푧풖 + 휺

DWT coefficient histogram of noise-free

and Gaussian-noise image 20 Noise Removal --- Non-local Method

 Non local means (NL-means) [Buades, et al. 2005]

Given a discrete noisy image 풗 = 풗 풊 |풊 ∈ 푰

푵푳 풗 풊 = 풘 풊, 풋 풗(풋) 풋∈푰

ퟐ 풗(퓝풊)−풗(퓝풋) ퟏ − ퟐ,풂 풘 풊, 풋 = 풆 풉ퟐ 풁(풊)

21 Noise Removal --- Non-local Method

 BM3D [Dabov, et al. 2007]

Block matching + 3D transform + Thresholding

22 Noise Removal --- Non-local Method

 BM3D-SAPCA [Dabov, et al. 2009]

23 Noise Removal --- TV Minimization

풏

푻푽 = 푰풊풋 − 푰풊,풋+ퟏ + 푰풊풋 − 푰풊+ퟏ,풋 풊,풋=ퟏ

124 100 30 124 100 30 Lower TV 푰 = 69 80 200 69 80 200 66 92 211 66 92 211

24 Noise Removal --- TV Minimization

 ROF [Rudin, Osher and Fatemi, 1992] 풏 ퟏ ퟐ ퟐ ퟐ ퟐ 퐦퐢퐧 퐊퐮 − 퐛 ퟐ + 흀 훁풙 풖풊 + 훁풚 풖풊 퐮 풊=ퟏ

Fidelity term Regularization term (Gaussian noise) (TV) K --- linear operator (identity in denoising). b --- the observation . u --- denoised image. 25 Noise Removal --- TV Minimization

풏 ퟏ ퟐ ퟐ ퟐ ퟐ 퐦퐢퐧 퐮 − 퐛 ퟐ + 흀 훁풙 풖풊 + 훁풚 풖풊 퐮 풊=ퟏ

풏 Impulse noise 퐮 − 퐛 ퟎ 훁풙풖풊 + 훁풚풖풊 풊=ퟏ Laplace noise 퐮 − 퐛 ퟏ 풏 훁풙풖풊 ퟎ + 훁풚풖풊 Uniform noise 퐮 − 퐛 ∞ 풊=ퟏ ퟎ

Achieve best performance in [Xu, et al. 2011]

26 Noise Removal --- Sparse Coding

Dictionary 퐃

Observation 퐲

≈ 0.8 × + 0.3 × + 0.5 ×

Sparse coefficients 훂

27 Noise Removal --- Sparse Coding

 Sparse coding [Elad and Aharon, et al. 2006]

ퟐ 휶 = 퐚퐫퐠 퐦퐢퐧 퐃휶 − 퐲 ퟐ + 휆 휶 ퟎ 휶

 Dictionary learning • K-SVD [Aharon and Elad, et al. 2006] • K-LLD (learned local dictionary) [Chatterjee, et al. 2009]

28 Noise Removal --- Sparse Coding

 Learned simultaneous sparse coding (LSSC) [Mairal, et al. 2009] BM3D + grouped sparsity 풏 퐀풊 풑,풒 퐦퐢퐧풏 풑 퐀풊 풊=ퟏ,퐃∈퓒 풊=ퟏ 푺풊

ퟐ s.t. ∀풊 퐲 − 퐃휶 ≤ 휺 풋∈푺풊 풊 풊풋 ퟐ 풊

ퟐ 푺 ≜ 풋 = ퟏ, ⋯ , 풏 퐬. 퐭. 퐲 − 퐲 ≤ 흃 풊 풊 풋 ퟐ

Sparsity vs. simultaneous sparsity 퐀풊 = 휶풊풋 풋∈푺풊

29 Noise Removal --- Sparse Coding

 Clustering-based sparse representation (CSR) [Dong, et al. 2011]

푲 1 ퟐ 휶 = 퐚퐫퐠 퐦퐢퐧 퐃휶 − 퐲 ퟐ + 휆1 휶 1 + 휆2 휶풊 − 휷풌 1 휶 2 풌=ퟏ 풊∈푪풌

• Combine global thinking with local fitting Cluster-based regularization • Combine clustering and sparsity under a uniform framework

30 Noise Removal --- Deep Learning

 Stacked denoising auto-encoder (SDAE) [Vincent, et al. 2010]

31 Noise Removal --- Deep Learning

 Stacked denoising auto-encoder (SDAE) [Vincent, et al. 2010]

32 Noise Removal --- Deep Learning

Can neural networks compete with BM3D? [Burger, et al. 2012]

• Clean image 풙 • Noisy image 풛 by corrupting 풙 with noise • Denoised image 풚

• Minimize 풙 − 풚 ퟐ

33 Conclusion --- Development Tendency

Spatial domain  Transform domain Local statistics  Non-local statistics Thresholding  Statistical modeling Direct estimation  Regularized optimization

34 Conclusion --- State-of-the-Art

 Local in spatial domain Kernel regression [Takeda, et al. 2007]  Neighborhood in transform domain GSM [Portilla, et al. 2003]  Non-local in transform domain BM3D [Dabov, et al. 2007]  BM3D-SAPCA [Dabov, et al. 2009]  Sparse coding in transform domain K-SVD [Aharon and Elad, 2006]  CSR [Dong, et al. 2011]

35 Conclusion --- The Future

Non-local + Transform-domain + Sparsity

Intensity similarity, DFT, DWT, DCT Geometrical similarity PCA, SVD [Chatterjee, et al. 2012] [Rajwade, et al. 2013]

36 Thank You!