Image Filtering: Noise Removal, Sharpening, Deblurring

Yao Wang Polytechnic University, Brooklyn, NY11201 http://eeweb.poly.edu/~yao Outline

• Noise removal by averaging filter • Noise removal by median filter • Sharpening (Edge enhancement) • Deblurring

©Yao Wang, 2006 EE3414: Image Filtering 2 Noise Removal (Image Smoothing)

• An image may be “dirty” (with dots, speckles,stains) • Noise removal: – To remove speckles/dots on an image – Dots can be modeled as impulses (salt-and-pepper or speckle) or continuously varying (Gaussian noise) – Can be removed by taking mean or median values of neighboring pixels (e.g. 3x3 window) – Equivalent to low-pass filtering • Problem with low-pass filtering – May blur edges – More advanced techniques: adaptive, edge preserving

©Yao Wang, 2006 EE3414: Image Filtering 3 Example

©Yao Wang, 2006 EE3414: Image Filtering 4 Averaging Filter

• Replace each pixel by the average of pixels in a square window surrounding this pixel • Trade-off between noise removal and detail preserving: – Larger window -> can remove noise more effectively, but also blur the details/edges

©Yao Wang, 2006 EE3414: Image Filtering 5 Example: 3x3 average

100 100 100 100 100

100 200 205 203 100

100 195 200 200 100

100 200 205 195 100

100 100 100 100 100

100 144 167 145 100

100 167 200 168 100

100 144 166 144 100

100 100 100 100 100

©Yao Wang, 2006 EE3414: Image Filtering 6 Example

©Yao Wang, 2006 EE3414: Image Filtering 7 Weighted Averaging Filter

• Instead of averaging all the pixel values in the window, give the closer-by pixels higher weighting, and far-away pixels lower weighting. L L g(m,n) = ∑ ∑h(k,l)s(m − k,n − l) l=−L k=−L

• This type of operation for arbitrary weighting matrices is generally called “2-D convolution or filtering”. When all the weights are positive, it corresponds to weighted average. • Weighted average filter retains low frequency and suppresses high frequency = low-pass filter

©Yao Wang, 2006 EE3414: Image Filtering 8 Graphical Illustration

©Yao Wang, 2006 EE3414: Image Filtering 9 Example Weighting Mask

1 1 1 1 2 1 1 × 1 1 1 1 × 2 4 2 9 16 1 1 1 1 2 1

All weights must sum to one

©Yao Wang, 2006 EE3414: Image Filtering 10 Example: Weighted Average

100 100 100 100 100

100 200 205 203 100

100 195 200 200 100 1 1 1 1 2 1 1 1 × 1 1 1 100 200 205 195 100 × 2 4 2 9 16 1 1 1 100 100 100 100 100 1 2 1

100 100 100 100 100 100 100 100 100 100 100 144 167 145 100 100 156 176 158 100

100 167 200 168 100 100 174 201 175 100

100 144 166 144 100 100 156 175 156 100

100 100 100 100 100 100 100 100 100 100

©Yao Wang, 2006 EE3414: Image Filtering 11 Filtering in 1-D: a Review

• Continuous-Time Signal τ ∞ τ Time Domain : g(t) = s(t)∗h(t) = h( )s(t − )dτ ∫−∞ Frequency Domain : G( f ) = S( f )H ( f ) ∞ Filter frequency response : H ( f ) = h(t)e− j2πft dt ∫−∞

• Extension to discrete time 1D signals

Time Domain (linear convolution) : s(n) = s(n)∗h(n) = ∑ h(m)s(n − m) m Frequency Domain : G( f ) = S( f )H ( f ) Filter frequency response (DTFT) : H ( f ) = ∑ h(n)e− j 2πfn n H ( f ) is periodic, only needs to look at the range f ∈(-1/ 2,1/ 2 )

1/2 corresponds to fs / 2

©Yao Wang, 2006 EE3414: Image Filtering 12 Filtering in 2D

Weighted averaging = 2D Linear Convolution

l1 k1 g(m,n) = ∑ ∑ h(k,l)s(m − k,n − l) = = l l0 k k0

= In 2D frequency domain G( f1, f2 ) S( f1, f2 )H ( f1, f2 ) Frequency response of the 2D Filter

l1 k1 − π + = j2 ( f1m f2n) H ( f1, f2 ) ∑ ∑ h(m,n)e = = m l0 n k0

H ( f1, f2 ) is periodic, only needs to look at the square region ∈ − ∈ − f1 ( 1/ 2,1/ 2), f2 ( 1/ 2,1/ 2).

©Yao Wang, 2006 EE3414: Image Filtering 13 Frequency Response of Averaging Filters

0 0 0 0 0 • Averaging over a 3x3   0 1 1 1 0 1 window h(m,n) = 0 1 1 1 0 9   0 1 1 1 0 0 0 0 0 0 π  

π π − ⋅ − + ⋅ − − ⋅ + ⋅ − − ⋅ + ⋅ − 1 j2 ( f1 ( 1) f2 ( 1)) j2 ( f1 (0) f2 ( 1)) j2 ( f1 (1) f2 ( 1)) H ( f , f ) = ()e π + e + e 1 2 π π 9

1 − ⋅ − + ⋅ − ⋅ + ⋅ π − ⋅ + ⋅ + ()j2 ( f1 ( 1) f2 (0)) + j2 ( f1 (0) f2 (0)) + j2 ( f1 (1) f2 (0)) e πe e 9 π 1 − ⋅ − + ⋅ − ⋅ + ⋅ − ⋅ + ⋅ + ()e j2 ( f1 ( 1) f2π(1)) + e j2 ( f1 (0) f2 (1)) +πe j2 ( f1 (1) f2 (1)) 9 π π π 1 j2 f − j2 f j2 f 1 j2 f − j2 f 1 j2 f − j2 f − j2 f = ()1 + + π 1 2 + ()1 + + 1 ⋅ + ()1 + + 1 2 e 1 e e e 1 e π 1 e 1 e e 9 π 9 9π π 1 − − π 1 = ()()e j2 f1 +1+ e j2 f1 e j2 f2 +1+ e j2 f2 = ()()1+ 2cos 2 f 1+ 2cos 2πf 9 9 1 π 2 π

©Yao Wang, 2006 EE3414: Image Filtering 14 π = H ( f1, f2 ) H ( f1)H ( f2 ) 1 1 H ( f ) = ()1+ 2cos 2 f , H ( f ) = ()1+ 2cos 2πf 1 3 1 2 3 2

Sketch H(f1)

0.5

-0.5 30 25 20 20 15 10 10 5 0 0

©Yao Wang, 2006 EE3414: Image Filtering 15 Frequency Response of Weighted Averaging Filters

1 b 1 1     = 1 2 = 1 [] H 2 b b b 2 b 1 b 1 ; (1+ b) (1+ b) 1 b π1 1 H (u,v) = ()()b + 2cos(2 u) b + 2cos(2πv) /(1+ b)2

0.8

0.6

0.4

0.2

0 30 25 20 20 15 10 10 5 0 0

©Yao Wang, 2006 EE3414: Image Filtering 16 Averaging vs. Weighted Averaging

1 1 1 1 1 b 1 1 1     1   1   H = 1 1 1 = 1 []1 1 1; H = b b2 b = b []1 b 1 ;     + 2   + 2   9 π (1 b) (1 b) 1 1 1 1 1 b π1 1 H (u,v) = ()()1+ 2cos(2 u) 1+ 2cos(2πv) / 9 H (u,v) = ()()b + 2cos(2 u) b + 2cos(2πv) /(1+ b)2

1 1

0.8 0.5 0.6

0 0.4

0.2 -0.5 30 0 30 25 20 20 25 20 15 20 10 10 15 5 10 10 0 0 5 0 0

b=2

©Yao Wang, 2006 EE3414: Image Filtering 17 Interpretation in Freq Domain

Filter response

Low-passed image spectrum

Original image spectrum

Noise spectrum

0 f

Noise typically spans entire frequency range, where as natural images have predominantly lower frequency components

©Yao Wang, 2006 EE3414: Image Filtering 18 Median Filter

• Problem with Averaging Filter – Blur edges and details in an image – Not effective for impulse noise (Salt-and-pepper) • Median filter: – Taking the median value instead of the average or weighted average of pixels in the window • Median: sort all the pixels in an increasing order, take the middle one – The window shape does not need to be a square – Special shapes can preserve line structures • Order-statistics filter – Instead of taking the mean, rank all pixel values in the window, take the n-th order value. – E.g. max or min

©Yao Wang, 2006 EE3414: Image Filtering 19 Example: 3x3 Median

100 100 100 100 100

100 200 205 203 100

100 195 200 200 100

100 200 205 195 100

100 100 100 100 100 100 100 100 100 100

100 100 200 100 100

100 200 200 200 100

100 100 195 100 100

100 100 100 100 100

Matlab command: medfilt2(A,[3 3])

©Yao Wang, 2006 EE3414: Image Filtering 20 Example

©Yao Wang, 2006 EE3414: Image Filtering 21 Matlab Demo: nrfiltdemo

Original Image Corrupted Image Filtered Image

Can choose between mean, median and adaptive (Wiener) filter with different window size

©Yao Wang, 2006 EE3414: Image Filtering 22 Noise Removal by Averaging Multiple Images

©Yao Wang, 2006 EE3414: Image Filtering 23 Image Sharpening

• Sharpening : to enhance line structures or other details in an image • Enhanced image = original image + scaled version of the line structures and edges in the image • Line structures and edges can be obtained by applying a difference operator (=high pass filter) on the image • Combined operation is still a weighted averaging operation, but some weights can be negative, and the sum=1. • In frequency domain, the filter has the “high- emphasis” character

©Yao Wang, 2006 EE3414: Image Filtering 24 Frequency Domain Interpretation

Filter response (high emphasis)

sharpened image spectrum

Original image spectrum

0 f

• Spatial operation: taking difference between current and averaging (weighted averaging) of nearby pixels – Can be interpreted as weighted averaging = linear convolution – Can be used for edge detection • Example filters

0 1 0  0 −1 0  1 1 1 −1 −1 −1         1 − 4 1; −1 4 −1; 1 −8 1; −1 8 −1; 0 1 0  0 −1 0  1 1 1 −1 −1 −1

– All coefficients sum to 0!

 0 −1 0  − − −    1 1 1 −1 4 −1     −1 8 −1  −   0 1 0  −1 −1 −1

6 12

10 4 8

2 6

4 0 30 2

0 20 30 25 20 10 25 20 20 15 15 10 10 10 0 5 0 5 0 0

Original image Isotropic edge detection Binary image

• The desired image is the original plus an appropriately scaled high-passed image • Sharpening filter = + λ fs f fh = δ + λ hs (m,n) (m,n) hh (m,n)

f(x) g(x)=f(x)*hh(x) fs(x)=f(x)+ag(x)

x x x

f(x) g(x)=f(x)*hh(x) fs(x)=f(x)+ag(x)

x x x  0 −1 0   0 −1 0  = 1 − −  ⇒ = 1 − −  λ = H h  1 4 1 H s  1 8 1 with 1. 4 4  0 −1 0   0 −1 0 

−1 −1 −1 −1 −1 −1 = 1 − −  ⇒ = 1 − −  λ = H h  1 8 1 H s  1 16 1 with 1. 8 8 −1 −1 −1 −1 −1 −1

−1 −1 −1 = 1 − −  H h  1 8 1 4 −1 −1 −1

−1 −1 −1 = 1 − −  H s  1 16 1 8 −1 −1 −1

λ = 4

λ = 8

• How to smooth the noise without blurring the details too much? • How to enhance edges without amplifying noise? • Still a active research area

Courtesy of Ivan Selesnick

Taking an image without injecting a contrast agent first. Then take the image again after the organ is injected some special contrast agent (which go into the bloodstreams only). Then subtract the two images --- A popular technique in medical imaging

• Noise removal considered thus far assumes the image is corrupted by additive noise – Each pixel is corrupted by a noise value, independent of neighboring pixels • Image blurring – When the camera moves while taking a picture – Or when the object moves – Each pixel value is the sum of surrounding pixels The blurred image is a filtered version of the original • Deblurring methods: • Inverse filter: can adversely amplify noise • Wiener filter = generalized inverse filter • Many advanced adaptive techniques

• How does averaging and weighted averaging filter works? • How does median filter works? • What method is better for additive Gaussian noise? • What method is better for salt-and-pepper noise? • How does high-pass filtering and sharpening work? • For smoothing and sharpening: – Can perform spatial filtering using given filters – Deriving frequency response is not required, but should know the desired shape for the frequency responses in different applications • What is the challenge in noise removal and sharpening? • What causes blurring? • Principle of deblurring: technical details not required

Gonzalez and Woods, Digital image processing,2nd edition, Prentice Hall, 2002. Chap 4 Sec 4.3, 4.4; Chap 5 Sec 5.1 – 5.3 (pages 167-184 and 220-243)