Computational Photography Filtering: Smoothing Objectives Example

• Point-process and Neighboring Pixels Computations on an Image for Image Smoothing Computational Photography

CS 4475/6475 Maria Hybinette

1 Maria Hybinette 2 Maria Hybinette

Objectives From Pixel/Point Operations to Groups of Pixels

• Smooth an image while considering a neighborhood n of pixels – Average filtering (use the average of n) – Median filtering (use the median of n) • special non-linear filtering and smoothing • Previously we used pixel to pixel arithmetic (e.g., add(), sub())) approach • How to Smooth a Signal? • Approach: Use a group of pixels : Neighboring Values of a neighborhood. 1. Moving Average [1 1 1 1 1] X 1/5 2. Weighted Moving Average [1 4 6 4 1] X 1/16

3 Maria Hybinette 4 Maria Hybinette

Example: 3x3 Kernel Example: 3x3 Kernel

• Smoothing Process over an Image using Averages • 1*90/9=10, 2*90/9=20, ….

5 Maria Hybinette 6 Maria Hybinette Results in Shades of Gray

• What about the edges?

7 Maria Hybinette 8 Maria Hybinette

Edges Mathematical Notation, Observations, and Computing the Result Matrix • A Small image ‘h’ is “rubbed” over (larger) image ‘F’ that we are smoothing • Kernel: h – Example: 3x3 area around each original pixel used – Neighborhood size – k=1, window size = 3. – k >1, • Window size is 2k+1 – k=1, 2(1)+1 = 3 – Our kernel is 3x3 – Recall : A pixel is referenced by (i, j) • Expand the image we are smoothing: • Want Result Matrix ‘G’ – Wrap around pixels. (3x3 ﬁlter needs one more pixel – F à G using h. on each border, 5x5 needs 2) – G = h (operation) F – 9 Copy edges Maria Hybinette 10 Maria Hybinette

Computing the Values ofA Result Mathematical Matrix RepresentationSmoothing when for a- iSmoothing are all 1s.

Observations (continued)1 G[3, 3] = (A + B + C + D + E + FG[3, 3] = 1/9 (A + B + C + D + E + F + G + H + I ) + G + H + I) a b c 20 20 10 20 10 20 10 10 13 9 30 0 0 0 0 0 0 0 30 1 k k d e f k k G[i, j] = F[i + u, j + v] Let’s refer to F[i,j] as input and 1 2 ∑ ∑ 20 0 90A 90B 90C 90 90 0 20 20 20 10 20 10 20 10 10 13(2k +1) u k v k • Let’s refer to G[i, j]= F [i + u, j + v] =− =− g h i 20 0 90 0 90 90 90 0 20 2 30 0 0 0 0 0 0 0 30 20 0 D E F 90 90 0 20 G[i,j] as output, h[i,j] as the kernel(2k + 1) 1 – F[i,j] as input and u= k v=20 0k 90A 90B 90C 90 90 0 20 10 0 90 90 90 90 90 0 10 X X • ( 2 k + 1 ) 2 “Standardizes” the weight to 10 0 G H I 90 90 0 10 – G[i,j] as output 20 0 90D E0 90F 90 90 0 20 10 0 90 90 90 90 90 0 10 a b c A B C 10 0 90G 90H 90I 90 each90 0 10 pixel . (k=1, then 1/9) 10 0 90 90 90 90 90 0 10 G[3, 3] = a A + b B + c C + d D 10 0 90 90 90 90 90 0 10 10 0 90 90 90 90 k90 k 0 10 10 0 90 90 90 90 90 0 10 – h[i,j] as the kernel⇤ ⇤ ⇤ ⇤ d e f D E F F[i u, j v] + e E + f F + h H + i I A 10Mathematical0 90 90 90•90 ∑ 90 ∑ 0 10 + + Loop over allRepresentation pixels in 20 0 0 0 0 0 0 for0 20 Smoothing u=−k v=−k 20 0 0 0 0 0 0 0 20 • Example: pixel⇤ (3,3) ⇤(expanded⇤ edges⇤ above)g h i G H I 20 0 0 0 0 0 neighborhood0 0 20 around particular 20 20 10 20 10 20 10 10 13 20 20 10 20 10 20 10 10 13 20 20 10 20 10 20 10 10 13 – G[3, 3] = a*A + b*B + c*C + d*D +e*E + f*F + h*H + i*I 1 image pixel F[i,j] Attribute uniform1/9 1/9 1/9 Loop over all pixels in a b c 20 20 10 20 10 20 10 10 13 1 G[3, 3] = (A +– Bstarting+ C from+ upperD + rightE corner,+ F -k,+ -k,G if+ H + I) 1/9 1/9 1/9 9 G[3, 3] = (A + B + C + D + E + F + G + H + I) k=1, then 1 left, from center and up 1. 30 0 0 0 0 0 0 0 30 9 k k d e f – If all weights (a through i) are 1: weight to1/9 each1/9 1/9 neighborhood1 around 20 0 90A 90B 90C 90 90 0 20 G[i, j]= F [i + u, j + v] g h i 11 – G[3, 3] = 1/9 (A + B + C + D + E + F + G + H + I ) Maria Hybinette (2k 12+ 1)2 Maria Hybinette 20 0 90D 0E 90F 90 90 0 20 © 2015 Irfan Essa, Georgia Tech, All Rights Reserved pixel imageu= k v =pixelk F[i,j] 10 0 90 90 90 90 90 0 10 X X 10 0 G H I 90 90 0 10 10 0 90 90 90 90 90 0 10 10 0 90 90 90 90 90 0 10 10 0 90 90 90 90 90 0 10 10 0 90 90 90 90 90 0 10 20 0 0 0 0 0 0 0 20 20 0 0 0 0 0 0 0 20 © 2015 Irfan Essa, Georgia Tech, All Rights Reserved 20 20 10 20 10 20 10 10 13 Attribute uniform Loop over all pixels in20 20 10 20 10 20 10 10 13 weight to each neighborhood around pixel image pixel F[i,j]

1 G[3, 3] = (A + B + C + D + E +G[3, 3] = 1/9 (A + B + C + D + E + F + G + H + I ) F + G + H + I) a b c 20 20 10 20 10 20 10 10 13

9 k k 30 0 0 0 0 0 0 0 30 k k 1 d e f G[i, j] = F[i + u, j + v] 20 0 90A 90B 90C 90 90 0 20 1 (2k +1)2 ∑ ∑ G[i, j]= F [i + u, j + v] u=−k v=−k g h i (2k + 1)2 20 0 90D 0E 90F 90 90 0 20 u= k v= k 10 0 90 90 90 90 90 0 10 X X • More Generally, 10 0 G H I 90 90 0 10 10 0 90 90 90 90 90 0 10 10 0 90 90 90 90 90 0 10 10 0 90 90 90 90 90 0 10 10 0 90 90 90 90 90 0 10 G[i, j] = a * A + b* B + c*C + d * D + e* E + f * F + h * H +i* I 20 0 0 0 0 0 0 0 20 A Mathematical Representation20 0 0 0 0 0 0 0for20 Smoothing k k 20 20 10 20 10 20 10 10 13 G[i, j] = ∑ ∑ h[u, v]* F[i + u, j + v] 20 20 10 20 10 20 10 10 13 1 u=−k v=−k • Uniform Kernel (all 1s). Attribute uniformG[3, 3] = Loop( A + B over+ C + Dall+ E +pixelsF + G + inH + I) a b c 20 20 10 20 10 20 10– Box10 Area13 above is 21x21 the larger the kernel the ‘smoother’ the 9 result. • Formula is Referred to Cross 30 0 0 0 0 0 0 0 30 k k d e f • The larger the box the smoother it gets. weight to each neighborhood1Correlation more onaround this later. 20 0 90A 90B 90C 90 90 0 20 G[i, j]= F [i + u, j + v] g h i (2k 13+ 1)2 Maria Hybinette 20 0 90D 0E 90F14 90 h7ps://90 0 en.wikipedia.org20 /wiki/Box_blur Maria Hybinette pixel imageu= k v =pixelk F[i,j] 10 0 90 90 90 90 90 0 10 X X 10 0 G H I 90 90 0 10 10 0 90 90 90 90 90 0 10 10 0 90 90 90 90 90 0 10 10 0 90 90 90 90 90 0 10 10 0 90 90 90 90 90 0 10 20 0 0 0 0 0 0 0 20 20 0 0 0 0 0 0 0 20 © 2015 Irfan Essa, Georgia Tech, All Rights Reserved 20 20 10 20 10 20 10 10 13 20 20 10 20 10 20 10 10 13 SpecialSpecialAttribute Case:Case: uniform MedianMedian Loop over Filtering all pixels in Specialweight Example:to eachCase: Median neighborhoodMedian Filter for smoothing aroundFiltering Comparison 20 20 1 20 10 20 10 13 13 Special Case: Median Filtering 20 20 1 20 10 20 10 13 13 pixel 20 20 1 image pixel F[i,j] 0 0 200 200 120 2020 10 20 10 13 13 20 20 1 20 0 0 0 0 0 0 20 20 20 20 1 20 10 20 10 13 13 20 0 0 0 0 0 0 20 20 20 0 0 20 2020 20 1 median(20 0 0 0 0 0 0 20)20 median( 0 0 0 1 20 20 20 20 90 ) 90 902090 090 020 020 0 0 0 20 20 20 0 0 20 0 0 0 0 0 1 20 20 20 20 90 = 20 20 0 90 90 90 90 90 20 20 median(20 0 90 90 90 90 90 20)20 median(median( ) median( 0 0 0 1 20 20 ) 20= 20 90 ) 20 0 90 90 90 90 90 20 20 20 0 90 20 200 0 900 0 0 0 1 20=20 2020 20 90 20 0 90 90 90 90 90 20 2020 02090median(0 090 90 90 20 20 ) Medianmedian( = 20 (removes outliers) ) = 20 20 0 90 0 90 90 90 20 20 10 0 90 90 90 90 90 10 10 20 0 90 0 0 0 1 20 20 20 20 90 20 0 90 0 90 90 90 20 20 average( ) = 19 20 0 90 0 90 90 90 20 2010 0 90 90 90 90 90 10 10 10 0 90 90 90 90 ©90 2015 Irfan10 Essa,10 Georgia Tech, All Rights Reserved 10 0 90 90 90 90 90 10 10 0 0 0 1 20 20 20 20 90 10 0 90 90 90 90 90 10 10 average( 0 0 0 1 20 20 20 20 90 ) = 19 10 0 90 90 90 90 90 10 1020 0 0 0 0 0 0 20 20 average(F [i,j] 0 0 0 )1 20= 20 1920 20 90 10 0 90 90 90 90 90 10 10 20 20 10 20 10 20 10 13 13 average( ) = 19 10 0 90 90 90 90 90 10 10 Average=171/9 = 19 10 0 90 90 901090 090 9010 9010 90 90 90 10 10 10 0 90 90 90 90 90 10 10 Apply median filter to all 3x3 regions over the 20 0 0 0 0 100 00 9020 9020 90 90 90 10 10 20 0 0 0 0 0 0 20 20 FF [i,j][i,j] • Medianwhole Filterimage :(as Nonlinear we would filter for averaging) 20 20 10 20 102020 010 013 013 0 0 0 20 20 F [i,j] 20 20 10 20 10 20 10 13 13 © 2015 Irfan Essa,– Georgia Reduces Tech, All Rights Reserved noise (the outliers are ‘basically’ noise and 20 20 10 20 10 20 10 13 13 removed). • Median Filter Preserves Edges (11x11 median filter) – Preserves edges • Sharper Edges ApplyApply medianmedian filterfilterh7ps:// totoen.wikipedia.org allall /wiki/3x33x3Median_filter regionsregions over the Apply median15 filter to all 3x3 regionsMaria Hybinette over the16 Maria Hybinette wholewhole imageimagewhole (as(asimage wewe wouldwould(as we for forwould averaging) for averaging)

Median Filter vs. Average Filter Example: Median Filter for Noise Removal

• Salt and Pepper Noise (3x3 median ﬁlter) • Median (11x11) on left, Average Filter on Right

17 Maria Hybinette 18 Maria Hybinette Noise Removal Median vs. Average

• Image Smoothing • Applying a kernel to smooth an image (cross correlation) • Applying a function (median) to smooth an image.

• 3x3 Median Filter on left, 3x3 Average Filter on Right

19 Maria Hybinette 20 Maria Hybinette

Cross Correlation Cross-CorrelationCross Correlation Method k k k k G[i, j]= k k h[u, v]F [i + u, j + v] G[i, j] = h[u, v]* F[i + u, j + v] G[i, j] = h[u, v]* F[i + u, j + v] ∑ ∑ u= ∑k v∑= k u=−k v=−k Xu=−k vX=−k Denoted by G = h F • In signal processing, cross-correlation is a measure ⌦ • Filtering an image: of similarity of two waveforms as a function of a Filtering an image: time-lag applied to one of them. – Replace each pixel with a linear combination of its neighbors – Can ﬁnd pattern in images with this approach, the Replace each pixel with a linear combination of its neighbors kernel is the pattern we are looking for. • Filter “kernel” or “mask” Filter “kernel– h[u,v”] is theor prescription“mask” for weights in the linear • Also known as a sliding dot product or sliding combination inner-product. h[u,v] is the prescription for weights in the linear combination 21 Maria Hybinette 22 Maria Hybinette

Example: Box Filter Box/Average Filter • Revisit:Size: 21x21 Box Filter Example:Example: Gaussian Gaussian Filter Filter Values:Example: Uniform Box Filter • Gaussian Filter Box/Average Filter • NormalSize: 21x21 or Gaussian Distribution of kernel • Box/AverageSize: 21x21 Filter • – Size• 21 x 21 – Give center values more importance. Values: Uniform • Values: Gaussian – Values• are uniform Example: Gaussianor FilterNormal distribution © 2015 Irfan Essa, Georgia Tech, All Rights Reserved Gaussian Filter • Size: 21x21 • Values: Gaussian Example: Box Filter or Normal distribution

h7ps://en.wikipedia.org/wiki/Gaussian_blur Original Result 23 © 2015 Irfan Essa, Georgia Tech, All Rights Reserved Maria Hybinette 24 Maria Hybinette Varying the ‘Spread’ or Standard Average Filter vs. Gaussian Filter Deviation (σ)

• Average is more blurry • Gaussian ﬁlter Gives more ‘detail’ to center pixel, • As sigma gets larger more blurry. Extends how far it – Preserves edges more smoothens the images.

25 Maria Hybinette 26 Maria Hybinette

Correlation to Convolution Result of Impulse Function

• Example : Take Cross Correlation of Impulse Kernel. • Notice : Reﬂected Result (switching rows and column of the kernel into result image)

27 Maria Hybinette 28 Maria Hybinette

Another way to look at result Convolution Method

• Convolution is a mathematical operation on two functions F and h, • Produces a third function that is typically viewed as a modiﬁed version • Gives the area of overlap between the two functions • In a form of the amount that one of the original • Reverse response functions is translated.

29 Maria Hybinette 30 Maria Hybinette Convolution Method k k GConvolution[i, j]= h [u,Method v]F [i u, j(*)v ] Example: Linear Filter u= k v= k Linear Filters X X original, 64 X 64 Denoted kbyk G = h F G[i, j] = ∑ ∑ h[u, v]* F[i − u, j⇤− v] Flip filter in u=both−k v=−k dimensions: • Flip ﬁlter in both dimensions: 0 0 0 0 0 0 1/9 1/9 1/9 a b c g h i i h g – BottomBottom to top to top 0 1 0 0 0 1 1/9 1/9 1/9 d e f d e f f e d – Right to left 0 0 0 0 0 0 1/9 1/9 1/9 Right to left g h i a b c c b a • Then apply cross-correlation ( (X) ) • CommutativeThen apply : F *cross-correlation G = G * F

© 2015 Irfan Essa, Georgia Tech, All Rights Reserved © 2015 Irfan Essa, Georgia• Tech, AllAssociative: Rights Reserved (F*G) * H = F * (G*H) • Nothing • Shift by 1 • Identity (Unit Impulse E): F * E = F • Average Box Blur

31 Maria Hybinette 32 Maria Hybinette

Example: Combing Linear Filters Sources & Inspiration Linear Filters

0 0 0 1/9 1/9 1/9 0 1 0 x 2 - 1/9 1/9 1/9 = Contributors of Course Material: 0 0 0 1/9 1/9 1/9 • Irfan Essa (main) & Frank Dellaert (Georgia Tech) -0.11 -0.11 -0.11 original, – Also early adopters 64x64 -0.11 1.9 -0.11 • Marc Levoy (Stanford)– taught computational

-0.11 -0.11 -0.11 photography since 2002: – A leader in the ﬁeld : Frankecamera • Frédo Durand (MIT) © 2015 Irfan Essa, Georgia Tech, All Rights Reserved • Magnify importance of middle pixel subtract a • Jack Tumblin (Northwestern) smooth version of the image • Wikipedia • http://www.all-art.org/ – Sharper image. history658_photography1.html • “Photography”, London, Stone, Upton

33 Maria Hybinette 34 Maria Hybinette