What is image filtering? Modify the pixels in an image based on some function of a local neighborhood of Image Filtering the pixels. & Edge Detection Some function Reading: Chapter 7 and 8, F&P Linear Functions Simplest: linear filtering. Replace each pixel by a linear combination of its neighbors. The prescription for the linear combination is called the “convolution kernel”. Convolution Key properties Let I be the image and g be the kernel. The Linearity: output of convolving I with g is denoted I ∗ g filter(I1 + I2 ) = filter(I1) + filter(I2) Shift invariance: f[ m,n] = I ∗g = ∑I[ m − k,n − l]g[ k,l] same behavior regardless of pixel location k ,l filter(shift(I)) = shift(filter(I)) Theoretical result: Any linear shift-invariant operator can be represented as a convolution I 1 Properties in more detail Commutative: a * b = b * a Conceptually no difference between filter and signal Associative: a * (b * c) = (a * b) * c Often apply several filters one after another: (((a * b1) * b2) * b3) This is equivalent to applying one filter: a * (b1 * b2 * b3) Distributes over addition: a * (b + c) = (a * b) + (a * c) Scalars factor out: ka * b = a * kb = k (a * b) Identity: unit impulse e = […, 0, 0, 1, 0, 0, …], a * e = a 2 Yucky details What is the size of the output? MATLAB: filter2(g, I, shape) shape = ‘full’: output size is sum of sizes of I and g shape = ‘same’: output size is same as I shape = ‘valid’:output size is of difference sizes for I & g full same valid g g g g g g I I I g g g g g g Implementation details Implementation details What about near the edge? What about near the edge? the filter window falls off the edge of the image the filter window falls off the edge of the image need to extrapolate need to extrapolate methods: methods (MATLAB): clip filter (black) clip filter (black): imfilter(f, g, 0) wrap around wrap around: imfilter(f, g, ‘circular’) copy edge copy edge: imfilter(f, g, ‘replicate’) reflect across edge reflect across edge: imfilter(f, g, ‘symmetric’) Source: S. Marschner Source: S. Marschner 3 Linear filtering (warm-up slide) Linear filtering (warm-up slide) 1.0 1.0 coefficient ? coefficient 0 0 Pixel offset Pixel offset original original Filtered 0 0 0 (no change) 0 1 0 0 0 0 Linear filtering Shift 1.0 1.0 coefficient ? coefficient 0 0 Pixel offset Pixel offset original original shifted 0 0 0 0 0 1 0 0 0 Linear filtering Blurring 0.3 0.3 coefficient ? coefficient 0 0 Pixel offset Pixel offset original original Blurred (filter 1 1 1 applied in both dimensions). Box filter: 1 1 1 1 1 1 4 Blur examples Blur examples 8 2.4 8 2.4 impulse impulse 0.3 0.3 coefficient coefficient 0 0 original Pixel offset filtered original Pixel offset filtered 8 8 edge 4 0.3 4 coefficient 0 original Pixel offset filtered Smoothing with box filter revisited Smoothing with box filter revisited Smoothing with an average actually doesn’t Smoothing with an average actually doesn’t compare at compare at all well with a defocused lens all well with a defocused lens Most obvious difference is that a single point of Most obvious difference is that a single point of light light viewed in a defocused lens looks like a fuzzy viewed in a defocused lens looks like a fuzzy blob; but blob; but the averaging process would give a little square the averaging process would give a little square Better idea: to eliminate edge effects, weight contribution of neighborhood pixels according to their closeness to the center, like so: Source: D. Forsyth “fuzzy blob” Gaussian Kernel Choosing kernel width Gaussian filters have infinite support, but discrete filters use finite kernels 0.003 0.013 0.022 0.013 0.003 0.013 0.059 0.097 0.059 0.013 0.022 0.097 0.159 0.097 0.022 0.013 0.059 0.097 0.059 0.013 0.003 0.013 0.022 0.013 0.003 5 x 5, σ = 1 Constant factor at front makes volume sum to 1 (can be ignored, as we should re-normalize weights to sum to 1 in any case) Source: C. Rasmussen Source: K. Grauman 5 Gaussian filtering Example: Smoothing with a A Gaussian kernel gives less weight to pixels further Gaussian from the center of the window 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 1 2 1 0 0 0 90 90 90 90 90 0 0 2 4 2 0 0 0 90 0 90 90 90 0 0 1 2 1 0 0 0 90 90 90 90 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 This kernel is an approximation of a Gaussian function: Mean vs. Gaussian filtering Separability of the Gaussian filter Source: D. Lowe Separability example Gaussian filters Remove “high-frequency” components from the image 2D convolution (low-pass filter) (center location only) Convolution with self is another Gaussian So can smooth with small-width kernel, repeat, and get same The filter factors result as larger-width kernel would have into a product of 1D Convolving two times with Gaussian kernel of width σ is same filters: as convolving once with kernel of width sqrt(2) σ Separable kernel Perform convolution Factors into product of two 1D Gaussians along rows: * = Useful: can convolve all rows, then all columns How does this change the computational complexity? Followed by convolution = Linear vs. quadratic in mask size along the remaining column: * Source: K. Grauman Source: K. Grauman 6 Review: Linear filtering Noise What are the defining mathematical properties Salt and pepper of a convolution? noise: contains random occurrences What is the difference between blurring with a of black and white box filter and blurring with a Gaussian? pixels Impulse noise: What happens when we convolve a Gaussian Original Salt and pepper noise contains random with another Gaussian? occurrences of white What is separability? pixels Gaussian noise: How does separability affect computational variations in intensity complexity? drawn from a Gaussian normal distribution Impulse noise Gaussian noise Source: S. Seitz Gaussian noise Reducing Gaussian noise Mathematical model: sum of many independent factors Good for small standard deviations Assumption: independent, zero-mean noise Smoothing with larger standard deviations suppresses noise, but also blurs the image Source: K. Grauman Reducing salt-and-pepper noise Alternative idea: Median filtering 3x3 5x5 7x7 A median filter operates over a window by selecting the median intensity in the window What’s wrong with the results? Is median filtering linear? No. ÍÎNot a convolution Source: K. Grauman 7 Median filter Median filter Salt-and-pepper noise Median filtered What advantage does median filtering have over Gaussian filtering? Robustness to outliers MATLAB: medfilt2(image, [h w]) Source: K. Grauman Source: K. Grauman Median vs. Gaussian filtering Linear filtering (warm-up slide) 3x3 5x5 7x7 2.0 Gaussian 1.0 ? 0 0 original Median Linear filtering (no change) Linear filtering 2.0 2.0 1.0 0.33 ? 0 0 0 0 original Filtered original (no change) 8 (Remember blurring) Sharpening 2.0 0.3 0.33 coefficient 0 0 0 Pixel offset Sharpened original Blurred (filter original original applied in both dimensions). Sharpening Sharpening example 1.7 11.2 8 8 coefficient -0.25 -0.3 original Sharpened (differences are accentuated; constant areas are left untouched). Sharpening Sharpening 0 0 0 1 1 1 0 0 0 1 1 1 0 2 0 - 1 1 1 0 2 0 - 1 1 1 0 0 0 1 1 1 ? 0 0 0 1 1 1 (Note that filter sums to 1) Original Original Sharpening filter - Accentuates differences with local average Source: D. Lowe Source: D. Lowe 9 Sharpening Unsharp mask filter I +α( I − I ∗ g ) = ( 1+α ) I −α I ∗ g = I ∗(( 1+α ) e − g ) image blurred unit impulse image (identity) before after unit impulse Gaussian Laplacian of Gaussian Sharpening Revisited Edge detection What does blurring take away? Goal: Identify sudden changes (discontinuities) in an image – = Intuitively, most semantic and shape information from original smoothed (5x5) detail the image can be encoded in the edges Let’s add it back: More compact than pixels + α = Ideal: artist’s line drawing (but artist is also using original detail sharpened object-level knowledge) Source: D. Lowe Origin of Edges Characterizing edges An edge is a place of rapid change in the image surface normal discontinuity intensity function depth discontinuity intensity function image (along horizontal scanline) first derivative surface color discontinuity illumination discontinuity Edges are caused by a variety of factors edges correspond to extrema of derivative Source: Steve Seitz 10 Image gradient Differentiation and convolution The gradient of an image: Recall, for 2D We could function, f(x,y): approximate this as The gradient points in the direction of most rapid change in intensity ∂f f (x + ε, y) f (x, y) ∂f fx()n+1,y − fx( n , y) = lim − ≈ ∂x ε→0 ε ε ∂x ∆x This is linear and shift (which is obviously a convolution) The gradient direction is given by: invariant, so must be the result of a convolution. -1 1 how does this relate to the direction of the edge? perpendicular The edge strength is given by the gradient magnitude Source: D.