CS 4495 Computer Vision – A. Bobick Templates/Edges CS 4495 Computer Vision Linear Filtering 2: Templates, Edges Aaron Bobick School of Interactive Computing CS 4495 Computer Vision – A. Bobick Templates/Edges Last time: Convolution • Convolution: • Flip the filter in both dimensions (right to left, bottom to top) • Then apply cross-correlation G= HF ∗ H* H* Notation for F convolution operator K. Grauman CS 4495 Computer Vision – A. Bobick Templates/Edges Convolution vs. correlation Convolution G= HF ∗ (Cross-)correlation • When H is symmetric, no difference. We tend to use the terms interchangeably. • Convolution with an impulse (centered at 0,0) is the identity K. Grauman CS 4495 Computer Vision – A. Bobick Templates/Edges Filters for features • Previously, thinking of filtering as a way to remove or reduce noise • Now, consider how filters will allow us to abstract higher- level “features”. • Map raw pixels to an intermediate representation that will be used for subsequent processing • Goal: reduce amount of data, discard redundancy, preserve what’s useful K. Grauman CS 4495 Computer Vision – A. Bobick Templates/Edges Template matching • Filters as templates: Note that filters look like the effects they are intended to find --- “matched filters” • Use (normalized) cross-correlation score to find a given pattern (template) in the image. • Normalization needed to control for relative brightness. More in problem sets. K. Grauman CS 4495 Computer Vision – A. Bobick Templates/Edges Template matching Template (mask) Scene A toy example K. Grauman CS 4495 Computer Vision – A. Bobick Templates/Edges Template matching Template Detected template K. Grauman CS 4495 Computer Vision – A. Bobick Templates/Edges Template matching Detected template Correlation map K. Grauman CS 4495 Computer Vision – A. Bobick Templates/Edges Where’s Waldo? Template Scene K. Grauman CS 4495 Computer Vision – A. Bobick Templates/Edges Where’s Waldo? Template Detected template K. Grauman CS 4495 Computer Vision – A. Bobick Templates/Edges Template demo… • In directory C:\Bobick\matlab\CS4495\Filter • echodemo waldotemplate CS 4495 Computer Vision – A. Bobick Templates/Edges Where’s Waldo? Detected template Correlation map K. Grauman CS 4495 Computer Vision – A. Bobick Templates/Edges Template matching Template Scene What if the template is not identical to some subimage in the scene? K. Grauman CS 4495 Computer Vision – A. Bobick Templates/Edges Template matching Template Detected template Match can be meaningful, if scale, orientation, and general appearance is right. K. Grauman CS 4495 Computer Vision – A. Bobick Templates/Edges Generic features… • When looking for a specific object or pattern, the features can be defined for that pattern – we will do this later in the course for specific object recognition. • But for generic images, what would be good features? What are the parts or properties of the image that encode its “meaning” for human (or other biological) observers? • Some examples of greatly reduced images… CS 4495 Computer Vision – A. Bobick Templates/Edges Edges seem to be important… CS 4495 Computer Vision – A. Bobick Templates/Edges Origin of Edges surface normal discontinuity depth discontinuity surface color discontinuity illumination discontinuity • Edges are caused by a variety of factors • Information theory view: edges encode change, change is what is hard to predict, therefore edges efficiently encode an image CS 4495 Computer Vision – A. Bobick Templates/Edges In a real image Depth discontinuity: Reflectance change: object boundary appearance information, texture Cast shadows Change in surface orientation: shape CS 4495 Computer Vision – A. Bobick Templates/Edges Contrast and invariance CS 4495 Computer Vision – A. Bobick Templates/Edges Edge detection • Convert a 2D image into a set of curves • Extracts salient features of the scene • More compact than pixels CS 4495 Computer Vision – A. Bobick Templates/Edges Edge detection • How can you tell that a pixel is on an edge? CS 4495 Computer Vision – A. Bobick Templates/Edges Images as functions… • Edges look like steep cliffs CS 4495 Computer Vision – A. Bobick Templates/Edges Edge Detection Basic idea: look for a neighborhood with strong signs of change. Problems: 81 82 26 24 82 33 25 25 • neighborhood size 81 82 26 24 • how to detect change CS 4495 Computer Vision – A. Bobick Templates/Edges Derivatives and edges An edge is a place of rapid change in the image intensity function. intensity function image (along horizontal scanline) first derivative edges correspond to extrema of derivative Source: L. Lazebnik CS 4495 Computer Vision – A. Bobick Templates/Edges Differential Operators • Differential operators – here we mean some operation that when applied to the image returns some derivatives. • We will model these “operators” as masks/kernels which when applied to the image yields a new function that is the image gradient function. • We will then threshold the this gradient function to select the edge pixels. • Which brings us to the question: What’s a gradient? CS 4495 Computer Vision – A. Bobick Templates/Edges Image gradient The gradient of an image: The gradient points in the direction of most rapid increase in intensity The gradient direction is given by: • how does this relate to the direction of the edge? The edge strength is given by the gradient magnitude CS 4495 Computer Vision – A. Bobick Templates/Edges Discrete gradient For 2D function, f(x,y), the partial derivative is: ∂f (x, y) f (x + ε, y) − f (x, y) = lim ∂x ε →0 ε For discrete data, we can approximate using finite differences: ∂f (x, y) f (x +1, y) − f (x, y) ≈ ∂x 1 ≈+−fx( 1,) y fxy (,)“right derivative” But is it??? CS 4495 Computer Vision – A. Bobick Templates/Edges Finite differences Computer Vision - A Modern Approach Set: Linear Filters Slides by D.A. Forsyth CS 4495 Computer Vision – A. Bobick Templates/Edges Partial derivatives of an image ∂f (x, y) ∂f (x, y) ∂x ∂y -1 ? 1 or -1 1 1 -1 Which shows changes with respect to x? (showing correlation filters) CS 4495 Computer Vision – A. Bobick Templates/Edges Differentiation and convolution • For 2D function, f(x,y), the partial derivative is: ∂f (x, y) f (x + ε, y) − f (x, y) = lim ∂x ε →0 ε • For discrete data, we can approximate using finite differences: ∂f (x, y) f (x +1, y) − f (x, y) ≈ ∂ x 1 • To implement above as convolution, what would be the associated filter? CS 4495 Computer Vision – A. Bobick Templates/Edges The discrete gradient • We want an “operator” (mask/kernel) that we can apply to the image that implements: ∂f (x, y) f (x + ε, y) − f (x, y) = lim ∂x ε →0 ε How would you implement this as a cross-correlation? (not flipped) Average of 0 0 Not symmetric 0 0 0 “left” and -1 +1 around image -1/2 0 +1/2 “right” 0 0 point; which is 0 0 0 derivative . “middle” pixel? See? CS 4495 Computer Vision – A. Bobick Templates/Edges Example: Sobel operator -1 0 1 1 2 1 -2 0 2 0 0 0 -1 0 1 -1 -2 -1 On a pixel of the image I •Let gx be the response to mask Sx (sometimes * 1/8) •Let gy be the response to mask Sy What is the gradient? T (Sobel) Gradient is ∇I = [gx gy] 2 2 1/2 g = (gx + gy ) is the gradient magnitude. θ = atan2(gy , gx) is the gradient direction. CS 4495 Computer Vision – A. Bobick Templates/Edges Sobel Operator on Blocks Image original image gradient thresholded magnitude gradient magnitude CS 4495 Computer Vision – A. Bobick Templates/Edges Some Well-Known Masks for Computing Gradients Sx Sy -1 0 1 1 2 1 • Sobel: -2 0 2 0 0 0 -1 0 1 -1 -2 -1 -1 0 1 1 1 1 • Prewitt: -1 0 1 0 0 0 -1 0 1 -1 -1 -1 • Roberts 0 1 1 0 -1 0 0 -1 CS 4495 Computer Vision – A. Bobick Templates/Edges Matlab does edges >> My = fspecial(‘sobel’); >> outim = imfilter(double(im), My); >> imagesc(outim); >> colormap gray; CS 4495 Computer Vision – A. Bobick Templates/Edges But… • Consider a single row or column of the image • Plotting intensity as a function of x • Apply derivative operator…. Uh, where’s the edge? CS 4495 Computer Vision – A. Bobick Templates/Edges Finite differences responding to noise Increasing noise -> (this is zero mean additive gaussian noise) D. Forsyth CS 4495 Computer Vision – A. Bobick Templates/Edges Solution: smooth first f h hf∗ ∂ (hf∗ ) ∂x ∂ (hf∗ ) Where is the edge? Look for peaks in ∂x CS 4495 Computer Vision – A. Bobick Templates/Edges Derivative theorem of convolution ∂∂ (hf∗=)( h ) ∗ f • This saves us one operation: ∂∂xx f ∂ h h ∂x ∂ ()hf∗ ∂x How can we find (local) maxima of a function? CS 4495 Computer Vision – A. Bobick Templates/Edges 2nd derivative of Gaussian ∂2 • Consider (hf∗ ) ∂x2 f ∂ h ∂ x 2 ∂ Second derivative of Gaussian h operator ∂x2 ∂2 (hf∗ ) ∂x2 Where is the edge? Zero-crossings of bottom graph CS 4495 Computer Vision – A. Bobick Templates/Edges What about 2D? CS 4495 Computer Vision – A. Bobick Templates/Edges Derivative of Gaussian filter (I ⊗ g)⊗ h = I ⊗ (g ⊗ h) 0.0030 0.0133 0.0219 0.0133 0.0030 0.0133 0.0596 0.0983 0.0596 0.0133 0.0219 0.0983 0.1621 0.0983 0.0219 0.0133 0.0596 0.0983 0.0596 0.0133 ⊗ [ 1 −1 ] [ 0.0030 0.0133 0.0219 0.0133 0.0030 ] Why is this preferable? CS 4495 Computer Vision – A. Bobick Templates/Edges Derivative of Gaussian filters x-direction y-direction Is this for correlation And for y it’s always a problem! or convolution? Source: L. Lazebnik CS 4495 Computer Vision – A. Bobick Templates/Edges Smoothing with a Gaussian Parameter σ is the “scale” / “width” / “spread” of the Gaussian kernel, and controls the amount of smoothing.