Computer Vision: Filtering Raquel Urtasun TTI Chicago Jan 10, 2013 Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 1 / 82 Today's lecture ... Image formation Image Filtering Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 2 / 82 Readings Chapter 2 and 3 of Rich Szeliski's book Available online here Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 3 / 82 How is an image created? The image formation process that produced a particular image depends on lighting conditions scene geometry surface properties camera optics [Source: R. Szeliski] Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 4 / 82 Image formation Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 5 / 82 What is an image? !"#"$%&'(%)*+%' 0*1&&'23456'37'$-*6*'"7'$-"6'4&%66' ,-*'./*' [Source: A. Efros] Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 6 / 82 From photons to RGB values Sample the 2D space on a regular grid. Quantize each sample, i.e., the photons arriving at each active cell are integrated and then digitized. [Source: D. Hoiem] Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 7 / 82 What is an image? A grid (matrix) of intensity values #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #%" %" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" &$" &$" &$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" &$" '$" '$" &$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" '(" )#&" )*$" )&$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" )#&" )*$" )&$" )&$" )&$" #$$" #$$" #$$" #$$" #$$" !" #$$" #$$" )#&" )*$" #%%" #%%" )&$" )&$" '$" #$$" #$$" #$$" #$$" #$$" )#&" )*$" #%%" #%%" )&$" )&$" '$" *&" #$$" #$$" #$$" #$$" )#&" )*$" )*$" )&$" )#&" )#&" '$" *&" #$$" #$$" #$$" #$$" &*" )#&" )#&" )#&" '$" '$" '$" *&" #$$" #$$" #$$" #$$" #$$" &*" &*" &*" &*" &*" &*" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" Common to use one byte per value: 0=black, 255=white) [Source: N. Snavely] Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 8 / 82 What is an image? We can think of a (grayscale) image as a function f : <2 ! < giving the intensity at position (x; y) f (x, y) x y A digital image is a discrete (sampled, quantized) version of this function [Source: N. Snavely] Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 9 / 82 Image Transformations As with any function, we can apply operators to an image !g (x,y) = f (x,y) + 20 !g (x,y) = f (-x,y) We'll talk about special kinds of operators, correlation and convolution (linear filtering) [Source: N. Snavely] Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 10 / 82 Filtering Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 11 / 82 What's the next best thing? [Source: S. Seitz] Question: Noise reduction Given a camera and a still scene, how can you reduce noise? Take lots of images and average them! Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 12 / 82 What's the next best thing? [Source: S. Seitz] Question: Noise reduction Given a camera and a still scene, how can you reduce noise? Take lots of images and average them! Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 12 / 82 Question: Noise reduction Given a camera and a still scene, how can you reduce noise? Take lots of images and average them! What's the next best thing? [Source: S. Seitz] Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 12 / 82 Question: Noise reduction Given a camera and a still scene, how can you reduce noise? Take lots of images and average them! What's the next best thing? [Source: S. Seitz] Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 12 / 82 Image filtering Modify the pixels in an image based on some function of a local neighborhood of each pixel #'" !" &" 5).0"678*9)8" $" !" #" %" #" #" %" ()*+,"-.+/0"1+2+" 3)1-401"-.+/0"1+2+" [Source: L. Zhang] Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 13 / 82 Detect patterns, e.g., template matching. Applications of Filtering Enhance an image, e.g., denoise, resize. Extract information, e.g., texture, edges. Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 14 / 82 Applications of Filtering Enhance an image, e.g., denoise, resize. Extract information, e.g., texture, edges. Detect patterns, e.g., template matching. Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 14 / 82 Applications of Filtering Enhance an image, e.g., denoise, resize. Extract information, e.g., texture, edges. Detect patterns, e.g., template matching. Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 14 / 82 [Source: S. Marschner] Noise reduction Simplest thing: replace each pixel by the average of its neighbors. This assumes that neighboring pixels are similar, and the noise to be independent from pixel to pixel. Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 15 / 82 Noise reduction Simplest thing: replace each pixel by the average of its neighbors. This assumes that neighboring pixels are similar, and the noise to be independent from pixel to pixel. [Source: S. Marschner] Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 15 / 82 Noise reduction Simplest thing: replace each pixel by the average of its neighbors. This assumes that neighboring pixels are similar, and the noise to be independent from pixel to pixel. [Source: S. Marschner] Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 15 / 82 Noise reduction Simplest thing: replace each pixel by the average of its neighbors This assumes that neighboring pixels are similar, and the noise to be independent from pixel to pixel. Moving average in 1D: [1, 1, 1, 1, 1]/5 [Source: S. Marschner] Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 16 / 82 Noise reduction Simpler thing: replace each pixel by the average of its neighbors This assumes that neighboring pixels are similar, and the noise to be independent from pixel to pixel. Non-uniform weights [1, 4, 6, 4, 1] / 16 [Source: S. Marschner] Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 17 / 82 Moving Average in 2D [Source: S. Seitz] Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 18 / 82 Moving Average in 2D [Source: S. Seitz] Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 18 / 82 Moving Average in 2D [Source: S. Seitz] Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 18 / 82 Moving Average in 2D [Source: S. Seitz] Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 18 / 82 Moving Average in 2D [Source: S. Seitz] Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 18 / 82 Moving Average in 2D [Source: S. Seitz] Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 18 / 82 The entries of the weight kernel or mask h(k; l) are often called the filter coefficients. This operator is the correlation operator g = f ⊗ h Linear Filtering: Correlation Involves weighted combinations of pixels in small neighborhoods. The output pixels value is determined as a weighted sum of input pixel values X g(i; j) = f (i + k; j + l)h(k; l) k;l Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 19 / 82 This operator is the correlation operator g = f ⊗ h Linear Filtering: Correlation Involves weighted combinations of pixels in small neighborhoods. The output pixels value is determined as a weighted sum of input pixel values X g(i; j) = f (i + k; j + l)h(k; l) k;l The entries of the weight kernel or mask h(k; l) are often called the filter coefficients. Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 19 / 82 Linear Filtering: Correlation Involves weighted combinations of pixels in small neighborhoods. The output pixels value is determined as a weighted sum of input pixel values X g(i; j) = f (i + k; j + l)h(k; l) k;l The entries of the weight kernel or mask h(k; l) are often called the filter coefficients. This operator is the correlation operator g = f ⊗ h Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 19 / 82 Linear Filtering: Correlation Involves weighted combinations of pixels in small neighborhoods. The output pixels value is determined as a weighted sum of input pixel values X g(i; j) = f (i + k; j + l)h(k; l) k;l The entries of the weight kernel or mask h(k; l) are often called the filter coefficients. This operator is the correlation operator g = f ⊗ h Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 19 / 82 Smoothing by averaging What if the filter size was 5 x 5 instead of 3 x 3? [Source: K. Graumann] Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 20 / 82 Gaussian filter What if we want nearest neighboring pixels to have the most influence on the output? Removes high-frequency components from the image (low-pass filter). [Source: S. Seitz] Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 21 / 82 Smoothing with a Gaussian [Source: K. Grauman] Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 22 / 82 Mean vs Gaussian Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 23 / 82 Gaussian filter: Parameters Size of kernel or mask: Gaussian function has infinite support, but discrete filters use finite kernels. [Source: K. Grauman] Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 24 / 82 Gaussian filter: Parameters Variance of the Gaussian: determines extent of smoothing. [Source: K. Grauman] Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 25 / 82 Gaussian filter: Parameters [Source: K. Grauman] Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 26 / 82 Is this the most general Gaussian? No, the most general form for x 2 <d 1 1 N (x; µ, Σ) = exp − (x − µ)T Σ−1(x − µ) (2π)d=2jΣj1=2 2 But the simplified version is typically use for filtering. Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 27 / 82 Amount of smoothing proportional to mask size. Remove high-frequency components; low-pass filter. [Source: K. Grauman] Properties of the Smoothing All values are positive. They all sum to 1. Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 28 / 82 Remove high-frequency components; low-pass filter. [Source: K. Grauman] Properties of the Smoothing All values are positive. They all sum to 1.