Reading Required: Shirley, Ch. 9 Recommended: Ron Bracewell, The Fourier Transform and Its Applications, McGraw-Hill. Fourier analysis and Don P. Mitchell and Arun N. Netravali, “Reconstruction Filters in Computer Computer sampling theory Graphics ,” Computer Graphics, (Proceedings of SIGGRAPH 88). 22 (4), pp. 221-228, 1988. Brian Curless CSE 557 Fall 2009 1 2 What is an image? Images as functions We can think of an image as a function, f, from R2 to R: f(x,y) gives the intensity of a channel at position (x,y) Realistically, we expect the image only to be defined over a rectangle, with a finite range: • f: [a,b]x[c,d] Æ [0,1] A color image is just three functions pasted together. We can write this as a “vector-valued” function: ⎡⎤rxy(, ) f (,xy )= ⎢⎥ gxy (, ) ⎢⎥ ⎣⎦⎢⎥bxy(, ) We’ll focus in grayscale (scalar-valued) images for now. 3 4 Digital images Motivation: filtering and resizing In computer graphics, we usually create or operate What if we now want to: on digital (discrete)images: smooth an image? Sample the space on a regular grid sharpen an image? Quantize each sample (round to nearest enlarge an image? integer) shrink an image? If our samples are Δ apart, we can write this as: In this lecture, we will explore the mathematical underpinnings of these operations. f [n ,m] = Quantize{ f (n Δ, m Δ) } 5 6 Convolution Convolution in 2D One of the most common methods for filtering a In two dimensions, convolution becomes: function, e.g., for smoothing or sharpening, is called convolution. gxy(,)=∗ f (,) xy hxy (,) ∞∞ In 1D, convolution is defined as: =−−∫∫fxyhxxyydxdy(',')( ', ')' ' −∞ −∞ gx()=∗ f () x hx () ∞∞ ∞ =−−∫∫fxyhxxyydxdy(',')(' ,' ) ' ' =−∫ fxhx(')( xdx ')' −∞ −∞ −∞ ∞ where hxy( , ) =−− h( x , y ). =−∫ fxhx(')(' xdx ) ' −∞ where hx( ) ≡− h( x ). * = f(x,y) h(x,y) g(x,y) 7 8 Fourier transforms Fourier transforms (cont’d) ∞ Convolution, while a bit cumbersome looking, Fs()= ∫ f ( xe )−isx2π dx actually has a beautiful structure when viewed in Spatial −∞ Frequency domain domain terms of Fourier analysis. ∞ fx()= ∫ Fse ()isx2π ds We can represent functions as a weighted sum of −∞ sines and cosines. f(x) is usually a real signal, but F(s) is generally complex: We can think of a function in two complementary ways: Fs()=+ As () iBs () = Fs () eis2()πθ Spatially in the spatial domain Spectrally in the frequency domain where magnitude |F(s)| and phase θ(s) are: The Fourier transform and its inverse convert Fs()=+ A22 () s B () s between these two domains: θ()sBsAs= tan−1[] ()/() ∞ Where do the sines and cosines come in? Fs()= ∫ f ( xe )−isx2π dx Spatial −∞ Frequency domain domain ∞ fx()= ∫ Fse ()isx2π ds −∞ 9 10 Fourier transform example More 1D Fourier examples ⎧ 11/2x < ⎪ ∏=()xx⎨ 1/2 = 1/2 ⎪ ⎩ 01/2x > I derived the cosine version, but actually \int_-inf^inf cos()dx is not well-defined, so problematic. 11 12 Some properties of FT’s 2D Fourier transform ∞ ∞∞ −isx2π Fs()= f ( xe ) dx −+isxsy2(π xy ) ∫ Fs(,)xy s= f (,) xye dxdy −∞ ∫∫ −∞ −∞ Amplitude scaling: Spatial Frequency domain domain ∞∞ fxy(,)= Fs ( , s ) eisxsy2(π xy+ ) dsds ∫∫ xy x y −∞ −∞ Additivity: Spatial domain Frequency domain Domain scaling: fxy(,) Fs(,)x sy 13 14 2D Fourier examples Fourier transforms and convolution What is the Fourier transform of the convolution of two functions? (The answer is very cool!) fh∗←⎯→?? Spatial Frequency domain domain fxy(,) Fs(,)x sy 15 16 Convolution theorems 1D convolution theorem example Convolution theorem: Convolution in the spatial domain is equivalent to multiplication in the frequency domain. fh∗←⎯→⋅ FH Symmetric theorem: Convolution in the frequency domain is equivalent to multiplication in the spatial domain. fh⋅←⎯→∗ F H 17 18 2D convolution theorem example Convolution properties Convolution exhibits a number of basic, but important properties…easily proved in the Fourier domain. Commutativity: |F(s ,s )| f(x,y) x y ax()∗=∗ bx () bx () ax () * Associativity: [()ax∗∗=∗∗ bx ()]() cx ax ()[() bx cx ()] h(x,y) |H(sx,sy)| Linearity: ax( )∗⋅ [ k bx ( )] =⋅ k [ ax ( ) ∗ bx ( )] g(x,y) ax()(()∗+=∗+∗ bx cx ()) ax () bx () ax () cx () |G(sx,sy)| 19 20 The delta function Sifting and shifting The Dirac delta function (or impulse function), δ(x), For sampling, the delta function has two important is a handy tool for sampling theory. properties. It has zero width, infinite height, and unit area. Sifting: fxxa()(δ −= ) faxa ()(δ − ) Can be computed as a limit of various functions, e.g.: 11⎛⎞−xx2 ⎛⎞ δ(x )==∏ lim exp⎜⎟2 lim ⎜⎟ σ →→00σπ2 ⎝⎠2σ w ww⎝⎠ Shifting: fx()∗−=−δ ( x a ) fx ( a ) It is usually drawn as: 21 22 The shah/comb function The shah/comb function, cont’d A string of delta functions is the key to sampling. The If we multiply an input function by the impulse train, resulting function is called the shah or comb we get: function or impulse train: ∞ ∞ fx()III (;) xT=− fx ()∑ δ ( x nT ) III()xxn=−∑ δ ( ) n=−∞ n=−∞ which looks like: We can also define an impulse train in terms of a desired delta function spacing, T: ∞ III(;)xT=−∑ δ ( x nT ) n=−∞ which looks like: 23 24 The shah/comb function, cont’d Sampling Amazingly, the Fourier transform of the shah Now, we can talk about sampling. function is also the shah function: III()xs←⎯→ III () One can also show that: 1 III(;)xT←⎯→= III() s ;1/ T s III() ss ; T oo where so = 1/T. We can visualize this as: ←⎯→ The Fourier spectrum gets replicated by spatial For convenience, I won’t draw the delta functions as sampling! scaled vertically, though mathematically, one must keep track of these scale factors. How do we recover the signal? 25 26 Sampling and reconstruction Sampling and reconstruction in 2D 27 28 Sampling theorem Reconstruction filters This result is known as the Sampling Theorem and is The sinc filter, while “ideal”, has two drawbacks: due to Claude Shannon who first discovered it in 1949: It has large support (slow to compute) It introduces ringing in practice A signal can be reconstructed from its samples without loss of information, if the original signal We can choose from many other filters… has no frequencies above ½ the sampling frequency. For a given bandlimited function, the minimum rate at which it must be sampled is the Nyquist frequency. 29 30 Cubic filters Reconstruction filters in 2D Mitchell and Netravali (1988) experimented with We can also perform reconstruction in 2D… cubic filters, reducing them all to the following form: ⎧(1296)−−BCx32 +−+ (18126) BCx + +− (62) B x < 1 ⎪ 1 ⎪ 32 rx( )=−−++⎨ (( B 6 C ) x (6 B 30 C ) x +−− ( 12 B 48 C ) x ++ (8 B 24 C ) 1 ≤< x 2 6 ⎪0 otherwise ⎩⎪ The choice of B or C trades off between being too blurry or having too much ringing. B=C=1/3 was their “visually best” choice. The resulting reconstruction filter is often called the “Mitchell filter.” 1 0.5 0 2 1.5 1 0.5 0 0.5 1 1.5 2 31 32 Aliasing Anti-aliasing What if we go below the Nyquist frequency? Anti-aliasing is the process of removing the frequencies before they alias. 33 34 Anti-aliasing by analytic prefiltering Filtered downsampling We can fill the “magic” box with analytic pre-filtering Alternatively, we can sample the image at a higher of the signal: rate, and then filter that signal: We can now sample the signal at a lower rate. The whole process is called filtered downsampling or Why may this not generally be possible? supersampling and averaging down. 35 36 Practical upsampling Practical upsampling When resampling a function (e.g., when resizing an This can also be viewed as: image), you do not need to reconstruct the complete 1. putting the reconstruction filter at the desired continuous function. location 2. evaluating at the original sample positions For zooming in on a function, you need only use a reconstruction filter and evaluate as needed for each 3. taking products with the sample values new sample. themselves 4. summing it up Here’s an example using a cubic filter: Important: filter should always be normalized! 37 38 Practical downsampling 2D resampling Downsampling is similar, but filter has larger support We’ve been looking at separable filters: and smaller amplitude. rxyrxry(,)= () () Operationally: 211DDD 1. Choose reconstruction filter in downsampled space. How might you use this fact for efficient resampling 2. Compute the downsampling rate, d, ratio of in 2D? new sampling rate to old sampling rate 3. Stretch the filter by 1/d and scale it down by d 4. Follow upsampling procedure (previous slides) to compute new values 39 40.