CS 89.15/189.5, Fall 2015 COMPUTATIONAL ASPECTS OF DIGITAL PHOTOGRAPHY Filtering & convolution Wojciech Jarosz [email protected] Your Spanish castle illusions CS 89/189: Computational Photography, Fall 2015 2 Yuan Tian Yuan Tian Tianyuan Zhang Tianyuan Zhang Weichen Weichen Maxim Maxim Timelapse photography in the news CS 89/189: Computational Photography, Fall 2015 11 https://youtu.be/MPql1VHbYl4 Today’s agenda Linear filtering & convolution - blurring - sharpening Complexity analysis - Optimizations Denoising from a single image - Bilateral filtering CS 89/189: Computational Photography, Fall 2015 13 Blur, sharpen Image processing motivation Sharpen images Downsample images Fake depth of field Smooth out noise, skin blemishes ... We must understand convolution! After a slide by Frédo Durand a slide by Frédo After CS 89/189: Computational Photography, Fall 2015 15 Sharpening After a slide by Frédo Durand a slide by Frédo After CS 89/189: Computational Photography, Fall 2015 16 Downsampling Yikes! Herringbone patterns Downsample by a scale of 0.2 After a slide by Frédo Durand a slide by Frédo After 17 Downsampling We “randomly” pick a color in the high frequency pattern Downsample by a scale of 0.2 After a slide by Frédo Durand a slide by Frédo After 18 Downsampling Solution: blur the pattern to get average color over new pixels Down Blur sample After a slide by Frédo Durand a slide by Frédo After 19 Fake tilt shift http://www.tiltshiftphotography.net/photoshop-tutorial.php After a slide by Frédo Durand a slide by Frédo After CS 89/189: Computational Photography, Fall 2015 20 Blur in optics Diffraction Lens aberrations Object movement Camera shake Can we remove blur computationally? - invert the blur equation - deconvolution After a slide by Frédo Durand a slide by Frédo After CS 89/189: Computational Photography, Fall 2015 21 Lens diffraction http://luminous-landscape.com/ tutorials/understanding-series/u- diffraction.shtml (heavily cropped) See also http://www.cambridgeincolour.com/ tutorials/diffraction-photography.htm After a slide by Frédo Durand a slide by Frédo After Blur example: spherical aberration Pixel value: weighted average of local color object with color variation Sensor lens After a slide by Frédo Durand a slide by Frédo After CS 89/189: Computational Photography, Fall 2015 23 Remove optical artifacts Calibrate lenses and remove blur e.g. DXO After a slide by Frédo Durand a slide by Frédo After CS 89/189: Computational Photography, Fall 2015 24 Removing camera shake Original Naïve Sharpening Fergus et al’s algorithm After a slide by Frédo Durand a slide by Frédo After CS 89/189: Computational Photography, Fall 2015 25 Convolution 101 Blur as convolution Replace each pixel by a linear combination of its neighbors. - only depends on relative position of neighbors The prescription for the linear combination is called the “convolution kernel”. local image data kernel modified image 10 5 3 0 0 0 4 5 1 0 0.5 0 7 1 1 7 0 1 0.5 After a slide by Frédo Durand a slide by Frédo After CS 89/189: Computational Photography, Fall 2015 27 Linear shift-invariant filtering Replace each pixel by a linear combination of its neighbors. - only depends on relative position of neighbors The prescription for the linear combination is called the “convolution kernel”. local image data kernel modified image - same kernel for all pixels 10 5 3 0 0 0 4 5 1 0 0.5 0 7 1 1 7 0 1 0.5 After a slide by Frédo Durand a slide by Frédo After CS 89/189: Computational Photography, Fall 2015 28 Example of linear NON-shift invariant transformation? e.g. neutral-density graduated filter (darken high y): - J(x,y) = I(x,y)*(1-y/ymax) Formally, what does linear mean? - For two scalars a & b and two inputs x & y: F(ax+by) = aF(x)+bF(y) What does shift invariant mean? - For a translation T: F(T(x)) = T(F(x)) - If I blur a translated image, I get a translated blurred image After a slide by Frédo Durand a slide by Frédo After CS 89/189: Computational Photography, Fall 2015 29 Questions? CS 89/189: Computational Photography, Fall 2015 30 Convolution algorithm set output image to zero for all pixels (x,y) in output image for all (x’,y’) in kernel out(x,y) += input(x+x’,y+y’)*kernel(x’,y’) (this assumes the kernel coordinates are centered) local image data kernel modified image 10 5 3 0 0 0 4 5 1 0 0.5 0 7 1 1 7 0 1 0.5 After a slide by Frédo Durand a slide by Frédo After CS 89/189: Computational Photography, Fall 2015 31 Questions? s for all pixels (x,y) in for all (x’,y’) out(x,y) += input(x+x’,y+y’)*kernel(x’,y’) (this assumes the kernel coordinates are centered) local image data kernel modified image 10 5 3 0 0 0 4 5 1 0 0.5 0 7 1 1 7 0 1 0.5 After a slide by Frédo Durand a slide by Frédo After CS 89/189: Computational Photography, Fall 2015 32 Convolution (warm-up slide) 1 ⨂ coefficient = 0 pixel offset ? original After a slide by Frédo Durand a slide by Frédo After CS 89/189: Computational Photography, Fall 2015 33 Convolution (warm-up slide) 1 ⨂ coefficient = 0 pixel offset original filtered (no change) After a slide by Frédo Durand a slide by Frédo After CS 89/189: Computational Photography, Fall 2015 34 Convolution (warm-up slide) 1 ⨂ coefficient = 0 pixel offset original filtered (no change) f d f = f d After a slide by Frédo Durand a slide by Frédo After ⌦ CS 89/189: Computational Photography, Fall 2015 35 Convolution 1 ⨂ coefficient = 0 pixel offset ? original After a slide by Frédo Durand a slide by Frédo After CS 89/189: Computational Photography, Fall 2015 36 Convolution 1 ⨂ coefficient = 0 pixel offset ? 0 0 0 0 0 0 0 0 0 0 0 0 0 0 original 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 After a slide by Frédo Durand a slide by Frédo After 0 0 0 0 0 0 0 CS 89/189: Computational Photography, Fall 2015 37 Convolution 1/3 ⨂ coefficient = 0 pixel offset ? original After a slide by Frédo Durand a slide by Frédo After CS 89/189: Computational Photography, Fall 2015 38 Blurring 1/3 ⨂ coefficient = 0 pixel offset original blurred (applied in both dimensions) After a slide by Frédo Durand a slide by Frédo After CS 89/189: Computational Photography, Fall 2015 39 Blur examples 8 impulse 1/3 8/3 ⨂ coefficient = original 0 filtered pixel offset After a slide by Frédo Durand a slide by Frédo After CS 89/189: Computational Photography, Fall 2015 40 Blur examples 8 impulse 1/3 8/3 ⨂ coefficient = original 0 filtered pixel offset 8 8 edge 4 1/3 4 ⨂ coefficient = original 0 filtered After a slide by Frédo Durand a slide by Frédo After pixel offset CS 89/189: Computational Photography, Fall 2015 41 Questions? CS 89/189: Computational Photography, Fall 2015 42 Formally More formally: Convolution (I g)(x)= I(x0) g(x x0) dx0 ⌦ Zx0 − g I After a slide by Frédo Durand a slide by Frédo After CS 89/189: Computational Photography, Fall 2015 44 Questions? (I g)(x)= I(x0) g(x x0) dx0 ⌦ Zx0 − g I After a slide by Frédo Durand a slide by Frédo After CS 89/189: Computational Photography, Fall 2015 45 What’s up with the flipping? Convolution & probability Convolution was first used by Laplace to study the probability of the sum of two random variables After a slide by Frédo Durand a slide by Frédo After CS 89/189: Computational Photography, Fall 2015 47 Random variables How can X+Y=0? Probability? - X=-1, Y=1 - P(X=-1)*P(Y=1) - X=0, Y=0 - P(X=0)*P(Y=0) - X=1, Y=-1 X 50% Y 50% - P(X=1)*P(Y=-1) 40% 30% 20% 10% After a slide by Frédo Durand a slide by Frédo After -1 0 1 -1 0 1 CS 89/189: Computational Photography, Fall 2015 48 Sum of random variables P(X + Y = k)=Â P(X = k0)P(Y = k k0) k − How can X+Y=0? 0 Probability? - X=-1, Y=1 - P(X=-1)*P(Y=1) X 50% Y 50% - X=0, Y=0 40% - P(X=0)*P(Y=0) - X=1, Y=-1 30% - P(X=1)*P(Y=-1) 20% 10% After a slide by Frédo Durand a slide by Frédo After -1 0 1 -1 0 1 CS 89/189: Computational Photography, Fall 2015 49 Questions? P(X + Y = k)=Â P(X = k0)P(Y = k k0) k − How can X+Y=0? 0 Probability? - X=-1, Y=1 - P(X=-1)*P(Y=1) X 50% Y 50% - X=0, Y=0 40% - P(X=0)*P(Y=0) - X=1, Y=-1 30% - P(X=1)*P(Y=-1) 20% 10% After a slide by Frédo Durand a slide by Frédo After -1 0 1 -1 0 1 CS 89/189: Computational Photography, Fall 2015 50 Compare P(X + Y = k)= P(X = k0)P(Y = k k0) Â − k0 (I g)(x)= I(x0) g(x x0) dx0 ⌦ Zx0 − Forward model: light goes from x to x+x’ Backward model: light at x comes from x-x’ After a slide by Frédo Durand a slide by Frédo After CS 89/189: Computational Photography, Fall 2015 51 Image processing I will often use the term “convolution” improperly and fail to flip the kernel - Called correlation - Won’t matter most of the time because our kernels are symmetric After a slide by Frédo Durand a slide by Frédo After CS 89/189: Computational Photography, Fall 2015 52 Questions? CS 89/189: Computational Photography, Fall 2015 53 Movie break https://youtu.be/HyZfIlxwsfI Blur zoo http://graphics.stanford.edu/courses/cs178/applets/ convolution.html After a slide by Frédo Durand a slide by Frédo After CS 89/189: Computational Photography, Fall 2015 57 Box filter ⌦ After a slide by Frédo Durand a slide by Frédo After CS 89/189: Computational Photography, Fall 2015 58 Nice and smooth: Gaussian ⌦ After a slide by Frédo Durand a slide by Frédo After CS 89/189: Computational Photography, Fall 2015 59 Gaussian formula http://en.wikipedia.org/wiki/Gaussian_function r2 ae− 2s2 r is the distance to the center a is a normalization constant - I usually just normalize my kernels after the fact σ is the standard deviation and controls the width of the Gaussian After a slide by Frédo Durand a slide by Frédo After CS 89/189: Computational Photography, Fall 2015 60 Gaussian formula http://en.wikipedia.org/wiki/Gaussian_function r2 ae− 2s2 Gaussians have infinite support - >0 everywhere but are often truncated - consider Gaussian to be zero beyond e.g.