Guided Image Filtering Kaiming He The Chinese University of Hong Kong Jian Sun Microsoft Research Asia Xiaoou Tang The Chinese University of Hong Kong Introduction • Edge-preserving filtering – An important topic in computer vision • Denoising, image smoothing/sharpening, texture decomposition, HDR compression, image abstraction, optical flow estimation, image super- resolution, feature smoothing… – Existing methods • Weighted Least Square [Lagendijk et al. 1988] • Anisotropic diffusion [Perona and Malik 1990] • Bilateral filter [Aurich and Weule 95], [Tomasi and Manduchi 98] • Digital TV (Total Variation) filter [Chan et al. 2001] Introduction • Bilateral filter qi Wij ( p)p j jN (i) spatial Gs(xi-xj) input p bilateral output q W=GsGr range Gr(pi-pj) Introduction • Joint bilateral filter [Petschnigg et al. 2004] qi Wij (I)p j jN (i) bilateral filter: I=p spatial Gs(xi-xj) input p bilateral output q W=GsGr range Gr(Ii-Ij) E.g. p: noisy / chrominance channel I: flash / luminance channel guide I Introduction • Advantages of bilateral filtering – Preserve edges in the smoothing process – Simple and intuitive – Non-iterative Introduction • Problems in bilateral filtering – Complexity • Brute-force: O(r2) • Distributive histogram: O(logr) [Weiss 06] Approximate • Bilateral grid: band-dependent [Paris and Durand 06], [Chen et al. 07] (quantized) • Integral histogram: O(1) [Porikli 08], [Yang et al. 09] Introduction • Problems in bilateral filtering – Complexity Example: detail enhancement gradient – Gradient distortion reversal • Preserves edges, but not gradients gradient reversal input enhanced Introduction • Our target - to design a new filter – Edge-preserving filtering Advantages of bilateral filter – Non-iterative – O(1) time, fast and non-approximate Overcome bilateral filter’s – No gradient distortion problems Guided filter q p n 2 2 i i i min (aIi b pi ) a (a,b) i ni - noise / texture Linear regression input p output q q aI cov(I, p) i i a var(I) q aI b i i b p aI guide I Bilateral/joint bilateral filter does not have this linear model Guided filter Definition covk (I, p) • Extend to the entire image ak vark (I) – In all local windows ωk ,compute the linear coefficients bk pk aIk – Compute the average of akIi+bk in all ωk that covers pixel qi 1 qi (ak Ii bk ) k|ik ai Ii bi qi ω2 ω1 ω3 Guided filter Definition covk (I, p) • Parameters ak vark (I) – Window radius r bk pk aIk – Regularization ε 1 qi (ak Ii bk ) k|ik ai Ii bi qi ω 2 2r ω1 ω3 Guided filter: smoothing a cascade of mean filters cov(I, p) a var(I) a 0 var(I) q aI b p cov(I, p) b p i i b p aI input p output q var(I) r : determines band-width guide I (like σs in BF) Guided filter: edge-preserving qi aIi b qi aIi Iia b ε : degree of cov(I, p) edge-preserving a var(I) (like σr in BF) Ii qi guide I output q input & Example – edge-preserving smoothing guide guided filter (let I=p) r=4, ε=0.12 r=4, ε=0.22 r=4, ε=0.42 bilateral filter σs=4, σr=0.1 σs=4, σr=0.2 σs=4, σr=0.4 • Our target - to design a new filter – Edge-preserving filtering Advantages of bilateral filter – Non-iterative – O(1) time, fast and non-approximate Overcome bilateral filter’s – No gradient distortion problems Definition Complexity covk (I, p) • mean, var, cov in all local windows ak vark (I) • Integral images [Franklin 1984] bk pk aIk – O(1) time – independent of r – Non-approximate qi ai Ii bi O(1) bilateral O(1) bilateral O(1) guided (32-bin, 40ms/M) (64-bin, 80ms/M) (exact, 80ms/M) [Porikli 08] Gradient Preserving bilateral filter guided filter input filtered q aI large fluctuation detail (input - filtered) enhanced (detail * 5 + input) gradient reversal gradient reversal Example – detail enhancement bilateral filter guided filter input (I=p) bilateral filter guided filter 2 σs=16, σr=0.1 r=16, ε=0.1 gradient reversal Example – detail enhancement bilateral filter guided filter input (I=p) bilateral filter guided filter 2 σs=16, σr=0.1 r=16, ε=0.1 Example – HDR compression gradient keep reversal anti-aliased bilateral filter guided filter input HDR bilateral filter guided filter 2 σs=15, σr=0.12 r=15, ε=0.12 Example – flash/no-flash denoising gradient reversal input p joint bilateral filter (no-flash) σs=8, σr=0.02 joint bilateral guided filter guide I guided filter (flash) r=8, ε=0.022 Beyond smoothing • Applications: feathering/matting, haze removal very small ε guide I preserve most gradients output q q aI input p Example – feathering guide I (size 3000x2000) Example – feathering filter input p (binary segmentation) Example – feathering filter output q (alpha matte) Example – feathering filter output q matting Laplacian guide I filter input p 0.3s [Levin et al. 06] image size 6M 2 min Example – haze removal filter input p guide I (dark channel prior filter output q [He et al. 09]) Example – haze removal guided filter global optimization guide I (<0.1s, 600x400p) (10s) Limitation • “What is an edge” – inherently ambiguous, context-dependent weaker halo halo edge stronger texture Input Bilateral filter Guided filter 2 σs=16, σr=0.4 r=16, ε=0.4 Conclusion • We go from “BF” to “GF” – Edge-preserving filtering – Non-iterative – O(1) time, fast, accurate – Gradient preserving – More generic than “smoothing” Thank you! .