Guided Image Filtering

Kaiming He The Chinese University of Hong Kong Jian Sun Microsoft Research Asia Xiaoou Tang The Chinese University of Hong Kong Introduction

• Edge-preserving filtering – An important topic in computer vision • Denoising, image /sharpening, texture decomposition, HDR compression, image abstraction, optical flow estimation, image super- resolution, feature smoothing… – Existing methods

• Weighted Least Square [Lagendijk et al. 1988]

• [Perona and Malik 1990]

• Bilateral filter [Aurich and Weule 95], [Tomasi and Manduchi 98]

• Digital TV (Total Variation) filter [Chan et al. 2001] Introduction

• Bilateral filter qi  Wij ( p)p j jN (i)

spatial Gs(xi-xj)

input p bilateral output q

W=GsGr

range Gr(pi-pj) Introduction

• Joint bilateral filter [Petschnigg et al. 2004] qi  Wij (I)p j jN (i)

bilateral filter: I=p

spatial Gs(xi-xj)

input p bilateral output q

W=GsGr

range Gr(Ii-Ij)

E.g. p: noisy / chrominance channel I: flash / luminance channel guide I Introduction

• Advantages of bilateral filtering – Preserve edges in the smoothing process – Simple and intuitive – Non-iterative

Introduction

• Problems in bilateral filtering – Complexity • Brute-force: O(r2)

• Distributive histogram: O(logr) [Weiss 06] Approximate • Bilateral grid: band-dependent [Paris and Durand 06], [Chen et al. 07] (quantized) • Integral histogram: O(1) [Porikli 08], [Yang et al. 09]

Introduction

• Problems in bilateral filtering – Complexity Example: detail enhancement

gradient – Gradient reversal • Preserves edges, but not gradients

gradient reversal

input enhanced Introduction

• Our target - to design a new filter – Edge-preserving filtering Advantages of bilateral filter – Non-iterative – O(1) time, fast and non-approximate Overcome bilateral filter’s – No gradient distortion problems Guided filter

q  p  n 2 2 i i i min (aIi  b  pi )  a (a,b)  i

ni - / texture Linear regression input p

output q q  aI cov(I, p) i i a  var(I)   q  aI  b i i b  p  aI guide I Bilateral/joint bilateral filter does not have this linear model Guided filter Definition

covk (I, p) • Extend to the entire image ak  vark (I)   – In all local windows ωk ,compute the linear coefficients bk  pk  aIk

– Compute the average of akIi+bk in

all ωk that covers pixel qi 1 qi  (ak Ii  bk )  k|ik

 ai Ii  bi qi

ω2 ω1 ω3 Guided filter Definition

covk (I, p) • Parameters ak  vark (I)   – Window radius r bk  pk  aIk – Regularization ε 1 qi  (ak Ii  bk )  k|ik

 ai Ii  bi qi ω 2 2r ω1 ω3 Guided filter: smoothing a cascade of mean filters cov(I, p) a  var(I)   a  0 var(I)   q  aI  b  p cov(I, p)   b  p i i b  p  aI

input p output q

var(I)   r : determines band-width guide I (like σs in BF) Guided filter: edge-preserving

qi  aIi  b qi  aIi  Iia  b

ε : degree of cov(I, p) edge-preserving   a   var(I)   (like σr in BF)

Ii qi

guide I output q input & Example – edge-preserving smoothing guide

guided filter (let I=p)

r=4, ε=0.12 r=4, ε=0.22 r=4, ε=0.42

bilateral filter

σs=4, σr=0.1 σs=4, σr=0.2 σs=4, σr=0.4 • Our target - to design a new filter – Edge-preserving filtering Advantages of bilateral filter – Non-iterative – O(1) time, fast and non-approximate Overcome bilateral filter’s – No gradient distortion problems Definition Complexity

covk (I, p) • mean, var, cov in all local windows ak  vark (I)   • Integral images [Franklin 1984] bk  pk  aIk – O(1) time – independent of r

– Non-approximate qi  ai Ii  bi

O(1) bilateral O(1) bilateral O(1) guided (32-bin, 40ms/M) (64-bin, 80ms/M) (exact, 80ms/M) [Porikli 08] Gradient Preserving

bilateral filter guided filter

input filtered q  aI large fluctuation detail (input - filtered)

enhanced (detail * 5 + input) gradient reversal gradient reversal Example – detail enhancement

bilateral filter guided filter

input (I=p) bilateral filter guided filter 2 σs=16, σr=0.1 r=16, ε=0.1 gradient reversal Example – detail enhancement

bilateral filter guided filter

input (I=p) bilateral filter guided filter 2 σs=16, σr=0.1 r=16, ε=0.1 Example – HDR compression

gradient keep reversal anti-aliased

bilateral filter guided filter input HDR

bilateral filter guided filter 2 σs=15, σr=0.12 r=15, ε=0.12 Example – flash/no-flash denoising

gradient reversal

input p joint bilateral filter

(no-flash) σs=8, σr=0.02

joint bilateral guided filter

guide I guided filter (flash) r=8, ε=0.022 Beyond smoothing

• Applications: feathering/matting, haze removal

very small ε guide I

preserve most gradients output q q  aI

input p Example – feathering

guide I (size 3000x2000) Example – feathering

filter input p (binary segmentation) Example – feathering

filter output q (alpha matte) Example – feathering

filter output q matting Laplacian guide I filter input p 0.3s [Levin et al. 06] image size 6M 2 min Example – haze removal

filter input p guide I (dark channel prior filter output q [He et al. 09]) Example – haze removal

guided filter global optimization guide I (<0.1s, 600x400p) (10s) Limitation

• “What is an edge” – inherently ambiguous, context-dependent

weaker halo halo edge

stronger texture

Input Bilateral filter Guided filter 2 σs=16, σr=0.4 r=16, ε=0.4 Conclusion

• We go from “BF” to “GF” – Edge-preserving filtering – Non-iterative – O(1) time, fast, accurate – Gradient preserving – More generic than “smoothing”

Thank you!