Bilateral filtering Bilateral Filters

Input Gaussian Log(Intensity) Digital Visual Effects Bilateral Smoothing Yung-Yu Chuang

with slides by Fredo Durand, Ramesh Raskar, Sylvain Paris, Soonmin Bae [Ben Weiss, Siggraph 2006]

Imaggge Denoising A Wide Ranggpe of Options

• Diffusion, Bayesian, … – All have their pros and cons.

• Bila tera l filter – not always the best result [Buades 05] but often good – easy to understand, adapt and set up noisy image naïve denoising better denoising edge-preserving filter

Smoothing an image without blurring its edges. Basic denoising Basic denoising

Noisy input Median 5x5 Noisy input Bilateral filter 7x7 window

Tone Mapping Tone Mapping [Duran d 02] [Duran d 02]

HDR input output Photographic Style Transfer Photographic Style Transfer [Bae 06] [Bae 06]

input output

Cartoon Rendition Cartoon Rendition [Winnemöller 06] [Winnemöller 06]

6 pppapers at SIGGRAPH’ 07

input output Gaussian Blur input

per-pixel multiplication input * output

average

box average Gaussian blur 1  x2  G (x)  exp  Equation of Gaussian Blur Gaussian Profile   2   2  2  pixel Same idea: weighted average of pixels. weight

GB[I]p G ||pq|| Iq qS

normalized Gaussian function 1

pixel position unrelated uncertain related uncertain unrelated 0 pixels pixels pixels pixels pixels

Spatial Parameter How to set  • Depends on the application.

input GB[I]p  G || p  q ||Iq qS • Common strategy: proportional to image size size of the window – e.g. 2% of the image diagonal – property: independent of image resolution

small  large 

limited smoothing strong smoothing Properties of Gaussian Blur Properties of Gaussian Blur • Weights independent of spatial location • Does smooth images input • But smoothes too much: – linear convolution edges are blurred. – Only spatial distance matters – well-known operation –No edge term

– efficient computation (recursive algorithm, FFT…) output

GB[I]p  G || p  q || Iq qS space

Blur Comes from Averaging across Edges Bilateral Filter No[Aurich Averaging 95 , Smith 97 , Tomasi across 98] Edges

input * output input * output * * * * SGiklSame Gaussian kernel everyw here. The k ernel s hape depen ds on the image con ten t. Illustration a 1D Image Bilateral Filter Definition •1D imaggpe = line of pixels Same idea: weighted average of pixels. new  not new new 1 BF [I ]  G || p  q || G | I  I | I • Better visualized as a plot p  s  r p q q Wp qS

normalizati on space weihtight range weihtight pixel factor I intensity

pixel position

Gaussian Blur and Bilateral Filter Bilateral Filter on a Height Field

Gaussian blur 1 BF [I ]  G || p  q || G | I  I | I p   s  r p q q p Wp qS

q GB[I]p  G || p  q ||Iq qS space

space

Bilateral filter [Aurich 95 , Smith 97 , Tomasi 98]

p  r a

nge 1 BF [I ]  G || p  q ||G | I  I |I p  s  r p q q q W qS p space range output input normalization space reproduced from [Durand 02] Space and Range Parameters Influence of Pixels

 Only pixels close in space and in range are considered. 1 BF [I ]  G || p  q || G | I  I |I space p  s  r p q q Wp qS

• space s : spatial extent of the kernel, size of range the considered neighborhood. p •range r : “minimum” amplitude of an edge

Exploring the Parameter Space Varying the Range Parameter

r =  r =  r =01= 0.1 r =025= 0.25 (Gauss ian bl ur) r =01= 0.1 r =025= 0.25 (Gauss ian bl ur)

input input

s = 2 s = 2

s = 6 s = 6

s = 18 s = 18 input r = 0.1

r = 0.25 r =  (Gaussi an bl ur) = 6 s

input  ) ur bl  an i = r = 2

 s auss  (G i bl ) 25 . 0

=025 = r  1 . 0

=01 = r  Varying the Space Parameter = 2 = 6 s s = 18

  s  input input – e.g., 2% of image diagonal –of image gradients or median e.g., mean How to Set the Parameters Depends on the application. For instance: • space parameter: proportional to image size • parameter: range proportional to edge amplitude • of resolution and exposure independent = 18 s  ] ) n ( I [ lications such as NPR. BF pp  ) 1  n ( I could be useful for a Iterating the Bilateral Filter •piecewise-flat images Generate more • not needed Often in computational photo, but 1 iteration 2 iterations

4 iterations Advantages of Bilateral Filter • Easy to understand – We ig hte d mean of nearby pilixels

• Easy to adapt – Distance between pixel values

• Easy to set up – Non-iterative Hard to Compute But Bilateral Filter is Nonlinear  1 • Nonlinear BF [I ]  G || p  q || G | I  I | I p  s  r p q q Wp qS • Slow but some accelerations exist: • Complex, spatially varying kernels – [Elad 02]: Gauss-Seidel iterations

– Cannot be precomputed, no FFT… • Only for many iterations

– [Durand 02, Weiss 06]: fast approximation

• No formal understanding of accuracy versus speed

• [Weiss 06]: Only box function as spatial kernel

• Brute-force implementation is slow > 10min

Definition of Bilateral Filter A Fast Approximation • [Smith 97, Tomasi 98] ItInput RltResult of the Bila tera l Filter • Smoothes an image and preserves edges using a Signal Processing • Weighted average Approach of neighbors •Weights Sylvain Paris and Frédo Durand – Gaussian on space distance Computer Science and Artificial Intelligence Laboratory – Gaussian on range distance Massachusetts Institute of Technology –sum to 1

space range Contributions Intuition on 1D Signal

• Link with linear filtering

• Fast and accurate approximation

BF

Basic idea Basic idea

1D Gaussians 1D Gaussians   1 1 BF [I ]  Gq;p, GI ;I , I BF [I ]  Gq;p, GI ;I , I p W  s q p r q p W  s q p r q p qS p qS 2D Gaussians 1 BF [I ]  Gq, I ;p, I , , I p W  q p s r q,I 'q  p q,I 'q S ' a special 2D image Intuition on 1D Signal Intuition on 1D Signal Weighted Average of Neighbors

weights p applied to pixels

BF • Near and similar pixels have influence. • Far pixels have no influence. • Pixels with different value have no influence.

Link with Linear Filtering 1. Handling the Division Formalization: Handling the Division

p

• Normalizing factor as homogeneous coordinate sum of weights • Multiply both sides by

Handling the division with a projective space. Link with Linear Filtering Formalization: Handling the Division space: 1D Gaussian 2. Introducing a Convolution  range: 1D Gaussian

combination: 2D Gaussian p with Wq=1 q

• Similar to homogeneous coordinates in projective space

• Division delayed until the end

• Next step: Adding a dimension to make a convolution appear space range

Link with Linear Filtering space: 1D Gaussian Link with Linear Filtering 2. Introducing a Convolution  range: 1D Gaussian 2. Introducing a Convolution

combination: 2D Gaussian p

q

sum all values black = zero

space-range GiGaussian space x range

Corresponds to a 3D Gaussian on a 2D image. sum all values multiplied by kernel  convolution Link with Linear Filtering Link with Linear Filtering 2. Introducing a Convolution 2. Introducing a Convolution

result of the convolution result of the convolution

space-range GiGaussian space-range GiGaussian

Reformulation: Summary higher dimensional functions

wiw i w

Gaussian convolution 1. Convolution in higher dimension • expensive but well understood (linear, FFT, etc) 2. Division and slicing division • nonlinear but simple and pixel-wise

slicing Exact reformulation higher dimensional functions higher dimensional functions wiw i w wiw i w

D O W N S A M P L E Low-pass filter Gaussian convolution Gaussian convolution

U P S A M P L E division division

slicing slicing

Fast Convolution by Downsampling Accuracy versus Running Time • Finer sampling increases accuracy. • Downsampling cuts frequencies •More ppprecise than previous work. above Nyquist limit

– Less data to process PSNR as function of Running Time – But induces error Digital photograph 1200  1600 • Evaluation of the approximation Straightforward – Precision versus running time implementation is – Visual accuracy over 10 minutes. Visual Results Conclusions • Comparison with previous work [Durand 02] – running time = 1s for both techniques hig her dimens ion  “be tter ” computa tion 1200  1600 itinput exact BF our result prev. work Practical gain Theoretical gain

• Interactive running time • Link with linear filters

• Visually similar results • Separation linear/nonlinear

• Simple to code (100 lines) • Signal processing framework difference 0.1 with exact computation (intensities in [0:1]) 0

Ansel Adams

Two-scale Tone Management for Photoggpraphic Look Soonmin Bae, Sylvain Paris, and Frédo Durand MIT CSAIL

Ansel Adams, Clearing Winter Storm An Amateur Photographer A Variety of Looks

Goals Aspects of Photographic Look

• Control over photographic look • Subject choice • Transfer “look” from a model photo •Framing and composition  Specified by input photos For example , Input • Tone distribution and contrast we want Modified based on model photos with the look of

Model Tonal Aspects of Look Tonal aspects of Look - Global Contrast

Ansel Adams Kenro Izu Ansel Adams Kenro Izu

High Global Contrast Low Global Contrast

Tonal aspects of Look - Local Contrast Overview

Model

Input Image Result

Ansel Adams Kenro Izu • Transfer look between photographs Variable amount of texture Texture everywhere – Tonal aspects Overview Global contrast Overview Global contrast

Split Careful combination

Input Input PtPost- Result process Image Image Local contrast

• Separate glbllobal an dld loca l contrast Local contrast

Result

Overview Global contrast Split Global vs. Local Contrast

• Naïve decomppgqyosition: low vs. high frequency – Problem: introduce blur & halos

Split Careful combination Halo Input PtPost- process Image Blur

Local contrast Low frequency High frequency Result Global contrast Local contrast Bilateral Filter Bilateral Filter

•Edge-ppgreserving smoothin g[]g [Tomasi 98] •Edge-ppgreserving smoothin g[]g [Tomasi 98] • We build upon tone mapping [Durand 02] • We build upon tone mapping [Durand 02]

BASE layer DETAIL layer

After bilateral filtering Residual after filtering After bilateral filtering Residual after filtering Global contrast Local contrast Global contrast Local contrast

Global contrast Global contrast

Bila tera l Careful Bilat era l Careful Filter combination Filter combination

Input PtPost- Input PtPost- process process Image Image

Local contrast Local contrast

Result Result Global Contrast Global Contrast (()Model Transfer)

• Intensity remapping of base layer • Histogram matching – Remapping function given input and model histogram Remapped Model intensity base

Input Output base base

Input baseInput intensity After remapping

Global contrast Global contrast

Intensity Intensity matching matching

Bila tera l Careful Bilat era l Careful Filter combination Filter combination

Input PtPost- Input PtPost- process process Image Image

Local contrast Local contrast

Result Result The amount of local contrast Local Contrast: Detail Layer is not uniform • Uniform control: – Multiply all values in the detail layer

Smooth region

Textured region

ItInput B3Base + 3  DtilDetail

Local Contrast Variation “Textureness”: 1D Example

• We define “textureness”: amount of local Textured region contrast  Large high-frequency Previous work: – at each pixel based on surrounding region Smooth region Low pass of |H|  Small high-frequency [Li 05, Su 05] Smooth region Textured region  Low textureness  Hig h tex tureness

Smooth region Textured region  LttLow textureness  High textureness Input signal High frequency H Amplitude |H| Edge-preserving filter Textureness Transfer Textureness

Step 1: Model Histogram transfer textureness

Input Desired tttextureness Hist. transfer tttextureness

Step 2: x 0.5 Scaling detail layer x 2.7 (per pixel) to match Input Textureness desired textureness x 4.3 Input detail Output detail

Global contrast Global contrast

Intensity Intensity matching matching

Bila tera l Careful Bilat era l Careful Filter combination Filter combination

Input PtPost- Input PtPost- process process Image Image Textureness Textureness matching matching Local contrast Local contrast

Result Result A Non Perfect Result Intensity Remapping

• Decoupled and large modifications (up to 6x) Limited defects may appear • Some intensi ti es may be outsid e disp laya ble range.

result after  Compress histogram to fit visible range. input (HDR) global and local adjustments

initial remapped corrected result intensities result

Preserving Details Effect of Detail Preservation

1. In the gradient domain: uncorrected result corrected result – Compare graditdient amplitdlitudes of inpu t an d curren t – Prevent extreme reduction & extreme increase 2. Solhlve the Poisson equation.

initial remapped corrected result intensities result Global contrast Global contrast

Intensity Intensity matching matching

Bila tera l Constrained Bilat era l Constrained Filter Poisson Filter Poisson

Input PtPost- Input Post- process Image Image process Textureness Textureness matching matching Local contrast Local contrast

Result Result

Global contrast Additional Effects model Intensity matching • Soft focus (high frequency manipulation) • Film grain (texture synthesis [Heeger 95]) • Color toning (chrominance = f (luminance) )

Bilat era l Constrained Filter Poisson

Input SftfSoft focus Toning Image Grain before after effects Textureness effects matching Local contrast

Result Recap Global contrast Results Intensity matching User provides input and modldel photograp hs.  Our system automatically produces the result. Bila tera l Constrained Filter Poisson Running times: Input SftfSoft focus Toning – 6 seconds for 1 MPixel or less Image Grain – 23 seconds for 4 MPixels Textureness matching . multi-grid Poisson solver and fast bilateral filter [Paris 06] Local contrast

Result

ResultInput Model ResultInput Comparison with Naïve Histogram Matching ResultInput Model

Model Input Snapshot, Alfred Stieglitz

Naïve Histogram Matching Our result Local contrast, sharpness unfaithful

Comparison with Naïve Histogram Matching Color Images

• Lab color space: modify only luminance

Input Output

Model Clearing Winter Storm, Ansel Input Adams

Histogram Matching Our Result Local contrast too low Limitations Conclusions

• TfTransfer “lk”“look” from a mo dldel photo • and JPEG artifacts – amplified defects • Two-scale tone management – Global and local contrast – New edge-preserving textureness • Can lead to unexpected – Constrained Poisson reconstruction results if the image content is – Additional effects too different from the model – Portraits, in particular, can suffer

Video Enhancement Using Joint bilateral filtering Per Pixel Exposures (Bennett, 06) From this video:

ASTA: Adaptive Spatio- Temporal Accumultilation Filter Flash / No-Flash Photo Improvement Overview (Petschnigg04) (Eisemann04)

Mergg,yg(e best features: warm, cozy candle light (no-flash) Basic approach of both flash/noflash papers low-noise, detailed flash image

Remove noise + details from image A,

Keep as image A Lighting No-flash ------

Obtain noise -free details from image B,

Discard Image B Lighting Result

Petschnigg: Petschnigg: •Flash •No Flash, Petschnigg: Our Approach •Result Registration Registration

Our Approach Decomposition

Decomposition Color / Intensity:

= *

original intensity color Our Approach Our Approach

Decomposition Decoupling

Decoupling Large-scale Layer • Lighting : Large-scale variation • Texture : SSllmall-scale variat ion • Bilateral filter – edge preserving filter Smith and Brady 1997; Tomasi and Manducci 1998; Durand et al. 2002

ItInput OtOutput

Lighting Texture Large-scale Layer Cross Bilateral Filter • Bilateral filter • Similar to joint bilateral filter by Petschnigg et al. • When no-flash imaggye is too noisy • Borrow similarity from flash image  edge stopping from flash image

Bilateral Cross Bilateral

Detail Layer Recombination

/ = * =

Intensity Large-scale Detail Large-scale Detail Intensity No-flash Flash Result

Recombination: Large scale * Detail = Intensity Recombination: Large scale * Detail = Intensity Recombination Our Approach

shadows

* ~

Intensity Color Result Result Flash

Recombination: Intensity * Color = Original

Our Approach Results

Shadow Detection/Treatment

No-flash

Flash Result Joint bilateral upsampling Joint bilateral upsampling

Joint bilateral upsampling Joint bilateral upsampling Joint bilateral upsampling Joint bilateral upsampling

Joint bilateral upsampling Joint bilateral upsampling