Bilateral filtering Bilateral Filters
Input Gaussian Log(Intensity) Digital Visual Effects Smoothing Bilateral Smoothing Yung-Yu Chuang
with slides by Fredo Durand, Ramesh Raskar, Sylvain Paris, Soonmin Bae [Ben Weiss, Siggraph 2006]
Imaggge Denoising A Wide Ranggpe of Options
• Diffusion, Bayesian, Wavelets… – All have their pros and cons.
• Bila tera l filter – not always the best result [Buades 05] but often good – easy to understand, adapt and set up noisy image naïve denoising better denoising Gaussian blur edge-preserving filter
Smoothing an image without blurring its edges. Basic denoising Basic denoising
Noisy input Median 5x5 Noisy input Bilateral filter 7x7 window
Tone Mapping Tone Mapping [Duran d 02] [Duran d 02]
HDR input output Photographic Style Transfer Photographic Style Transfer [Bae 06] [Bae 06]
input output
Cartoon Rendition Cartoon Rendition [Winnemöller 06] [Winnemöller 06]
6 pppapers at SIGGRAPH’ 07
input output Gaussian Blur input
per-pixel multiplication input * output
average
box average Gaussian blur 1 x2 G (x) exp Equation of Gaussian Blur Gaussian Profile 2 2 2 pixel Same idea: weighted average of pixels. weight
GB[I]p G ||pq|| Iq qS
normalized Gaussian function 1
pixel position unrelated uncertain related uncertain unrelated 0 pixels pixels pixels pixels pixels
Spatial Parameter How to set • Depends on the application.
input GB[I]p G || p q ||Iq qS • Common strategy: proportional to image size size of the window – e.g. 2% of the image diagonal – property: independent of image resolution
small large
limited smoothing strong smoothing Properties of Gaussian Blur Properties of Gaussian Blur • Weights independent of spatial location • Does smooth images input • But smoothes too much: – linear convolution edges are blurred. – Only spatial distance matters – well-known operation –No edge term
– efficient computation (recursive algorithm, FFT…) output
GB[I]p G || p q || Iq qS space
Blur Comes from Averaging across Edges Bilateral Filter No[Aurich Averaging 95 , Smith 97 , Tomasi across 98] Edges
input * output input * output * * * * SGiklSame Gaussian kernel everyw here. The k ernel s hape depen ds on the image con ten t. Illustration a 1D Image Bilateral Filter Definition •1D imaggpe = line of pixels Same idea: weighted average of pixels. new not new new 1 BF [I ] G || p q || G | I I | I • Better visualized as a plot p s r p q q Wp qS
normalizati on space weihtight range weihtight pixel factor I intensity
pixel position
Gaussian Blur and Bilateral Filter Bilateral Filter on a Height Field
Gaussian blur 1 BF [I ] G || p q || G | I I | I p s r p q q p Wp qS
q GB[I]p G || p q ||Iq qS space
space
Bilateral filter [Aurich 95 , Smith 97 , Tomasi 98]
p r a
nge 1 BF [I ] G || p q ||G | I I |I p s r p q q q W qS p space range output input normalization space reproduced from [Durand 02] Space and Range Parameters Influence of Pixels
Only pixels close in space and in range are considered. 1 BF [I ] G || p q || G | I I |I space p s r p q q Wp qS
• space s : spatial extent of the kernel, size of range the considered neighborhood. p •range r : “minimum” amplitude of an edge
Exploring the Parameter Space Varying the Range Parameter
r = r = r =01= 0.1 r =025= 0.25 (Gauss ian bl ur) r =01= 0.1 r =025= 0.25 (Gauss ian bl ur)
input input
s = 2 s = 2
s = 6 s = 6
s = 18 s = 18 input r = 0.1
r = 0.25 r = (Gaussi an bl ur) = 6 s
input ) ur bl an i = r = 2
s auss (G i bl ) 25 . 0
=025 = r 1 . 0
=01 = r Varying the Space Parameter = 2 = 6 s s = 18
s input input – e.g., 2% of image diagonal –of image gradients or median e.g., mean How to Set the Parameters Depends on the application. For instance: • space parameter: proportional to image size • parameter: range proportional to edge amplitude • of resolution and exposure independent = 18 s ] ) n ( I [ lications such as NPR. BF pp ) 1 n ( I could be useful for a Iterating the Bilateral Filter •piecewise-flat images Generate more • not needed Often in computational photo, but 1 iteration 2 iterations
4 iterations Advantages of Bilateral Filter • Easy to understand – We ig hte d mean of nearby pilixels
• Easy to adapt – Distance between pixel values
• Easy to set up – Non-iterative Hard to Compute But Bilateral Filter is Nonlinear 1 • Nonlinear BF [I ] G || p q || G | I I | I p s r p q q Wp qS • Slow but some accelerations exist: • Complex, spatially varying kernels – [Elad 02]: Gauss-Seidel iterations
– Cannot be precomputed, no FFT… • Only for many iterations
– [Durand 02, Weiss 06]: fast approximation
• No formal understanding of accuracy versus speed
• [Weiss 06]: Only box function as spatial kernel
• Brute-force implementation is slow > 10min
Definition of Bilateral Filter A Fast Approximation • [Smith 97, Tomasi 98] ItInput RltResult of the Bila tera l Filter • Smoothes an image and preserves edges using a Signal Processing • Weighted average Approach of neighbors •Weights Sylvain Paris and Frédo Durand – Gaussian on space distance Computer Science and Artificial Intelligence Laboratory – Gaussian on range distance Massachusetts Institute of Technology –sum to 1
space range Contributions Intuition on 1D Signal
• Link with linear filtering
• Fast and accurate approximation
BF
Basic idea Basic idea
1D Gaussians 1D Gaussians 1 1 BF [I ] Gq;p, GI ;I , I BF [I ] Gq;p, GI ;I , I p W s q p r q p W s q p r q p qS p qS 2D Gaussians 1 BF [I ] Gq, I ;p, I , , I p W q p s r q,I 'q p q,I 'q S ' a special 2D image Intuition on 1D Signal Intuition on 1D Signal Weighted Average of Neighbors
weights p applied to pixels
BF • Near and similar pixels have influence. • Far pixels have no influence. • Pixels with different value have no influence.
Link with Linear Filtering 1. Handling the Division Formalization: Handling the Division
p
• Normalizing factor as homogeneous coordinate sum of weights • Multiply both sides by
Handling the division with a projective space. Link with Linear Filtering Formalization: Handling the Division space: 1D Gaussian 2. Introducing a Convolution range: 1D Gaussian
combination: 2D Gaussian p with Wq=1 q
• Similar to homogeneous coordinates in projective space
• Division delayed until the end
• Next step: Adding a dimension to make a convolution appear space range
Link with Linear Filtering space: 1D Gaussian Link with Linear Filtering 2. Introducing a Convolution range: 1D Gaussian 2. Introducing a Convolution
combination: 2D Gaussian p
q
sum all values black = zero
space-range GiGaussian space x range
Corresponds to a 3D Gaussian on a 2D image. sum all values multiplied by kernel convolution Link with Linear Filtering Link with Linear Filtering 2. Introducing a Convolution 2. Introducing a Convolution
result of the convolution result of the convolution
space-range GiGaussian space-range GiGaussian
Reformulation: Summary higher dimensional functions
wiw i w
Gaussian convolution 1. Convolution in higher dimension • expensive but well understood (linear, FFT, etc) 2. Division and slicing division • nonlinear but simple and pixel-wise
slicing Exact reformulation higher dimensional functions higher dimensional functions wiw i w wiw i w
D O W N S A M P L E Low-pass filter Gaussian convolution Gaussian convolution
U P S A M P L E division division
slicing slicing
Fast Convolution by Downsampling Accuracy versus Running Time • Finer sampling increases accuracy. • Downsampling cuts frequencies •More ppprecise than previous work. above Nyquist limit
– Less data to process PSNR as function of Running Time – But induces error Digital photograph 1200 1600 • Evaluation of the approximation Straightforward – Precision versus running time implementation is – Visual accuracy over 10 minutes. Visual Results Conclusions • Comparison with previous work [Durand 02] – running time = 1s for both techniques hig her dimens ion “be tter ” computa tion 1200 1600 itinput exact BF our result prev. work Practical gain Theoretical gain
• Interactive running time • Link with linear filters
• Visually similar results • Separation linear/nonlinear
• Simple to code (100 lines) • Signal processing framework difference 0.1 with exact computation (intensities in [0:1]) 0
Ansel Adams
Two-scale Tone Management for Photoggpraphic Look Soonmin Bae, Sylvain Paris, and Frédo Durand MIT CSAIL
Ansel Adams, Clearing Winter Storm An Amateur Photographer A Variety of Looks
Goals Aspects of Photographic Look
• Control over photographic look • Subject choice • Transfer “look” from a model photo •Framing and composition Specified by input photos For example , Input • Tone distribution and contrast we want Modified based on model photos with the look of
Model Tonal Aspects of Look Tonal aspects of Look - Global Contrast
Ansel Adams Kenro Izu Ansel Adams Kenro Izu
High Global Contrast Low Global Contrast
Tonal aspects of Look - Local Contrast Overview
Model
Input Image Result
Ansel Adams Kenro Izu • Transfer look between photographs Variable amount of texture Texture everywhere – Tonal aspects Overview Global contrast Overview Global contrast
Split Careful combination
Input Input PtPost- Result process Image Image Local contrast
• Separate glbllobal an dld loca l contrast Local contrast
Result
Overview Global contrast Split Global vs. Local Contrast
• Naïve decomppgqyosition: low vs. high frequency – Problem: introduce blur & halos
Split Careful combination Halo Input PtPost- process Image Blur
Local contrast Low frequency High frequency Result Global contrast Local contrast Bilateral Filter Bilateral Filter
•Edge-ppgreserving smoothin g[]g [Tomasi 98] •Edge-ppgreserving smoothin g[]g [Tomasi 98] • We build upon tone mapping [Durand 02] • We build upon tone mapping [Durand 02]
BASE layer DETAIL layer
After bilateral filtering Residual after filtering After bilateral filtering Residual after filtering Global contrast Local contrast Global contrast Local contrast
Global contrast Global contrast
Bila tera l Careful Bilat era l Careful Filter combination Filter combination
Input PtPost- Input PtPost- process process Image Image
Local contrast Local contrast
Result Result Global Contrast Global Contrast (()Model Transfer)
• Intensity remapping of base layer • Histogram matching – Remapping function given input and model histogram Remapped Model intensity base
Input Output base base
Input baseInput intensity After remapping
Global contrast Global contrast
Intensity Intensity matching matching
Bila tera l Careful Bilat era l Careful Filter combination Filter combination
Input PtPost- Input PtPost- process process Image Image
Local contrast Local contrast
Result Result The amount of local contrast Local Contrast: Detail Layer is not uniform • Uniform control: – Multiply all values in the detail layer
Smooth region
Textured region
ItInput B3Base + 3 DtilDetail
Local Contrast Variation “Textureness”: 1D Example
• We define “textureness”: amount of local Textured region contrast Large high-frequency Previous work: – at each pixel based on surrounding region Smooth region Low pass of |H| Small high-frequency [Li 05, Su 05] Smooth region Textured region Low textureness Hig h tex tureness
Smooth region Textured region LttLow textureness High textureness Input signal High frequency H Amplitude |H| Edge-preserving filter Textureness Transfer Textureness
Step 1: Model Histogram transfer textureness
Input Desired tttextureness Hist. transfer tttextureness
Step 2: x 0.5 Scaling detail layer x 2.7 (per pixel) to match Input Textureness desired textureness x 4.3 Input detail Output detail
Global contrast Global contrast
Intensity Intensity matching matching
Bila tera l Careful Bilat era l Careful Filter combination Filter combination
Input PtPost- Input PtPost- process process Image Image Textureness Textureness matching matching Local contrast Local contrast
Result Result A Non Perfect Result Intensity Remapping
• Decoupled and large modifications (up to 6x) Limited defects may appear • Some intensi ti es may be outsid e disp laya ble range.
result after Compress histogram to fit visible range. input (HDR) global and local adjustments
initial remapped corrected result intensities result
Preserving Details Effect of Detail Preservation
1. In the gradient domain: uncorrected result corrected result – Compare graditdient amplitdlitudes of inpu t an d curren t – Prevent extreme reduction & extreme increase 2. Solhlve the Poisson equation.
initial remapped corrected result intensities result Global contrast Global contrast
Intensity Intensity matching matching
Bila tera l Constrained Bilat era l Constrained Filter Poisson Filter Poisson
Input PtPost- Input Post- process Image Image process Textureness Textureness matching matching Local contrast Local contrast
Result Result
Global contrast Additional Effects model Intensity matching • Soft focus (high frequency manipulation) • Film grain (texture synthesis [Heeger 95]) • Color toning (chrominance = f (luminance) )
Bilat era l Constrained Filter Poisson
Input SftfSoft focus Toning Image Grain before after effects Textureness effects matching Local contrast
Result Recap Global contrast Results Intensity matching User provides input and modldel photograp hs. Our system automatically produces the result. Bila tera l Constrained Filter Poisson Running times: Input SftfSoft focus Toning – 6 seconds for 1 MPixel or less Image Grain – 23 seconds for 4 MPixels Textureness matching . multi-grid Poisson solver and fast bilateral filter [Paris 06] Local contrast
Result
ResultInput Model ResultInput Comparison with Naïve Histogram Matching ResultInput Model
Model Input Snapshot, Alfred Stieglitz
Naïve Histogram Matching Our result Local contrast, sharpness unfaithful
Comparison with Naïve Histogram Matching Color Images
• Lab color space: modify only luminance
Input Output
Model Clearing Winter Storm, Ansel Input Adams
Histogram Matching Our Result Local contrast too low Limitations Conclusions
• TfTransfer “lk”“look” from a mo dldel photo • Noise and JPEG artifacts – amplified defects • Two-scale tone management – Global and local contrast – New edge-preserving textureness • Can lead to unexpected – Constrained Poisson reconstruction results if the image content is – Additional effects too different from the model – Portraits, in particular, can suffer
Video Enhancement Using Joint bilateral filtering Per Pixel Exposures (Bennett, 06) From this video:
ASTA: Adaptive Spatio- Temporal Accumultilation Filter Flash / No-Flash Photo Improvement Overview (Petschnigg04) (Eisemann04)
Mergg,yg(e best features: warm, cozy candle light (no-flash) Basic approach of both flash/noflash papers low-noise, detailed flash image
Remove noise + details from image A,
Keep as image A Lighting No-flash ------
Obtain noise -free details from image B,
Discard Image B Lighting Result
Petschnigg: Petschnigg: •Flash •No Flash, Petschnigg: Our Approach •Result Registration Registration
Our Approach Decomposition
Decomposition Color / Intensity:
= *
original intensity color Our Approach Our Approach
Decomposition Decoupling
Decoupling Large-scale Layer • Lighting : Large-scale variation • Texture : SSllmall-scale variat ion • Bilateral filter – edge preserving filter Smith and Brady 1997; Tomasi and Manducci 1998; Durand et al. 2002
ItInput OtOutput
Lighting Texture Large-scale Layer Cross Bilateral Filter • Bilateral filter • Similar to joint bilateral filter by Petschnigg et al. • When no-flash imaggye is too noisy • Borrow similarity from flash image edge stopping from flash image
Bilateral Cross Bilateral
Detail Layer Recombination
/ = * =
Intensity Large-scale Detail Large-scale Detail Intensity No-flash Flash Result
Recombination: Large scale * Detail = Intensity Recombination: Large scale * Detail = Intensity Recombination Our Approach
shadows
* ~
Intensity Color Result Result Flash
Recombination: Intensity * Color = Original
Our Approach Results
Shadow Detection/Treatment
No-flash
Flash Result Joint bilateral upsampling Joint bilateral upsampling
Joint bilateral upsampling Joint bilateral upsampling Joint bilateral upsampling Joint bilateral upsampling
Joint bilateral upsampling Joint bilateral upsampling