2690 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 20, NO. 9, SEPTEMBER 2011 [26] D. L. Donoho and I. M. Johnstone, “Adapting to unknown smoothness At a pixel location x a@xIYxPA, the output of the bilateral filter is via wavelet shrinkage,” J. Amer. Statist. Assoc., vol. 90, no. 432, pp. calculated as follows: 1200–1224, 1995. [27] A. Benazza-Benyahia and J.-C. Pesquet, “Building robust wavelet es- I s@xAa u @ky xkAu @js@yA s@xAjAs@yA timators for multicomponent images using Stein^{\prime}s principle,” g@xA d r (1) IEEE Trans. Image Process., vol. 14, no. 11, pp. 1814–1830, Nov. yPx @xA 2005. [28] F. Luisier, T. Blu, and M. Unser, “A new SURE approach to image where ud@A is the spatial domain kernel, ur@A is the intensity range denoising: Interscale orthonormal wavelet thresholding,” IEEE Trans. x @xA x Image Process., vol. 16, no. 3, pp. 593–606, Mar. 2007. kernel, is the set of pixels within a spatial neighborhood of , [29] S. Ramani, T. Blu, and M. Unser, “Monte-Carlo SURE: A black-box and g@xA is the normalization term optimization of regularization parameters for general denoising algo- rithms,” IEEE Trans. Image Process., vol. 17, no. 9, pp. 1540–1554, g@xAa ud@ky xkAur@js@yA s@xAjAX (2) Sep. 2008. [30] D. Van De Ville and M. Kocher, “SURE-based non-local means,” IEEE yPx @xA Signal Process. Lett., vol. 16, no. 11, pp. 973–976, Nov. 2009. [31] T. Blu and F. Luisier, “The SURE-LET approach to image denoising,” The kernels ud@A and ur@A determine how the spatial and intensity IEEE Trans. Image Process., vol. 16, no. 11, pp. 2778–2786, Nov. 2007. differences are treated. The contribution (weight) of a pixel s@yA is [32] J. Salmon, “On two parameters for denoising with non-local means,” determined by the product of ud@A and ur@A. The bilateral filter in IEEE Signal Process. Lett., vol. 17, no. 3, pp. 269–272, Mar. 2010. P P [33] J. Salmon and Y. Strozecki, “From patches to pixels in non-local [1] uses the Gaussian kernel, q'@zAaexp@z aP' A, for both the methods: Weighted-average reprojection,” in Proc. 2010 IEEE 17th domain and range kernels: Int. Conf. Image Process., 2010, pp. 1929–1932. ud@ky xkAaq' @ky xkA (3) and Fast Bilateral Filter With Arbitrary Range and ur@js@yA s@xAjAaq' @js@yA s@xAjAX (4) Domain Kernels On the other hand, the sigma filter [3] and the neighborhood filter [4] Bahadir K. Gunturk, Senior Member, IEEE use different kernels. The sigma filter [3] first calculates the local stan- dard deviation around s@xA; the standard deviation is then used to de- termine a threshold value for pixel intensities, and pixels that are within s@xA Abstract—In this paper, we present a fast implementation of the bilat- the threshold of the center pixel are averaged (with equal weights) eral filter with arbitrary range and domain kernels. It is based on the his- to calculate the filter output at that pixel. In case of the neighborhood togram-based fast bilateral filter approximation that uses uniform box as filter [4], the range kernel is a Gaussian as in (3), and the spatial kernel the domain kernel. Instead of using a single box kernel, multiple box ker- is a uniform box kernel. Among different kernel options, the Gaussian nels are used and optimally combined to approximate an arbitrary domain kernel is the most popular choice for both the range and spatial kernels, kernel. The method achieves better approximation of the bilateral filter compared to the single box kernel version with little increase in compu- as it gives an intuitive and simple control of the behavior of the filter tational complexity. We also derive the optimal kernel size when a single with two parameters, 'd and 'r. box kernel is used. The bilateral filter has found a wide range of applications in image Index Terms—Image enhancement, nonlinear filtering. processing and computer vision. The immediate application of the bi- lateral filter is image denoising as it can do spatial averaging without blurring edges. [5] presents a multiresolution extension of the bilateral I. INTRODUCTION filter for image denoising and an empirical study on optimal parameter selection. It is shown that the optimal value of 'd is relatively insensi- The bilateral filter is a nonlinear weighted averaging filter, where the tive to noise power, while the optimal 'r value is linearly proportional weights depend on both the spatial distance and the intensity distance to the noise standard deviation. Other applications of bilateral filter in- with respect to the center pixel. The main feature of the bilateral filter clude tone mapping in high-dynamic range imaging [6], contrast en- is its ability to preserve edges while doing spatial smoothing. The term hancement [7], [8], fusion of flash and no-flash images [9], [10], fusion “bilateral filter” was first used by Tomasi and Manduchi in [1]; the of visible spectrum and infrared spectrum images [11], compression same filter was earlier called the SUSAN (Smallest Univalue Segment artifact reduction [12], 3-D mesh denoising [13], [14], depth map es- Assimilating Nucleus) filter by Smith and Brady in [2]. The variants of timation [15], video stylization [16], video enhancement [17], texture the bilateral filter have been published even earlier as the sigma filter and illumination separation [18], orientation smoothing [19], and op- [3] and the neighborhood filter [4]. tical flow estimation [20]. This paper presents a fast approximation of the bilateral filter with Manuscript received April 14, 2010; revised November 02, 2010; accepted arbitrary range and domain kernels. It is based on a method presented February 18, 2011. Date of publication March 14, 2011; date of current ver- by Porikli in [21]. The method in [21] (which uses a box domain sion August 19, 2011. This work was supported in part by the National Science Foundation under Grant 0528785 and the National Institutes of Health under kernel) is extended by optimally combining multiple box kernels to Grant 1R21AG032231-01. The associate editor coordinating the review of this approximate an arbitrary domain kernel. As there is no restriction on manuscript and approving it for publication was Dr. Luminita Aura Vese. the range kernel either, any range and domain kernels can be used The author is with the Department of Electrical and Computer Engi- with this fast bilateral filter implementation. Section II reviews the neering, Louisiana State University, Baton Rouge, LA 70803 USA (e-mail: fast bilateral filter techniques in the literature. The proposed method is [email protected]). Color versions of one or more of the figures in this paper are available online explained in Section III. In Section IV, the question of optimal kernel at http://ieeexplore.ieee.org. size in case of a single box kernel is addressed. Section V provides Digital Object Identifier 10.1109/TIP.2011.2126585 some experimental results, and Section VI concludes the paper. 1057-7149/$26.00 © 2011 IEEE IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 20, NO. 9, SEPTEMBER 2011 2691 PSS II. FAST BILATERAL FILTER METHODS I a irxYx @iAur@ji s@xAjA (5) gx @xA The direct implementation of the bilateral filter is given here. iaH • For each pixel x P, where is the set of all pixels in the image i P HY PSS r • Initialize: s@xAaHand g@xAaH. where in for an 8-bit image, xYx is the local histogram Px CI Px CI x f @ A • For each y in the local neighborhood x of x: in the neighborhood around , x is the box • Calculate the weight: w a ud@ky xkAur@js@yA s@xAjA. kernel, given by • Update: s@xAas@xACws@yA. IY jzj j xY j aIY P • Update: g@xAag@xACw. fx @zAa (6) HY otherwise • Normalize: s@xAas@xAag@xA. The computational complexity of this implementation is y@jjjx jA, and gx @xA is the normalization term, given by where jj is the number of pixels in the entire image and jx j is the number of pixels in the neighborhood x . The local neighborhood is PSS typically chosen such that kyxk Q'd; therefore, the neighborhood gx @xAa rxYx @iAur@ji s@xAjAX (7) P P size jx j is proportional to 'd . While the overall complexity y jj'd iaH is acceptable for small 'd, it quickly becomes limiting with increasing There are several advantages of this formulation [21], including 'd. To address this issue, a number of fast implementation/approxima- tion methods have been proposed. those given here. 1) ur@ji s@xAjA can be calculated for all values of i and at all locations x independently and therefore in parallel. Simi- A. Kernel Separation larly, rxYx @iA and irxYx @iA, which are also required in the One method of speeding up the bilateral filter is to separate the 2-D bilateral filter calculation (5), can be obtained independent of filter kernel into two 1-D kernels. First, the rows of an image are fil- ur@ji s@xAjA. tered; the result is then filtered along the columns [22], [23]. This re- 2) rxYx @iA can be calculated efficiently using the integral histogram duces the complexity to y@jj'dA. Although its performance is good technique [25]. in smooth regions and horizontal/vertical edges, the algorithm may not 3) The algorithm can be further speeded up by quantizing the perform satisfactorily on texture regions because of 1-D handling of histogram. spatial domain. As a result, the box bilateral filter can be implemented independent of the kernel size.