Mean Shift Analysis and Applications Dorin Comaniciu Peter Meer Department of Electrical and Computer Engineering Rutgers University, Piscataway, NJ 08854-8058, USA fcomanici, [email protected] to the average of the data p oints within. Details in the Abstract image are preserved due to the nonparametric charac- A nonparametric estimator of density gradient, the ter of the analysis which do es not assume a priori any mean shift, is employed in the joint, spatial-range particular structure for the data. value domain of gray level and color images for dis- The pap er is organized as follows. Section 2 dis- continuity preserving ltering and image segmentation. cusses the estimation of the density gradient and de- Prop erties of the mean shift are reviewed and its con- nes the mean shift vector. The convergence of the vergence on lattices is proven. The prop osed ltering mean shift pro cedure is proven in Section 3 for discrete metho d asso ciates with each pixel in the image the clos- data. Section 4 de nes the pro cessing principle in the est lo cal mo de in the density distribution of the joint do- joint spatial-range domain. Mean shift ltering is ex- main. Segmentation into a piecewise constant structure plained and ltering examples are given in Section 5. requires only one more step, fusion of the regions asso- The prop osed mean shift segmentation is intro duced ciated with nearby mo des. The prop osed technique has and analyzed in Section 6. two parameters controlling the resolution in the spatial and range domains. Since convergence is guaranteed, 2 Density Gradient Estimation the technique does not require the intervention of the Let fx g b e an arbitrary set of n p oints in the i i=1:::n user to stop the ltering at the desired image quality. d d-dimensional Euclidean space R . The multivariate Several examples, for gray and color images, show the kernel density estimate obtained with kernel K x and versatilityof the metho d and compare favorably with window radius h, computed in the p oint x is de ned as results describ ed in the literature for the same images. [12, p.76] n X 1 Intro duction 1 x x i ^ f x = K : 1 d Low level computer vision tasks are misleadingly dif- nh h i=1 cult and often yield unreliable results, since the em- ployed techniques rely up on the correct choice by the The optimum kernel yielding minimum mean inte- user of the tuning parameter values. Today,itisanac- grated square error MISE is the Epanechnikovkernel cepted fact in the vision community that the execution 1 1 T T c d + 21 x x if x x < 1 of low level tasks should b e task driven, i.e., supp orted d 2 K x = E 0 otherwise by indep endent high level information. To be able to 2 successfully complement this paradigm, the low-level where c is the volume of the unit d-dimensional sphere techniques must b ecome more autonomous. In this pa- d [12, p.76]. per we prop ose such a technique for image smo othing The use of a di erentiable kernel allows to de ne the and for segmentation. estimate of the density gradient as the gradient of the The mean shift estimate of the gradient of a density kernel density estimate 1 function and the asso ciated iterative pro cedure of mo de seeking have b een develop ed byFukunaga and Hostetler n X 1 x x i in [6]. Only recently, however, the nice prop erties of ^ ^ rf x rf x = rK : 3 d nh h data compaction and dimensionality reduction of the i=1 mean shift have b een exploited in low level computer Conditions on the kernel K x and the window radius vision tasks color space analysis [3], face tracking [1]. h to guarantee asymptotic unbiasedness, mean-square In this pap er we describ e a new application based consistency, and uniform consistency are derived in [6]. on the theoretical results obtained in [4]. We show that For the Epanechnikovkernel 2 the density gradient high quality edge preserving ltering and image seg- estimate 3 b ecomes mentation can b e obtained by applying the mean shift in the combined spatial-range domain. The metho ds we X 1 d +2 ^ develop ed are conceptually very simple b eing based on [x x] rf x = i d 2 nh c h d the same idea of iteratively shifting a xed size window x 2S x i h 0 1 X 8 d +2 1 n x @ A [x x ] 4 = i 6 d 2 nh c h n d x x 2S x i h 4 2 where the region S x is a hyp ersphere of radius h h 0 d ving the volume h c , centered on x, and containing ha −2 d −4 n data p oints. The last term in 4 x −6 X X 1 1 −8 M x [x x]= x x 5 h i i −10 −5 0 5 10 n n x x x 2S x x 2S x i i h h Figure 1: Successive computations of the mean shift is called the sample mean shift. Using a kernel di er- de ne a path leading to a lo cal density maximum. ent from the Epanechnikovkernel results in a weighted mean computation in 5. n the direction of the gradient of the density estimate at x The quantity is the kernel density estimate d nh c d x, it is not apparent that the density estimate at lo- ^ f x computed with the hyp ersphere S x the uni- h cations fy g is a monotonic increasing sequence. k k =1;2::: form kernel, and thus we can write 4 as Moving in the direction of the gradient guarantees hill climbing only for in nitesimal steps. The following the- d +2 ^ ^ rf x=f x M x; 6 h 2 orem asserts the convergence for discrete data. h which yields o n ^ ^ 2 f y ;K be the se- Theorem 1 Let f = ^ k E E k rf x h k =1;2::: : 7 M x = h ^ quence of density estimates obtained using Epanech- d +2 f x nikov kernel and computed in the points fy g de- k k =1;2::: The expression 7 was rst derived in [6] and shows ned by the successive locations of the mean shift proce- that an estimate of the normalized gradient can b e ob- dure with uniform kernel. The sequenceisconvergent. tained by computing the sample mean shift in a uniform kernel centered on x . The mean shift vector has the di- Pro of Since the data set fx g has nite car- i i=1:::n rection of the gradient of the density estimate at x when ^ dinality n, the sequence f is b ounded. Moreover, we E this estimate is obtained with the Epanechnikovkernel. ^ will show that f is strictly monotonic increasing, i.e., E Since the mean shift vector always p oints towards ^ ^ if y 6= y then f k < f k + 1, for all k =1; 2 :::. E E k k +1 00 0 00 0 the direction of the maximum increase in the density,it b e the num- + n with n = n , and n Let n , n k k k k k k can de ne a path leading to a lo cal density maximum, b er of data p oints falling in the d-dimensional windows i.e., to a mo de of the density Figure 1. 0 00 Figure 2 S y , S y =S y S y , and h h h h k k k k T The mean shift procedure, obtained by successive 00 S y =S y S y . h h h k k k +1 computation of the mean shift vector M x h Without loss of generalitywe can assume the origin translation of the window S xby M x , h h lo cated at y . Using the de nition of the density esti- k mate 1 with the Epanechnikovkernel 2 and noting is guaranteed to converge, as it will be shown in the 2 2 that ky x k = kx k wehave i i k next section. 3 Convergence ^ ^ f k = f y ;K E k E k Let fy g denote the sequence of successive lo- X k k =1;2::: 1 y x i k = K cations of the mean shift pro cedure. By de nition we E d nh h x 2S y have for each k=1,2. i h k 2 X kx k d +2 i X 1 1 : 9 = d 2 y = x ; 8 i 2nh c h k +1 d n x 2S y k i h k x 2S y i h k where y is the center of the initial window and n is the k 1 Since the kernel K is nonnegativewe also have E numberofpoints falling in the window S y centered h k ^ ^ on y . f k +1= f y ;K k E k +1 E k +1 The convergence of the mean shift has b een justi ed X y x 1 i k +1 K as a consequence of relation 7, see [2]. However, E d nh h 00 while it is true that the mean shift vector M x has x 2S y h i k h 4 Pro cessing in Spatial-Range Domain An image is typically represented as a 2-dimensional lattice of r-dimensional vectors pixels, where r is1in the gray level case, 3 for color images, or r>3 in the multisp ectral case.