IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 24, NO. 5, MAY 2002 603 : A Robust Approach Toward Feature Space Analysis

Dorin Comaniciu, Member, IEEE, and Peter Meer, Senior Member, IEEE

AbstractÐA general nonparametric technique is proposed for the analysis of a complex multimodal feature space and to delineate arbitrarily shaped clusters in it. The basic computational module of the technique is an old pattern recognition procedure, the mean shift. We prove for discrete data the convergence of a recursive mean shift procedure to the nearest stationary point of the underlying density function and, thus, its utility in detectingthe modes of the density. The relation of the mean shift procedure to the Nadaraya- Watson estimator from regression and the robust M-estimators of location is also established. Algorithms for two low-level vision tasks, discontinuity preserving smoothing and image segmentation, are described as applications. In these algorithms, the only user set parameter is the resolution of the analysis and either gray level or color images are accepted as input. Extensive experimental results illustrate their excellent performance.

Index TermsÐMean shift, clustering, image segmentation, image smoothing, feature space, low-level vision. æ

1INTRODUCTION

OW-LEVEL tasks are misleadingly diffi- significant feature is pooled together,providing excellent Lcult. Incorrect results can be easily obtained since the tolerance to a noise level which may render local decisions employed techniques often rely upon the user correctly unreliable. On the other hand,features with lesser support guessing the values for the tuning parameters. To improve in the feature space may not be detected in spite of being performance,the execution of low-level tasks should be task salient for the task to be executed. This disadvantage, driven,i.e.,supported by independent high-level informa- however,can be largely avoided by either augmenting the tion. This approach,however,requires that,first,the low- feature space with additional 'spatial) parameters from the level stage provides a reliable enough representation of the input domain or by robust postprocessing of the input input and that the feature extraction process be controlled domain guided by the results of the feature space analysis. only by very few tuning parameters corresponding to Analysis of the feature space is application independent. intuitive measures in the input domain. While there are a plethora of published clustering techni- Feature space-based analysis of images is a paradigm ques,most of them are not adequate to analyze feature which can achieve the above-stated goals. A feature space is spaces derived from real data. Methods which rely upon a mapping of the input obtained through the processing of a priori knowledge of the number of clusters present the data in small subsets at a time. For each subset,a 'including those which use optimization of a global parametric representation of the feature of interest is criterion to find this number),as well as methods which obtained and the result is mapped into a point in the implicitly assume the same shape 'most often elliptical) for multidimensional space of the parameter. After the entire all the clusters in the space,are not able to handle the input is processed,significant features correspond to denser complexity of a real feature space. For a recent survey of regions in the feature space,i.e.,to clusters,and the goal of such methods,see [29,Section 8]. the analysis is the delineation of these clusters. In Fig. 1,a typical example is shown. The color image in The nature of the feature space is application dependent. Fig. 1a is mapped into the three-dimensional L*u*v* color The subsets employed in the mapping can range from space 'to be discussed in Section 4). There is a continuous individual pixels,as in the color space representation of an transition between the clusters arising from the dominant image,to a set of quasi-randomly chosen data points,as in colors and a decomposition of the space into elliptical tiles the probabilistic Hough transform. Both the advantage and will introduce severe artifacts. Enforcing a Gaussian the disadvantage of the feature space paradigm arise from mixture model over such data is doomed to fail,e.g.,[49], the global nature of the derived representation of the input. and even the use of a robust approach with contaminated On one hand,all the evidence for the presence of a Gaussian densities [67] cannot be satisfactory for such complex cases. Note also that the mixture models require the number of clusters as a parameter,which raises its own . D. Comaniciu is with the Imaging and Visualization Department, Siemens Corporate Research, 755 College Road East, Princeton, NJ08540. challenges. For example,the method described in [45] E-mail: [email protected]. proposes several different ways to determine this number. . P. Meer is with the Electrical and Computer Engineering Department, Arbitrarily structured feature spaces can be analyzed Rutgers University, 94 Brett Road, Piscataway, NJ08854-8058. only by nonparametric methods since these methods do not E-mail: [email protected]. have embedded assumptions. Numerous nonparametric Manuscript received 17 Jan. 2001; revised 16 July 2001; accepted 21 Nov. clustering methods were described in the literature and 2001. they can be classified into two large classes: hierarchical Recommended for acceptance by V. Solo. For information on obtaining reprints of this article, please send e-mail to: clustering and density estimation. [email protected], and reference IEEECS Log Number 113483. techniques either aggregate or divide the data based on

0162-8828/02/$17.00 ß 2002 IEEE 604 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 24, NO. 5, MAY 2002

Fig. 1. Example of a feature space.

d some proximity measure. See [28,Section 3.2] for a survey points xi, i ˆ 1; ...;n in the d-dimensional space R ,the of hierarchical clustering methods. The hierarchical meth- multivariate kernel density estimator with kernel K x† and a ods tend to be computationally expensive and the definition symmetric positive definite d  d bandwidth matrix H, of a meaningful stopping criterion for the fusion 'or computed in the point x is given by division) of the data is not straightforward. 1 Xn The rationale behind the density estimation-based non- f^ x†ˆ K †x À x ; 1† n H i parametric clustering approach is that the feature space can iˆ1 be regarded as the empirical probability density function 'p.d.f.) of the represented parameter. Dense regions in the where feature space thus correspond to local maxima of the p.d.f., K x†ˆjH jÀ1=2 K HÀ1=2x†: 2† that is,to the modes of the unknown density. Once the H location of a is determined,the cluster associated The d-variate kernel K x† is a bounded function with with it is delineated based on the local structure of the compact support satisfying [62,p. 95] feature space [25],[60],[63]. Z Our approach to mode detection and clustering is based on K x†dx ˆ 1 lim kxkdK x†ˆ0 the mean shift procedure,proposed in 1975 by Fukunaga and d kxk!1 Z R Z 3† Hostetler [21] and largely forgotten until Cheng's paper [7] > rekindled interest in it. In spite of its excellent qualities,the xK x†dx ˆ 0 xx K x†dx ˆ cK I; Rd Rd mean shift procedure does not seem to be known in statistical literature. While the book [54,Section 6.2.2] discusses [21],the where cK is a constant. The multivariate kernel can be advantages of employing a mean shift type procedure in generated from a symmetric univariate kernel K1 x† in two density estimation were only recently rediscovered [8]. different ways As will be proven in the sequel,a computational module Yd based on the mean shift procedure is an extremely versatile P S K x†ˆ K1 xi† K x†ˆak;dK1 kxk†; 4† tool for feature space analysis and can provide reliable iˆ1 solutions for many vision tasks. In Section 2,the mean shift P procedure is defined and its properties are analyzed. In where K x† is obtained from the product of the univariate S d S Section 3,the procedure is used as the computational kernels and K x† from rotating K1 x† in R R,i.e., K x† is radially symmetric. The constant aÀ1 ˆ K kxk†dx module for robust feature space analysis and implementa- k;d Rd 1 S tional issues are discussed. In Section 4,the feature space assures that K x† integrates to one,though this condition analysis technique is applied to two low-level vision tasks: can be relaxed in our context. Either type of multivariate discontinuity preserving filtering and image segmentation. kernel obeys '3),but,for our purposes,the radially Both algorithms can have as input either gray level or color symmetric kernels are often more suitable. images and the only parameter to be tuned by the user is We are interested only in a special class of radially the resolution of the analysis. The applicability of the mean symmetric kernels satisfying shift procedure is not restricted to the presented examples. K x†ˆc k kxk2†; 5† In Section 5,other applications are mentioned and the k;d procedure is put into a more general context. in which case it suffices to define the function k x† called the profile of the kernel,only for x  0. The normalization constant c ,which makes K x† integrate to one,is 2THE MEAN SHIFT PROCEDURE k;d assumed strictly positive. Kernel density estimation 'known as the Parzen window Using a fully parameterized H increases the complexity technique in pattern recognition literature [17,Section 4.3]) is of the estimation [62,p. 106] and,in practice,the bandwidth 2 2 the most popular density estimation method. Given n data matrix H is chosen either as diagonal H ˆ diag‰h1; ...;hdŠ, COMANICIU AND MEER: MEAN SHIFT: A ROBUST APPROACH TOWARD FEATURE SPACE ANALYSIS 605 or proportional to the identity matrix H ˆ h2I. The clear We define the function advantage of the latter case is that only one bandwidth parameter h>0 must be provided; however,as can be seen g x†ˆÀk0 x†; 13† from '2),then the validity of an Euclidean metric for the assuming that the derivative of the kernel profile k exists for feature space should be confirmed first. Employing only all x 2‰0; 1†,except for a finite set of points. Now,using one bandwidth parameter,the kernel density estimator '1) g x† for profile,the kernel G x† is defined as becomes the well-known expression  n  2 1 X x À x G x†ˆcg;dg kxk ; 14† f^ x†ˆ K i : 6† nhd h iˆ1 where cg;d is the corresponding normalization constant. The The quality of a kernel density estimator is measured by kernel K x† was called the shadow of G x† in [7] in a slightly the mean of the square error between the density and its different context. Note that the Epanechnikov kernel is the estimate,integrated over the domain of definition. In practice, shadow of the uniform kernel,i.e.,the d-dimensional unit however,only an asymptotic approximation of this measure sphere,while the normal kernel and its shadow have the same 'denoted as AMISE) can be computed. Under the asympto- expression. tics,the number of data points n!1,while the bandwidth Introducing g x† into '12) yields, h!0 at a rate slower than nÀ1. For both types of multivariate kernels,the AMISE measure is minimized by the Epanechni- ^ rfh;K x† kov kernel [51,p. 139],[62,p. 104] having the profile  2c Xn x À x 2  ˆ k;d †x À x g i 1 À x 0  x  1 nhd‡2 i h k x†ˆ 7† iˆ1 E 0 x>1; 2P  3 "#n xÀx 2 Xn x g i 2ck;d x À xi 2 iˆ1 i h which yields the radially symmetric kernel ˆ g 4  À x5; d‡2 Pn 2 nh h g xÀxi  iˆ1 iˆ1 h 1 À1 2 cd d ‡ 2† 1 Àkxk †kxk1 KE x†ˆ 2 8† 15† 0 otherwise;  Pn 2 where g xÀxi is assumed to be a positive number. where c is the volume of the unit d-dimensional sphere. iˆ1 h d This condition is easy to satisfy for all the profiles met in Note that the Epanechnikov profile is not differentiable at practice. Both terms of the product in '15) have special the boundary. The profile significance. From '11),the first term is proportional to the  1 density estimate at x computed with the kernel G kN x†ˆexp À x x  0 9† 2 n  c X x À x 2 f^ x†ˆ g;d g i : 16† yields the multivariate normal kernel h;G nhd h iˆ1  Àd=2 1 2 The second term is the mean shift KN x†ˆ 2† exp À kxk 10† 2  Pn 2 x g xÀxi for both types of composition '4). The normal kernel is often iˆ1 i h mh;G x†ˆ  À x; 17† Pn 2 symmetrically truncated to have a kernel with finite support. g xÀxi While these two kernels will suffice for most applications iˆ1 h we are interested in,all the results presented below are valid i.e.,the difference between the weighted mean,using the for arbitrary kernels within the conditions to be stated. kernel G for weights,and x,the center of the kernel Employing the profile notation,the density estimator '6) can 'window). From '16) and '17),'15) becomes be rewritten as 2ck;d n  r^ f x†ˆf^ x† m x†; 18† c X x À x 2 h;K h;G 2 h;G f^ x†ˆ k;d k i : 11† h cg;d h;K nhd h iˆ1 yielding The first step in the analysis of a feature space with the 1 r^ f x† underlying density f x† is to find the modes of this density. m x†ˆ h2c h;K : 19† h;G ^ The modes are located among the zeros of the gradient 2 fh;G x† rf x†ˆ0 and the mean shift procedure is an elegant way to locate these zeros without estimating the density. The expression '19) shows that,at location x,the mean shift vector computed with kernel G is proportional to the normal- 2.1 Density Gradient Estimation ized density gradient estimate obtained with kernel K. The The density gradient estimator is obtained as the gradient of normalization is by the density estimate in x computed with the density estimator by exploiting the linearity of '11) the kernel G. The mean shift vector thus always points toward the direction of maximum increase in the density. This is a Xn 2ck;d x À xi 2 more general formulation of the property first remarked by r^ f x†rf^ x†ˆ †x À x k0 : h;K h;K nhd‡2 i h iˆ1 Fukunaga and Hostetler [20,p. 535],[21],and discussed in [7]. The relation captured in '19) is intuitive,the local mean is 12† shifted toward the region in which the majority of the 606 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 24, NO. 5, MAY 2002 points reside. Since the mean shift vector is aligned with the procedures to chose the adequate step sizes. This is a major local gradient estimate,it can define a path leading to a advantage over the traditional gradient-based methods. stationary point of the estimated density. The modes of the For discrete data,the number of steps to convergence density are such stationary points. The mean shift procedure, depends on the employed kernel. When G is the uniform obtained by successive kernel,convergence is achieved in a finite number of steps since the number of locations generating distinct mean . computation of the mean shift vector mh;G x†, values is finite. However,when the kernel G imposes a . translation of the kernel 'window) G x† by mh;G x†, weighting on the data points 'according to the distance is guaranteedto converge atanearby pointwhere the estimate from its center),the mean shift procedure is infinitely '11) has zero gradient,as will be shown in the next section. The convergent. The practical way to stop the iterations is to set presence of the normalization by the density estimate is a a lower bound for the magnitude of the mean shift vector. desirable feature. The regions of low-density values are of no interest for the feature space analysis and,in such regions,the 2.3 Mean Shift-Based Mode Detection ^c ^ mean shift steps are large. Similarly,near local maxima the Let us denote by yc and fh;K ˆ fh;K yc† the convergence ^ steps are small and the analysis more refined. The mean shift points of the sequences fyjgjˆ1;2... and ffh;K j†gjˆ1;2..., procedure thus is an adaptive gradient ascent method. respectively. The implications of Theorem 1 are the following. First,the magnitude of the mean shift vector converges to 2.2 Sufficient Condition for Convergence zero. Indeed,from '17) and '20) the jth mean shift vector is Denote by fyjgjˆ1;2... the sequence of successive locations of the kernel G,where,from '17), mh;G yj†ˆyj‡1 À yj 22†  Pn 2 and,at the limit, mh;G yc†ˆyc À yc ˆ 0. In other words,the x g xÀxi iˆ1 i h gradient of the density estimate '11) computed at y is zero y ˆ j ˆ 1; 2; ... 20† c j‡1 P 2 n xÀxi iˆ1 g h ^ rfh;K yc†ˆ0; 23† is the weighted mean at y computed with kernel G and y ^ j 1 due to '19). Hence, yc is a stationary point of fh;K. Second, ^ is the center of the initial position of the kernel. The since ffh;K j†gjˆ1;2... is monotonically increasing,the mean corresponding sequence of density estimates computed shift iterations satisfy the conditions required by the Capture ^ Theorem [4,p. 45],which states that the trajectories of such with kernel K, ffh;K j†gjˆ1;2...,is given by gradient methods are attracted by local maxima if they are ^ ^ fh;K j†ˆfh;K yj† j ˆ 1; 2...: 21† unique 'within a small neighborhood) stationary points. ^ That is,once yj gets sufficiently close to a mode of fh;K,it As stated by the following theorem,a kernel K that obeys converges to it. The set of all locations that converge to the some mild conditions suffices for the convergence of the ^ same mode defines the basin of attraction of that mode. sequences fyjgjˆ1;2... and ffh;K j†gjˆ1;2.... The theoretical observations from above suggest a Theorem 1. If the kernel K has a convex andÈÉ monotonically practical algorithm for mode detection: decreasing profile, the sequences y and j jˆ1;2... ff^ j†g converge and ff^ j†g is monotoni- . Run the mean shift procedure to find the stationary h;K jˆ1;2... h;K jˆ1;2... points of f^ , cally increasing. h;K . Prune these points by retaining only the local maxima. The proof is given in the Appendix. The theorem generalizes the result derived differently in [13],where K The local maxima points are defined,according to the was the Epanechnikov kernel and G the uniform kernel. The Capture Theorem,as unique stationary points within some theorem remains valid when each data point x is associated small open sphere. This property can be tested by i perturbing each stationary point by a random vector of with a nonnegative weight wi. An example of nonconver- gence when the kernel K is not convex is shown in [10,p. 16]. small norm and letting the mean shift procedure converge The convergence property of the mean shift was also again. Should the point of convergence be unchanged 'up to discussed in [7,Section iv]. 'Note,however,that almost all the a tolerance),the point is a local maximum. discussion there is concerned with the ªblurringº process in 2.4 Smooth Trajectory Property which the input is recursively modified after each mean shift The mean shift procedure employing a normal kernel has step.) The convergence of the procedure as defined in this an interesting property. Its path toward the mode follows a paperwasattributedin[7]tothegradientascentnatureof'19). smooth trajectory,the angle between two consecutive mean However,as shown in [4,Section 1.2],moving in the direction shift vectors being always less than 90 degrees. of the local gradient guarantees convergence only for Using the normal kernel '10),the jth mean shift vector is infinitesimal steps. The step size of a gradient-based algo- given by rithm is crucial for the overall performance. If the step size is  toolarge,thealgorithmwilldiverge,whileifthestepsizeistoo P 2 n xÀxi iˆ1 xi exp h small,the rate of convergence may be very slow. A number of m y †ˆy À y ˆ À y : 24† h;N j j‡1 j P 2 j costly procedures have been developed for step size selection n xÀxi iˆ1 exp h [4,p. 24]. The guaranteed convergence 'as shown by Theorem 1) is due to the adaptive magnitude of the mean The following theorem holds true for all j ˆ 1; 2; ..., shift vector,which also eliminates the need for additional according to the proof given in the Appendix. COMANICIU AND MEER: MEAN SHIFT: A ROBUST APPROACH TOWARD FEATURE SPACE ANALYSIS 607

0 Theorem 2. The cosine of the angle between two consecutive 2 f x† E‰ x^ À x†jX1; ...;Xn Šh ; 29† mean shift vectors is strictly positive when a normal kernel is f x†2‰gŠ employed, i.e., which is similar to '19). The mean shift procedure thus > exploits to its advantage the inherent bias of the zero-order mh;N y † mh;N y † j j‡1 > 0: 25† kernel regression. km y †kkm y †k h;N j h;N j‡1 The connection to the kernel regression literature opens As a consequence of Theorem 2,the normal kernel many interesting issues,however,most of these are more of appears to be the optimal one for the mean shift procedure. a theoretical than practical importance. The smooth trajectory of the mean shift procedure is in 2.6 Relation to Location M-Estimators contrast with the standard steepest ascent method [4,p. 21] 'local gradient evaluation followed by line maximization) The M-estimators are a family of robust techniques which can whose convergence rate on surfaces with deep narrow handle data in the presence of severe contaminations,i.e., valleys is slow due to its zigzagging trajectory. outliers. See [26],[32] for introductory surveys. In our context In practice,the convergence of the mean shift procedure only,the problem of location estimation has to be considered. based on the normal kernel requires large number of steps, Given the data xi;iˆ 1; ...;n; and the scale h,will as was discussed at the end of Section 2.2. Therefore,in define ^,the location estimator as most of our experiments,we have used the uniform kernel, ! for which the convergence is finite,and not the normal Xn 2 ^  À xi kernel. Note,however,that the quality of the results almost  ˆ argmin J †ˆargmin  ; 30†   h always improves when the normal kernel is employed. iˆ1 where,  u† is a symmetric,nonnegative valued function, 2.5 Relation to Kernel Regression with a unique minimum at the origin and nondecreasing for Important insight can be gained when '19) is obtained u  0. The estimator is obtained from the normal equations approaching the problem differently. Considering the 0 1 univariate case suffices for this purpose. 2 ^À x Kernel regression is a nonparametric method to estimate ^ À2 ^ @ i A rJ †ˆ2h  À xi†w ˆ 0; 31† complex trends from noisy data. See [62,chapter 5] for an h introduction to the topic,[24] for a more in-depth treatment. Let n measured data points be Xi;Zi† and assume that the where values X are the outcomes of a random variable x with i d u† probability density function f x†, xi ˆ Xi;iˆ 1; ...;n, w u†ˆ : du while the relation between Zi and Xi is Therefore,the iterations to find the location M-estimate are Z ˆ m X †‡ i ˆ 1; ...;n; 26† i i i based on where m x† is called the regression function and i is an  2 Pn ^ independently distributed,zero-mean error, E‰iŠˆ0. Àxi iˆ1 xiw h A natural way to estimate the regression function is by ^ ˆ ; 32† 2 locally fitting a degree p polynomial to the data. For a window Pn ^ w Àxi centered at x,the polynomial coefficients then can be obtained iˆ1 h byweightedleastsquares,theweightsbeingcomputedfroma which is identical to '20) when w u†g u†. Taking into symmetric function g x†. The size of the window is controlled À1 account '13),the minimization '30) becomes by the parameter h, gh x†ˆh g x=h†. The simplest case is that of fitting a constant to the data in the window,i.e., p ˆ 0.It ! Xn  À x 2 can be shown,[24,Section 3.1],[62,Section 5.2],that ^ ˆ argmax k i ; 33†  h the estimated constant is the value of the Nadaraya- iˆ1 Watson estimator, which can also be interpreted as P n g x À X †Z Piˆ1 h i i ^ ^ m^ x; h†ˆ n ; 27†  ˆ argmax fh;K  j x1; ...; xn†: 34† iˆ1 gh x À Xi†  introduced in the statistical literature 35 years ago. The That is,the location estimator is the mode of the density asymptotic conditional bias of the estimator has the estimated with the kernel K from the available data. Note that expression [24,p. 109],[62,p. 125], the convexity of the k x† profile,the sufficient condition for the convergence of the mean shift procedure 'Section 2.2) is in E‰ m^ x; h†Àm x†† j X1; ...;Xn Š accordance with the requirements to be satisfied by the 00 0 0 28† 2 m x†f x†‡2m x†f x†  h 2‰gŠ; objective function  u†. 2f x† The relation between location M-estimators and kernel R 2 where 2‰gŠˆ u g u†du. Defining m x†ˆx reduces the density estimation is not well-investigated in the statistical Nadaraya-Watson estimator to '20) 'in the univariate case), literature,only [9] discusses it in the context of an edge while '28) becomes preserving smoothing technique. 608 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 24, NO. 5, MAY 2002

3ROBUST ANALYSIS OF FEATURE SPACES that for a synthetic,bimodal normal distribution,the technique achieves a classification error similar to the Multimodality and arbitrarily shaped clusters are the defin- optimal Bayesian classifier. The behavior of this feature ing properties of a real feature space. The quality of the mean space analysis technique is illustrated in Fig. 2. A two- shift procedure to move toward the mode 'peak) of the hill on dimensional data set of 110; 400 points 'Fig. 2a) is decom- which it was initiated makes it the ideal computational posed into seven clusters represented with different colors module to analyze such spaces. To detect all the significant in Fig. 2b. A number of 159 mean shift procedures with modes,the basic algorithm given in Section 2.3 should be run multiple times 'evolving in principle in parallel) with uniform kernel were employed. Their trajectories are shown initializations that cover the entire feature space. in Fig. 2c,overlapped over the density estimate computed Before the analysis is performed,two important 'and with the Epanechnikov kernel. The pruning of the mode somewhat related) issues should be addressed: the metric of candidates produced seven peaks. Observe that some of the the feature space and the shape of the kernel. The mapping trajectories are prematurely stopped by local plateaus. from the input domain into a feature space often associates 3.1 Bandwidth Selection a non-Euclidean metric to the space. The problem of color The influence of the bandwidth parameter h was assessed representation will be discussed in Section 4,but the employed parameterization has to be carefully examined empirically in [12] through a simple image segmentation even in a simple case like the Hough space of lines,e.g., task. In a more rigorous approach,however,four different [48],[61]. techniques for bandwidth selection can be considered. The presence of a Mahalanobis metric can be accommo- . The first one has a statistical motivation. The optimal dated by an adequate choice of the bandwidth matrix '2). In bandwidth associated with the kernel density esti- practice,however,it is preferable to have assured that the mator '6) is defined as the bandwidth that achieves the metric of the feature space is Euclidean and,thus,the best compromise between the bias and variance of the bandwidth matrix is controlled by a single parameter, estimator,over all x 2 Rd,i.e.,minimizes AMISE. In 2 H ˆ h I. To be able to use the same kernel size for all the the multivariate case,the resulting bandwidth for- mean shift procedures in the feature space,the necessary mula [54,p. 85],[62,p. 99] is of little practical use,since condition is that local density variations near a significant it depends on the Laplacian of the unknown density mode are not as large as the entire support of a significant being estimated,and its performance is not well mode somewhere else. understood [62,p. 108]. For the univariate case,a The starting points of the mean shift procedures should reliable method for bandwidth selection is the plug-in be chosen to have the entire feature space 'except the very rule [53],which was proven to be superior to least- sparse regions) tessellated by the kernels 'windows). squares cross-validation and biased cross-validation Regular tessellations are not required. As the windows [42],[55,p. 46]. Its only assumption is the smoothness evolve toward the modes,almost all the data points are of the underlying density. visited and,thus,all the information captured in the feature . The second bandwidth selection technique is related space is exploited. Note that the convergence to a given to the stability of the decomposition. The bandwidth mode may yield slightly different locations due to the is taken as the center of the largest operating range threshold that terminates the iterations. Similarly,on flat over which the same number of clusters are obtained plateaus,the value of the gradient is close to zero and the for the given data [20,p. 541]. mean shift procedure could stop. . For the third technique,the best bandwidth max- These artifacts are easy to eliminate through postproces- imizes an objective function that expresses the quality sing. Mode candidates at a distance less than the kernel of the decomposition 'i.e.,the index of cluster bandwidth are fused,the one corresponding to the highest validity). The objective function typically compares density being chosen. The global structure of the feature the inter- versus intra-cluster variability [30],[28] or space can be confirmed by measuring the significance of the evaluates the isolation and connectivity of the valleys defined along a cut through the density in the delineated clusters [43]. direction determined by two modes. . Finally,since in most of the cases the decomposition The delineation of the clusters is a natural outcome of the is task dependent,top-down information provided mode seeking process. After convergence,the basin of by the user or by an upper-level module can be used attraction of a mode,i.e.,the data points visited by all the to control the kernel bandwidth. mean shift procedures converging to that mode,automati- cally delineates a cluster of arbitrary shape. Close to the We present in [15],a detailed analysis of the bandwidth boundaries,where a data point could have been visited by selection problem. To solve the difficulties generated by the several diverging procedures,majority logic can be em- narrow peaks and the tails of the underlying density,two ployed. It is important to notice that,in computer vision, locally adaptive solutions are proposed. One is nonpara- most often we are not dealing with an abstract clustering metric,being based on a newly defined adaptive mean shift problem. The input domain almost always provides an procedure,which exploits the plug-in rule and the sample independent test for the validity of local decisions in the point density estimator. The other is semiparametric, feature space. That is,while it is less likely that one can imposing a local structure on the data to extract reliable recover from a severe clustering error,allocation of a few scale information. We show that the local bandwidth uncertain data points can be reliably supported by input should maximize the magnitude of the normalized mean domain information. shift vector. The adaptation of the bandwidth provides The multimodal feature space analysis technique was superior results when compared to the fixed bandwidth discussed in detail in [12]. It was shown experimentally, procedure. For more details,see [15]. COMANICIU AND MEER: MEAN SHIFT: A ROBUST APPROACH TOWARD FEATURE SPACE ANALYSIS 609

Fig. 2. Example of a 2D feature space analysis.

3.2 Implementation Issues performance through a single parameter,the resolution of An efficient computation of the mean shift procedure first the analysis 'i.e.,bandwidth of the kernel). Since the control requires the resampling of the input data with a regular grid. parameter has clear physical meaning,the new algorithms This is a standard technique in the context of density can be easily integrated into systems performing more estimation which leads to a binned estimator [62,Appendix complex tasks. Furthermore,both gray level and color D]. The procedure is similar to defining a histogram where images are processed with the same algorithm,in the linear interpolation is used to compute the weights associated former case,the feature space containing two degenerate with the grid points. Further reduction in the computation dimensions that have no effect on the mean shift procedure. time is achieved by employing algorithms for multidimen- Before proceeding to develop the new algorithms,the sional range searching [52,p. 373] used to find the data points issue of the employed color space has to be settled. To obtain falling in the neighborhood of a given kernel. For the efficient a meaningful segmentation, perceived color differences Euclidean distance computation,we used the improved should correspond to Euclidean distances in the color space absolute error inequality criterion,derived in [39]. chosen to represent the features 'pixels). An Euclidean metric,however,is not guaranteed for a color space [65, Sections 6.5.2,8.4]. The spaces L*u*v* and L*a*b* were 4APPLICATIONS especially designed to best approximate perceptually uni- The feature space analysis technique introduced in the form color spaces. In both cases, LÃ,the lightness 'relative previous section is application independent and,thus,can brightness) coordinate,is defined the same way,the two be used to develop vision algorithms for a wide variety of spaces differ only through the chromaticity coordinates. The tasks. Two somewhat related applications are discussed in dependence of all three coordinates on the traditional the sequel: discontinuity preserving smoothing and image RGB color values is nonlinear. See [46,Section 3.5] for a segmentation. The versatility of the feature space analysis readily accessible source for the conversion formulae. The enables the design of algorithms in which the user controls metric of perceptually uniform color spaces is discussed in 610 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 24, NO. 5, MAY 2002 the context of feature representation for image segmentation A recently proposed noniterative discontinuity preserving in [16]. In practice,there is no clear advantage between using smoothing technique is the bilateral filtering [59]. The relation L*u*v* or L*a*b*; in the proposed algorithms,we employed between bilateral filtering and diffusion-based techniques L*u*v* motivated by a linear mapping property [65,p.166]. was analyzed in [3]. The bilateral filters also work in the joint Our first image segmentation algorithm was a straightfor- spatial-range domain. The data is independently weighted in ward application of the feature space analysis technique to an the two domains and the center pixel is computed as the L*u*v* representation of the color image [11]. The modularity weighted average of the window. The fundamental differ- ence between the bilateral filtering and the mean shift-based of the segmentation algorithm enabled its integration by other smoothing algorithm is in the use of local information. groups to a large variety of applications like image retrieval [1],face tracking [6],object-based video coding for MPEG-4 4.1.1 Mean Shift Filtering [22],shapedetectionandrecognition[33],andtextureanalysis Let xi and zi;iˆ 1; ...;n,be the d-dimensional input and [47],to mention only a few. However,since the feature space filtered image pixels in the joint spatial-range domain. For analysis can be applied unchanged to moderately higher each pixel, dimensional spaces 'see Section 5),we subsequently also 1. Initialize j ˆ 1 and y ˆ x . incorporated the spatial coordinates of a pixel into its feature i;1 i 2. Compute y according to '20) until convergence, space representation. This joint domain representation is i;j‡1 y ˆ yi;c. employed in the two algorithms described here. 3. Assign z ˆ xs; yr †. An image is typically represented as a two-dimensional i i i;c lattice of p-dimensional vectors 'pixels),where p ˆ 1 in the The superscripts s and r denote the spatial and range gray-level case,three for color images,and p>3 in the components of a vector,respectively. The assignment s multispectral case. The space of the lattice is known as the specifies that the filtered data at the spatial location xi will r spatial domain,while the gray level,color,or spectral have the range component of the point of convergence yi;c. information is represented in the range domain. For both The kernel 'window) in the mean shift procedure moves in domains,Euclidean metric is assumed. When the location the direction of the maximum increase in the joint density and range vectors are concatenated in the joint spatial-range gradient,while the bilateral filtering uses a fixed,static domain of dimension d ˆ p ‡ 2,their different nature has to window. In the image smoothed by mean shift filtering, be compensated by proper normalization. Thus,the multi- information beyond the individual windows is also taken into variate kernel is defined as the product of two radially account. symmetric kernels and the Euclidean metric allows a single An important connection between filtering in the joint domain and robust M-estimation should be mentioned. The bandwidth parameter for each domain improved performance of the generalized M-estimators 'GM ! ! or bounded-influence estimators) is due to the presence of a C xs 2 xr 2 K x†ˆ k k ; 35† secondweightfunctionwhichoffsetstheinfluenceofleverage hs;hr 2 p h h hshr s r points,i.e.,outliers in the input domain [32,Section 8E]. A similar 'at least in spirit) twofold weighting is employed in the where xs is the spatial part, xr is the range part of a feature bilateral and mean shift-based filterings,which is the main vector, k x† the common profile used in both two domains, reason for their excellent smoothing performance. h and h the employed kernel bandwidths,and C the s r Mean shift filtering with uniform kernel having h ;h †ˆ corresponding normalization constant. In practice,an s r 8; 4† has been applied to the often used 256  256 gray-level Epanechnikov or a 'truncated) normal kernel always cameraman image 'Fig. 3a),the result being shown in Fig. 3b. provides satisfactory performance,so the user only has to The regions containing the grass field have been almost set the bandwidth parameter h ˆ hs;hr†,which,by completely smoothed,while details such as the tripod and the controlling the size of the kernel,determines the resolution buildings in the background were preserved. The processing of the mode detection. required fractions of a second on a standard PC '600 Mhz 4.1 Discontinuity Preserving Smoothing Pentium III) using an optimized C++ implementation of the algorithm.Ontheaverage,3:06iterationswerenecessaryuntil Smoothing through replacing the pixel in the center of a the filtered value of a pixel was defined,i.e.,its mean shift window by the 'weighted) average of the pixels in the procedure converged. window indiscriminately blurs the image,removing not To better visualize the filtering process,the 40Â20 window only the noise but also salient information. Discontinuity marked in Fig. 3a is represented in three dimensions in Fig. 4a. preserving smoothing techniques,on the other hand, Note that the data was reflected over the horizontal axis of the adaptively reduce the amount of smoothing near abrupt window for a more informative display. In Fig. 4b,the mean changes in the local structure,i.e.,edges. shift paths associated with every other pixel 'in both There are a large variety of approaches to achieve this coordinates) from the plateau and the line are shown. Note goal,from adaptive Wiener filtering [31],to implementing that convergence points 'black dots) are situated in the center isotropic [50] and anisotropic [44] local diffusion processes, of the plateau,away from the discontinuities delineating it. a topic which recently received renewed interest [19],[37], Similarly,the mean shift trajectories on the line remain on it. [56]. The diffusion-based techniques,however,do not have As a result,the filtered data 'Fig. 4c) shows clean quasi- a straightforward stopping criterion and,after a sufficiently homogeneous regions. large number of iterations,the processed image collapses The physical interpretation of the mean shift-based into a flat surface. The connection between anisotropic filtering is easy to see by examining Fig. 4a,which,in fact, diffusion and M-estimators is analyzed in [5]. displays the three dimensions of the joint domain of a COMANICIU AND MEER: MEAN SHIFT: A ROBUST APPROACH TOWARD FEATURE SPACE ANALYSIS 611

Fig. 3. Cameraman image.

Fig. 4. Visualization of mean shift-based filtering and segmentation for gray-level data.

Fig. 5. Baboon image. Original and filtered.

The attraction force field defined in [57] is computed at 4.2.1 Mean Shift Segmentation each pixel as a vector sum of pairwise affinities between the Let x and z ;iˆ 1; ...;n,be the d-dimensional input and current pixel and all other pixels,with similarity measured i i filtered image pixels in the joint spatial-range domain and in both spatial and range domains. The region boundaries L the label of the ith pixel in the segmented image. are then identified as loci where the force vectors diverge. It i is interesting to note that,for a given pixel,the magnitude 1. Run the mean shift filtering procedure for the image and orientation of the force field are similar to those of the and store all the information about the d-dimensional joint domain mean shift vector computed at that pixel and convergence point in zi,i.e., zi ˆ yi;c. ÈÉ projected into the spatial domain. However,in contrast to 2. Delineate in the joint domain the clusters C [57],the mean shift procedure moves in the direction of this p pˆ1...m by grouping together all z which are closer than h vector,away from the boundaries. i s in the spatial domain and h in the range domain, The edge flow in [34] is obtained at each location for a r i.e.,concatenate the basins of attraction of the given set of directions as the magnitude of the gradient of a corresponding convergence points. smoothed image. The boundaries are detected at image locations which encounter two opposite directions of flow. 3. For each i ˆ 1; ...;n,assign Li ˆfp j zi 2 Cpg. The quantization of the edge flow direction,however,may 4. Optional: Eliminate spatial regions containing less introduce artifacts. Recall that the direction of the mean than M pixels. shift is dictated solely by the data. The cluster delineation step can be refined according to The mean shift procedure-based image segmentation is a a priori information and,thus,physics-based segmentation straightforward extension of the discontinuity preserving algorithms,e.g.,[2],[35],can be incorporated. Since this smoothing algorithm. Each pixel is associated with a process is performed on region adjacency graphs,hierarch- significant mode of the joint domain density located in its ical techniques like [36] can provide significant speed-up. neighborhood,after nearby modes were pruned as in the The effect of the cluster delineation step is shown in Fig. 4d. generic feature space analysis technique 'Section 3). Note the fusion into larger homogeneous regions of the COMANICIU AND MEER: MEAN SHIFT: A ROBUST APPROACH TOWARD FEATURE SPACE ANALYSIS 613

Fig. 6. MIT image.

Fig. 7. Room image.

Fig. 8. Lake image.

Fig. 9. Hand image.

17  17 spatial window,while all 512  512 images used hs ˆ The code for the discontinuity preserving smoothing and 16 corresponding to a 31  31 window. The range image segmentation algorithms integrated into a single parameter hr and the smallest significant feature size system with graphical interface is available at http:// M control the number of regions in the segmented image. www.caip.rutgers.edu/riul/research/code.html. The more an image deviates from the assumed piecewise constant model,larger values have to be used for hr and M to 5DISCUSSION discard the effect of small local variations in the feature space. For example,the heavily textured background in the hand The mean shift-based feature space analysis technique introduced in this paper is a general tool which is not image is compensated by using hr ˆ 19 and M ˆ 40,values which are much larger than those used for the room image restricted to the two applications discussed here. Since the hr ˆ 5;M ˆ 20† since the latter better obeys the model. As quality of the output is controlled only by the kernel with any low-level vision algorithm,the quality of the bandwidth,i.e.,the resolution of the analysis,the technique segmentation output can be assessed only in the context of should be also easily integrable into complex vision systems the whole vision task and,thus,the resolution parameters where the control is relinquished to a closed loop process. should be chosen according to that criterion. An important Additional insights on the bandwidth selection can be advantage of mean shift-based segmentation is its modularity obtained by testing the stability of the mean shift direction which makes the control of segmentation output very simple. across the different bandwidths,as investigated in [57] in Other segmentation examples in which the original the case of the force field. The nonparametric toolbox image has the region boundaries superposed are shown in developed in this paper is suitable for a large variety of Fig. 10 and in which the original and labeled images are computer vision tasks where parametric models are less compared in Fig. 11. adequate,for example,modeling the background in visual As a potential application of the segmentation,we return to surveillance [18]. the cameraman image. Fig. 12a shows the reconstructed image The complete solution toward autonomous image seg- after the regions corresponding to the sky and grass were mentation is to combine a bandwidth selection technique manually replaced with white. The mean shift segmentation 'like the ones discussed in Section 3.1) with top-down task- has been applied with hs;hr;M†ˆ 8; 4; 10†. Observe the related high-level information. In this case,each mean shift preservation of the details which suggests that the algorithm process is associated with a kernel best suited to the local can also be used for image editing,as shown in Fig. 12b. structure of the joint domain. Several interesting theoretical COMANICIU AND MEER: MEAN SHIFT: A ROBUST APPROACH TOWARD FEATURE SPACE ANALYSIS 615

Fig. 10. Landscape images. All the region boundaries were delineated with hs;hr;M†ˆ 8; 7; 100† and are drawn over the original image. issues have to be addressed,though,before the benefits of dimension of the space. This is mostly due to the empty space such a data driven approach can be fully exploited. We are phenomenon[20,p.70],[54,p.93]bywhichmostofthemassin currently investigating these issues. a high-dimensional space is concentrated in a small region of The ability of the mean shift procedure to be attracted by the space. Thus,whenever the feature space has more than the modes 'local maxima) of an underlying density function, 'say) six dimensions,the analysis should be approached can be exploited in an optimization framework. Cheng [7] carefully. Employing projection pursuit,in which the density already discusses a simple example. However,by introdu- is analyzed along lower dimensional cuts,e.g.,[27],is a cing adequate objective functions,the optimization problem possibility. can acquire physical meaning in the context of a computer To conclude,the mean shift procedure is a valuable vision task. For example,in [14],by defining the distance computational module whose versatility can make it an between the distributions of the model and a candidate of the important component of any computer vision toolbox. target,nonrigid objects were tracked in an image sequence under severe distortions. The distance was defined at every APPENDIX pixel in the region of interest of the new frame and the mean ProofofTheorem1.If the kernel K has a convex and shift procedure was used to find the mode of this measure monotonically decreasing profile, the sequences fyjgjˆ1;2... and ^ ^ nearest to the previous location of the target. ffh;K j†gjˆ1;2... converge, and ffh;K j†gjˆ1;2... is monotonically The above-mentioned tracking algorithm can be re- increasing. ^ garded as an example of computer vision techniques which Since n is finite,the sequence fh;K '21) is bounded, ^ are based on in situ optimization. Under this paradigm,the therefore,it is sufficient to show that fh;K is strictly solution is obtained by using the input domain to define the monotonic increasing,i.e.,if yj 6ˆ yj‡1,then optimization problem. The in situ optimization is a very ^ ^ fh;K j† < fh;K j ‡ 1†; powerful method. In [23] and [58],each input data point was associated with a local field 'voting kernel) to produce for j ˆ 1; 2.... Without loss of generality,it can be a more dense structure from where the sought information assumed that yj ˆ 0 and,thus,from '16) and '21) 'salient features,the hyperplane representing the funda- ^ ^ fh;K j ‡ 1†Àfh;K j†ˆ  mental matrix) can be reliably extracted. Xn c y À xi 2 x 2 A:1† The mean shift procedure is not computationally expen- k;d k j‡1 À k i : nhd h h sive. Careful C++ implementation of the tracking algorithm iˆ1 allowed real time '30 frames/second) processing of the video The convexity of the profile k x† implies that stream. While it is not clear if the segmentation algorithm k x †k x †‡k0 x † x À x † A:2† describedinthispapercanbemadesofast,giventhequalityof 2 1 1 2 1 0 the region boundaries it provides,it can be used to support for all x1;x2 2‰0; 1†, x1 6ˆ x2,and since g x†ˆÀk x†, edge detection without significant overhead in time. 'A.2) becomes Kernel density estimation,in particular,and nonpara- k x †Àk x †g x † x À x †: A:3† metric techniques,in general,do not scale well with the 2 1 1 1 2 616 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 24, NO. 5, MAY 2002

Fig. 11. Some other segmentation examples with hs;hr;M†ˆ 8; 7; 20†. Left: original. Right: segmented.

Now,using 'A.1) and 'A.3),we obtain and,recalling '20),yields  ^ ^ Xn 2 fh;K j ‡ 1†Àfh;K j† ^ ^ ck;d 2 xi  fh;K j ‡ 1†Àfh;K j† kyj‡1k g : A:5† Xn hi nhd‡2 h c x 2 iˆ1  k;d g i kx k2 Àky À x k2 d‡2 i j‡1 i nh h The profile k x† being monotonically decreasing for all iˆ1 n  hi Pn 2 c X x 2 x  0,the sum g xi † is strictly positive. Thus,as ˆ k;d g i 2y> x Àky k2 iˆ1 h nhd‡2 h j‡1 i j‡1 iˆ1 long as yj‡1 6ˆ yj ˆ 0,the right term of 'A.5) is strictly "#  ^ ^ Xn 2 Xn 2 positive,i.e., fh;K j ‡ 1† > fh;K j†. Consequently,the ck;d > xi 2 xi ˆ 2y xig Àkyj‡1k g sequence ff^ j†g is convergent. nhd‡2 j‡1 h h h;K jˆ1;2... iˆ1 iˆ1 To prove the convergence of the sequence fyjgjˆ1;2..., A:4† 'A.5) is rewritten for an arbitrary kernel location yj 6ˆ 0. After some algebra,we have COMANICIU AND MEER: MEAN SHIFT: A ROBUST APPROACH TOWARD FEATURE SPACE ANALYSIS 617

Fig. 12. Cameraman image. x exp À j‡1 < 0: B:2† h;K h;K nhd‡2 j‡1 j h j‡1 j‡1 i h iˆ1 iˆ1 A:6† The space Rd can be decomposed into the following three domains: Now,summing the two terms of 'A.6) for indices  j; j ‡ 1...j ‡ m À 1,it results that 1 D ˆ x 2 Rd y> x  ky k2 1 j‡1 2 j‡1 ^ ^  fh;K j ‡ m†Àfh;K j† 1  d 2 > 2 B:3† Xn 2 D2 ˆ x 2 R kyj‡1k < yj‡1x kyj‡1k ck;d 2 yj‡mÀ1 À xi 2  ky À y k g ‡ ... no nhd‡2 j‡m j‡mÀ1 h iˆ1 D ˆ x 2 Rd ky k2 < y> x  3 j‡1 j‡1 Xn c y À xi 2 ‡ k;d ky À y k2 g j nhd‡2 j‡1 j h and after some simple manipulations from 'B.1),we can  iˆ1  derive the equality c k;d 2 2   kyj‡m À yj‡mÀ1k ‡ ...‡kyj‡1 À yjk M X 2 nhd‡2 2 > xi kyj‡1k À yj‡1xi exp À ck;d 2 h  ky À y k M; xi2D2 d‡2 j‡m j  B:4† nh X x 2 ˆ y> x Àky k2 exp À i : A:7† j‡1 i j‡1 h xi2D1[D3 where M represents the minimum 'always strictly 2 > In addition,for x 2 D2,we have kyj‡1k À yj‡1x  0, P 2 n yjÀxi which implies positive) of the sum iˆ1 g h † for all fyjgjˆ1;2.... ^ Since ffh;K j†gjˆ1;2... is convergent,it is also a Cauchy 2 2 2 > 2 2 kyj‡1 À xik ˆkyj‡1k ‡kxik À 2y xi kxik Àkyj‡1k sequence. This property in conjunction with 'A.7) implies j‡1 B:5† that fyjgjˆ1;2... is a Cauchy sequence,hence,it is con- vergent in the Euclidean space. tu from where Proof of Theorem 2. The cosine of the angle between two X  y À xi 2 ky k2 À y> x exp À j‡1 consecutive mean shift vectors is strictly positive when a j‡1 j‡1 i h xi2D2   normal kernel is employed. 2 X  2 yj‡1 2 > xi  exp kyj‡1k À yj‡1xi exp À : We can assume,without loss of generality that yj ˆ 0 and h h xi2D2 yj‡1 6ˆ yj‡2 6ˆ 0 since,otherwise,convergence has already B:6† been achieved. Therefore,the mean shift vector m 0† is h;N Now,introducing 'B.4) in 'B.6),we have  Pn 2  x exp À xi X  2 iˆ1 i h 2 > yj‡1 À xi m 0†ˆy ˆ : B:1† kyj‡1k À yj‡1xi exp À h;N j‡1 P 2 n xi h exp À xi2D2 iˆ1 h   y 2 X x 2  exp j‡1 y> x Àky k2 exp À i We will show first that,when the weights are given by a h j‡1 i j‡1 h normal kernel centered at y ,the weighted sum of the xi2D1[D3 ÀÁj‡1 B:7† projections of yj‡1 À xi onto yj‡1 is strictly negative,i.e., 618 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 24, NO. 5, MAY 2002

and,by adding to both sides of 'B.7),the quantity [4] D.P. Bertsekas, Nonlinear Programming. Athena Scientific,1995. [5] M.J. Black,G. Sapiro,D.H. Marimont,and D. Heeger,ªRobust X  Anisotropic Diffusion,º IEEE Trans. Image Processing, vol. 7,pp. 421- y À xi 2 ky k2 À y> x exp À j‡1 ; 432,1998. j‡1 j‡1 i h xi2D1[D3 [6] G.R. Bradski,ªComputer Vision Face Tracking as a Component of a Perceptual User Interface,º Proc. IEEE Workshop Applications of after some algebra,it results that Computer Vision, pp. 214-219,Oct. 1998.  [7] Y. Cheng,ªMean Shift,Mode Seeking,and Clustering,º IEEE Xn  2 Trans. Pattern Analysis and Machine Intelligence, vol. 17,no. 8, 2 > yj‡1 À xi kyj‡1k À yj‡1xi exp À pp. 790-799,Aug. 1995. h [8] E. Choi and P. Hall,ªData Sharpening as a Prelude to Density iˆ1   2 X  2 Estimation,º Biometrika, vol. 86,pp. 941-947,1999. yj‡1 2 > xi  exp kyj‡1k À yj‡1xi exp À [9] C.K. Chu,I.K. Glad,F. Godtliebsen,and J.S. Maron,ªEdge- h h Preserving Smoothers for Image Processing,º J. Am. Statistical xi2D1[D3  Assoc., vol. 93,pp. 526-541,1998. 2  2 > [10] D. Comaniciu,ªNonparametric Robust Methods for Computer  exp À ky k À y xi À 1 : h2 j‡1 j‡1 Vision,º PhD thesis, Dept. of Electrical and Computer Eng., Rutgers Univ.,1999. Available at http://www.caip.rutgers.edu/ B:8† riul/research/theses.html. The right side of 'B.8) is negative because [11] D. Comaniciu and P. Meer,ªRobust Analysis of Feature Spaces: Color Image Segmentation,º Proc. 1997 IEEE Conf. Computer Vision  and Pattern Recognition, pp. 750-755,June 1997. 2 > kyj‡1k À yj‡1xi [12] D. Comaniciu and P. Meer,ªDistribution Free Decomposition of Multivariate Data,º Pattern Analysis and Applications, vol. 2,pp. 22- and the last product term has opposite signs in both the 30,1999. D and D domains. Therefore,the left side of 'B.8) is [13] D. Comaniciu and P. Meer,ªMean Shift Analysis and Applica- 1 3 tions,º Proc. Seventh Int'l Conf. Computer Vision, pp. 1197-1203,Sept. also negative,which proves 'B.2). 1999. We can use now 'B.2) to write [14] D. Comaniciu,V. Ramesh,and P. Meer,ªReal-Time Tracking of  Non-Rigid Objects Using Mean Shift,º Proc. 2000 IEEE Conf. P y Àx 2 Computer Vision and Pattern Recognition, vol. II,pp. 142-149,June n x exp À j‡1 i iˆ1 i h 2000. ky k2 < y> ˆ y> y ; B:9† j‡1 j‡1 P 2 j‡1 j‡2 [15] D. Comaniciu,V. Ramesh,and P. Meer,ªThe Variable Bandwidth n yj‡1Àxi iˆ1 exp À h Mean Shift and Data-Driven Scale Selection,º Proc Eighth Int'l Conf. Computer Vision, vol. I,pp. 438-445,July 2001. from where [16] C. Connolly,ªThe Relationship between Colour Metrics and the ÀÁ Appearance of Three-Dimensional Coloured Objects,º Color > Research and Applications, vol. 21,pp. 331-337,1996. yj‡1 yj‡2 À yj‡1 > 0 B:10† [17] R.O. Duda and P.E. Hart, Pattern Classification and Scene Analysis. kyj‡1kkyj‡2 À yj‡1k Wiley,1973. [18] A. Elgammal,D. Harwood,and L. Davis,ªNon-Parametric Model or by taking into account '24) for Background Subtraction,º Proc. Sixth European Conf. Computer Vision, vol. II,pp. 751-767,June 2000. > m y † m y † [19] B. Fischl and E.L. Schwartz,ªAdaptive Nonlocal Filtering: A Fast h;N j h;N j‡1 > 0: km y †kkm y †k Alternative to Anisotropic Diffusion for Image Enhancement,º h;N j h;N j‡1 IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 21,no. 1, tu pp. 42-48,Jan. 1999. [20] K. Fukunaga, Introduction to Statistical Pattern Recognition, second ed. Academic Press,1990. [21] K. Fukunaga and L.D. Hostetler,ªThe Estimation of the Gradient of a Density Function,with Applications in Pattern Recognition,º ACKNOWLEDGMENTS IEEE Trans. Information Theory, vol. 21,pp. 32-40,1975. [22] J. Guo,J. Kim,and C. Kuo,ªFast and Accurate Moving Object The support of the US National Science Foundation under Extraction Technique for MPEG-4 Object Based Video Coding,º grants IRI 95-30546 and IRI 99-87695 is gratefully Proc. SPIE Visual Comm. and Image Processing, vol. 3653,pp. 1210- acknowledged. Preliminary versions for some parts of the 1221,1999. [23] G. Guy and G. Medioni,ªInference of Surfaces,3D Curves,and material were presented in [13] and [14]. The authors would Junctions from Sparse,Noisy,3D Data,º IEEE Trans. Pattern like to thank John Kent from the University of Leeds and Analysis and Machine Intelligence, vol. 19,pp. 1265-1277,1997. David Tyler of Rutgers for discussions about the relation [24] W. HaÈrdle, Applied Nonparameteric Regression. Cambridge Univ. Press,1991. between the mean shift procedure and M-estimators. [25] M. Herbin,N. Bonnet,and P. Vautrot,ªA Clustering Method Based on the Estimation of the Probability Density Function and on the Skeleton by Influence Zones,º Pattern Recognition Letters, REFERENCES vol. 17,pp. 1141-1150,1996. [1] G. Aggarwal,S. Ghosal,and P. Dubey,ªEfficient Query [26] P.J. Huber, Robust Statistical Procedures, second ed. SIAM,1996. Modification for Image Retrieval,º Proc. 2000 IEEE Conf. [27] J.N. Hwang and S.R. Lay,and A. Lippman,ªNonparametric Computer Vision and Pattern Recognition, vol. II,pp. 255-261, Multivariate Density Estimation: A Comparative Study,º IEEE June 2000. Trans. Signal Processing, vol. 42,pp. 2795-2810,1994. [2] R. Bajcsy,S.W. Lee,and A. Leonardis,ªDetection of Diffuse and [28] A.K. Jain and R.C. Dubes, Algorithms for Clustering Data. Prentice Specular Interface Reflections and Inter-Reflections by Color Hall,1988. Image Segmentation,º Int'l J. Computer Vision, vol. 17,pp. 241- [29] A.K. Jain,R.P.W. Duin,and J. Mao,ªStatistical Pattern Recogni- 272,1996. tion: A Review,º IEEE Trans. Pattern Analysis and Machine [3] D. Barash,ªBilateral Filtering and Anisotropic Diffusion: Towards a Intelligence, vol. 22,no. 1,pp. 4-37,Jan. 2000. Unified Viewpoint,º IEEE Trans. Pattern Analysis and Pattern [30] L. Kauffman and P. Rousseeuw, Finding Groups in Data: An Analysis, to appear. Introduction to . J. Wiley & Sons,1990. COMANICIU AND MEER: MEAN SHIFT: A ROBUST APPROACH TOWARD FEATURE SPACE ANALYSIS 619

[31] D.T. Kuan,A.A. Sawchuk,T.C. Strand,and P. Chavel,ªAdaptive [60] A. Touzani and J.G. Postaire,ªClustering by Mode Boundary Noise Smoothing Filter for Images with Signal Dependent Noise,º Detection,º Pattern Recognition Letters, vol. 9,pp. 1-12,1989. IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 7,no. 2, [61] T. Tuytelaars,L. Van Gool,M. Proesmans,and T. Moons,ªThe pp. 165-177,Mar. 1985. Cascaded Hough Transform as an Aid in Aerial Image Interpreta- [32] G. Li,ªRobust Regression,º Exploring Data Tables, Trends, and Shapes, tion,º Proc. Sixth Int'l Conf. Computer Vision, pp. 67-72,Jan. 1998. D.C. Hoaglin,F. Mosteller,and J.W. Tukey,eds.,pp. 281-343. Wiley, [62] M.P. Wand and M. Jones, Kernel Smoothing. Chapman and Hall, 1985. 1995. [33] L. Liu and S. Sclaroff,ªDeformable Shape Detection and [63] R. Wilson and M. Spann,ªA New Approach to Clustering,º Description via Model-Based Region Grouping,º Proc. 1999 IEEE Pattern Recognition, vol. 23,pp. 1413-1425,1990. Conf. Computer Vision and Pattern Recognition, vol. II,pp. 21-27, [64] C. Wren,A. Azarbayejani,T. Darrell,and A. Pentland,ªPfinder: June 1999. Real-Time Tracking of the Human Body,º IEEE Trans. Pattern [34] W.Y. Ma and B.S. Manjunath,ªEdge flow: A Framework of Analysis and Machine Intelligence, vol. 19,no. 7,pp. 780-785,July 1997. Boundary Detection and Image Segmentation,º IEEE Trans. Image [65] G. Wyszecki and W.S. Stiles, Color Science: Concepts and Methods, Processing, vol. 9,pp. 1375-1388,2000. Quantitative Data and Formulae. second ed. Wiley,1982. [35] B.A. Maxwell and S.S. Shafer,ªSegmentation and Interpretation of [66] S.C. Zhu and A. Yuille,ªRegion Competition: Unifying Snakes, Multicolored Objects with Highlights,º Computer Vision and Image Region Growing,and Bayes/MDL for Multiband Image Segmen- Understanding, vol. 77,pp. 1-24,2000. tation,º IEEE Trans. Pattern Analysis and Machine Intelligence, [36] A. Montanvert,P. Meer,and A. Rosenfeld,ªHierarchical Image vol. 18,no. 9,pp. 884-900,Sept. 1996. Analysis Using Irregular Tessellation,º IEEE Trans. Pattern Analysis [67] X. Zhuang,Y. Huang,K. Palaniappan,and Y. Zhao,ªGaussian and Machine Intelligence, vol. 13,no. 4,pp. 307-316,Apr. 1991. Mixture Density Modeling Decomposition,and Applications,º [37] J. Monteil and A. Beghdadi,ªA New Interpretation and IEEE Trans. Image Processing, vol. 5,pp. 1293-1302,1996. Improvement of Nonlinear Anisotropic Diffusion for Image Enhancement,º IEEE Trans. Pattern Analysis and Machine Intelli- Dorin Comaniciu received the Dipl. Engn. and gence, vol. 21,no. 9,pp. 940-946,Sept. 1999. PhD degrees in electronics from the Polytechnic [38] Y. Ohta,T. Kanade,and T. Sakai,ªColor Information for Region University of Bucharest in 1988 and 1995 and the Segmentation,º Computer Graphics and Image Processing, vol. 13, PhD degree in electrical engineering from Rut- pp. 222-241,1980. gers University in 1999. From 1988 to 1990, he [39] J. Pan,F. McInnes,and M. Jack,ªFast Clustering Algorithms for was with ICE Felix Computers in Bucharest. In Vector Quantization,º Pattern Recognition, vol. 29,pp. 511-518,1996. 1991, he joined the Department of Electronics [40] T.N. Pappas,ªAn Adaptive Clustering Algorithm for Image and Telecommunications at the Polytechnic Segmentation,º IEEE Trans. Signal Processing, vol. 40,pp. 901-914, University of Bucharest and he held research 1992. appointments in Germany and France. From [41] N. Paragios and R. Deriche,ªGeodesic Active Contours for 1996 to 1999, he was with the Center for Advanced Information Supervised Texture Segmentation,º Proc. 1999 IEEE Conf. Compu- Processingassociated with RutgersUniversity. Since 1999, he has been ter Vision and Pattern Recognition, vol. II,pp. 422-427,June 1999. a member of the technical staff at Siemens Corporate Research in [42] B. Park and J. Marron,ªComparison of Data-Driven Bandwidth Princeton, New Jersey. His research interests include robust methods for Selectors,º J. Am. Statistical Assoc., vol. 85,pp. 66-72,1990. autonomous computer vision, nonparametric analysis, real-time vision [43] E.J. Pauwels and G. Frederix,ªFinding Salient Regions in Images,º systems, video surveillance, content-based access to visual data, and Computer Vision and Image Understanding, vol. 75,pp. 73-85,1999. data compression. He has coauthored numerous papers, conference [44] P. Perona and J. Malik,ªScale-Space and Edge Detection Using papers, and book chapters in the area of visual information processing. Anisotropic Diffusion,º IEEE Trans. Pattern Analysis and Machine He received the Best Paper Award at the IEEE Conference Computer Intelligence, vol. 12,no. 7,pp. 629-639,July 1990. Vision and Pattern Recognition 2000. He is a member of the IEEE. [45] K. Popat and R.W. Picard,ªCluster-Based Probability Model and Its Application to Image and Texture Processing,º IEEE Trans. Peter Meer received the Dipl. Engn. degree Image Processing, vol. 6,pp. 268-284,1997. from the Bucharest Polytechnic Institute, Bu- [46] W.K. Pratt, , second ed. Wiley,1991. charest, Romania, in 1971, and the DSc degree [47] D. Ridder,J. Kittler,O. Lemmers,and R. Duin,ªThe Adaptive from the Technion, Israel Institute of Technol- Subspace Map for Texture Segmentation,º Proc. 2000 Int'l Conf. ogy, Haifa, Israel, in 1986, both in electrical Pattern Recognition, pp. 216-220,Sept. 2000. engineering. From 1971 to 1979, he was with the [48] T. Risse,ªHough Transform for Line Recognition: Complexity of Computer Research Institute, Cluj, Romania, Evidence Accumulation and Cluster Detection,º Computer Vision workingon research and development of digital Graphics and Image Processing, vol. 46,pp. 327-345,1989. hardware. From 1986 to 1990, he was an [49] S.J. Roberts,ªParametric and Non-Parametric Unsupervised assistant research scientist at the Center for Cluster Analysis,º Pattern Recognition, vol. 30,pp. 261-272,1997. Automation Research, University of Maryland at College Park. In 1991, [50] P. Saint-Marc and J.S. Chen,G. Medioni,ªAdaptive Smoothing: A he joined the Department of Electrical and Computer Engineering, General Tool for Early Vision,º IEEE Trans. Pattern Analysis and Rutgers University, Piscataway, New Jersey, where he is currently an Machine Intelligence, vol. 13,no. 6,pp. 514-529,June 1991. associate professor. He has held visitingappointments in Japan, Korea, [51] D.W. Scott, Multivariate Density Estimation. Wiley,1992. Sweden, Israel and France and was on the organizing committees of numerous International workshops and conferences. He is an associate [52] R. Sedgewick, Algorithms in C++. Addison-Wesley,1992. editor of the IEEE Transaction on Pattern Analysis and Machine [53] S. Sheather,M. Jones,ªA Reliable Data-Based Bandwidth Intelligence, a member of the editorial board of Pattern Recognition, and Selection Method for Kernel Density Estimation,º J. Royal Statistics he was a guest editor of Computer Vision and Image Understanding for Soc. B, vol. 53,pp. 683-690,1991. the April 2000 special issue on ªRobust Statistical Techniques in Image [54] B. W. Silverman, Density Estimation for Statistics and Data Analysis. Understanding.º He is coauthor of an award winning paper in Pattern Chapman and Hall,1986. Recognition in 1989, the best student paper in 1999, and the best paper [55] J. Simonoff, Smoothing Methods in Statistics. Springer-Verlag,1996. in the 2000 IEEE Conference Computer Vision and Pattern Recognition. [56] Special Issue on Partial Differential Equations and Geometry- His research interest is in application of modern statistical methods to Driven Diffusion in Image Processing and Analysis, IEEE Trans. image understanding problems. He is a senior member of the IEEE. Image Processing, vol. 7,Mar. 1998. [57] M. Tabb and N. Ahuja,ªMultiscale Image Segmentation by Integrated Edge and Region Detection,º IEEE Trans. Image Processing, vol. 6,pp. 642-655,1997. [58] C.K. Tang,G. Medioni,and M.S. Lee,ªEpipolar Geometry Estimation by Tensor Voting in 8D,º Proc. Seventh Int'l Conf. . For more information on this or any other computing topic, Computer Vision, vol. I,pp. 502-509,Sept. 1999. please visit our Digital Library at http://computer.org/publications/dlib. [59] C. Tomasi and R. Manduchi,ªBilateral Filtering for Gray and Color Images,º Proc. Sixth Int'l Conf. Computer Vision, pp. 839-846, Jan. 1998.