Scale-Space Color

Ekaterina V. Semeikina, Dmitry V. Yurin Department of Computational Mathematics and Cybernetics Moscow State University, Moscow, Russia [email protected], [email protected]

dependent on second image derivatives. In this paper two variants Abstract of color blob detection are proposed. In our method sizes of blobs are defined adaptively in scale-space. We know the only work [9] Feature detection in color images frequently consists in image on color blob detection, comparative discussion of the proposed conversion from color to grayscale and then application of one of method and [9] will be given. Also comparison with detection in many known grayscale detectors. This approach has a few disad- converted to grayscale images is presented. vantages: some features become indistinguishable in grayscale and features ordering based on grayscale detector response do not 2. SCALE-SPACE accord with features order of importance from human’s percep- tion point of view. In this paper the method for direct detection of Blobs in image can be described using derivatives of an image blobs in color images is proposed. The proposed algorithm is brightness function. However, the input image  , yxI  is given at based on approach and estimates blobs sizes. Two modifications of the proposed method are given and compared. discrete pixel mesh in the plane , yx . Scale-space theory [7] proposes to use instead discrete image , yxI its version Keywords: color blob detection, feature points, scale-space, hes-  sian matrix.  ,, tyxL  blurred with the Gaussian kernel ,, tyxG :   yxItyxGtyxL ),(),,(),,( , 1. INTRODUCTION 1 222 where tyxG ),,(  e  yx 2 , t  2 (1) Point and linear feature detection is a base problem of image mo- 2  2 saicing, image registration, 3D recovery, pattern recognition, and scene analysis. It has been shown [8] that blobs are the most ap- Blurred function  ,, tyxL  is infinitely differentiable. Using propriate point features for the applications which need feature convolution properties the derivatives of  ,, tyxL  can be cal- matching. In contrast to corners [4], blobs [8] have more stable culated via convolution of  , yxI  with the corresponding deriva- location and size. The algorithm for color blob detection will be proposed in the paper. tives of Gauss function. Thus image derivatives depend on the 2 Feature detection in color images often consists of conversion blurring parameter t  called scale [7]. from color to grayscale mode and application of one of grayscale In blob detection there is no a priory known scale, which should detectors. This approach has disadvantages described below. be used for derivatives calculation. A blob should be detected at Most of feature detectors [1, 4, 11] consist of three steps. The first the scale where it is visible better. Such scale is proportional to of them is application of some transform to image in order to the blob size. This leads to consideration of one-dimensional fam- construct Feature Response Image (FRI). Typical examples of ily of images  ,, tyxL  blurred with different t . FRI are gradient absolute value image (for Canny Let us notice that since Gauss function is used as the blurring [1]), (grayscale blob detection [8]), Harris kernel, then function  ,, tyxL  satisfies diffusion equation [7]: functional ( [4]). The second is extrema detection (or non maxima suppression [1]) in FRI. The third step is thresh- 1 olding or hysteresis [1]: sufficiently large extrema are considered t   LLL yyxx . (2) to be features. Using of thresholds gives rise to the first argument 2 for color image analysis without conversion to grayscale: a possi- bility of feature distinguishability reduction. Equal brightness of a 3. GRAYSCALE BLOB DETECTION AT A FIXED feature and background is a rare situation, but visibility decreas- SCALE ing of some features after conversion to grayscale is typical. Let us consider Taylor series (3) of image brightness at the point For a wide class of algorithm correct feature sorting in order of  , yx  in order to explain popular blob [8] detection techniques their importance is significant. [5] can be considered as an exam- ples of the algorithms which estimate model parameters (homo- basics. graphy or essential matrix) using matched pairs of features from T different views. Estimation is fulfilled in two stages: using pairs T dx 1 dx dx 0  g yxtyxdL ),(,,      H, yx   ..., of “most important” features (with high FRI value) and refine- dy 2dy dy ment with all feature pairs. If feature ordering is not stable, sets of  LL  (3) most important features from different views can contain images where HH ,,, tyxyx  xyxx  is Hessian ma- 0   of different points of 3D scene and matching will be incorrect.  LL yyxy  The corner [10] and edge [2, 3] detectors for color images are trix, scale  tt 0 is fixed. known. Unfortunately such approach is not applicable to features In the middle of a blob first derivatives (gradient) are small and  T 1  the second term becomes the main. Image brightness  ,, tyxL   cccc BGR  CC (6) 0 3 defines the surface and eigenvalues 1 , 2 (   21 |||| ) of Hes- In order to form Hessian matrix, derivatives of the adaptively sian matrix characterizes curvature of this surface in direction of projected image should be constructed. Derivatives calculation in eigenvectors v1 , v2 . If both curvature values have the same sign the point  , yx ii  via convolution with Gaussian derivatives would and similar magnitudes then the feature is blob. Thus grayscale require forming of separate image Li , using the fixed coefficients blob detection procedure consists of FRI construction   , yx   2  , yxc ii  in all points of Li . and local extrema detection.  ,      ,, yxRyxcyxL  In order to avoid square root calculation Laplacian (4) is fre- i iiR . (7) quently used as FRI for blob detection:  iiG  ,,  iiB  ,, yxByxcyxGyxc

Fortunately we can avoid building of this image set Li and cal- trace H LL yyxx   21 . (4) culate derivatives in all points directly from color components Another popular FRI for blob detection is determinant of Hessian derivatives: matrix det H . ˆ   ,   ,  ˆ   , yxRDyxcyxLD   R , 4. COLOR BLOB DETECTION AT A FIXED SCALE ˆ ˆ G ,   , B ,   , yxBDyxcyxGDyxc (8) ˆ 4.1 Color variation vector and Hessian matrix for where D is differentiation operator. color image Since only adaptive image derivatives are needed for blob detec- Feature detection using Hessian matrix eigenvalues cannot be tion, the proposed method does not contain image conversion directly applied to color images. Usually in order to solve this from color to grayscale. problem grayscale image is constructed as a projection of color 4.2 Detection at a fixed scale image to some direction in color space, for example, 114.0,587.0,299.0 . In this work we propose to select color Hessian matrix can be formed using derivatives (8). Further blob projection direction adaptively in every image point as a direction detection is analogous to detection in grayscale images using in color space of the fastest color change. eigenvalues 1 , 2 (   21 ) of Hesse matrix. In order to introduce vector of the fastest color change let us ap- Let us chose threshold  characterizing maximum allowed ob- ply Laplace operator to each of image color channels and let us  longness of a blob. Local extrema of FRI 2 yx ),( , where introduce auxiliary vector C : 12   01 , are detected as blobs. These local extrema can  1 T  ,  ,  BBGGRRC yyxxyyxxyyxx . (5) be detected via scanning by 33 frame and comparing central 3 pixel on FRI with its 8 neighbour pixels. In our experiments Application of the Laplace operator, i.e. convolution with second   8 have been used. derivatives of Gauss function, means difference between weighted  1 In the previous work [6] 2 yx ),( was used as FRI and local mean  ,, BGRC T over the point neighbourhood of 3  maxima needed to be detected. Detection of maxima instead of maxima and minima slightly saves computation time but leads to  1 T radius r and weighted mean  ,, BGRC over artifacts. 0 3  4.3 “Continuity” and “compatibility” properties outer ring neighbourhood (Figure 1). Here 0   Nr G , where Like classical color extensions of corner [10] and edge [2, 3] de- N is the dimensionality of used Gaussian function, in our case G tectors the proposed method obeys an important “continuity”    NG  2 . So we can say that  CCC  is a color variation property: small changes in RGB channel values results in small vector in the feature neighbourhood. FRI changes. It also obeys property of “compatibility” with gray- scale detector: when image color components are equal:

  yxByxGyxR ),(),(),( , (9) formulae used to construct FRI of color image come to formulae used for grayscale images. Thus the results of detection using color and grayscale algorithms are the same when (9) is true.

5. TWO VARIANTS OF SCALE-SPACE BLOB DETECTION Figure 1: Second derivative of the 1D Gauss function. Unlike edge detection, where incorrect scale selection frequently  leads only to some change of the edges shape, blobs will be Direction of C is a vector of the fastest color change (6): missed if incorrect scale is used for detection. In the current sec- tion the method for scale-space detection which allows to detect H ,, tyx . Each diagram in the right part of Figure 3 contains feature at the scale where it is better visible is proposed. the same row of FRI taken at sequential scales. Rows ycc and yec For blob detection in scale-space we construct Hessian matrices of FRIs are shown, where circle blob in the original image has (3) for a set of sequential scales t using derivatives (8). Then we j center  , yx cccc  and elliptic blob has center , yx ecec . It can be construct a set of FRIs 2  ,, tyx j  and detect local extrema seen that in the middle of circle or elliptic blob there is evident via scanning with window  333 . extremum of 2  ,, tyx  and both eigenvalues 1 , 2 have close values. This scale-space blob detection method can be modified if simul- taneous blob and ridge detection is needed. Ridge detection [6] 6. COMPARISON uses Hessian matrix H,, tyx , characterizing brightness curva- ture in 3D space ,, tyx : Adaptive projection (8) preserves features distinguishability to- wards background unlike methods using a fixed direction or the  LLL   xtxyxx  adaptive method [9] proposed by Ming and Ma (11): H tyx ),,(   LLL ytyyxy   diffusion equation )2(    ˆ  , yxR  ˆ  LLL  , yxLD  , yxRD   ttytxt  , yxS ,   LL xyyxxx  , yxG ˆ , yxB ˆ  Lxx Lxy  (10)  , yxGD  , yxBD (11)  2  , yxS , yxS   LL yyyxxy   L L  xy yy  where     ,,,, yxByxGyxRyxS  2  .   xyyxxx  yyyxxy xxxx 2 xxyy  LLLLLLL yyyy   2 2 4  Unlike the proposed algorithm method [9] take into consideration   only the color in current point and do not use background color. Using of second eigenvalue of Hessian matrix H,, tyx (10) as Let us consider an example blob of color 0,0,255 against a FRI gives result equivalent to result of detection with H , yx  background of color  5,0,255 . In formula (11) red component, (3): sets of detected local extrema are almost the same (see exam- which does not distinguish feature and background, has the high- ple in Figure 2) and contrasts of the corresponding extrema are est contribution while the most important blue component is sup- close. To compare extrema contrast two values have been calcu- pressed, so projection direction is 02.0,0,98.0 . In our method lated: in case of local maxima c1  mean  max and (8) a component has the higher weight the higher change between feature and background for this component. The color variation c2  pre  maxmax , where max is detected maximum value, vector is proportional to  1,0,0  for the above example.  premax is maximum value among   333 - neigbourhood of max Comparative examples of blob detection with the proposed algo- of size  333  is mean value among  . Analogi- , mean  333 rithm and detection after conversion to grayscale are given in cally in case of local minima: c1  mean  min and Figure 4. It can be seen that after conversion to grayscale most noticeable from the human perception point of view color features c2 pre  minmin . Mean values over the test base, containing syn- are missed. At the same time these features have been detected by thetic and natural images, are c1  3.2 and c2  02.1 . the proposed method with high response.

7. CONCLUSION

Two variants of scale-space algorithm for blob detection in color images have been developed. Up today the color detectors of features dependent on first derivatives (edges and corners) were known. In this paper we have proposed an other approach to using color information. In contrast to previous approach our method can be applied to features dependent on second derivatives, par- ticularly blobs. It has been showed that the proposed method has a) b) advantages in comparison with the only previously known color blob detector and with detection after conversion to grayscale. c) Figure 2: The blobs detected (response threshold 1.25): a) using 8. ACKNOWLEDGEMENTS H , yx , b) using H ,, tyx , c) color of ellipses correspond to   The work was supported by the Federal Program “Scientific and blob response. Scientific Pedagogical Personnel of Innovative Russia” in 2009– 2013 and the Russian Foundation for Basic Research, project no. 09-07-92000-HHC_a. Figure 3 illustrates eigenvalues of H, yx and H,, tyx at a set of scales. For a figure in the left part four FRI sets have been calculated: 1,, tyx and 2 ,, tyx of H , yx  and Figure 3: Eigenvalues of H , yx  and H,, tyx at a set of scales. For a figure in the left part four FRI sets have been calculated: 1 ,, tyx 

and 2 ,, tyx of H , yx  and H ,, tyx . Each diagram in the right part contains the same row of FRI taken at sequential scales. Shown rows are taken in the middle of circle and elliptic blobs. Values in diagrams are normalized to [0, 255].

Image Processing, 1991, V. 53, No. 1, P. 40 - 51. [3] S. Di Zenzo. A Note on the Gradient of Multi-Image. Comp. Vision Graphics Image Processing, 1986, V. 33, P. 116 - 125. [4] C. Harris, M. Stephens. A combined corner and edge detec- tor. Proc. 4th Alvey Vision Conf., V. 15, 1988, P. 147–151. [5] R. Hartley, A. Zisserman. Multiple View Geometry in Com- puter Vision. Cambridge University Press, 2004, P. 672.

a) b) [6] N. Khanina, E. Semeikina, D. Yurin. Color Blob and Ridge Detection. Proc. of 10th Conf. on Pattern Recognition and Image Analysis: New Inform. Technologies, 2010, V. 1, P. 289−292. [7] T. Lindeberg. Scale–Space Theory in . Kluwer Academic Publishers, Dordrecht, 1994. [8] D. G. Lowe. Distinctive image features from scale-invariant keypoints. IJCV 2004, V. 60, No 2, P. 91 - 110. [9] A. Ming, H. Ma. A Blob Detector in Color Images. Proc. of the 6th ACM CIVR, 2007, P. 364 - 370. c) d) [10] P. Montesinos, V. Gouet, R. Deriche. Differential Invariants for Color Images. Proc. of 14th ICPR, 1998, P.838–840. [11] J. Shi, C. Tomasi. Good features to track. Proceedings of the IEEE CVPR'94, 1994, P. 593 - 600.

About the authors

Ekaterina V. Semeikina is a Ph.D. student at Chair e) f) of Mathematical Physics, Faculty of Computational Figure 4: Comparison of results obtained from the proposed algo- Mathematics and Cybernetics, Lomonosov Moscow rithm ((c) and (d)) and detection after conversion to grayscale ((e) State University, Russia. and (f)). Response threshold is 0.75. After conversion to grayscale Her contact email is [email protected] most noticeable color features are missed.

9. REFERENCES Dmitry V. Yurin (PhD) is a senior researcher at laboratory of Mathematical Methods of Image Proc- [1] J. Canny. A computational approach to edge detection. IEEE essing, Faculty of Computational Mathematics and Trans. PAMI, 1986, V. 8, P. 34 - 43. Cybernetics, Lomonosov Moscow State University, [2] A. Cumani. Edge Detection in Multispectral Images. Com- Russia. His contact email is [email protected] puter Vision, Graphics and Image Processing: Graphical Models