Tensor Based Feature Detection for Color Images J. van de Weijer and Th. Gevers Intelligent Sensory Information Systems ,University of Amsterdam, The Netherlands Abstract tion estimator for orientated patterns (e.g. fingerprint im- ages). Oriented patterns differ from non-oriented ones in Extending differential-based operations to color images is that they consist of ridges which have a differential struc- hindered by the multi-channel nature of color images. The ture of opposing vectors within a small neighborhood. Just derivatives in different channels can point in opposite di- as for color images, the solution was found in tensor math- rections, hence cancellation might occur by simple addi- ematics and a set of tensor based features for oriented pat- tion. The solution to this problem is given by the structure terns was proposed [8] [9]. tensor for which opposing vectors reinforce each other. In addition to the apparent loss of information which We review the set of existing tensor based features which occurs by converting an RGB image to a luminance im- are applied on luminance images and show how to expand age, photometric information is also lost. In [10] Shafer them to the color domain. We combine feature detectors introduces the dichromatic reflection model. The theory with photometric invariance theory to construct invariant provides a physical model which allows for discrimina- features. Experiments show that color features perform tion of different photometric events in the image, such as better than luminance based features and that the addi- changes caused by shadows, shading or specularities. On tional photometric information is useful to discriminate be- the basis of this model, others provided algorithms invari- tween different physical causes of features. ant to various photometric events [11] [12]. Combining the photometric invariance theory with geometric operations has been investigated by Geusebroek et al. [13]. More re- Introduction cently, van de Weijer et al. [9] proposed the quasi-invariants, which is a set of derivatives invariant to photometric changes. Feature detection is an important tool in computer vision These derivative filters have the advantage over existing [1]. Differential based features for detecting events such methods that they remain stable over the entire RGB-space. as edges, corners, salient points, are used abundantly in a variety of applications such as matching, object recogni- In this paper, we start from the observation made by tion, tracking [2] [3] [4]. Although the majority of images DiZenzo that tensors are suited to combine first order deriva- is in color format nowadays, the computer vision commu- tives for color images. The first contribution that we col- nity still uses luminance based feature extraction. Obvi- lect a number of existing features which are based on the ously, extensions of these feature detection techniques to structure tensor and show how to extend these to color im- the color domain is desired. ages. The second contribution is that we combine these A pioneering work on extending edge detection to color features with the photometric derivatives which allows for images was proposed by DiZenzo [5]. The paper addresses photometric invariant feature detection. the problem of opposing vectors for different color chan- nels. Opposing vectors occur on edges where for one chan- nel the signal decreases while for another the signal in- Extending Differential Based Operations to creases. A simple addition of the opposing derivative sig- Color Images nals of the different channels reduces the total derivative strength. DiZenzo solves this problem by proposing the The extension of differential based operations to color im- tensor based gradient for which opposing vectors reinforce ages can be done in various ways. The main challenge one another. Sapiro and Ringach [6] further investigated here is how to project differential structure back to a scalar the local structure tensor and the interpretation of its eigen- representation. For the first order differential structure of values within the context of anisotropic diffusion. color images this has been explored in [5] and [6]. Here, Similar equations as found by DiZenzo [5] were pre- we describe several consideration which will result in the sented by Kass and Witkin [7], who proposed an orienta- color tensor framework given in section 3. (a) (b) Figure 1: a) The subspace of measured light RGB in the Hilbert space of possible spectra. b) The RGB coordinate system and an alternative orthonormal color coordinate system which spans the same subspace. Mathematical Viewpoint red, green and blue channels (see fig. 1). For operations on the color coordinate system to be physically meaningful As pointed out in [5], simply adding the differential struc- they should be independent of orthonormal transformation ture of different channels may result in a cancellation even of the three axes in Hilbert space. As an example of an when evident structure exists in the image. E.g. for a red- orthonormal color coordinate system the opponent color green edge the derivatives in the red and green channel space can be mentioned (see fig. 1). The opponent color point in opposing directions. Instead of adding the direc- space spans the same subspace as the subspace defined by tion information (defined on [0; 2π ) of the different chan- i the RGB-axes and hence physically meaningful features nels, it is more appropriate to sum the orientation informa- computed from both subspace should yield the same re- tion (defined on [0; π ). In the example of the red-green i sults. We will verify the color features to be invariant of edge the derivatives in the red and green channel have op- the accidental choice of the color coordinate frame. posing direction, but the same orientation. A well known mathematical method for which vectors in opposite direc- tions reinforce each other is provided by tensor mathemat- Tensor Based Feature Detection for Color ics. Tensors describe the local orientation rather than the Images direction. More precise, the tensor of a vector and the ten- sor of the same vector rotated over 180◦ are equal. In this section we extend several tensor based features to color images. As stated before, the tensor basis ensures Photometric Viewpoint that vectors pointing in opposite direction reinforce each other. Further, the feature detectors are verified to be in- A good reason for using color images is the photometric variant for orthonormal rotations of the RGB-space. information which can be exploited. Photometric invari- ance theory provides invariants for different photometric variations. Well known results are photometric invariant Structure Tensor Based Features colorspaces such as normalized RGB or HSI. Opposing derivative vectors are common for invariant colorspaces. In [5] Di Zenzo pointed out that the correct method to com- Actually, for normalized RGB the summed derivative is bine the first order derivative structure is by using a local per definition zero. Hence, the structure tensor is indis- tensor. Analysis of the shape of the tensor leads to an ori- pensable for computing the differential structure of photo- entation and a gradient norm estimate. metric invariant representations of images. Given an image f, the structure tensor is given by f 2 f f Physical Viewpoint G = x x y (1) f f f 2 x y y Next to the photometric invariance discussed above, we will look into invariance with respect to coordinate trans- where the subscripts indicates spatial derivatives and the formations. For color images, values are represented in bar ¯: indicates the convolution with a Gaussian filter. As the RGB coordinate system. The -dimensional Hilbert discussed in section 2 tensors can be added for different 1 1 2 T space is sampled with three probes which results in the channels. For a multichannel image f = f ; f ; :::; f n @ Rf Rf the structure tensor is given by since @x = x and f T f f T f Rf T Rf f T RT Rf f T f G = x x x y ( x) y = x y = x y (7) f T f f T f (2) y x y y ! Here R is a rotation operator on the channels of f. where superscript T indicates the transpose operation. For f T color images = (R; G; B) , this results in the color Adapted Structure Tensor Based Features structure tensor Similar equations to Di Zenzo's equations for orientation R2 + G2 + B2 R R + G G + B B x x x x y x y x y : estimation are found by Kass and Witkin [7]. They stud- R R + G G + B B R2 + G2 + B2 x y x y x y y y y ied orientation estimation for oriented patterns (e.g. fin- (3) gerprint images). Oriented patterns are defined as patterns The color structure tensor describes the 2D first order dif- with a dominant orientation everywhere. For oriented pat- ferential structure at a certain point in the image. Eigen- terns other mathematics are needed than for regular ob- value analysis of the tensor leads to two eigenvalues which ject images. The local structure of object images is de- are defined by scribed by a step edge, whereas for oriented patterns the local structure is described as a set of lines (roof edges). 2 2 1 f T f f T f f T f f T f f T f Lines have the property that they have opposing vectors on λ1 = 2 x x + y y + x x y y + 2 x y − ! a small scale. Hence for geometric operations on oriented r 2 2 patterns, mathematical methods are needed for which op- 1 f T f f T f f T f f T f f T f λ2 = 2 x x + y y x x y y + 2 x y posing vectors enforce one another.