Corner Detection in Textured Color Images

In IEEE Seventh International Conference on Computer Vision, Volume 2, pages 1039-1045,

September 1999

Mark A. Ruzon Carlo Tomasi

Computer Science Department

Stanford University

Stanford, CA 94305

Abstract [4]showed that edges found by rst-derivative op era-

tors tend to \round o " corners. Without using the

Corner models in the literature have laggedbehind

image itself, it is imp ossible to distinguish true cor-

edge models with respect to color and shading. We use

ners from curved b oundaries. Therefore, we opt for a

botharegion model, based on distributions of pixel col-

direct approach.

ors, and an edge model, which removes false positives,

At a conceptual level, corner and edge detection

to perform corner detection on color images whose re-

algorithms b oth compute the degree to whichtwo ad-

gions contain texture. We show results on a variety

jacent regions are dissimilar. Corners do not bisect an

of natural images at di erent scales that highlight the

op erator's supp ort, however, and the resulting asym-

problems that occur when boundaries between regions

metry must b e accounted for. Also, corners are p oint

have curvature.

features, so only one resp onse to the same part of the

1 Intro duction

image can b e accepted. Detecting junctions, though,

requires accepting multiple resp onses of the corner de-

Corners and junctions (multiple corners at the same

tector in the same or nearly the same image lo cation.

image lo cation) are crucial for high-level vision tasks

b ecause they represent o cclusions useful to stereo and Our approach uses b oth a region mo del, from which

motion algorithms, and they provide shap e informa- we create a set of corner candidates, and an edge

tion for ob ject recognition. They are arguably at least mo del, which decides whether to accept or reject a

as imp ortant as edges, yet currentedgemodelsinvoke

candidate. We mo del a corner as twoadjacent regions

fewer assumptions and are more robust than current that di er in their color distributions. The resulting

corner mo dels. op erator generalizes edge detection to asymmetric re-

gions with multiple colors p er region. Multiple colors

Sp eci cally, edge mo dels prop osed in the literature

are represented by a set of p oint masses in a color

are sup erior to existing corner mo dels with resp ect to

space. The distance b etween two such sets is found

color and shading. Many color edge detectors have

using the Earth Mover's Distance, which measures the

b een prop osed ([1], [5], and [11] form a representative

minimum amountof\work" required to transform one

sample), but corner detectors have b een con ned to

set into the other in that space.

greyscale images. Certain algorithms (e.g. [8]) app ear

to b e easily extendable to color images. The e ects The advantage of this mo del is that we can detect

of shading on the direction of the image gradientwere corners (and edges, see [10]) in textured regions where

mo deled byWang and Binford [12] to create an edge other detectors cannot. Two textures mayhavethe

detector insensitive to shading, while most corner de- same \mean color," for example, even though they

tectors assume that regions are of constantintensity have no color in common. Furthermore, the texture

(Alvarez and Morales [2] assumed level sets). need not b e homogeneous as long as its colors are suf-

ciently di erent from its neighb ors.

Many corner detectors start with an edge map

rather than an image (e.g. [6] and [7]), whichwould An edge mo del is also necessary, b ecause corners

app ear to mitigate such e ects. However, wear- cannot exist indep endently of edges. Our mo del, in-

gue against using these indirect metho ds for tworea- spired by Deriche and Giraudon's, presumes that at

sons: (1) using the output of an algorithm whose goal the twoendpoints of the corner, there is a strong

is something other than corner detection causes un- edge resp onse in the same direction. Furthermore, the

known biases and errors to propagate into the corner edge resp onse b etween the two endp oints of the corner

detector, and (2) the analysis of Deriche and Giraudon should b e weaker than the corner resp onse. Wecan A B R

(x,y) α SO S

θ I

partial normalized

(a) (b)

A B A B

S 1 0 S 1 0

I I

Figure 1: Parts of the region mo del. (a) Illustra-

S 1 6 S 1=7 6=7

O O

tion of op erator parameters. (b) The pixel weighting

EMD: 0 EMD: 6=7

function, a surface of revolution of half of a Gaussian

derivative function.

Figure 2: The normalized EMD can detect corners

that the partial EMD cannot.

compare corner candidates with this mo del to exclude

most false p ositives in the op erator's resp onse. If mul-

equally breaks the circle into wedges, allowing ecient

tiple candidates all resp ond well to the same corner,

up dating of the signatures. Weuse15 wedges.

we group them and cho ose the \b est" candidate.

We represent colors in the CIE-Lab color space [13],

The next two sections explain the region and edge

in which short, Euclidean distances are p erceptually

mo dels, resp ectively, after whichwe present the results

accurate. Toaccount for the fact that long distances

and our conclusions.

are not, we use a normalized measure that saturates:

2 The Region Mo del

d =1 exp(E = );

ij ij

In this section we develop a mo del of two adjacent

where E is the Euclidean distance b etween color i

regions and the p erceptual distance b etween them. In

and color j ,and =14:0 is a constant determining

Section 2.1 we summarize the representation of a re-

the steepness of the function.

gion as a color signature; details are in [10]. Section 2.2

The distance b etween two color signatures is found

tackles the problem of asymmetry b etween the two re-

using the Earth Mover's Distance (EMD) [9]. The

gions, and Section 2.3 explains how initial corner can-

EMD measures the minimum amountofphysical work

didates are selected.

needed to move the masses of one signature into cor-

2.1 Color Signatures

resp ondence with the other. In our formulation, the

A color signature is a set of p oint masses that rep-

EMD lies in [0, 1] since the maximum amountofmass

resents one of the two regions. There are ve param-

that can b e moved and the maximum distance it can

eters that determine which pixels will b elong to each

move are b oth 1.

region: (x; y ), the lo cation of the center of the win-

2.2 Partial EMD vs. Normalized EMD

dow; 2 [0; 360), the orientation of the corner (de-

After creating two color signatures, S inside the

ned as the angle formed by the p ositive x-axis and

corner and S outside it, we can use the EMD to

the \clo ckwise" side of the corner); 2 (0; 180], the

measure the similaritybetween the two regions. An

angle subtended by the corner; and R, the scale pa-

imp ortant issue in this computation that is not present

rameter (Figure 1(a)). Because it is natural to con-

when using this mo del for edge detection is that S

sider a corner as a wedge, the window is a circle of

always has more mass than S .

radius R.

We normalize S to have a mass of 1, regardless

Vector quantization applied to the circle deter-

of the value of , to preserve the same output range.

mines the numb er and lo cation of the p oint masses.

There are twoways to normalize S :wecanusethe

Each pixel contributes a weight dep endent onlyonits

same constant and nd the EMD b etween signatures

distance to the center. The p olar function f (r )=

of unequal mass (\partial" EMD), or we can assign

cr e S a mass of 1 also (\normalized" EMD). , where c is a normalizing constantand =

Eachtyp e of EMD has di erent advantages. In R=3, is the p ositive half of a 1-D Gaussian deriva-

Figure 2 the normalized EMD detects a corner that the tive function revolved around the y -axis (Figure 1(b)).

partial EMD do es not. A 45 corner consists entirely Isotropy simpli es computations over all combinations

of color A, while the oustide region has amounts of of and b ecause the mass that each pixel contributes

colors A and B (a p erceptual distance of 1 from A)in remains constant. Sampling the ranges of and

or the corners that we do detect,

B in natural images. F it is b est to describ e them as accurately as p ossible.

SO Finding Corner Candidates

2.3

y measur- The pro cess of corner detection b egins b

ver all circular windows and for all ing the EMD o

combinations of and . The result is a list of three-

30 partial normalized

dimensional tensors, one for eachvalue of . Corner

A B A B

candidates are maximum values over x, y , and that

S 1 0 S 1 0

I I

are ab ove a threshold. Parab olic interp olation over

S 0.8 10.2 S 0:07 0:93

O O

gives the actual strength and orientation of a candi-

EMD: 0.2 EMD: 0.93

date.

Output for di erentvalues of cannot b e compared

60 partial normalized

directly,however, even though the range of values is

A B A B

the same. From a purely statistical standp oint, it is

S 0.9 0.1 S 0.9 0.1

I I

less likely that a large EMD will result from a smaller

S 0 5 S 0 1

O O

value of b ecause of the greater imbalance in the

EMD: 0.9 EMD: 0.9

amount of mass. The net e ect is that wemust vary

our threshold linearly with .

In addition, wemust cho ose b ounds on b ecause

the output at small values is less reliable due to noise,

and large corners are hard to distinguish from edges.

Wehavechosen =30 and = 150 .

min max

3 Corner Detection

partial normalized

In this section, we present our edge mo del (Sec-

tion 3.1) and consider the problem of pruning multi-

Figure 3: The partial EMD can more accurately de-

ple resp onses to the same corner that all satisfy this

scrib e a corner.

mo del (Section 3.2).

3.1 The Edge Mo del

a 1:6 ratio. We de ne the mass of S to b e 1, and the

In order for a true corner to exist, there must b e

mass of S can b e either 1 or 7. If wecho ose the mass

evidence of strong edges that are consistent with the

to b e 7 (partial EMD), then S b ecomes a subset of

lo cation, orientation, and angle of the corner. A basic

S , and the distance is 0. If instead wecho ose the

schematic of our mo del is illustrated in Figure 4(a).

mass of S to b e 1 (normalized EMD), the distance is

Sp eci cally, our mo del incorp orates three ideas:

6=7, and a corner is likely to b e found.

1. Edge direction at each end of a corner must match

Figure 3 shows a situation where the partial EMD

the orientation of the side of that corner.

describ es a corner more accurately than the normal-

ized EMD. A 60 corner consists entirely of color A

2. Edge resp onse at each end of a corner must b e

except for the two edges, which contain some pixels of

high.

color B . If 10% of the pixels inside the corner have

color B , then the values of the EMD are those shown

3. Between the two ends of a corner, the edge re-

in the accompanying table. Both typ es of EMD detect

sp onse must b e weaker than the corner resp onse.

a corner, but the normalized EMD estimates to b e

In theory, none of these conditions is satis ed when

30 while the partial EMD correctly estimates to

be 60 . An example of the di erences for real image a corner candidate falsely resp onds to an edge (Fig-

data is shown b elow the table. The corner found by ure 4(b)).

the partial EMD runs along the edges, while the other Before we can measure the degree to which a corner

do es not. matches our mo del, wemust have edge information.

This is found by applying our op erator with =180 ,

Wehavechosen the partial EMD for our exp eri-

and nding the orientation at each image lo cation that

ments. Although wemayhave false negatives, the

numberislikely to b e small b ecause the frequency of maximizes the resp onse (see [10] for details).

the phenomenon illustrated in Figure 2 is inversely Once this is done, we ndtwo angles: , the di er-

prop ortional to , and such corners are less frequent ence in orientation b etween the clo ckwise side of the

(a) (b)

Figure 4: (a) A true corner. The edge is aligned with

(a) (b) (c)

the corner at the endp oints and \rounded o " in the

middle, where the resp onse weakens. (b) A false p os-

Figure 5: Corner detection steps applied to op era-

itive due to inhomogeneities on one side of an edge.

tor output of cut stone o ccluding an ivy-covered wall.

(a) Initial corner candidates. (b) After applying the

edge mo del. (c) Final result after pruning multiple

corner and the edge resp onse at the corner's endp oint,

resp onses.

and , the corresp onding angle for the counterclo ck-

wise side. Because edges havetwovalid orientations

(di ering by180 ), wecho ose the one that yields the

and one of the following conditions is true: (1) the

smaller angle. Our measurement P can b e expressed

twoclockwise orientations di er bynomorethan10 ,

(2) the twocounterclo ckwise orientations di er byno

P =cos + cos ;

more than 10 , or (3) the sum the di erences is no C CC

more than 40 . These conditions group \nested" cor-

whichliesbetween 0 and 2. We threshold P at 1.97.

ners while preserving multiple corners near junctions.

Finally, where the edge crosses the line that bisects

An ambiguity arises when corner X is close to cor-

the corner, the pro jection of the edge resp onse onto

ners Y and Z ,butY and Z are not close to each other.

the line normal to the bisector must b e weaker than

If our notion of \closeness" is global, then the order

the corner resp onse. Wecheck resp onses on a small

in whichwe examine corners a ects the nal output.

interval along the bisector line centered at a p oint

Since this is unacceptable, we compute the transitive

R(sec tan ) pixels away from the corner. This

2 2

closure of \closeness," that is, X , Y ,andZ will all

quantity is the distance from the corner p ointtothe

b ecome part of the same set. It is theoretically p os-

circumference of an imaginary circle tangenttothe

sible that corners in distant parts of the image could

sides of the corner at its endp oints.

b ecome part of the same set; in practice, however, the

Applying all three parts of the mo del to the set

application of the edge mo del removes enough candi-

of initial corner candidates greatly reduces their num-

dates that this do es not happ en often.

b er. Figure 5(a) shows the candidates for an image

Once wehave computed the transitive closure, we

patch consisting of cut stone against an ivy-covered

pickthememberofeach set that maximizes the ex-

wall. Though all the candidates are near the b ound-

pression 2C + P + E , where C is the corner resp onse,

ary b etween the two regions, most of them do not t

P is the degree of orientation match describ ed earlier,

the b oundary well. Figure 5(b) shows the results after

and E is the sum of the edge resp onses at the end-

applying the edge mo del.

points of the two sides of the corner. C is doubled so

3.2 Pruning Multiple Resp onses

that each term contributes equally. The nal corner

Using edge data, however, do es not completely

of our example is shown in Figure 5(c).

solve our problem. Figure 5(b) contains two corners,

4 Results

each of whichgives a strong corner resp onse to the

same area of the image and matches the edges well.

In this section we present results on a variety

Obviously,wewould liketohave only one.

of image patches in order to convey the versatil-

ity of the op erator. All the results in this pap er The question of which corners to group together in

that use the partial EMD were computed with the this image is trivial, but the general question of when

same thresholds. The lengths of the sides of the cor- two or more corners are \close enough" to each other

ners in the images are equal to the chosen value of that one should b e accepted and the rest rejected has

R. Color versions of the results are available from no de nitive answer in natural images.

http://vision.stanford.edu/public/publication/. We de ne two corners as b eing \close enough" if

Figure 6 shows one fabric o ccluding another. Al- the corner p oints are within 3R=4 pixels of each other

Figure 7: Corners found b etween three regions.

sumption, which is violated here. Neither app ears

to have b een tested on b oundaries with continuously

Figure 6: Two fabrics. Note the heterogeneityofeach

changing curvature or on corners as large as 150 ,

texture, as well as the existence of shadows.

though.

Finally,we wish to mention the running time. The

op erator is implemented in C and can pro cess image

though each contains texture that varies greatly in

lo cations (each including all combinations of and )

color and has regions in partial shadow, the corner is

at a rate on the order of 1500 p er minute on an SGI

correctly detected.

Indigo 2, dep ending on R.

Figure 7 is more complicated b ecause three textures

5 Conclusion

are involved: trees (upp er left), ro ck (lower left), and

ro ck in deep shadow (right). Five corners are found

Wehave presented an op erator that outputs high

that separate the regions and, incidentally, form most

values when two regions inside and outside of a cor-

of the b oundary of the illuminated ro ck.

ner have di erent color distributions. Using color sig-

In Figure 8 we show output at three di erent scales. natures and the Earth Mover's Distance allows us to

The image contains a junction, but the textured re- detect corners in situations that others havenoteven

gion subtends an angle greater than , and the considered b ecause they are forced to assume that each

max

region b oundaries are not rays emanating from the region is constant.

junction. At all three scales we ndthetwo smaller

The edge mo del eliminates those corners that are

corners with compatible orienations and corner p oints

not supp orted by evidence of strong edges. This basic

near each other. We emphasize that corners are esti-

dep endence of corners on edges is b oth conceptually

mated indep endently; a true junction detector would

imp ortant and practically e ective.

combine these corners, p erhaps with knowledge of the

The application of corner detection to natural im-

lack of symmetry near the junction [10], to estimate

ages that are not comp osed mostly of p olygons brings

its lo cation and parameters.

up manyinteresting issues, the most imp ortantof

Other researchers have examined the evolution of which is the de nition of a corner. Wehave not sp ec-

corners across scales in more detail. Mokhtarian and i ed an optimality condition from which our op erator

Suomela [7] detected corners at a large scale and used can b e derived, b ecause deciding that a corner exists

smaller scales to lo calize them, an approach that might is mostly a question of thresholding the curvature of

work well here. Alvarez and Morales' framework [2] an edge with resp ect to the chosen scale. By the same

caused corners to evolve along the line bisecting the token, there is no ground truth in natural images to

corner. Their framework dep ends on the level set as- compare our results to. The results are b est evaluated

Figure 9: Arock with edges (black) and corners

(white) drawn. The features are complementary.

in the context of an application, suchastracking land-

marks for rob otic navigation or, more generally, shap e

recovery for ob ject recognition.

Another fundamental issue is the integration of cor-

ners and edges. One approachmay b e to detect

them separately and use the corners to p erturb the

edges toward the true b oundary. In Figure 9 wehave

drawn, in addition to the two corners, a set of edges

found by thresholding and p erforming non-maximal

suppression [3] on the op erator's output at =180 .

Through a suitable pro cess (e.g. energy minimiza-

tion), it may b e p ossible to conform the edges to the

corners and pro duce a more accurate b oundary.

However, the fact that we can compute edge and

corner information in the same framework bychang-

ing the value of b egs the question of whether the two

are fundamentally di erent at all. It is unsatisfactory

that wemust exclude corners b etween 150 and 180 .

In Figure 10 wehavedrawn b oth edges and corners,

including =165 , on top of a natural image. The

large amountofoverlap indicates that the two di er-

ent algorithms are pro ducing nearly the same output,

though they do complementeach other in a few places.

Future researchwillinvestigate a b oundary mo del in

which b oth edges and corners can b e detected and re-

lated to each other without creating an arti cial dis-

tinction b etween them.

Acknowledgment

WethankYossi Rubner for the EMD co de.

Figure 8: Comparison of output at three di erent

References

scales near a junction. One corner of the junction

[1] R. Alb erto Salinas, C. Richardson, M.A. Abidi, and

is greater than 150 and cannot b e recovered directly. R.C. Gonzalez. Data fusion: Color edge detection and

Figure 10: Canyon with corners (left) and edges (right). Note the high amountofoverlap. 165 corners have

b een included.

surface reconstruction through regularization . IEEE [8] L. Parida, D. Geiger, and R. Hummel. Junctions:

Trans. on Ind. Elec., 43(3):355{363, 1996. Detection, classi cation, and reconstruction. PAMI,

20(7):687{698, 1998.

[2] L. Alvarez and F. Morales. Ane morphologica l

[9] Y. Rubner, C. Tomasi, and L.J. Guibas. A metric for

multiscale analysis of corners and junctions. IJCV,

distributio ns with applicati ons to image databases. In

25(2):95{107, 1997.

ICCV98, pages 59{66, 1998.

[3] J. Canny. A computational approach to edge detec-

[10] M. Ruzon and C. Tomasi. Color edge detection with

tion. PAMI, 8(6):679{698, 1986.

the compass op erator. In CVPR99,volume 2, pages

160{167, 1999.

[4] R. Deriche and G. Giraudon. A computational

[11] P.E. Trahanias and A.N. Venetsanop oulos. Vec-

approach for corner and vertex detection. IJCV,

tor order-statistics op erators as color edge detectors.

10(2):101{124, 1993.

IEEE SMC-B, 26(1):135{143, 1996.

[5] S. Di Zenzo. A note on the gradientofamulti-image.

[12] S.-J. Wang and T.O. Binford. Generic, mo del-based

CVGIP, 33(1):116{125, 1986.

estimation and detection of discontinuiti es in image

surfaces. In ARPAIUW,volume I I, pages 113{116,

[6] Q. Ji and R. Haralick. Breakp oint detection using

1994.

covariance propagation. PAMI, 20(8):845{851, 1998.

[13] G. Wyszecki and W.S. Stiles. Color Science: Concepts

[7] F. Mokhtarian and R. Suomela. Robust image cor-

and Methods, Quantitative Data and Formulae.John

ner detection through curvature scale space. PAMI,

Wiley and Sons, New York, NY, 1982. 20(12):1376{1381, 1998.