IMPROVED HARRIS’ ALGORITHM FOR CORNER AND EDGE DETEC- TIONS Soo-Chang Pei, Jian-Jiun Ding, Department of Electrical Engineering, National Taiwan University, Taipei, Taiwan, R.O.C Email address: [email protected], [email protected]

ABSTRACT However, the algorithm has some problems, such as the A more accurate algorithm for corner and edge detections ability to distinguish the corner from the peak or the dip that is the improved form of the well-known Harris’ algo- and the robustness to noise should be improved. In [5], an rithm is introduced. First, instead of approximating |L[m+x, improved algorithm based on modifying the detection n+y]–L[m, n]|2 just in terms of x2, xy, and y2, we will ap- criterion was proposed. In this paper, we apply many new proximate |L[m+x, n+y]–L[m, n]|(L[m+x, n+y]–L[m, n]) by ideas listed in Section 2 to improve Harris’ algorithm for the linear combination of x2, xy, y2, x, y, and 1. With the . modifications, we can observe the sign of variation. It can avoid misjudging the pixel at a dot or on a ridge as a cor- 2. IDEAS FOR IMPROVING PERFORMANCE ner and is also helpful for increasing the robustness to (A) Instead of approximating |L[m+x, n+y]L[m, n]|2, we noise. Moreover, we also use orthogonal polynomial ex- use a quadratic polynomial to approximate L [m, n, x, y]: pansion and table looking up and define the cornity as the 1 L [m, n, x, y] “integration” of the quadratic function to further improve 1 the performance. From simulations, our algorithm can = L>@>@m  x, n  y  L m, n (L>@>@m  x, n  y  L m, n ) . (5) much reduce the probability of regarding a non-corner This modification is helpful for improving the robust- pixel as a corner. In addition, our algorithm is also effec- ness to noise. Suppose that the image is interfered by a tive for . noise K[m, n], i.e., H[m, n] = L[m, n] + K[m, n], and

E[K[m, n]] = 0, E[K[m, n]K[m+x, n+y]] = Wx,,y, (6) Index terms:  corner detection, edge detection, prob{K[m, n] = k} = prob{K[m, n] = k}, (7) quadratic polynomial, , noise immunity then we can prove that

2 E{ H>@>@m  x, n  y  H m, n } 1. INTRODUCTION 2 L>@>@m  x, n  y  L m, n  2(W 0,0  W x, y ) , (8) Corner detection is important for feature extraction and pattern recognition. In [1], Harris and Stephens proposed a E{ H>@>@m  x, n  y  H m, n (H>@>@m  x, n  y  H m, n )} corner detection algorithm. First, they used a quadratic L>@>@m  x, n  y  L m, n (L>@>@m  x, n  y  L m, n ) . (9) polynomial to approximate the variation around [m, n]: Thus, the mean of (1) is affected by noise, but from (9) 2 L>@>@m  x, n  y  L m, n the mean of L1[m, n, x, y] will be not affected by noise. The sign of variation is also important to distinguish A x 2  2C x y  B y 2 + remained terms, (1) m,n m,n m,n a corner from a peak. For both a corner pixel and a peak, where Am,n, Bm,n, and Cm,n were calculated from the corre- the variations along all the directions are large, as in Fig. lations between the variations and a window function: 1. If we observe only the amplitude of variations, as Har- 2 2 Am,n X w , Bm,n Y w , Cm,n XY w , (2) ris’ algorithm, they are hard to distinguish. In contrast, if T the sign of variations can be observed, we can distin- X L>@>m, n 1, 0,1 @, Y L>@>m, n 1, 0,1 @, guish them. For a peak, the signs of variations along all 2 2 2 wx,y = exp[(x +y )/2V ]. (3) the directions are negative. For a corner, sometimes Then the variations along the principal axes can be calcu- variation is positive and sometimes it is negative. lated from the eigenvalues of the following 2u2 matrix: (B) We will approximate the variation by the linear com- bination of x2, xy, y2, x, y, and 1, see (16). For Harris’ al- ªAm,n Cm,n º H . (4) gorithm in (1), only the terms of x2, xy, and y2 are used. m,n «C B » ¬ m,n m,n ¼ Note that x2 and y2 are even-even bases (i.e., even re- If both the two eigenvalues of Hm,n are large, then we rec- spect to both x and y) and xy is an odd-odd basis. Using ognize the pixel [m, n] as a corner. Harris and Stephens them is hard to observe even-odd and odd-even varia- proposed a systematic and effective way to detect corners. tions. Thus, to observe all types of variations, we should

1-4244-1437-7/07/$20.00 ©2007 IEEE III - 57 ICIP 2007 use the terms of x (an odd-even basis) and y (an even- odd basis). With these terms, we can observe more Corner Edge + Xm,n(n) +Xm,n(n) styles of variations and classify the pixel into more types. Y (p) higher gray m,n higher gray (C) Instead of (3), we use “orthogonal polynomial expan- Ym,n(~) level re- level re- sion” to find the coefficients. It can assure that the mean i i +Ym,n(~) square error of approximation to be minimal [2]. [m, n] lower gray [m, n] (D) In [1], Harris classified the pixel into only three types: level region lower gray level re- corners, edges, and plains. In this paper, with the help of Xm,n(p) +Ym,n(p) i Case table (see Table 1), we classify the pixel into cor- Xm,n(p) ners, edges, ridges, valleys, peaks, saddles, and plains. Peak Ridge + X m , n (~) Ym,n(p) +X (p) (E) From derivations and experiments, we find that, to m,n Y (p) lower gray achieve the best accuracy, it is better to define the cornity lower gray m,n lower gray level re- [ m , n ] level re- as the “integration” of the quadratic function. See Step 6. level re- i i Xm,n(p) i +Ym,n(p) 3. ALGORITHM Xm,n(~) +Ym,n(p) (Step 1) First, find the orthonormal polynomial set that is Fig. 1 Variations of gray levels along four principal directions orthogonal respect to a weighting function w[m, n]. That is, for the pixel at a corner, on an edge, at a peak, and on a ridge. 2 2 M k >@x, y a1,k  a2,k x  a3,k y  a4,k x  a5,k xy  a6,k y , Table 1 Case table. for k = 1, 2, 3, 4, 5, 6, (10) v1, v2 (p, p)(p, ~) (p, n) (~, ~) (~, n) (n, n) v , v and the parameters aj,k should be chosen properly such that 3 4 (~, p) (n, p) (~, n) . (11) (p, p) Q1 C C E1 E1 Q3 ¦¦M k >@>@>@x, y M h x, y w x, y G k ,h xy (p, ~) (~, p) C E E P CY E2 We can use the Gram-Schmidt algorithm to find aj,k. For (p, n) (n, p) C E E E E C example, if we choose the weighting function as (~, ~) E1 P E P P E2 w[x, y] exp[(x 2  y 2 ) / 20] for (x2 + y2)1/2 < 6.5, (12) (~, n) (n, ~) E1 CY E P E C w[x, y] = 0 for (x2 + y2)1/2 > 6.5, (13) (n, n) S E2 C E2 C Q2 then the orthogonal polynomials are C: corner, E: edge, E1: ridge, E2: valley, M1>@x, y 0.1339 , M2 >@x, y 0.0498x , M3 >@x, y 0.0498y , Q1: peak Q2: dip, Q3: saddle, CY: Y-corner, P: plain. M >@x, y 0.0157y  0.1131, M >@x, y 0.0206xy , 4 5 (Step 4) From (19), we can observe the variation along the 2 2 M6 >@x, y 0.0158y  0.00213x  0.1296 . (14) two principal axes. For example, the variation along the (Step 2) Then, we do the inner product for L1[m, n, x, y] is positive part of Xm,n-axis (denoted by +Xm,n) is: t (defined in (5)) and Mk[x, y], where k = 1, 2, 3, 4, 5, 6: v (c X 2  c X )dX 1 ³0 1,m,n m,n 3,m,n m,n m,n bk,m,n M k >@>x, y L1 m, n, x, y @>@w x, y , (15) ¦¦ 3 2 xy c1,m,nt / 3  c3,m,nt / 2 . (20)

(Step 3) Then we express the variation around [m, n] by: Similarly, the variations along the negative part of Xm,n- 6 axis (denoted by –Xm,n) and along the positive and nega- Qm,n >@x, y bk,m,nMk >@x, y ¦ tive parts of Ym,n-axis (denoted by +Ym,n and –Ym,n) are: k 1 2 2 for X : v c t3 / 3  c t 2 / 2 . (21) = p1,m,nx +p2,m,nxy +p3,m,ny +p4,m,nx +p5,m,ny +p6,m,n. (16) m,n 2 1,m,n 3,m,n Note that Qm,n[x, y] | L1[m, n, x, y] (defined in (5)). Then, 3 2 for +Ym,n: v3 c2,m,nt / 3  c4,m,nt / 2 , we expand Q [x, y] as a function of principal axes m,n 3 2 Qm,n[x, y], for  Ym,n: v4 c2,m,nt / 3 c4,m,nt / 2 . (22) 2 2 c1,m,nXm,n  c2,m,nYm,n  c3,m,nXm,n  c4,m,nYm,n  p6,m,n , (17) We may choose t as 2~3. Then, we choose a threshold 'd c vk > ' o denoted by n, d vk < ' o denoted by p, where Xm,n = xcosTm,n+ysinTm,n, Ym,n = xsinTm,n+ycosTm,n, T e ' d vk d ' o denoted by ~. (23) and the principal axes [cosTm,n, sinTm,n] , [sinTm,n, cosT ]T are the eigenvectors of Harris’ algorithm classifies the variation into only 3 types. m,n For our algorithm, there are four directions (rX , rY ) ª p 0.5p º m,n m,n H 1,m,n 2,m,n , (18) and for each direction there are 3 types of variations (n, p, m,n « » 4 ¬0.5p2,m,n p3,m,n ¼ ~) . In sum, there are 3 = 81 types of variations. c1,m,n and c2,m,n are the corresponding eigenvalues, and (Step 5) For the typical corner in Fig. 1, the variation c3m,n = p4m,ncosTm,n + p5m,nsinTm,n, along one axis is (p, p) and along another axis is (p, n). c4m,n = p4m,nsinTm,n + p5m,ncosTm,n. (19) Thus, we conclude that if (v1, v2) = (p,p) and (v3, v4) = (p,

III - 58 n), the pixel should be at the corner. If a pixel is on an (a) corner candidates (b) edge candidates edge, then the variation along one principal axis is (p, n) and along another principal axis is (~, ~). We use Table 1 20 20 (Case table) to summarize the relations between the varia- 40 40 tion style and the most possible location of a pixel. With it, 60 60 we can conclude the pixel is at a corner, on an edge, on a 80 80 plane, on a ridge, in a valley, or at a peak. 100 100 120 120 (Step 6) In Step 5, we just obtain the “corner candi- dates”. Sometimes, the pixel that is not at a corner but 20 40 60 80 100120 20 40 60 80 100120 near the corner will be recognized as a corner after Step 5. (c) detected corner (d) detected edge Thus we should use the cornity score to choose the corner 20 20 from the corner candidates. We find that the most effec- 40 40 tive way is to define the cornity as the integration of the 60 60 quadratic polynomial Qm,n[x, y] in a circular region: 80 80 R 2S cornity = Q >@x, y rdrdT , (24) 100 100 ³³0 0 m,n 120 120 where Xm,n = rcosT, Ym,n = rsinT, and Qm,n[x, y] is defined in (16) and (17). Eq. (24) has an analytic solution: 20 40 60 80 100120 20 40 60 80 100120 4 2 Fig. 2 Corner detection and edge detection for Star image. x cornity = (c1,m,n  c2,m,n )R S / 4  p6,m,n R S . (25) d moreover, we classify the adjective pixels into four Why is the cornity defined as the integration of Q [x, y]? m,n quadrants (W >0 & U >0, W >0 & U <0, W <0 & U >0, and W Note that Q [x,y] approximates the variation around [m,n] m,n <0 & U <0) and for each quadrant, there should be at least Qm,n[x, y] | |L[m+x, n+y]  L[m, n]|(L[m+x, n+y]L[m, n]) one pixel that satisfies (33), then, we can treat [m, n] as a when x and y are small. (26) pixel on an edge. If [m, n] is on a plain (L[m+x, n+y]L[m, n]=0) or an edge (L[m+x, n+y]L[m, n]=(L[mx, ny]L[m, n])), then the 5. EXPERIMENTS integration of Qm,n(x, y) is near to zero. If [m, n] is at a We first give an example in Fig. 2. We follow Steps corner, then the integration of Qm,n[x, y] will be far from 1~4 and use the Case table in Step 5 to find the corner and zero. Thus, Qm,n[x, y] is good for cornity measurement. the edge candidates (shown in Figs. 2(a)(b)). Then, we use (Step 7) If a pixel [m, n] is a corner candidate and the “cornity” and “E score” and to select corners and cornity[m, n] t cornity[m+W, n+U] for W, U =-2~2. (27) edges from the corner and edge candidates. The results are then [m, n] is recognized as a corner. shown in Fig. 2(c)(d), which show corners and edges are detected successfully. 4. EDGE DETECTION Then we do some experiments to compare the per- Our algorithm can not only detect the corner but also formances of the proposed and the Harris’ algorithm for detect the edge, the ridge, the valley, and the plain regions. the image in Fig. 3(a). When using Harris’ algorithm, the To detect the edge, Steps 1a5 are the same as those of results of corner detection is shown in Fig. 3(b). Note that corner detection. In Step 6, we define the edge score as: some pixels at the dots, on the ridge, in the valley, or in a R 2S noise-interfered plain are treated as corners. From com- E score = T x, y rdrdT , (28) mon senses, they should not be corners. However, since ³³0 0 m,n 3 Harris’ algorithm observes just the “amplitudes” of varia- tions, and these pixels have large variations along two Tm,n >@x, y ¦bk,m,nMk >@x, y t1,m,n x  t2,m,n y  t3,m,n . (29) k 1 principle axes, they are misjudged as corners.

E score also has an analytic solution. If |t3,m,n/a|d R, If we use the proposed algorithm, we can distinguish a corner from an isolated pixel, a belt valley, and a belt x E score = aR3[cos(3M ) / 3  cosM ] 0 0 ridge according to the “sign” of variation. The result is 2  t3,m,n R [2M0  sin(2M0 )], (30) shown in Fig. 3(c). Note that no non-corner pixel is mis- 2 2 2 judged as a corner. Especially, no pixel in the lower-right where a = t1,m,n + t2,m,n and M0 = asin[t3,m,n/(aR)], S/2 part (the noised-interfered plain) is regarded as a corner. It d M0 d S/2. If |t3,m,n/a| > R, then 2 proves the theory in (9), i.e., the proposed algorithm is x E score = |t3,m,n|R S. (31) much more robust to noise. For an “edge candidate” pixel [m, n], if c At least 17 ad- In Fig. 4, we also do s experiment and show that the jective pixels satisfy: proposed algorithm can avoid regarding the Gaussian-like E score[m+W, n+U] d E score[m, n] for |W |, |U | d 2. (32) noise interfered region as a corner.

III - 59 (a) Input Image (b) Using Harris’ algorithm

20 20 50

40 40 60 100 60 80 80 150 100 100 120 20 40 60 80 100 20 40 60 80 100 120 200 (c) Using the proposed algorithm 250 50 100 150 200 250 20 Fig. 5 Using Harris’ algorithm to do corner detection 40

60

80 50

100 100 20 40 60 80 100 Fig. 3 (a) Image consists of three dots (upper-left), a valley (up- per-right), a ridge (lower-left), and a noise-interfered region 150 (lower-right). (b)(c) The results of corner detections. 200 (a) Input Image (b) by Harris’ method (c) by proposed method

10 10 10 250 50 100 150 200 250 20 20 20 30 30 30 Fig. 6 Using the proposed algorithm to do corner detection. 40 40 40

50 50 50 10 20 30 40 50 10 20 30 40 50 10 20 30 40 50 Fig. 4 Corner detection results for the plain interfered by Gaus- 50 sian-like noise. We also do the experiment for a natural image in Figs. 100 5 and 6. It is obvious that, when using the proposed algo- rithm, the probability of misjudging the pixel in a valley, 150 on a ridge, or at the isolated dots as a corner can be much reduced. Our algorithm can also detect most of the edges 200 of Lena image successfully, as in Fig. 7.

6. CONCLUSIONS 50 100 150 200

We introduced an improved algorithm for corner and Fig. 7 Using the proposed algorithm to do edge detection. edge detections. For the proposed algorithm, since the [2] A. F. Nikiforov, S. K. Suslov, and V. B. Uvarov, Clas- “sign” of variations is considered, the difference among a sical Orthogonal Polynomials of a Discrete Variables, corner, a belt ridge, and an isolated dot can be observed. Berlin; New York: Springer-Verlag, 1991. Due to the same reason, our algorithm is more robust to [3] S. M. Smith and M. Brady, “SUSAN–a new approach noise, which was proven in Figs. 3, 4 and 6. We also ap- to low level image processing”, Int. J. , ply orthogonal polynomial expansion, case table and the vol. 23, no. 1, 1997, pp. 45-78. new definition of cornity to improve the performance. [4] H. Wang and M. Brady, “Real-time corner detection algorithm for motion estimation”, Image and Vision 7. REFERENCES Computing, vol. 13, no. 9, 1995, pp. 695-703. [1] C. Harris and M. Stephens, “A combined corner and edge [5] N. C. Overgaard, “On a Modification to the Harris detection”, Proc. 4th Alvey Vision Conf., 1988, pp. 189-192. Corner Detector”, Symposium Svenska Sällskapet för Bildanalys, Stockholm, 2003.

III - 60