Degeneracy from Twisted Cubic Under Two Views

Home , Homography

Wu YH, Lan T, Hu ZY. Degeneracy from twisted cubic under two views. JOURNAL OF COMPUTER SCIENCE AND TECHNOLOGY 25(5): 916–924 Sept. 2010. DOI 10.1007/s11390-010-1072-9

Yi-Hong Wu (吴毅红), Tian Lan (兰添), and Zhan-Yi Hu (胡占义)

National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China E-mail: {yhwu, tlan, huzy}@nlpr.ia.ac.cn Received November 9, 2009; revised May 20, 2010.

Abstract Fundamental matrix, drawing geometric relationship between two images, plays an important role in 3- dimensional computer vision. Degenerate configurations of the space points and the two camera optical centers affect stability of computation for fundamental matrix. In order to robustly estimate fundamental matrix, it is necessary to study these degenerate configurations. We analyze all the possible degenerate configurations caused by twisted cubic and give the corresponding degenerate rank for each case. Relationships with general degeneracies, the previous ruled quadric degeneracy and the homography degeneracy, are also reported. Keywords fundamental matrix, twisted cubic, ruled quadric, degenerate configuration

1 Introduction can give more geometric intuition, which could be of guidance for placing cameras to avoid degeneracy in Fundamental matrix describes geometric relation practice. Furthermore, if we can judge the degeneracy between two 2-dimensional views and is a key concept by applying geometric knowledge, RANSAC will work in 3-dimensional computer vision. It plays an impor- much easier. tant role in image matching, epipolar geometry, camera Due to the importance of degeneracy analyses, many motion determination, camera self-calibration and 3- of such works have been reported previously. The dimensional reconstruction[1−9]. Robust and accurate planar scene is a trivial degenerate configuration for estimation for fundamental matrix has been the re- computing fundamental matrix, where the images can search focus of extensive researchers[10−17]. provide only six independent constraints[1,16] but the From at least seven pairs of point-point correspon- general fundamental matrix has seven degrees of free- dences between two views, the fundamental matrix dom in a nonlinear space. Degeneracy from twisted can be estimated. Sometimes, no matter how many cubic configuration has also been discussed. In [22], correspondences are used, accuracy of the estimation Buchanan stated that camera calibration from known is low. One of the main reasons is that the cameras space points under a single view is not unique if the and the scene lie on a degenerate or quasi-degenerate optical center and the space points lie on a twisted configuration. If a space configuration is degenerate cubic. The corresponding detection as well as emen- mathematically but the noise from the measured image dations including other unreliability was given by Wu makes it non-degenerate, any estimation under such a et al.[23] Then, under two views, Maybank[24] analyzed configuration would be useless[7,18-20]. It follows that characterizations of horopter curve and relations be- we should know what configurations might cause de- tween the curve and ambiguous case of reconstruc- generacy for estimating the fundamental matrix. More- tion. The horopter curve is regarded as a twisted cu- over in order for a robust random sample consensus bic, which intersects the plane at infinity at three par- (RANSAC) like [16, 21], we still need to know how ticular points. The ambiguous case of reconstruction great the degenerate degree is, namely, to know the de- implies ambiguity of fundamental matrix. Luong and generate rank of the coefficient matrix of the equations Faugeras reported stability of computing fundamental for computing fundamental matrix. In [12], RANSAC matrix caused by quadric critical surface in [25]. Hart- loop to estimate relation from quasi-degenerate data is ley and Zisserman[1] also gave systematic discussions reported, where the degenerate configurations need not on degeneracy of camera projection estimation from be known. This does not mean the studies on the degen- twisted cubic under a single view and on degeneracy erate configurations are useless. At least, such studies from ruled quadric surface under two views. Under

Regular Paper Supported by the National Natural Science Foundation of China under Grant Nos. 60835003 and 60773039. ©2010 Springer Science + Business Media, LLC & Science Press, China Yi-Hong Wu et al.: Twisted Cubic Degeneracy 917 three views, critical configurations are provided in [26], Section 4 and Section 5 is some conclusions. which is an extension of the critical surface under two views. Degeneracy under a sequence of images is also 2 Preliminaries investigated[2,27]. Maybank and Shashua[27] pointed The camera model used is a perspective camera. A out there is a three-way ambiguity for reconstruction space point or its homogeneous coordinates is denoted from images of six points when the six points and the by M , an image point or its homogeneous coordinates camera optical centers lie on a hyperboloid of one sheet. i is denoted by m , P denotes the camera projection In [2], Hartley and Kahl presented a classification of all i matrix, and O denotes the camera optical center. If possible critical configurations for any number of points two views are used, P 0 denotes the second camera pro- from three images and showed that in most cases, the jection matrix, O0 denotes this second camera optical ambiguity could extend to any number of cameras. center, and m0 denotes the corresponding image point Relative to the above works on degenerate config- i of M under this camera. Let F be the fundamen- urations, there are fewer in-depth studies on degener- i tal matrix between these two views. Other vectors or acy degrees of degenerate configurations. Torr et al.[16] matrices are also denoted in boldface. The symbol ≈ catalogued all two-view non-degenerate and degenerate means equality up to a scale. cases in a logical way by dimensions of the right null • Camera Projection Matrix space of equations on fundamental matrix and then M , i = 1..N are N 3-dimensional space points. proposed a PLUNDER-DL method to detect degene- i And, their 2-dimensional image points under a single racy and outliers. Chum et al.[21] also analyzed those view are m , i = 1..N. The camera projection matrix dimensions when the two views or most of the point i P is a 3 × 4 matrix such that m ≈ PM for all i. correspondences are related by a homography and pre- i i P is equal to K(R, t) up to a scale, where K is the sented an algorithm to estimate fundamental matrix 3 × 3 camera intrinsic parameter matrix, R is the 3 × 3 through detecting the homography degeneracy. They camera rotation matrix, t the camera translation vec- all[16,21] generalized the robust estimator RANSAC[28]. tor relative to the world coordinate system. For the The plane degeneracy in [16, 21] is consistent with camera optical center O, we have the equation: the ruled quadric degeneracy proposed by Hartley and Zisserman[1] because a plane and two camera optical PO = 0. (1) centers always lie on a degenerate ruled quadric. What are the degeneracy degrees when estimating the fun- • Fundamental Matrix damental matrix for other non-trivial degenerate confi- Fundamental matrix describes the geometric rela- gurations? In this paper, we discuss all possible dege- 0 tionship between two 2-dimensional views. Let mi, nerate situations caused by twisted cubic and give the i = 1..N be the 2-demensional image points of the same corresponding degeneracy degrees. Let S be a set of 0 space points M i under another view. Then, mi and mi space points and the two camera optical centers. We are related by the 3×3 fundamental matrix F through: find that if all the points in S lie on a twisted cubic, the configuration is degenerate for estimating fundamental 0T mi F mi = 0, i = 1..N. (2) matrix and the corresponding rank of the coefficient   matrix is five; if all the points other than one in S lie f1 f2 f3 on a twisted cubic, the corresponding rank is six; if all We denote F as F =  f4 f5 f6 . If mi ≈ the points other than two in S lie on a twisted cubic, the f7 f8 f9 corresponding rank is seven. The previous general de- T 0 0 0 0 T (ui, vi, wi) and mi ≈ (ui, vi, wi) , we expand (2) generacies are twisted cubic degeneracy, ruled quadric and have: degeneracy, and homography degeneracy. However, few   studies are given on relationships among them. We in- ···  0 0 0 0 0 0 0 0 0  vestigate relationships of the twisted cubic degeneracy uiui, uivi, uiwi, viui, vivi, viwi, wiui, wivi, wiwi f ··· with the pervious ruled quadric degeneracy and homog- N×9 raphy degeneracy, where some interesting theoretical = 0 (3) results are obtained such as a complete homography T relation between two views. where f = (f1, f2, f3, f4, f5, f6, f7, f8, f9) . The N × 9 Organization of the paper is as follows. Some pre- coefficient matrix of f is denoted by G. liminaries are listed in Section 2. The complete and uni- • Plane Homography fied degeneracy study from twisted cubic is elaborated When scene points lie on the same plane in space, 0 in Section 3. Some experimental results are reported in their corresponding image points mi and mi under two 918 J. Comput. Sci. & Technol., Sept. 2010, Vol.25, No.5 views are related by a 3 × 3 matrix H: then F can be determined uniquely by linear 8-point algorithm[13]. Otherwise, if the rank of G is 7, the so- 0 mi ≈ Hmi, i = 1..N. lution of f from (3) has one degree of freedom and the freedom can be removed by det(F ) = 0 to obtain three H is called the homography induced by this plane. or one solution. But if we only rely on the linear equa- • Twisted Cubic tions (3), the freedom cannot be removed. If the rank is The locus of points X = (X,Y,Z,T )T in a 3- 6 or less than 6, solutions of f has two or more degrees dimensional projective space satisfying the parametric of freedom and so F is under determined. The configu- equation: ration making the rank of G deficient is degenerate for (X,Y,Z,T )T ≈ A(θ3, θ2, θ, 1)T, (4) computing F . Due to noise of image data, generally we always can calculate a unique solution of f from (3) is a twisted cubic, where A is a 4 × 4 matrix and θ is with N = 8. However, the degenerate configurations or the parameter[29]. A twisted cubic is an extension of the configurations near to degeneracy will terribly in- a conic to 3-dimesional space by increasing the degree fluence the stability of the calculation for F . Therefore, of curve parameter from two to three. Generally six in order for robust estimation of fundamental matrix, points uniquely determine a twisted cubic. we need to know the degenerate configurations. The The properties of a twisted cubic underlie many of degenerate configurations from twisted cubic and the the ambiguous cases that arise in 3-dimensional recon- corresponding degeneracy degrees are provided in the struction. following theorem. • Ruled Quadric Theorem 1. If all the space points and the optical The locus of points X = (X,Y,Z,T )T in a 3- centers of two cameras are on a twisted cubic, then the dimensional projective space satisfying the equation: rank of the coefficient matrix G for computing F is five.

T If all the space points and the optical centers except one (X,Y,Z,T )B(X,Y,Z,T ) = 0, of them are on a twisted cubic, the rank of G is six. If is a quadric, where B is a 4 × 4 symmetric matrix. all the space points and the optical centers except two B can be transformed into a canonical diagonal form. of them are on a twisted cubic, the rank of G is seven. The number of positive entries in the canonical form Proof. At first we prove the case when the space is called the positive index of inertia of the quadric. points and the optical centers of the two cameras are Quadrics are classified into two classes — ruled and on a twisted cubic. unruled quadrics. A ruled quadric is one that contains According to (4), assume the parametric equation 3 2 T a straight line. For example, quadrics with positive in- of this twisted cubic is A(θ , θ , θ, 1) , in which A dex of inertia 2, namely hyperboloids of one sheet and is a 4 × 4 matrix. Since all the space points and the cones, are ruled quadrics. And, the degenerate quadrics two camera optical centers lie on this twisted cubic, let except one point case are all ruled ones[Chapter 2.2.4, 1]. the parameter of the space point M i be θi and the pa- 0 rameters of the two camera optical centers be θ0, θ0. 3 Degeneracies from Twisted Cubic When Then, Estimating Fundamental Matrix M = A(θ3, θ2, θ , 1)T, The previously known degenerate configuration of i i i i 3 2 T two views for fundamental matrix estimation or pro- O = A(θ0, θ0, θ0, 1) , 0 03 02 0 T jective reconstruction is that the two camera optical O = A(θ0 , θ0 , θ0, 1) . centers and the space points lie on a ruled quadric. For such a general ruled quadric, the right null space of G in By (1), we have: (3) is of dimension two as given in Subsection 2.1 of [16] 3 2 T and in the fifth paragraph of the introduction section 0 = PO = PA(θ0, θ0, θ0, 1) , of [26]. The more critically degenerate configuration is 0 0 0 03 02 0 T 0 = P O = P A(θ0 , θ0 , θ0, 1) , (5) from a plane, of which the right null space of G in (3) is of dimension three[16,21]. This is not the most cri- where P , P 0 are the two camera projection matrices. 0 0 tical case since the nontrivial degenerate configuration Because mi ≈ PM i, mi ≈ P M i, we also have: — twisted cubic can cause more critically degeneracy 3 2 T than a plane as shown below. mi ≈ PM i = PA(θi , θi , θi, 1) , m0 ≈ P 0M = P 0A(θ3, θ2, θ , 1)T. (6) 3.1 Degeneracy Degree from Twisted Cubic i i i i i In (3), if the rank of the coefficient matrix G is 8, Do subtraction from both sides for the first equations Yi-Hong Wu et al.: Twisted Cubic Degeneracy 919

  in (5) and (6), we obtain: c11 ··· c1j ··· c19  c21 ··· c2j ··· c29  3 2 T 3 2 T   mi ≈ PA(θi , θi , θi, 1) − PA(θ0, θ0, θ0, 1)  c31 ··· c3j ··· c39  3 3 2 2 T   = PA(θi − θ0, θi − θ0, θi − θ0, 0) c41 ··· c4j ··· c49 2 2 T c51 ··· c5j ··· c59 5×9 ≈ PA(θi + θiθ0 + θ0, θi + θ0, 1, 0) . (7)   q q q q 1 2 3 4 From the above expression, we know the rank of G is Denote PA as Q =  q q q q . Then, (7) 5 6 7 8 generally five. By now, we proved that if all the space q q q q 9 10 11 12 points and the optical centers of the two cameras are is changed to:     on a twisted cubic, the rank of the coefficient matrix G q1 q2 is generally five.   2 2   mi ≈ q5 (θi + θiθ0 + θ0) + q6 (θi + θ0)+ If all the space points and the optical centers except q9 q10   for one of them are on a twisted cubic, the proof is as q3 follows. If one of the camera optical centers does not  q7  . (8) lie on the twisted cubic determined by another camera q11 optical center and the space points, assume O is not, then the degree of θ for representing m in (6) cannot Under the first view, θ varies with m varying while q i i i i k decrease to two but the degree for representing m0 can and θ are unchanged. Similarly, do subtraction from i 0 be, i.e., m0 is still in the form (9). Thus, the degrees of both sides for the second equations in (5) and (6), there i θ in the obtained coefficient matrix G of (10) become is: i       five. Then by the same reason as above, we can prove 0 0 0 q1 q2 q3 the corresponding rank is six. If the point not lying 0  0  2 0 02  0  0  0  mi ≈ q5 (θi +θiθ0+θ0 )+ q6 (θi+θ0)+ q7 on the twisted cubic is one of the space points other 0 0 0 q9 q10 q11 than one of the camera optical centers, assume M i0 is (9) not, then the row in G from the image pair m , m0   i0 i0 0 0 0 0 q1 q2 q3 q4 is not in the polynomial form of some θ. It follows that 0 0  0 0 0 0  where Q = P A = q5 q6 q7 q8 . Under the this row is not linearly related to other rows in gen- 0 0 0 0 q9 q10 q11 q12 eral. Thus, the rank of G increases from five to six. 0 0 0 second view, θi varies with mi varying while qk and θ0 Similarly, if all the space points and the camera optical are unchanged. centers except two of them are on a twisted cubic, the Substitute (8) and (9) into (3), we get the coefficient rank of G is seven. The theorem is proved. ¤ matrix G with each element of the i-th row being a four- In the above theorem, we analyze all the possible de- order polynomial in θ as c θ4 + c θ3 + c θ2 + c θ + c . i 1 i 2 i 3 i 4 i 5 generate configurations for computing F caused from The coefficients c of θ in these four-order polynomi- s i twisted cubic. In all the cases, F cannot be determined als are functions on q , q0 , θ , θ0 . Since q , q0 , θ , θ0 k k 0 0 k k 0 0 finitely by linear 8-point algorithm and the dimensions are not varying with image pair varying, c are also not s of the right null space of G in (3) are respectively 4, varying with the row number varying. It follows that 3, 2 which give 5, 6, 7 linearly independent equations G is in this form:   on fundamental matrices. By 7-point algorithm, F still g1(θ1) g2(θ1) ··· g8(θ1) g9(θ1) cannot be solved in rank 5 or 6 cases but can be in rank  ···  7 case. G =    g (θ ) g (θ ) ··· g (θ ) g (θ )  1 i 2 i 8 i 9 i The degeneracy makes the condition number of ··· N×9 the coefficient matrix G larger and thus makes the (10) computation of F unstable. The normalized 8-point algorithm[13] could improve the computed result of where g (θ ) = c θ4 + c θ3 + c θ2 + c θ + c , j i 1j i 2j i 3j i 4j i 5j F and makes the condition number smaller to some i = 1..N, j = 1..9. We equivalently change G into: extent. However, for the twisted cubic degenerate  4 3 2  θ1 th1 θ1 θ1 1 configuration, no matter what linear transformation is  ···  performed on the image data, improvement on the con-   G ≈  θ4 θ3 θ2 θ 1  · dition number is not marginal. This is because a linear  i i i i  ··· transformation cannot change the degeneracy nature of 4 3 2 the twisted cubic. θN θN θN θN 1 N×5 920 J. Comput. Sci. & Technol., Sept. 2010, Vol.25, No.5

3.2 Relationship with Ruled Quadric where e is the vector form of ATBA with seven Degeneracy elements. This coefficient matrix is a Vandermonde matrix[30], whose determinant is always non-zero inde- The degenerate configuration of two views for re- 3 2 [1] pendent θi. So e should be zero. Therefore (θi , θi , construction is well known as a ruled quadric . The T 3 2 T θi, 1)A BA(θi , θi , θi, 1) = 0 is an identical equa- theorem in Subsection 3.1 is consistent with the ruled tion. This means if seven points of a twisted cubic lie on quadric degeneracy. In this subsection, we first give a quadric, the whole twisted cubic lies on this quadric. two lemmas about twisted cubic and ruled quadric for ¤ the consistency. Then, the contribution of our work is By Lemma 1 and Lemma 2, we can conclude that a discussed. twisted cubic plus one or two points can be embedded Lemma 1. In a 3-dimensional projective space, a in a ruled quadric. The reason is: we take seven points proper real twisted cubic can always be embedded in on the twisted cubic and combine the additional one a ruled quadric, conversely, any quadric containing a or two points to generate a quadric. This is reasonable proper real twisted cubic is a ruled one. because generally nine space points uniquely determine Proof. All proper twisted cubics are projectively a quadric. Since this quadric contains seven points of equivalent. They have the canonical representation the twisted cubic, by Lemma 2, we know it contains X : Y : Z : T = θ3 : θ2 : θ : 1. Then from the whole twisted cubic. Furthermore by Lemma 1, we these parametric equations, we have three quadrics Q : 1 know the generated quadric is ruled. It follows that the XT − YZ = 0, Q : XZ − Y 2 = 0, Q : YT − Z2 = 0. 2 3 theorem in Subsection 3.1 is consistent with the previ- So, a proper twisted cubic can lie on every quadric of ous ruled quadric degeneracy. the net Q ≡ λ1Q1 + λ2Q2 + λ3Q3 = 0. Now, we prove every quadric in this net is ruled. At first, it is easy The contribution of the theorem in Subsection 3.1 is that it gives more intuitive degeneracy and the degener- to check Q1, Q2 and Q3 are all ruled quadrics by the preliminaries in Section 2. For other Q in this net, the acy degrees of all possible cases caused by twisted cubic. λ2 For the general ruled quadric degeneracy, there are a fi- determinant of the symmetric matrix is ( 1 − λ2λ3 )2. 4 4 nite number of solutions for the fundamental matrix by It follows that the positive index of inertia of a proper combining with the additional constraint of det(F ) = 0. quadric Q is either 2 or 4. If the positive index of iner- This degeneracy degree is the same as the rank 7 case in tia is 4, Q is a virtual quadric with no real point which the theorem. For rank 5 and 6 cases in the theorem, the is impossible here[Chapter 2.2.4, 1]. Thus, the positive in- degeneracy is more critical which makes the fundamen- dex of inertia of Q is 2. So, such a proper Q is a ruled tal matrix undermind up to 4 or 3 degrees of freedom. one[chapter 2.2.4, 1]. If the determinant of Q is zero, the Even by the additional constraint det(F ) = 0, it can- quadric is degenerate. The degenerate quadric cannot not be solved. These details are not discussed in the be a point because it contains a proper twisted cubic. previous ruled quadric degeneracy. Usually, six points Thus it is also ruled according to the fact that the de- determine a unique twisted cubic and nine points deter- generate quadrics except for the one point case are all mine a unique quadric. A twisted cubic is not a class in ruled ones[Chapter 2.2.4, 1]. For any quadric containing the ruled quadrics. Therefore, from fewer non-incidence a proper real twisted cubic, it is a member of the net points in F computations, quadric degeneracy may not Q[p. 297, 29] and thereafter ruled by the above proof. come to mind but this configuration may be the twisted A ruled quadric is still a ruled one after a projective cubic degeneracy which might be ignored. However in- transformation. Then, the lemma is proved. ¤ deed the twisted cubic can make the F computation Lemma 2. In a 3-dimensional projective space, degenerate severely as shown in the theorem in Subsec- if seven points of a real proper twisted cubic lie on tion 3.1. a quadric, then the whole twisted cubic lies on the quadric. 3.3 Relationship with H-Degeneracy Proof. Let the twisted cubic be X ≈ A(θ3, θ2, θ, 1)T and a quadric be XTBX = 0. If seven The previous work closely related to ours is the H- points of the twisted cubic lie on the quadric, we have degeneracy by Chum et al.[21], where the H-degeneracy

3 2 T 3 2 T means that when scene points are all on a plane, to (θi , θi , θi, 1)A BA(θi , θi , θi, 1) = 0, i = 1..7. compute fundamental matrix between two views is degenerate. At the time, the two images are related by a This equation could be changed to homography as described in Section 2. Chum et al.[21]   ··· also discussed the degeneracy degrees for the F com-  6 5 4 3 2  putation and mentioned the twisted cubic degeneracy. θi θi θi θi θi θi 1 e = 0, ··· There are differences between our work and theirs. In Yi-Hong Wu et al.: Twisted Cubic Degeneracy 921 this subsection, we discuss the contribution of our work a homography; (ii) all image point pairs other than one relative to the study in [21]. pair are related by a homography; (iii) all image point Firstly, we give the complete cases that two views pairs other than two pairs are related by a homography. are related by a 3 × 3 homography. Then based on the degrees, they developed a DEGEN- Lemma 3. If the image point correspondences SAC algorithm to compute F unaffected by a dominant 0 (mi, mi) between two views are related by a homogra- plane by detecting H-degeneracy. 0 phy H, that is mi ≈ Hmi, then generally there are the The relationship and differences between our work following three situations: and Chum et al.’s[21] are: (i) the camera performs a pure rotation with either The cases in Theorem 1 of Subsection 3.1 related to varying or unvarying intrinsic parameters; the H-degeneracy are: (ii) the space points are coplanar; (a1) the two camera optical centers and all the space (iii) the space points and the two camera optical cen- points lie on a twisted cubic; ters lie on a twisted cubic. (a2) the two camera optical centers and all the space Proof. Let the two camera projection matrices be points except one lie on a twisted cubic; 0 K(I 0), K (R t). Then, the relation between the cor- (a3) the two camera optical centers and all the space 0 0 −1 0 responding image points is mi ≈ K RK mi + K t. points except two lie on a twisted cubic. 0 If the image point correspondences (mi, mi) are related According to Lemma 3, the two views in (a1) are by a homography H, then related by a homography, in (a2) the image point pairs (i) if t = 0, we know the camera performs a pure except one pair are related by a homography, and the 0 rotation and the relation is mi ≈ Hmi, with H = image point pairs except two pairs are related by a ho- 0 −1[p.194, 1] K RK ; mography in (a3). Although these geometric relations (ii) if t 6= 0 and all the space points lie on a space between the two views in the three cases are the same plane, this is the well known case of which the two views as Chum et al.’s, the degeneracy degrees are different. are related by a homography as described in Section Here, the degeneracy is more critical. For case (a1), [Chapter 12, 1] 2 ; since the coefficient matrix has rank 5, the linear space (iii) if t 6= 0 and not all the space points lie on a space of F has dimension 4 while in [21] for two views related plane, this is the case that all the space points and the with a homography the dimension is 3. For case (a2), two camera optical centers lie on a twisted cubic. The the corresponding dimension is 3 while that in [21] is 2. 0T reason is as follows. Since t 6= 0, there is mi F mi = 0. For case (a3), the corresponding dimension is 2 while 0 T T Also by mi ≈ Hmi, we have mi H F mi = 0. The that in [21] could be 1 if linear 8-point algorithm is ap- T T symmetric matrix for the conic mi H F mi = 0 is plied. It follows that the twisted cubic cases could cause T T H F +F H C = 2 . If C = 0, all space points lie on more critical degeneracy than the plane cases, though a plane[p.314, 1] which is not consistent with the condi- they have the same geometric H-relations between the tion of this case (iii). If C 6= 0 and det(C) = 0, the two views. image locus is a degenerate conic which consists of two The cases in the theorem of Subsection 3.1 not in- lines or one line. We discuss under the general case volved in [21] are: and so do not consider this case of lines. If C 6= 0 and (b1) All the space points and one of the camera opti- det(C) 6= 0, the image locus is a proper conic. Simi- cal centers lie on a twisted cubic. But the other camera larly in the second view, the image locus is a proper optical center is not on this twisted cubic. −T T −1 0 T 0 0 0 H F +FH (b2) All the space points except one and one of the conic (mi) C mi = 0 with C = 2 . Let the two epipoles in the two views be e and e0. Be- camera optical centers lie on a twisted cubic. But the cause F e = 0 and e0TF = 0, we find eTCe = 0 and other camera optical center is not on this twisted cubic. e0TCe0 = 0. It follows that the re-back projection of (b3) All the space points but the two camera optical the two conics is two quadric cones with a common tan- centers lie on a twisted cubic. gent line OO0 and with the respective vertexes O, O0. The three cases do not fall into the work of Chum The line OO0 is a common generator of the two quadric et al.[21]. In the three cases at least one of the optical cones. Thus, the intersection of the two quadric cones centers does not lie on the twisted cubic and the space is a twisted cubic containing O, O0[29,31]. points are also not coplanar. Thus according to Lemma The above classification conditions are complete and 3 all or most of the image point pairs in each case do so are the three cases. ¤ not agree to a homography relation. In [21], Chum et al. analyzed the degrees of the H- Therefore, our work not only improves the work in degeneracy on three cases: (i) two views are related by [21] but also makes some new contribution. 922 J. Comput. Sci. & Technol., Sept. 2010, Vol.25, No.5

4 Experiments matrix of these equations could be computed out the rank is ﬁve. We performed both simulations and experiments on real data. The results verify the established theorem. One group of the experiments is reported below.

4.1 Simulations

The parametric equation of a space twisted cubic is:     2 5 −3 2.5 θ3  1 −1 12 1   θ2  M ≈     . (11)  6 −15 −2 3   θ  −7 5 3 2 1 Fig.2. A simulated image when the optical center and space Ten points M , i = 1..10 on this twisted cubic are i points M i, i = 1..10, lie on a twisted cubic. taken, of which the parameters are −1.1, −0.35, −0.75, −0.22, −0.6, 0.1, −0.1, 0.2, 1.9, −2 respectively. At ﬁrst, we consider the case that both the two optical centers and the space points lie on the same twisted cubic. Take the two points of the twisted cubic with parameters 1.25, 1.5 as the two optical centers O, O0. The space point distribution is shown as Fig.1.

Fig.3. Another simulated image when the optical center and the

same M i, i = 1..10, lie on the same twisted cubic as Fig.2. We also tested all the other cases in Theorem 1 of Subsection 3.1 under noise level zero and the results validate the rank conclusions. According to the simulations, we find that the direct computation on the matrix rank or the rank computa- Fig.1. The space points and the two optical centers lie on a tion by the singular values is only correct in the absence twisted cubic, where ∗ denotes the space points, and ◦ denotes of noise. When we add noise to the image, the rank of the camera optical centers. the coefficient matrix becomes 8 and the computation becomes very unstable, that means the computation is Then, the corresponding camera projection matrices very sensitive to noise. Therefore, in order to robustly consistent with the optical centers are set as follows: estimate the fundamental matrix, it is necessary to de-   velop a method of detection on the degenerate config- 1000 0 512 43198 uration. We will explore a detection method on the P =  0 900 384 95484  , degeneracy caused from twisted cubic and then can ap- 0 0 1 −103   ply the RANSAC on degenerate data like in [23, 28] to −529 −648.1 287.4 −4321.3 robustly compute the fundamental matrix. P 0 =  338.6 −295.4 748.7 −1810.1  . −0.7 −0.1 0.7 −3.9 4.2 Experiments on Real Data

We generated two simulated images of the ten space The degeneracy of six points from real data is also points and the results are shown in Figs. 2 and 3. We tested. The experiments of more points on real data established the equations on the fundamental matrix need to be performed after the detection on degenerate between the two images from the data of Figs. 2 and 3. data and the corresponding RANSAC are proposed. Under the noise level of zero, the rank of the coefficient We took the images of six space points at different Yi-Hong Wu et al.: Twisted Cubic Degeneracy 923 viewpoints. Four of them with the size of 640 × 480 in Figs. 4 and 5 are 1.0655, 1.0504, 2.2934, and 2.5091, pixels are shown in Figs. 4 and 5. In order to know respectively. Then, by the method in [23], we know whether the six space points and the corresponding op- that the six points and the two corresponding optical tical center lie on a twisted cubic, we measured the centers in Fig.4 are on the same twisted cubic, while, space coordinates of the six points and then by the cri- the six points and the two corresponding optical cen- terion function proposed in [23] detected the situation. ters in Fig.5 are not. We have computed the condition The values of the criterion function on the four images numbers of the coefficient matrices when estimating F in Figs. 4 and 5 and then tested the degeneracy ranks. However, usually we cannot compute singular values to get condition numbers, then to find the matrix ranks because singular values are very sensitive to noise and presetting a threshold to discriminate the degeneracy from the non-degeneracy is not easy, as pointed out in [32]. Also, we found sometimes condition number of the degeneracy is yet smaller than that of the non-degeneracy. This is why we would like to pursue a detection method for the degeneracy from two image data in the future.

5 Conclusion

This paper provides all the possible degenerate con- ﬁgurations caused by a twisted cubic and the corresponding degeneracy degrees for estimating fundamental matrix. Relationships with the ruled quadric degeneracy and the homograghy degeneracy are also reported. In future robust detection on the twisted cubic conﬁg- urations will be investigated.

Fig.4. Two images of six points, of which the space points and References the two camera optical centers are on a same twisted cubic. [1] Hartley R, Zisserman A. Multiple View Geometry in Com- puter Vision. Cambridge University Press, 2000. [2] Hartley R, Kahl F. Critical configurations for projective reconstruction from multiple views. International Journal of Computer Vision, 2006, 71(1): 5-47. [3] Horn B.K.P. Relative orientation. International Journal of Computer Vision, 1990, 4(1): 59-78. [4] Kahl F, Triggs B, Astrom K. Critical motions for auto- calibration when some intrinsic parameters can vary. Journal of Mathematical Imaging and Vision, 2000, 13(2): 131-146. [5] Luong Q T, Faugeras O. Self-calibration of a moving camera from point correspondences. International Journal of Com- puter Vision, 1997, 22(3): 261-289. [6] Maybank S, Faugeras O. A theory of self-calibration of a moving camera. International Journal of Computer Vision, 1992, 8(2): 123-151. [7] Weng J, Huang T S, Ahuja N. Motion and structure from two perspective views: Algorithms, error analysis, and error estimation. IEEE Transactions on Pattern Analysis and Ma- chine Intelligence, 1989, 11(5): 451-476. [8] Xu G, Zhang Z. Epipolar Geometry in Stereo, Motion and Object Recognition: A Unified Approach. Kluwer Academic Publishers, Norwell, MA, USA, 1996. [9] Zhang Z, Deriche R, Faugeras O, Luong Q T. A robust technique for matching two uncalibrated images through the recovery of the unknown epipolar geometry. Artificial Intelligence Journal, 1995, 78(1/2): 87-119. Fig.5. Two images of six points, of which the space points and [10] Bartoli A, Sturm P. Non-linear estimation of the fundamen- the two camera optical centers are not on a same twisted cubic. tal matrix with minimal parameters. IEEE Transactions on 924 J. Comput. Sci. & Technol., Sept. 2010, Vol.25, No.5

Pattern Analysis and Machine Intelligence, 2004, 26(4): 426- Computer Vision, Bombay, India, Jan. 4-7, 1998, pp.703-708. 432. [28] Fischler M A, Bolles R C. Random sample consensus: A [11] Bober M, Georgis N, Kittler J. On accurate and robust esti- paradigm for model fitting with applications to image analysis mation of fundamental matrix. Computer Vision and Image and automated cartography. Communications of the ACM, Understanding, 1998, 72(1): 39-53. 1981, 24(6): 381-395. [12] Frahm J, Pollefeys M. RANSAC for (quasi-)degenerate data [29] Semple J G, Kneebone G T. Algebraic Projective Geometry. (QDEGSAC). IEEE Computer Society Conference on Com- Oxford University Press, 1952. puter Vision and Pattern Recognition, New York, USA, June [30] Hoffman K, Kunze R. Linear Algebra, 2nd Edition. Engle- 17-22, 2006, pp.453-460. wood Cliffs, NJ: Prentice Hall, 1971. [13] Hartley R. In defense of the eight-point algorithm. IEEE [31] Wu Y, Hu Z. Invariant representations of a quadric cone and Transactions on Pattern Analysis and Machine Intelligence, a twisted cubic. IEEE Transactions on Pattern Analysis and 1997, 19(6): 580-593. Machine Intelligence, 2003, 25(10): 1329-1332. [14] Luong Q T, Faugeras O. The fundamental matrix: Theory, [32] Kahl F, Henrion D. Globally optimal estimates for geomet- algorithms, and stability analysis. International Journal of ric reconstruction problems. In Proc. ICCV 2005, Beijing, Computer Vision, 1996, 17(1): 43-75. China, Oct. 17-20, 2005, pp.978-985. Also in International [15] Torr P H S, Murray D W. The development and compari- Journal of Computer Vision, 2007, 74(1): 3-15. son of robust methods for estimating the fundamental matrix. International Journal of Computer Vision, 1997, 24(3): Yi-Hong Wu is currently a pro- 271-300. fessor at the Institute of Automa- [16] Torr P H S, Zisserman A, Maybank S J. Robust detection of tion of the Chinese Academy of Sci- degenerate configurations while estimating the fundamental matrix. Computer Vision and Image Understanding, 1998, ences. She received her Ph.D. de- 71(3): 312-333. gree from the Institute of Systems [17] Zhang Z. Determining the epipolar geometry and its uncer- Science, the Chinese Academy of Sci- tainty: A review. International Journal of Computer Vision, ences in 2001. Her research inter- 1998, 27(2): 161-195. ests include geometric invariant ap- [18] Gurdjos P, Bartoli A, Sturm P. Is dual linear self-calibration plication, camera calibration, camera artificially ambiguous? International Conference on Com- puter Vision, Kyoto, Japan, Sept. 29-Oct. 2, 2009, pp.88-95. pose determination, image matching, [19] Sturm P. A case against Kruppa’s equations for camera self- and 3D reconstruction. calibration. IEEE Transactions on Pattern Analysis and Ma- chine Intelligence, 2000, 22(10): 1199-1204. Tian Lan is currently a Ph.D. [20] Sturm P. Critical motion sequences for the self-calibration of candidate at the Institute of Au- cameras and stereo systems with variable focal length. Image tomation, the Chinese Academy of and Vision Computing, 2002, 20(5/6): 415-426. Sciences. He received his Bachelor’s [21] Chum O, Werner T, Matas J. Two-view geometry estimation unaffected by a dominant plane. In Proc. IEEE Con- degree from Huazhong University of ference on Computer Vision and Pattern Recognition, San Science and Technology in 2004. Diego, CA, USA, June 20-26, 2005, pp.772-779. [22] Buchanan T. The twisted cubic and camera calibration. Com- puter Vision, Graphics and Image Processing, 1988, 42(1): 130-132. [23] Wu Y, Li Y, Hu Z. Detecting and handling unreliable points for camera parameter estimation. International Journal of Computer Vision, 2008, 79(2): 209-223. Zhan-Yi Hu is currently a pro- [24] Maybank S. Theory of Reconstruction from Image Motion. fessor at the Institute of Automation, Springer-Verlag, 1992. the Chinese Academy of Sciences. He [25] Luong Q T, Faugeras O. A stability analysis of the funda- received his Ph.D. degree from the mental matrix. In Proc. European Conference on Computer University of Liege, Belgium in 1993. Vision, Prague, Czech, May 11-14, 1994, pp.577-588. His research interests are in com- [26] Hartley R. Ambiguous configurations for 3-view projective re- puter vision, including camera cali- construction. In Proc. European Conference on Computer Vision, Dublin, Ireland, Jun. 26-Jul. 1, 2000, pp.922-935. bration, 3D reconstruction, active vi- [27] Maybank S, Shashua A. Ambiguity in reconstruction from sion, and geometric primitive extrac- images of six points. In Proc. International Conference on tion.