Cayley Transformation and Numerical Stability of Calibration Equation

Int J Comput Vis (2009) 82: 156–184 DOI 10.1007/s11263-008-0193-x

F.C. Wu · Z.H. Wang · Z.Y. Hu

Received: 25 May 2008 / Accepted: 22 October 2008 / Published online: 5 December 2008 © US Government 2008

Abstract The application of Cayley transformation to en- it is proved that the standard calibration equation, the Cay- hance the numerical stability of camera calibration is inves- ley calibration family and the S-Cayley calibration family tigated. First, a new calibration equation, called the standard are all some special cases of this generic calibration family. calibration equation, is introduced using the Cayley transformation and its analytical solution is obtained. The standard Keywords Camera calibration · The absolute conic · calibration equation is equivalent to the classical calibration Calibration equation · Cayley transformation · Numerical equation, but it exhibits remarkable better numerical stabil- stability ity. Second, a one-parameter calibration family, called the Cayley calibration family which is equivalent to the standard calibration equation, is obtained using also the Cayley 1 Introduction transformation and it is found that this family is composed Camera calibration is a necessary step to recover 3D met- of those infinite homographies whose rotation has the same ric information from 2D images. Many methods and tech- axis with the rotation between the two given views. The con- niques for camera calibration have been proposed in the last dition number of equations in the Cayley calibration family decades. The existing methods can be roughly classified into varies with the parameter value, and an algorithm to deter- two categories: methods based on a reference calibration mine the best parameter value is provided. Third, the gen- object and self-calibration methods. In the first kind, there eralized Cayley calibration families equivalent to the stan- are mainly the methods based on 3D object (Brown 1971; dard calibration equation are also introduced via general- Faig 1975;Tsai1986; Faugeras 1993), 2D pattern (Zhang ized Cayley transformations. An example of the generalized 1999, 2000; Sturm and Maybank 1999) and 1D object Cayley transformations is illustrated, called the S-Cayley (Zhang 2004;Wuetal.2005; Wang et al. 2007). In these calibration family. As in the Cayley calibration family, the methods, camera calibration is performed by observing a numerical stability of equations in a generalized Cayley cal- 3D calibration object, a 2D calibration pattern or a 1D ibration family also depends on the parameter value. In addi- calibration object. Since the Euclidean geometries of cal- tion, a more generic calibration family is also proposed and ibration objects are used, the methods can provide a calibration of high-precision. In the second kind, the methods widely used are the calibration based on the Kruppa equa- · · F.C. Wu ( ) Z.H. Wang Z.Y. Hu tion (Faugeras et al. 1992; Maybank and Faugeras 1992; National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, PO Box 2728, Hartley 1997a; Luong and Faugeras 1997), the absolute Beijing 100190, People’s Republic of China conic (Pollefeys et al. 1996; Hartley 1997b; Pollefeys and e-mail: [email protected] Gool 1999) and the dual absolute quadric (Triggs 1997; Z.H. Wang Ponce et al. 2000). These methods do not need any refer- e-mail: [email protected] ence calibration objects and use only the correspondences of Z.Y. Hu image entities, such as points, lines. It is remarkable and de- e-mail: [email protected] sirable that the self-calibration methods require no reference Int J Comput Vis (2009) 82: 156–184 157 objects. Regrettably, in addition to their low calibration pre- family, the numerical stability of equations in this general cision, the numerical stabilities of self-calibration methods family varies also with the parameter value. An example are usually poor. Hartley and Zisserman (2000) pointed out of the generalized Cayley transformations is illustrated, that the algorithms absolutely depended on self-calibration called the S-Cayley calibration family. should be restrainedly used in practice. Though the people • A more generic calibration family is proposed and it is have paid great attention to improve the precision and the proved that the standard calibration equation, the Cayley numerical stability of self-calibration methods, no satisfac- calibration family and the S-Cayley calibration family are tory results are obtained up to now. all some special cases of this generic calibration family. In this paper, we mainly investigate the numerical sta- This paper is organized as follows. Section 2 reviews bilities of the self-calibration methods based on the ab- briefly the classical calibration algorithm based on the absolute conic. The calibration equation induced by the absolute conic. In the Sect. 3, the Cayley transformation is in- T = T = −1 solute conic, H∞ωH∞ ω or, equivalently H∞ω ωH∞ , troduced. The standard calibration equation, the Cayley cal- is called the classical calibration equation in this paper. ibration family, and the generalized Cayley calibration fam- Here H∞ is the infinite homography and ω the image of ily are elaborated in Sects. 4, 5 and 6, respectively. Section 7 the absolute conic. The previous studies are mainly fo- gives more generic equivalent equations. Section 8 reports cused on the conditions and computational means to solve the experiment results. Section 9 concludes this paper. the infinite homography (such as, Pollefeys et al. 1996; Pollefeys and Gool 1999; Hartley 1997b, etc.). For a long time the numerical stability of the calibration equation itself 2 Classical Calibration Equation has been neglected in research community. In computation term, generally speaking different forms of the same equa- The infinite homography between the two views is denoted tion can usually have different numerical stabilities. Thus we as H∞ in this paper. If the motion between the two views is have the following problem: For the self-calibration based (R, t), R is the rotation and t the translation, and the intrin- on the absolute conic, do there exist more stable forms than sic matrix of camera is K, then the infinite homography can the classical calibration equation in computation? This prob- be expressed as lem is answered affirmatively in this paper. More specifi- = −1 cally, we use the Cayley transformation to enhance the nu- H∞ KRK . (1) merical stability of camera calibration and here are our main Let ω be the image of the absolute conic (IAC) i.e., ω = contributions: K−T K−1. Then there is the classical calibration equation • Using the Cayley transformation of the infinite homogra- (CCE): phy, a new calibration equation, called the standard cali- T H ωH∞ = ω. (2) bration equation, is introduced and its analytical solution ∞ is obtained. The Cayley transformation of the infinite ho- Evidently, this equation can be changed to the following mography transforms image point to its conjugate point equivalent form: on the image of absolute conic. The standard calibration T −1 equation is equivalent to the classical calibration equa- H∞ω = ωH∞ . (3) tion, but it exhibits remarkable better numerical stability. For the convenience of statement, (2) is called CCE of the • A one-parameter family equivalent to the standard cali- first type (CCE-I), and (3) is called CCE of the second type bration equation, called the Cayley calibration family, is (CCE-II). introduced using also the Cayley transformation and it is Hartley (1997b) used the CCE-II to compute ω and shown that the Cayley calibration family is composed of pointed out that the CCE can only provide 4 independent those infinite homographies whose rotation has the same constraints on ω. Thus at least 3 views are necessary to lin- axis with the rotation between the two given views. The early solve ω.Afterω is determined, the intrinsic matrix K condition number of equations in the Cayley calibration can be determined by the Cholesky factorization of ω, then family varies with the parameter value, and an algorithm the rotation R can in turn be determined. The classical cali- is also provided to determine the best parameter which bration algorithm can be sketched as: gives the best numerical stability among the Cayley family. Classical Algorithm • Generalized Cayley calibration family equivalent to the standard calibration equation is introduced via general- Goal Given N views of a scene and assume the infinite ho- ized Cayley transformations. Given a generalized Cay- mographies Hi∞ between the 0th and ith views are known. ley transformation, a generalized Cayley calibration fam- Compute the intrinsic matrix K and the rotation Ri between ily can be obtained. Similarly to the Cayley calibration the 0th and ith views. 158 Int J Comput Vis (2009) 82: 156–184 Algorithm The algorithm consists of the following three Here e is the epipole on the 2nd view, steps: F T e = 0, ˜ −1/3 1. Let Hi∞ = Det (Hi∞)Hi∞. Compute the least squares solution of the following set of linear equations a is an unknown 3-vector. Given an infinite correspondence on ω: m(0) ↔ m(1),wehave: − ˜ T ˜ = ˜ T = ˜ 1 (1) T (0) Hi∞ωHi∞ ω(or Hi∞ω ωHi∞), [m ]×([e ]F + e a )m = 0. i = 1, 2,...,N − 1. In the above three equations on a, there are only one in- 2. Determine the intrinsic matrix K by taking the Cholesky dependent constraint. And thus, at least 3 infinite correspon- factorization of ω. dences are necessary to determine linearly the vector a. 3. Determine the rotation Ri by finding the best rota- ˜ K = −1 ˜ tion approximation of Hi∞ K Hi∞K, i.e., com- Method 3 Computation using the modulus constraint. For pute firstly the singular value decomposition of the more than 2 views, using the modulus constraint shown in ˜ K ˜ K = T matrix Hi∞, Hi∞ Udiag(t1,t2,t3)V , then set up Pollefeys et al. (1996), i.e., the three eigenvalues of infinite T Ri = UV . homography have the same modulus, the infinite homographies between views can also be derived without resort to the Remark 1 Since the infinite homography computed in prac- infinite correspondences. But it needs to solve a set of quar- − tice is equal to matrix KRK 1 up to a scale factor, it is tic equations in three variables, and thus is highly non-linear. necessary that the determinant of the computed infinite ho- The computation steps are as follows. First, by the trifocal mography be normalized to 1 in the step 1, otherwise (2) tensor method in Hartley and Zisserman (2000)orthematrix and (3) do not hold. factorization methods in Sturm and Triggs (1996), Oliensis and Hartley (2007), a projective reconstruction is computed Computing Infinite Homography with a set of image correspondences from 3D space points:

Here, we sketch three different methods for the infinite ho- P0 = (I|0), Pi = (Hi|ei)(j= 1, 2,...,N − 1). mography computation. Then, the infinite homography between the 0th view and ith Method 1 Computation with infinite correspondences (i.e., view is expressed as: image point correspondences from infinite plane). Given = + T = − two views of a scene and a set of infinite correspondences: Hi∞ Hi eia (i 1, 2,...,N 1). (0) ↔ (1) = Here a is an unknown 3-vector, the direction of infinite plane mj mj ,j 1, 2,...,M. in the reconstructed projective space. And thus, the vector a We can linearly compute the infinite homography H∞ from can be non-linearly computed by using the modulus con- the following set of linear equations: straints. [ (1)] (0) = = mj ×H∞mj 0 (j 1, 2,...,M). 3 Cayley Transformation This method needs at least 4 infinite correspondences because each infinite correspondence can only give rise to two The following function of complex variable is called the independent constraints on H∞. Cayley transformation (Bell 1992): Hartley (1997b) pointed out that if the motion between 1 − λ two views is a rotation around the camera center, then infi- ξ = ϕ(λ) = . (4) nite correspondences in the above can be replaced by image 1 + λ correspondences from 3D space points. It is a conformal mapping on the extended complex plane. It is ready to verify that Method 2 Computation with the epipolar geometry. Given the foundational matrix between two views, F , which can 1 − ξ λ = ϕ−1(ξ) = = ϕ(ξ). be computed with image correspondences from 3D space 1 + ξ points, three infinite correspondences are needed in order So, the inverse transformation is also the Cayley transforma- to linearly compute the infinite homography H∞.Thisis because the infinite homography H∞ can be expressed as: tion. For a matrix A ∈ Cn×n,letσ(A) denote the set of all T H∞ =[e ]F + e a . eigenvalues of A. Assume −1 ∈/ σ(A) i.e., det(I + A) = 0, Int J Comput Vis (2009) 82: 156–184 159 then the Cayley transformation of A is defined by By right-multiplying the two sides of this equation by C, and notice that BT C + CB = 0, have B = ϕ(A) = (I − A)(I + A)−1 AT C(I − B) = C(I + B). − = (I + A) 1(I − A). (5) So, From this equation, we have (I + A)B = (I − A) or, equiv- T = T − + −1 alently, A CA A C(I B)(I B) = C(I + B)(I + B)−1 = C. A(I + B) = (I − B). (6) Hence, the necessary condition holds too. If Bx =−x, then by (5) there must be x = 0, and thus, −1 ∈/ σ(B). Therefore, from (6)wehave 4 Standard Calibration Equation A = ϕ(B) = (I − B)(I + B)−1 − 4.1 Cayley Transformation of H∞ = (I + B) 1(I − B). (7) The infinite homography H∞ satisfies This indicates that the matrix A is also the Cayley transfor- −1 mation of B. det(I + H∞) = det(K(I + R)K ) = det(I + R). According to the above discussions, the Cayley transfor- ∈ mation of matrices is a one-to-one mapping on the matrix If the rotation angle of R is not equal to iπ (i Z), + = set {A ∈ Cn×n|−1 ∈/ σ(A)} and we can prove that: det(I R) 0. So, in this case, there is the Cayley transformation of H∞: − 1 λ −1 λ ∈ σ(A) ⇔ β = ∈ σ (ϕ(A)). (8) H = ϕ(H∞) = (I − H∞)(I + H∞) 1 + λ −1 = (I + H∞) (I − H∞). (10) For the Cayley transformation of matrices, we have the following proposition, which is the fundamental basis of our And H∞ is also the Cayley transformation of H : current work. −1 H∞ = ϕ(H) = (I − H )(I + H) × Proposition 1 Assume matrices A,B ∈ Cn n satisfy (5) − = (I + H) 1(I − H). (11) and (7) and let C ∈ Cn×n be an arbitrary matrix, then, In this paper, we always assume that the rotation angle θ of T T A CA = C ⇔ B C + CB = 0. (9) R is not equal to iπ (i ∈ Z).

Proof By left-multiplying the two sides of (5)by(I + A), Proposition 2 H is a conjugate of an anti-symmetric ma- and transposed, we obtain trix.

T T T − B (I + A) = (I − A) . Proof Substituting H∞ = KRK 1 into (10) yields

−1 −1 −1 By right-multiplying the two sides of this equation by CA, H = (I − KRK )(I + KRK ) T and notice that A CA = C,wehave = − + −1 −1 = −1 K(I R)(I R) K Kϕ(R)K . (12) BT C(I + A) =−C(I − A). ϕ(R) Where ϕ(R) is the Cayley transformation of rotation R. So, Since T −1 − B C =−C(I − A)(I + A) =−CB. (ϕ(R))T = (I + RT ) 1(I − RT ) − − − Therefore the sufficient condition holds. Now, we prove the = (I + R 1) 1(I − R 1) necessary condition next. − − = (R + I) 1RR 1(R − I) By left-multiplying the two sides of (7)by(I + B), and transposed, we obtain = (R + I)−1(R − I) =− AT (I + B)T = (I − B)T . ϕ(R), 160 Int J Comput Vis (2009) 82: 156–184 ϕ(R) is an anti-symmetric matrix, and thus by (12)the Here q¯ is the conjugate of q. Then, we obtain proposition holds. ¯ = ¯ − H(p1, q, q) (p1, q, q)diag(0,is, is). Let the eigenvalues of rotation R be {1,eiθ,e−iθ}. Then it is not difficult to see that ϕ(R) has the eigenvalues By right-multiplying the two sides of this equation by { − } ⎛ ⎞ 0,itan(θ/2), i tan(θ/2) and has the same eigenvectors 10 0 with R, i.e., ⎜ √ √ ⎟ = ⎝ ⎠ ⎧ ⎧ U 01/√21/i√2 , ⎨ϕ(R)x = 0 ⎨Rx = x 01/ 2 −/i 2 = ⇔ = iθ ⎩ϕ(R)y i tan(θ/2)y ⎩Ry e y ϕ(R)y = i tan(θ/2)y ϕ(R)y = e−iθy. since q + q¯ q − q¯ According to Proposition 2, H has also the eigenvalues ¯ = √ √ ∈ 3×3 (p1, q, q)U p1, , R and {0,itan(θ/2), −i tan(θ/2)} and the corresponding eigenvec- 2 i 2 tors are {Kx,Ky,Ky}. ¯ − = ¯ [ ] (p1, q, q)diag(0,is, is)U (p1, q, q)U a × q + q¯ q − q¯ Corollary 1 = √ √ [ ] p1, , a ×, (a) rank(H ) = 2. 2 i 2 K −1 (b) Let H = K HK. Then the rotation R is the Cayley we have transformation of H K : q + q¯ q − q¯ q + q¯ q − q¯ − H p , √ , √ = p , √ , √ [a]× R = ϕ(HK ) = (I − H K )(I + H K ) 1. (13) 1 1 2 i 2 2 i 2 P P Proposition 3 Let the eigenvalues of H be {0,is,−is}, then, and thus (14) holds. We now prove (b). Assume H = − P [a]×P 1 is another standard anti-symmetric factoriza- (a) H can be factorized as − − tion, then P [a]×P 1 = P [a]×P 1, and thus, = [ ] −1 H P a ×P , (14) −1 −1 P P[a]× −[a]× P P = 0. where P is a real matrix, [a]× is the anti-symmetric ma- W W trix defined by the vector a = (s, 0, 0)T , i.e. ⎛ ⎞ According to Lemma 1 below, solutions of W are 000 ⎛ ⎞ ⎝ ⎠ x 00 [a]× = 00s . ⎜ ⎟ = + [ ] + [ ]2 = − 2 0 −s 0 W xI y a × z a × ⎝ 0 x zs ys ⎠ , 0 −ys x − zs2 Equation (14) is called the standard anti-symmetric factorization of H in this paper. x(y2s2 + (x − zs2)2) = 0. (b) If H has another standard anti-symmetric factorization − = = − 2 = H = P [a]×P 1, then P must be of the form Let α x,β x zs , γ ys, then, −1 P = PW(α,β,γ), (15) W = W(α,β,γ)= P P . where Hence, P = PW. ⎛ ⎞ α 00 Lemma 1 Assume c is a nonzero 3-vector, then solutions of W = ⎝ 0 βγ⎠ the equation [c]×X − X[c]× = 0 are: 0 −γβ 2 X = xI + y[c]× + z[c] , x,y,z∈ R. (16) is an invertible matrix. × ∈ n×n { ¯} Proof Suppose A,B R and let Proof Let p1, q, q be the unit eigenvectors of H corre- { − } sponding to the eigenvalues 0,is, is ,i.e. vec(B) = (bT , bT ,...,bT )T , ⎧ 1 2 n ⎨H p = 1 0 where bT is the ith row of B, then we can obtain = i ⎩H q isq ¯ =− ¯ Hq isq. vec(AB) = (A ⊗ In)vec(B), (17) Int J Comput Vis (2009) 82: 156–184 161

T vec(BA) = (In ⊗ A )vec(B). (18) (b) Symmetrical positive-deﬁnite solutions of the SCE are: − − Here ⊗ denotes the tensor product of matrices. By applying ω = xP T diag(1,w,w)P 1,x>0,w>0. (22) (17) and (18), [c]×X − X[c]× = 0 can be changed as − Proof Substituting H = P [a]×P 1 into (20) and by some

([c]× ⊗ I3 + I3 ⊗[c]×)vec(X) = 0. (19) simple manipulations, we have T T −[a]×P ωP + P ωP[a]× = 0. (23) Since the three eigenvalues of anti-symmetric matrix [c]× { 2 − 2} are 0,i c , i c , the coefﬁcient matrix of (19) has By denoting X = P T ωP in (23), we have eigenvalue zero of multiplicity 3. It is easy to verify that 2 {vec(I), vec([c]×), vec([c]×)} are three linear independent [a]×X − X[a]× = 0. (24) solutions of (19). And thus its solutions are According to Lemma 1, its solutions are = + [ ] + [ ]2 vec(X) xvec(I) yvec( c ×) zvec( c ×), 2 X = xI + z[a]× + y[a]×, x,y,z∈ R. x,y,z ∈ R. Since It yields that the solutions of [c]×X − X[c]× = 0are X − XT = 0 2 X = xI + y[c]× + z[c]×, x,y,z∈ R. 2 ⇔ (xI + z[a]× + y[a]×) 2 T − (xI + z[a]× + y[a] ) = 0 4.2 Standard Calibration Equation × ⇔ 2z[a]× = 0 ⇔ z = 0, By Proposition 1, the following proposition holds. symmetrical solutions of (24)are

2 Proposition 4 The CCE-I (or CCE-II) is equivalent to the X = xI + y[a]×,x,y∈ R. equation below: 2 2 2 Notice that [a]× = diag(0, −s , −s ),wehave H T ω + ωH = 0. (20) X = diag(x, x − ys2,x− ys2), x, y ∈ R.

This equation is called the standard calibration equation Hence symmetrical solutions of the SCE are (SCE) in this paper, which has the similar form with the − − CCE-II. Though the CCE-I and the SCE are equivalent to ω = P T diag(x, x − ys2,x− ys2)P 1,x,y∈ R. each other, their forms have remarkable differences. First, the CCE-I is of a symmetrical form, while the SCE is of an Evidently, ω is a symmetrical positive-definite matrix if and anti-symmetric form because it implies that the matrix ωH only if X is a symmetrical positive-definite matrix, and thus is anti-symmetric. Second, the CCE-I is homogeneous on symmetrical positive-definite solutions of the SCE are only the matrix ω, while the SCE is homogeneous on not − − − ω = P T diag(x, x −ys2,x−ys2)P 1,x>0,y0,w>0. the following proposition. Remark 2 The symmetrical solutions (21) and symmetrical − positive-definite solutions (22) of the SCE do not depend Proposition 5 Let H = P [a]×P 1 is a standard anti- P symmetric factorization of H (see Proposition 3), then, on the specific selection of matrix in the standard anti- symmetric factorization of H . This is because: Assume H = − (a) Symmetrical solutions of the SCE are: P [a]×P 1 is another standard anti-symmetric factorization, then according to Proposition 3(b), P = PW(α,β,γ). − − ω = P T diag(x, x − ys2,x− ys2)P 1, It is not difficult to verify that ∈ x,y R. (21) Wdiag(a,b,b)= diag(a,b,b)W, ∀a,b 162 Int J Comput Vis (2009) 82: 156–184 or, equivalently, Proof By Proposition 5, the symmetrical positive-definite solutions of H T ω + ωH = 0are − i i Wdiag(a,b,b)W 1 = diag(a,b,b), ∀a,b. −T −1 − − − − ω = x P ( ,w ,w )P (x > ,w > ), i = , . And thus, P T diag(a,b,b)P 1 = P T diag(a,b,b)P 1. i i diag 1 i i i i 0 i 0 1 2 Hence, the symmetrical solutions (21) and symmetrical (27) positive-definite solutions (22) do not depend on the selec- = tion of matrix P in the standard anti-symmetric factoriza- Assume x1 1. Then from (27), we have tion. T diag(1,w1,w1) = x2Q diag(1,w2,w2)Q = T + T + T Proposition 5 tells us that from the SCE of a pair of views, x2(q1q1 w2(q2q2 q3q3 )). the IAC ω can be determined up to a one-parameter family. It is easy to see that this equation is equivalent to the follow- Thus under the 4-parameter camera model with zero skew, ing equation: the IAC ω can be uniquely determined from the SCE of a ⎧ pair of views and its computation is very easy with (22): ⎪ × = ⎨diag(1,w1,w1)(q2 q3) x2 det(Q)q1 × =− diag(1,w1,w1)(q1 q3) w2 det(Q)q2 (28) = −T − q11q12 ⎩⎪ ω P diag 1, , diag(1,w ,w )(q × q ) = w det(Q)q . q21q22 + q31q32 1 1 1 2 2 3 q q (a) If the rotation axes of the two rotations are parallel to − 11 12 P −1, q21q22 + q31q32 each other, then from the proof of Propositions 2 and 3,the first rows of P1 and P2 are equal up to a nonzero scale. And −1 where qij is the (i, j)-element of P . Given two SCEs thus Q can be expressed as from three views, the following proposition provides an an- ⎛ ⎞ ∗∗ alytical solution of ω. α = −1 = ⎝ ∗∗⎠ = Q P2 P1 0 ,α0. 0 ∗∗ Proposition 6 Assume Ri is the rotation between the 0th and ith views and the corresponding SCE is: Hence, (28) is equivalent to the equation below: ⎧ T + = = = Hi ω ωHi 0,i1, 2. (25) ⎨det(Q)x2 α =− =− ⎩Q22w1 q22w2,Q23w1 q23w2 Then has a unique solution if and only if that the (25) Q32w1 =−q32w2,Q33w1 =−q33w2. rotation axes of the two rotations are not parallel. And if we let the standard anti-symmetric factorization of Hi be Clearly, its solution is not unique, and thus the solution of = [ ] −1 (25) is not unique either. Hi Pi ai ×Pi , then its unique solution is: (b) If the rotation axes of two rotations are not parallel, − − = T 1 then the two elements q21 and q31 of Q are not all zero. ω P1 diag(1,w1,w1)P1 Rewriting the 2nd and 3rd equations in (28)as = −T −1 x2P2 diag(1,w2,w2)P2 , (26) (Q21,Q22w1,Q23w1) =−det(Q)(q21w2,q22w2,q23w2) where (Q31,Q32w1,Q33w1) = det(Q)(q31w2,q32w2,q33w2). 2 + 2 3 2 + 2 Since q2 + q2 = 0, we have (Q21 Q31) j=2(Q2j Q3j ) 21 31 w = , 1 2 + 2 3 2 + 2 (q21 q31) j=2(q2j q3j ) Q2 + Q2 w = 21 31 , 2 2 2 + 2 Q2 + w2(Q2 + Q2 ) det (Q)(q21 q31) 11 1 12 13 x2 = , 2 2 + 2 + 2 3 det (Q)(q11 q12 q13) (Q2 + Q2 ) (Q2 + Q2 ) 21 31 j=2 2j 3j w1 = . 2 + 2 2 + 2 3 2 + 2 Q21 Q31 (q21 q31) j=2(q2j q3j ) w2 = , det2(Q)(q2 + q2 ) 21 31 Then, from the first equation in (28)wehave − (q ) = (q , q , q )T = P 1P = Q, ij 1 2 3 2 1 Q2 + w2(Q2 + Q2 ) x = 11 1 12 13 . (Q ,Q ,Q )T = q × q , 2 2 2 + 2 + 2 i1 i2 i3 j k det (Q)(q11 q12 q13) i = 1, 2, 3; j,k = i and j

4.3 Standard Algorithm 4.4 Geometrical Interpretation of H

Goal Given N views of a scene and assume the infinite ho- We will give a geometrical interpretation of H in this sub- mographies Hi∞ between the 0th and ith views are known. section. Compute the intrinsic matrix K and the rotation Ri between Let m ↔ m be an image correspondence, both m and the 0th and ith views. m being projections of an infinite point X∞ = (XT , 0)T . Algorithm The algorithm consists of the following three Then, steps: αm = KX,βm = KRX, ˜ −1/3 1. Let Hi∞ = Det (Hi∞)Hi∞ and compute its Cayley ˜ transformation Hi = ϕ(Hi∞). Then find the least squares and thus, solution of the set of linear equations on ω: αm − βm = K(I − R)X, T + = = − Hi ω ωHi 0 (i 1, 2,...,N 1). αm + βm = K(I + R)X. 2. Compute the intrinsic matrix K by the Cholesky factorization of ω. By the above two equations, we obtain K −1 3. Let H = K HiK and compute its Cayley trans- i − − ˜ K − = − + 1 1 + formation, Ri = ϕ(H ). Then, determine the rotation Ri αm βm K(I R)(I R) K (αm βm ). by finding the best rotation approximation of the matrix ˜ Ri , i.e., firstly compute the singular value decomposi- Therefore, ˜ ˜ = T tion of the matrix Ri , Ri U diag(t1,t2,t3)V , then set T m − γ m = H(m + γ m )(γ= β/α). (29) up Ri = UV .

This algorithm is called the standard algorithm in this paper. From (29), we can see that the matrix H transforms the In the absence of data error and with the infinite precision of linear combination m + γ m of an infinite correspondence computation, the CCE and the SCE should give the same ω. m ↔ m into another linear combination m − γ m. This in- However, in the presence of data error and with numerical dicates that the matrix H is not a transformation between computation, their numerical performances manifest a great the two views. difference (see Sect. 5.3). Equation (29) cannot be used for a linear computation of the matrix H from infinite correspondences due to the Remark 3 First, the matrix H has an important constraint unknown scale γ . But a nonlinear solution is possible. By of rank 2. In practice due to the data error, the computed eliminating the unknown scale γ ,(29) becomes H does not satisfy this constraint. If in the step 1 this constraint is enforced, both success rate and calibration preci- [m + Hm ]×(m − H m) = 0. (30) sion of the algorithm can be improved. The following shows an Rank-enforced method: Firstly, compute the eigenvalues It is a nonlinear constraint on H . factorization of the matrix H , H = Udiag(s, t, t)U−1, then If we regard (I − H∞) and (I + H∞) as two trans- s = H let its real eigenvalue 0, and replace with the matrix formation from the first view into the second view, then − − = + 1 − H = Re(Udiag(0,t,t)U 1). H (I H∞) (I H∞) is a transformation on the first view, and is also a transformation on the second view be- − Here Re(A) denotes the real part of A. In theory cause (I + H∞) 1 is commutative with (I − H∞) i.e., Udiag(0,t,t)U−1 is a real matrix, but a very small imag- −1 −1 inary part is possible owing to the numerical computation. H = (I + H∞) (I − H∞) = (I − H∞)(I + H∞) . 164 Int J Comput Vis (2009) 82: 156–184

(a) (b)

Fig. 1 The intrinsic geometrical meaning of the transformation H : an arbitrary x ∈/ l∞ ∪{v∞}, y = H x ∈ l∞ is a conjugate point of x the vanishing point v∞ of the rotation axis of R is a 1-dimensional on the IAC and H is a transposition on {y,Hy}.(b) Except for v∞, zero space of H and the vanishing line l∞ of plane orthogonal to the H transforms all points on l to its conjugate point c(l) on the IAC and rotation axis of R is 2-dimensional invariant subspace of H .(a)For H transforms c(l) to the intersection of l and l∞

By considering H as a transformation on the same view, we By the proof of Proposition 2, H has the same eigenvec- can give its intrinsic geometrical meaning as stated in the tors with H∞. Since l∞ is a 2-dimensional invariant sub- following proposition. space of H∞, it is also a 2-dimensional invariant subspace of H , and thus {H x,H2x,H3x}⊂l∞. By (a), both H 2x 3 2 Proposition 7 Let v∞ be the vanishing point of the rotation and H x are the conjugate points of H x and H x = H x, 3 axis of R and l∞ be the vanishing line of plane orthogonal therefore there must be H x = H x. Hence, H is a transpo- with the rotation axis of R. Assume x is an arbitrary image sition on {H x,H2x}. point, then, (c) Let x be a fixed point on the line l but not equal to v∞, then ∀y ∈ l\{v∞} there is (a) If x is not the vanishing point v∞, then H transforms x to its conjugate point y on the IAC, i.e., if y = H x, then y = αv∞ + βx,β= 0. (x, y) satisfies xT ωy = 0. = (b) ∀x ∈/ l∞ ∪{v∞}, there must be H x ∈ l∞ and H is a And thus, H y βHx, i.e., H y and H x are the same point. transposition on {H x,H2x}, i.e., H(Hx) = H 2x and By (a), H x is the conjugate point of the line l on the IAC, { }⊂ H(H2x) = H x, asshowninFig. 1(a). c(l).By(b), c(l), H c(l) l∞ and H c(l) is a conjugate c l c l (c) Let l be an arbitrary line passing through v∞, and c(l) point of ( ) on the IAC, and thus H ( ) must be the inter- be its conjugate point on the IAC. Then, except for the section of l and l∞. vanishing point v∞,H transforms all points on the line Remark 4 This is an interesting proposition. First, this l to the same point c(l) and H transforms c(l) to the proposition tells us that using H we can obtain the con- intersection of the line l and the line l∞, q = l∞ ∩ l, as straints on the IAC from the vanishing points of two orthog- showninFig. 1(b). onal directions to each other. Obviously, by the matrix H we cannot obtain the constraints on the IAC from all orthogonal Proof (a) From the SCE (20), the matrix ωH is anti- directions, for example, the constraints from two orthogonal symmetric. And thus for any image point x, there must directions of the rotation axis of R and its orthogonal line. be xT ωHx = 0. Note that m ∈ N(H) if and only if But, from (b) and (c), it is not difficult to see that H can = m H∞m, namely, m is the vanishing point v∞. There- provide the four independent constraints. = = fore, H x 0, and thus y H x is a conjugate point of x on Second, this proposition tells also us that H 2 is a the IAC. 2-dimensional projection with the center v∞ and the image ∀x ∈/ l∞ ∪{v∞} (b) , there must be line l∞ as shown Fig. 2(a), and H 3 is H self, i.e., H 3 = H . Observing the transformation property of H on two lines x = αv∞ + βx + γ x , 1 2 passing through v∞ and a pair of conjugate points on l∞, we can have the Euclidean properties of H ,asshownin where x , x ∈ l∞ and α, β, γ = 0. Then, we have 1 2 Fig. 2(b). In essence, the constraints on the IAC induced by H are from the orthogonality of 3D structure, while H∞ are H x = αHv∞ + βHx1 + γHx2 from the orthogonality of 3D motion R. Hence, the Cayley = βHx1 + γHx2 ∈ l∞. transformation, H = ϕ(H∞), is a conversion between the Int J Comput Vis (2009) 82: 156–184 165

(a) (b)

Fig. 2 (a) H 2 is a 2-dimensional projection with the center v∞ and L(v∞)⊥ (l∞) and L(p)⊥L(q).Sincex is on line v∞ ×p,L(x) is on the image line l∞. It maps point x to the intersection of line l∞ and the plane L(v∞)×L(p), similarly L(y) is on the plane L(v∞)×L(q). the line passing through v∞ and x.(b)LetL(m) and (l) be the di- Proposition 7 shows that H transforms L(x) to its orthogonal direction rections of the line and the plane in 3D space corresponding to m and L(q), L(y) to its orthogonal direction L(p), and transposes the two or- l and assume p and q are a pair of conjugate points on the IAC, then thogonal directions L(p) and L(q) to each other

3D motion orthogonality and the 3D structure orthogonal- Equation (31) is called the Cayley transformation family ity. of H . Next, we show some properties of the Cayley trans- Third, this proposition gives a linear constraint on H as formation family. follows. If a line L and a plane P are orthogonal to each other in 3D space, then their vanishing point x and vanish- Proposition 8 (a) ing line l satisfy lT H x = 0. The only exception is the case ϕ(1 · H)= H∞,ϕ(0 · H)= I. (32) of rotation axis of R and its orthogonal plane. A proof is ready: Assume the line L is not parallel to the rotation axis (b) For τ ∈ (−∞, +∞), of R, then the relation of its vanishing point x ∈/ N(H) and − the vanishing line of its orthogonal plane is l = ωx.By(a), ϕ 1(τH ) = ϕ(−τH) or, xT ω(Hx) = 0, i.e., lT H x = 0 which is a linear constraint (33) ϕ(τH) · ϕ(−τH)= I, on H . τH = ϕ2(τH ). (34)

5 Cayley Calibration Family Proof We only prove (33) here. Both (I − τH) and (I + τH) are invertible for τ ∈ (−∞, +∞), therefore 5.1 Cayley Transformation Family of H ϕ−1(τH ) = (I + τH )(I − τH)−1 = − + −1 − Since H (I H∞)(I H∞) has the eigenvalues: = (I − (−τH ))(I + (−τH)) 1 = ϕ(−τH). { − } 0,itan(θ/2), i tan(θ/2) , The matrix ϕ(τH) can be computed from the inﬁnite homography H∞ by (35) via the following proposition. for τ ∈ (−∞, +∞) the eigenvalues of matrix I ± τH must be Proposition 9

{1, 1 ± iτ tan(θ/2), 1 ∓ iτ tan(θ/2)}. −1 ϕ(τH) = ((1 + τ)I + (1 − τ)H∞)

Then, × ((1 − τ)I + (1 + τ)H∞). (35)

2 2 det(1 ± τH)= 1 + τ tan (θ/2) = 0. Proof

Hence (τH ) has the Cayley transformation: ϕ(τH) = (I + τH)−1(I − τH)

− −1 −1 ϕ(τH) = (I − τH )(I + τH) 1 = (I + τ(I + H∞) (I − H∞)) −1 for τ ∈ (−∞, +∞). (31) × (I − τ(I + H∞) (I − H∞)) 166 Int J Comput Vis (2009) 82: 156–184 −1 = ((I + H∞) + τ(I − H∞)) (I + H∞) equations are equivalent to the CCE-I. Evidently, the CCF-I

−1 has the following equivalent form: × (I + H∞) ((I + H∞) − τ(I − H∞)) T = −1 = −1 ϕ (τH )ω ωϕ (τH ) (τ 0). = ((1 + τ)I + (1 − τ)H∞) −1 = − × ((1 − τ)I + (1 + τ)H∞). By Proposition 8, ϕ (τH ) ϕ( τH), and thus the above equation can be rewritten as Let ϕ (τH ) = K−1ϕ(τH)K. If the intrinsic matrix K is K ϕT (τH )ω = ωϕ(−τH) (τ = 0). (38) known, then we can compute the rotation R by (36) via the following proposition. The advantage of this form is to avoid the matrix inverse op- erations, hence improve numerical stability. Equation (38)is Proposition 10 called Cayley calibration family of the second type, denoted as CCF-II(τ). = + + − −1 R ((τ 1)I (τ 1)ϕK (τH )) Next, we give an explanation of the CCF in term of × ((τ − 1)I + (τ + 1)ϕK (τH )). (36) the inﬁnite homography. In other words, we will prove that for τ = 0, the Cayley transformation ϕ(τH) is an inﬁnite Proof From (35), we have homography.

−1 ϕK (τH ) = K ϕ(τH)K Proposition 12 For τ = 0, the Cayley transformation ϕ(τH) of τH is an inﬁnite homography and the correspond- = + + − −1 ((1 τ)I (1 τ)R) ing rotation has the same rotation axis with the rotation be- × ((1 − τ)I + (1 + τ)R) tween the two views. = − + + ((1 τ)I (1 τ)R) Proof Let R be the rotation between the two given views. − × ((1 + τ)I + (1 − τ)R) 1. Then according to Proposition 9,wehave

Hence, ϕ(τH) − − = K((1 + τ)I + (1 − τ)R) 1((1 − τ)I + (1 + τ)R)K 1. ϕK (τH )((1 + τ)I + (1 − τ)R) U(τ) = ((1 − τ)I + (1 + τ)R). (39) By some simple manipulations, we can obtain (36). In order to prove Proposition 12, we need only to prove that U(τ) is a rotation and it has the same rotation axis with R. 5.2 Cayley Calibration Family For this purpose, let the eigenvalues factorization of R be: ∈ −∞ +∞ − − Since for τ ( , ), ϕ(τH ) is the Cayley trans- R = V diag(1,eiθ,e iθ)V 1. (40) formation of τH, according to Proposition 1 the equa- T tion ϕ (τH )ωϕ(τH ) = ω is equivalent to the equation Then θ is the rotation angle of R,V = (v1, v2, v3) is a uni- T (τH ) ω +ω(τH) = 0. When τ = 0, both the two equations tary matrix and v1 is the rotation axis of R. Substituting (40) become the identity equation, and thus we do not consider into U(τ)in (39) and by some simple manipulations, we can this trivial case. Evidently, for τ = 0 the SCE is equivalent obtain to the equation (τH )T ω + ω(τH) = 0. Thus, we have the 1 − τ + (1 + τ)eiθ following proposition. U(τ)= V diag 1, , 1 + τ + (1 − τ)eiθ Proposition 11 The SCE is equivalent to the following 1 − τ + (1 + τ)e−iθ V −1. equation: 1 + τ + (1 − τ)e−iθ

ϕT (τH )ωϕ(τH ) = ω(τ= 0). (37) It is not difficult to prove that − τ + ( + τ)eiθ The above equation gives a one-parameter family equiv- 1 1 + + − iθ alent to the SCE. Since it has the same form with the CCE-I, 1 τ (1 τ)e we call it Cayley calibration family of the first type, denoted + 2 cos(θ/2) iτ sin(θ/2) as CCF-I(τ). In the CCF-I(τ), the equation corresponding to = = z(τ), 2 2 2 τ = 1is just the CCE-I, i.e. CCF-I(1) = CCE-I, and all the cos (θ/2) + τ sin (θ/2) Int J Comput Vis (2009) 82: 156–184 167 − 1 − τ + (1 + τ)e iθ usually means less numerical stability, and a small condition 1 + τ + (1 − τ)e−iθ number indicates good numerical stability. If the rank of the 2 matrix A is k, then its condition number is defined as the cos(θ/2) − iτ sin(θ/2) = = z(τ). ratio of the largest singular value to the kth smallest singular cos2(θ/2) + τ 2 sin2(θ/2) value (Hartley 1997c). And thus the condition number of the matrix A is computed by taking the singular value decompo- Thus, U(τ) can be rewritten as sition of this matrix. Figure 3 gives the condition numbers of equations in 5 CCF-Is and 5 CCF-IIs. From this figure, we T U(τ)= V diag(1, z(τ), z(τ))V . (41) can see that different equation in the CCF-I (or CCF-II) has different condition number, and thus has different numerical Since z(τ)is a unit complex number, U(τ) is a rotation and stability. Hence, we can improve the numerical stability of v1is its rotation axis. calibration by selecting a parameter τ such that corresponding equation has a smaller condition number. Remark 5 From the proof of Proposition 12, we can see that We next give an algorithm for selecting the best parame- the rotation angle of rotation U(τ) corresponding to infinite ter value. Let the set of m CCF-Is (or CCF-IIs) be homography ϕ(τH) is T ϕ (τiHi)ωϕ(τiHi) − ω = 0,i= 1, 2,...,m θ(τ)= 2Arctan(τ tan(θ/2)). (42) T (or,ϕ (τiHi)ω − ωϕ(−τiHi) = 0,i= 1, 2,...,m). Thus if τ varies in (−∞, +∞), we can obtain all the ro- The m-parameter family of the above homogeneous linear v tations with the fixed rotation axis 1. In other words, given systems can be rewritten as the following matrix form: an infinite homography H∞ with the rotation R (rotation ×m axis v and rotation angle θ), by the Cayley transformation of A(τ)c = 0, τ = (τ1,τ2,...,τm) ∈ (−∞, +∞) , (43) τϕ(H∞) or, τH we can obtain all the infinite homographies = T with the fixed rotation axis v but arbitrary rotation angle, here c (ω11,ω12,ω13,ω22,ω23,ω33) . In theory, the rank and thus Proposition 6 tells us again that the configuration of of matrix A is 5, therefore our problem is to solve the fol- cameras with the same rotation axis is a degenerate config- lowing minimization: uration. Of course, by changing the rotation angle from the σ1 ×m eigenvalues factorization of H∞, we can also obtain these min J = (τ)|τ ∈ (−∞, +∞) , (44) σ5 infinite homographies, but it needs to compute the eigensys- tem of H∞. In the contrast, we here need merely taking two where σ1 ≥ σ2 ≥···≥σ5 ≥ σ6 are singular values of A(τ). times of the Cayley transformation, ϕ(τϕ(H∞)), in order to For solving the above problem, we firstly compute the change the rotation angle. gradient of J . By taking the singular value decomposition = T = T of A, A UDV , then σj uj Avj where uj , vj are the 5.3 Improving Calibration Conditioning jth row of the orthogonal matrices U and V , respectively. And thus we have Given the infinite homography H∞ between two views, we T ∂σj ∂(uj Avj ) can obtain the CCF-I(τ) equivalent to the CCE-I: = ∂τi ∂τi CCF-I(τ) : ϕT (τH )ωϕ(τH ) − ω = 0,τ= 0, ∂uT = j + T ∂A + T ∂vn Avj uj vj uj A ∂τi ∂τi ∂τi and CCE-I = CCF-I(1)(ϕ(1 · H)= H∞). In the CCF-I(τ), if we define CCF-I(0) by H T ω + ωH = 0 (SCE), then ∂uT ∂v = j + T ∂A + T j τ σj uj uj vj σj vj . by Proposition 10 we obtain an equivalent family with ∂τi ∂τi ∂τi in (−∞, +∞), denoted as CCF-I also. Similarly, for the ∂vT ∂u CCF-II(τ), we define CCF-II(0) also by the SCE, and thus Notice j v = uT j = 0 from uT u = vT v = 1, we ∂τi j j ∂τi j j j j obtain an equivalent family of the second type with τ in have (−∞, +∞). ∂σ ∂A The numerical stability of a homogeneous linear system, j = T uj vj . such as Ax = 0, can be assessed by examining the condition ∂τi ∂τi number of its coefficient matrix A, it is also called the con- Hence the gradient of J on τi is: dition number of this homogeneous linear system. The con- ∂J 1 ∂A ∂A dition number is a typical measure of the sensitivity of ho- = σ uT v − σ uT v . 2 5 1 1 1 5 5 (45) mogeneous linear system to noise. A large condition number ∂τi σ5 ∂τi ∂τi 168 Int J Comput Vis (2009) 82: 156–184

(a) Condition numbers of CCF-Is (b) Condition numbers of CCF-IIs Fig. 3 Condition numbers of the CCF-Is and CCF-IIs corresponding the SCE has the smallest condition number among the CCF-I. For the −1 to 5 inﬁnite homographies H∞ = KRK where K : fu = fv = 1000, CCF-II, the condition numbers of equations vary with the parameter τ = = = = · · s 0.0, pu pv 512 and the rotation Ri Rx (θi1) Ry (θi2) also, but the condition numbers reach their minimums at different τ . = Rz(π/θi3)(i 1, 2, 3, 4, 5), θij is randomly generated by the uniform The condition numbers of the CCF-I and the CCF-II for the same pa- distribution on the interval [−π/4,π/4]. The abscissa and the ordinate rameter are different and the CCF-II has always a smaller condition are parameter τ and condition number respectively. We can see that: For the CCF-I, the condition numbers of equations vary with the para- number, thus the numerical performance of CCF-II is better than that meter τ and the condition number reaches its minimum at τ = 0 i.e., of CCF-I in general

With the computed gradient ∇J(τ) of J , we can use some 1. Let s0 = 0,s1 = 1, n = 1 and given a termination δ. k k k k numerical methods to solve the minimization problem (44), 2. Compute e = J(τ + sn−1p ) − J(τ + snp ). such as steepest descent method or conjugate gradient 3. In the case e>0, set up sn+1 = sn + 10λ and let method (Press et al. 1988). In the experiments of this pa- n ← n + 1, return to Step 2. In the case e<0, if per, we use the steepest descent method as outlined below. |sn+1 − sn| <δ, terminate iteration and put out sn,or else do Step 4. Determining the Parameters τ The algorithm consists of 4. (a) t = λ/10. the following four steps: (b) Set up sn+1 = sn + t and do Step 2. (c) In the case e>0, let n ← n + 1 and return to (b). In 1. Given an initialization τ 0 and a termination ε.Letk = 0. the case e<0, if |sn+1 − sn| <δ, terminate iteration 2. Compute the gradgent ∇J(τ k) of J with (45), and put out sn,orelselett ← t/10 and return to (b). ∂J ∂J ∂J = −5 = ∇J(τ k) = , ,..., . In our work, δ 10 and λ 0.1 are used. Third, in ∂τ1 ∂τ2 ∂τm τ=τ k practice finding an exact solution of the minimization (44)is unnecessary. This is because the condition number is a qual- k k If ∇J(τ ) <ε, terminate iteration and put out τ ,or itative analysis of numerical stability; the condition numbers else do Step 3. with the same order of magnitude are unable to lead to a big k k 3. Let p =−∇J(τ ). Solve sk such that difference in numerical stability. And thus, we can terminate iteration once the condition number is less than a given k k k k J(τ + skp ) = min J(τ + sp )|s ≥ 0 . threshold. In this work, 50 is used as the threshold. 5.4 Cayley Algorithm k+1 k k 4. Let τ = τ + skp and k ← k + 1. Return to Step 2. By the discussions in Sects. 5.1, 5.2 and 5.3, we can also Remark 6 First, in this paper the initialization is set to give a calibration algorithm similar to the standard one, τ 0 = (1, 1,...,1), corresponding to the set of CCEs. Sec- called the Cayley algorithm in this paper. ond, in Step 3, the exact solution s cannot be obtained since Goal Given N views of a scene and assume the infinite ho- k + k J(τ sp ) has no analytical expression. We can use an al- mographies Hi∞ between the 0th and ith views are known. gorithm similar to Levenberg-Marquardt algorithm (Press et Compute the intrinsic matrix K and the rotation Ri between al. 1988) to find an approximate solution of sk as follows: the 0th and ith views. Int J Comput Vis (2009) 82: 156–184 169 Algorithm The algorithm consists of the following four For example, let steps: − − φ (τA) = (I + e τA)(I + eτA) 1, (48) ˜ −1/3 e 1. Let Hi∞ = det (Hi∞)Hi∞ and compute its Cayley ˜ = − + −1 transformation Hi = ϕ(Hi∞) and CCF-I (or CCF-II): φs(τA) (I sin(τA))(I sin(τA)) . (49)

T Then for −∞ <τ<∞ the transformation (48) is a gen- ϕ (τiHi)ωϕ(τiHi) = ω, i = 1, 2,...,N eralized Cayley transformation of matrix A.Ifr(A) ≤ 1 T (or ϕ (τiHi)ω = ωϕ(−τiHi), i = 1, 2,...,N). (here r(A) = max{|λ|:λ ∈ σ(A)} is called the spectrum ra- dius of matrix A), then for −π/2 <τ<π/2, the transfor- = ∗ ∗ ∗ 2. Determine the parameters τ (τ1 ,τ2 ,...,τN ) by solv- mation (49) is also a generalized Cayley transformation of ing the minimization (44), then ﬁnd the least squares so- matrix A. lution of the set of linear equations on ω: For generalized Cayley transformations, we have the following proposition. T ∗ ∗ = = ϕ (τi Hi)ωϕ(τi Hi) ω, i 1, 2,...,N T ∗ = − ∗ = Proposition 13 Assume φ(τA) (−ε<τ<ε)is a gen- (or ϕ (τi Hi)ω ωϕ( τi Hi), i 1, 2,...,N). eralized Cayley transformation of matrix A ∈ Cn×n and × 3. Determine the intrinsic matrix K by taking the Cholesky C ∈ Cn n is an arbitrary matrix. Then factorization of ω. T ∗ = −1 ∗ φ(τA) Cφ(τA) = C for 0 < |τ| <ε 4. Let ϕK (τi H) K ϕ(τi Hi)K and compute ⇔ AT C + CA = . = ∗ + + ∗ − ∗ −1 0 (50) Ri ((τi 1)I (τi 1)ϕK (τi Hi)) × ∗ − + ∗ + ∗ Proof g(τA) = φ(τA)T Cφ(τA) ((τi 1)I (τi 1)ϕK (τi Hi)). Let , then, ∂ ∂ Then, determine the rotation Ri by ﬁnding the best rotation = T ˜ g(τA) φ(τA) Cφ(τA) approximation of the matrix Ri , as shown in the standard ∂τ ∂τ algorithm in Sect. 4.3. ∂ = φ(τAT )Cφ(τA) ∂τ = AT φ (τAT )Cφ(τA) + φ(τAT )CAφ (τA). 6 Generalized Cayley Calibration Families τ τ (51)

6.1 Generalized Cayley Transformation Assume equation φ(τA)T Cφ(τA) = C holds for 0 < |τ| <ε, then for 0 < |τ| <εthere must exist If a function of complex variable, ξ = φ(λ), satisﬁes the fol- T T T lowing three conditions, then it is called a generalized Cay- A φτ (τA )Cφ(τA) + φ(τA )CAφτ (τA) = 0. (52) ley transformation: Because φ(λ) is analytical in −ε<τ<ε, the left side (a) φ(λ) is an analytical function with real coefﬁcients in in (52) is a continuous mapping on the variable τ . Thus, | | λ <δ; letting τ → 0in(52) leads to (b) φ(λ)φ(−λ) = 1for|λ| <δ; (c) φ(0)>0 and φ (0) = 0. φ (0)φ(0)(AT C + CA) = 0.

It is easy to see that the Cayley transformation is a special By the conditions (b) and (c), φ(0)φ(0) = 0, and thus there case of the generalized Cayley transformation. Below are must be two examples of generalized Cayley transformations: AT C + CA = 0. 1 + e−λ 1 − sin(λ) φe(λ) = ,φs(λ) = . 1 + eλ 1 + sin(λ) Now assume equation AT C +CA = 0 holds, then we can verify that For a matrix A ∈ Cn×n, a generalized Cayley transforma- T T tion of A can be deﬁned as φ(τA )C = Cφ(−τA), φτ (τA )C = Cφτ (−τA)

Bτ = φ(τA) (−ε<τ<ε), (46) holds for any analytical function φ(λ). Then we obtain

T T T where ε is a positive number such that A φτ (τA )Cφ(τA) = A Cφτ (−τA)φ(τA), T σ(εA)⊂{λ :|λ| <δ}. (47) φ(τA )CAφτ (τA) = CAφ(−τA)φτ (τA). 170 Int J Comput Vis (2009) 82: 156–184 So, by (51), we have Proposition 15 The generalized Cayley transformation φ(τH) is an inﬁnite homography and its rotation has the ∂ g(τA) = AT Cφ (−τA)φ(τA) + CAφ(−τA)φ (τA). same rotation axis with the rotation between the two views. ∂τ τ τ By the condition (b), there is Proof Let R be the rotation between the two given views and its eigenvalues factorization be: φ (τA)φ(−τA)= φ(τA)φ (−τA). τ τ − − R = V diag(1,eiθ,e iθ)V 1, And thus, for 0 < |τ| <ε, where θ is its rotation angle, V = (v1, v2, v3) is an unitary ∂ T matrix and v is the rotation axis of R. Then according to g(τA) = (A C + CA)φ(τA)φτ (−τA)= 0. 1 ∂τ Proposition 2, Therefore, for 0 < |τ| <ε, − τH = Kτϕ(R)K 1 = T = − − g(τA) φ(τA) Cφ(τA) constant matrix. = KVdiag(0,iτtan(θ/2), −iτ tan(θ/2))V 1K 1. = By the continuity of φ(τA) at τ 0, we have Then, T 2 φ(τA) Cφ(τA) = g(τA) = φ (0)C = − −1 −1 φ(τH) KV diag(φ(0), φ(is(τ)), φ( is(τ)))V K , for 0 < |τ| <ε. Q(τ)

By the condition (b), φ2(0) = 1, and thus there must be here s(τ) = τ tan(θ/2). In order to prove Proposition 15,we need only to prove that Q(τ) is a rotation and it has the same T φ(τA) Cφ(τA) = C. rotation axis with R. From the conditions (b) and (c) of the function φ, there is φ(0) = 1. By the conditions (b) of the 6.2 Generalized Cayley Calibration Family function φ,wehave

Applying a generalized Cayley transformation to the Cayley φ(is(τ)) · φ(−is(τ)) = 1. transformation of the infinite homography, we can obtain the following proposition according to the Proposition 13. Then from the conditions (a) of the function φ,wehave φ(−is(τ)) = φ(is(τ)). Proposition 14 The SCE is equivalent to the following one- parameter family Therefore {φ(is(τ)), φ(−is(τ))} is a conjugate pair of unit complex numbers, and Q(τ) can be written as: φ(τH)T ωφ(τH) = ω for 0 < |τ| <ε. (53a) − − Q(τ) = V diag(1,eiσ(τ),e iσ(τ))V 1, Given a generalized Cayley transformation, by (53a)we obtain a one-parameter family equivalent to the SCE. Since where it has the same form with the CCE-I and CCF-I, it is called 1 1 the generalized Cayley calibration family of the first type σ(τ)= log(φ(is(τ))) = log(φ(iτ tan(θ/2))). i i (GCCF-I). Because φ−1(τH ) = φ(−τH) for −ε<τ<ε, the SCCF-I has the following equivalent form, it is called Hence, Q(τ) is a rotation and v1 is its rotation axis. the generalized Cayley calibration family of the second type (GCCF-II): Remark 7 The GCCF-I contains also the CCE-I. Let θ be the rotation angle between the two given views, then in the φ(τH)T ω = ωφ(−τH) for 0 < |τ| <ε. (53b) GCCF-I the parameter τ corresponding to the CCE-I is the solution of the following equation: As in the CCFs, the numerical stability of equations in a GCCF varies also with the parameter τ . For a given gener- log(φ(iτ tan(θ/2))) = iθ. alized Cayley calibration family, we can use the method in Sect. 5.3 to reduce the condition number of calibration equa- 6.3 S-Cayley Calibration Family tions by selecting a proper parameter value and can give a calibration algorithm similar to the algorithm in Sect. 5.3, As an example, we examine the generalized Cayley calibra- called as generalized Cayley algorithm. Next, we give an tion family induced by the transformation (49): explanation of (53a) also in terms of the infinite homogra- π BT (τH )ωB(τH ) = ω for 0 < |τ| < . (54) phy. 2 Int J Comput Vis (2009) 82: 156–184 171 Here B(τH) = (I − sin(τH ))(I + sin(τH ))−1. B(τH) that the condition numbers of equations in the SCCF-I (or is the Cayley transformation of the matrix sin(τH ) i.e., SCCF-II) have no notable difference, having the same mag- B(τH) = ϕ(sin(τH )). And thus, the calibration family (54) nitude almost. Compared with the CCF-I and the CCF-II can be rewritten as (see Fig. 3), Both the SCCF-I and SCCF-II have smaller π condition numbers and the condition number of SCCF-I is ϕ(sin(τH ))T ωϕ(sin(τH )) = ω for 0 < |τ| < . (55) 2 far smaller than that of the CCF-I. This indicates that, on the whole, the numerical performance of the SCCF is superior We call it S-Cayley calibration family of the first type to that of the CCF. (SCCF-I). Corresponding S-Cayley calibration family of the second type (SCCF-II) is Remark 8 For computational simplicity, in practice the gen- π eralized Cayley transformation ϕ(sin(x)) can be approxi- ϕ(sin(τH ))T ω = ωϕ(sin(−τH)) for 0 < |τ| < . (56) 2 mated with the following rational function: By Proposition 15, the S-Cayley transformation − = 1 p2n+1(x) ϕ(sin(τH )) is an infinite homography whose rotation axis is qn(x) , (57) 1 + p2n+1(x) the same with the rotation axis between the two given views + and rotation angle is where p2n+1(x) is the (2n 1)th-order of Taylor approximation of sin(x): 1 σ(τ)= log (ϕ(sin(iτ tan(θ/2)))) 3 2n+1 x n x i p + (x) = x − +···+(− ) . 2n 1 ! 1 + ! (58) 1 i + sinh(τ tan(θ/2)) 3 (2n 1) = log , i i − sinh(τ tan(θ/2)) The first-order rational approximation of ϕ(sin(x)), q1(x), is just the Cayley transformation. It is not difficult to ver- where θ is the rotation angle between the two given views. ify that the (2n + 1)th-order rational approximation q (x) By the above equation, we can obtain the parameter corre- n is also a generalized Cayley transformation. In other words, sponding to the CCE-I in the SCCF-I: the generalized Cayley transformation ϕ(sin(x)) can be ap- τ =−cotan(θ/2) · arcsinh(tan(θ/2)). proximated with a series of rational Cayley transformations.

Figure 4 gives the condition numbers of equations in From at least 3 views, we can linearly compute the IAC 5 SCCF-Is and 5 SCCF-IIs. From this ﬁgure, we can see with the SCCFs. After the IAC is determined, we can com-

(a) Condition numbers of the SCCF-Is (b) Condition numbers of the SCCF-IIs

Fig. 4 Condition numbers of the SCCF-Is and SCCF-IIs correspond- that: Compared with the CCF-I and the CCF-II (see Fig. 3). Both the −1 ing to 5 infinite homographies H∞ = KRK where K : fu = fv = SCCF-I and SCCF-II have smaller condition numbers and the con- 1000, s = 0.0, pu = pv = 512 and the rotation Ri = Rx (θi1) · Ry (θi2) · dition number of SCCF-I is far smaller than that of the CCF-I. This Rz(π/θi3)(i= 1, 2, 3, 4, 5), θij is randomly generated by the uniform indicates that, on the whole, the numerical performance of the SCCF distribution on the interval [−π/4,π/4]. The abscissa and the ordi- is superior to that of the CCF. Note the SCCF has no definition at τ = 0 nate are parameter τ and condition number respectively. We can see 172 Int J Comput Vis (2009) 82: 156–184 pute the rotations between the views using (59)inthefol- T T + T = H ξ (τA)ωξ(τA) ξ (τA)ωξ(τA) H 0. (62a) lowing proposition. Proposition 16 Assume ω or, equivalently K is known. For example, let A = H,ξ(λ) = 1 + tanh(λ) and by sub- Then the rotation between two views is: stituting ξ(τH) = I + tanh(τH ) into (62a), we obtain the following one-parameter calibration family with τ ∈ 1 − R = ϕ K 1Arcsin(ϕ(B(τH )))K . (59) (−∞, +∞): τ H T (I + tanh(τH ))T ω(I + tanh(τH )) Proof As matrix Bτ is the Cayley transformation of sin(τH ), =− + T + the matrix sin(τH ) is also the Cayley transformation of Bτ : (I tanh(τH )) ω(I tanh(τH ))H. (63) 1 sin(τH ) = ϕ(B ) and H = Arcsin(ϕ(B )). In this calibration family, the equation corresponding to τ τ τ τ = 0 is just the SCE. Figure 5 gives the condition numbers Thus letting of the calibration families from 5 infinite homomgraphies. They all reach their minimums at τ = 0, corresponding to K −1 1 −1 H = K HK = K Arcsin(ϕ(Bτ ))K, the SCE. τ From (61a), we can have a new interesting observation. by Corollary 1(b) we have The infinite homography preserves not only the IAC but also other conic, such as ψ = ξ T (τA)ωξ(τA) which is not 1 −1 the IAC. R = ϕ K Arcsin(ϕ(B(τH )))K . τ Given an invertible matrix A and an analytical function ξ(λ), the numerical performances of equations in the one- parameter family (61)or(62) are different, they vary with 7 A More Generic Calibration Family parameter τ . For improving numerical stability of calibration, we can also use the method in Sect. 5.3 to deter- 7.1 More Generic Equivalent Equations mine a proper parameter value such that the corresponding equation has a smaller condition number. For differ- Let G3×3 is a group of invertible 3 × 3 matrices i.e., ent matrices and different analytical functions, correspond- × × G3 3 ={A ∈ R3 3| det(A) = 0}. ing one-parameter families have different numerical per-

Then, ∀A ∈ G3×3 we have

T H∞ωH∞ − ω = 0 T T T ⇔ A H∞ωH∞A − A ωA = 0 ⇔ AT H T ωA + AT ωHA = 0. (60)

Given an invertible matrix A and an analytical function with real coefﬁcients ξ(λ), we can deﬁne a one-parameter family of the matrix, ξ(τA).Ifforτ ∈ (a, b), there is ξ(τμ) = 0, ∀μ ∈ σ(A), then ξ(τA) is a one-parameter family of invertible matrices with τ ∈ (a, b). And thus, we can obtain two one-parameter calibration families equivalent to the CCE:

T T T ξ (τA)H∞ωH∞ξ(τA)− ξ (τA)ωξ(τA) = 0, (61) ξ T (τA)H T ωξ(τA) + ξ T (τA)ωH ξ(τA) = 0. (62) Fig. 5 Condition numbers of calibration families (63) corre- = −1 An interesting case is that the matrix A is commutative sponding to 5 inﬁnite homographies H∞ KRK where K : fu = fv = 1000, s = 0.0, pu = pv = 512 and the rotation H∞ ξ(τA) with . In this case, since is also commutative with Ri = Rx (θi1) · Ry (θi2) · Rz(π/θi3)(i= 1, 2, 3, 4, 5), θij is randomly both H∞ and H ,(61) and (62) become generated by the uniform distribution on the interval [−π/4,π/4]. The abscissa and the ordinate are parameter τ and condition number T T T H∞ ξ (τA)ωξ(τA) H∞ − ξ (τA)ωξ(τA) = 0, (61a) respectively. We can see that: All the condition numbers reach their minimums at τ = 0, corresponding to the SCE. Note their condition ψ ψ numbers are not continues at τ = 0 Int J Comput Vis (2009) 82: 156–184 173 formances on the whole. For example, the numerical performance of CCF-II is superior to that of CCF-I in general. Here there is a more generic problem: How to ﬁnd amatrixA and an analytical function ξ(λ) such that the condition number of corresponding calibration equations is the smallest among all invertible matrices and analytical functions or, an optimal combination of matrix A and ξ(λ)? Unfortunately, we cannot give a clear-cut answer cur- rently although it is an interesting one. Next, we point out that by selecting matrix A and analytical function ξ(λ),we can obtain the SCE, the CCF and the SCCF from (61a) or (62a).

7.2 The SCE, CCF and SCCF

−1 Let A = H∞, ξ(λ) = (1 + λ) . Evidently, the matrix Fig. 6 Condition numbers of calibration families (64) corre- − − ξ(τH∞) = (I +τH∞) 1 is commutative with H∞. So, sub- sponding to 5 infinite homographies H∞ = KRK 1 where K : fu = fv = 1000, s = 0.0, pu = pv = 512 and the rotation stituting it into (61a) we obtain the following one-parameter = · · = = − + Ri Rx (θi1) Ry (θi2) Rz(π/θi3)(i 1, 2, 3, 4, 5), θij is randomly calibration family with τ 1 (note the matrix (I τH∞) generated by the uniform distribution on the interval [−π/4,π/4].The is not invertible at τ =−1): abscissa and the ordinate are parameter τ and condition number respectively. We can see that: The condition numbers of these calibration fam- T −T −1 = H∞ (I + τH∞) ω(I + τH∞) H∞ ilies all reach their minimums at τ 1, corresponding to the SCE. The condition numbers of the CCE-Is (τ = 0) are far bigger than those of ψ(τ) the SCE-Is. And thus the numerical performance of the SCE is superior = + −T + −1 to that of the CCE-I (I τH∞) ω(I τH∞) . (64) ψ(τ) equation (65) is true. We have (66) from This is a very interesting calibration family. First, it has an −1 I − ϕ(H∞) = I − (I − H∞)(I + H∞) explicit geometrical meaning. The matrix I + τH∞ trans- −1 forms point on the IAC, m ∈ ω,intom + τm (m = H∞m). = (I + H∞) (I + H∞ − I + H∞) Since H∞ preserves the IAC, m + τm is a linear combi- −1 −1 = 2(I + H∞) H∞ = 2H∞(I + H∞) . nation of a pair of corresponding points of H∞ on the IAC, = called here the projective τ -fractile of H∞. And thus the By setting τ 1in(64) and from (65) and (66), we can conic ψ(τ) is composed of all projective τ -fractiles of H∞, obtain. T called the projective τ -fractile curve of H∞. Equation (64) (I − ϕ(H∞)) ω(I − ϕ(H∞)) indicates the infinite homography preserves its projective τ - T fractile curve (τ = −1). = (I + ϕ(H∞)) ω(I + ϕ(H∞)). Second, the calibration family (64) contains the CCE-I By simplifying this equation, we have and the SCE. Let τ = 0, then (64) becomes the CCE-I. Next T + = T + = we show that the equation corresponding to τ = 1 is just the ϕ (H∞)ω ωϕ(H∞) 0i.e.,H ω ωH 0. SCE. Firstly, we give two properties of the Cayley transfor- Third, we can study the numerical performance of both mation ϕ(H∞) as follows: the CCE and the SCE, from the calibration family (64). + Figure 6 gives the condition numbers of the calibration I ϕ(H∞) −1 = (I + H∞) , (65) families from 5 infinite homographies. From this figure, 2 − we can see that: The condition numbers of these calibra- I ϕ(H∞) −1 = H∞(I + H∞) . (66) tion families all reach their minimums at τ = 1, corre- 2 sponding to the SCEs. The condition numbers of the CCEs A proof of (64) can be done as: Because (τ = 0) are far bigger than those of the SCEs. And thus −1 the numerical performance of the SCE is superior to that I + ϕ(H∞) = I + (I − H∞)(I + H∞) of the CCE-I. −1 = (I + H∞) (I + H∞ + I − H∞) Next, we will show that the CCF can be obtained with

−1 (62a) by selecting proper matrix A and analytical func- = 2(I + H∞) , − tion ξ(λ).LetA = H,ξ(λ) = (1 + λ) 1. Since the matrix 174 Int J Comput Vis (2009) 82: 156–184 ξ(τH) = (I + τH)−1 is invertible for τ ∈ (−∞, +∞) and is commutative with H , substituting it into (62a) we obtain another interesting calibration family:

H T (I + τH)−T ω(I + τH)−1 − − + (I + τH) T ω(I + τH) 1H = 0, τ ∈ (−∞, +∞). (67)

In this calibration family, the two equations corresponding to τ = 0 and 1 are the SCE and the CCE-I respectively. Figure 7 gives the condition numbers of the calibration families corresponding to 5 infinite homographies. From this figure, we can see that: The condition numbers of these calibration families all reach their minimums at τ = 0, corre- Fig. 7 Condition numbers of calibration families (67) corresponding to the SCEs. The condition numbers of the CCE-Is − sponding to 5 infinite homographies H∞ = KRK 1 where τ = ) ( 1 are far bigger than those of the SCEs. This indicates K : fu = fv = 1000, s = 0.0,pu = pv = 512 and the rotation again that the numerical performance of the SCE is superior Ri = Rx (θi1) · Ry (θi2) · Rz(π/θi3)(i= 1, 2, 3, 4, 5), θij is randomly to that of the CCE-I. generated by the uniform distribution on the interval [−π/4,π/4].The τ By comparing Fig. 7 with Fig. 3(a), we can find that the abscissa and the ordinate are parameter and condition number respectively. We can see that: The condition numbers of these calibration two figures are very similar. This is not a more coincidence, families all reach their minimums at τ = 0, corresponding to the SCE. because in the next we can prove that when τ = 0, the cal- The condition numbers of the CCE-Is (τ = 1) are far bigger than those ibration family (67) is just the CCF (from this, we see that of the SCEs. This indicates again that the numerical performance of the SCE is superior to that of the CCE-I the definition “CCF(0) =SCE” is very natural in Sect. 5.3). A proof can be done as follows. When τ = 0, (67) can be rewritten as Notice that ∀n ∈ Z+, from the SCE we have

1 − − T 2n+1 + 2n+1 = T + 2n (I + τH) T (2(τH )T ω + 2ω(τH ))(I + τH) 1 = 0. (τH ) ω ω(τH) (τH ω ω(τH ))(τH ) . 2τ (68) Then, It is not difficult to see that sinT (τH )ω + ω sin(τH ) = ((τH )T ω + ω(τH ))f (τH ), 2(τH )T ω + 2ω(τH) where = + T + − − T − (I τH) ω(I τH) (I τH) ω(I τH). ∞ (τH )2n f(τH)= I + (−1)n (|τ| <π/2) Substituting it into (68), we have (2n + 1)! n=1 1 ((I + τH)−T (I − τH)T ω(I − τH )(I + τH)−1 −ω) is invertible and is commutative with (I + sin(τH ))−1.And 2τ thus, ϕT (τH ) ϕ(τH) = 0. (τH )T ω + ω(τH) = T + −1 And thus, ϕT (τH )ωϕ(τH ) − ω = 0. This is just the CCF. (sin (τH )ω ω sin(τH ))f (τH ) Finally, we prove that by selecting proper matrix A and or, equivalently, analytical function ξ(λ), the SCCF can also be obtained from (62a). 1 − H T ω + ωH = (sinT (τH )ω + ω sin(τH ))f 1(τH ). τ Proof Let A = H , ξ(λ) = (1 + sin(λ))−1. Then the matrix = + −1 − π π Substituting it into (69), we have ξ(τH) (I sin(τH )) has definition in ( 2 , 2 ). So, substituting it into (62a)wehave − (I + sin(τH )) T (sinT (τH )ω + ω sin(τH )) − − + T T + + 1 = − (I sin(τH )) (H ω ωH )(I sin(τH )) 0. (69) × (I + sin(τH )) 1 = 0. (70) Int J Comput Vis (2009) 82: 156–184 175 Notice that results, for each noise level we perform 1000 trials. We use the RMS errors relative to the ground truth to evaluate the T + 2(sin (τH )ω ω sin(τH )) numerical stability of algorithms. The experimental results = (I + sin(τH ))T ω(I + sin(τH )) are shown in Fig. 8. From the Fig. 8, we can see that: • For the magnificationf in the u direction, among all − (I − sin(τH ))T ω(I − sin(τH )), u the algorithms, the CCE-I performs the worst and the and substituting it into (70), we have CCE-II performs the best. The performance of the CCF-II (SCCF-II) is remarkably better than that of the CCF-I (ϕ(sin(τH )))T ωϕ(sin(τH )) − ω = 0. (SCCF-I). The SCE lies between the CCF and the SCCF. • For the magnification fv in the v direction, all the algorithms perform comparably. 8 Experiments • For the skew, among all the algorithms, the SCE performs the best and the CCE-I performs the worst. The perfor- In this section, we report the results of experiments with mance of the CCF-II (SCCF-II) is superior to that of the simulated and real data to evaluate the numerical stability CCF-I (SCCF-I). The performance of the CCE-II is better of algorithms. In all Figures and Tables, CCE, SCE, CCF than that of the CCE-I, but inferior to the rest of all the and GCCF stand for the classical algorithm, the standard al- other algorithms. gorithm, the Cayley algorithm and the generalized Cayley • For the u-coordinate of the principal point, among all algorithm, respectively. the algorithms, both the CCE-I and the CCE-II perform the worst and the SCE performs the best. The CCF-I 8.1 Experiments with Simulated Data (SCCF-I) is superior to that of the CCF-II (SCCF-II). • For the v-coordinate of the principal point, among all the The intrinsic matrix of the simulated camera is: algorithms, the SCE performs the best and the CCE-I performs the worst. The performance of the CCF-I (SCCF-I) f = f = 1000,p= p = 512,s= 0.02. u v u v is superior to that of the CCF-II (SCCF-II). The perfor- The extrinsic matrices of three views are respectively: mance of the CCE-II is better than that of the CCE-I, but inferior to the rest of all the other algorithms. E0 = (I|0), E1 = (R1|0), E2 = (R2|0). • For the two rotation axes, among all the algorithms, the SCE performs also the best and the CCE-I performs Here the worst. The performance of the CCF-II (SCCF-II) is ⎛ ⎞ 0.8536 −0.3536 −0.3827 slightly better than that of the CCF-I (SCCF-I). The per- ⎝ ⎠ formance of the CCE-II is also better than that of the R1 = 0.2183 0.9096 −0.3536 , 0.4731 0.2183 0.8536 CCE-I, but inferior to the rest of the other algorithms. ⎛ ⎞ • For the two rotation angles, all the algorithms have almost 0.8924 −0.3696 −0.2588 the same performance. ⎝ ⎠ R2 = 0.2119 0.8496 −0.4830 . 0.3984 0.3762 0.8365 On the whole, the SCE performs the best; the CCE-I performs the worst; the performance of the CCE-II is supe- Their respective rotation axes and angles are: rior to that of the CCE-I, but inferior to the rest of all the other algorithms; the performance of the CCF-II (SCCF-II) ◦ a1 = (0.4857, −0.7268, 0.4857), θ1 = 36.06 , is slightly better than that of the CCF-I (SCCF-I). ◦ It is quite natural that both the CCF and SCCF are su- a2 = (0.6996, −0.5352, 0.4735), θ2 = 37.88 . perior to the CCE, because they optimize the conditioning Because the motions between the views are pure rota- of calibration equations with the CCE as their initialization. tions around the camera center, we can linearly compute At the first glace, since the CCF and the SCCF used the op- the infinite homography with image correspondences from timal technique, they would provide a better/same precision 3D space points (Hartley 1997b). In the experiments, we of calibration than/as the SCE, however it is not validated by choose randomly 50 space points in the cube [0, 50]× our experiment. This is most probably because: Though the [0, 50]×[80, 130] such that all their image points are in CCF and the SCCF improve the conditioning of the prob- [0, 1024]×[0, 1024]. And then, Gaussian noise with zero lems by optimizing the parameters in the calibration fami- mean and standard deviation σ is added to each image point. lies, their condition numbers have the same order of magni- The noise level σ is varied from 0.3 to 3 pixels with a step of tude with the condition number of the SCE. Since the com- 0.3 pixels. In order to provide more statistically meaningful plexities of calibration equations in the CCF and the SCCF 176 Int J Comput Vis (2009) 82: 156–184

(a) (b)

(e) (f)

Fig. 8 The RMS errors of the estimated intrinsic matrix K and the ro- pal point. (f): The rotation axis a1.(g): The rotation axis a2.(h): The tations (R1,R2).(a): The magnification in the u direction. (b): The rotation angle θ1.(i): The rotation angle θ2. Here the RMS error of ro- magnification in the v direction. (c): The skew parameter. (d): The tation axis is defined as the RMS of the angle between the computed u-coordinate of the principal point. (e): The v-coordinate of the princi- value and the ground truth Int J Comput Vis (2009) 82: 156–184 177

(g) (h)

(i)

Fig. 8 (Continued)

are far higher than that of the SCE and the more inverse isfying angle (a1, a2) = φ is chosen randomly to construct and nonlinear matrix computations are involved, the more the other rotation R2, whose rotation angle θ2 is randomly numerical errors are introduced in the CCF and the SCCF. selected. The angle φ is increasingly varied from 0.5◦ to 10◦ As a result, neither the CCF nor the SCCF achieves a better with the steps of 0.5◦. In order to provide more statistically calibration precision than the SCE in our experiments. meaningful results, we perform 1000 trials for each angle φ. The experimental results are shown in Fig. 9. 8.2 Critical Motions For the success rate, from Fig. 9(a) we can see that: the SCE performs the best. The CCF-I and SCCF-I have almost Calibration algorithms usually do not provide a positive def- the same performance, and both are quite close to the SCE. inite ω, i.e., usually fail, in the case that views are close to a The algorithms of the second type have also almost the same critical motion sequence. Being a good algorithm, it should performance, but they perform the worst. The performance have a higher success rate and provides a reliable calibra- of CCE-I lies between the performances of the algorithms tion when succeeded. Rotating camera around the same axis of the ﬁrst type (CCF-I, SCCF-I) and the second type. The leads to a critical motion sequence for the CCE. Here, the success rates of all algorithms increase with the increase of success rate and the calibration accuracy when the motion is angle φ. case close to the critical motion sequence are used to evalu- For the accuracy of calibration, Fig. 9(b–f) show that: ate the CCE, SCE, CCF and GCCF. Firstly we choose ran- The SCE, the CCF-I and the SCCF-I have almost the same domly a unit vector a1 and an angle θ1 to construct a rota- performance, they all are remarkably superior to the other tion R1. Then given an angle φ, the other unit vector a2 sat- algorithms. Among the CCE-I and the algorithms of the sec- 178 Int J Comput Vis (2009) 82: 156–184

(a) (b)

(e) (f)

Fig. 9 The success rate and the RMS error of calibration in the case with zero mean and standard deviation 1. (a): The success rate. (b): The that the motion is close to the critical motion sequence. Here the ground magnification in the u direction. (c): The magnification in the v direc- truth of intrinsic parameters is (1000, 1000, 0.02, 512, 512) and the in- tion. (d): The skew parameter. (e): The u-coordinate of the principal finite homographies are computed by 50 image point correspondences point. (f): The v-coordinate of the principal point Int J Comput Vis (2009) 82: 156–184 179 ond type, they are of no comparability because their perfor- • The principal points by The SCE and the algorithms of the mances are different with the different intrinsic parameters. second type are close to the image center. The principal On the whole, the SCE always performs the best, the points from the algorithms of the first type are far away CCE-I and the algorithms of second type perform the worst from the image center. among all algorithms and the performances of the CCF-I and This experiment indicates that the means of K computed SCCF-I are very close to that of the SCE. It is not surprising by the SCE and the algorithms of the second type are the that the algorithms of the second type have the worst per- most viable ones. formances, because in these algorithms additional inverse For the standard deviation, from Fig. 13 we can see that: matrix computations, which are very unstable to the image data close to a critical motion sequence, are involved. This • Among all algorithms, the SCE always has the smallest experiment tells us that we should use the SCE, the CCF-I standard deviation, but the CCE-I the biggest one. or the SCCF-I for camera calibration in case the image data • For the magnifications in the two direction, the standard close to a critical motion sequence. Considering computa- deviations from the algorithms of the second type are tional complexity of these algorithms, we recommend the slightly bigger than those from the algorithms of the first SCE be used for camera calibration in practice. type except for the CCF-I. • For the skew, except for the CCE-I and CCE-II, the al- 8.3 Experiments with Real Data gorithms of the two types give almost the same standard deviation. But, the standard deviation from the CCE-II is In this section, we report two experiments with real data. remarkably smaller than those from the algorithms of the The first is an experiment on the images of a real scene cap- first type. • tured by an rotating camera around its center; the second is For the principal point, the standard deviations from the an experiment on the images of a 3D reference object. algorithms of the second type are remarkably smaller than those from the algorithms of the first type. Real Scene On the whole, the SCE performs the best; the CCE-I performs the worst; the algorithms of the second type are su- In this experiment, the used three images are shown in perior to the algorithms of the first type. A smaller standard Fig. 10. The image size is of 2048× 1536. The experimental deviation means a good numerical stability. And thus, this procedure is as follows. experiments indicate that the SCE and the CCE-I have the Firstly, the corner points in the images are extracted by best and worst numerical stability among all the algorithms, the Harris detector (Harris and Stephens 1988), and matched respectively. The numerical stability of the algorithms of the by the SFIT descriptor (Lowe 2004) and the epipolar geom- second type is better than that of the algorithms of the first etry computed with RANSAC technique (Hartley and Zis- type. serman 2000) is used to remove false matches. We obtain 432 correct matches between the images (a) and (b) and 244 Reference Object correct matches between the images (a) and (c), as shown in Fig. 11. Then, we compute the infinite homographies with In this experiment, the used three images are shown in N (= 40, 55, 70, 85, 100, 115, 130, 145, 160, 175, 190, Fig. 14, the image size is of 2490 × 1920. The experimental 205, 220) matches chosen randomly in the two match sets. procedure is as follows. Finally, the intrinsic matrix K is computed by different algo- 1. Calibration based on the 3D reference object. Firstly, we rithms. For each match number N, we perform 1000 trials, use the correspondences between 3D points on the ref- and then compute the mean and the standard deviation of the erence object and their images to compute the camera estimated K. projection matrices by the DLT method. Then, we deter- The experimental results are given in Figs. 12 and 13. mine the intrinsic matrix by taking QR factorization of From Fig. 12, we can see that: the first 3 × 3 sub-matrices of the camera matrices. Since • The aspect ratios by the SCE and the algorithms of second the errors of image localizations and the numerical com- type (CCE-II, CCF-II and SCCF-II) are closer to 1 than putation, the estimated intrinsic matrices corresponding the algorithms of first type (CCE-I, CCF-I and SCCF-I). to the three views are susceptible to small variations. The • The SCE and CCE-II provide almost the same skew fac- average value is here regarded as the final estimation of tor, they are much closer to zero than those by other algo- the intrinsic matrix K. The experimental results are given rithms. The skew factors from the CCF-II and the SCCF- in Table 1. II are also close to zero. The algorithms of the first type 2. Calibration from the IAC. Firstly, we use the image cor- produce big skew factors far away from zero. respondences from 3D space points on the reference 180 Int J Comput Vis (2009) 82: 156–184

Fig. 10 The three images a real scene

Fig. 11 The image point correspondences used in the experiments. (a): 432 image point correspondences between the ﬁrst and second images. (b): 244 image point correspondences between the ﬁrst and third images

object to compute the epipolar geometries between the algorithms. The algorithms of the second type are superior views, and then compute the infinite homographies using to the algorithms of the first type. the vanishing points of three orthogonal directions on the Both the two experiments with real data indicate that reference object. Then, we use the SCE, the CCE, the among all the algorithms the SCE and the CCE-I have the CCF and the SCCF to compute the intrinsic matrix K, best and worst numerical stability respectively. The numer- respectively. The experimental results are given also in ical stability of algorithms of the second type is superior to Table 1. that of the algorithms of the first type. And thus, in practice we should use the SCE or the algorithms of the second type For the convenience of comparison, we regard the cali- for camera calibration. Besides, the SCE is highly recom- bration from the 3D reference object as the true values of mended due to its high calibration precision and low com- the intrinsic parameters and for each parameter, compute its putational complexity. absolute error (AE(·)).WeuseSAE(f ), SAE(p) and AE(s) to evaluate the calibration accuracy of the algorithms, where SAE(f ) is the sum of AE(fu) and AE(fv), SAE(p) the sum 9 Conclusions of AE(pu) and AE(pv). Table 2 gives the statistics of the data in Table 1. From Table 2, we can see that: The SCE The numerical stability of self-calibration methods based provides the highest precision of calibration and the cali- on the AC is investigated in this paper. The main contribu- bration precision of the CCE-I is the lowest among all the tions are as follows: The SCE, the CCF and the GCCFs are Int J Comput Vis (2009) 82: 156–184 181

(a) (b)

(e)

Fig. 12 The means of the intrinsic matrix K.(a): The magniﬁcation in the u direction. (b): The magniﬁcation in the v direction. (c): The skew parameter. (d): The u-coordinate of the principal point. (e): The v-coordinate of the principal point derived from the Cayley transformation or the generalized analyses and large numbers of experiments, it is found that transformations. All the calibration equations in this paper the SCE can always provide a better numerical stability and are mathematically equivalent to each other, but they man- has a lower computational complexity. And thus, the SCE is ifest different numerical performances. Through numerical highly recommended for calibrate camera in practice. 182 Int J Comput Vis (2009) 82: 156–184

(a) (b)

(e)

Fig. 13 The standard deviations of the intrinsic matrix K.(a): The magniﬁcation in the u direction. (b): The magniﬁcation in the v direction. (c): The skew parameter. (d): The u-coordinate of the principal point. (e): The v-coordinate of the principal point Int J Comput Vis (2009) 82: 156–184 183

Fig. 14 The three images a reference object and point correspondences used in the experiment

Table 1 The experimental results with a reference object: Methods Results the intrinsic matrix K.Here fu fv spu pv 3DA denotes the algorithm based on 3D reference object 3DA 2574.4 2574.5 −1.0 1254.7 943.3 SCE 2568.9 2573.6 −20.0 1261.7 898.6 CCE-I 2552.5 2548.5 −35.1 1269.3 873.7 CCE-II 2558.3 2575.8 −25.2 1265.7 897.8 CCF-I 2566.2 2573.6 −25.6 1262.5 897.2 CCF-II 2568.6 2574.6 −25.1 1259.8 897.1 SCCF-I 2566.5 2571.7 −26.6 1262.9 895.3 SCCF-II 2569.0 2576.5 −25.8 1259.7 897.0

Table 2 The experimental results with a reference object: the absolute ies on numerical stability of calibration are in essence based errors on affine construction since the assumption of known in- Methods Results finite homography. We do not consider the numerical sta- SAE(f ) SAE(p) AE(s) bility of self-calibration algorithms in projective construction, such as the self-calibration algorithm based on the SCE 6.4 51.7 19.0 modulus constraint (Pollefeys et al. 1996) and the self- CCE-I 47.9 84.2 34.1 calibration algorithm based on the dual absolute quadric CCE-II 17.4 56.5 24.1 (Triggs 1997; Ponce et al. 2000). Can the Cayley transfor- CCF-I 9.1 53.9 24.1 mation be used to improve numerical stability of these al- . CCF-II 5 9 51.3 24.6 gorithms? It is still an open question, and will be our future SCCF-I 10.7 56.2 25.6 work. SCCF-II 7.4 51.3 24.8 The studies in this paper indicate that the different expressions of the same problem can result in different numerical performances, even if these expressions are mathemat- We reported only our studies on the numerical stabil- ically equivalent to each other. The existing algorithms for ity of camera calibration based on the AC. If the infinite many problems in computer vision mostly have a poor nu- homography is known, the calibration equation based on merical stability, such as those for three dimensional recon- the dual absolute quadric (DAC) has similar form with struction and motion analysis. By investigating equivalent the calibration equation based on the AC (Hartley 1997b). forms of these problems, we could probably obtain more The Cayley transformation can be used to improve the nu- stable algorithms in the future. merical stability of calibration equation based on the DAC and we can also obtain the corresponding SCE, CCF and Acknowledgements We wish to thank the anonymous reviewers GCCF. These calibration equations and calibration fami- for their inspiring comments and suggestions on this paper. Also, we gratefully acknowledge the support from the National Natural Science lies have similar numerical performances with the SCE, Foundation of China (60835003, 60575019) and the National High- CCF and GCCF based on the AC. In this paper, the stud- Tech Research and Development Program of China (2006AA01Z116). 184 Int J Comput Vis (2009) 82: 156–184

References Pollefeys, M., & Gool, L. (1999). Stratified self-calibration with the modulus constraint. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(8), 707–724. Brown, D. C. (1971). Close-range camera calibration. Photogrammet- Pollefeys, M., Gool, L., & Osterlinck, A. (1996). The modulus con- ric Engineering, 37(8), 855–866. straint: A new constraint for self-calibration. In: Proc. of interna- Bell, S. (1992). The Cayley transform, potential theory and conformal tional conference of pattern recognition (pp. 31–42). mapping. Boca Raton: CRC Press. Ponce, J., McHenry, K., Papadopoulo, T., Teillaud, M., & Teiggs, B. Faig, W. (1975). Calibration of close-range photogrammetry systems: (2000). On the absolute quadratic complex and its application to Photogrammetric Engineering and Re- Mathematical formulaton. autocalibration. In Proc. of computer vision and pattern recogni- mote Sensing, 41(21), 1479–1486. tion (pp. 780–787). Faugeras, O. (1993). Three-dimensional computer vision: A geometric Press, W. H., Flannery, B. P., Teukolsky, S. A., & Vetterling, W. T. viewpoint. Cambridge: MIT Press. (1988). Numerical recipes in C: The art of scientific computing. Faugeras, O., Luong, Q.-T., & Manybank, S. J. (1992). Camera self- Cambridge: Cambridge University Press. calibration: Theroy and experiments. In Proc. of European con- Sturm, P., & Maybank, S. J. (1999). On plane-based camera calibra- ference on computer vision (pp. 321–334). tion: A general algorithm, singularities, applications. In Proc. of Hartley, R. (1997a). Kruppa’s equations derived from the fundamen- computer vision and pattern recognition (pp. 432–437). tal matrix. IEEE Transactions on Pattern Analysis and Machine Sturm, P., & Triggs, B. (1996). A factorization based algorithm for Intelligence, 19(2), 133–135. mulit-image projective structure and motion. In Proc. of European Hartley, R. (1997b). Self-calibration of stationary cameras. Interna- conference on computer vision (pp. 709–720). tional Journal of Computer Vision, 22(1), 5—23. Tsai, R. (1986). An efficient and accurate camera calibration technique Hartley, R. (1997c). In defense of the eight-point algorithm. IEEE for 3D machine vision. In Proc. of computer vision and pattern Transactions on Pattern Analysis and Machine Intelligence, recognition (pp. 364–374). Miami Beach, USA. 19(6), 580–593. Triggs, B. (1997). Auto-calibration and the absolute quadric. In Proc. Hartley, R., & Zisserman, A. (2000). Multiple view geometry in com- of computer vision and pattern recognition (pp. 609–614). puter vision. Cambridge: Cambridge University Press. Wang, L., Wu, F. C., & Hu, Z. Y. (2007). Multi-camera calibration Harris, C., & Stephens, M. J. (1988). A combined corner and edge with one-dimensional object under general motions. In Proc. of detector. In Proceeding 4th alvey vision conference. international conference on computer vision (pp. 1–7). Luong, Q. T., & Faugeras, O. (1997). Self-calibration of a moving cam- Wu, F. C., Hu, Z. Y.,& Zhu, H. J. (2005). Camera calibration with mov- era from point correspondence and fundamental matrices. Inter- ing one-dimensional objects. Pattern Recognition, 38(5), 755– national Journal of Computer Vision, 22(3), 261–289. 765. Lowe, D. G. (2004). Distinctive image feature from scale invariant key- Zhang, Z. (1999). Flexible camera calibration by viewing a plane from point. International Journal of Computer Vision, 60(2), 91–110. unknown orientations. In Proc. of international conference on Maybank, S. J., & Faugeras, O. (1992). A theory of self-calibration of computer vision (pp. 666–673). a moving camera. International Journal of Computer Vision, 8(2), Zhang, Z. (2000). A flexible new technique for camera calibration. 123–152. IEEE Transactions on Pattern Analysis and Machine Intelligence, Oliensis, J., & Hartley, R. (2007). Iterative extensions of the 22(11), 1330–1334. Sturm/Trigs algorithm: Convergence and nonconvergence. IEEE Zhang, Z. (2004). Camera calibration with one-dimensional objects. Transactions on Pattern Analysis and Machine Intelligence, IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(12), 2217–2233. 26(7), 892–899.