Calibration of Ultra-Wide Fisheye Lens Cameras by Eigenvalue Minimization

Memoirs of the Faculty of Engineering, Okayama University, Vol. 46, pp. 10-20, January 2012

Kenichi KANATANI∗ Department of Computer Science, Okayama University Okayama 700-8530 Japan

(Received November 21, 2011)

We present a new technique for calibrating ultra-wide fisheye lens cameras by imposing the constraint that collinear points be rectified to be collinear, parallel lines to be parallel, and orthogonal lines to be orthogonal. Exploiting the fact that line fitting reduces to an eigenvalue problem, we do a rigorous perturbation analysis to obtain a Levenberg-Marquardt procedure for the optimization. Doing experiments, we point out that spurious solutions exist if collinearity and parallelism alone are imposed. Our technique has many desirable properties. For example, no metric information is required about the reference pattern or the camera position, and separate stripe patterns can be displayed on a video screen to generate a virtual grid, eliminating the grid point extraction processing.

1. Introduction et al. [8], and Okutsu et al. [13]. Komagata et al. [9] further introduced the parallelism constraint and the Fisheye lens cameras are widely used for surveil- orthogonality constraint, requiring that parallel lines lance purposes because of their wide angles of view. be rectified to be parallel and orthogonal lines to be They are also mounted on vehicles for various pur- orthogonal. However, the cost function has been di- poses including obstacle detection, self-localization, rectly minimized by brute force means such as the and bird’s eye view generation [10, 13]. However, Brent method and the Powell method [16]. fisheye lens images have a large distortion, so that In this paper, we adopt the collinearity- in order to apply the computer vision techniques ac- parallelism-orthogonality constraint of Komagata et cumulated for the past decades, one first needs to al. [9] and optimize it by eigenvalue minimization. rectify the image into a perspective view. Already, The fact that imposing collinearity implies eigenvalue there is a lot of literature for this [3, 5, 7, 8, 11, 12, minimization and that the optimization can be done 13, 14, 17, 18]. by invoking the perturbation theorem was pointed The standard approach is to place a reference out by Onodera and Kanatani [15]. Using this princi- grid plane and match the image with the reference, ple, they rectified perspective images by gradient de- whose precise geometry is assumed to be known cent. Here, we consider ultra-wide fisheye lenses and [3, 4, 5, 7, 18]. However, this approach is not very do a more detailed perturbation analysis to derive the practical for recently popularized ultra-wide fisheye ◦ Levenberg-Marquardt procedure, currently regarded lenses, because they cover more than 180 angles of as the standard tool for efficient and accurate opti- view and hence any (even infinite) reference plane mization. cannot cover the entire field of view. This difficulty The eigenvalue minimization principle has not can be circumvented by using the collinearity con- been known in the past collinearity-based work [1, straint pointed out repeatedly, first by Onodera and 8, 12, 13, 17]. Demonstrating its usefulness for ultra- Kanatani [15] in 1992, later by Swaminathan and Na- wide fisheye lens calibration is the first contribution yar [17] in 2000, and by Devernay and Faugeras [1] in of this paper. We also show by experiments that 2001. They pointed out that camera calibration can the orthogonality constraint plays an essential role, be done by imposing the constraint that straight lines pointing out that a spurious solution exists if only should be rectified to be straight. This principle was collinearity and parallelism are imposed. This fact applied to fisheye lenses by Nakano, et al. [12], Kase has not been known in the past collinearity-based ∗E-mail [email protected] work [1, 8, 12, 13, 15, 17]. Pointing this out is the

This work is subjected to copyright. All rights are reserved by this author/authors. 10 Kenichi KANATANI MEM.FAC.ENG.OKA.UNI. Vol. 46

second contribution of this paper. For data acquisition, we take images of stripes of different orientations on a large-screen display by placing the camera in various positions. Like the past θ m collinearity-based methods [1, 8, 12, 13, 15, 17], our method is non-metric in the sense that no metric information is required about the camera position or the reference pattern. Yet, many researchers pointed f out the necessity of some auxiliary information. For example, Nakano et al. [11, 12] proposed vanishing r point estimation using conic fitting to straight line images (recently, Hughes et al. [5] proposed this same Figure 1: The imaging geometry of a fisheye lens and the technique again). Okutsu et al. [14] picked out the incident ray vector m. images of antipodal points by hand. Such auxiliary information may be useful to suppress spurious so- and determine the values of f, a1, a2, ... along with lutions. However, we show that accurate calibra- the principal point position. Here, f0 is a scale con- tion is possible without any auxiliary information stant to keep the powers rk within a reasonable nu- by our eigenvalue minimization for the collinearity- merical range (in our experiment, we used the value parallelism-orthogonality constraint. f0 = 150 pixels). The linear term r/f0 has no coeffi- This paper is organized as follows. In Sec. 2, we cient because f on the right-hand side is an unknown describe our imaging geometry model. Section 3 gives parameter; a1 = a2 = ··· = 0 corresponds to the derivative expressions of the fundamental quantities, stereographic projection. Since a sufficient number of followed by a detailed perturbation analysis of the correction terms could approximate any function, the collinearity constraint in Sec. 4, of the parallelism right-hand side of (2) could be any function of θ, e.g., constraint in Sec. 5, and of the orthogonality con- the perspective projection model (f/f0) tan θ or the straint in Sec. 6. Section 7 describes our Levenberg- equidistance projection model (f/f0)θ. We adopt the Marquardt procedure for eigenvalue minimization. In stereographic projection model merely for the ease of Sec. 8, we experiment our non-metric technique, using initialization. stripe images on a video display. We point out that In (2), even power terms do not exist, because the a spurious solution arises if only collinearity and par- lens has circular symmetry; r is an odd function of allelism are imposed and that it can be eliminated θ. We assume that the azimuthal angle of the projec- without using any auxiliary information if orthogo- tion is equal to that of the incident ray. In the past, nality is introduced. We also show some examples of these two were often assumed to be slightly different, real scene applications. In Sec. 9, we conclude. and geometric correction of the resulting “tangential distortion” was studied. Currently, the lens manu- 2. Geometry of Fisheye Lens Imaging facturing technology is highly advanced so that the tangential distortion can be safely ignored. If not, We consider recently popularized ultra-wide fish- we can simply include the tangential distortion terms eye lenses with the imaging geometry modeled by the in (2), and the subsequent calibration procedure re- stereographic projection mains unchanged. In the literature, the model of the form r = c1θ + θ 3 5 r = 2f tan , (1) c2θ + c3θ + ··· is frequently assumed [7, 8, 13, 12]. 2 As we see shortly, however, the value of θ for a spec- where θ is the incidence angle (the angle of the inci- ified r is necessary in each step of the optimization dent ray of light from the optical axis) and r (in pix- iterations. So, many authors computed θ by solv- els) is the distance of the corresponding image point ing a polynomial equation using a numerical means from the principal point (Fig. 1). The constant f [7, 8, 13, 12], but this causes loss of accuracy and ef- is called the focal length. We consider (1) merely be- ficiency. It is more convenient to express θ in terms cause our camera is as such, but the following calibra- of r from the beginning. From (2), the expression of tion procedure is identical whatever model is used. θ is given by

For a manufactured lens, the value of f is unknown ( ( ( )3 ( )5 )) −1 f0 r r r or may not be exact if provided by the manufacturer. θ = 2 tan + a1 + a2 + ··· . Also, the principal point may not be at the center 2f f0 f0 f0 (3) of the image frame. Moreover, (1) is an idealization, and a real lens may not exactly satisfy it. So, we generalize (1) into the form 3. Incident Ray Vector ( ) ( ) 3 5 Let m be the unit vector in the direction of the r r r ··· 2f θ + a1 + a2 + = tan , (2) incident ray of light (Fig. 1); we call m the incident f0 f0 f0 f0 2

11 January 2012 Calibration of Ultra-Wide Fisheye Lens Cameras by Eigenvalue Minimization

( ∞ ( ) ) ∑ 2k − ray vector. In polar coordinates, it has the expression ∂θ − 1 2 θ − r y v0   = cos 1 + (2k 1)ak . ∂v0 f 2 f0 r sin θ cos φ k=1 m =  sin θ sin φ  , (4) (10) cos θ Substituting these into (9), we can evaluate ∂m/∂u0 where θ is the incidence angle from the Z-axis and φ and ∂m/∂v0. is the azimuthal angle from the X-axis. Since φ, by Next, we consider derivation with respect to f. our assumption, equals the azimuthal angle on the Differentiating (2) with respect to f on both sides, image plane, the point (x, y) on which the incident we have light focuses is specified by 2 θ 2f 1 ∂θ 0 = tan + 2 . (11) x − u0 = r cos φ, y − v0 = r sin φ, f0 2 f0 2 cos (θ/2) ∂f √ 2 2 Hence, we obtain r = (x − u0) + (y − v0) , (5) where (u , v ) is the principal point. Hence, (4) is ∂θ 2 θ θ 1 0 0 = − sin cos = − sin θ. (12) rewritten as ∂f f 2 2 f   − ((x u0)/r) sin θ It follows that the derivative of (6) with respect to f   m = ((y − v0)/r) sin θ . (6) is cos θ   ((x − u )/r) cos θ ∂m 1 0 Differentiating (5) with respect to u and v , we ob- = − sin θ  ((y − v )/r) cos θ  . (13) 0 0 ∂f f 0 tain − sin θ

∂r x − u0 ∂r y − v0 = − , = − . (7) Finally, we consider derivation with respect to ak. ∂u0 r ∂v0 r Diﬀerentiating (2) with respect to ak on both sides, Hence, the derivatives of (6) with respect to u and we have 0 ( ) v0 become r 2k+1 2f 1 ∂θ   = 2 . (14) −1/r + (x − u )2/r3 f0 f0 2 cos (θ/2) ∂ak ∂m 0  − − 3  = (x u0)(y v0)/r sin θ Hence, we obtain ∂u0 0 ( )   ∂θ f r 2k+1 θ (x − u )/r) cos θ 0 2 0 ∂θ = cos . (15) +  ((y − v )/r) cos θ  , ∂ak f f0 2 0 ∂u − sin θ 0   It follows that the derivation of (6) with respect to 3 (x − u )(y − v )/r ak is ∂m 0 0  − − 2 3    = 1/r + (y v0) /r sin θ − ∂v0 ( )2k+1 ((x u0)/r) cos θ 0 ∂m f0 r 2 θ     = cos ((y − v0)/r) cos θ ((x − u )/r) cos θ ∂ak f f0 2 − 0 ∂θ sin θ +  ((y − v )/r) cos θ  . (8) 0 ∂v (16) − sin θ 0 All the cost functions in the subsequent optimization are expressed in terms of the incident ray vector m. Diﬀerentiating (3) with respect to u0 and v0 on both Hence, we can compute derivatives of any cost func- sides, we obtain tion with respect to any parameter simply by com- ( ( ) ( ) ( ) ) 1 3a r 2 5a r 4 7a r 6 ∂r bining the above derivative expressions. + 1 + 2 + 3 + ··· f0 f0 f0 f0 f0 f0 f0 ∂u0 2f 1 ∂θ 4. Collinearity Constraint = 2 , f0 2 cos (θ/2) ∂u0 S ( ( ) ( ) ( ) ) Suppose we observe a collinear point sequence κ 1 3a r 2 5a r 4 7a r 6 ∂r + 1 + 2 + 3 + ··· (the subscript κ enumerates all existing sequences) f0 f0 f0 f0 f0 f0 f0 ∂v0 consisting of N points p1, ..., pN , and let m1,..., mN 2f 1 ∂θ be their incident ray vectors. If the camera is pre- = . (9) 2 cisely calibrated, the computed incident ray vectors f0 2 cos (θ/2) ∂v0 should be coplanar. Hence, if nκ is the unit normal to Hence, ∂θ/∂u0 and ∂θ/∂v0 can be written as the plane passing through the origin O (lens center) S ( ∞ ( ) ) and κ, we should have (nκ, mα) = 0, α = 1, ..., N ∑ 2k − ∂θ 1 2 θ r x u0 (Fig. 2). In the following, we denote the inner prod- = − cos 1 + (2k − 1)ak , ∂u0 f 2 f0 r k=1 uct of vectors a and b by (a, b). If the calibration

12 Kenichi KANATANI MEM.FAC.ENG.OKA.UNI. Vol. 46

Computing the inner product with nκ on both sides, we obtain

p1 (κ) ∂M (κ) ∂nκ (nκ, nκ) + (nκ, M ) nκ ∂c ∂c (κ) m1 ∂λ ∂n = min (n , n ) + λ(κ) (n , κ ). (22) ∂c κ κ min κ ∂c O p 2 mN N Since nκ is a unit vector, we have (nκ, nκ) = knκk = 1. Variations of a unit vector should be orthog- (κ) onal to itself, so (nκ, ∂nκ/∂c) = 0. Since M is (κ) a symmetric matrix, we have (nκ, M ∂nκ/∂c) = Figure 2: The incident ray vectors mα of collinear points (κ) (κ) (M nκ, ∂nκ/∂c) = λ (nκ, ∂nκ/∂c) = 0. Thus, p1, ..., pN are coplanar. min (22) implies

is incomplete, however, (nκ, mα) may not be strictly ∂λ(κ) ∂M (κ) zero. So, we adjust the parameters by minimizing min = (n , n ). (23) ∂c κ ∂c κ ∑ ∑ 2 > > This result is well known as the perturbation theorem (nκ, mα) = nκ mαmα nκ

α∈Sκ α∈Sκ of eigenvalue problems [6]. From the deﬁnition of ∑ (κ) > (κ) M in (49), we see that = (nκ, mαmα nκ) = (nκ, M nκ), (17)

α∈Sκ N ( ( ) ) ∂M (κ) ∑ ∂m ∂m > = α m> + m α ∂c ∂c α α ∂c where we deﬁne α=1 ∑ > ∑N (κ) ∂mα > M = mαmα . (18) S ≡ (κ) = 2 [ mα ] M c , (24) ∈S ∂c α κ α=1

Since (17) is a quadratic form of M (κ), its minimum where S[ · ] denotes symmetrization (S[A] = (A + > (κ) (κ) A )/2). Thus, the first derivatives of the function J1 equals the smallest eigenvalue λmin of M , nκ being the corresponding unit eigenvector. To enforce the with respect to c = u0, v0, f, a1, a2, ... are given as collinearity constraint for all collinear point sequences follows: ∑ ∂J1 S , we determine the parameters so as to minimize = (n , M (κ)n ). (25) κ ∂c κ c κ ∑ all κ (κ) J1 = λmin. (19) 4.2 Second derivatives all κ 0 Differentiating (23) with respect to c (= u0, v0, f, Because we minimize the sum of eigenvalues, we a1, a2, ...), we obtain call our approach eigenvalue minimization, which was first proposed by Onodera and Kanatani [15] in 1992. ∂2λ(κ) ∂n ∂2M (κ) min = ( κ , M (κ)n ) + (n , n ) The reason that this technique has not been widely ∂c∂c0 ∂c0 c κ κ ∂c∂c0 κ used may be that at first sight it appears that one ∂n +(n , M (κ) κ ) cannot “differentiate” eigenvalues. However, differ- κ c ∂c0 entiating eigenvalue is very easy, as we now show in 2 (κ) ∂ M ∂nκ (κ) the following, if one uses the perturbation theorem = (nκ, 0 nκ) + 2( 0 , M c nκ). (26) well known in physics. ∂c∂c ∂c First, consider the first term. Differentiation of 4.1 First derivatives (24) with respect to c0 is (κ) Consider the first derivatives of λ with respect (κ) N ( ( ) ) min ∂2M ∑ ∂2m ∂m ∂m > to c, which represents the parameters u , v , f, a , = 2S[ α m> + α α ], 0 0 1 ∂c∂c0 ∂c∂c0 α ∂c ∂c0 a2, ... Differentiating the defining equation α=1 (27) (κ) (κ) and hence we have M nκ = λminnκ (20) N ( ∂2M (κ) ∑ ∂2m with respect to c on both sides, we have (n , n ) = 2 (n , α )(m , n ) κ ∂c∂c0 κ κ ∂c∂c0 α κ α=1 ) ∂M (κ) ∂n ∂λ(κ) ∂n ∂m ∂m n +M (κ) κ = min n +λ(κ) κ . (21) +(n , α )( α , n ) .(28) ∂c κ ∂c ∂c κ min ∂c κ ∂c ∂c0 κ

13 January 2012 Calibration of Ultra-Wide Fisheye Lens Cameras by Eigenvalue Minimization

If the calibration is complete, we should have (mα, nκ) = 0. In the course of the optimization, we can expect that (mα, nκ) ≈ 0. Hence, (28) can be nκ lg approximated by lg

2 (κ) ∑N ∂ M ∂mα ∂mα (n , n ) ≈ 2 (n , )( , n ) O κ ∂c∂c0 κ κ ∂c ∂c0 κ α=1 (κ) = 2(nκ, M cc0 nκ), (29)

∑N ( )( )> (κ) ∂mα ∂mα M 0 ≡ . (30) Figure 3: The surface normals nκ to the planes deﬁned cc ∂c ∂c0 α=1 by parallel lines are orthogonal to the common direction lg of the lines. This is a sort of the Gauss-Newton approximation. Next, consider the second term of (26). Because Combining (29) and (36), we can approximate (26) nκ is a unit vector, its variations are orthogonal to (κ) ≥ (κ) ≥ (κ) in the form itself. Let λ1 λ2 λmin be the eigenvalues of ( (κ) 2 (κ) M with nκ1, nκ2, and nκ the corresponding unit ∂ λmin (κ) ≈ 2 (nκ, M 0 nκ) eigenvectors. Since the eigenvectors of a symmetric ∂c∂c0 cc matrix are mutually orthogonal, any vector orthogo- ∑2 (κ) (κ) ) (nκi, M c nκ)(nκi, M c0 nκ) nal to n is expressed as a linear combination of n − . (37) κ κ1 λ(κ) − λ(κ) and nκ2. Hence, we can write i=1 i min Thus, the second derivatives of the function J with ∂n 1 κ = β n + β n , (31) respect to c and c0 are given by ∂c 1 κ1 2 κ2 2 ∑( ∂ J1 (κ) for some β and β . Substitution of (23) and (31) = 2 (n , M 0 n ) 1 2 ∂c∂c0 κ cc κ into (21) results in all κ ∑2 (κ) (κ) ) (κ) (κ) (nκi, M c nκ)(nκi, M c0 nκ) M c nκ + M (β1nκ1 + β2nκ2) − . (38) (κ) − (κ) (κ) (κ) i=1 λi λmin = (nκ, M c nκ)nκ + λmin(β1nκ1 + β2nκ2). (32) 5. Parallelism Constraint (κ) (κ) (κ) Noting that M nκ1 = λ1 nκ1 and M nκ2 = (κ) Let Gg be a group of parallel collinear point λ2 nκ2, we have sequences (the subscript g enumerates all existing (κ) − (κ) (κ) − (κ) β1(λ1 λmin)nκ1 + β2(λ2 λmin)nκ2 groups) with a common orientation lg (unit vector). (κ) − (κ) The normals nκ to the planes passing through the = (nκ, M c nκ)nκ M c nκ. (33) origin O (lens center) and lines of Gg are all orthog- Computing the inner product with nκ1 and nκ2 on onal to lg (Fig. 3). Hence, we should have (lg, nκ) both sides, we obtain = 0, κ ∈ Gg, if the calibration is complete. So, we adjust the parameters by minimizing (κ) − (κ) − (κ) ∑ ∑ β1(λ1 λmin) = (nκ1, M c nκ), 2 > > (lg, nκ) = l nκn lg (κ) (κ) (κ) g κ β2(λ − λ ) = −(nκ2, M nκ). (34) κ∈G κ∈G 2 min c g∑ g > (g) Thus, (31) is written as follows: = (lg, nκnκ lg) = (lg, N lg), (39)

κ∈Gg ∂n (n , M (κ)n )n (n , M (κ)n )n κ = − κ1 c κ κ1 − κ2 c κ κ2 . where we deﬁne ∂c (κ) − (κ) (κ) − (κ) ∑ λ1 λmin λ2 λmin (g) > (35) N = nκnκ . (40) ∈G This is also a well known result of the perturbation κ g theorem of eigenvalue problems [6]. Thus, the second Since (39) is a quadratic form of N (g), its minimum term of (26) can be written as (g) (g) equals the smallest eigenvalue µmin of N , lg be- (κ) (κ) ing the corresponding unit eigenvector. To enforce ∂nκ (κ) 2(nκ1, M c nκ)(nκ1, M c0 nκ) 2( , M nκ) = − the parallelism constraint for all groups of parallel ∂c0 c (κ) − (κ) λ1 λmin collinear sequences, we determine the parameters so (κ) (κ) 2(n , M n )(n , M 0 n ) as to minimize − κ2 c κ κ2 c κ . ∑ (κ) − (κ) (g) λ2 λmin J2 = µmin. (41) (36) all g

14 Kenichi KANATANI MEM.FAC.ENG.OKA.UNI. Vol. 46

l g' 6.1 First derivatives

The ﬁrst derivatives of the function J3 with respect lg to parameters c are given by ∑ ( ) ∂J3 ∂lg ∂lg0 = 2 (l , l 0 ) ( , l 0 ) + (l , ) . (47) ∂c g g ∂c g g ∂c all orthogonal {G G0 } pairs g, g

The ﬁrst derivative ∂lg/∂c is given by

Figure 4: If two sets of parallel lines make right angles, ∑2 (g) ∂lg (lgi, N c lg)lgi their directions lg and lg0 are orthogonal to each other. = − , (48) ∂c (g) − (g) i=1 µi µmin 5.1 First derivatives and ∂lg0 /∂c similarly. Doing the same perturbation analysis as in Sec. 4, 6.2 Second derivatives we obtain the first derivatives of the function J2 with respect to parameters c in the form Using the Gauss-Newton approximation (lg, lg0 ) ≈ ∑ ∂J 0, we obtain the second derivatives of the function J3 2 = (l , N (g)l ), (42) 0 ∂c g c g with respect to parameters c and c in the form all g 2 ∑ ( ) ∑ ∂ J3 ∂lg ∂lg0 ∂n = 2 ( , l 0 ) + (l , ) N (g) = 2S[ κ n>], (43) ∂c∂c0 ∂c g g ∂c c ∂c κ all orthogonal ∈G {G G0 } κ g pairs g, g ( ) where ∂nκ/∂c is given by (35). ∂lg ∂lg0 ( , l 0 ) + (l , ) . (49) ∂c0 g g ∂c0 5.2 Second derivatives 7. Levenberg-Marquardt Procedure Doing the same perturbation analysis as in Sec. 4, we obtain the second derivatives of the function J 2 To incorporate all of the collinearity, parallelism, with respect to parameters c and c0 in the form and orthogonality constraints, we minimize 2 ∑( ∂ J2 (g) = 2 (l , N 0 l ) J J J ∂c∂c0 g cc g J = 1 + 2 + 3 , (50) all g γ1 γ1 γ1 ∑2 (g) (g) ) (lgi, N lg)(lgi, N 0 lg) − c c , (44) where γi, i = 1, 2, 3, are the weights to balance the (g) − (g) À i=1 µi µmin magnitudes of the three terms. Note that J1 J2 À ( )( ) J3, since J1 is proportional to the number of all ∑ ∂n ∂n > (g) κ κ points, J2 to the number of all lines, and J3 to the N 0 ≡ , (45) cc ∂c ∂c0 number of orthogonal pairs of parallel lines. In our κ∈Gg experiment, we used as γi the initial value of Ji, so (g) where µi , i = 1, 2, are the first and the second that J is initially 1 + 1 + 1 = 3. Our collinearity- (g) largest eigenvalues of the matrix N and lgi are the parallelism-orthogonality constraint “effectively” fits corresponding unit eigenvectors. straight lines to collinear point sequences, but unlike the past collinearity-based work [1, 8, 12, 13, 17] we 6. Orthogonality Constraint never explicitly compute the fitted lines; all the constraints are expressed in terms of the incident ray Suppose we observe two groups Gg and Gg0 of par- vectors m. allel line sequences with mutually orthogonal direc- Now that we have derived the first and second tions lg and lg0 (Fig. 4). The orientation lg of the derivatives of all Ji with respect to all the parameters, sequences in the group Gg is the unit eigenvector of we can combine them into the following Levenberg- the matrix N (g) in (40) for the smallest eigenvalue. Marquardt (LM) procedure [16]: If the calibration is complete, we should have (lg, lg0 ) = 0, so we adjust the parameters by minimizing 1. Provide initial values, e.g., let the principal point ∑ (u0, v0) be at the frame center, f be a default 2 J3 = (lg, lg0 ) . (46) value and, let a1 = a2 = ··· = 0. Let J0 be the all orthogonal value of the function J for these initial values, {G G0 } pairs g, g and let C = 0.0001.

15 January 2012 Calibration of Ultra-Wide Fisheye Lens Cameras by Eigenvalue Minimization

2. Compute the incidence angle θκα of the αth point pα in the κth sequence Sκ by (3), and compute its incidence ray vector mκα by (6). Then, compute the derivatives ∂mκα/∂c by (8), (13), and (16) for c = u0, v0, f, a1, a2, ... .

3. Compute the ﬁrst derivatives Jc and the second 0 derivatives Jcc0 of the function J for c, c = u0, v0, f, a1, a2, ... .

4. Determine the increments ∆u0, ∆v0, ∆f, ∆a1, ... by solving the linear equation

··· 0 (1+C)Ju0u0 Ju0v0 Ju0f Ju0a1 1 ··· Jv0u0 (1+C)Jv0v0 Jv0f Jv0a1 B ··· C B Jfu0 Jfv0 (1+C)Jff Jfa1 C B ··· C B Ja1u0 Ja1v0 Ja1f (1 + C)Ja1a1 C B C @ . . . . . A ......

0 ∆u0 1 0 Ju0 1 ∆v0 Jv0 B C B C B ∆f C − B Jf C B C = B C. (51) B ∆a1 C B Ja1 C B C B C @ . A @ . A . .

5. Tentatively update the parameter values in the form ˜ u˜0 = u0 + ∆u0, v˜ = v0 + ∆v0, f = f + ∆f, Figure 5: Stripe patterns in four directions are displayed on a large video screen.

a˜1 = a1 + ∆a1, a˜2 = a2 + ∆a2, ... (52) and evaluate the resulting value J˜ of the function J.

6. If J˜ < J0, proceed. Else, let C ← 10C and go back to Step 4. ˜ 7. Let u0 ← u˜0, v0 ← v˜0, f ← f, a1 ← a˜1, a2 ← a˜2, ... . If |∆u0| < ²0, |∆v0| < ²0, |∆f| < ²f , |∆a1| < ²1, |∆a2| < ²2, ..., return J, u0, v0, f, (a) (b) a1, a2, ..., and stop. Else, let J0 ← J and C ← C/10, and go back to Step 2. Figure 6: (a) Fisheye lens image of a stripe pattern. (b) Detected edges. 8. Experiments to create connected edge segments. On each segment The four stripe patterns shown in Fig. 5 (above) was imposed the collinearity constraint; on the seg- were displayed on a large video screen (Fig. 5 below). ments in one frame were imposed the parallelism con- We took images by placing the camera in various straint; on the segments in one frame and the frame positions so that the stripe pattern appears in var- after the next with the same camera position were ious parts of the view (recall that the view cannot be imposed the orthogonality constraint. In all, we ob- covered by a single planar pattern image). The four tained 220 segments, consisting of 20 groups of par- patterns were cyclically displayed with blank frames allel segments and 10 orthogonal pairs, to which the in-between, and the camera is ﬁxed in each position LM procedure was applied. for at least one cycle to capture the four patterns; The conventional approach using a reference grid Fig. 6(a) shows one shot. The image size is 640 × 480 board [3, 5, 7, 17, 18] would require precise localiza- pixels. From each image, we detected edges; Fig. 6(b) tion of grid points in the image, which is a rather shows the edges detected from the image in Fig. 6(a). diﬃcult task. Here, all we need to do is detect “con- We manually removed those edges outside the dis- tinuous edges.” We can use a video display instead of play area. We also removed too small clusters of edge a specially designed grid board, because our method points. Then, we ran an edge segmentation algorithm is non-metric: we need no metric information about

16 Kenichi KANATANI MEM.FAC.ENG.OKA.UNI. Vol. 46

the pattern or the camera positions. We do not even ily understandable if one considers perspective cam- know where each edge point corresponds to in the ref- eras. If no image distortion exists, the 3-D interpreta- erence pattern. tion of images using wrong camera parameters, such Let us call the number K of the terms on the left as the focal length, the principal point, the aspect hand side of (2) the correction degree, meaning that ratio, and the skew angle, is a projective transforma- the left-hand side of (2) is approximated by a (2K + tion of the real scene, called projective reconstruction 1)th degree polynomial. The results up to the fifth [2]. Projective transformations preserve collinearity correction degree are shown in Table 1. We set the but not parallelism or orthogonality. Imposing paral- frame center to be (0, 0) to specify the principal point lelism, we obtain affine reconstruction. Further im- (u0, v0). For the convergence thresholds in Step 7 of posing orthogonality, we obtain so called Euclidean −3 −5 the LM iterations, we let ²0 = ²f = 10 , ²1 = 10 , reconstruction. Thus, orthogonality is essential for −6 −7 −8 −9 ²2 = 10 , ²3 = 10 , ²4 = 10 , and ²5 = 10 . camera calibration; collinearity and parallelism alone Using various different initial values, we confirmed are insufficient. This fact has been overlooked in the that the LM always converges to the same solution past collinearity-based work [1, 8, 12, 13, 15, 17], after around 10 to 20 iterations. partly because image distortion is so dominant for Figure 7(a) plots the graph of (3) for different cor- fisheye lens cameras, partly because spurious solu- rection degrees. For the convenience of the subse- tions can be prevented by using auxiliary informa- quent applications, we numerically converted (3) to tion such as vanishing point estimation [3, 5, 11, 12] express the angle θ in terms of the distance r. As we or antipodal point extraction [14], and partly because see, the stereographic projection model in (1) holds usually a small number, typically three, of collection fairly well even without any correction terms (degree terms are retained [1, 8, 12, 13, 15, 17], providing 0). The result is almost unchanged for the degrees insufficient degrees of freedom to fall into spurious 3, 4, and 5, i.e., including powers up to r7, r9, and solution. r11. Thus, there is no need to increase the correction The top-left of Fig. 9 is an image of a street scene terms any further. taken from a moving vehicle with a fisheye lens cam- For comparison, Fig. 7(b) shows the same result era mounted below the bumper at the car front. The top-right of Fig. 9 is the rectified perspective image. using collinearity alone, i.e., J = J1/γ1 instead of (50); Fig. 7(c) shows the result using collinearity and The second and third rows show the rectified perspec- parallelism, i.e., J = J /γ + J /γ instead of (50). tive images to be observed if the camera is rotated by 1 1 2 2 90◦ left, right, up and down, confirming that we are In both cases, the graph approaches, as the degree ◦ increases, some r-θ relationship quite different from really seeing more than 180 angles of view. Using a the stereographic projection. In order to see what fisheye lens camera like this, we can detect vehicles this means, we did a rectification experiment. Fig- approaching from left and/or right or create an image ure 8(a) shows a fisheye lens image viewing a square as if viewing the road from above the car. grid pattern in approximately 30 degree direction, and Fig. 8(b) is the rectified perspective image, using 9. Concluding Remarks the parameters of correction degree 5 in Table 1. The image is converted to a view as if observed by rotat- We have presented a new technique for calibrating ing the camera by 60 degrees to face the pattern (see ultra-wide fisheye lens cameras. Our method can be Appendix for this computation). The black area near contrasted to the conventional approach of using a the left boundary corresponds to 95 degrees or more grid board as follows: from the optical axis. Thus, we can obtain a correct perspective image to the very boundary of the view. 1. For ultra-wide fisheye lens cameras with more than 180◦ degrees of view, the image of any For comparison, Fig. 8(c) shows the result ob- large (even infinite) plane cannot cover the image tained by the same procedure using the spurious so- frame. Our method allows us take multiple par- lution. We can see that collinear points are certainly tial images of the reference by freely moving the rectified to be collinear and parallel lines to be (a camera so that every part of the frame is covered skewed view of) parallel lines. We have confirmed by some reference images. that this spurious solution is not the result of a local minimum. Let us call the parameter values obtained 2. The grid-based approach requires detection of by imposing collinearity, parallelism, and orthogonal- “grid points” in the image, but accurate pro- ity the “correct solution”. The cost functions J = cessing of a grid image is rather difficult. In J1/γ1 and J = J1/γ1 + J2/γ2 are certainly smaller our method, we only need to detect “continuous at the spurious solution than at the correct solution, edges” of a stripe pattern. meaning that the spurious solution is not attributed to the minimization algorithm but is inherent in the 3. The grid-based approach requires the correspon- formulation of the problem itself. dence of each detected grid point to the location The fact that spurious solution should exist is eas- on the reference. This is often difficult due to the

17 January 2012 Calibration of Ultra-Wide Fisheye Lens Cameras by Eigenvalue Minimization

Table 1: Computed parameters for each correction degree. degree 0 1 2 3 4 5

u0 −1.56761 −1.56968 −1.57134 −1.59349 −1.59385 −1.60081 v0 0.517809 0.488646 0.430145 0.423562 0.422805 0.426097 f 146.647 149.572 148.112 146.727 146.796 146.501 −2 a1/10 — 0.646891 −0.305581 −1.41589 −1.68775 −1.80506 −3 a2/10 — — 2.39013 7.57212 11.1001 9.34256 −4 a3/10 — — — 8.05471 22.2419 0.387907 −5 a4/10 — — — — 18.0654 61.6627 −6 a5/10 — — — — — 0.935994

(a) (b) (c)

Figure 7: The dependence of the distance r (pixels) from the focal point on the incidence angle θ (degree) obtained by (a) using the collinearity, parallelism, and orthogonality constraints; (b) using only the collinearity constraints; (c) using the collinearity and parallelism constraints.

periodicity of the grid. In our method, we need The basic principle of our calibration is the impo- not know where the detected edge points corre- sition of the constraint that collinear points be rec- spond to in the reference. tified to collinear, parallel lines to parallel, and orthogonal lines to be orthogonal. Exploiting the fact 4. In the grid-based approach, one needs to mea- that line fitting reduces to eigenvalue problems, we sure the camera position relative to the reference optimized the constraint by invoking the perturba- board by a mechanical means or by computing tion theorem, as suggested by Onodera and Kanatani the homography from image-reference matching [15]. Then, we derived a Levenberg-Marquardt pro- [3, 18]. Our method does not require any infor- cedure for eigenvalue minimization. By experiments, mation about the camera position. we have found that a spurious solution exists if the collinearity constraint alone is used or even com- 5. In the grid-based approach, one needs to create bined with the parallelism constraint. This fact has a reference pattern. This is not a trivial task. If not been noticed in existing collinearity-based work a pattern is printed out on a sheet of paper and [1, 8, 12, 13, 15, 17]. However, we have shown pasted to a planar surface, wrinkles and creases that incorporating the orthogonality constraint al- may arise and the glue may cause uneven defor- lows an accurate calibration without using any aux- mations of the paper. In our method, no metric iliary information such as vanishing point estimation information is required about the pattern, so we [3, 5, 11, 12]. We have also shown a real image exam- can display it on any video screen. ple using a vehicle-mounted fisheye lens camera. It is expected that our procedure is going to be a standard 6. In the grid-based approach, one can usually ob- tool for fisheye lens camera calibration. tain only one image of the reference pattern from Define a world XYZ coordinate system with the one camera position. In our method, we can fix origin O at the lens center and the Z-axis along the the camera anywhere and freely change the ref- optical axis (Fig. 1). As far as the camera imaging erence pattern on the video screen. is concerned, the outside scene can be regarded as if painted inside a sphere of a specified radius R sur- 7. The difficulty of processing a grid image with rounding the lens. The angle θ of the incident ray crossings and branches is circumvented by gen- from point (X,Y,Z) on the sphere is erating a “virtual grid” from separate stripe im- √ √ ages of different orientations; each image has no X2 + Y 2 R2 − Z2 θ = tan−1 = tan−1 . (53) crossings or branches. Z Z

18 Kenichi KANATANI MEM.FAC.ENG.OKA.UNI. Vol. 46

(a) (b) (c)

Figure 8: (a) Fisheye lens image viewing a square grid pattern in approximately 30 degree direction. (b) Rectiﬁed perspective image to be observed if the camera is rotated by 60 degrees to face the pattern. (c) Similarly rectiﬁed image using a spurious solution.

(X, Y, Z) ρ (X, Y, Z)

θ R

f f (x, y) Original ﬁsheye lens image Rectiﬁed front view (x, y) (0,0) r (u 0 ,v 0 )

Figure 10: To the pixel (¯x, y¯) of the rectiﬁed perspective image of focal length f¯ should be an image of the 3-D point (X,Y,Z), which is actually projected to (x, y) on the ﬁsheye lens image of focal length f.

point (X,Y,Z) given by Rectified left 90◦ view Rectified right 90◦ view X Y x¯ = f¯ , y¯ = f¯ . (55) Z Z From X2 Y 2 f¯2R2 x¯2 +y ¯2 + f¯2 = f¯2 + f¯2 + f¯2 = , (56) Z2 Z2 Z2 we see that ¯ √ fR Rectified down view Rectified up view Z = , (57) x¯2 +y ¯2 + f¯2 and hence Figure 9: Fisheye lens image of an outdoor scene taken     from a moving vehicle, rectified front images, and rectified X x¯ R images after virtually rotating the camera to left, right,  Y  = √  y¯  . (58) up, and down. 2 2 ¯2 Z x¯ +y ¯ + f f¯

Let r(θ) be the expression of r in terms of θ described Thus, the rectification can be done as follows in Fig. 7. The point (X,Y,Z) is projected to an image (Fig. 10): point (x, y) such that 1. For each pixel (¯x, y¯), compute the 3-D coordi- ( ) ( ) ( ) nates (X,Y,Z) in (58), where the radius R is x u0 r(θ) X = + √ . (54) arbitrary (we may let R = 1). y v0 R2 − Z2 Y 2. Compute the corresponding pixel position (x, y) Suppose we want to obtain a rectified perspective im- by (54) and copy its pixel value to (¯x, y¯). If (x, y) age with focal length f¯. Then, the pixel (¯x, y¯) of the are not integers, interpolate its value from sur- rectified image should be the projection of the 3-D rounding pixels.

19 January 2012 Calibration of Ultra-Wide Fisheye Lens Cameras by Eigenvalue Minimization

We can also obtain the perspective image to be [8] S. Kase, H. Mitsumoto, Y. Aragaki, N. Shi- observed if the camera is rotated by R. Since rotation momura and K. Umeda, “A method to con- of the camera by R is equivalent to the rotation of the struct overhead view images using multiple fish- − scene sphere by R 1, the 3-D point (X,Y,Z) given eye cameras” (in Japanese), , J. JSPE, vol. 75, by (58) on the rotated sphere corresponds to the 3-D no. 2, pp. 251–255, Feb. 2009. point     [9] H. Komagata, I. Ishii, A. Takahashi, D. Wak- X x¯ abayashi and H. Imai, “A geometric calibration R  Y  = √ R  y¯  . (59) method of internal camera parameters for fish- 2 2 ¯2 Z x¯ +y ¯ + f f¯ eye lenses” (in Japanese), IEICE Trans. Infor- mation and Systems, vol. J89-D, no. 1 (2006-1), on the original sphere. Its fisheye lens image should 64–73, Jan. 2006. be held at the pixel (x, y) given by (54). Thus, the mapping procedure goes as follows: [10] Y.-C. Liu, K.-Y. Lin, and Y.-S. Chen, “Bird’s eye view vision system for vehicle surrounding mon- 1. For each pixel (¯x, y¯), compute the 3-D coordi- itoring,” Proc. 2nd Int’l Workshop, Robvis2008 , nates (X,Y,Z) in (59). pp. 207–218, Feb. 2008. 2. Compute the corresponding pixel position (x, y) [11] M. Nakano, S. Li and N. Chiba, “Calibration of by (54) and copy its pixel value to (¯x, y¯). fish-eye camera for acquisition of spherical image” (in Japanese), IEICE Trans. Information Acknowledgments. The author thank Ryota Moriyasu and Systems, vol. J89-D-II, no. 9, 1847–1856, of ERD Corporation for helping the experiments. This work was supported in part by the Ministry of Education, Sept. 2005. Culture, Sports, Science, and Technology, Japan, under a [12] M. Nakano, S. Li and N. Chiba, “Calibrating Grant in Aid for Scientific Research (C 21500172). fisheye camera by stripe pattern based upon spherical model” (in Japanese), IEICE Trans. References Information and Systems, vol. J89-D, no. 1, 73– [1] F. Devernay and O. Faugeras, “Straight lines 82, Jan. 2007. have to be straight: Automatic calibration and [13] R. Okutsu, K. Terabayashi, Y. Aragaki, N. Shi- removal of distortion from scenes of structured momura, and K. Umeda, “Generation of over- environments,” Machine Vision Applications, head view images by estimating intrinsic and ex- vol. 13, no. 1, pp. 14–24, Aug. 2001. trinsic camera parameters of multiple fish-eye cameras,” Proc. IAPR Conf. Machine Vision [2] R. Hartley and O. Zisserman, Multiple View Ge- Applications, pp. 447–450, May 2009. ometry in Computer Vision, 2nd ed., Cambridge Univ. Press, 2004. [14] R. Okutsu, K. Terabayashi and K. Umeda, “Cal- ibration of intrinsic parameters of a fish-eye cam- [3] R. Hartley and S. B. Kang, “Parameter-free ra- era using a sphere” (in Japanese), IEICE Trans. dial distortion correction with center of distor- Information and Systems, vol. J89-D, no. 12 tion estimation,” IEEE Trans. Pattern Analysis (2010-12), 2645–2653, Dec. 2010 and Machine Intelligence, vol. 28, no. 8, 1309– 1321, Aug. 2007. [15] Y. Onodera and K. Kanatani, “Geometric correction of images without camera registration” [4] J. Heikkilä,“Geometric camera calibration us- (in Japanese), IEICE Trans. Information and ing circular control points,” IEEE Trans. Pat- Systems, vol. J75-D-II, no. 5, 1009–1013, May tern Analysis and Machine Intelligence, vol. 22, 1992. no. 10, Oct. 2000. [16] W. H. Press, S. A. Teukolsky, W. T. Vetterling, [5] C. Hughes, P. Denny, M. Glavin and E. Jones, and B. P. Flannery, Numerical Recipes in C: The “Equidistant fish-eye calibration and rectifi- Art of Scientific Computing, second ed., Cam- cation by vanishing point extraction,” IEEE bridge Univ. Press, 1992. Trans. Pattern Analysis and Machine Intelli- gence, vol. 32, no. 12, 2289–2296, Dec. 2010. [17] F. Swaminathan and S. K. Nayar, “Nonmetric calibration of wide-angle lenses and polycam- [6] K. Kanatani, Statistical Optimization for Geo- eras,” IEEE Trans. Pattern Analysis and Ma- metric Computation: Theory and Practice, El- chine Intelligence, vol. 22, no. 10, 1172–1178, sevier, 1996; reprinted Dover, 2005. Oct. 2000. [7] J. Kannala and S. S. Brandt, “A general cam- [18] Z. Zhang, “Flexible new technique for camera era model and calibration method for conven- calibration,” IEEE Trans. Pattern Analysis and tional, wide angle, and fisheye-lenses,” IEEE Machine Intelligence, vol. 22, no. 11, 1330–1334, Trans. Pattern Analysis and Machine Intelli- Nov. 2000. gence., vol. 28, no. 8, 1335–1340, Aug. 2006.