1 An Efficient Solution to Non-Minimal Case Essential Matrix Estimation Ji Zhao Abstract—Finding relative pose between two calibrated images is a fundamental task in computer vision. Given five point correspondences, the classical five-point methods can be used to calculate the essential matrix efficiently. For the case of N (N > 5) inlier point correspondences, which is called N-point problem, existing methods are either inefficient or prone to local minima. In this paper, we propose a certifiably globally optimal and efficient solver for the N-point problem. First we formulate the problem as a quadratically constrained quadratic program (QCQP). Then a certifiably globally optimal solution to this problem is obtained by semidefinite relaxation. This allows us to obtain certifiably globally optimal solutions to the original non-convex QCQPs in polynomial time. The theoretical guarantees of the semidefinite relaxation are also provided, including tightness and local stability. To deal with outliers, we propose a robust N-point method using M-estimators. Though global optimality cannot be guaranteed for the overall robust framework, the proposed robust N-point method can achieve good performance when the outlier ratio is not high. Extensive experiments on synthetic and real-world datasets demonstrated that our N-point method is 2 ∼ 3 orders of magnitude faster than state-of-the-art methods. Moreover, our robust N-point method outperforms state-of-the-art methods in terms of robustness and accuracy. Index Terms—Relative pose estimation, essential manifold, non-minimal solver, robust estimation, quadratically constrained quadratic program, semidefinite programming, convex optimization F 1 INTRODUCTION INDNG relative pose between two images using 2D- provide high robustness. F 2D point correspondences is a cornerstone in geometric Once the maximal consensus set has been found, the vision. It makes a basic building block in many structure- standard RANSAC optionally re-estimates a model by using from-motion (SfM), visual odometry, and simultaneous lo- all inliers to reduce the influence of noise [1], [6]. Thus a non- calization and mapping (SLAM) systems [1]. Relative pose minimal solver is needed in this procedure. This RANSAC estimation is a difficult problem since it is by nature non- framework is called the gold standard algorithm [1]. Fig- convex and known to be plagued by local minima and ure 1 illustrates this framework by taking line fitting as an ambiguous solutions. Relative pose for uncalibrated and example. In addition to the usage as post-processing, the calibrated cameras are usually characterized by fundamen- non-minimal solvers can also be integrated more tightly into tal matrix and essential matrix, respectively [1]. A matrix RANSAC variants. For example, LO-RANSAC [7] attempts is a fundamental matrix if and only if it has two non- to enlarge the consensus set of an initial RANSAC estimate zero singular values. An essential matrix has an additional by generating hypotheses from larger-than-minimal subsets of property that the two nonzero singular values are equal. the consensus set. The rationale is that hypotheses fitted on Due to these strict constraints, essential matrix estimation is a larger number of inliers typically lead to better estimates arguably thought to be more challenging than fundamental with higher support. matrix estimation. This paper focuses on optimal essential In this paper, the non-minimal case relative pose esti- matrix estimation. mation is called N-point problem, and its solver is called N- arXiv:1903.09067v3 [cs.CV] 7 Oct 2020 Due to the scale ambiguity of translation, relative pose point method. As it has been investigated in [8], [9], [10], N- for calibrated cameras has 5 degrees-of-freedom (DoFs), point methods usually lead to more accurate results than including 3 for rotation and 2 for translation. Except for de- five-point methods. Thus the N-point method is useful generate configurations, 5 point correspondences are hence for scenarios that require accurate pose estimation, such enough to determine the relative pose. Given five point as visual odometry and image-based visual servoing. The correspondences, the five-point methods using essential well-known direct linear transformation (DLT) technique [1] matrix [2], [3] or rotation matrix parametrization [4] can with proper normalization [11] can be used to estimate the be used to calculate the relative pose efficiently. The afore- essential matrix using 8 or more point correspondences. mentioned solvers are the so-called minimal solvers. When However, DLT ignores the inherent nonlinear constraints of point correspondences contain outliers, minimal solvers are the essential matrix. To deal with this problem, an essential usually integrated into a hypothesize-and-test framework, matrix is recovered after an approximated essential matrix such as RANSAC [5], to find the solution corresponding is obtained from the DLT solution [1]. An eigenvalue-based to the maximal consensus set. Hence this framework can formulation and its variant were proposed to solve N- point problem [9], [10]. However, all the aforementioned methods fail to guarantee global optimality and efficiency • J. Zhao is with TuSimple, Beijing, China. simultaneously. E-mail: [email protected] Since the N-point problem is challenging, the progress 2 • Robust N-point method. We propose a robust essen- tial matrix estimation method by integrating N-point Input observations with outliers method into M-estimators. Considering that the robust components of the overall framework are not certifiably and provably optimal, we can only demonstrated em- pirical performance assurances. … • Theoretical aspects. We provide theoretical proofs of the semidefinite relaxation (SDR) in the proposed N- point method, including SDR tightness and local stabil- consensus set maximization (minimal solver is used) ity with small observation noise. error minimization (non-minimal solver is used) The paper is organized as follows. Section 2 introduces active point to determine line the related work. In Section 3, we propose a simple param- best model maximizing consensus set eterization of essential manifold and provide novel formu- best model minimizing fitting error lations of N-point methods. Based on these formulations, Section 4 derives a convex optimization approach by SDR. Fig. 1. The RANSAC framework contains the collaboration of a minimal Section 5 proves tightness and local stability of SDR. A solver and a non-minimal solver. This framework is known as the gold robust N-point method based on robust loss function is standard algorithm [1]. This paper focuses on relative pose estimation. Here we take line fitting for an example due to its convenient visualiza- proposed in Section 6. Section 7 presents the performance tion. of our method in comparison to other approaches, followed by a concluding discussion in Section 8. of its solvers is far behind the absolute pose estimation (perspective-n-point (PnP)) and point cloud registration. 2 RELATED WORK For example, the EPnP algorithm [12] in PnP area is very efficient and has linear complexity with the number Estimating an essential matrix for a calibrated camera from of observations. It has been successfully used in RANSAC point correspondences is an active research area in computer framework given an arbitrary number of 2D-3D correspon- vision. Finding the optimal essential matrix by L1 norm dences. Though relative pose estimation is more difficult cost and branch-and-bound (BnB) was proposed in [18]. It than absolute pose estimation, it is desirable to find a prac- achieves global optima but is inefficient. There are several tical solver to N-point problem, whose efficiency and global works for N-point fundamental/essential matrix estimation optimality are both satisfactory. This is one motivation of using local optimization [19], [20] or manifold optimiza- this paper. tion [21], [22], [23]. A method for minimizing an algebraic Another motivation of this paper is developing an M- error was investigated in [8]. Its global optimality was estimator based method for relative pose estimation. In obtained by a square matrix representation of homogeneous practice, it arises frequently that the data have been con- forms and relaxation. An eigenvalue-based formulation was taminated by large noise and outliers. Existing robust es- proposed to estimate rotation matrix [9], in which the prob- timation methods are mainly classified into two main cat- lem is optimized by local gradient descent or BnB search. It egories, i.e., inlier set maximization [13] and M-estimator was later improved by a certifiably globally optimal solution based method [14]. Inlier set maximization can be achieved by relaxation and semidefinite programming (SDP) [10]. by randomized sampling methods [5], [7] or deterministic However, none of the aforementioned methods can find optimization methods [15], [16]. For M-estimator based the globally optimal solution efficiently. Though BnB search methods, the associated optimization problems are non- methods [9], [18] can obtain global optima in theory, they convex and difficult to solve. In this paper, the proposed have the exponential time complexity in the worst case. robust N-point method uses M-estimators. To solve the In [8], [10], a theoretical guarantee of the convexification associated optimization problem effectively, the line process procedures is not provided. The most related paper to is adopted [17]. The line process uses a continuous iterative this work