Robust Multiple Homography Estimation: An Ill-Solved Problem Zygmunt L. Szpak, Wojciech Chojnacki, Anton van den Hengel School of Computer Science, The University of Adelaide, SA 5005, Australia fzygmunt.szpak,wojciech.chojnacki,[email protected] Abstract namely the failure to recognise that a set of homographies that each of these schemes produces is actually not a gen- The estimation of multiple homographies between two uine set of homographies between two views of the same piecewise planar views of a rigid scene is often assumed scene. A collection of homography matrices forms a valid to be a solved problem. We show that contrary to popular set only if the matrices satisfy consistency constraints im- opinion various crucial aspects of the task have not been plied by the rigidity of the motion and the scene. If the adequately emphasised. We are motivated by a growing constraints are not deliberately enforced, they are not sat- body of literature in robust multi-structure estimation that isfied in typical scenarios. Hence, one of the fundamen- purports to solve the multi-homography estimation prob- tal problems in estimating multiple homography matrices lem but in fact does not. We demonstrate that the estima- is to find a way to enforce the consistency constraints—a tion of multiple homographies is an ill-solved problem by task reminiscent of that of enforcing the rank-two constraint deriving new constraints that a set of mutually compatible in the case of the fundamental matrix estimation [17, Sect. homographies must satisfy, and by showing that homogra- 11.1.1]. phies estimated with prevailing methods fail to satisfy the Explicit formulae for all constraints that must be satisfied requisite constraints on real-world data. We also explain have eluded the vision community. It was only as recently why incompatible homographies imply inconsistent epipo- as 2011 that a decisive answer pertaining to even just the lar geometries. The arguments and experiments presented number of constraints was given [10, 11]. Over the years in this paper signal the need for a new generation of robust various researchers have managed to identify and enforce a multi-structure estimation methods that have the capacity reduced set of constraints. For example, Shashua and Avi- to enforce constraints on projective entities such as homog- dan [31] found that homography matrices induced by four raphy matrices. or more planes in a 3D scene appearing in two views span a four-dimensional linear subspace. Chen and Suter [5] de- rived a set of strengthened constraints for the case of three 1. Introduction or more homographies in two views. Zelnik-Manor and Images in two views of world points lying on a planar Irani [36] have shown that another rank-four constraint ap- surface are related by a homography matrix. Since pla- plies to a set of so-called relative homographies generated nar surfaces are ubiquitous in urban environments, estimat- by two planes in four or more views. These latter authors ing multiple homography matrices from image measure- also derived constraints for larger sets of homographies and ments between two views is an important step in many ap- views. plications such as augmenting reality, stitching and warp- Once isolated, the constraints are typically put to use ing images, calibrating cameras, finding a metric recon- in a procedure whereby first individual homography ma- struction, and detecting non-rigid motion. Because of the trices are estimated from image data, and then the result- diverse utility of homography matrices, the task of es- ing estimates are upgraded to matrices satisfying the con- timating multiple homographies is often used to demon- straints. Following this pattern, Shashua and Avidan as strate the merits of robust multi-structure estimation meth- well as Zelnik-Manor and Irani used low-rank approxima- ods [6,7,12,13,19,29,30,34,35]. In fact, some robust multi- tion under the Frobenius norm to enforce the rank-four con- structure estimation methods, such as multiRANSAC [39], straint. Chen and Suter enforced their set of constraints also were specifically designed to address the multi-homography via low-rank approximation, but then employed the Maha- estimation problem. However, an inadvertent oversight has lanobis norm with covariances of the input homographies. crept into multi-structure estimation methods, one that per- All of these estimation procedures produce matrices that sists in all state-of-the-art methods that we are familiar with, satisfy only incomplete constraints so their true consistency cannot be guaranteed. first and second views described by the 3 × 3 matrix A few researchers managed to enforce some consis- > tency without appealing to rank constraints. For example, Hi = wiA + bvi ; (1) Lopez-Nicol´ as´ et al. [22] used the geometry underpinning an epipole constraint, whereas Kirchhof [20] required a fun- where damental matrix in order to estimate homographies consis- A = K R R−1K−1; w = n>t − d ; tently. 2 2 1 1 i i 1 i −> (2) Without knowledge of explicit formulae for all of the b = K2R2(t1 − t2); vi = K1 R1ni: constraints, it is still possible to implicitly enforce full con- sistency by recoursing to a parametrisation of the set of In the case of calibrated cameras when one may assume that all intervening homography matrices. Following this path, K1 = K2 = I3, t1 = 0, R1 = I3, R2 = R, system (2) Chojnacki et al. [8,9] employed an appropriate parametri- reduces to A = R; wi = −di; sation and a distinct cost function to develop an upgrade (3) procedure based on unconstrained optimisation. Szpak et b = t; vi = ni; al. [32] used the same parametrisation and the Sampson with t = −Rt2, and equality (1) becomes the familiar di- distance to develop an alternative technique with a sound rect nRt representation statistical basis. Details of this parametrisation will be pre- > sented in Section2. Hi = −diR + tni While a literature review shows that considerable progress has been made on the problem of multi- (cf. [1], [25, Sect. 5.3.1]). We stress that all of our subse- homography estimation in recent years, it is also apparent quent analysis concerns the general uncalibrated case, with that many researchers are not aware that bona fide sets of A, b, wi’s and vi’s to be interpreted according to (2) rather homography matrices need to satisfy constraints. The ne- than (3). science concerning constraints is particularly evident in the A natural object associated with the matrices Hi is discourse on robust multi-structure estimation. Our contri- the 3 × 3I concatenation matrix H = [H1;:::; HI ]: It bution, therefore, is two-fold: (1) we demonstrate, from proves convenient to consider also the 9 × I matrix H = a practical perspective, that failing to enforce consistency [h1;::: hI ]; where, with vec denoting column-wise vec- constraints on multiple homographies leads to inconsistent torisation [23], hi = vec(Hi) for each i = 1;:::;I. It estimates of the epipolar geometry between two views, and turns out that H = ST; where, with ⊗ denoting Kro- (2) we derive two new sets of explicit constraints that multi- necker product [23], S = [I3 ⊗ b; a] is a 9 × 4 matrix v1 ::: vI ple homographies between two views need to satisfy. Some and T = [ w1 ::: wI ] is a 4 × I matrix [32]. An immedi- benefits of having explicit constraints are presented in Sec- ate consequence of this factorisation is that H has rank at tion3. most four. Whenever I ≥ 5 the requirement that H should have rank no greater than four places a genuine constraint 2. Enforcing consistency implicitly on H, and hence also on H. This is the rank-four constraint of Shashua and Avidan mentioned earlier. Since the real As mentioned in the introduction, when estimating a set 9 × I matrices of rank at most 4 form a manifold of dimen- of homographies associated with multiple planes from im- sion 4(9 + I − 4) = 4I + 20 [16, Proposition 12.2], the age correspondences between two views, one must recog- rank-four constraint implies that the dimension of the set of nise that the homographies involved are interdependent. To all H’s is no greater than 4I + 20 for I ≥ 5. The ensuing get an idea of the relevant dependencies, consider two fixed inequality 4I + 20 < 9I for I ≥ 5 makes it clear that H uncalibrated cameras giving rise to two camera matrices resides in a proper subset of all 3 × 3I matrices for I ≥ 5. P1 = K1R1[I3; −t1] and P2 = K2R2[I3; −t2]. Here, the The dimensionality count for the H’s can be re- length-3 translation vector tk and the 3 × 3 rotation matrix fined and the subsequent conclusions sharpened. Letting Rk represent the Euclidean transformation between the k-th > > > > > η = [a ; b ; v1 ;:::; vI ; w1; : : : ; wI ] and Π(η) = (k = 1; 2) camera and the world coordinate system, Kk is a > [Π1(η);:::; ΠI (η)]; where Πi(η) = wiA+bvi for each 3 × 3 upper triangular calibration matrix encoding the inter- i = 1;:::;I, H can be represented as nal parameters of the k-th camera, and I3 denotes the 3 × 3 identity matrix. Suppose, moreover, that a set of I planes in H = Π(η): (4) a 3D scene have been selected. Given i = 1;:::;I, let the i-th plane from the collection have a unit outward normal In this formulation, η appears as the vector of latent vari- ni and be situated at a distance di from the origin of the ables that link all the constituent matrices together and pro- world coordinate system. Then, for each i = 1;:::;I, the vide a natural parametrisation of the set of all H’s. Since i-th plane gives rise to a planar homography between the η has a total of 4I + 12 entries, the totality of all matrices of the form Π(η) has dimension no greater than 4I + 12.