Homography estimation on the Special Linear Group based on direct point correspondence Tarek Hamel, Robert Mahony, Jochen Trumpf, Pascal Morin and Minh-Duc Hua Abstract— This paper considers the question of obtaining instantaneous estimates, the observer still requires individual a high quality estimate of a time-varying sequence of image image homographies to be computed for each image in the homographies using point correspondences from an image se- sequence to compute the observer innovation. quence without requiring explicit computation of the individual homographies between any two given images. The approach In this paper, we consider the question of deriving an uses the representation of a homography as an element of the observer for a sequence of image homographies that directly Special Linear group and defines a nonlinear observer directly takes point feature correspondences as input. The proposed on this structure. We assume, either that the group velocity approach is only valid in the case where a sequence of images of the homography sequence is known, or more realistically, used as data is associated with a continuous variation of that the homographies are generated by rigid-body motion of a camera viewing a planar surface, and that the angular velocity the reference image. The most common case encountered is of the camera is known. where the images are derived from a moving camera viewing a planar scene. The nonlinear proposed observer is posed I. INTRODUCTION on the Special Linear group SL(3) that is in one-to-one A homography is an invertible mapping relating two correspondence with the group of homographies [2] and uses images of the same planar scene. Homographies play a velocity measurements to propagate the homography estimate key role in many computer vision and robotics problems, and fuse this with new data as it becomes available [13], especially those that involve manmade environments typically [14]. A key advance on prior work by the authors is the constructed of planar surfaces, and those where the camera is formulation of a point feature innovation for the observer sufficiently far from the scene viewed that the relief of surface that incorporates point correspondences directly in the ob- features is negligible, such as the situation encountered in server without requiring reconstruction of individual image vision sequences of the ground taken from a flying vehicle. homographies. The proposed approach has a number of Computing homographies from point correspondences has advantages. Firstly, it avoids the computation associated with been extensively studied in the last fifteen years and different the full homography construction. This saves considerable techniques have been proposed in the literature that provide computational resources and makes the proposed algorithm an estimate of the homography matrix [5]. The quality of suitable for embedded systems with simple point tracking the homography estimate depends strongly on the algorithm software. Secondly, the algorithm is well posed even when used and the size of the set of points considered. For a well there is insufficient data for a full reconstruction of a homog- textured scene, state of the art methods provide high quality raphy. For example, if the number of corresponding points homography estimates but require significant computational between two images drops below four it is impossible to effort (see [17] and references therein). For a scene with algebraically reconstruct an image homography and the exist- poor texture, the existing homography estimation algorithms ing algorithms fail. In such situations, the proposed observer perform poorly. In a recent paper by the authors [13], [14] a will continue to operate, incorporating what information is nonlinear observer for homography estimation was proposed available and relying on propagation of prior estimates where based on the group structure of the set of homographies, the necessary. Finally, even if a homography can be reconstructed Special Linear group SL(3) [2]. This observer uses velocity from a small set of feature correspondences, the estimate information to interpolate across a sequence of images and is often unreliable and the associated error is difficult to to improve the overall homography estimate between any characterize. The proposed algorithm integrates information two given images. Although this earlier approach addresses from a sequence of images, and noise in the individual the problem partly by using temporal information to improve feature correspondences is filtered through the natural low- pass response of the observer, resulting in a highly robust T. Hamel and M.D. Hua are with I3S UNSA-CNRS, Nice-Sophia An- tipolis, France. e-mails: thamel(hua)@i3s.unice.fr. estimate. As a result, the authors believe that the proposed R. Mahony and J. Trumpf are with the School of Engineer- observer is ideally suited for poorly textured scenes and real- ing, Australian National University, ACT, 0200, Australia. e-mails: time implementation. We initially consider the case where Robert.Mahony(Jochen.Trumpf)@anu.edu.au. P. Morin is with ISIR UPMC, Paris, France. email: the group velocity is known, a situation that is rarely true in [email protected]. practice but provides considerable insight. The main result of the paper considers the case of a moving camera where the the equation written in a simple form: angular velocity of the camera is measured. This is a practical ∼ ˚ ∼ scenario where a camera is equipped with gyrometers. The ˚p = P ; p = P: (3) primary focus of the paper is on the presentation of the B. Homography observers and analysis of their stability properties, however, we do provide simulations to indicate the performance of the Assumption 2.1: Assume a calibrated camera and that proposed scheme. there is a planar surface π containing a set of n target points The paper is organized into five sections including the (n ≥ 4) so that introduction and the conclusion sections. Section II presents n o π = P˚ 2 R3 : ˚ηT P˚ − d˚= 0 ; a brief recap of the Lie group structure of the set of homo- graphies and relates it to rigid-body motion of the camera. where ˚η is the unit normal to the plane expressed in fA˚g and Section III provides an initial lemma in the case where it is ˚ ˚ assumed the group velocity is known and then considers the d is the distance of the plane to the origin of fAg. From the rigid-body relationships (1), one has P = RT P˚ − case of a moving camera where the angular velocity of the T T camera is known. Simulation results are provided in Section R ξ. Define ζ = −R ξ. Since all target points lie in a single IV to verify performance of the proposed algorithms. planar surface one has T T ζ˚η II. PRELIMINARY MATERIAL Pi = R P˚i + P˚i; i = f1; : : : ; ng; (4) d˚ A. Projection and thus, using (3), the projected points obey Visual data is obtained via a projection of observed T images onto the camera image surface. The projection is ∼ T ζ˚η pi = R + ˚pi; i = f1; : : : ; ng: (5) parameterised by two sets of parameters: intrinsic (“internal” d˚ parameters of the camera such as the focal length, the pixel T −1 The projective mapping H : A! A˚, H :=∼ RT + ζ˚η aspect ratio, etc.) and extrinsic (the pose, i.e. the position and d˚ orientation of the camera). Let A˚ (resp. A) denote projective is termed a homography and it relates the images of points coordinates for the image plane of a camera A˚ (resp. A), on the plane π when viewed from two poses defined by the and let fA˚g (resp. fAg) denote its (right-hand) frame of coordinate systems A and A˚. It is straightforward to verify reference. Let ξ 2 R3 denote the position of the frame fAg that the homography H can be written as follows: with respect to fA˚g expressed in fA˚g. The orientation of the ξη> frame fAg with respect to fA˚g, is given by a rotation matrix, H =∼ R + (6) element of the Special Orthogonal group, R 2 SO(3) : d ˚ fAg ! fAg. The pose of the camera determines a rigid where η is the normal to the observed planar surface ex- ˚ body transformation from fAg to fAg (and visa versa). One pressed in the frame fAg and d is the orthogonal distance of has the plane to the origin of fAg. One can verify that [2]: ˚ P = RP + ξ (1) η = RT ˚η (7) T T as a relation between the coordinates of the same point in d = d˚− ˚η ξ = d˚+ η ζ: (8) the reference frame (P˚ 2 fA˚g) and in the current frame The homography matrix contains the pose information (R; ξ) (P 2 fAg). The camera internal parameters, in the commonly of the camera from the frame fAg (termed current frame) to used approximation, define a 3 × 3 matrix K so that we can the frame fA˚g (termed reference frame). However, since the write1: relationship between the image points and the homography ˚p ∼ KP˚ ; p ∼ KP; (2) = = is a projective relationship, it is only possible to determine where p 2 A is the image of a point when the camera is H up to a scale factor (using the image points relationships aligned with frame fAg, and can be written as (x; y; w)T alone). using the homogeneous coordinate representation for that 2D C. Homography versus element of the Special Linear Goup image point. Likewise, ˚p 2 A˚ is the image of the same point SL(3) viewed when the camera is aligned with frame fA˚g. If the camera is calibrated (the intrinsic parameters are Recall that the Special Linear Lie-group SL(3) is defined known), then all quantities can be appropriately scaled and as the set of all real valued 3 × 3 matrices with unit determinant 1Most statements in projective geometry involve equality up to a multi- 3 plicative constant denoted by =∼.