Complete Endomorphisms in Computer Vision Applied to Multiple View Geometry

Complete Endomorphisms in Computer Vision Applied to multiple view geometry

Finat, Javier · Delgado-del-Hoyo, Francisco

July 26, 2021

Abstract Correspondences between k-tuples of points 1 Introduction are key in multiple view geometry and motion analysis. Regular transformations are posed by homographies A classical issue regarding 3D reconstruction and mo- between two projective planes that serves as structural tion analysis concerns the preservation of the continuity models for images. Such transformations can not in- of the scene or the flow, although small changes in input clude degenerate situations. Fundamental or essential occurs. There are a lot of answers where different con- matrices expand homographies with structural infor- straints have been introduced from the early eighties. mation by using degenerate bilinear maps. The projec- The most common constraint is the structural bilinear tivization of the endomorphisms of a three-dimensional - or multilinear - tensor created with k-tuples of corre- vector space includes all of them. Hence, they are able sponding elements (points and/or lines) for the camera to explain a wider range of eventually degenerate trans- pose. All these constraints are very sensitive to noise. formations between arbitrary pairs of views. To include Classical approaches are based on minimal solutions ex- these degenerate situations, this paper introduces a com- tracted from noise measurements following a RANSAC1 pletion of bilinear maps between spaces given by an scheme. However, indeterminacies persist for some de- equivariant compactification of regular transformations. generate situations - created by low rank matrices - that This completion is extensible to the varieties of fun- can arise for mobile cameras. damental and essential matrices, where most methods Classic literature in computer vision [4,9] display based on regular transformations fail. The construction a low attention to degenerate cases in structural mod- of complete endomorphisms manages degenerate pro- els. These cases arise when independence conditions for jection maps using a simultaneous action on source and features are not fulfilled, including situations where the target spaces. In such way, this mathematical construc- camera turns around its optical axis or it is in fronto- tion provides a robust framework to relate correspond- parallel position w.r.t. a planar surface (a wall or the ing views in multiple view geometry. ground, e.g.). Then, the problem is ill-posed, and conventional solutions consists in performing a “small per- arXiv:2002.09003v1 [cs.CV] 20 Feb 2020 turbation” or reboot the process. Both strategies dis- Keywords Epipolar Geometry · Essential Matrix · play issues concerning the lack of control about the Fundamental Matrix · Degeneracies · Secant Varieties · perturbation to be made that generate undesirable dis- Adjoint Representation continuities. Thus, it is important to develop alternative strategies which can maintain some kind of “coher- MoBiVAP research group ence” by reusing the “recent history” of the trajectory. Universidad de Valladolid History is continuously modeled in terms of generically Campus Miguel Delibes regular conditions for tensors in previously sampled im- Paseo de Belén,11 ages with a discrete approach of a well-defined path in 47010, Valladolid, Spain Tel.: +34-983-184-398 the space of structural tensors. Unfortunately, degener- https://www.mobivap.es E-mail: franciscojavier.fi[email protected] 1 Random Sample Consensus 2 Finat, Javier, Delgado-del-Hoyo, Francisco acy conditions for typical features give indeterminacy containing their degenerate cases, which are usually for limits of structural tensors, which must be removed. excluded from the analysis. They are recovered by in- Our approach consists of considering Kinematic infor- troducing a “compactification” where degenerate cases mation of the matrix version of the gradient field for are managed in terms of successive envelopes by linear indeterminacy loci. subspaces W of V . All arguments can be extended to Less attention has been paid to preserve the “conti- higher dimensions and even to hypermatrices represent- nuity” of eventually singular trajectories in the space of ing more sophisticated tensors. However, for simplifica- bilinear maps linked to the automatic correspondence tion purposes, we constraint ourselves only to endomor- between pairs of views. In this case, singular maps are phisms extending planar or spatial homographies to the the responsible for indeterminacies in tensors and lie on singular case. singular strata of the space of bilinear maps[24]. In this The rest of the paper is organized as follows. Section work, we develop a more down-to-earth approach using 2 provides the mathematical background to understand some basic properties of the projectivization of spaces the rest of the paper and frames our approach in the of endomorphisms, including homographies H, funda- state of the art. Section 3 analyzes the simplest cases mental F and essential E matrices. All of them can be including regular transformations defined by homogra- described in terms of orbits by a group action on the phies for the planar case that relate 2D views using the space of endomorphisms End(V ), i.e. linear maps of a fundamental variety. Section 4 extends the approach vector space V in itself. Their simultaneous algebraic to rigid transformations in the third dimension, includ- treatment allows to extend the algebraic completion to ing metric aspects in terms of the essential variety in- the space of eventually degenerated central projection volving source and target spaces in P3. Section 5 stud- maps MC with center C. Intuitively, the key for the ies the structural connection between them using the control of degenerate cases is to select appropriate lim- simultaneous action on source and target spaces of a its of tangents in a “more complete” space. variable projection linked to the camera pose; left-right Therefore, the main goal of our work is to model A and contact K-equivalences are explained. Section 6 a continuous solution that also considers degenerated provides additional insight concerning the details and cases for simplest tensors (fundamental and essential practical considerations for implementing this approach 2 matrices, e.g.) such those appearing in structural mod- in VO systems. Finally, Section 7 concludes the paper els for 3D Reconstruction. In order to achieve this goal, with a summary of the main results and guidelines for we introduced an “equivariant compactification” of the further research. space of matrices w.r.t. group actions linked to kinematic properties visualized in the dual space. This double representation (positions and “speed”) stores the 2 Background “recent history” represented by a path in the tangent bundle τEnd(V ) to End(V ) for the trajectory of a mobile Local symmetries are ubiquitous in a lot of problems camera C(t). Our approach is based on a dual presen- in Physics and Engineering involving propagation phe- tation for the rank stratification of matrices. This dual nomena. Most approaches in applied areas consider only representation encodes tangential information at each regular regions, by ignoring any kind of degenerations point represented by the adjoint matrix, a 3 × 3-matrix linked to rank deficient matrices linked to linearization whose entries represent the gradient ∇(det(A)) of the of phenomena. To include them, we develop a “locally determinant of A. symmetric completion” of eventually singular transfor- In the simplest case, after fixing a basis BV of V , en- mations for involved tensors such those appearing in domorphisms of a 3D vector space V ' R3 are given by Reconstruction issues. Our approach is not quite origi- arbitrary 3×3-matrices; they are naturally stratified by nal; a similar idea can be found in [26], which introduces the rank giving three orbits with rank r ≤ 3. In partic- a locally symmetric structure in a differential frame- ular, from the differential viewpoint, sets of homogra- work concerning geodesics on the essential manifold. phies H and regular (i.e. rank 2) fundamental matrices Nevertheless, the initial geometric description as sym- F can be considered as two G-orbits of the Lie algebra metric space (union of orbits linked to the rank preser- g = End(V 3) of G = GL(3) = Aut(V 3) by the action of vation) can not be extended to include a differential the projectivization PGL(3) of the general linear group approach to degenerate cases. Due to the occurrence of corresponding to rank 3 and rank 2 matrices, respec- singularities, the support given as a quotient variety is tively. More generally, the description of End(V n+1) as not a smooth manifold but a singular algebraic variety a union of orbits by the action of GL(n + 1; R) gives a structure as an “orbifold”, i.e. a union of G-orbits 2 Visual Odometry Complete Endomorphisms in Computer Vision 3 where the methods for Riemannian manifolds no longer double conjugacy classes from the algebraic viewpoint. apply. The A-action is very useful for decoupled models (im- A larger description of a locally symmetric struc- plicit in KLT algorithms or SfM4, e.g.), and conse- ture may be performed by extending the ordinary al- quently very useful by computational reasons. Despite gebraic approach. Roughly speaking, it suffices to add the wide interest for the above approaches, the A-action limits of tangent subspaces along different “branches” is less plausible than the K-action, which incorporates at singularities and “extend the action” to obtain a the graph preservation (corresponding to quadratic con- more complete description including kinematic aspects. tact between a manifold M and its tangent space TpM In this way, one obtains a “local replication” compati- at each contact point p ∈ M) as the structural con- ble with the presence of singularities in the adherence of straint. “augmented” orbits by tangent spaces at regular points. In our case, contact equivalence is based on a cou- So, for the “subregular” case - codimension one orbit pling between images and scene representations. Al- - it suffices to construct pairs of eventually degenerate though contact equivalence is well known in Local Dif- transformations involving the original one and a “gen- ferential Topology, its use in Computer Vision is very eralized dual” transformation (given by the adjoint map scarce. It is implicitly embedded in some recognition ap- in the regular case) representing neighbor tangent di- proaches where one exchanges information about con- rections. So, first order differential approaches of even- trol points and envelopes. However, to our best knowl- tually degenerate maps allow to propagate - and conse- edge, it has not been applied to multiple view geometry quently, anticipate - partial representations of expected issues. We constraint ourselves to almost generic phe- views, even in presence of rank-deficient matrices. nomena given by low-corank c ≤ 2 singularities. In this This extended duality allows a simultaneous treat- way, a more stable “geometric control” of limit positions ment of incidence and tangency conditions (both are using envelopes of linear subspaces can be performed. projectively invariant), and to manage degenerate cases in terms of “complete objects” as limits in an enlarged space (including the original space and their duals) which 3 Completing planar homographies can be managed as a locally symmetric space in terms of extended transformations (original ones and their ex- This section extends conventional homographies to in- terior powers). Besides its differential description as a clude the degenerate cases by considering arbitrary - gradient in the space of matrices, a more geometric de- including eventually singular - endomorphisms (i.e. lin- scription can be developed in terms of pairs of loci and ear maps on a vector space) acting on configurations their envelopes. The extended transformations act on of points. In particular, regular transformations up to the source or ambient space (right action), and on its scale of a 3-dimensional vector space V 3 belong to the dual space which can be considered as a target space group of homographies which is an open subset of the representing envelopes by tangent subspaces. This idea projective space 8 = End(V 3). Its complementary is reminiscent of the contact action K which preserves P P is the set of singular endomorphisms up to scale, a the graph and it provides a natural extension of the cubic hypersurface defined by det(X) = 0 for X = right-left action A (see next paragraph). A discrete ver- (x ) and containing the fundamental subvariety sion of last action has been used by Kanade, Tomasi ij 0≤i,j≤2 F and the essential manifold E. and Lucas KLT3 along the early nineties in regard to Planar homographies represent regular transforma- Structure from Motion approaches to 3D Reconstruc- 2 3 tion. Both actions are commonly used for the infinites- tions between two projective planes P = PV of 2D imal classification of map-germs in differential classifi- views. Thus, any homography is an element of the pro- cation of map-germs. However, our approach is more jectivized linear group PGL(3, R), where GL(3, R) is focused towards a local description of the space of gen- the general linear group acting on the projective model eralized transformations and/or projection maps as a of each view. Given a reference for V , each element locally symmetric space. This structure has the addi- of GL(3, R) can be represented by a (3 × 3) regular tional advantage of allowing the extension of Rieman- matrix, i.e. with non-vanishing determinant. By con- nian properties given in terms of geodesics. struction, homographies (regular transformations up to The simplest simultaneous action on source and tar- scale) can not include degenerate transformations such get spaces is the Cartesian or direct product of ac- those appearing in fundamental or essential matrices. tions. It is denoted by the A-action where A := R × Then, these matrices can be considered as “degenerate” L is the right-left action. Its orbits are given by the endomorphisms (represented by defficient rank matri-

3 Kanade-Lucas-Tomasi 4 Structure from Motion 4 Finat, Javier, Delgado-del-Hoyo, Francisco ces up to scale) of an abstract real 3D space V with is a homography away from the variety D of degenerate PV = P2. endomorphisms5. For n = 2, the intersection L ∩ D of a general pro- Fundamental matrices F ∈ F are defined by degen- 8 0 0 jective line L ⊂ P with the cubic algebraic variety D erate bilinear forms xFx = 0 linking pairs (x, x ) of defined by det(D) = 0 gives generically three differ- corresponding points. The set of pairs of correspond- 2 ent degenerate endomorphisms. In particular, if L is ing points is called the join of two copies of P . This tangent to D at least two elements of L ∩ D can coa- join is defined by the image of the Segre embedding 2 2 8 lesce. An ordinary tangency condition is represented by s2,2 : P × P ,→ P giving a four-dimensional variety 0 0 8 2d+d , where 2d (resp. d ) represents a tangency (resp. of P that determines the 7D subvariety F of singular simple) contact point corresponding to the intersection endomorphisms up to scale. Addition of the singular of L with D. Linear pencils of matrices representing cases “completes” the homographies (regular transfor- endomorphisms are interpreted as secant lines in the mations), treating fundamental and essential matrices projective ambient space. as degenerate transformation between two projective In general, k-secant varieties Sec(k; X) to a vari- planes inside the set of a completion or homogeneous ety X ⊂ An are defined by the set of points lying in endomorphisms. the closure of k-dimensional subspaces Lk generated by To understand how transformations can be extended (k + 1)-tuples of affine independent points generating from a geometric to a kinematic framework, it is con- k linearly independent vectors. They can be formally k venient to introduce the differential approach for the constructed by using the k-th exterior power ∧ V of regular subset. In terms of algebraic transformations, the underlying vector space V that allows to manage one mus replace the Lie group G of regular transforma- k + 1-tuples of points for k ≥ 1. This statement can be adapted to the underlying vector space of the Lie tions by its Lie algebra g := TeG where e is the neutral element of F (the identity matrix for matrix groups); in algebra g := TeG with its natural stratification by the rank of any classical group G. The locally symmetric particular, TeAut(V ) = End(V ). As usual in Lie theory, A, B, C ... denote the elements of the group G, and structure is the key for extending the concept in the presence of singularities. Although this construction is X, Y, Z,... the elements of its Lie algebra g := TeG. In general for End(V ), it can be constrained to manage particular, any endomorphism X ∈ g`(3) := TI GL(3) can be described by a matrix representing a point x ∈ eventually degenerate tensors. Actually, this approach 8 allows to connect old based-perspective methods using P up to scale. homographies with tensor-based methods. The exponential map exp : g → G is a local diffeomorphism (with the logarithm as inverse) that can be applied to degenerate matrices for n = 3. In general, 3.1 Fundamental variety the set of homographies is an open set of PN where This subsection highlights the geometry of subvarieties N = (n + 1)2 − 1, whose complementary is given by the parameterizing rank deficient endomorphisms (up to algebraic variety of degenerate matrices, i.e. D := {D ∈ scale) for a three-dimensional vector space V . PN | det(D) = 0}), where det(D) is the determinant The graph of a planar homography H is given by the of D. We are interested in a better understanding of de- set of pairs of corresponding points (x , x0 ) ∈ 2 × 2 generacy arguments from the analysis of pencils (in fact i i P P contained in two views modeled as projective planes tangent directions) passing through a lower rank endo- fulfilling H(x ) = x0 . From a global point of view, the morphism. The “moral” consists of the following simple i i ambient space is given as the image of the Segre embed- remark: the original action given by a matrix product, ding s : 2 × 2 ,→ 8, i.e. it is a 4-dimensional alge- induces an action on linear (k + 1) − dimensional sub- 2,2 P P P braic variety given at each point by the intersection of spaces by means the (k + 1)-exterior power of the orig- four functionally independent quadrics[4]. As Im(s ) inal action. Next paragraph illustrates this idea with a 2,2 parameterizes the set of bilinear relations between two simple example. projective planes, and each projective plane has a pro- In particular, a line L represents a pencil (uni-parameterjective reference given by 4 points, a general homog-

1 raphy H can be described in terms of two 4-tuples of family) of endomorphisms {Hλ}λ∈P , i.e. a linear trajectory in the ambient space PN where N = (n+1)2 −1. points that can be re-interpreted as the eight (nine up A general line has n + 1 degenerate endomorphisms to scale) projective parameters of a general matrix H. corresponding to the intersection L ∩ D denoted by 5 This justiﬁes perturbation arguments or, alternately, it D1, D2,..., Dn+1 ∈ D. Inversely, the generic element prevents against the indiscriminate use of linear interpolation of the linear pencil µiDi + µjDj for 1 ≤ i < j ≤ n + 1 for a non-linear variety as the cubic hypersurface D. Complete Endomorphisms in Computer Vision 5

The space of (3×3)-matrices X up to scale is a pro- nantal variety representing rank 1 matrices) is locally jective space P8, which is homogeneous by the action given by of the projective linear group PGL(9). The projective a − a a linear group PGL(3) induces an action on PEnd(V ) 11 01 10 a − a a that breaks the initial homogeneity of P8 due to the 12 02 10 (1) rank stratification given by three orbits. Each orbit is a21 − a01a20 characterized by the rank constancy of a representative a22 − a02a20 matrix. In particular, if Mr denotes the algebraic sub- 8 in the open coordinate set D+(a00) := {a ∈ P | a00 6= variety of matrices (up to scale) of rank 1 ≤ r ≤ 3, then 0} of P8. They are functionally independent (i.e. its ja- there is a natural rank stratification M1 ⊂ M2 ⊂ M3. cobian matrix has maximal rank) between them. Hence, Planar homographies may be viewed as elements of they define a smooth variety of codimension 4 (differ- M3\M2 up to scale. This rank decomposition is re- ential map of the above equations has maximal rank), stricted in a natural way to the fundamental variety which is locally diffeomorphic to P3. The induced group F. action allows to extend the local diffeomorphism to a global diffeomorphism. In particular, it is locally parameterized by a11, a12, a21, a22 corresponding to elements in the complementary box of a00 (obtained by 3.1.1 Canonical rank stratification eliminating the row and the column of a00). Formally, the involution on spaces that exchanges This subsection includes some results regarding the en- subindexes (fixed points for transposition) leaves invari- domorphisms End (V ) of a vector space V ⊂ 3, where r R ant the first and fourth generators, and identifies the r denotes the rank of a generic element. The first result second and third generators between them. Such invo- provides a description of endomorphisms End (V ) and 3 lution corresponds to a representation of the symmetric its singular locus corresponding to degenerate endomor- group, giving the local generators for the Veronese va- phisms End (V ). The second result gives its structure 2 riety of double lines, which is isomorphic to the dual as a locally homogeneous space, i.e. as a disjoint union (P2)ν counted twice. of G-orbits by the action of GL(3) on the vector space Anyway, the rank stratification can be reformulated of g. As usual, their elements are regular or eventually in homogeneous coordinates as follows: degenerate matrices, but their meaning is different as Lie group or Lie algebra, respectively. Corollary 1 The action of PGL(3) on PEnd(V ) gives an equivariant decomposition in three orbits characterized by the rank of the representative matrix up to Proposition 1 For any three-dimensional vector space scale. In particular: 1) the set of rank 1 endomorphisms V : a) the set of singular endomorphisms End2(V ) is End1(V ) (up to scale) is a 4D smooth manifold whose a algebraic variety of codimension 1 given by a cubic projectivization is diffeomorphic to P3, which is a closed hypersurface for n = 2, which is a subregular orbit orbit by the induced action; 2) the set of rank 2 endo- by the action of PGL(3) corresponding to “subregular” morphisms End2(V ) (up to scale) is a 7D subregular elements located in the adherence of the set of homo- orbit; and 3) the set of homographies corresponding to 8 graphies in P ; b) its singular locus is given by rank 1 regular endomorphisms (up to scale) End3(V ) is the degenerate endomorphisms End1(V ), which is a codi- regular orbit. mension 4 manifold (smooth subvariety) diffeomorphi- 3 The stratification of endomorphisms up to scale in- cally equivalent to P volving the projective model of planar views can be geometrically reinterpreted by reconstructing the va- Proof a) It is proved taking into account the character- riety E2 of degenerate endomorphisms as the secant 8 ization of singular endomorphisms by the vanishing of variety Sec(1, End1(V )) in P of the smooth manifold the determinant of a generic 3 × 3 matrix. b) By taking End1(V ). Secant varieties are explained in Section 3.1.2. the gradient field in P8, its singular locus is locally de- The action of GL(n+1) can be extended in a natural scribed by the vanishing of determinants of all (2 × 2) way to the k-th exterior power involving (k + 1)-tuples minors of a generic matrix A representing any endo- of vectors and their transformations for 0 ≤ k ≤ n. morphism up to scale. Using (aij) as local coordinates Thus, a locally symmetric structure is obtained for ar- 8 in P , if a00 6= 0, then a local system of independent bitrary configurations of (k + 1)-tuples of vectors (or equations (local generators for the ideal of the determi- k-tuples of points). It is extended in a natural way to 6 Finat, Javier, Delgado-del-Hoyo, Francisco linear envelopes of (k+1)-dimensional vector subspaces Obviously, if 2m ≥ N the secant variety Sec(1,M) or, in the homogeneous case, to k-dimensional projec- fills out the ambient projective space. More generally, tive subspaces giving linear envelopes for any geometric the following result is true: object contained in the ambient space. Lemma 1 If M is a m-dimensional smooth connected The set of (k+1)-dimensional linear subspaces W k+1 manifold, the expected dimension of Sec(1,M) is equal are elements of a Grassmann manifold Grass(k +1, n+ n to min(2m + 1,N). 1); its projective version is denoted as Grk(P ). Grass- mann manifolds are a natural extension of projective The lemma is a consequence of a computation of spaces. They also provide non-trivial “examples” for parameters on connected smooth varieties. The dimen- homogeneous spaces and their generalization to sym- sion of the secant variety can be lower, but exceptions metric spaces or spherical varieties, jointly with super- are well-known for a specific type of low-dimensional imposed universal structures (fiber bundles). They have varieties called Severi varieties[28]. In particular, the been overlooked over the years despite the presence of chordal variety of the m-dimensional Veronese variety the analysis based on subspaces in a lot of tasks. A brief has dimension 2m, instead of the expected dimension introduction to Grassmannian manifolds and their ap- 2m+1, providing the first non-trivial example of a Sev- plications is provided by [27]. eri variety. More generally, if M is a m-dimensional connected manifold and 2m+1 ≥ N, as Sec(1,M) is a connected variety, then Sec(1,M) = N (see [22, Page 40] 3.1.2 Secant varieties P for more details about the Veronese embedding). An alternative description for a secant variety can Homographies, fundamental or essential matrices can be provided in a purely topological way. Let define the be viewed as PL-uniparametric families (linear pencils) diagonal of the product M × M as ∆ , i.e. the set of matrices that can be represented by secant lines. Sim- M of pairs (x, y) ∈ M × M such that x = y. If M is ilarly, secant planes would correspond to PL-biparametric a smooth m-dimensional variety, then M is diffeomor- families (linear nets) of matrices, and so on. Additional phic to ∆ through the diagonal embedding. Hence, formalism is required for a systematic treatment of these M the normal bundle N is isomorphic to the tangent families. ∆M bundle τ . Note that τ = τ ⊕ τ . This topolog- Secant varieties provide a PL-approach to any vari- M M×M M M ical description is useful to detect “regular” directions ety X relative to any immersion f : X → N . They are P thorugh the singular locus: given as the closure Sec(k, X) of points z ∈ Lk ⊂ N P Each pair (x, y) ∈ M × M − ∆ can be mapped to where Lk =< x , . . . , x > with x , . . . , x ∈ f(X) are M 0 k 0 k the line ` =< x, y > (also denoted as x × y). This map linearly independent. The k-th secant map associates defines a morphism σ : M × M − ∆ → Grass ( N ) to each collection of k + 1 l.i. points x , . . . , x their M 1 P 0 k called the secant map of lines. The closure of its image linear span Lk =< x , . . . , x >. The closure of the 0 k contains the set of tangent lines to M, which correspond graph is called the secant incidence variety. The projec- to limit positions of secant lines when x, y coalesce in tion of the last component on the Grassmann manifold one point (x, x) ∈ ∆ . Grass ( N ) = Grass(k + 1,N + 1) is called the k-th M k P The closure of the graph Γ of the secant map of secant variety (of secant k-dimensional subspaces) of X σ lines is a (2m + 1)-dimensional incidence smooth pro- and it is denoted by S (X) as subvariety of G ( N ). k k P jective variety in the product N × N × Grass ( N ) Since ordinary incidence conditions p ∈ L are in- P P 1 P whose projection on last component is, by definition, variant by the action of the projective group, secant in- the 1-secant (2m + 1)-dimensional subvariety S (M) of cidence varieties represent projectively invariant condi- 1 Grass ( N ). The tangent space at each point x ∈ M tions too. These conditions extend the well-known tan- 1 P can be described as the set of tangent lines to curves gency conditions z ∈ T Y . Next, we provide a classical x having a contact of order 2 with x. This definition is definition for smooth manifolds: formally extended to the singular case by taking local derivations. However, we use a more simplistic approach Definition 1 For any embedding M m ,→ N of a con- P based on the continuity arguments and a simple compu- nected regular m-dimensional manifold M, the secant tation of parameters, which gives the following result: variety Sec(1,M) (also called “chordal variety” in the old terminology) is defined by the closure of points Proposition 2 The incidence variety of secant lines z ∈ PN lying on lines xy (called “chords”, also), where contains the 2m-dimensional space TM of the tangent (x, y) ∈ M × M − ∆M are different points belonging bundle τM of the manifold M. Its projection to M, where ∆M := {(x, y) ∈ M × M | x = y} is the N diagonal of M × M. S1(M) ⊂ Grass1(P ) Complete Endomorphisms in Computer Vision 7

on the image of Sec(1,M) by the secant map σ is a The extension of the Corollary 2 to the space P9 of (m + 1)-dimensional subvariety of the Grasmannian of quadrics in P3 is meaningful for 3D reconstruction is- lines. sues. Indeed, a projective compactification of essential manifold E is isomorphic to a hyperplane section of the These descriptions show how secant lines can be un- variety of quadrics in P3 of rank r ≤ 2 (see below), derstood in terms of the geometry of the ambient pro- whose elements are pairs of planes (eventually coinci- jective space or, alternately, in terms of the geometry of dent). Grassmannians of lines G ( n). The arguments are ex- 1 P Nevertheless their simplicity, these results are useful tended to higher dimension and singular varieties in [6]. because provide a general strategy to manage degen- Furthermore, they correspond to decomposable tensors eracies. In particular, for each degenerate fundamental which are useful for estimation issues, also. 0 matrix F ∈ F1, a generic segment F + λF connecting two rank 1 fundamental matrices gives a generic 3.1.3 Secant to degenerate fundamental matrices rank 2 fundamental matrix. As consequence, a generic perturbation with any PL-path removes the indeter- Results in previous subsection allow to manage degen- minacy, and recovers a generic rank 2 fundamental ma- eracies in regular transformations and to perform a PL- trix. This perturbation method (valid for stratifications control in terms of limit positions of secant varieties. Its with “good incidence properties” for adjacent strata) extension to singular cases requires also the following provides a structural connection between fundamental result: matrices and homographies, which can be extended to Proposition 3 Let denote the three-dimensional alge- essential matrices (see Section 4.1.4). braic variety of degenerate rank 1 fundamental matrices 3 3 7 7 8 as F1 , then Sec(1, F1 ) = F2 and Sec(1, F2 ) = P . 3.2 Removing indeterminacies Proof It suffices to prove that if rank(F) = rank(F0) = 0 0 1, then rank(F + λF ) ≤ 2, i.e. | F + λF |= 0. So, let There are different kinds of indeterminacies for mul- define F = (f1 f2 f3), where fi is the i-th column of the tilinear approaches including structural relations be- 0 matrix F for 1 ≤ i ≤ 3. Then, | F+λF | is computed as tween pairs (given by fundamental and essential matri- the arithmetic sum (up to sign) of determinants which ces, typically) or triplets of views (trifocal tensors, e.g.). are always null. More explicitly, All of them can be interpreted as singularities of the variety of corresponding tensors, including the fundamen- 0 tal variety F or the essential variety E of End(V 3). | F + λF | =| f1 f2 f3 | P 0 0 Indeterminacies can be removed using secant lines to +λ(| f1 f2 f3 | + | f1 f2 f3 | + | f1 f2 f3 |) the singular locus F1 := Sing(F2) of F (introduced in 2 0 0 0 0 0 +λ (| f1 f2 f3 | + | f1 f2 f3 | + | f1 f2 f3 |) Section 3.1) or more generally k-th exterior powers. 3 0 0 0 +λ | f1 f2 f3 | Secant lines allows to control degenerated matrices using chords cutting singular varieties. These chords The first and last summand vanish since F, F0 ∈ F . 1 interpolate between more degenerate cases to recover a By developing each determinant by the elements of the valid case. This subsection explains an alternative re- column (Laplace) and by using that all 2 × 2-minors of covery strategy based on the “recent history” of tangent F and F0 vanish the proposition is proved. vectors to the camera poses. This also requires a careful This result can be also applied to symmetric ma- analysis of tangent spaces to matrices in terms of their trices up to scale in the projective space P5 of plane adjoint matrices. conics:

Corollary 2 Let denote the two-dimensional algebraic 3.2.1 A naive approach 2 variety of degenerate rank 1 symmetric matrices as Q1, 2 4 4 5 A regular transformation can be described by a matrix then Sec(1, Q1) = Q2 and Sec(1, Q2) = Q3 = P . A with non-vanishing determinant or, from a dual view- Proof The second equation is trivial by connectedness point, by the adjoint matrix adj(A). Entries Aij of the properties and dimensional reasons. An intuitive proof adjoint matrix are the adjoint of each element aij ∈ A for the ﬁrst equation is obtained from Proposition 3 up to scale; in the regular case, one must multiply with by the involution that exchanges subindex coordinates. det(A)−1. Hence, entries of the adjoint matrix repre- Intuitively, the generic element of any pencil generated sent up to scale the gradient ﬁeld of the array a linked by two double lines is a pair of intersecting lines[28]. to A. 8 Finat, Javier, Delgado-del-Hoyo, Francisco

For non-regular matrices (i.e. matrices A with van- `1 = P < X1,X3 > and `2 = P < X2,X4 >. In this ishing determinant) both descriptions are no longer equiv- case, the adjoint map induces an involution that can alent between them. Let study the simplest case for be translated to a complex conjugation between the n = 3 where the adjoint can be extended to non-regular generators for each ruling. matrices. In the regular case, the adjoint is formally described as the second exterior power V2 A of A. The 3.2.2 A more formal approach action of GL(3, R) = Aut(V ) on End(V ) = g induces an action of the second exterior power on the space From a local topological viewpoint, degenerate endo- of 2-dimensional subspaces (bivectors), whose projec- morphisms can be studied in terms of limits of tangent tivization represents lines of the projective plane as- vectors X = tAγ(t) to curves γ(t) through A. Such sociated to the image plane. Thus, the Adjoint map curves represent trajectories in the matrix space con- adj : A 7→ adj(A) replaces the study of loci character- necting “consecutive” poses for a camera. Regularity of ized by 0D features by their dual 1D features supported generic points of such curves allows to define tangent by projective lines. Loci and enveloping hyperplanes are vectors at isolated degeneracies by means secant lines. equivalent between them for regular matrices. A more intrinsic approach to tangency conditions This naive approach has some implications to re- must include the dual matrix for any X ∈ End(V ). It move indeterminacy when rank(F) = 1). In order to is defined at each point by the adjoint matrix adj(X) understand them F1 must be replaced by an enlarged whose entries Xij are the signed determinants of com- space that includes the different ways of approaching plementary minors of xij. If X is a regular matrix, then each element F ∈ M1 by an “exceptional divisor”. Each X.adj(X) is a power of det(X) 6= 0, i.e. a unit from divisor is supported by a finite collection of hypersur- a projective viewpoint. We are interested in extending faces in the ambient space M3 representing different ap- this construction to singular endomorphisms by using proaching ways to the singular locus, including elements intermediate exterior powers. Their closure in the corre- of M2. This process is known in Algebraic Geometry as sponding projective space forms the variety of complete a blowing-up or σ-process [22]. endomorphisms. To begin with, a first order complete endomorphism An almost-trivial example Bilinear maps on a vector representing a degenerate planar transformation is given, space are represented by a 4-dimensional arbitrary space up to conjugation, by a pair of matrices (X, Xν ), where coupled with an inner product < X, Y >= tr(XY ). A X is an endomorphism of a 3D vector space V , and direct computation shows that an orthogonal basis with ν 2 X represents its dual given up to scale by the ad- T r(X ) 6= 0 and T r(X X ) = 0 is given by 2 i i j joint matrix adj(X) = V X. The replacement of a 1 0 0 0 0 1 0 1 matrix with its adjoint transforms any incidence con- , , , 0 0 0 1 1 0 −1 0 dition (pass through a point for a conic, e.g.) into a tangency condition (dual line becomes tangent at a Note that X1,X2 and X3 + X4 are second order point, e.g.). Moreover, this exchange between projec- nilpotent operators and that the last generator X4 pro- tively invariant conditions does not depend of the di- vides the bilinear structural constraint for the symplec- mension. For regular homographies, there exists a nat- 2 tic geometry on V [16,8]. In this case, ad(X1) = X2, ural duality between descriptions in terms of the origi- ad(X2) = X1, ad(X3) = −X3, and ad(X4) = X4, with nal matrix A and its adjoint matrix adj(A), giving the det(X1) = det(X2) = 0, det(X3) = −1 and det(X4) = natural duality between incidence and tangency con- 1. ditions for smooth “objects” (endomorphisms, in our From a projective viewpoint, the set of bilinear maps case). Hence, the only novelty appears linked to singu- 1 on the projective line P can be reinterpreted in terms lar strata which can be illustrated by its application 1 1 3 of the image of the Segre embedding s1,1 : P ×P → P to the variety of singular fundamental matrices when given by n = 3. The dual construction is compatible with the induced action of PGL(3) on PEnd(V ) and its restriction ν ∗ to F. Let X ∈ g , g = TeG, then X ∈ g . ([x1 : x2], [y1, y2]) 7→ [x1y1 : x1y2 : x2y1 : x2y2] For arbitrary dimension, the entries Xij of the ad- Vn n Let be zij := xiyj then z11z22 − z12z21 = 0 defines a joint matrix adj(X) = X = det(X) for X ∈ End(V ) one-sheet hyperboloid in P3. Hence, the set of bilinear are interpreted as determinants corresponding to the 2 maps on V (up to scale) can be modelled with such components of ∇det(X). The next iteration for Xij hyperboloid. The main novelty here concerns the rul- gives the determinants of (n − 2) × (n − 2)-minors as ing given by the group action on the projective lines generators for the ideal of ∇Xij. Their vanishing defines Complete Endomorphisms in Computer Vision 9 the singular locus of the variety det(X) = 0. Symbol- (3, 3), (2, 1), (1, 2) and (1, 1). Hence, the most degener- ically, ∇2det(X) can be formulated as a double itera- ate orbits of complete quadrics have multiranks (1, 3, 3), tion of the adjoint map generated by the vanishing of (1, 2, 1), (1, 1, 2) and (1, 1, 1). If k ≥ 2 the exterior pow- determinants of minors of size n − 2 of X. These deter- ers represent geometrically the matrices acting on en- minants are the generators of Vn−1 End(V n+1), which velopes by k-dimensional tangent linear subspaces to V2 n+1 n is the dual of End(V ). “any object” contained in P of increasing dimension. The description of the previous paragraph can be In this case, (a) the action of GL(n + 1) induces a con- geometrically reinterpreted in terms of arbitrary codi- jugation action of Vk GL(n + 1) on itself; (b) the k-th mension k subspaces. In particular, the extension of the exterior powers of X can be considered as elements of adjoint map can be algebraically interpreted as a gra- the k-th exterior power of g∗. dient field. For any endomorphism X let consider the Essential manifold E5 of regular essential matrices is n-tuples embedded in P(End(V 4)), a 15-dimensional projective space. The extension of complete homographies on 2 2 n P ^ ^ to 3 creates complete collineations (X, V2 X, V3 X). (X, X,..., X) , (2) P The third component V3 X is in fact the adjoint matrix k of X. This construction provides a general framework where V X is the k-th exterior power of X, whose to obtain compactifications (as complete varieties) of entries are given by the determinants of the k×k-minors ∗ orbifolds corresponding to F and E as degenerate en- of X. It is an element of the exterior algebra V End(V ) domorphisms. defined by the direct sum of exterior powers of End(V ). The basic idea for extending regular to singular cases The iteration of the gradient field given by the deter- is based on adding infinitesimal information from suc- minant function det : End(V ) → R as X 7→ det(X) cessive adjoint maps, which is interpreted as the itera- can be interpreted as “successive derivatives” on the tion of the gradient operator applied to the determinant space of endomorphisms. Let us remark that traces of of square submatrices. For generic singularities (i.e. for exterior powers are the coefficients of the charatceristic corank c = 1) it suffices to replace the original formula- polynomila | λI − A |;, which can be reinterpreted (in tion by its dual, which gives the tangent vector for small the complex case) in terms of eigenvalues. Thus, in this displacements. For singularities with corank c ≥ 2 suc- case all the information is computable in terms of SVD cessive exterior powers and complete endomorphisms with usual interpretation for the ordered collection of must be considered. This differential description allows eigenvalues. to interpret complete endomorphisms in terms of the In arbitrary dimension, the generic case corresponds “recent history” along the trajectory. In order to sim- to the regular orbit, i.e. endomorphisms X with rank(X) = plify the developments we consider the particular case n+1 (automorphisms). By iterating the construction of c = 2, n = 3. exterior powers, one can associate an algebraic invariant given by the multirank rank(Vk+1 X) = rank(Vn−k X). Looking at Figure 2, the regular case for n = 2 (resp. 3.2.3 A symbolic representation n = 3) corresponds to bilinear forms with birank (3, 3) (resp. the multirank (4, 6, 4) with self-duality for the Adjacency relations between closures of orbits for rank mid term), which can be reinterpreted in terms of quadratic stratification of End(V ) can be symbolically represented forms. The case for non-regular orbits is constructed by the oriented graph of Figure 1. The vertices of the recurrently: let k = corank(X) be the dimension of triangle represent an orbit labeled with 3, 2, 1, accord- Lk = ker(X), the indeterminacy is removed by adding ing to the rank of the matrix representing the endomor- the complete bilinear as the linked quadratic forms on phism. Oriented edges are denoted as e32, e21 and e31 Lk. and represent the following degeneracies: For example, for any symmetric endomorphism X – 3 → 2, corresponding to degeneracies of endomor- whose projectivization is a rank 1 plane conic, there phisms to fundamental matrices. 2 exists a double line ` = 0 whose kernel is the whole – 2 → 1, corresponding to degeneracies from funda- line. The reduced 1D kernel ` is also the support for a mental to degenerate fundamental matrices. 0D conic on the line given by two different points (rank – 3 → 1, corresponding to degeneracies from rank 3 2) or a double point (rank 1), which define two orbits endomorphisms to degenerate fundamental matrices labeled as (1, 2) and (1, 1) in Figure 2. Similarly, for a rank 1 quadric supported by a double plane L2, the Right-side vertices of Figure 1 represent the biranks kernel is the whole plane that supports an embedded corresponding to the original matrix and its general- complete conic (q, qν ) in the double plane with biranks ized adjoint. A simple example is useful to illustrate 10 Finat, Javier, Delgado-del-Hoyo, Francisco

(1,2) (1,1) 3.3 Complete fundamental matrices 1 This section adapts constructions of endomorphisms in any dimension shown in precedent section to fundamental matrices. The notion of complete fundamental matrix is crucial to perform a control of degenerate cases from a quasi-static approach. This means that we are 3 2 not taken into account the kinematics of the camera. a (3,3) (2,1) Nevertheless, as the adjoint f represents the gradient vector field ∇ a F at the element f, there is a measure- Figure 1 Oriented graphs for the adjacency relations be- f tween the closures of orbits for the rank stratification of ment of “local variation” around each element f ∈ F. End(V ). Nodes represent a rank stratification whilst edges represent degeneracies in an orbit. Definition 2 The set of complete fundamental matri- 0 a a ces is the closure F2 × F2 of pairs (F, F ) ∈ where a a F ∈ F2 is a fundamental matrix of rank 2, and F ∈ F1 the idea. Let suppose X ∈ End(V ) as a diagonalizable is its adjoint matrix whose entries are given by the determinants of all 2 × 2-minors of F. matrix and λ1, λ2 and λ3 as the eigenvalues of A, then fundamental matrices have at least one vanishing eigen- Degeneracies of endomorphisms up to scale linked value. In the complex case canonical diagonal forms to camera poses can be controlled with the insertion of would be equivalent to diagonal matrices (λ , λ , λ ) 1 2 3 tangential information and the restriction of the con- whose adjoint would be (λ λ , λ λ , λ λ ), (λ λ , 0, 0), 2 3 1 3 1 2 2 3 struction to F. This information allows to represent and (0, 0, 0), respectively. The adjoint of the third type “instantaneous” directions in terms of all possible di- is identically null so, initially it does not provide ad- rectional derivatives represented by the adjoint matrix. ditional information about the most degenerate case. Thus, despite rank(F) = 2 and consequently rank(Fa) = Information about regular elements in the degenerate 1, the adjoint matrix provides a parameterized sup- support corresponding to neighbor tangent directions port (in fact a variety) to represent “directions” along (infinitesimal neighborhood) must be added to avoid which the degeneracy occurs. However, this remark is the indeterminacy. must be replaced by the excep- M1 not longer true for the most degenerate cases corre- tional divisor E of the blowing-up of 8 with center the P sponding to rank 1 matrices that require a set of pairs smooth manifold . E has the orbits by the adjoint M1 of degenerate fundamental matrices. action as components, represented by the pairs (1, 2) and (1, 1). 3.3.1 Complete pairs of fundamental matrices The blow-up of the graph G replaces the oriented triangle whose vertices are labeled as 3, 2 and 1 with a new In the absence of perspective models supporting arbi- (1) graph G . This graph is an oriented square whose ver- trary homographies, fundamental matrices are used to tices are labeled as (3, 3), (2, 1), (1, 2) and (1, 1). These avoid indeterminacies. For consecutive camera poses, labels represent the biranks of pairs of endomorphisms two fundamental matrices F (for views V1 and V2), and (A, adj(A)). Biranks are relative to the generic ele- 0 0 0 F (for views V1 and V2 ) provide a structural relation ments of the topological closure for the orbits of the between both views. V2 product action G × G, where G = Aut(V ) = GL(3), The dual construction X 7→ adj(X) corresponding V2 on the space g × g, where g = End(V ) = g`(3), to the gradient ∇ at each point can be restricted just to acting on the graph of the adjoint map. rank 2 fundamental matrices. This provides a tangential Hence, biranks encapsulate numerical invariants for description with information about the first order evo- the natural extension of the action of the Lie alge- lution of F according to tangential constraints. From a bra g`(3) corresponding to End(V ) = TI Aut(V ), us- more practical viewpoint, tangential information can be ing pairs of infinitesimal transformations on the origi- approached by secant lines in a PL-approach that con- nal and dual spaces. The simultaneous management of nects rank 2 fundamental matrices f, f 0 ∈ F. However, complete endomorphisms (X, Xν ) is more suitable from this construction becomes ill-defined when rank(F) = 1 the computational viewpoint. Indeed, a simple exten- since the adjoint map is identically null. Then, tan- sion of SVD methods to pairs of endomorphisms allows gential directions corresponding to endomorphisms of an estimation of the generators of Lie algebras g and W = Ker(X) are added to avoid this indeterminacy. its dual g∗ easier than the estimation of the generators For regular matrices the extended approach for com- for the original Lie group G. plete objects is compatible with the differential approach Complete Endomorphisms in Computer Vision 11 given by a smooth interpolation along a geodesic path Proposition 4 The 7D secant variety Sec1(F1) is a γi,i+1 connecting consecutive complete fundamental ma- generically triple covering of F2 that ramifies along the a a trices (fi, fi ) and (fi+1, fi+1). These are obtained as the 6-dimensional tangent variety T1(F1) (total space of the lift of a geodesic path to the tangent space. From a tangent bundle) corresponding to tangent lines having a theoretical viewpoint, lifting is performed by restrict- double contact at each element F ∈ F2. ing the logarithm map. In practice, secant lines provide a first order approach to geodesics. 3.3.3 Triplets of fundamental matrices A more detailed study of the geometry of degenerate fundamental matrices is required to recover a well- This Section is intended to provide an algebraic visu- defined limit of tangent spaces in the singular case. alization of neighboring matrices at the most degener- This study must include procedures for selecting a PL- ate case, which avoids the indeterminacy at rank 1 ele- path (supported on chords) connecting the degenerated ments. We use a geometric interpretation of the blowing- f 0 with its neighboring generic fundamental matrices. up process explained in [22] in terms of secant varieties. 3 In practice, if the sampling rate is high enough there In our case, the blowing-up of the variety F1 re- will be no meaningful difference between pairs of ma- places each rank 1 degenerate fundamental matrix F ∈ 3 trices. This generates uncertainty about the direction F1 with a 3D subspace generated by four linearly in- to approach the tangent vector. It can be solved with a dependent vectors. They can be interpreted in terms of coarser sampling rate along the “precedent story”. secant lines connecting the point f for matrix F with four independent points belonging to the subregular 3.3.2 Incidence varieties and canonical bundle orbit F2\F1. Hence, each extended face of a generic “tetrahedral configuration” (see Figure 2) represents a 3D secant projective space to F displaying degeneracies A local neighborhood of F ∈ F1 relative to F2 is the unit sphere corresponding to the fiber of the punctured at each F1 ∈ F1. normal bundle of F in F . Normal bundles are given The cubic hypersurface representing fundamental 1 2 8 by a quotient of tangent bundles. Hence, tangent bun- matrices in P is not a ruled variety. Thus, a generic triplet of fundamental matrices generates a 2-dimensional dles of F1 and F2 = Sec(1, F1) must be computed. The secant plane to the hypersurface det(F) = 0. Obvi- latter can be described as the blow-up of F1 × F1 with center in the diagonal ∆ . This diagonal is isomor- ously, the variety of secant lines to F2 and trisecant F1 8 2-planes to F1 fills out the projective space F3 = phic to the smooth manifold F1 so its tangent bundle P of endomorphisms up to scale. More specifically, any is also isomorphic to the tangent bundle of F1. Hence, it suffices to compute the latter and reinterpret it in homography can be expressed as a linear combination geometric terms. of three generic fundamental matrices (similarly for essential matrices). Note that F1 is a smooth manifold diffeomorphic The nearest 3-secant 2-plane < F , F , F > for to P3, so their tangent bundles are isomorphic. Thus, 1 2 3 each rank 1 fundamental matrix F ∈ F can be com- 3 1 τF1 ' τP can be reinterpreted in terms of incidence varieties. Simplest incidence varieties in the projective puted using a metric on the Grassmannian of 2-planes. 2 2 2 ν The most common metric is the inner product plane P are given by I1,2 := {(p, `) ∈ P × (P ) | p ∈ `}. < X, Y >= tr(X.Y) There are two projections on P2 and (P2)ν to interpret the incidence variety as the canonical bundle of g = End(V ). In particular, at each degenerate en- of the projective plane. This elementary construction domorphism X corresponding to a fundamental matrix is extended to subspaces of any dimension k with the F (rep. essential matrix E), two different eigenvalues canonical bundle on the Grassmannian of k-dimensional λi1, λi2 (resp. a non-zero double eigenvalue λi) are ob- subspaces. An example is the incidence variety I1,8 := tained for 1 ≤ i ≤ 3. 8 8 8 {(a, `) ∈ P × Gr1(P ) | a ∈ `}, where Gr1(P ) denotes In practice, the direction to choose as “escape path” 8 the Grassmannian of lines in P . should correspond to the nearest Fi with maximal dis- The restriction to the fundamental variety F of the tance in the plane (λ1, λ2) of non-null eigenvalues. This 8 second projection on Gr1(P ) gives the secant variety distance is determined w.r.t. the eigenvalues (λ, 0) or Sec1(F) to F that fills out all the ambient projective (0, λ) of the degenerate fundamental matrix F (simi- space. In other words, along each point F ∈ P8 pass a larly for the essential matrix). The application of this secant line to F. The same is also true for E instead of theoretical remark would must allow to escape from F . Furthermore, an interesting result can be formulated degenerate situations to avoid collisions against pla- using the same notation: nar surfaces (corresponding to walls, floor, ground, e.g.) 12 Finat, Javier, Delgado-del-Hoyo, Francisco whose elements do not impose linearly independent con- terms of ordinary rotations. In addition, this construc- ditions to determine F. From a more practical view- tion can be adapted to bilateral (product or contact) point, the problem is the design of a control device able actions in terms of double conjugacy classes. We are in- of identifying the “best” escape path in a continuous terested in the locally symmetric structure of the rank- way, i.e. without applying switching procedures. In the stratified set of projection matrices linking scene and next paragraph we give some insight about this issue. views models. A simple description of this structure for any space allows to propagate control strategies by using local symmetries, without using differential meth- 4 Extending the algebraic approach ods, no longer valid in the presence of singularities. The completion of planar homographies in P8 to in- This section explains how complete matrices, which can clude fundamental matrices can be extended to any di- be read in terms of successive envelopes, provide specific mension and re-interpreted in matrix terms. To achieve control mechanisms to avoid degeneracies appearing in this goal, it is required to consider the projectivization rank 1 fundamental or essential matrices. P(End(V n+1)) of a (n+1)-dimensional space V , to con- A basic strategy to analyze and solve the indetermi- struct the rank stratification of matrices in End(V ) and nacy locus of an endomorphism consists in augmenting to take the (k + 1)-th exterior power (up to scale) of the original endomorphism by their successive exterior them for 0 ≤ k ≤ n. The locally symmetric structure powers. The properties of the adjoint matrix provide for the resulting completed space is obtained by the in- a geometric interpretation in terms of successive gen- duced action of GL(n+1) = Aut(V n+1) on the (k +1)- eralized secant envelopes by k-dimensional subspaces6. th exterior power Vk+1 V of V . This structure justifies Consecutive iteration on the adjoints can be viewed as positional arguments for minimal collections of corre- a Taylor development so that when rank(X) decreases, sponding elements (points, lines, or more generally, lin- all complete objects contained in ker(X) can be added ear subspaces) and controls their possible degeneracies to remove degeneracies. in terms of adjacent orbits. The lifting of the action of Aut(V ) on End(V ) to The construction of the above completion poses some their k-th exterior powers delivers a structure as lo- challenges. For example, its topological description in cally symmetric spaces for the set of complete objects PN for N = (n + 1)2 − 1 requires additional (k + 1)- linked to End(V ). Here Aut(V ) can be replaced with dimensional linear subspaces in the ambient projective a group G, the dual of End(V ) (corresponding to take space Pn. Using contact constraints for linear subspaces adjoint matrices) with g∗ and the induced action by has an equivalence to scene objects in terms of PL- the adjoint map ad : G → g∗ giving the adjoint ac- envelopes by successive higher dimensional linear sub- tion ad : G × g∗ → g∗. Then, the k-th exterior power spaces Lk+1. Due to space limitations, we constrain our- k k k V ad : V G → V g∗ is the natural extension of the selves to the case n = 3 and the completion of essential adjoint map. This map induces the corresponding k-th matrices. k k adjoint action of V G on V g∗ that extends the origi- The fact that End(V ) (including degeneracies) is nal action of Aut(V ) on End(V )∗. This simple construc- the Lie algebra of Aut(V ) (only regular transformation is applicable for all actions of classical groups to tions) links algebraic with differential aspects. Hence, remove their possible indeterminacies on Lie algebras. exponential and logarithm maps provide a natural re- The simplest non-trivial example is the pair (k, n) = lation between both of them. However, as fundamental 4 (2, 4), which defines a space of dimension 6 = 2 for and essential matrices play a similar role for affine and V2 V 4. In this case, the 6 × 6 regular matrices of V2 G euclidean frameworks (as non-degenerate bilinear rela- (automorphisms of V2 V 4) act on the 6 × 6 arbitrary tions), a common framework where both interpretations matrices of Vk g∗ (endomorphisms of V2 V 4). In prac- are compatible is required for an unified treatment of tice it requires to compute the determinants of 2 × 2- degenerated cases. minors of a 4 × 4 regular matrix acting on 2 × 2-minors of a 4×4 arbitrary matrix. This explains the jump from 4.1 Essential manifold original ranks (4, 3, 2) of arbitrary 4 × 4-matrices and their second powers to ranks (6, 3, 1). The essential constraint for pairs (p, p0) of correspond- An adaptation of the general linear approach to the ing points from a calibrated camera is given by T pEp = euclidean case provides a decomposition of the 6 × 6- 0. The set of essential matrices E is globally character- matrices in 3 × 3-blocks that can be reinterpreted in ized as a determinantal variety in [11, Section 2.2]. The 6 This is valid for tangent subspaces as limits of secant author explores the structure of E as a locally symmet- subspaces in the Grassmannian ric variety and a completion (not necessarily unique) Complete Endomorphisms in Computer Vision 13 obtained using elementary properties of adjoint matri- – The essential variety E is a 5-dimensional degree 10 ces. subvariety of P8, which is isomorphic to a hyper- Any ordinary essential matrix E has a decompo- plane section of the variety V2 of complex symmet- sition E = RS, where R is a rotation matrix and ric matrices of rank ≤ 2[5]. The singular locus is S is the skew-symmetric matrix of a translation vec- isomorphic to P3 via the degree 2 Veronese embed- tor t. Essential and fundamental matrices are related ding v2,3 whose image is the variety V1 of double T through E = Mr FM` where Mr (resp. M`) is an planes. In particular Sec(1,V1) = V2. affine transformation acting on right (resp. left) on the – If TSO(3) ' SO(3) × R3 ' SO(3) × so(3) denotes source (resp. target space). In algebraic terms, they be- the total space of the tangent bundle of SO(3), then long to the same double conjugacy class by the diagonal the essential regular manifold E0 is isomorphic to of the A-action of two copies of the affine group, where the total space of unit tangent bundle of SO(3) A = R × L is the direct product of right R and left given by SO(3) × S2. L actions. Hence, essential matrices can be considered Nevertheless their local description in terms of Lie as equivalence classes of fundamental matrices. The fol- algebras, all the above isomorphisms are global since lowing result gives a synthesis of the above considera- any Lie group G is a parallelizable manifold. If G is tions: connected, the isomorphisms are infinitesimally given by the translation T G = A.g. In the euclidean frame- Proposition 5 The variety E of extended essential ma- A i work, this description allows to decouple rotations and trices of rank ≤ i is a quotient of the variety F of i translations. The rest of this Section explains local and extended fundamental matrices of rank ≤ i. More gen- global properties of extended E and their relations with erally, the stratified map F → E is an equivariant fi- fundamental matrices. bration between stratified analytic varieties for natural rank stratifications in PEnd(V ). 4.1.1 Parameterizing the essential manifold The exchange between projective, affine and euclidean An essential matrix E ∈ E0 is a 2-rank matrix with information and the analysis of degenerate situations two equal eigenvalues and a diagonal form (λ, λ, 0), requires a general framework where rank transitions can which in the complex case is projectively equivalent (up be controlled in simple terms. To accomplish this goal, to scale) to (1, 1, 0). SVD8 decomposes E in a prod- a locally symmetric framework that represents degen- uct UΣV T where U, V are orthogonal and det(U) = erate cases must be developed. +1 = det(V ). The sign of the determinant can be cho- Degeneracies can be studied in the space of multilin- sen without modifying the SVD. Then, the fibration ear relations between corresponding points, from which Φ : SO(3) × SO(3) → E given by Φ(U, V ) = UΣV T the essential matrix is estimated [12]. However, these es- is a submersion with a 1-dimensional kernel represent- timations are performed in the space of configurations ing the ambiguity for choosing the basis of the space of points without considering the degeneracies of the generated by the first two columns of U and V . matrices in the space of endomorphisms arising from The description of E in terms of the fibration Φ al- an algebraic viewpoint. The secant line connecting two lows to decompose any element E ∈ E in “horizontal” degenerate endomorphisms is translated in 8+8+1 = 17 and “vertical” components for the tangent space to the linear parameters in the same way as in [12]. Additional product SO(3) × SO(3). This is crucial to reinterpret constraints relative to fundamental or essential matri- the decomposition in locally symmetric terms and to ces help to reduce the number of parameters. bound errors linked to large baselines[23]. If X is a variety with singular locus Sing(X), the regular locus is denoted as X0 := X\Sing(X). In par- 4.1.2 A differential approach ticular, the set of regular fundamental matrices is de- 0 noted by F = F2\F1 and the set of essential matri- The set E0 of regular essential matrices is an open man- 0 ces as E = E2\E1. Rank stratification of End(V ) is ifold that can be globally described in terms of the unit F1 ⊂ F2 (resp. E1 ⊂ E2), where the subindex k denotes tangent bundle τuSO(3) to the special orthogonal group the algebraic subvariety of matrices with rank ≤ k, up SO(3) representing spatial rotations[18]. Also, o(3) := to scale 7. TI SO(3) is the Lie algebra of SO(3), i.e. the vector Two global algebraic and differential results to con- space of skew-symmetric 3 × 3-matrices that can be in- sider are the following ones: terpreted as translations in the tangent plane TASO(3) =

7 Fundamental matrices are considered in Section 3.1. 8 Singular Value Decomposition 14 Finat, Javier, Delgado-del-Hoyo, Francisco

A.so(3) at each A ∈ SO(3). Then, E0 is a 5-dimensional distortions must be avoided when approaching to the algebraic variety contained in the total space TuSO(3) ' singular locus. 2 4 SO(3) × S of the unit tangent bundle τuSO(3). In this The vector space End(V ) of 4 × 4-matrices X has bundle each fiber takes only unit tangent vectors so we a natural stratification by the rank rank(X) denoted restrict to unit vectors X ∈ so(3). by W1 ⊂ W2 ⊂ W3 ⊂ W4. Here, Wi represents the al- The isomorphism S2 = SO(3)/SO(2) enables a rein- gebraic variety defined by the vanishing of all determi- terpretation of the unit vectors as spatial rotations mod- nants of size (i + 1) × (i + 1), which are endomorphisms ulo planar rotations. Each element determines a unique of rank ≤ i. More concretely, ordinary homographies rotation axis, where the rotation through angles θ + π A ∈ PGL(4) are represented by points of the open set and θ − π are identical. Hence, SO(3) is homeomorphic W4\W3. Obviously, W3 is the natural generalization of to RP3, which provides a general framework for a pro- fundamental matrices given by det(X) = 0. jective interpretation in terms of space lines (see [18, In this framework, the variety of secant lines Sec1(E) Section 3.2]). Inversely, euclidean reduction of projec- to E' SO(3)×S2 in P15 is a 11-dimensional projective tive information can be viewed as a group reduction variety. Similarly, the variety of secant planes Sec2(E) to fix the absolute quadric that plays the role of non- to E fills out the ambient space P15. This means that degenerate metric (see Section 4.1.3). any spatial homography of P3 can be described by three essential matrices, but not in a unique way. Even more, 4.1.3 A local homogeneous description secant varieties to rank-stratified varieties of projective endomorphisms can be described in terms of locally The projective ambiguity of projective lines as rota- symmetric varieties. The simplest case corresponds to tion axis gives two possible solutions for corresponding symmetric endomorphisms representing eventually de- elements of SO(3) [3, Section 3]. This ambiguity can generate conics or quadrics. be modelled as a reflection that exchanges the current In particular, Figure 2 illustrates how to extend Fig- phase with the opposite phase between them. Hence, ure 1 to the third dimension for the symmetric case. pairs of regular essential matrices in SO(3) × SO(3), These orbits are induced by the action of PGL(4; C) where the the second component is generated by the on complete symmetric complex endomorphisms. The logarithmic map, generate a quadruple ambiguity cor- first blow-up in the graph replaces the vertex labeled responding to two simultaneous reflections. as 1 with the opposite face (with the same orientation). Using the topological equivalence between SO(3) This vertex represents the endomorphisms of rank 1 up 3 to scale or , They are completed in the new face rep- and RP , the ambiguity can be represented by the prod- E1 uct of two copies of the tangent bundle of the projective resenting the three orbits of the variety E2 of 1-secants space where elements (x, v) and (−x, −v) are identi- Sec(1, E1). Next, the second blow-up at each minimal fied by the antipodal map. This natural identification vertex at each height replaces such vertex by the oppo- 3 site side in the height with the same orientation. The of the tangent bundle τRP has not a kinematic meaning from the viewpoint of the “recent history”. Hence, result is an oriented cubical 3D graph whose vertices in order to solve the ambiguity, a C1-constraint must be represent orbits by the induced action of GL on suc- inserted into the essential matrices for precedent cam- cessive exterior powers. Vertices are labeled according era poses. However, this constraint is only valid under to the multirank of each symmetric endomorphism and non-degeneracy conditions for essential matrices, i.e. their respective envelopes by linear subspaces. the eigenvalue λ must be non-null. Otherwise, essential matrix “vanishes” and it cannot be recovered. 4.2 The adjoint representation

4.1.4 A remark for matrix representation Any element of a Lie group G can be lifted to its Lie algebra g with the log map log : G → g, the local inverse The manifold E0 of regular essential matrices can be of the exponential map exp : g → G. The adjoint repre- visualized as a smooth submanifold of spatial homogra- sentation G → Aut(G) models G as a matrix group in phies PGL(4) given as the open set of regular transfor- terms of its conjugation automorphism Φg defined by 15 −1 mations as points of P . These regular transformations Φg(h) := ghg for every g ∈ G. From the differential are described by automorphisms A ∈ Aut(V 4) of a 4- viewpoint, the adjoint representation of G is computed dimensional vector space V whose Lie algebra is given as the differential map of Φg at the identity e ∈ G, i.e. 4 by End(V ), including possible degeneracies. Our aim A : G → Aut(g) defined by A(g) := deΦg. is to study these degenerate cases by using locally sym- Its dual is given by the coadjoint action K : G → ∗ metric properties extending TRSO(3) = Rso(3). Metric Aut(g ) defined by the adjoint representation of the in- Complete Endomorphisms in Computer Vision 15

(1) the action of G. The corresponding algebraic equivariant stratification corresponds to an algebraic action. In the presence of motion, it is interesting to construct a (1,1,1) (3) (2) symplectic equivariant stratification to manage packs of solutions for structural motion equations [14]. This sub- (1,2,1) (1,1,2) (4) section refers only to basic aspects of algebraic equiv- (1,3,3) ariant stratifications. More formally, the compactification of the coadjoint (2,1,1) (1,2) (1,1) action for GL(n) creates an equivariant decomposition

(3,3,1) (2,1,2) of orbits for many general groups whose canonical forms are well-known (Jordan) [2]. Furthermore, this decom- (1,3) position can be restricted to the coadjoint actions for (4,6,4) any subgroup of G, such as SO(n) (preservation of (3,3) (2,1) metric properties), SL(n) (volume preservation, despite shape changes), or Sp(n) (preservation of motion equa- (4,6) tions). Figure 2 Symbolic representation of the orbits generated by The above conservation laws w.r.t. the algebraic G- the action of PGL(4). Dotted edges represent hidden edges. actions provide ideal theoretical structural constraints. Grey-colored edges are new edges generated after a blow-up of The problem is solved by minimizing their infinitesi- a vertex in a previous graph. The oriented tetrahedral graph gives the oriented triangular prism and this is finally con- mal variation in their Lie algebra space. The transfer- verted into the oriented cube. ence of information requires a more careful study of algebraic and differential relations, developed in Sec- tion 4.3. For the applications concerning this paper we verse element g−1. Orbits by the coadjoint action sup- have constraint ourselves to only regular or subregular port a symplectic structure[13]. orbits.

4.2.1 Some meaningful examples 4.3 Relating algebraic and differential approaches There are two meaningful examples of the adjoint representation for our case: This subsection explores the relations between an equivariant completion of the essential manifold and the fun- – If G = GL(3) = Aut(V ), where V is a 3D vector damental variety in the algebraic framework given by space, then g = End(V ) is naturally stratified by coadjoint actions. More concretely, we provide descrip- the rank. The coadjoint action corresponds to orbits tions of such completions in terms of locally symmetric of adjoint matrices. Its compactification has been spaces obtained by compactifying the original descrip- described in Section 3.1. tions as homogeneous spaces. From a topological view- – If G = SO(3), then the adjoint representation is point, a compactification incorporates the behavior at SO(3) → Aut(so(3)). It maps each spatial rota- boundaries of a topological space to preserve regularity tion g ∈ SO(3) into the automorphism so(3) → conditions. These compactifications include additional so(3) representing an “infinitesimal” displacement orbits corresponding to degeneracies in the topological between two consecutive poses as a translation in adherence of regular orbits. Therefore, computations for the tangent space. The adjoint action can be inter- the regular case can be extended to degenerate cases preted as the action induced by the differential of too. the Adjoint representation of G in Aut(G) given by ordinary algebraic conjugation9. 4.3.1 An affine reinterpretation

4.2.2 Equivariant decompositions Essential matrices have two equal non-vanishing eigenvalues and a null eigenvalue. A basic specialization prin- A topological equivariant stratiﬁcation of a G-space X ciple suggests to obtain an essential matrix with a re- w.r.t. a G-action G × X → X is a decomposition of X duction to the diagonal (on the space of eigenvalues), in a union of G-orbits, i.e. subsets of X invariant by and the addition of constraints relative to SO(3) and its 9 The adjoint starts with uppercase when it refers to a Lie algebra so(3). By the triviality of its tangent bundle, group. TRSO(3) is the translation Rso(3) of the Lie algebra. 16 Finat, Javier, Delgado-del-Hoyo, Francisco

Considering the degeneracies of fundamental and essen- provides the relation between fundamental and essential matrices, it is convenient to visualize how topolog- tial matrices. ical arguments for regular points can be extended to More generally, the matrix expression of the A-action singularities. In particular, complete objects allow to is given by the linearization of the double conjugacy control degeneracies in the boundary of orbits and to classes for the A-action. From the differential view- minimize errors in the presence of such singularities. point, it can be written as the first order term of the 1 In affine geometry, a completion of degenerate fun- 1-jet je,eA = A. This case comprises two linear actions damental matrices allows to recover affine reconstruc- acting on the Jacobian matrix at right and left simulta- tions. Euclidean reconstructions (up to scale) can be neously. In practice, the Lie algebras are preferred over achieved by preserving the absolute conic. However, eu- the Lie groups so instead of taking the direct product clidean information from the completion of degenerate of Lie groups GL(n, R)×GL(p; R), its infinitesimal ver- fundamental matrices is more difficult to retrieve. A for- sion can be chosen. It is defined by the direct product ward construction for fundamental matrices is not true g`(n) × g`(p) of their Lie algebras, which provide a uni- because these matrices would become identically null. fied treatment including degenerate cases involving the Thus, it is convenient to consider pairs (resp. triplets) restriction of endomorphisms to the corresponding Lie corresponding to secant lines (resp. planes). algebras g for each classical group G. The description of secant spaces as locally symmet- Furthermore, this approach connects directly with ric spaces requires some considerations about products the extended adjoint representation from Section 4.2. of algebraic or infinitesimal actions. The following sub- These constructions can be adapted to any other classi- sections describe the topological case, the most general cal subgroup H ⊂ G, i.e. a closed subgroup preserving a one, with a basic distinction between the simplest prod- non-degenerate quadratic, bilinear or multilinear form. uct action and the contact action, which incorporates This includes SO(n), SL(n), Sp(n), and similarly for coupling constraints. their Lie algebras. In all cases, a specific G-equivariant decomposition as union of G-orbits for the adjoint representation in the Lie algebra g is obtained. 4.3.2 Product action

Let Homeom(Z) denote the group of homeomorphisms 4.3.3 The contact action (bijective and bicontinuous maps) of Z such that for any continuous map f : X → Y between two topological In the differentiable framework, contact equivalence pre- spaces the A-action is defined by the direct or Cartesian serves the graph Γf of any transformation f : X → group Homeom(X)×Homeom(Y ). It acts on the space Y . Moreover, it introduces a natural coupling between of maps as (kfg−1)(x) = k(f(g−1)x) for any x ∈ X actions on source X and target Y spaces for f, and and for any (g, k) ∈ Homeom(X) × Homeom(Y ). Ob- consequently for Cr-equivalences acting simultaneously viously, R-action on the source space X, and L-action on X × Y . A meaningful example for our purposes on the target space Y are two particular cases of the corresponds to linear maps of central projections of a above A-action. bounded region of the image plane. From a topological viewpoint, it is convenient to Topological invariants for the transformation f : restrict the above topological action to the differen- X → X are linked to the fixed locus of f [6]. This tiable case using Diffeom(Z) instead of Homeom(Z) locus is the intersection product Γf · ∆X , which cor- for Z = X or Z = Y . Homeomorphisms and diffeomor- responds to a weighted sum of pairs (x, y) ∈ X × X phisms have been used in deformation problems [19]. such that: a) y = f(x), i.e. they belong to the graph Since a global description of Diffeom(Z) is complex Γf ; b) y = x, i.e. they belong to the diagonal ∆X := and we are interested in the local aspects, only local {(x, y) ∈ X × X | x = y}). Therefore, transformation diffeomorphisms Diffeomz(Z) are considered, i.e. dif- represented by a group action can be computed from feomorphisms preserving a point z ∈ Z. pairs of corresponding points. The linearization of the differentiable A-action gives This argument can be extended to k-tuples of points the A := R × L-action on the differentiable map dxf : related by transformations acting on source and target TxX → TyY (locally represented by the Jacobian map spaces linked by the projection map. So, instead of re- Jf ) at x ∈ X with y = f(x). If p = dim(Y ), then moving 3D points generated from the pairs, which re- −1 the A-action is given by KJf H for any (H,K) ∈ quires a posterior resolution of the ambiguity, they can GL(n, R) × GL(p; R). When n = p = 3, the action be saved for a low-level interpretation in the ambient of GL can be reduced to the action of the orthogonal space. More formally, this ambiguity can be viewed as group O. Its restriction to the diagonal in O(3) × O(3) the dependence loci for sections of a topological fibra- Complete Endomorphisms in Computer Vision 17 tion that takes values in the space of configurations. specifically, spatial homographies PGL(4). Two mean- 3 This idea is further explained in Section 5. ingful subgroups are the affine group AG := GL(3) n R3 (semidirect group of general linear group and the 3 group of translations), and the euclidean group EG := 3 5 Spatial homographies and projections SO(3) n R (semidirect group of the special orthogonal group and the group of translations). 3 2 This section is devoted to outline a procedure for an- A central projection Pi : P → P with center Ci ticipating changes in camera poses that can include de- is the conjugate of the standard projection (I3O) by generate cases too in order to provide some insight for the left-right action denoted by A := R × L. Here R = the corresponding control devices in autonomous navi- GL(4) (resp. L = GL(3)) acts on right (resp. left) by gation including degenerate cases. The key is to exploit matrix multiplication up to scale. Hence, description of the locally symmetric structure of the space of projec- basic algebraic invariants for projection maps must be tion maps arising from the double action. Our strategy posed using double actions on the space of maps. considers linear actions on source and target spaces described by endomorphisms. These actions can act in a 5.1.1 The left-right equivalence decoupled way (left-right actions) or in a coupled way (contact action). Moreover, they can be described in For any pair (K, H) of regular transformations, the ac- algebraic terms (using Lie groups) or in infinitesimal tion on any central projection matrix Pi corresponding 3 2 −1 terms (using Lie algebras). The second approach allows to P → P is defined by KPiH up to scale. The to incorporate degeneracies in a natural way, and de- double conjugacy class of the (3 × 4)-matrix Pi for a velop “completion strategies” by using exterior powers, regular central projection can be obtained by varying as described in Section 4. (K, H) in GL(3) × GL(4). Eventual degeneracies in endomorphisms are man- The simplest double actions in multiple view geom- aged by simultaneous actions on source and target spaces, etry are pairs of rigid motions or affine transformations which are subsets of the scene and views. Endomor- acting on source P3 and target P2 spaces for the central phisms can be completed using exterior powers to achieve projection Pi with center Ci an equivariant stratification in terms of double conjugacy classes. This results in the aforementioned struc- Proposition 6 Let define the double action by R × L- ture as a locally symmetric variety for the space of or- action of pairs of diffeomorphisms acting on (germs of) bits. maps f : Rn → Rp by double conjugacy, i.e. Af = {kfh−1 | for all (k, h) ∈ Diff(Rn) × Diff(Rp)}. Let (h, k) ∈ g`(n) × g`(p) be a pair of vector fields for 5.1 Decoupled vs coupled actions (K,H), then the tangent space to the left-right orbit Af is given by k ◦ f − df ◦ h. Our strategy comprises two steps. The first step consist Proof By using the underlying topology of the set of in decoupling source spaces (completed by a projective projection matrices [P] ∈ 11 (up to scale), a “small model for the 3D scene, e.g.) from target spaces (com- P perturbation” P of the projection map P around any pleted by a projective model for each view) for maps. ε element x = {(x ) | 1 ≤ i ≤ 3 , 1 ≤ j ≤ j} represent- The second step incorporates a more realistic coupling ij ing a 3 × 4-matrix (up to scale) is given by between both spaces, which is natural since each view is a projection of the scene. In the first step there is a decoupling between left Pε(x) = P(x − εh(P(x)) + εk(P(x)) + ... = and right actions. This decoupling is firstly formulated ∂P in algebraic terms, and next extended to infinitesimal P + ε[k(P(x)) − h(x)] + ε2[...] + ... ∂x terms with complete endomorphisms to include degenerate cases. The main novelty w.r.t. precedent sections Hence, the result is proved by taking limits in : is the management of pairs of orbits involving source and target spaces. This structure can be adapted to P (x) − P(x ∂P any pair of classical groups related to the projective, lim ε = k(P(x)) − h(x) ε→0 ε ∂x affine or euclidean framework. The most regular algebraic transformations of the Remark: This is a particular case of the description source space (a three-dimensional projective space P3 = of the tangent orbit to the A-action for infinitesimally PV 4) are defined by the group of collineations or, more stable maps used in [1, Section 1.6]. 18 Finat, Javier, Delgado-del-Hoyo, Francisco

Additionally, local homeomorphisms arising from in- as a particular case of the left-right or A-equivalence tegrating the above vector fields can be constrained to for smooth map-germs f : Rn → Rp. More specifically, those preserving a quadratic form (euclidean metric, or the A := L × R-action is defined by f 7→ k ◦ f ◦ h−1 n the absolute quadric in the projective version) or an for any pair of diffeomorphisms (k, h) ∈ Diff0(R ) × p invariant bilinear form (such as the symplectic form), Diff0(R ) that preserve a point (the origin). then GL(3) × GL(4) can be replaced by the product of The linearization of the topological A-action (pairs the corresponding affine or euclidean groups. of diffeomorphisms) is an algebraic A-action that can be described in terms of pairs of automorphisms (acting on Corollary 3 The tangent space to the double conju- groups). From an infinitesimal viewpoint (Lie algebra −1 gacy class KPH of any projection matrix P is the g), they can be seen as pairs of endomorphisms acting (3 × 4)-matrix X · P − dP · Y, where · is the ordinary on the supporting vector space of g, to be completed if product of matrices, and (X, Y) is the pair of fields the rank is deficient. This is an immediate consequence (k, h) ∈ g`(3) × g`(4) corresponding to vector fields on n of the description of TeDiff0(R ) = GL(n; R) for the (K, H) ∈ GL(3) × GL(4). source space (similarly for the target space) of any map. Orbits are created by both actions as double conjugacy In our case X = ad(K) = ∇K and Y = ad(H) = classes, but their meaning is not exactly the same: A- ∇H, with a slight abuse of notation. Actually, this dou- action allows true deformations, whereas A-action only ble action is implicit in the original formulation of the allows linear deformations, which can be reinterpreted KLT-algorithm [25]. as perspective transformations, e.g.

5.1.2 Incorporating the singularities 5.3 Incidence varieties and multilinear constraints From the topological viewpoint, spatial homographies define an open dense subset of P15 representing the or- Incidence conditions between linear subspaces are given a b a b dered (4×4) array up to scale of entries of A ∈ PGL(4). by relations, such as Li ⊂ Lj or Li ∩ Lj 6= ∅ where a b The Lie algebra of endomorphisms corresponding to ho- a = dim(L ) and b = dim(L ). All of them are repre- mographies can also be stratified by the rank but only sented by linear equations that can display deficiency a small 6D submanifold of these homographies arise rank conditions. In our case, they can also be expressed from a rigid motion. Thus, information from internal with multilinear constraints posed by the projections and external camera parameters can be recovered with of the geometric objects of the scene (points and lines, the double conjugacy classes (K, H) of upper triangular mainly). Fundamental or essential matrices, trifocal ten- matrices K and euclidean transforms H. sor, or more generally multilinear tensors provide the In order to include degeneracies in the space of pairs most common examples for structural constraints be- of endomorphisms, (k, h) must display some kind of tween corresponding elements, such as points and lines. “infinitesimal stability”. It suffices to prove that the Multilinear constraints are not easy to manage due (3 × 4)-matrix X · P − dP · Y is always regular, i.e. it to the ambiguity in correspondences between objects has 3 as maximal rank. In this way, it fills out the tan- and the need of efficient optimization procedures. In- gent space to the left-right A-orbit of any projection deed, both problems are related since the ambiguity matrix P. This result is true for the regular orbit, but is solved using enough functionally independent con- not necessarily for degeneracies in the Lie algebras rep- straints, which require optimization if the set of equa- resented by degenerate End(R3) × End(R4). However, tions is not minimal. The simplest example is the eight- the construction of complete endomorphisms with the point linear algorithm that estimates the fundamental exterior powers removes the indeterminacy in degenera- matrix[10]. This matrix is a point F in the 7-dimensional 8 cies of the fundamental matrix. variety F of P . A forward approach based in seven points leads to a highly non-linear procedure which is unstable and more difficult to solve. 5.2 Extending double conjugacy actions Optimization procedures in the space of multilinear tensors are required when redundant noisy information Indeterminacies in projection matrices appears when is available; from the algebraic viewpoing, noise can be their corresponding fundamental or essential matrices linked to the smallest or near-zero eigenvalues. Regu- are rank-deficient. In order to avoid them, the left-right lar tensors do not include degenerations in boundary equivalence for matrices (representing endomorphisms components, because they are elements on an open set. or automorphisms of vector spaces) must consider de- Thus, to include degenerate tensors it is necessary to generate cases. In fact, this approach can be considered extend regular analysis which is performed in terms of Complete Endomorphisms in Computer Vision 19 complete objects (extending complete endomorphisms, These manifolds are homogeneous spaces serving as for simplest tensors given by matrices). The rest of the base space of a canonical bundle that extends the this Section remarks how eventual degeneracies can be properties of the simplest case of the previous para- solved by completing the information with appropriate graph. Indeed, they are “classifying spaces” not only for compactifications of incidence varieties. To ease their incidence, but for tangency conditions too. In general, interpretation we adopt a geometric language instead any (eventually degenerated) collections of points and of the more formal language of [24]. lines in the projective space can be viewed as configurations in an appropriate flag manifold B(r1, . . . , rk). 5.3.1 Incidence varieties In this case, a locally symmetric structure in terms of cellular decompositions must be recovered to deal with The simplest incidence variety in a projective space Pn degeneracies. n n fulfills p ∈ `, where ` ∈ Grass1(P ) is a line in P , i.e. an element of the Grassmannian of projective lines. The n n 5.3.2 Equivariant cellular decompositions set of pairs (p, `) ∈ P × Grass1(P ) with p ∈ ` forms the total space E(γ1,n) of the canonical bundle γ1,n of n A cellular decomposition of a N-dimensional variety X Grass1(P ). A direct consequence of this construction splits the variety in a disjoint union of k-dimensional is the following result: k k k k−1 k−1 cells (ei , ∂ei ) ∼top (B , S ) for 1 ≤ k ≤ N. Here S Proposition 7 With the above notation: is the (k−1)-dimensional sphere as the boundary of the k-dimensional ball Bk. Hence, each k-dimensional stra- – Epipolar constraints for corresponding points are el- tum of a locally symmetric variety X is a “replication” ements of the total space E(γ1,3). of a basic k-dimensional cell by some elementary alge- – The set of pencils λ1F1 + λ2F2 connecting two de- braic operation (ordinary or curved reflections, e.g.). It generate fundamental matrices F1, F2 ∈ Sing(F) is can be represented by an oriented graph. 8 the constraint of E(γ ) to F ⊂ . The topological 3 1,8 P For example, the cellular decomposition of P is the closure of this set of pencils is the 1-secant variety 3 complementary of a complete flag p0 ∈ `0 ⊂ π0 ⊂ P . to F1, including cases where F1 and F2 coalesce so 1 They are isomorphic to affine spaces A = `0\p0 (one- that the secant becomes a tangent. 2 dimensional cells), A = π0\`0 (two-dimensional cells), 3 3 The construction of the incidence variety can be and A = P \π0 (three-dimensional cells), correspond- n ing to the fixation of elements at infinity. Furthermore, extended to any kind of Grassmannians Grassk(P ), such cells are invariant by the action of the affine group. and flag manifolds denoted by B(r1, . . . , rk) whose ele- A similar reasoning can be outlined using the reduction ments are finite collections of nested subspaces Lr1 ⊂ r1+r2 r1+...+r n to the euclidean group by fixing the absolute quadric, L ⊂ ...L k = P , where (r1, . . . , rk) is a n n the absolute conic and the circular points [9]. partition of n + 1. If k = 1, then Grass0(P ) = P , and if k = 2, then B(k + 1, n − k) = Grass ( n) with A less trivial example is the cellular decomposition k P 3 3 partition (k + 1, n − k)10 of the Grassmannian Grass1(P ) of lines ` ⊂ P , which In particular, the first non-trivial example of a com- is the relative localization of lines ` w.r.t. a complete 3 plete flag manifold is associated to the partition (1, 1, 1) flag p0 ∈ `0 ⊂ π0 ⊂ P . They are isomorphic to affine of 3. This collection represents a nested sequence of spaces given by the complementary of consecutive Schu- linear subspaces V 1 ⊂ V 2 ⊂ V 3 with projectivization bert cycles σ(a0, a1) or their dual representation (β0, β1) p ∈ ` ⊂ π. The stabilizer subgroup of a generic ele- with 2 ≥ β0 ≥ β1 ≥ 0. ment in B(1, 1, 1) are the 3 × 3-upper triangular ma- The extension to arbitrary Grassmmannians is a cumbersome problem (see [6, Chapter 14] and [7]). How- trices K (up to scale for projective flags) acting at left for the double conjugacy action. This formulation pro- ever, their geometric meaning is simpler: they represent vides a locally symmetric structure for the left action elements with an excedentary intersection for incidence to model changes in camera calibration. For instance, conditions. Their dual interpretation expresses rank de- changes in focal length can be interpreted in terms of ficiency conditions for block matrices whose entries are uni-parameter subgroups of the orbit in the associated subspaces, such as tangent spaces to submanifolds. In flag manifold. particular, for Grass1(3) the dual description (β0, β1) of Schubert cycles can be easily visualized in an oriented 10 The notation B(r1, . . . , rk) for flag manifold is not stan- graph with nodes located at the vertices of an “increas- dard; very often F(r1, . . . , rk) (F for flag in English) or ing stair”. Hence, the cellular decomposition contains D(r1, . . . , rk) (D for drapeau in French) generate confusion with the space of Fundamental Matrices and the set of dis- one 4D cell, one 3D cell, two 2D cells, and one 1D cell, tributions (of vector fields, e.g.). Thus, we have chosen B. which are described as follows: 20 Finat, Javier, Delgado-del-Hoyo, Francisco

– (0, 0) does not impose conditions about lines so it T pEp0 = 0, where E ∈ E is the essential matrix linked represents the whole Grassmannian; to each pair of views. – (1, 0) imposes the constraint ` ∩ `0 6= ∅, which is a More specifically, if {ei}1≤i≤3 represents the canon- hyperplane section of the Grassmannian; ical basis for E3, then the collection of subspaces rep- – (2, 0) is {` ∈ Grass1(3) | ` ⊂ π0}, which imposes resented by e1, e1 ∧ e2 and ei ∧ e2 ∧ e3 are a positive two conditions and it is isomorphic to the dual of oriented flag. This flag can be viewed as the starting P2 flag to interpret essential matrices in terms of rotations – (1, 1) is {` ∈ Grass1(3) | p0 ∈ `}, which imposes and translations. two conditions and it is isomorphic to an ordinary 2 Proposition 8 The local description of a essential ma- projective plane P ; trix as a product of SO(3) and 2 can be formulated – (2, 1) = {` ∈ Grass1(3) | p0 ∈ ` ⊂ π0}, which S imposes three conditions and it is isomorphic to a globally as follows: projective line 1; P 1. The essential manifold E is the unit sphere bundle – (0, 0) is the point representing the fixed line `0 of SτSO(3) of the tangent bundle τSO(3). the flag. 2. The evolution of the tangent bundle is τ(SO(3) × 2 The same decomposition can be formulated in the 2 S ) ' so(3) ⊕ τS . euclidean terms on the underlying vector space V ⊂ Rn+1 by selecting nested subspaces linked to the stan- Proof The former is proved in [23]. The proof of the 2 3 dard reference {e1, . . . , e4 as an oriented fixed flag. Fur- latter is based on the embedding i : S ,→ R , such thermore, this decomposition can be extended to any that there exists an isomorphism n Grass1(P ) adding the vertices linked to n − 1 steps of ∗ 3 τ 2 ⊕ N 2 ' i τ 3 = ε 2 the stair. Two specially relevant cases for secant lines S S R S (including degeneracies) on spaces of bilinear epipo- 1 where N 2 ' ε 2 is the normal bundle to the ordi- lar constraints are Grass ( 8) and Grass ( 15)) that S S 1 P 1 P nary embedding i of S2 and represent the ambient space for planar and volumetric collineations, respectively. 3 1 1 1 ε 2 = ε 2 ⊕ ε 2 ⊕ ε 2 S S S S 5.3.3 Evolving fundamental matrices is the 3-rank trivial bundle on S2. Thus,

0 2 2 1 3 The epipolar constraint (p, p ) ∈ × can be ex- 2 P P τS ⊕ ε 2 ' ε 2 pressed in the projective framework as T pFp0 = 0, S S 1 However any copy of ε 2 cannot be simpliﬁed since where F is the fundamental matrix. But it can also be S τ 2 is not topologically trivial, i.e. it is not isomorphic interpreted as the simplest incidence relations p ∈ `0 or S 1 0 to the Whitney sum of two copies of ε 2 . The quotient as p ∈ ` in each projective plane, which are mutually S dual. by the action of Z2 (corresponding to the lifting of the The set of pairs (p, `) fulﬁlling the epipolar con- antipodal map to the canonical bundle) gives the de- straint are isomorphic to the total space of the canoni- scription of the tangent space to the projective space. 2 2 This result links the information of fundamental matri- cal bundle γ1 on each projective plane P . From a global viewpoint, the evolution in space and time of the epipo- ces with the similar information for essential matrices.

2 lar constraint can be described on the tangent space τP to the projective plane. It is well-known that 6 Practical considerations 1 1 1 1 ε 2 ⊕ τ 2 ' γ ⊕ γ ⊕ γ , (3) P P 2 2 2 1 Despite the fact that most results presented in this pa- where ε 2 is the trivial vector bundle on the projec- P 2 per are posed in a static framework, our main mo- tive plane P , ⊕ is the Whitney sum of vector bundles, 1 2 tivation arise from the indeterminacies appearing at and γ2 is the canonical line bundle on P . This justifies the twist of a fundamental matrix for the secant bootstrapping a mobile calibrated camera in a non- line (uniparametric pencil) connecting a pair of camera structured environment. There are many approaches in locations. the state of the art based on perspective models working in structured or man-made scenes that take advantage 5.3.4 Evolving essential matrices of the support provided by perspective lines, vanishing points, horizon lines, e.g. However, the problem be- In the euclidean space the epipolar constraint for cor- comes more challenging in low-structured environments responding points p, p0 ∈ E2 × E2 is represented by where a “weighted combination” of homographies and Complete Endomorphisms in Computer Vision 21 fundamental matrices provides a practical solution for Our approach retrieve the recent history expressed bootstrapping. in terms of the kinematics of the trajectory as a lifted Generally, planar or structured scenes are better ex- path from the group G of transformations to its tangent plained by a homography, whereas non-planar or un- bundle G × g, which adds the unit vector linked to the structured scenes are better explained by a fundamen- gradient to avoid the indeterminacy in the completion tal matrix. Usually, it is preferable to not assume a of PEnd(V ). In presence of uncertainty due to rank de- specific geometric model but to compute both of them ficiency, the local secant cone to the regular part of the in parallel [20]. Later, the best model is selected using a fundamental or the essential variety can be computed. simple heuristic that avoids the low-parallax cases and Then, the shortest chord that minimizes the angle w.r.t. the well-known twofold ambiguity solution arising when the precedent trajectory is selected to avoid abrupt dis- all points in a planar scene are closer to the camera continuities which make more difficult the control. centers [17]. In low-parallax cases both models are not Actually, this approach is independent of the dimen- well-constrained and the solution yields an initial cor- sion of the underlying vector space V, and thus it could rupted map that should be rejected. Indeed, the quality be extended to support dynamic scenes with additional of the tracking relies heavily in the bootstrapping of the structural homogeneous constraints. It is also indepen- system and, more specifically, in the choice of the most dent of the subgroup so it can also be adapted to other suitable geometric model. classical groups, such as SL(2) or SL(3), which leave Our heuristic approach is developed using weighted invariant the area or volume elements, or even the sym- PL-paths in the secant variety Sec(1, F) to the funda- plectic groups leaving invariant Hamilton-Jacobi mo- 8 tion equations [21]. Hence, motion analysis in dynamic mental variety F that fills out the whole space P = 8 environments can be performed theoretically using the PEnd(V ). Each secant line ` ∈ Gr1(P ) cuts out F generically in three elements, which can coalesce in a same approach. double tangency point plus an ordinary point. The tangent hyperplane at each point of F ∈ F provides the 7 Conclusions and future work homography H related to the fundamental matrix F. Overall, the main challenge consists of retrieving a Degeneracies in fundamental and essential matrix are a valid fundamental matrix when the estimation degener- common issue for hand-held cameras traveling around ates into a 1-rank matrix, instead of the expected 2-rank a non-controlled environment. These singular matri- matrix (when the camera points towards a planar scene, ces can be incorporated to the analysis using results e.g.). The most common approaches are based on the from Classical Algebraic Geometry. This paper intro- introduction of additional sensors, the manual specifica- duces the concept of complete endomorphisms to man- tion of two keyframes in a structured scene from which age degeneracies, providing a geometric reinterpreta- the system must bootstrap [15], or the perturbation of tion in terms of secant varieties. Instead of looking at the camera pose to find a close keyframe from which to configurations of (k +1)-tuples of corresponding points, recover. Each approach has its own drawbacks: our alternative approach focuses on the projective geometry of ambient spaces where tensors and projection 1. Additional sensors or devices improve the robust- maps live. The graph of the adjoint map for End(V ) = ness and safety of the systems (a requirement for T Aut(V ) viewed as the gradient field for matrix space autonomous navigation), but it does not provide a provides the first example of completion. This construc- scientific solution to the problem. Moreover, it is not tion is applied to the fundamental F and the essential always possible to modify the hardware of a system. E varieties by adding limits of tangent directions ap- 2. Manual selection of the initial keyframe pair breaks proached by secants in the ambient space PEnd(V ) the autonomy of the system since it requires the Completions of regular transformations (automor- user interaction to bootstrap. In addition, not all phisms in a Lie group G) in terms of their tangent scenes contain a structured region to compute the spaces (endomorphisms in the Lie algebra g := TeG) are homography. also extended to include the degenerate cases for projec- 3. The perturbation of the camera pose degrades the tion matrices. These completions are always managed in continuity in the image and scene flows since the terms of rank stratifications of spaces of matrices. Ex- direction of the perturbation is randomized. Also, it terior algebra of the underlying vector spaces and its ignores the recent history of the trajectory, leaving projectivization provides a framework to manage this the system in an inconsistent state. Proposition 6 rank stratification. These stratifications can be inter- provides an infinitesimally stable structural result preted geometrically in terms of secant subspaces and to avoid this problem. their adjacent tangent subspaces of any dimension. The 22 Finat, Javier, Delgado-del-Hoyo, Francisco simultaneous completion of the transformations w.r.t. 11. Kileel, J.D.: Algebraic geometry for computer vision. left-right action A on the source and target spaces or Ph.D. thesis, University of California, Berkeley (2017) contact action K on the graph of the projection map 12. Kim, J.S., Kanade, T.: Degeneracy of the linear seventeen-point algorithm for generalized essential ma- 3 2 P → P allows a more robust feedback between image trix. Journal of Mathematical Imaging and Vision 37(1), interpretation and scene reconstruction. 40–48 (2010) The constructions of this paper admit extension con- 13. Kirillov, A.A.: Lectures on the orbit method. In: Grad- cerning several topics, such as analytic presentation of uate Studies in Mathematics, vol. 64. American Mathe- matical Society (2004) limits of tangent spaces (by using appropriate compact- 14. Kirwan, F.C.: Cohomology of quotients in symplectic and ifications, e.g.), intrinsic localization of degeneracy loci algebraic geometry, vol. 31. Princeton University Press in terms of cellular decompositions (inverse image of (1984) the secant map, e.g.), preservation of locally symmetric 15. Klein, G., Murray, D.: Parallel tracking and mapping for small ar workspaces. In: 6th IEEE and ACM Inter- structure of complete spaces (involving the adjoint rep- national Symposium on Mixed and Augmented Reality, resentations), intrinsic formulation of image and scene 2007. ISMAR 2007., pp. 225–234. IEEE (2007) flows (for mobile cameras, e.g.), or a relation between 16. Libermann, P., Marle, C.M.: Symplectic geometry and the motion and structure tensors (in the moment-map analytical mechanics, vol. 35. Springer Science & Busi- ness Media (2012) framework, e.g.). The results can also be adapted to the 17. Longuet-Higgins, H.: The reconstruction of a plane sur- geometry of different kinds of infinitesimal transforma- face from two perspective projections. In: Proc. Royal tions involving arbitrary deformations of the geometric Soc. London Ser. B, Biol, Sci., vol. 227, pp. 399–410 models of views. (1986) 18. Ma, Y., Koˇsecká,J., Sastry, S.: Optimization criteria and These developments have a direct application for geometric algorithms for motion and structure estima- bootstrapping and tracking the transformations of a tion. International Journal of Computer Vision 44(3), camera pose in video sequences recorded in non-restricted 219–249 (2001) environments with arbitrary movements. Therefore, far 19. Miller, M.I., Younes, L.: Group actions, homeomorphisms, and matching: A general framework. Interna- from being just a theoretical curiosity, the infinitesimal tional Journal of Computer Vision 41(1-2), 61–84 (2001) completion of regular transformations provides a natu- 20. Mur-Artal, R., Montiel, J.M.M., Tardos, J.D.: Orb-slam: ral and continuous framework for a unified treatment of a versatile and accurate monocular slam system. IEEE kinematics. In particular, degenerate cases can be man- Transactions on Robotics 31(5), 1147–1163 (2015) 21. Osher, S., Sethian, J.A.: Fronts propagating with aged in motion prediction to increase the robustness of curvature-dependent speed: algorithms based on Visual Odometry algorithms. hamilton-jacobi formulations. Journal of computational physics 79(1), 12–49 (1988) 22. Shafarevich, I.R.: Basic Algebraic Geometry 1, vol. 3. References Berlin: Springer-Verlag (2013) 23. Subbarao, R., Genc, Y., Meer, P.: Robust unambiguous parametrization of the essential manifold. In: IEEE Con- 1. Arnold, V., Varchenko, A., Gusein-Zade, S.: Singularities ference on Computer Vision and Pattern Recognition, of Differentiable Maps: Volume I: The Classification of 2008., pp. 1–8. IEEE (2008) Critical Points Caustics and Wave Fronts. Monographs 24. Thorup, A., Kleiman, S.: Complete bilinear forms. In: Al- in Mathematics. BirkhäuserBoston (1985) gebraic Geometry Sundance 1986, pp. 253–320. Springer 2. Bahturin, Y.A., Shestakov, I.P., Zaicev, M.V.: Gradings (1988) on simple jordan and lie algebras. Journal of Algebra 25. Tomasi, C., Kanade, T.: Shape and motion from image 283(2), 849–868 (2005) streams under orthography: a factorization method. In- 3. Dubbelman, G., Dorst, L., Pijls, H.: Manifold statistics ternational Journal of Computer Vision 9(2), 137–154 for essential matrices. In: Computer Vision ECCV 2012, (1992) pp. 531–544. Springer (2012) 26. Tron, R., Daniilidis, K.: The space of essential matrices 4. Faugeras, O.: Three-dimensional computer vision: a geo- as a riemannian quotient manifold. SIAM Journal on metric viewpoint. The MIT press (1993) Imaging Sciences 10(3), 1416–1445 (2017) 5. Fløystad, G., Kileel, J.D., Ottaviani, G.: The chow form 27. Ye, K., Lim, L.H.: Schubert varieties and distances be- of the essential variety in computer vision. Journal of tween subspaces of different dimensions. SIAM Journal Symbolic Computation 86, 97–119 (2018) on Matrix Analysis and Applications 37(3), 1176–1197 6. Fulton, W.: Intersection theory, vol. 2. Springer Science (2016) & Business Media (2013) 28. Zak, F.L.: Severi varieties. Sbornik: Mathematics 54(1), 7. Griffiths, P., Harris, J.: Principles of algebraic geometry. 113–127 (1986) John Wiley & Sons (2014) 8. Guillemin, V., Sternberg, S.: Symplectic techniques in physics. Cambridge university press (1990) 9. Hartley, R., Zisserman, A.: Multiple view geometry in computer vision. Cambridge university press (2003) 10. Hartley, R.I.: In defense of the eight-point algorithm. IEEE Transactions on Pattern Analysis and Machine In- telligence 19(6), 580–593 (1997)