7 Projective Plane 68 7.1Motivation–Perspectiveprojectioninaffinespace

Geometry of Computer Vision

Translation of Euclid’s Elements by Adelardus Bathensis (1080–1152) http://en.wikipedia.org/wiki/File:Woman teaching geometry.jpg

Tom´aˇsPajdla [email protected]

Monday 2nd May, 2011 Contents

1Notation 1

2 Linear algebra 2 2.1 Change of coordinates induced by the change of basis ...... 3 2.2Dualspaceanddualbasis...... 6 2.3Operationswithmatrices...... 8

3Affinespace 11 3.1Vectors...... 12 3.1.1 Geometricscalars...... 13 3.1.2 Geometricvectors...... 13 3.1.3 Boundvectors...... 14 3.2Linearspace...... 15 3.3Freevectors...... 17 3.4Affinespace...... 18 3.5Coordinatesysteminaffinespace...... 19 3.6Examplesofaffinespaces...... 21 3.6.1 Affine space of solutions of a system of linear equations . . . 21 3.6.2 Smallaffinespaces...... 22 3.7Incidenceaxiomsofaffinespaces...... 24

4 Image coordinate system 27

5 Perspective camera 29 5.1Perspectivecameramodel...... 30 5.2 Computing camera projection matrix from image of six points . . . . 34 5.3Camerapose...... 36 5.4Cameracalibrationandanglebetweenprojectionrays...... 42 5.5Calibratedcameraposecomputation...... 44

6 Homography 53 6.1Homographybetweenimageswiththesamecenter...... 54 6.2Homographybetweenimagesofaplane...... 56 6.2.1 Imageofaplane...... 56 6.2.2 Twoimagesofaplane...... 57 6.3 Spherical image ...... 60 6.4Homography–summary...... 61 6.5Imageofimageofanimage...... 63 6.6Computinghomographyfromimagematches...... 66 6.6.1 Generalperspectivecameras...... 66 6.6.2 Calibratedcameras...... 67

ii T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

7 Projective plane 68 7.1Motivation–perspectiveprojectioninaffinespace...... 69 7.2Realprojectiveplane...... 72 7.2.1 Geometricalmodeloftherealprojectiveplane...... 72 7.2.2 Algebraicmodeloftherealprojectiveplane...... 73 7.2.3 Linesoftherealprojectiveplane...... 74 7.2.4 Idealline...... 76 7.2.5 Homogeneouscoordinates...... 77 7.2.6 Incidenceofpointsandlines...... 78 7.2.7 Joinofpoints...... 79 7.2.8 Meetoflines...... 80 7.3 Line coordinates under homography ...... 81 7.4Vanishingpoints...... 82 7.5Vanishinglineandhorizon...... 82 7.6Incidenceaxiomsofprojectiveplane...... 83

8 Projective space 85 8.1Motivation–unionofidealpointsofallaffineplanes...... 85 8.2Perspectivecamerainprojectivespace...... 87 8.2.1 Parallelprojection...... 88

9 Camera Auto-Calibration and Metrology 92 9.1Internalendexternalcalibration...... 93 9.2Single-viewcameracalibration...... 94 9.3Single-viewmetrology...... 94 9.4Multiple-viewcameracalibration...... 94

Index 96

iii 1Notation

À ... emptyset[1] exp U . . . the set of all subset of set U [1]

U ¢ V . . . Cartesian product of sets U and V [1] Z ... wholenumbers[1] Q . . . rational numbers [2] R ... realnumbers[2]

i . . . imaginary unit [2]

Õ ÔS, , . . . space of geometric scalars

A . . . affine space (space of geometric vectors)

 Õ ÔAo, , . . . space of geometric vectors bound to point o A2 ... realaffineplane A3 . . . three-dimensional real affine space P2 . . . real projective plane

P3 . . . three-dimensional real projective space

Õ ÔV, , . . . space of free vectors x . . . vector A ... matrix Aij ... ij element of A

A Â ... transposeof A  A . . . determinant of A I ... identitymatrix R . . . rotation matrix

. . . Kronecker product of matrices

× β Öb1, b2, b3 . . . basis (an ordered triple of independent generator vectors) β¯ . . . the dual basis to basis β

xβ . . . column matrix of coordinates of x w.r.t. the basis β

Â

¤ x ¤ y . . . Euclidean scalar product of x and y (x y xβ yβ in an orthonormal basis β)

x ¢ y . . . cross (vector) product of x and y



   ¤ x . . . Euclidean norm of x ( x x x) orthogonal vectors . . . mutually perpendicular and all of equal length orthonormal vectors . . . unit orthogonal vectors

P ¥ l ... point P is incident to line l

P  Q . . . line(s) incident to points P and Q

k  l . . . point(s) incident to lines k and l

1 2 Linear algebra

M. C. Escher. Drawing Hands. http://www.worldofescher.com/gallery/A13.html

2 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

We rely on linear algebra [3, 4, 5, 6, 7, 8]. We recommend excellent text books [6, 3] for acquiring basic as well as more advanced elements of the topic. Monograph [4] provides a number of examples and applications and provides a link to numerical a computational aspects of linear algebra. We will next review the most crucial topics needed in this text.

2.1 Change of coordinates induced by the change of basis

Let us discuss the relationship between the coordinates of a vector in a linear space, which is induced by passing from one basis to another. We shall derive the rela- tionship between the coordinates in a three-dimensional linear space, which is the most important when modeling the geometry around us. The formulas for all other

n-dimensional spaces are obtained by passing from 3 to n.

 3 § 1Coordinates Let us consider an ordered basis β b1 b2 b3 of V .Avector 3

x È V is uniquely expressed as a linear combination of the basic vectors by its real

coordinates x, y, z, i.e. x x b1 y b2 z b3, and can be represented as an ordered

  Â

triple of coordinates, as a coordinate vector xβ xyz .

 

½

½ ½

½

§ 2Twobases Having two ordered bases β b1 b2 b3 and β b1 b2 b3

leads to expressing one vector x in two ways as x x b1 y b2 z b3 and x

½ ½ ½ ½ ½ ½

½ x b1 y b2 z b3. The vectors of the basis β can also be expressed in the basis β

using their coordinates. Let us introduce

½ ½

½

b1 a11 b1 a21 b2 a31 b3

½ ½

½

b2 a12 b1 a22 b2 a32 b3 (2.1)

½ ½

½

b3 a13 b1 a23 b2 a33 b3

§ 3 Change of coordinates We will next use the above equations to relate the coordinates of x w.r.t. the basis β to the coordinates of x w.r.t. the basis β ½

x x b1 y b2 z b3

½ ½ ½ ½ ½ ½ ½ ½

½

Ô Õ Ô Õ Ô Õ

x a11 b1 a21 b2 a31 b3 y a12 b1 a22 b2 a32 b3 z a13 b1 a23 b2 a33 b3

½ ½

½

Ô Õ Ô Õ Ô Õ

a11 x a12 y a13 z b1 a21 x a22 y a23 z b2 a31 x a32 y a33 z b3

½ ½ ½ ½ ½

½

x b1 y b2 z b3 (2.2)

Since coordinates are unique, we get

½

x a11 x a12 y a13 z (2.3)

½

y a21 x a22 y a23 z (2.4)

½

z a31 x a32 y a33 z (2.5)

Coordinate vectors xβ and xβ ½ are thus related by the following matrix multiplication

     

x½ a11 a12 a13 x

½

     

y a21 a22 a23 y (2.6)

z ½ a31 a32 a33 z

3 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

which we concisely write as ½ xβ A xβ (2.7)

The columns of matrix A can be viewed as vectors of coordinates of basic vectors,

b1,b2,b3 of β in the basis β ½

 

    

b1 b2 b3 (2.8)

½ ½

A β ½ β β

   and the matrix multiplication can be interpreted as the linear combination of columns of A by coordinates of x w.r.t. β

½

xβ x b1 y b2 z b3 (2.9)

½ ½ β ½ β β

Matrix A plays such an important role here that it deserves its own name. Matrix

A is very often called the change of basis matrix from basis β to β ½ or the transition matrix from basis β to basis β ½ [4, 9] since it can be used to pass from coordinates w.r.t. β to coordinates w.r.t. β ½ by Equation 2.7.

However, literature [5, 10] calls A the change of basis matrix from basis β ½ to β, i.e. it (seemingly illogically) swaps the bases. This choice is motivated by the fact that A canalsobeusedtoobtainthevectorsofβ from vectors of β ½ using

Equation 2.1 as



½ ½ ½ ½ ½

½

b1 b2 b3 a11 b1 a21 b2 a31 b3 a12 b1 a22 b2 a32 b3



½ ½

½

a13 b1 a23 b2 a33 b3 (2.10)

  

a11 a12 a13

½ ½ ½   b1 b2 b3 a21 a22 a23

a31 a32 a33



½ ½ ½ b1 b2 b3 A (2.11)

or equivalently

 

¡1

½ ½ ½ b1 b2 b3 b1 b2 b3 A (2.12) where the multiplication of a row of column vectors by a matrix from the right in Equation 2.11 has the meaning given by Equation 2.10 above. Yet another variation 1 of the naming appeared in [7, 8] where A ¡ was named the change of basis matrix from basis β to β ½ . We have to conclude that the meaning associated with the change of basis matrix varies in the literature and hence we will avoid this confusing name and talk about

A as about the matrix transforming coordinates of a vector from basis β to basis β ½ .

There is the following interesting variation of Equation 2.11

   

b1½ b1

¡Â

½

    b2 A b2 (2.13)

b3½ b3

4 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

where the basic vectors of β and β ½ are understood as elements of column vectors.

For instance, vector b1½ is obtained as

½

b1 a¯11 b1 a¯12 b2 a¯13 b3 (2.14)

¡Â × where Öa¯1, a¯2, a¯3 is the first row of A .

§ 4Example We demonstrate the relationship between vectors and bases on a concrete example. Consider two bases α and β represented by coordinate vectors,

which we write into matrices 

110

 

 

α a1 a2 a3 011 (2.15)

001

  

111 



β b1 b2 b3 001 , (2.16) 011

and a vector x with coordinates w.r.t. the basis α 

 1

 

xα 1 (2.17) 1 We see that basic vectors of α can be obtained as the following linear combinations of basic vectors of β

a1 1 b1 0 b2 0 b3 (2.18)

¡ a2 1 b1 1 b2 1 b3 (2.19)

¡ a3 1 b1 0 b2 1 b3 (2.20)

(2.21)

or equivalently

 

¡

 

111

  



¡

a1 a2 a3 b1 b2 b3 0 10 b1 b2 b3 A (2.22) 011

Coordinates of x w.r.t. β are hence obtained as

 

11¡1

 

¡ xβ A xα, A 0 10 (2.23)

011

     

1 11¡1 1

     

¡ ¡ 1 0 10 1 (2.24) 2 011 1 We see that

α β A (2.25)

     

110 111 11¡1

     

011 001 0 ¡10 (2.26) 001 011 011

5 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

2.2 Dual space and dual basis

Let us start with a three-dimensional linear space L over scalars S and consider the ¯ set L of all linear functions f : L  S, i.e. the functions on L for which the following

holds true

Õ Ô Õ Ô Õ

f Ôax by af x bf y (2.27) È for all a, b È S and all x, y L.

¯ ¯ ¯ ¯

¢  È Let us next define the addition ¯ : L L L of linear functions f,g L and

¯ ¯ ¯ ¯

¢  È È the multiplication ¯¤ : L L L of a linear function f L by a scalar a S such

that

ÕÔ Õ Ô Õ Ô Õ

Ôf ¯ g x f x g x (2.28)

¤ ÕÔ Õ Ô Õ Ôa¯f x af x (2.29)

¯

È Ô ¤Õ Ô Õ holds true for all a È S and for all x L. One can verify that L, ¯ ,¯ over S, , is itself a linear space [3, 6, 5]. The linear space L¯ is derived from and naturally connected to the linear space L and hence deserves a special name. Linear space L¯ is called [3] the dual (linear) space to L.

¯ ¯ × Now, consider a basis β Öb1, b2, b3 of L. We will construct a basis β of L,ina ¯ certain natural and useful way. Let us take three linear functions f1,f2,f3 È L such that

Õ Ô Õ Ô Õ f1 Ôb1 1 f1 b2 0 f1 b3 0

Õ Ô Õ Ô Õ f2 Ôb1 0 f2 b2 1 f2 b3 0 (2.30)

Õ Ô Õ Ô Õ f3 Ôb1 0 f3 b2 0 f3 b3 1 where 0 and 1 are the zero and the unit element of S respectively. First of all, one has to verify [3] that such and assignment is possible with linear functions over L. Secondly one can show [3] that functions f1,f2,f3 are determined by this assignment uniquely on all vectors of L. Finally, one can observe [3] that the triple

¯ ¯ × β Öf1,f2,f3 forms an (ordered) basis of L. The basis β is called the dual basis of

L¯ to the basis β of L.

§ 5 Valuation of linear functions Consider a vector x È L with coordinates xβ

 ¯

× Ö × È Öx1,x2,x3 w.r.t. a basis β b1, b2, b3 and a linear function h L with coordinates

 ¯

Ö × Ö × Ô Õ È hβ¯ h1,h2,h3 w.r.t. the dual basis β f1, f2, f3 .Thevalueh x S is

6 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

obtained from the coordinates xβ and hβ¯ as

Õ Ô Õ hÔx h x1 b1 x2 b2 x3 b3 (2.31)

Ô ÕÔ Õ h1 f1 h2 f2 h3 f3 x1 b1 x2 b2 x3 b3 (2.32)

Ô Õ Ô Õ Ô Õ h1 f1 b1 x1 h1 f1 b2 x2 h1 f1 b3 x3

Ô Õ Ô Õ Ô Õ h2 f2 b1 x1 h2 f2 b2 x2 h2 f2 b3 x3 (2.33)

Ô Õ Ô Õ Ô Õ

h3 f3 b1 x1 h3 f3 b2 x2 h3 f3 b3 x3

  



Õ Ô Õ Ô Õ

f1 Ôb1 f1 b2 f1 b3 x1

 





Ô Õ Ô Õ Ô Õ  h1 h2 h3  f2 b1 f2 b2 f2 b3 x2 (2.34)

3

Õ Ô Õ Ô Õ

f3 Ôb1 f3 b2 f3 b3 x

   

100 x1

 

   

h1 h2 h3 010 x2 (2.35)

001 x3

 

x1

 

 

h1,h2,h3 x2 (2.36) x3

 hβ¯ xβ (2.37)

¯ È The value of h È L on x L is be obtained by matrix multiplication of the transpose of hβ¯ and xβ. Notice that the middle matrix on the right of Equation 2.33 evaluates into the identity. This is the consequence of using the pair of a basis and its dual basis. The formula 2.37 can be generalized to the situation when bases are not dual by evaluating the middle matrix accordingly, i.e. in general

Â

Õ Ö Ô Õ× h Ôx hβ¯ fi bj xβ (2.38)

¯ ¯

Ô Õ× where matrix Öfi bj is constructed from the respective bases β, β of L and L.

§ 6 Changing basis in a linear space and in its dual Let us now look at what

happens with coordinates of vectors of L¯ when passing from the dual basis β¯ to the ½ dual basis β¯ ½ induced by passing form basis β to basis β in L. Consider vector

¯

½

È È x L and a linear function h L and their coordinates xβ, xβ and hβ¯, hβ¯ ½ w.r.t. the respective bases. Introduce further matrix A transforming coordinates of vectors

in L as ½ xβ A xβ (2.39)

when passing from β to β ½ . ½

Basis β¯ is the dual basis to β and basis β¯ ½ is the dual basis to β and therefore Â

Â

½

Ô Õ hβ¯ xβ h x hβ¯ ½ xβ (2.40)

¯ È

for all x È L and all h L. Hence Â

Â

hβ¯ xβ hβ¯ ½ A xβ (2.41)

for all x È L and therefore Â

Â

hβ¯ hβ¯ ½ A (2.42)

7 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected]) or equivalently

¡Â

hβ¯ ½ A hβ¯ (2.43) Notice that the derived relationship is formally identical with Equation 2.13.

§ 7 Remark on higher dimensions We have introduced the dual space and the dual basis in a three-dimensional liner space. The definition of dual space is exactly the same for any linear space. The definition of dual basis is the same for all finite- dimensional linear spaces [3]. For any n-dimensional linear space L and its basis β, we get the corresponding n-dimensional dual space L¯ with the dual basis β¯.

2.3 Operations with matrices

Matrices are powerful tool which can be used in many ways. Here we review a few useful rules for matrix manipulation. The rules are often studied in multi-linear algebra and tensor calculus. We shall not review the theory of multi-linear algebra but will look at the rules from a phenomenological point of view. We will take them

as useful identities making an effective manipulation and concise notation possible. ¢

§ 8 Kronecker product Let A be a k ¢ l matrix and B be a m n matrix

 

a11 a12 ¤¤¤ a1l

a21 a22 ¤¤¤ a2l

k ¢l m¢n

R È R È

A . . . . and B (2.44)  . . .. . 

ak1 ak2 ¤¤¤ akl

then km ¢ ln matrix

 

a11 B a12 B ¤¤¤ a1l B

a21 B a22 B ¤¤¤ a2l B

C A B . . . . (2.45)   . . .. .

ak1 B ak2 B ¤¤¤ akl B

is the matrix of the Kronecker product of matrices A, B (in this order).

Õ Ô Õ

Notice that this product is associative, i.e. ÔA B C A B C , but it



is not commutative, i.e. A B B A in general. There holds a useful identity

  Â

Õ ÔA B A B .

§ 9 Matrix vectorization Let A be an m ¢ n matrix

 

a11 a12 ¤¤¤ a1n

a21 a22 ¤¤¤ a2n

m¢n

È R

A . . . . (2.46)   . . .. .

am1 am2 ¤¤¤ amn

8

T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

m¢n mn

Õ R  R ¢ We define v Ô. : operator which reshapes an m n matrix A into a

mn ¢ 1 matrix (i.e. into a vector) by stacking columns of A one above another

 

a11

a21

.

.

am1

a12

a22

Ô Õ

v A . (2.47)

.

am2

a1n

a2n

.  . 

amn

 Â

Õ Ô Õ Ô Õ

Let us study the relationship between v ÔA and v A . We see that vector v A Õ

contains only permuted elements of v ÔA and therefore we can construct permutation

 Â

n n ¢m

matrices m¢ and such that

Â

Õ Â Ô Õ Ô n

v A m¢ v A

Â

Õ Â Ô Õ Ô m v A n ¢ v A

We see that there holds

Â

Â Â Ô Õ Â Ô Õ Ô Õ

m m¢n n ¢m n ¢ v A v A v A (2.48)

for every m ¢ n matrix A. Hence

¡1 Â Â m

n ¢ m (2.49) ¢n

Consider a permutation Â. It has exactly one unit element in each row and each column. Consider the i-throwwith1inthej-th column. This row sends the j-th element of an input vector to the i-th element of the output vector. The i-the

column of the transpose of  has 1 in the j-th row. It is the only non-zero element Â

in that row and therefore the j-th row of  sends the i-th element of an input

  vector to the j-th element of the output vector. We see that  is the inverse of ,

i.e. permutation matrices are orthogonal. We see that

¡1 Â Â

 n (2.50)

m m¢ ¢n

and hence conclude

   m

n ¢ (2.51)

m¢n

§ 10 From matrix equations to linear systems Kronecker product of matrices and matrix vectorization can be used to manipulate matrix equations in order to get systems of linear equations. Consider, for instance, matrix equation

AXB C (2.52)

9

T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

m¢k k ¢l l ¢n m¢n

R È R È R È R with matrices A È , X , B , C . It can be verified by direct

computation that

Â

Õ Ô Õ Ô Õ v ÔAXB B A v X (2.53)

This is useful when matrices A, B and C are known and we use Equation 2.52 to compute X. Notice that matrix Equation 2.52 is actually mnscalar linear equations in klunknown elements of X. Therefore, we should be able to write it in the standard

form, e.g., as

Õ Ô Õ

M v ÔX v C (2.54)

ÔmnÕ¢ÔklÕ Â

È R with M . We can use Equation 2.52 to get M B A which yields the

linear system

Õ Ô Õ

v ÔAXB v C (2.55)

Â

Õ Ô Õ Ô Õ

ÔB A v X v C (2.56) Õ for unknown v ÔX , which is in the standard form. Let us next consider two variations of Equation 2.52. First consider matrix equation

AXB X (2.57)

Here unknowns X appear on both sides but we are still getting a linear system of

the form

Â

¡ Õ Ô Õ

ÔB A I v X 0 (2.58)

Õ¢Ô Õ

where I is the Ômn kl identity matrix. Â

Next, we add yet another constraints: X X, i.e. matrix X is symmetric, to get

Â

AXB X and X X (2.59)

whichcanberewritteninthevectorizedformas

Â

¡ Õ Ô Õ ÔÂ ¡ Õ Ô Õ Ô n B A I v X 0 and m¢ I v X 0 (2.60)

and combine it into a single linear system

¡ Â n

m¢ I Õ

v ÔX 0 (2.61)

 ¡ B A I

10 3Affinespace

Jacopo de’Barbari. Ritratto di Fra Luca Pacoli (1495) http://upload.wikimedia.org/wikipedia/commons/2/2a/Pacioli.jpg

11 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

Let us study affine space, an important structure underlying geometry and its algebraic representation. Affine space is closely connected to linear space. The

connection is so intimate that they are sometimes not even distinguished. Consider,

R È R

for instance, function f : R  with non-zero a, b

Õ f Ôx ax b (3.1)

R It is often called “linear” but it is not a linear function [5,6,4]sinceforeveryα È

there holds

Õ  Ô Õ Ô Õ f Ôαx αax b α ax b αf x (3.2)

In fact, f is an affine function, which becomes linear function only for b 0. In geometry, we need to be very precise and we have to clearly distinguish affine from linear. Let us therefore first review the very basics of linear spaces, and in particular their relationship to geometry, and then move to the notion of affine space.

3.1 Vectors

Let us start with geometric vectors and study rules of their manipulation. Figure 3.1(a) shows space of points P , which we live in and intuitively under- stand. We know what is an oriented line segment, which we also call a marked ruler (or just a ruler). A marked ruler is oriented from its origin towards its end, which is actually a mark (represented by an arrow) on a thought infinite ruler, Figure 3.1(b). We assume that we are able to align the ruler with any pair of points x, y,sothat the ruler begins in x and a mark is made at the point y. We also know how to align a marked ruler with any pair of distinct points u, v such that the ruler begins in u and aligns with the line connecting u and v in the direction towards point v.The mark on so aligned ruler determines point z, which is collinear with points u, v.We know how to translate, Figure 3.1(c), a ruler in this space. To define geometric vectors, we need to define geometric scalars.

z v u y

x

(a) (b) (c)

Figure 3.1: (a) Space consists of points. Rulers (marked oriented line segments) can be aligned (b) and translated (c) and thus used to transfer, but not measure, distances.

12 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

a a a 1 ¡1 a b ¡1 a ¡1 b

a b b

1 a 1 a a b b a b

a b ab (a) (b)

Figure 3.2: Scalars are represented by oriented rulers. They can be added (a) and multiplied (b) purely geometrically by translating and aligning rulers. Notice that we need to single out unit scalar “1” to perform geometric multiplication.

3.1.1 Geometric scalars Geometric scalars S are horizontal oriented rulers. The ruler, which has its origin identical with its end is called 0. Geometric scalars are equipped with two geometric

operations, addition a b and multiplication ab, defined for every two elements

a, b È S.

Figure 3.2(a) shows addition a b. We translate ruler b toalignoriginofb with

the end of a and obtain ruler a b.

Figure 3.2(b) shows multiplication ab. To perform multiplication, we choose

Ô¡ Õ a unit ruler “1” and construct its additive inverse ¡1using1 1 0. This introduces orientation to scalars. Scalars aiming to the same side as 1 are positive

and scalars aiming to the same side as ¡1arenegative. Scalar 0 is neither positive,

¡ ¡ ¡ nor negative. Next we define multiplication by ¡1 such that 1 a a, i.e. 1 times a equals the additive inverse of a. Finally, we define multiplication of non- negative (i.e. positive and zero) rulers a, b as follows. We align a with 1 such that origins of 1 and a coincide and they contain an acute non-zero angle. We align b with 1 and construct ruler ab by translation as shown in the figure. All constructions were purely geometrical and were performed with real rulers. We can verify that so defined addition and multiplication of geometric scalars satisfy all rules of addition and multiplication of real numbers. Geometric scalars form a

field [10, 11] w.r.t. to a b and ab.

3.1.2 Geometric vectors Õ

Ordered pairs of points, such as Ôx, y in Figure 3.3(a), are called geometric vectors

   Õ and denoted as xy, i.e. xy Ôx, y .Symbolxy is often replaced by a simpler one, e.g. by a. The set of all geometric vectors is denoted by A.

13 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

z z y

x  y x y x

o x x o (a) (b) (c) (d) (e) z z y y

y  x y y x o x o

(f) (g) (h) (i) (j)

Õ Ô Õ Figure 3.3: Bound vectors are (ordered) pairs of points Ôo, x , i.e. arrows x o, x . Addition of the bound vectors x, y is realized by parallel transport (using a ruler). We see that the result is the same whether we add x to y or y to x. The addition is commutative.

3.1.3 Bound vectors Õ Let us now choose one point o and consider all pairs Ôo, x ,wherex can be any point, Figure 3.3(a). We obtain a subset Ao of A,whichwecallgeometric vectors

bound to o,orjustbound vectors when it is clear to which point they are bound. We

Õ Ô Õ will write x Ôo, x . Figure 3.3(f) shows another bound vector y. The pair o, o is

special. It will be called the zero bound vector and denoted by 0. We will introduce 

two operations , with bound vectors.

¢  First we define addition of bound vectors  : Ao Ao Ao. Let us add vector x to y as shown on Figure 3.3(b). We take a ruler and align it with x, Figure 3.3(c). Then we translate the ruler to align its begin with point y, Figure 3.3(d). The

end of the ruler determines point z. We define a new bound vector, which we

Ô Õ denote x  y, as the pair o, z , Figure 3.3(e). Figures 3.3(f-j) demonstrate that

addition gives the same result when we exchange (commute) vectors x and y, i.e.

½



x  y y x. We notice that for every point x, there is exactly one point x such

½ ½

½

Ô ÕÔ Õ 

that o, x o, x Õ Ôo, o , i.e. x x 0. Bound vector x is the inverse to x and  is denoted as ¡x. Bound vectors are invertible w.r.t. operation . Finally, we see

ÕÔ Õ Ô Õ 

that Ôo, x o, o o, x , i.e. x 0 x.Vector0istheidentity element of the  operation . Clearly, operation behaves exactly as addition of scalars – it is a commutative group [10, 11].

Secondly, we define the multiplication of bound vector by a geometric scalar

¢   : S Ao Ao,whereS are geometric scalars and Ao are bound vectors. Oper-

ation  is a mapping which takes a geometric scalar (a ruler) and a bound vector

and delivers another bound vector. Õ Figure 3.4 shows that to multiply a bound vector x Ôo, x by a geometric scalar a, we consider the ruler b whose origin can be aligned with o and end with x.We multiply scalars a and b to obtain scalar ab and align ab with x such that the origin

14 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

of ab coincides with o and ab extends along the line passing through x.Weobtain

 Ô Õ

end point y of so placed ab and construct the resulting vector y a x o, y .  We notice that addition  and multiplication of horizontal bound vectors coincides exactly with addition and multiplication of scalars.

3.2 Linear space

We can verify that for every two geometric scalars a, b È S and every three bound

vectors x, y,z È Ao with their respective operations, there holds the following eight

rules

 Õ Ô  Õ

x Ôy z x y z (3.3)

 x  y y x (3.4)

x  0 x (3.5)

x ¡x 0 (3.6)

1  x x (3.7)

Õ Ô  Õ

Ôab x a b x (3.8)

 Õ Ô  ÕÔ  Õ

a Ôx y a x a y (3.9)

Õ Ô  ÕÔ  Õ Ôa b x a x b x (3.10)

These rules are known as axioms of linear space [5, 6, 3]. Bound vectors are one particular model of linear space. There are many other very useful models, e.g. n-tuples of real or rational numbers for any natural n, polynomials, series of real numbers and real functions. We will give some particularly simple examples useful in geometry later. The next concept we will introduce are coordinates of bound vectors. To illustrate this concept, we will work in a plane. Figure 3.5 shows two non-collinear bound vectors b1, b2,whichwecallbasis, and another bound vector x. We see that there is

 only one way how to choose scalars x1 and x2 such that vectors x1  b1 and x2 b2 add to x, i.e.

   x x1 b1 x2 b2 (3.11)

× Scalars x1, x2 are coordinates of x in (ordered) basis Öb1, b2 .

y  y a x x x o b a ab

Figure 3.4: Multiplication of the bound vector x by a geometric scalar a is realized by aligning rulers to vectors and multiplication of geometric scalars.

15 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

x2  b2 x

x

b2 x1  b1 o b1

Figure 3.5: Coordinates are the unique scalars that combine independent basic vec-

tors b1, b2 into x. ½ ½

x2  b2 x ½ b½ x 2 x2  b2 x b2½ ½ 2 1  b x ½ x b1

o ½ b ½ 1 b1 b2 x1  b1 o b1 b1

½ ×

Figure 3.6: Two sets of bound vectors Ao and Ao . Coordinates of x w.r.t. Öb1, b2

½ ½

½ × are equal to coordinates of x w.r.t. Öb1, b2 .

16 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

x x y y o o o

(a) (b)

Figure 3.7: The set A of all geometric vectors (a) can be partitioned into subsets

o,x Õ Ôo,y Õ which are called free vectors. Two free vectors AÔ and A , i.e. subsets of A,areshownin(b).

3.3 Free vectors

We can choose any point from A to construct bound vectors and all such choices will lead to the same manipulation of bound vector and to the same axioms of linear space. Figure 3.6 shows two such choices for points o and o½ .

Ô Õ Ô Õ Ô Õ

We take bound vectors b1 o, b1 , b2 o, b2 , x o, x at o and construct

½ ½ ½ ½ ½ ½ ½ ½ ½ ½

½

Õ Ô Õ Ô Õ

bound vectors b1 Ôo ,b1 , b2 o ,b2 , x o ,x at o by translating x to x , b1 ½

½ ×

to b1 and b2 to b2 by the same translation. Coordinates of x w.r.t. Öb1, b2 are equal

½ ½

½ × to coordinates of x w.r.t. Öb1, b2 . This interesting property allows us to construct another model of linear space, which plays an important role in geometry. Let us now consider the set of all geometric vectors A. Figure 3.7(a) shows an example of a few points and a few geometric vectors. Let us partition the set A of

geometric vectors into disjoint subsets A such that we choose one bound vector

Ôo,x Õ Õ

Ôo, x and put to A all geometric vectors that can be obtained by a translation

Ôo,x Õ Õ

of Ôo, x . Figure 3.7(b) shows two such partitions A , A . It is clear that

Ôo,x Õ Ôo,y Õ

½

 À 

A A ½ for x x and that every geometric vector is in some (and in

Ôo,x Õ Ôo,x Õ

exactly one) subset A .

Ôo,x Õ

½ ½

Õ Ô Õ Ô ½ ½

o,x Õ Ôo ,x Õ

Two geometric vectors o, x and o ,x form two subsets AÔ , A which

½ ½

Õ Ô Õ are equal if and only if Ôo ,x is related by a translation to o, x .

“To be related by a translation” is an equivalence relation [1]. All geometric

Ô Õ

o,x Õ vectors in AÔ are equivalent to o, x . There are as many sets of the partition as there are bound vectors at a point.

We can define the partition by geometric vectors bound to any point o because if ½

we choose another point o½ , then for every point x, there is exactly one point x such

½ ½

Ô Õ Ô that o, x can be translated to o ,x Õ.

We denote the set of subsets A by V . Let us see that we can equip set V

Ôo,x Õ

¢   ¢  with a meaningful addition : V V V and a multiplication : S V V by geometric scalars S such that it will become a model of linear space. Elements of V

will be called free vectors.

o,x Õ Ôo,y Õ Ôo,x ÕÔo,y Õ

We define the sum of x AÔ and y A , i.e. z x y is the set A .



o,x Õ

Multiplication of x AÔ by geometrical scalar a is defined analogically, i.e. a x



o,x Õ equals the set AaÔ . We see that the result of and does not depend on the choice of o. We have constructed the linear space V of free vectors.

17 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

x x o o

z z ½ y x½ q y y q

p p Õ

Figure 3.8: Free vector A is added to free vector A by translating Ôo, x to

Ôo,x Õ Ôp,y Õ

½ ½ ½ ½

Õ Ô Õ Ô Õ Ô Õ Ô ÕÔ Õ

Ôq,x and p, y to q,y , adding bound vectors q,z q,x q,y

and setting A A A

Ôo,x Õ Ôp,y Õ Ôq,z Õ

y v z

u w

x

t

Figure 3.9: Free vectors u, v and w defined by three points x, y and z satisfy triangle

identity u v w.

3.4 Affine space

We saw that bound vectors and free vectors were (models of) linear space. On the other hand, we see that the set of geometric vectors A is not (a model of) linear space because we do not know how to meaningfully add (by translation) geometric vectors which are not bound to the same point. The set of geometric vectors is affine space. Affine space connects points, geometric scalars, bound geometric vectors and free

vectors in a natural way. Õ

Two points x and y, in this order, give one geometric vector Ôx, y , which de- 

termines exactly one free vector v A . We define function ϕ: A V ,which

Ôx,y Õ

Ô Õ

assigns to two points x, y È P their corresponding free vector ϕ x, y A .

Ôx,y Õ È

Consider a point a È P and a free vector x V . There is exactly one geometric Õ

vector Ôa, x ,witha at the first position, in the free vector x. Therefore, point a

¢ 

and free vector x uniquely define point x. We define function  : P V P ,which

takes a point and a free vector and delivers another point. We write ax x and

Ô Õ require x ϕ a, x .

Consider three points x, y, z È P , Figure 3.9. We can produce three free vectors

Ô Õ Ô Õ Ô Õ

x,y Õ Ôx,y Õ Ôx,z Õ u ϕ x, y AÔ , v ϕ y,z A , w ϕ x, z A . Let us investigate

18

T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected]) Õ

ϕÔx, y

x,y Õ

u AÔ

o,aÕÔo,c Õ w u v AÔ

y Ô y,z Õ

a  z x w

b Õ

Ôx, y Õ

Ôx, z

y,z Õ v AÔ x o

t c

Õ Ô ÕÈ ¢

Figure 3.10: Affine space ÔP, L, ϕ , its geometric vectors x, y A P P and free Õ

vector space L and the canonical assignment of pairs of points Ôx, y to



x,y Õ thefreevectorAÔ .Operations , , combining vectors with vectors,

and , combining points with vectors, are illustrated.

the sum x y. Chose representatives of the free vectors, such that they are all bound

ÕÈ Ô ÕÈ Ô ÕÈ

Ô x,y

y,z Õ Ôx,z Õ to x, i.e. bound vectors x, y A , x, t AÔ and x, z A .Noticethat

we could choose pairs of the original points to represent the first and the third free Õ

vector but we had to introduce a new pair of points, Ôx, t , to represent the second

ÕÔ Õ Ô Õ

free vector. Clearly, there holds Ôx, y x, t x, z . We now see, Figure 3.9, that

Õ Ô Õ

Ôy,z is related to x, t by a translation and therefore

u v A A A A A A w (3.12)

Ôx,y Õ Ôy,z Õ Ôx,y Õ Ôx,tÕ Ôx,y ÕÔx,tÕ Ôx,z Õ

Figure 3.10 shows the operations explained above in Figure 3.9 but realized using the vectors bound to another point o. The above rules are known as axioms of affine space and can be used to define

even more general affine spaces.

Õ Ô Õ

§ 11 Affine space Triple ÔP, L, ϕ with a set of points P , linear space L, ,  (over some field of scalars) and surjective (onto) [1] function ϕ: P ¢ P L,isaffine

space when there holds for every three points x, y, z È P

Õ Ô Õ 

ϕÔx, y ϕ x, z y z (3.13)

Õ Ô Õ Ô Õ

ϕÔx, z ϕ x, y ϕ y,z (3.14)

¢  Ô  Õ È È We define function  : P L P by ϕ a, a x x for all a P and x L. In our geometrical model of A discussed above, function ϕ assigned to pairs of

points x, y their corresponding free vector A .

Ôx,y Õ

3.5Coordinatesysteminaffinespace

We see that function ϕ assigns the same vector from L to many different pairs of points from P . To represent uniquely points by vectors, we select a point o, called

19 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

x

½ ½

Ô Õ x ϕ o ,x

b2½

Ô Õ

x ϕ o, x ½ o b1½

b2

½ ½

Ô Õ o ϕ o, o o

b1

Figure 3.11: Point x is represented in two affine coordinate systems.

Ô Õ origin of affine coordinate system and represent point x È P by x ϕ o, x .Inour

geometric model of A discussed above, we thus represent point x by bound vector Õ

Ôo, x or by point o and free vector A .

Ôo,x Õ To be able to compute with points, we now pass to the representation of points

Õ in A by coordinate vectors. We choose a basis β Ôb1, b2,... in L. That allow us

to represent point x È P by a coordinate vector

 

x1

x2

¤¤¤  xβ  , such that x x1 b1 x2 b2 (3.15) .

.

Õ È

The pair Ôo, β ,whereo P and β a basis of L is called affine coordinate system Õ (often shortly called just coordinate system) of affine space ÔP, L, ϕ .

Let us now study what happens when we choose another point o½ and another

½ ½

½

Õ È

basis β Ôb1, b2,... to represent x P by coordinate vectors, Figure 3.11. Point x

Ô Õ

β β

o,x Õβ is represented twice: by coordinate vector x ϕ o, x AÔ and by coordinate

½ ½

½

½ ½

Ô Õ

½ β

o ,xÕβ

vector xβ ϕ o ,x AÔ . ½

To get the relationship between the coordinate vectors xβ and xβ ½ ,weemploy

the triangle equality

½ ½

Õ Ô Õ Ô Õ

ϕÔo, x ϕ o, o ϕ o ,x (3.16)

½ ½

x o x (3.17) which we can write in basis β as (notice that we replace by to emphasize that

we are adding coordinate vectors)

½ ½

xβ xβ oβ (3.18) and use the matrix A transforming coordinates of vectors from basis β ½ to β to get

the desired relationship

½ ½

xβ A xβ ½ oβ (3.19) ½ Columns of A correspond to coordinate vectors b1½ β,b2β,....Whenpresentedwith a situation in real affine space, we can measure those coordinates by a ruler on a particular representation of L by geometrical vectors bound to, e.g., point o.

20 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

y

2 u 1 x b o 021 p

V Õ

Figure 3.12: Affine space ÔP, V, ϕ of solutions to a linear system is the set of vectors Õ representing points on line p.IncoordinatesystemÔo, u , vector x has coordinate 1. The subspace V of solutions to the accompanied

homogeneous system is the associate linear space. Function ϕ assigns ¡

to two points o, x the vector u y x. Õ

§ 12 Remark on notation We were carefully distinguishing operations Ô , over

Õ Ô Õ  scalars, Ô, over bound vectors, , over free vectors, and function com- bining points and free vectors. This is very correct but rarely used. Often, only the

symbols introduced for geometric scalars are used for all operations, i.e.

  

, , , (3.20)

   , , (3.21)

3.6 Examples of affine spaces

Let us now present a few important examples of affine spaces.

3.6.1 Affine space of solutions of a system of linear equations

When looking at the following system of linear equations in R2

11 2

x (3.22)

¡ ¡ ¡1 1 2

we immediately see that there is an infinite number of solutions. They can be written

as

2 1

È R x τ ,τ (3.23)

0 ¡1

 Â

× Ö¡ × or as a sum of a particular solution Ö2, 0 and the set of solutions v τ 1, 1 of

accompanied homogeneous system

11 0

v (3.24) ¡

¡1 1 0 Õ Figure 3.12 shows that the affine space ÔP, V, ϕ of solutions to the linear sys- tem (3.22) is the set of vectors representing points on line p. The subspace V of

21 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

solutions to the accompanied homogeneous system (3.24) is the linear space associ-

¡ È ated to A by function ϕ, which assigns to two points x, y È A the vector u y x V .

Â

Ö × Ô Õ ¡ If we choose o 2, 0 as the origin in A and vector b ϕ o, x x o as the basis of V , vector x has coordinate 1. We see that, in this example, points of A are actually vectors of R2,whichare the solution to the system (3.22). The vectors of V are the vectors of R2,whichare solutions to the associated homogeneous linear system (3.24).

3.6.2 Small affine spaces Let us now look at small affine spaces. These are structures that satisfy axioms of affine space and have a very small number of points. They do not represent geometry of normal space around us but satisfy axioms, which we have distilled out from the properties of the geometric affine space studied above. They illuminate the

structure of space we live in. Ù § 13 One-point affine space The one-point affine space with P Øx has associ-

Ù Ô Õ ated trivial linear space L Ø0 , which has only zero vector and function ϕ x, x 0. This is trivial affine space.

§ 14 Two-point affine space Let us construct two-point affine space.Iftherewas

Ø Ù an affine space with P x, y , then there would be a linear space L associated

with P such that ϕ would exhaust all vectors in L and the axioms 3.13, 3.14 would

Õ Ô Õ Ô Õ Ô Õ hold. There are four ordered pairs Ôx, x , y,y , x, y , y,x . Due to triangle

Õ Ô Õ Ô Õ Ô Õ Ô Õ

equality ϕÔx, x ϕ y,y 0 (consider, e.g., ϕ x, x ϕ x, x ϕ x, x )and

Õ ¡ Ô Õ Ô Õ Ô Õ Ô Õ ϕÔx, y ϕ y,x (consider ϕ x, y ϕ y,x ϕ x, x ). Now, ϕ must exhaust

Õ Ô Õ all vectors in L. We have already seen that the ϕÔx, x ϕ y,y 0. Assigning

Õ Ô Õ Ô Õ  ϕÔx, y 0 is not possible since ϕ x, y ϕ x, x but x y. The same holds true for

Õ Ô Õ ¡ Ô Õ Ô Õ  È

ϕÔy,x . Therefore, ϕ x, y ϕ y,x ϕ y,x u for some 0 u L. This calls Ù

for a linear space with exactly two vectors. Let us consider linear space L Ø0, 1

with addition defined by 0 0 0, 0 1 1 0 1and1 1 0 and multiplication

¤ ¤ ¤ defined by 0 ¤ 0 0, 0 1 1 0 0and1 1 1 (Check that all axioms of linear

space hols true). We know this linear space also as binary numbers with modulo-two

Õ Ô Õ Ô Õ Ô Õ arithmetic. Define ϕ as ϕÔx, x ϕ y,y 0, ϕ x, y ϕ y,x 1. We see that ϕ is onto. There are no two pairs that would agree on the first component and

would be mapped by ϕ to the same vector. Hence axiom 3.13 holds true. Finally,

Õ Ô Õ Ô Õ Ô Õ Ô Õ Ô Õ

ϕÔx, x ϕ x, x 0 0 0 ϕ x, x , ϕ x, x ϕ x, y 0 1 1 ϕ x, y ,

Õ Ô Õ Ô Õ Ô Õ Ô Õ Ô Õ

ϕÔx, y ϕ y,x 1 1 0 ϕ x, x , ϕ x, y ϕ y,y 1 0 1 ϕ x, y ,

Õ Ô Õ Ô Õ

ϕÔy,y ϕ y,y 0 0 0 ϕ y,y verify that axiom 3.14 holds true as well.

Ø Ù

§ 15 Three-point affine space If there was affine space with P x, y, z ,then Õ

there would be a linear L space associated with P such that ϕÔx, y would exhaust

Õ Ô Õ Ô Õ Ô Õ Ô Õ all vectors in L. There are nine ordered pairs Ôx, x , y,y , z,z , x, y , y,x ,

Õ Ô Õ Ô Õ Ô Õ Ô Õ Ô Õ Ô Õ

Ôy,z , z,y , x, z , z,x . Due to triangle equality ϕ x, x ϕ y,y ϕ z,z 0

Õ ¡ Ô Õ Ô Õ ¡ Ô Õ Ô Õ ¡ Ô Õ and ϕÔx, y ϕ y,x , ϕ y,z ϕ z,y , ϕ x, z ϕ z,x . Clearly function

Ô Õ È ϕ can’t assign a pair of different points to 0. Hence 0  ϕ x, y u L and

Ô Õ È   Ô Õ 0  ϕ x, z v L and v u since x z.Letustryϕ y,z u.Then

¡ Ô Õ¡ Ô Õ Ô Õ 0 u u ϕ x, y ϕ y,z ϕ x, z v implies v 0, which is not possible.

22 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

Õ ¡ Ô Õ Ô Õ Ô Õ ¡ Let us try ϕÔy,z v.Then0 v v ϕ y,z ϕ z,x ϕ y,x u implies

Ô Õ    u 0, which is also not possible. Hence, ϕ y,z w with w 0, w u and w v.

2

Ø Ù Consider the linear space L 0, 1 , i.e. ordered pairs of binary numbers with

coordinate-wise addition and multiplication in modulo-two arithmetic (Verify that

  Â

× Ö × Ö ×

it indeed is a linear space). Now, make x Ö1, 0 , y 0, 1 and z 1, 1 and

Â

Õ ¡ È Ô Õ ¡ Ö × È

ϕÔu, v v u for u, v A.Weseethatϕ u, u u u 0, 0 for every u A,

Â

Õ ¡ ¡Ô ¡ Õ ¡ Ô Õ Ô Õ Ô Õ Ô Õ Ö ×

ϕÔu, v v u u v ϕ v,u .Wegetϕ x, x ϕ y,y ϕ z,z 0, 0 ,

  Â

Õ Ô Õ Ö × Ô Õ Ô Õ Ö × Ô Õ Ô Õ Ö × ϕÔx, y ϕ y,x 1, 1 , ϕ y,z ϕ z,y 1, 0 , ϕ x, z ϕ z,x 0, 1 . We see that ϕ is onto. There are no two pairs that would agree on the first component

and would be mapped by ϕ to the same vector. Hence axiom 3.13 holds true. Finally,

Õ Ô Õ ¡ ¡ ¡ Ô Õ ϕÔu, v ϕ v,w v u w v w u ϕ u, w verifies that axiom 3.14 holds true as well.

§ 16 Four-point affine space To construct the four-point affine space, we extend

Ø Ù

the previous example of the three-point affine space. Let us set P x, y, z, w ,

   Â

Ö × Ö × Ö × Ö × Ô Õ ¡

with x 1, 0 , y 0, 1 , z 1, 1 , w 0, 0 and ϕ u, v v u for

Â

Ô Õ ¡ Ö × È Ô Õ

u, v È P .Weseethatϕ u, u u u 0, 0 for every u, v, t P , ϕ u, v

¡Ô ¡ Õ ¡ Ô Õ Ô Õ Ô Õ ¡ ¡ ¡ Ô Õ v ¡ u u v ϕ v,u and ϕ u, v ϕ v,t v u t v t u ϕ u, t . Clearly ϕ is onto and the triangle equality, axiom 3.14, holds true. We only need to verify axiom 3.13. Let us consider sets of point pairs that map to the same vector.

On pairs from x, y, z,theϕ coincides with three-point affine plane and hence for

Â

Õ Ö ×

them axiom 3.13 holds true. We need to check assignments ϕÔx, w 1, 0 ,

  Â

Õ Ö × Ô Õ Ö × Ô Õ Ö ×

ϕÔy,w 0, 1 , ϕ z,w 1, 1 and ϕ w, w 0, 0 . We see that pre-image

  ¡1

× ÔÖ × Õ ØÔ Õ Ô Õ Ô Õ Ô ÕÙ

of Ö1, 0 , ϕ 1, 0 y,z , z,y , x, w , w, x has no repetition of the first

 ×

component of its elements. The same can be verified for preimages of Ö0, 1 and

Â × Ö1, 1 . Axiom 3.13 holds true.

The above linear space is not the only linear space with four vectors. The set

½ 2

Ø Ù L 0, 1,t,t 1 of polynomials in t with addition modulo polynomial t t

1 forms a linear space over itself. It is a finite field with four elements (Verify

Ø Ù

the axioms). Set P x, y, z, w is realized by x 0, y 1, z t and w

Ô Õ ¡

t 1andϕ u, v v u. Clearly ϕ is onto and axiom 3.14 holds true. Sets

¡1 ¡1

Õ ØÔ Õ Ô Õ Ô Õ Ô ÕÙ Ô Õ ØÔ Õ Ô Õ Ô Õ Ô ÕÙ ϕ Ô0 x, x , y,y , z,z , w, w , ϕ 1 y,z , z,y , x, w , w, x ,etc., demonstrate that axiom 3.13 holds true. The above two examples are interesting because they represent two quite different

affine spaces. Consider that the first example clearly represents a two-dimensional

 Â

× Ö × linear space since vectors Ö0, 1 and 1, 0 are linearly independent. The second

example, on the other hand, represents only one-dimensional space since all vectors

in the space can be generates as multiples of 1 as 0 01 , 1 11, t t 1,

Ô Õ t 1 t 1 1. We also see that the first example of the four-point affine space is actually the smallest affine plane, i.e. the smallest affine space that is based on a two dimensional linear space. It is clear that two dimensional linear space has to have at least four vectors because it has to have the zero vector, two basic vectors and the sum of the two basic vectors must give yet another non-zero vector.

23 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

P4

l1 l2 l3 l4 l5 l6 l6

¥ ¥ P1 ¥ l5

l3 P3

¥ ¥ P2 ¥ l2

l4

¥ ¥

P3 ¥

¥ ¥

P4 ¥ P1 l1 P2

(a) (b) t 0 1 1 1

1 t 1 1

0 1 t t 1

t 0 1

t 1 0 0 (c) (d)

Figure 3.13: Four-point affine space models. (a) Affine plane incidence table. (b) Affine plane geometrical incidence model. (c) Affine line linear algebraic model. (d) Affine plane linear algebraic model.

3.7 Incidence axioms of affine spaces

We saw that affine space with the same number of points can be represented by linear spaces of different dimensions. This suggests that models of affine spaces can be quite different. Indeed there are some very interesting affine spaces which can’t even be represented by a linear space. To get such spaces, however, we have to pass to a different set of axioms that will not be incorporating linear space into the foundations of affine space. We can verify that all affine spaces according to the definition in Section 3.4 satisfy the following rules that we shall adopt as axioms to arrive at more cases of affine spaces. Let us first define a few new concepts and formulate axioms of

synthetic affine spaces and show interesting examples.

P L E Õ P Let P be a set. We define triple Ô , , of points exp P , lines L exp P P and planes E exp P . Points, lines and planes are subsets of .Inparticular,

points are singleton sets, i.e. the sets containing exactly one element. We say that

P È L

point p È is incident [12] with (lies in, is on, belongs to, is contained in) line l ,

 À È E  À

and write p ¥ l when p l , Similarly, p is incident with π when p π .

L È L  À È L

Line l È intersects (meets) line m ,whenl m , line l intersects plane

E  À È E È E  À π È when l π and plane σ intersects π when σ π .Wesay that line l is parallel to line m if they do not intersect and are contained in a plane.

We say that points are collinear (coplanar) if they are contained in a line (plane). P L E Õ We say that Ô , , is syntheticaffinespacewhen the following incidence ax- ioms of affine space [13] hold for the triple.

AS1 Every two distinct points are incident with exactly one line.

AS2 Every three distinct non-collinear points are incident with exactly one plane.

AS3 If p is a point not incident with a line l, then there is a unique line m such that p is incident with m and m is parallel to l.

24 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

Figure 3.14: Moulton plane

AS4 If k, l, m are distinct lines with k parallel to l and l parallel to m,thenk is parallel to m. This new definition encompasses all cases defined above but also some other affine spaces that can’t be represented by linear space with ϕ given by vector subtraction. Figure 3.7 shows four models of the four-point affine plane. In Figure 3.7(a), the

plane is defined by an incidence table. Another model is purely set-theoretical. We

Ù Ø Ù Ø Ù Ø ÙÙ L

set P ØØP1 , P2 , P3 , P4 and line segments connecting points are lines

Ù Ø Ù Ø Ù Ø Ù Ø Ù Ø ÙÙ

ØØP1,P2 , P1,P3 , P1,P4 , P2,P3 , P2,P4 , P3,P4 . There is only one plane

Ø Ù E P P1,P2,P3,P4 . We can verify that axioms AS1–AS4 are satisfied but it is clear that axioms AS2 and AS4 were met trivially in this case. Figure 3.7(b) shows a geometrical model in which the four-point affine plane is represented by a tetrahedron. Vertices of the tetrahedron represent points. The edges of the tetrahedron represent lines. This model is actually just another visual- ization of the corresponding incidence table. Figure 3.7(c) shows four-point affine space represented by the two dimensional linear space over binary numbers. Notice that every point is represented by a vector and ϕ can be defined as the difference of vectors representing points. This is a linear model of affine space. Interestingly, it is possible to drop AS2 and AS4, simplify AS3, and to add another axiom to arrive at a set of axioms of the synthetic affine plane which are satisfied by all affine spaces represented by two dimensional linear spaces with ϕ equal to the difference of vectors representing points AP1 Every two distinct points are contained in exactly one line.

AP2 There are three non-collinear points.

AP3 If p is a point not contained in a line l, then there is a unique line m such that m does not intersect l.

25 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

Synthetic affine planes represent an important class of affine spaces, some of which do not have a linear model. Consider, for instance, the Moulton plane [13]. Set

R2 P

2 2 2 Â

ØÖ × È R  È R 

L x, y a, b, c ,a b 0,

Ô Õ Ô Õ Ô Õ ! Ù

ax t y by c, t y 1fory 0andt y 2fory 0

Â

Â

Ö ¥ È L Ö × x, y × l iff x, y l (3.25)

See Figure 3.14 for examples of a few lines of the Moulton plane. We can check that Moulton plane satisfies axioms of synthetic affine space as well as axioms of synthetic affine plane. Its points are represented by vectors of R2 but lines are not one-dimensional subspaces of R2 and ϕ is not obtained by subtracting vectors representing the points.

26 4 Image coordinate system

Digital image Im is a matrix of pixels. We assume that Im is obtained by measuring intensity of light by sensors (pixels) arranged in a grid, Figure 4.1. We will work with images in two ways. First, we will work with intensity values, which are stored in the memory as a three-dimensional array of bytes indexed by the row index i, the column index j and the color index k, Figure 4(a). Color index attains three values 1, 2, 3, with 1 corresponding to red, 2 corresponding to green and 3 corresponding to blue. In Matlab, image Im is accessed using the row index i, the column index j and color index k as >>Im(i,j,k). The most top left pixel has row as well as column index equal to 1. The red channel of the pixel with row index 2 and column index 3 is accessed as >>Im(2,3,1).

§ 17 Image coordinate system For geometrical computation, we introduce an im- age coordinate system as in Figure 4(b). The origin of the coordinate system is chosen to assign coordinates 1, 1 to the center of the most top left pixel. Horizontal axis b1 goes from left to right. The vertical axis b2 goes from top down. Pixel, which

is accessed as >>Im(i,j,k) is in the image coordinate system represented by the

Â

Ö × vector u j, i . Digital image with H rows and W columns will in Matlab be indexed as >>Im(1:H,1:W,1:3) and >>size(Im) will return [HW3]. In the image

coordinate system the center of the most bottom right pixel will have coordinates

Â × ÖW, H . The image coordinate system coincides with the Matlab coordinate system image, i.e. commands

>> axis ij >> axis equal >> plot(j,i,’.b’) plot a blue dot on the pixel accessed as Im(i,j,k); The image coordinate system is non-standard in two dimensions since it is a left-handed system. The reason for such a unnatural choice is that this system will be next augmented into a three-dimensional right-handed coordinate system in such a way that the b3 axis will be pointing towards the scene.

27 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

Figure 4.1: Image is digitized by a rectangular array of pixels

j 3 b1 3 i b2

2 2

(a) Image Im is a matrix of pixels. (b) The image coordinate system is In Matlab, it is accessed using the defined with horizontal axis b1 and row index i, the column index j and vertical axis b2. The origin of the co- color index k as >>Im(i,j,k).The ordinate system is chosen to to assign most top left pixel has row as well coordinates 1, 1tothemosttopleft as column index equal to 1. The red pixel. Notice that pixel, which is ac- channel of the pixel with row index cessed as >>Im(2,3,1), is represented

2 and column index 3 is accessed as in the image coordinate system by the

Â × >>Im(2,3,1). vector u Ö3, 2 .

Figure 4.2: Image coordinate system.

28 5 Perspective camera

A. Durer. Underweysung der Messung (1525) http://commons.wikimedia.org/wiki/File:Duerer Underweysung der Messung 181.jpg

29 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

(a) (b)

Figure 5.1: Perspective camera (a) is geometrically a point (red) and an image plane (green) (b).

Modern photographic camera, Figure 5.1, is an interesting and advanced device. We shall abstract from all physical and technical details of image formation and will concentrate solely on its geometry. From the point of view of geometry, a perspective camera projects point X from space into an image point x by intersecting the line connecting X with the projection center (red) and a planar image plane (green), Figure 5.1(b).

5.1 Perspective camera model

Let us now develop a mathematical model of the perspective camera. The model will allow us to project space point X into image point x and to find the ray p in space along the which point x has been projected.

§ 18 Camera coordinate system Figure 5.2 shows the geometry of the perspective camera. Point X is projected along ray p from three-dimensional space to point x into two-dimensional image. Point x is obtained as the intersection of ray p with planar image plane π.Rayp is constructed by joining point X with the projection center C. The plane through the projection center C, which is parallel to the image

plane is called principal plane. Õ The image plane is equipped with an image coordinate system (§ 17), Ôo, α ,

Ö × where o is the origin and α b1, b2 is the basis of the image coordinate system. Notice that the basis α is shown as non-orthogonal. We want to develop a general

camera model, which will be applicable even in the situation when image coordinate Õ

system is not rectangular. Point x is represented by vector u in Ôo, α

u

u u b1 v b2 i.e. uα (5.1)

v Õ Three-dimensional space is equipped with a world coordinate system ÔO, δ ,where

× O is the origin and δ Öd1, d2, d3 is a three-dimensional orthonormal basis. Point

Õ X is represented by vector X in ÔO, δ . The camera projection center is represented

Õ by vector C in ÔO, δ .

30 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

d3

O d2

X d1

o p X b1

u b2 C c3 x

b3

x π

C b1

b2 c1

c2

Figure 5.2: Coordinate systems of perspective camera.

31 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

Letusnextdefinethecamera coordinate system. The system will be derived from the image coordinate system to make the construction of coordinates of the

direction vector x of p extremely simple. Õ Camera coordinate system ÔC, β has the origin in the projection center C and

Ö ×

its basis β b1, b2, b3 is constructed by re-using the two basis vectors of α and

 adding the third basic vector b3, which corresponds to vector Co.Weseethat vectors in β form a basis when point C is not in π,whichissatisfiedforevery meaningful perspective camera. Notice also that the camera coordinate system, is three-dimensional.

Image points o and x are in plane π, which is in three-dimensional space, and Õ hence, we can consider them points of the space. Point x is in ÔC, β represented by vector x, which is the direction vector of the projection ray p along which point X has been projected into x. We see that vectors u, x, b3 form a triangle such that

x u b3 (5.2)

u b1 v b2 1 b3 (5.3) 

and therefore 

u

uα

 

xβ x    v . (5.4) ,b ,b × Öb1 2 3 1 1 Notice that basis β has been constructed in a very special way to facilitate con- struction of xβ.Wecanuseu, v directly since β re-uses vectors of α and the third coordinate is always 1 by the construction of b3. Although we do not know exact position of C w.r.t. the image plane, we know that it is not in the plane π and hence a meaningful camera coordinate system constructed this way exists. Notice next that the camera coordinate system is right-handed. This is because when looking at a scene from a point C through the image plane, the image is constructed by intersecting image rays with the image plane, which is in front and hence the vector b3 points towards the scene. We see that vectors of β form a right-handed system. Let us mention that we have used deeper properties of linear and affine spaces. In particular, we were making use of the concept of free vector in the following way. We look at vectors b1, b2 and u as on a free vectors. Therefore, coordinates of the representative of u beginning in o with respect to representatives of b1, b2 beginning in o equal the coordinates of the representative of u beginning in C with respect to representatives of b1, b2 beginning in C.Henceu, v reappear as the first two coordinates of x. For usual consumer cameras, vectorb3 is often much longer than vectorsb1,b2 and often not orthogonal to them. Therefore, basis β is in general neither orthonormal nor orthogonal! This has severe consequences since we can’t measure angles and distances in the space using β, unless we find out what are the lengths of its vectors and what are the angles between them.

§ 19 Perspective projection Point X has been projected along p into x.Sincex Õ is a direction vector of p,pointX can be represented in ÔC, β by

ηx (5.5)

32 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected]) for some real η. Value of η corresponds to scaled depth of X, i.e. the distance of X from the plane passing through C and generated by b1, b2 in units equal to the distance of C from π.Valueη is not known since it “has been lost” in the process of projection1 but will serve us to parametrize the projection ray in order to get coordinates of all possible points in space that could project into x. Let us now relate the coordinates uα, which are measured in the image, to the coordinates X δ, which are measured in the world coordinate system. First consider vectors X , C and x. They are all coplanar and hence

¡ ηx X C (5.6) To pass to coordinates, we will use the camera coordinate system, in which we can write

¡

ηxβ Xβ Cβ (5.7)

uα

η Xβ ¡ Cβ (5.8) 1 Next we shall pass to the coordinates w.r.t. basis δ on the right hand side of Equa- tion 5.8 by introducing a matrix A, which transforms coordinates of a general vector y from basis δ to basis β, i.e.

yβ A yδ (5.9)

We know from linear algebra (§ 3) that such matrix exists

uα

Ô ¡ Õ η A Xδ Cδ

1



uα Xδ

η A I ¡ Cδ

1 1

uα X δ

η P (5.10)

1 1

X δ

ηxβ P (5.11) 1

with 3 ¢ 4 camera (projection) matrix



P A ¡ A Cδ (5.12)

§ 20 Projection equation Equation 5.10 describes the relationship between mea- surement uα in the image and measurement X δ in space. It says that X δ is projected into uα since there exists η such that Equation 5.10 holds. Notice that η multiple of the vector on the left of Equation 5.10 is obtained by a linear mapping represented by matrix P from vector X δ on the right. When computing uα from X δ, we actually eliminate η using the last row of the

(matricidal) equation (5.10)

 

p1Â X

Â

p3 X

uα (5.13)

   p2 X

p3Â X 1It can be recovered when a point X is observed by two cameras with different projection centers.

33 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

where we introduced rows of p1, p2, p3 of P and a 4 ¢ 1 vector X as follows

 

Â

p1

X δ

Â

 

P p and X (5.14) 2 1 p3Â Notice that the projection equation is not linear. It is a rational function of the first order polynomials on elements of X.

§ 21 Projection ray Having an image point uα, we can construct its projection ray

Õ p in space. The ray consists of all points Y that can project to uα.InÔC, β ,the ray is emanating from the zero vector. We parametrize it by real η and express it

Õ

in ÔO, δ by vector Xδ

uα

Yβ η ηxβ 1 1

¡

Xδ η A xβ Cδ (5.15)

Notice that X δ (5.15) can also be obtained for a given η by solving the system of linear equations (5.11) for X δ.

5.2 Computing camera projection matrix from image of six points

Let us now consider the task of finding the P from measurements. We shall consider the situation when we can measure points in space as well as their projection in the

corr

 Â

× " Ö ×

image. Consider a pair of such measurements Öx, y, z u, v . There holds

    x

u

y

 

λ v Q QX (5.16)   z 1

1 ¢ for some real λ,3¢ 4matrixQ and 4 1 coordinate vector X.Noticethatweintro- duced new symbols λ and Q to emphasize that they are determined by Equation 5.16 up to a non-zero scale

Q ξ P (5.17) We will see that this will have further consequences.

Introduce symbols for rows of Q

 

q1Â

Â

  Q q2 (5.18)

q3Â

and rewrite the above matrix equation as Â

λu q1 X (5.19) Â

λv q2 X (5.20) Â

λ q3 X (5.21)

34 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

Eliminate λ from the first two equations using the third one

 Â

Õ

Ôq3 X u q1 X (5.22)

 Â

Õ

Ôq3 X v q2 X (5.23)

 Â

move all to the left hand side and reshape it using x y y x

 Â

Õ

X q1 ¡Ôu X q3 0 (5.24)

 Â

Õ X q2 ¡Ôv X q3 0 (5.25)

Introduce vector of parameters (which are elements of Q)

 

Â

  Â

q q1 q2 q3 (5.26)

and express the above two equations in matrix form

¡ ¡ ¡ xyz10000¡ux uy uz u

q 0

¡ ¡ ¡ 0000xyz1 ¡vx vy vz v

Mq 0 (5.27)

corr

 Â

× " Ö × Every correspondence Öx, y, z u, v bringstworowsintothematrix M (5.27). We need therefore at least 6 correspondences in general position to obtain 11 linearly independent rows in Equation 5.27 to obtain a one-dimensional space of solutions. If Q is a solution to Equation 5.27, then τ Q is also a solution and both determine

the same projection since

Õ Ô Õ Ô Õ Ô Õ

Ô τ Q X τ QX τ λxβ τλ xβ (5.28)

ß Assuming P τ Q leads to λ η τ. We see that we can’t recover P but only its non-zero multiple. Therefore, when solving Equation 5.27, we are looking for

one-dimensional subspace of 3 ¢ 4matricesofrank 3. Such a subspace determines one projection. Also note that the zero matrix does not represent any interesting projection.

Notice that when considering more correspondences, M becomes

 

¡ ¡ ¡

x1 y1 z1 10 0 00¡u1x1 u1y1 u1z1 u1

¡ ¡ ¡

x2 y2 z2 10 0 00¡u2x2 u2y2 u2z2 u2

.

.

M q 0 (5.29)

¡ ¡ ¡

0000x1 y1 z1 1 ¡v1x1 v1y1 v1z1 v1

¡ ¡ ¡

0000x2 y2 z2 1 ¡v2x2 v2y2 v2z2 v2  .  .

which can be more concisely rewritten as

 

  Â

X1 0 ¡u1 X1

  Â

¡

X2 0 u2 X2

.

.

.

M (5.30)

  Â

¡

0 X1 v1 X1

  Â

¡ 2   0 X2 v X2 .

.

Â × with 0 Ö0, 0, 0, 0 .

35 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

5.3 Camera pose

The projection formula 5.10 reveals that the perspective projection depends on ma- trix A and vector Cδ.ThevectorCδ represents the position of the camera projection center w.r.t. the world coordinate system. Columns of matrix A are coordinates of

the basic vectors of δ in the basis β

 A d1β d2β d3β (5.31)

To recover the orientation of the camera, we will introduce the focal length f

(in world units) and replace the product f A by the product of two 3 ¢ 3 matrices K and R

f A KR (5.32) We will see that this seemingly artificial construction is indeed justified. Rotation matrix R determines the orientation of the camera in space and alto- gether with Cδ defines the camera pose.Thecamera calibration matrix matrix K

does not change when moving its camera in the space. Õ

To obtain K and R, we define the camera cartesian coordinate system ÔC, γ with × center (again) in the camera projection center C and with basis γ Öc1,c2,c3 such that c1 k11 b1

c2 k12 b1 k22 b2 (5.33)

c3 k13 b1 k23 b2 1 b3

Parameters kij are determined to make the basis γ orthogonal. Notice that vector c3 is orthogonal to π since it is orthogonal to c1,c2,whichspanπ,byconstruction. Also notice that γ is (in general) not an orthonormal basis since the length of its vectors equals the distance of C from π, i.e. the focal length f in the world units.

Equations 5.33 define matrix K as

 

k11 k12 k13

 

 

K c1β c2β c3β 0 k22 k23 (5.34) 001

The world cartesian coordinate system has basic vectors of unit length. The Õ camera cartesian coordinate system ÔC, γ has basic vectors of length equal to f.

Therefore,

  Â

r1 ßf



1

Â

ß  d1 d2 d3 R  r f (5.35)

γ γ γ f 2 Â

r3 ßf

   for some 3 ¢ 3 orthonormal matrix R with rows r1 , r2 , r3 .

Consider that

 

1

A d1 d2 d3 K d1 d2 d3 KR (5.36) β β β γ γ γ f

36 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

o

b1

u0 b2

c3

b3

f π

C b1

b2 c1

Ô Õ  b1, b2

c2

Figure 5.3: Camera internal parameters are related to the geometry of basis β.

1 We can view the matrices f R and K as coordinate transformation matrices, which transform a general vector y from the coordinates w.r.t. δ to γ andthentoβ, i.e.

1

yβ K yγ KRyδ (5.37) f The basis γ is orthogonal and all basic vectors have the same length, which is equal

2

¤ to f. It follows from the orthogonality of the basis γ that c1 ¤ c1 f , c1 c2 0

2 and c2 ¤ c2 f and hence using Equation 5.33 leads to

¡ k11 b1 f 0

2 2

¤ Õ k11 k22 Ôb1 b2 k12 f 0 (5.38)

2 2 2 2 2 2

 ¡Ô Õ k11 k22 b2 k12 k11 f 0

Let us solve Equations 5.38 for k11, k12 and k22. The first Equation in (5.38) provides k11. Substitution the square of the first equation into the second one an dividing it 2 by k11 provides the second equation of (5.39). To get k22 from the third equation of (5.39), we express k11 from the first equation of (5.38) and k22 from the second

37 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected]) equation of (5.38) and substitute them into the third equation of (5.38). Altogether, we get

¡ k11 b1 f 0

2

 Ô ¤ Õ k12 b1 k22 b1 b2 0 (5.39)

2 2 2 2 2 2

   ¡Ô ¤ Õ Õ¡   k22 Ôb1 b2 b1 b2 f b1 0

Looking at the third equation of (5.38) we see that

2 2 2 

2 f b1 f

k (5.40)

22 2 2 2 2 2 2

   ¡Ô ¤ Õ   ¡  Ô Õ b1 b2 b1 b2 b2 b2 cos b1, b2 f

k22 (5.41)

 Ô Õ b2 sin b1, b2 since γ was constructed to make k22 positive. The second equation of (5.38) now gives

  Ô Õ

b1 ¤ b2 b2 cos b1, b2

¡ ¡ k12 k22 k22 (5.42)

2

   b1 b1

Ô Õ

f cos  b1, b2 ¡ (5.43)

 Ô Õ b1 sin b1, b2

Finally k11 follows from (5.39)

f

k11 (5.44)

 b1

Considering Figure 5.3 and Equation 5.33, we see that the coordinates of the vector u0, corresponding to the principal point, which is the perpendicular projection

of C onto π,areinβ

 

k13

k13

 

u0β k23 , i.e. u0α (5.45) k23 0

 The horizontal pixel size corresponds to b1 .Quantityk11 thus transforms the world units into the horizontal image units. It can be understood as f expressed in the horizontal image units. The angle between the image axes b1,b2 is obtained from

¡ Ô Õ k11 ßk12 tan b1, b2 . The ratio of the lengths of the image axes is determined

ß  Ô Õß by b2 b1 k11 k11 k12 k22. Let us now return to Equation 5.10 and substitute there the above results to

38 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

arrive at the final projection equation

X δ

ηxβ P (5.46)

1

uα

¡ Õ η A ÔXδ Cδ (5.47)

1

uα

Ô ¡ Õ fη f A Xδ Cδ (5.48)

1

uα

Ô ¡ Õ fη KR Xδ Cδ (5.49)

1

uα

Ô ¡ Õ ζ KR Xδ Cδ (5.50)

1



uα Xδ

ζ KR I ¡ Cδ (5.51) 1 1

We have introduced a new parameter ζ fη, which is the depth of X in world  units. We conclude that

1 1 ¡ P f KR f KRCδ (5.52)

Notice that the last row a3Â of A provides f since

       

 Â

a k11 k12 k13 r k11r1 k12r2 k13r3

1 1 1 1

 Â

       

A a2 0 k22 k23 r2 k22r2 k23r3 (5.53) Â

 f f a3 001 r3 r3

1 1

Â

  Ô Õ

and hence a3 f . Therefore P 3, 1:3 f .

1 ¡1 § 22 Coordinate systems generated by applying KR to yδ and R ¡ K to yβ We

have seen that the decomposition of A to K and R introduced the camera cartesian Õ coordinate system ÔC, γ , Figure 5.4(b) 1

yγ R yδ (5.54) f

yβ K yγ (5.55)

There are three more coordinate system to consider when looking at how matrices

1 ¡1 R, K, and their inverses R ¡ , K , apply to vectors yδ and yβ, Figure 5.4. Let us first consider coordinates of a vector y w.r.t. basis δ and apply successively

R and K. Coordinate vector R yδ can be interpreted as coordinates of y w.r.t. a new ×

basis  Öe1,e2,e3 , Figure 5.4(c). Applying further K to y gives the coordinate vec- × tor K y, which can be interpreted as y w.r.t. yet another new basis ν Ön1,n2,n3 . 1 We get from ν to β by using f I

y R yδ (5.56)

yν K y (5.57) 1

yβ I yν (5.58) f

39 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

d3

O d2 d3

d1 O d2 o b1 d1 b2 o 1 b1 f R c3 b3 b2 π C b3 b1 A K π b2 c1 C b1

c2

1

× Ö × Ö × (a) β Öb1, b2, b3 , δ d1, d2, d3 : yβ A yδ (b) γ c1,c2,c3 : yγ f R yδ

yβ K yγ

k3

d3

2 d3 O d k2 d1 O d2 o k1 b1 d1 o b2 1 b1 R ¡ c3 b3 b2 R π b3 C 1 ¡ b1 K n3 e3 π b2 c1 C n2 b1 K n1 e1 e2 b2 c2 1

¡

× Ö ×

(c)  Öe1,e2,e3 : y R yδ,(d)κ k1, k2, k3 : yγ K yβ,

¡1

× ν Ön1,n2,n3 : yν K y yκ R yγ

1

1 ¡1 Figure 5.4: Coordinate systems generated by applying f R, K, R, R ¡ and K . 40 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

1 δ κ y f y 1 δ f κ R R ¡ 1 1 R f R ¡ y f  1 1 R R ¡

f

¡1 K yγ K γ 1 1

fA f A ¡

¡1 yν K ν K 1 f f

yβ β

   10

100   01 010 00

yα α (a) (b)

Figure 5.5: Relationships between (a) coordinates in different bases. e.g. yβ K yγ

¡1

and (b) bases themselves, e.g. β γ K , associated with a perspective

camera.

Õ Ö × Ô Õ

We have introduced two new coordinate systems ÔO, ν , ν n1,n2,n3 and O,  , ×  Öe1,e2,e3 .

Next we consider coordinates of a vector y w.r.t. basis β and apply successively

1 ¡1 ¡1 ¡1 K ¡ and R .CoordinatevectorK yβ gives yγ.CoordinatevectorR yγ can be

× interpreted as coordinates of y w.r.t. a new basis κ Ök1, k2, k3 , Figure 5.4(d). To

get from yκ to yδ we need to employ fI

¡1

yγ K yβ (5.59)

¡1

yκ R yγ (5.60)

yδ fI yκ (5.61)

Õ Ö × We have thus introduced a new coordinate system ÔO, κ , κ k1, k2, k3 . Figure 5.5 summarizes the relationship between coordinates of a vector and be- tween bases associated with a perspective cameras.

§ 23 Recovering camera pose from its projection matrix Let us next consider that we have already computed the camera projection matrix

1

Ö  × Q ξ P ξ KR I Cδ (5.62)

f ¢

consisting of a 3 ¢ 3matrixM and 3 1vectorm

 × Q ÖM m (5.63)

To recover camera pose from Q,weneedtogetCδ from m and to decompose Q into

Â

  the product of K in the form of (5.34) and R such that R R I and R 1. Consider

M in the form

 

m1Â

Â

  M m2 (5.64) m3Â

41 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

Next we notice that the last row of KRhas unit norm since it is equal to the last row of rotation R. Therefore, we need to divide M by the norm of its last row to get a matrix

decomposable into the product of KR. Moreover, it follows from the construction

$      $ of β that k11 $ 0andk22 0. Thus, determinant KR K R k11 k22 0. Therefore, we also need to multiply M by the sign of its determinant to get a matrix

decomposable into KR.

      Â

m1Â k11 k12 k13 r1

  

sign M sign M

 Â

     

M m2 0 k22 k23 r2 (5.65)

  

 3 3  m m  m3 001 r3 which provides the following set of equations

m  m3

2 Â Â

2 k22 r2 r3 k23 r3 r3 k23 (5.66)  m3

m1Â m3

2 k13 (5.67)  m3

m2Â m2 2 2

2 k22 k23 (5.68)  m3

m1Â m2

2 k12 k22 k13 k23 (5.69)  m3

m1Â m1 2 2 2

2 k11 k12 k13 (5.70)  m3

from which k11, k12, k13, k22, k23 can be easily computed considering that the most

$ $ $ of consumer digital cameras have k11 $ 0, k22 0, k13 0, k23 0.

Having kij computed, we recover R from M as 

sign M

¡1

R K M (5.71)  m3

Camera projection center can be computed in two ways. Either we get

1 ¡ Cδ ¡M m (5.72) or we obtain it by finding a basis c of the one-dimensional right null space of matrix Q, i.e. solving

Qc 0 (5.73)

and then computing

Cδ 1

c (5.74) 1 c4 where c4 is the fourth coordinate of vector c.

5.4 Camera calibration and angle between projection rays

We have introduced matrices P, R and K,andvectorCδ which determine the pro-

jection from space to images. However, since K is introduced with K33 1, the triplet (K, R, Cδ) does not contain all information about the camera, which can be

42 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

d3 X2 X 2 O d2 d12 x3

η3 x 3 d23 d1

X3 X1 x2 d31

η2 x 2 x2 x1

η1 x C 1

x3 x1

π C

Figure 5.6: A calibrated camera pose can be computed from projections of three known points. obtained by direct measurement of its physical components in a world coordinate system equipped with a known world unit length 1W . The missing element is the scale of P, which is equivalent to knowing the value of the focal length or the size of

   pixel, i.e. f, b1 or b2 ,in1W .

   ß Knowing K and f allows to recover b1 from Equations 5.33 as b1 f k11.

  ß Knowing K and b1 , on the other hand, gives f b1 k11. Therefore, full calibration of the camera is encoded in matrix P, Equation 5.52, or, e.g., in one of the following four-tuples: (K, R, Cδ, f), (K, R, Cδ, k11)or(K, R, Cδ, k22).

We defined the camera calibration matrix K with K33 1 because we often do not have access to world units when working with images without knowing anything about the camera which was used to make them. Moreover, a number of important tasks can be done without knowing the world units.

§ 24 Angle between projection rays Consider two image points u1β and u2β.The

direction vectors of the rays are in β given by

u1β u2β

x1β ,x2β (5.75) 1 1 To obtain the angle between the direction vectors by evaluating the scalar product of the vectors, we need to pass to an orthogonal basis. The “closest” orthogonal

basis is γ.Hence

 ¡Â ¡  1

x1γx2γ x1β K K x2β

Ô Õ

cos  x1,x2 (5.76) ¡

¡1 1

     x1γ x2γ K x1β K x2β Notice that we could use the othogonal basis γ to measure angles instead of, e.g., the closest orthonormal basis  since the unknown scale factor f cancels in the following

43 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

formula

 Â

Â

ÕÔ Õ

x1x2 Ôfx1γ fx2 x1γx2γ

Ô Õ

cos x1,x2 (5.77)

        x1 x2 fx1 fx2 x1γ x2γ We conclude that we do not need to know f to measure the angle between projection rays.

5.5 Calibrated camera pose computation

We have seen how to find (uncalibrated) perspective camera pose from projections of known six points. In fact, we have recovered the calibration of the camera. Next we shall show that when the calibration is known, we are able to find the pose of the camera from projections of three points. This is one of very classical problem. Its solution is known since ... when it has been solved by Grunert. Figure 5.6 shows a camera with center C, which projects three points X1, X2 and

Õ X3, represented by vectors X1δ, X2δ and X3δ in ÔO, δ , into image points represented by x1β, x2β and x3β.

§ 25 Classical formulation of the calibrated camera pose computation We in- troduce distances between pairs of points as

¡   ¡   ¡  d12 X2δ X1δ ,d23 X3δ X2δ ,d31 X1δ X3δ (5.78)

Since we see three different points, we know that all distances are non-zero. Õ Points X1, X2 and X3 are in ÔC, γ represented by vectors

1

xiγ K ¡ xiβ

ηi ηi ,i 1, 2, 3 (5.79)

¡1

   xiγ K xiβ with ηi representing the distance from C to Xi.Distancesηi are non-zero since otherwise we could not see the points.

§ 26 Computing distances to camera center Calibrated perspective camera mea-

sures angles between projection rays

¡Â ¡ Â 1

xiβ K K xjβ

Ô Õ Ô ¡ Õ

cij cos xi,xj ,i 1, 2, 3,j i 1 mod 3 1 (5.80) ¡

¡1 1

 

K xiβ K xjβ

Ô Õ Hencewehaveallquantitiesηi,cos xi,xj and dij, which we need to construct

2 2 2

¡ Ô Õ a set of equations using the rule of cosines dij ηi ηj 2 ηi ηj cos xi,xj , i.e.

2 2 2

¡ d12 η1 η2 2 η1 η2 c12 (5.81)

2 2 2

¡ d23 η2 η3 2 η2 η3 c23 (5.82)

2 2 2

¡

d31 η3 η1 2 η3 η1 c31 (5.83)

Ô Õ with cij cos xi,xj . We have three quadratic equations in three variables. We shall solve this system by manipulating the three equations to generate one equation in one variable, solving it and then substituting back to get other variables.

44 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

§ 27 A classical (tedious) solution Let us first get two equations in two variables. Let us generate new equations by multiplying the left side of (5.81) and (5.83) right side of (5.82) and right side of (5.81) and (5.83) by left side of (5.82)

2 2 2 2 2 2

¡ Õ Ô ¡ Õ d12 Ôη2 η3 2 η2 η3 c23 d23 η1 η2 2 η1 η2 c12 (5.84)

2 2 2 2 2 2

¡ Õ Ô ¡ Õ d31 Ôη2 η3 2 η2 η3 c23 d23 η3 η1 2 η3 η1 c31 (5.85)

We could have made three different choices which equation to use twice but since all

dij  0, and hence all sides of the equations are nonzero, all the choices are equally valid. We have now two equations with three variables but since the equations are homogeneous, we will be able to reduce the number of variables to two by dividing 2

equations by (e.g.) η1 (which is non-zero) to get

¨ ¨

2 2 2 2 2

¡ ¡

d12 η12 η13 2 η12 η13 c23 d23 1 η12 2 η12 c12 (5.86)

¨ ¨

2 2 2 2 2

¡ ¡ d31 η12 η13 2 η12 η13 c23 d23 1 η13 2 η13 c31 (5.87)

η2 η3

12 13 with η η1 and η η1 . Notice that we have a simpler situation than before with only two quadratic equations in two variables. Let us proceed further towards a one equation in one variable. We rearrange the terms to get homogeneous polynomials in η13 on the left and

the rest on the right ¨

2 2 2 2 2 2 2

Õ ¡ ¡ d12 η13 Ô¡2 d12 η12 c23 η13 d23 1 η12 2 η12 c12 d(5.88)12η12

2 2 2 2 2 2 2 2

¡ Õ Ô ¡ Õ ¡ Ôd31 d23 η13 2 d23 c31 2 d31 η12 c23 η13 d23 d31 η12 (5.89) to get two quadratic equations

2

m1 η13 p1 η13 q1 (5.90)

2

m2 η13 p2 η13 q2 in η13 with

2

m1 d12 (5.91)

2 ¡

p1 2 d12 η12 c23 (5.92) ¨

2 2 2 2

¡ ¡ q1 d23 1 η12 2 η12 c12 d12η12 (5.93)

2 2 ¡ m2 d31 d23 (5.94)

2 2 ¡ p2 2 d23 c31 2 d31 η12 c23 (5.95)

2 2 2 ¡ q2 d23 d31 η12 (5.96)

We have “hidden” the variable η12 in the new coefficients. We can now look upon

Equations 5.90 as on a linear system

2 m1 p1 η13 q1

(5.97) m2 p2 η13 q2

The matrix of the system (5.97) either is or is not singular.

45 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

§ 28 Case A If it is not singular, we can solve the system by Cramer’s rule [5, 6, 4]

§ § § §

§ § §

2 § m1 p1 q1 p1

§ § § §

η13 (5.98)

§ § §

§ m2 p2 q2 p2

§ § § §

§ § §

§ m1 p1 m1 q1

§ § § §

η13 (5.99)

§ § § § m2 p2 m2 q2 giving

2

¡ Õ ¡

η13 Ôm1 p2 m2 p1 q1 p2 q2 p1 (5.100)

¡ Õ ¡ η13 Ôm1 p2 m2 p1 m1 q2 m2 q1 (5.101)

Eliminating η13 (by squaring the second equation, multiplying the first one by

m1 p2 ¡ m2 p1, which is non-zero, and comparing the left hand sides) yields

2

¡ ÕÔ ¡ Õ Ô ¡ Õ Ôm1 p2 m2 p1 q1 p2 q2 p1 m1 q2 m2 q1 (5.102)

Substituting Formulas 5.91-5.96 into Equation 5.102 yields

4 3 2

0 a4 η12 a3 η12 a2 η12 a1 η12 a0 (5.103) with coefficients

8 4 4 4 4 2 4 2 6 2 2 6

¡ ¡ ¡ ¡ a4 d23 d12 d23 d23 d31 2 d12 d23 d31 2 d23 d31 2 d12 d23 (5.104) 2 2 4 2

4 d12 c23 d23 d31

4 4 2 6 2 6 4 4

¡ ¡ a3 4 d12 d23 c31 c23 4 d12 d23 c12 4 d12 c23 d23 c31 4 d23 c12 d31 (5.105)

8 2 4 2 2 2 4 2 6 2

¡ ¡ ¡ 4 d23 c12 4 d12 d23 c31 d31 c23 8 d12 c23 d23 d31 c12 8 d23 c12 d31 2 4 2

4 d12 d23 c12 d31

6 2 2 6 2 4 4 4 4 4 4 2

¡ ¡ a2 8 d23 c12 d31 4 d23 d31 2 d23 d31 2 d12 d23 4 d12 d23 c31 (5.106)

8 2 4 2 4 8 2 6

¡ ¡ ¡4 d23 c12 4 d12 c23 d23 2 d23 8 d12 c23 d23 c31 c12

2 2 4 2 4 2 4 2 6 2 2 4 2

¡ 4 d12 c23 d23 d31 4 d23 c12 d31 4 d12 d23 c31 8 d12 d23 c31 d31 c23 c12

4 4 2 6 8 2 6

¡ a1 4 d23 c12 d31 4 d12 d23 c12 4 d23 c12 4 d12 c23 d23 c31 (5.107)

2 6 2 2 4 2 2 4 2

¡ ¡ ¡8 d12 d23 c31 c12 4 d12 d23 c31 d31 c23 4 d12 d23 c12 d31

4 4 6 2 ¡ 4 d12 d23 c31 c23 8 d23 c12 d31

6 2 2 4 2 4 4 4 4 2 6 2

¡ ¡ a0 2 d23 d31 2 d12 d23 d31 d23 d31 d12 d23 4 d12 d23 c31 (5.108)

8 2 6 ¡ ¡d23 2 d12 d23

We will use computation of eigenvalues to find a numerical solution to Equa-

tion 5.103. Construct the companion matrix

  a0

000¡ a4

a1

¡

100 a4

C a2 (5.109)

¡   010 a4 a3 001¡ a4 and observe that

4 a3 3 a2 2 a1 a0

¡   η12 I C η12 η12 η12 η12 (5.110) a4 a4 a4 a4

46 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

Therefore, a numerical approximation of η12 can be obtained by computing, e.g., >>eig(C) in Matlab. Complex solutions are artifacts of the method and should not be further considered. For every real solution, we can then substitute back to Equation 5.101 to obtain the corresponding

m1 q2 ¡ m2 q1

η13 (5.111)

m1 p2 ¡ m2 p1

2 2 2 2 2 2 2 2 2 2

¡ Õ Ô ¡ ÕÔ Ô ¡ Õ¡ Õ d12 Ôd23 d31 η12 d23 d31 d23 1 η12 2 η12 c12 d12 η12

2 2 2 2 2 2

¡ Õ Ô ¡ Õ 2 d12 Ôd23 c31 d31 c23 η12 2 d31 d23 d12 c23 η12

To get η1, η2 and η3, we consider Equation 5.81, which can be rearranged as

2 2 2

Ô ¡ Õ d12 η1 1 η12 2 η12 c12 (5.112) and hence yields positive

d12

η1 2 (5.113) ¡ 1 η12 2 η12 c12

η2 η1 η12 (5.114)

η3 η1 η13 (5.115)

§ 29 Case B Let us now look at what happens when the matrix of the system (5.97)

is singular. Then we have

m1 p2 ¡ m2 p1 0 (5.116)

2 2

Ô ¡ Õ ¡2 d12 d23 η12 c23 c31 0 (5.117)

η12 c23 c31 (5.118)

§ 30 Case B1 When c23  0, then we get c31

η12 (5.119) c23

Substituting it to Equations 5.86 we get

¢ ª ¢ ª

2 c31 2 2 c31 2 c31 2 c31

Õ ¡ Ô Õ ¡ d12 Ô η13 2 η13 c23 d23 1 2 c12 (5.120)

c23 c23 c23 c23

¨ ¨

2 2 2 2 2 2 2 2

¡ ¡ d12 c31 c23 η13 2 c31 c23 η13 d23 c23 c31 2 c31 c23 c12 (5.121) and after some more manipulation obtain a quadratic equation

2 2 2 2 2 2 2 2 2 2 2 2

Õ Ô¡ Õ ¡ ¡ Ôd12 c23 η13 2 d12 c23 c31 η13 d12 c31 d23 c23 d23 c31 2 d23 c12 c23 c31 0 (5.122)

in η13.Wegetη1, η2 and η3 from Equations 5.113, 5.114, 5.115.

§ 31 Case B2 When c23 0, then it follows from Equation 5.118 that c31 0as

well. Returning back to equations 5.86, 5.87 provides

¨ ¨

2 2 2 2 2

¡

d12 η12 η13 d23 1 η12 2 η12 c12 (5.123)

¨ ¨

2 2 2 2 2

d31 η12 η13 d23 1 η13 (5.124)

Expressing η13 from Equation 5.124 gives

2 2 2 2 2 2

¡ Õ ¡ Ôd23 d31 η13 d31 η12 d23 (5.125)

47 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

2 2

§ 32 Case B2.1 When d23  d31,thenwecanwrite

2 2 2

2 d31 η12 ¡ d23

η13 2 2 (5.126)

d23 ¡ d31

to substitute it into Equation 5.123 ª

¢ 2 2 2

¨

2 2 d31 η12 ¡ d23 2 2

¡ d12 η12 2 2 d23 1 η12 2 η12 c12 (5.127)

d23 ¡ d31

which we further manipulate to get quadratic equation in η12 ¨

2 2 2 2 2

d23 d12 ¡ d23 d31 η12 (5.128)

2 2 2 2 2 2 2

Ô ¡ ÕÕ Ô ¡ ¡ Õ Ô2 c12 d23 d23 d31 η12 d23 d31 d12 d23 0 (5.129)

We get η1, η2 and η3 from Equations 5.113, 5.114, 5.115.

2 2

§ 33 Case B2.2 Finally, when d23 d31, then we get from Equation 5.125

η12 1 (5.130) and from Equation 5.123

2

2 d23

Ô ¡ Õ¡ η13 2 2 2 c12 1 (5.131) d12

We get η1, η2 and η3 from Equations 5.113, 5.114, 5.115.

§ 34 A modern (more elegant) solution The classical solution is perfectly valid but it was quite tedious to derive it. Let us now present another, somewhat more elegant, solution, which exploits some of more recent results of algebraic geome- try [14, 15]. Let us consider Equations 5.81, 5.82, 5.83 and proceed to Equations 5.86, 5.87,

but this time with use all three pairs to get three equations in η12, η13

¨ ¨

2 2 2 2 2

¡ ¡ ¡

f1 d12 η12 η13 2 η12 η13 c23 d23 1 η12 2 η12 c12 0 (5.132)

¨ ¨

2 2 2 2 2

¡ ¡ ¡

f2 d31 η12 η13 2 η12 η13 c23 d23 1 η13 2 η13 c31 0 (5.133)

¨ ¨

2 2 2 2

¡ ¡

f3 d12 1 η13 2 η13 c31 ¡ d31 1 η12 2 η12 c12 0 (5.134)

It is known [14, 15] that the solution to a set of k algebraic equations

Õ fi Ôx1,...,xn 0,i 1 ...,k (5.135)

in n variables, can (provided it exists) be always obtained by deriving a polynomial

Õ

g Ôxn 0 in the last variable by the following procedure. If the system, does not have any solution, gn 1, i.e. a non-zero constant, leading to contradiction 1 0. First generate additional equations by multiplying all fi by all possible monomi- als up to degree m

2 2 3 2 m x1,...,xn,x1,x1 x2,...,xn,x1,x1 x2,...,xn (5.136)

48 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected]) to get equations

2 m

f1 0,...,fn 0,x1f1 0,...,xnfn 0,x1f1 0,x1 x2f1 0,...,xn fn 0 (5.137) The degree m needs to be chosen such that the next step yields the desired results. It is always possible to choose such m but it may sometimes be found only by using more and more monomials until the Gaussian elimination of the matrix of coefficients, which combine monomials, does not produce a row corresponding to an equation in xn only. Let us demonstrate this process by solving our problem. We use the following four monomials of maximal degree two

2 η12,η13,η12 η13,η12 (5.138)

2

Notice that we did not include the second degree monomial η13 since it turns out that equations generated by that monomial are not necessary. We obtain 15 3 43

equations

  f1 3

η12 η13 

 f2 3

  

 η13

  

 f3 2 2

  

 η12 η13

  

 η12 f1 2

  

 η13 η12

  

 η12 f2 2

  

 η13

  

 η12 f3 3

  

 η12 η13

  

 η13 f1 2

  

 η13 η12

   

 

η13 f2  M Mm 0 (5.139)

  

 η13 η12

  

 η13 f3

  

 η13

  

 η12 η13 f1 4

  

 η12

  

 η12 η13 f2 3

  

 η12

  

 η12 η13 f3 2

  

 2 η12

  

 η12 f1

  

2  η12   η12 f2 2 1 η12 f3

with

 

m 00 0m m ¡m

000 0m1 00¡ 7 4 8 2

m 00¡m 0 m

000 0m5 00m9 ¡ 10 3 2

000 0¡m 000m 00m ¡m m

1 11 3 12 6

000m 00¡m 000m m ¡m 0

1 7 4 8 2

000m m 00¡m 0 m 0

5 00m9 ¡ 10 3 2

m 00 0m 00m ¡m m 0

000¡ 1 11 3 12 6

0 m m 00m m ¡m 00000

1 0 ¡ 7 4 8 2

0 m m 0 ¡m 0 m 00000 5 0 m9 ¡ 10 3 2

M

m 00m 0 m ¡m m 00000

0 ¡ 1 11 3 12 6

m m 00m m ¡m 000000

1 0 ¡ 7 4 8 2

m m 0 ¡m 0 m 000000

5 0 m9 ¡ 10 3 2

m 00m 0 m ¡m m 000000

¡ 1 11 3 12 6

 

m 000m m ¡m 00 00m1 00¡ 7 4 8 2

00m m 00¡m 0 m 00

5 00m9 ¡ 10 3 2

m 000m 00m ¡m m 00 00¡ 1 11 3 12 6 (5.140) and

2 2 2 2 2

¡ m1 d12 m4 d12 d23 m7 2 d12 c23 m10 2 d23 c31

2 2 2 2 2

¡ m2 d23 m5 d23 d31 m8 2 d23 c12 m11 2 d12 c31

2 2 2 2 2

¡ m3 d31 m6 d31 d12 m9 2 d31 c23 m12 2 d31 c12 (5.141) Matrix M holds coefficients and vector m holds monomials. Notice that in the last five monomials on the right in Equation 5.139, there appears only η12. We have deliberately ordered monomials in this way. Next, we perform Gaussian elimination (with pivoting) of matrix M and get new matrix M ½ .

49 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

One can verify that that the 10th row of M ½ has the first nine elements equal to

zero. Therefore ½

M10,: m 0 (5.142) is a polynomial only in η12. In fact, it is exactly a non-zero multiple of polynomials obtained in cases A, B1, B2.1 and B2.2 above. Discussion of the cases happens in the Gaussian elimination with pivoting, which avoids dividing by elements close to zero. The resulting polynomial may be of degree four (case A) but will come out of a lower degree in other cases.

§ 35 Computing camera center coordinates To compute the coordinates of the camera center, we shall use the fact that the center must lie on the intersection of three spheres with radii η1, η2, η3 andwithcentersX1, X2, X3. To simplify the computation, we will choose a special coordinate system, in which the computation will be easier. We place the origin of the new coordinate system

to point X1 and construct new basis τ,inwhichvectorsY1τ , Y2τ , Y3τ ,representing Õ X1, X2, X3 in the new coordinates system ÔX1,τ will be particularly simple. We

require

    

 0 p q

    



Y1τ 0 , Y2τ 0 , Y3τ r (5.143) 0 0 0 To fully specify τ, we will require it to be orthonormal. We shall see that this requirement, altogether with Equation 5.143, fix τ uniquely. Vectors Yiτ are related to vectors X iδ as

Ô ¡ Õ Yiτ S Xiδ X1δ ,i 1, 2, 3 (5.144) where S is the the matrix transforming coordinates of a vector from basis δ to basis

τ.

Â

  Since both δ and τ are orthonormal, there holds S S I, S 1. We see that Yiτ becomes automatically zero vector by the choice of the origin. Let us now construct S and determine p, q and r.

Exploiting the required coordinates we get

 

¡

Y1τ Y2τ Y3τ S 0 X2δ ¡ X1δ X3δ X1δ (5.145)

  

0 pq

Â

 

¡ S 00r 0 X2δ ¡ X1δ X3δ X1δ (5.146)

000

  

0 pq

 

 

¡ s1 s2 s3 00r 0 X2δ ¡ X1δ X3δ X1δ (5.147) 000 where column matrices s1, s2, s3 denote columns of S Â . Passing to vector equations delivers

¡ ¡

p s1 X2δ X1δ,qs1 r s2 X3δ X1δ (5.148)  Since we require s1 1, we get

X2δ ¡ X1δ

¡  p X2δ X1δ , s1 (5.149)

¡  X2δ X1δ

50

T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

Â

   

Next we use s1 s2 0and s1 s2 1togetq and r Â

Â

Õ Ô ¡ Õ s1 Ôq s1 r s2 s1 X3δ X1δ (5.150)

Â

Ô ¡ Õ q s1 X3δ X1δ (5.151)

¡ ¡ r s2 X3δ X1δ q s1 (5.152)

 ¡ Õ¡  r X3δ X1δ q s1 (5.153) and we get s2 as

¡ X3δ ¡ X1δ q s1

s2 (5.154)

¡ ¡  X3δ X1δ q s1

Finally, we compute ¢ s3 s1 s2 (5.155)

Matrix S has been determined and hence we can get Y1τ , Y2τ , Y3τ ,intheform prescribed by (5.143).

Õ

Let us now compute vector Dτ , which represents the camera center C in ÔX1,τ .  Introduce coordinates of 

x   Dτ y (5.156) z to formulate that C is the intersection of three spheres with radii η1, η2, η3 and with centers X1, X2, X3 represented by Y1τ , Y2τ , Y3τ

2 2 2 2

x y z η1 (5.157)

2 2 2 2

¡ Õ Ôx p y z η2 (5.158)

2 2 2 2

¡ Õ Ô ¡ Õ Ôx q y r z η3 (5.159)

Subtracting Equation 5.157 from Equation 5.158 yields

2 2 2 2

Ô ¡ Õ ¡ ¡ 0 x p x η1 η2 (5.160)

2 2 2

¡ ¡ 0 2 px p η1 η2 (5.161)

2 2 2 ¡ p η1 η2

x (5.162) 2 p Subtracting Equation 5.158 from Equation 5.159 yields

2 2 2 2 2 2

Ô ¡ Õ ¡ ¡ Ô ¡ Õ ¡Ô ¡ Õ 0 x r y η3 η2 x q x p (5.163)

2 2 2 2 2

¡ ¡ Ô ¡ Õ ¡ 0 2 ry r η3 η2 2 p q x q p (5.164)

2 2 2 2 2

¡ ¡ Ô ¡ Õ r q p η3 η2 2 p q x

y (5.165) 2 r

Finally, we get z from Equation 5.157 

2 2 2

¡ ¡ z ¨ η1 x y (5.166)

which finally gives us D τ .TogetvectorCδ representing camera center C in the Õ world coordinate system ÔO, δ , we use Equation 5.144 to arrive at

Â

Cδ S Dτ X1δ (5.167)

51 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

§ 36 Computing camera orientation Camera orientation can be now recovered by Õ

relating direction vectors of rays, which we recover in ÔC,  , to the same direction Õ vectors measured in ÔC, δ . We know from Equation 5.56 that in general

y R yδ (5.168)

We construct unit ray direction vectors in δ as follows

Xiδ ¡ Ciδ

yiδ ,i 1, 2, 3 (5.169)

¡  Xiδ Ciδ

To get unit yi, we use the relationship between bases β, γ,and

1

xi fxiγ xiγ K ¡ xiβ

yi (5.170)

¡1

       xi fxiγ xiγ K xiβ

Therefore, we get 1 ¡

K xiβ Xiδ ¡ Cδ

R i 1, 2, 3 (5.171)

¡1 



¡  K xiβ Xiδ Cδ

which leads to the final equation determining R

 

¡1

1 ¡1 ¡1

¡ x x x      

¡ ¡

K 1β K 2β K 3β X1δ ¡Cδ X2δ Cδ X3δ Cδ

1 ¡1 ¡1

R ¡       (5.172)

x x x

¡ ¡ ¡ K 1β K 2β K 3β X1δ Cδ X2δ Cδ X3δ Cδ

52 6 Homography

Figure 6.1: Making copy of Une Odalisque by Jean-Auguste-Dominique Ingres(1814) http://www.ssqq.com/archive/louvre06.htm

53 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

X

π u x

π ½ o b1

u ½ ½

x ½ b2 o

b3 b1½ ½

b3 b2½ ½

C C

Figure 6.2: Cameras share a projections center. Image projections are related by a homography.

We shall next investigate the relationship between projections of 3D points by two perspective cameras into two images. In general, the projections depend on the shape of the scene and camera poses and this relationship may be very difficult to describe. However, there are several very important situations when the relationship can be given in a form of a special image transform, the homography. Let us first consider the situation when two (different) cameras share a common projection center. That means, the cameras may have different coordinate systems, different orientations but must have the same projection center. This situation often arises when photographing with a camera rotating around its projection center, e.g., when taking images for constructing a panorama capturing wide view angle. We shall see that the corresponding projections will be related by a homography. Next, we shall look at a different situation when the cameras are unconstrained, i.e. they can be anywhere in the space and with completely different poses and coordinate systems, but 3D points are forced to lie in a single plane not containing the camera centers. This situation arises, e.g., when photographing a flat screen, a poster or a board from different viewpoints. Again, the corresponding projections of the points in the plane (but not the projections of the points out of the plane) will be related by a homography.

6.1 Homography between images with the same center ½

Let us consider two perspective cameras with identical projection centers C C , which project point X from space to their respective image planes π and π ½ ,Fig-

54 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

O σ d1 X X

d3 d2

b1 uy

b3 b2 x

π C

Figure 6.3: All 3D points are in a single plane. Coordinates in the plane and in the image are related by a homography.

Õ Ö ×

ure 6.2. We introduce image coordinate systems Ôo, α with α b1, b2 in π

½ ½ ½ ½ ½

½

Ô × Ö

and o ,α Õ with α b1, b2 in π and use them to construct the correspond-



½

Õ Ö × Ô Õ

ingcameracoordinatesystemsÔC, β with β b1, b2, b3 Co and C, β with



½

½ ½ ½

½

× β Öb1, b2, b3 Co .

Point X is projected to image points along the projection rays, which are inter-

½ Â ×

sected with π and π . The projection of X in π is represented by vector uα Öu, v .

½ ½ ½ ½ Â

Ö × The projection of X in π is represented by vector uα½ u ,v .

Vectors x and x ½ are two direction vectors of the same ray and hence are linearly

dependent. Since they are both non-zero for X  C, their linear dependence is

equivalent with

½

È R &λ : λx x (6.1)

To arrive at the relationship between the available coordinates of vectors x and

x½ , we shall now pass from vectors to their coordinates. There holds ½

λx x (6.2)

½

½

λxβ ½ xβ (6.3)

½

λxβ ½ H xβ (6.4) for some 3 ¢ 3realmatrixH with rank H 3, which transforms coordinates of a vector from basis β to basis β ½ .

Considering the choices of camera coordinate systems, we see that

½

λxβ ½ H xβ (6.5)

   

u½ u

½

   

λ v H v (6.6) 1 1

55 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

We obtained an interesting relationship. The above equations tell us that the

image projections are related by a transformation, which is (up to C) independent ½ of the position of points X in space. This is the consequence of having C C . Matrix H is related to camera projection matrices. Consider two camera projec-

tions given by Equation 5.11



Xδ Xδ

¡

ηxβ P A ¡ C A Xδ Cβ (6.7)

1 β 1



½ ½ δ δ ½ ½ X X ½

½

¡

½ ¡ η xβ ½ P A Cβ A Xδ Cβ (6.8) 1 1 which gives 1

¡ ¡

η A xβ Xδ Cδ (6.9)

¡1

½ ½

½

¡ η A xβ ½ Xδ Cδ (6.10)

and therefore

¡1

½ ½ ½ ¡1

η A xβ ½ η A xβ (6.11)

η ½

½ ½ ¡1

x ½ A A xβ (6.12) η β

η ½

½ ¡1

We see that H A A and λ η . Let us go yet one step further by exploiting

1 1

½ ½ ½

A KR and A K R (6.13) f f ½ which combines into

f

½ ½ Â ¡1

H K R R K (6.14)

f ½

½ ½

This is particularly useful when K K and f f since then

½ Â ¡1

H KR R K (6.15)

which implies that H is similar [4] to a rotation

¡1 ½ Â

K HK R R (6.16) and hence has one eigenvalue equal to 1 and the other two of modulus equal to one [2].

6.2 Homography between images of a plane

6.2.1 Image of a plane Let study the relationship between the coordinates of 3D points X, which all lie in a plane σ and their projections into an image. Coordinates of points X are measured

Õ Ö ×

in a coordinate system ÔO, δ with δ d1, d2, d3 . Vectors d1, d2 span plane σ and  therefore 

x   Xδ y (6.17) 0

56 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected]) for some real x, y.

The points X are projected by a perspective camera with projection matrix P into

Â

× Ô Õ image coordinates uα Öu, v , w.r.t. an image coordinate system o, α with α

× Ô Õ Ô Õ Öb1, b2 . The corresponding camera coordinate system is C, β with β b1, b2, b3 . To find the relationship between the coordinates of X δ and uα, we project points

X by P into projections xβ as

 

  

 x

u x

  

X δ  y

   

η v ηxβ P p1 p2 p3 p4 p1 p2 p4 y H yτ  1  0 1 1 1 (6.18)

where p1, p2, p3, p4 are columns of P.

Â

Ö ×

Notice that 3 ¢ 1matrixyτ x, y, 1 represents point X in the coordinate





Õ Ô Õ system ÔC, τ with the basis τ d1, d2, d4 ,wherethed4 CO is the vector

Õ assigned to the pair of points ÔC, O .IfpointC is not in σ, then vectors d1, d2, d4

are independent and hence form a basis. Therefore, matrix

 

H p1 p2 p4 (6.19)

represents a change of coordinates and has rank 3. Õ

When we think about pair ÔC, σ as about a camera that shares its projection Õ center with camera ÔC, π and imagine that points X are all (accidentally) in the projection plane σ, we see that we have recovered the relationship between cameras sharing their projection center.

6.2.2 Two images of a plane We shall now consider the situation when all points in the scene are in a single plane. Then, as we shall see, the projections of the 3D points which are in the plane are again related by a homography even when the camera centers are located at different points in the space. Let us consider a plane σ and two perspective cameras with (in general different) projection centers C and C ½ , which do not lie in σ and corresponding projection

matrices P and P ½

 

P p1 p2 p3 p4 (6.20)

 

½

½ ½ ½ ½

P p1 p2 p3 p4 (6.21)

(6.22)

3 ½ 3 ½

R È R

where pi È and pi , i 1,...,4 stand for the columns of P, P .

½ ½

Õ Õ Ô Õ Ô We establish coordinate systems ÔO, δ , C, β , C ,β in the standard way, see

Figure 6.4 to get 

x    Xδ y (6.23) 0 for some real x, y.

57 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

O σ d1 X X

d3 d2

b1 u y

3 2 b b ½ y x π ½ π C u ½

x ½

b1½ ½ b2½ b3 C ½

Figure 6.4: All 3D points are in a single plane. Two images of the points are related by a homography.

Point X È σ is projected to the cameras as

  

x 

x

  

X δ  y

 

ηxβ P p1 p2 p3 p4 p1 p2 p4 y G yτ  1  0 1

1

  

x 

x

  

 y

½ ½ ½ ½

½ Xδ

½ ½ ½ ½ ½ ½ ½

 

½

η xβ ½ P p1 p2 p3 p4 p1 p2 p4 y G yτ  1  0 1 1

½

R ÞØ Ù Ô Õ Ô Õ

for some η, η È 0 and two new coordinate systems C, τ with τ d1, d2, d4 ,





½

½ ½ ½ ½

½

Õ Ô Ô Õ where the d4 CO and C ,τ with τ d1, d2, d4 ,wherethed4 CO .

We see that there are two different vectors y and y ½ , which appear on the right ½ hand side of the equations but they are in different bases, i.e. as yτ and yτ ½

ηxβ G yτ (6.24)

½ ½ ½ ½

½

η xβ ½ G yτ (6.25)

½ ½ ½ ½

× Ö ×

with G Öp1, p2, p4 and G p1, p2, p4 .

½ ½

Õ Õ Ô

Coordinate systems ÔC, τ and C ,τ are so special that

½

yτ yτ ½ (6.26)

58

T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

½ Ê

for all points in σ and therefore, considering that C Ê σ and C σ,weget

½ ½ ½ ¡1

η xβ ½ G G ηxβ (6.27)

which we can write as

½

λxβ ½ H xβ (6.28)

η ½

½ ¡1 3¢3

È R for λ η and H G G . Clearly, H ,rankH 3. We could interpret this situation also such that two images of a plane are related by the homography, which is a combination of the homographies relating the plane to its images.

§ 37 Homographies induced by a plane LetuslookatEquation6.27inmore detail. We can write

η ½

½ ½ ¡1 ½ ½ ½ ¡1

Ö ×Ö × x ½ G G xβ p1, p2, p4 p1, p2, p4 xβ (6.29)

η β

    ¡1

10 10

½ ¡1

½

  



¡ ¡ A 01 Cδ 01 Cδ A xβ (6.30)

00 00

  

 1 ¡

½ ¡

10¡x 10 x

½ ¡1

½

   

¡ ¡

A 01 y 01 y A xβ (6.31)

½ ¡ 00 ¡z 00 z

We have introduced new symbols to represent vectors

   

 Â

½

½ ½

½

Cδ xyz and Cδ x y z (6.32) and written the homography as a product of four matrices. Let us next compute

the product of the second matrix with the inverse of the third matrix

 

½

¡ Õß 10Ôx x z

η ½

½ ½ ¡1

½

 

Ô Õß ¡

xβ ½ A 01 y y z A xβ (6.33)

η ½

00 z ßz We see that the middle matrix on the right looks almost as the identity plus some-

thing. Let’s express it in that way

 

½

¡ Õß 10 Ôx x z

η ½

½ ½ ¡1

½

 

Ô ¡ Õß

xβ ½ A 01 y y z A xβ (6.34)

η ½

¡ Õß 001 Ôz z z

We can now further rearrange expressions as follows

  ¬ ¤

½

Ô ¡ Õß

½ x x z 

η 

½ ½ ¡1

½

  ­ ¥

¡ Ô Õß

xβ ½ A I y y z 001 A xβ (6.35)

η ½

¡ Õß

Ôz z z

£ « 

1 

½ ¡1

½

Ô ¡ Õ A I C Cδ 001 A xβ (6.36)

δ Õ

Cδ Ô3

£ « 

1 

¡1 ½ ¡1

½

¡Ô ¡ Õ A A I Cβ C 001A xβ (6.37)

β Õ Cδ Ô3

59 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

Õ

We denoted the third coordinate of Cδ by Cδ Ô3 .  1  1

Vector  001A ¡ has a geometrical interpretation. Consider the equa- 3Õ

Cδ Ô Õ

tion of plane σ in coordinate system ÔO, δ

  X δ

0010 0 (6.38) 1

Â × where Ö001 is the normal vector of plane σ containing point Xδ written w.r.t.

 ¯

Õ Ö × ÔO, δ , i.e. nδ¯ 001,whereδ is the dual basis to basis δ, Chapter 2.

Õ Ô ¡ Õ Next, consider the camera coordinate system ÔC, β with Yβ A Xδ Cδ .We

see that

¡1   A Yβ Cδ

0010 0 (6.39)

1

 

 Yβ

¡1

Õ 001A Cδ Ô3 0 (6.40) 1

provides the unit normal n of plane σ in the dual basis β¯ to basis β

 

 ¡1

nβ¯ 001A (6.41)

We have obtained the following formula for the homography between points xβ, ½

xβ ½ in the two images, which is generated by the plane σ

£ «

η ½ 1

½ ¡1 ½ Â

½

¡Ô ¡ Õ x ½ A A I Cβ C n ¯ xβ (6.42)

β β β Õ η Cδ Ô3

¯ Õ where nβ¯ is the normal vector of σ in β, Cδ Ô3 is the distance of σ from the camera center C,andη, η ½ are the distances of points from the respective principal planes in multiples of the respective focal lengths.

6.3 Spherical image

Consider a camera rotating around a center C and collecting n images all around such that every ray from C is captured in some image. We can choose one camera,

e.g. the first one, and relate all other cameras to it as

λi xβ1 Hi xβi ,i 1,...,n (6.43)

Since all vectors x were captured, there inevitably will appear a vector with coordi-  nates 

x

  xβ1 y (6.44) 0

Such vector does not represent any point in the affine image plane π1 of the first camera because it does not have the third coordinate equal to one. To be able to represent rays in all directions, we have to introduce spherical image,whichisthe set of all unit vectors in R3 (also called omnidirectional image). We sometimes use

60 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected]) only a subset of the sphere, typically a cylinder, to capture panoramic image.In such a case, we can remap pixels onto such cylinder and then unwarp the cylinder into a plane. Notice however, that in such a representation, straight lines in space do not project to straight lines in images. All equations we have developed so far work with minor modifications also for vectors with last zero coordinate. We will come back to it later when studying pro- jective plane which is somewhere between the affine image plane and full a spherical image.

6.4 Homography – summary

Let us summarize the findings related to homography to see where it appears.

Let us encounter one of the following situations

 ½ ½

× Ö × 1. Two images with one projection center Let Öu, v and u ,v be co- ordinates of the projections of 3D points into two images by two perspective

cameras with identical projection centers;

 ×

2. Image of a plane.LetÖu, v be coordinates of 3D points all in one plane

½ ½ Â × σ, w.r.t. a coordinate system in σ and Öu ,v coordinates of their projections

by a perspective cameras with projection center not in the plane σ;

½ ½ Â Â

× × Ö 3. Two images of a plane Let Öu, v and u ,v be coordinates of the pro- jections of 3D points all in one plane σ, into two images by two perspective cameras with projection centers not in σ;

then there holds

    u½ u

corr

3¢3 Â ½ ½ Â

½

   

& È R )Ö × " Ö × & È R H , rank H 3, so that u, v u ,v λ : λ v H v

w ½ w

(6.45)

½ where w w 1 for perspective images and may be general for spherical images. In all three cases, coordinates of points are related by a homography. We have used linear algebra to derive the relationship between the coordinates of image points in the above form. The homography can be also represented in a different way.

To see that, we shall eliminate λ as follows

       

u½ u h11 h12 h13 u

½

       

λ v H v h21 h22 h23 v (6.46)

1 1 h31 h32 h33 1

½

λu h11 u h12 v h13 (6.47)

½

λv h21 u h22 v h23 (6.48)

λ1 h31 u h32 v h33 (6.49)

h11 u h12 v h13 ½

u (6.50)

h31 u h32 v h33

h21 u h22 v h23 ½

v (6.51)

h31 u h32 v h33

61 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

We see that mapping h obtained as

 

¢ ª

h v h

h11 u 12 13

½

h v h

u u h31 u 32 33

h (6.52)

½

h v h

v v h21 u 22 23

h v h h31 u 32 33 is a mapping from a subset of R2 to R2 but it is not linear! It contains fractions of affine functions. On the other hand, we can understand the homography as a linear mapping in certain sense. However, it is not a linear mapping in the natural sense in which we really use it. Matrix H is a matrix and in this sense it represents a linear mapping from R3 to R3 . However, we are not interested in the individual vectors in R3 but in complete one-dimensional subspaces, which correspond to the direction vectors representing

projection rays.

  Notice that λ can accommodate for any change of the length of uv1 Â

(except for making it zero) since it can be split into ξ,ξ½ and used as

   

u½ u

½

½

   

ξ v H ξ v (6.53)

1 1 ½

x Hx (6.54) 3 We can now think ab out x and x ½ as about one-dimensional subspaces of R 1

generated by x and x ½ . The “equation” ½

x Hx (6.55)

then actually means

½ ½ ½

È & È &x x and x x such that x H x (6.56) Thus the homography can be seen as a mapping between one-dimensional subspaces of R3. While R3 itself is a linear space, the set of its one-dimensional subspaces, in thewayweusethem,is not a linear space and therefore the homography is not a linear mapping although it is represented by a matrix H, which is used to multiply vectors.

It is also important to notice the true relationship between homographies and ¢ 3 ¢ 3 real matrices. Any 3 3 real matrix of rank 3 represents a homography but

many different matrices represent the same homography. Let’s see why.

3¢3 3¢3

R È R  Let us consider H È and G such that ξ H G for some ξ 0. We can

write

½ ½

ξ x H x (6.57)

½ ½

ξξ x ξ H x (6.58)

½ ½

ξξ x G x (6.59)

½ ½

λ x G x (6.60) We see that H and G represent the same homography. Indeed, two matrices related by a non-zero multiple represent the same homography. Hence, it suggests itself to

associate homographies with one-dimensional subspaces of 3 ¢ 3matrices.

1Monograph [16] very often uses “=” exactly in this sense of equality of one-dimensional subspaces.

62 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

6.5 Image of image of an image

Let us now consider an interesting question. Figure 6 presents a photograph of an artist painting a scene with a painting. We see the original painting and its copy in the same photograph. The artist may have different goals. For instance, he might want to reproduce the scene exactly as it looks from his point of view, to imitate a photographic process with his canvas instead of the film and his eye instead of the lens. Alternatively, he might want to reproduce an exact copy of the original painting, or perhaps its somewhat smaller version. The question is: Can we find out what is he aiming at from the photograph? To answer this and other similar questions, we need to understand the relationship between a planar scene, its photograph and a photograph of its photograph.

Figure 6.5 shows a plane σ, e.g. a painting, with point X photographed by

Õ Ô Õ

camera ÔC, π .Thecamera C, π produces the first photograph with projection x ½

½ Ô

of X. The second camera C ,π Õ photographs the first photograph and produces

½ ½ ½ Õ the second photograph with projection y of x.CameraÔC ,π also projects point

X directly into x½ . Artist’s reproduction, when faithfully reproducing the scene, is equivalent to a camera. Likewise, a mere scaled copy of a painting is equivalent to

taking its fronto-parallel picture. ½

We see that x ½ is in general not equal to y . Let us find the relationship between ½

coordinates of general points X , x, x ½ and y . We know that two images of a

½ ½

Õ Õ Ô single plane are related by a homography. We see that ÔC, σ and C ,π are two perspective cameras observing plane π. Therefore, points y ½ are related to points

X by a homography. Secondly, we know that points x ½ are also related to X by a ½

homography. We shall see that the x ½ is also related to y by a homography.

½ ½

Õ Õ Ô Õ Ô We establish coordinate systems ÔO, δ , C, β , C ,β in the standard way as

Â

Ö × showninFigure6.5togetXδ xy0 for some real x, y. We consider a plane σ and two perspective cameras with (in general different) projection centers C and

C ½ , which do not lie in σ, with the corresponding projection matrices

 

½ ½

½

¡ ¡ P A Cβ , P A Cβ ½ (6.61)

½ The two images xβ, xβ ½ of X in σ are related (see Equation 6.42 in Section 6.2.2)

by the following homography

£ « ½

η 1 Â

½ ¡1 ½

½

¡Ô ¡ Õ Ô Õ x ½ A A I Cβ C n σ ¯ xβ (6.62)

β β β Õ η Cδ Ô3

¯ Õ

where n Ôσ β¯ is the normal vector of σ in β.

½ ½

Let us next find yβ ½ as a function of xβ andprojectionmatricesP and P .Itis

clear from Figure 6.5 that we can write

½ ½

½

¡

μ y x C C

½ ½

½

½ ½

¡

½

μ yβ ½ xβ Cβ Cβ

½ ½ ¡1 ½ ¡1 ½

½

¡ ½

μ yβ ½ A A xβ A A Cβ Cβ (6.63)

¡1 ½

with A ½ A transforming coordinates of vectors from β to β (see Equation 5.9) and

½ Õ real μ , which is equal to the distance of the point represented by x in ÔC, π from

63 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

O σ d1 X X

d3 d2

b1

b3 b2 x π ½ π b1 C

b3½ ½

½ b2 x

y ½ e ½

C ½ ½ Figure 6.5: Image y ½ of an image x of an image X is related to the image x of image X by a homography.

the principal plane of the second camera (in multiples of the focal length f ½ of the

second camera).

¡1

This relationship does not look as a homography since, besides A ½ A ,thereis

¡1 ½

½ ¡

also the additive term A A Cβ Cβ ½ . However, since all points of π are represented Õ

in ÔC, β by vector 

 u

 

xβ v (6.64) 1

for some real u and v,wecanwrite

½ ½ ¡1 ½ ¡1 ½

½

¡

½

μ yβ ½ A A xβ A A Cβ Cβ (6.65)

    

u  00 u

½ ½ ½ ¡1

½ ¡1 ½

    



¡ ½ μ yβ A A v 00 A A Cβ Cβ ½ v (6.66)

1 00 1

© ¡

 

½ ½ ¡1 ½

½

Ô ¡ Õ

μ yβ ½ A A I Cβ Cβ 001 xβ (6.67) ½ and hence we arrive again at a homography between xβ and yβ ½ . Notice that Equa-

tion 6.67 could also be obtained from Equation 6.42 for putting σ to the image

   

Â

¡1

Ô Õ ¡ plane, i.e. σ π and O o. That yields n π β¯ 001A 00 f and

Õ Cδ Ô3 f. We can now combine Equations 6.67 and 6.62 to get the final relationship between

64

T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

½ ½ ½

xβ ½ and yβ

¬ ¤

Â

¡ ©

½

Ô Õ

¡1 

η n σ β¯ 

½ ½ ¡1 ½ ½ ½ ¡1 ½ ½ ­

¥

¡Ô ¡ Õ Ô ¡ Õ Ô Õ

½ x ½ A A I Cβ C I Cβ C 001 A A μ y

β β β β Õ η Cδ Ô3

To proceed further, we exploit the following relationship, which holds true for every

two n ¢ 1matricesu, v

Â

¨

¡1 uv

Â

¡

I uv I (6.68) Â

1 v u

and get

£ «£ «

 ½

½

¡ Õ

η nβ¯ ÔCβ Cβ 001

½ ¡1 ½ ½ ¡1 ½ ½

½

 ¡Ô ¡ Õ ¡ Ô Õ

½

xβ ½ A A I Cβ Cβ I A A μ yβ ½

½

Ô Õ ¡ Õ ημ Cδ Ô3 1 001 Cβ Cβ

We now simplify the product of the two middle matrices using

      Â

Õ Ô Õ ¡

uv uw Ô1 w u uv c uw u v u w

¡ ÕÔ ¡ Õ ¡

ÔI I I

 Â

Ô Õ

c 1 w u c 1 w u

 ¡ which holds for every three n ¢ 1matricesu, v and w with w u 1 and a non-zero

scalar c,andobtain

¡ © ¡ ©

¤ ¬

½ Â ½

½

½ ¡ Õ Ô ¡ Õ Ô Õ

η 1 001ÔCβ Cβ Cβ Cβ nβ¯ Cδ 3 001

½ ½ ¡1 ½ ¡1 ½ ½

¥ ­

¡ ©

¡ Ô Õ  ½

x ½ μ y

β A A I A A β ½

ημ ½

Õ Ô ¡ Õ

Cδ Ô3 1 001 Cβ Cβ

£ £ ««

½ ½ ½

¡ Õ

η Cβ Cβ Cδ Ô3 001

½ ½ ¡1 Â ½ ¡1 ½ ½

 ¡ Ô Õ ½

xβ ½ A A I nβ¯ A A μ yβ ½

½

Õ Ô Õ¡ Ô Õ

ημ Cδ Ô3 1 Cβ 3 Cβ 3

£ £ ««

½ ½

½ ½

¡ ½ Õ β Ô

η C Cβ C 3 Â

 ½ ½

½ δ

 ¡

½ ½ xβ I n ¯½ b3 μ yβ (6.69)

β ¯½

½ β

½

Õ Ô Õ¡ Ô Õ ημ Cδ Ô3 1 Cβ 3 Cβ 3

There holds

¡1 ½ ½ ½ ½

½

½

¡ ¡

½ ½

A A Cβ Cβ ½ Cβ Cβ ξ eβ (6.70)

½ ½

where eβ ½ represents the projection of C into the second image and ξ is the distance

of C from the principal plane of the second camera in multiples of f ½ . Hence we can

rewrite Equation 6.69 as

¡ ©

½ ½ ½ ½ Â ½

¡

½ ½ ½ λ xβ I ψ eβ m β¯½ yβ (6.71)

with ½

½

Ô Õ

C 3 Â ξ

 ½

 δ

¯½ m ¯½ n b3 and ψ (6.72) β β β¯½

½

Ô Õ¡ Ô Õ Ô Õ

1 Cβ 3 Cβ 3 Cδ 3

½ Â

We see that the mapping is a homography with fixed point e ½ andfixedlinem β β¯½

since

© ¡

½ ½ Â ½ ½ ½ ½ Â ½ ½ Â ½ ½

¡ ¡ Ô Õ Ô ¡ Õ

½ ½ ½ ½ ½ ½ ½

½ ½ I ψ eβ m β¯½ eβ eβ ψ eβ m β¯ eβ 1 ψ m β¯ eβ eβ (6.73)

and

¡ ©

 ½ ½   ½  ½  ½  ½ Â

¡ Ô Õ Õ ¡ Ô ¡

½ ½ ½

¯½ ¯½ ¯½ ¯½ ¯½ ¯½ m β¯½ I ψ eβ m β m β ψ m β eβ m β 1 ψ m β eβ m β (6.74)

65 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

6.6 Computing homography from image matches

Let us turn to the computational aspect of the homography relationship between images. Our goal is to find the homography mapping from a few pairs of corre- sponding image points. We shall see that this problem leads to solving a system of linear equations.

6.6.1 General perspective cameras Our goal is to find matrix H in Equation 6.45 without assuming any knowledge about

cameras. Let us introduce symbols for rows of homography H

 

 

h1  u

Â

 

 H  h2 and for the vector x v (6.75)

 1 h3

and rewrite the above matrix Equation 6.45 as

½ Â

λu h1 x (6.76)

½ Â

λv h2 x (6.77) Â

λ h3 x (6.78)

Eliminate λ from the first two equations using the third one

 ½ Â

Õ

Ôh3 x u h1 x (6.79)

 ½ Â

Õ Ôh3 x v h2 x (6.80)

(6.81)

 Â

move all to the left hand side and reshape it using x y y x

 ½ Â

Õ

x h1 ¡Ôu x h3 0 (6.82)

 ½ Â

Õ x h2 ¡Ôv x h3 0 (6.83) (6.84)

Introduce notation

 

Â

  Â

h h1 h2 h3 (6.85)

and express the above two equations in a matrix form

½ ½ ½

¡ ¡ uv1000¡u u u v u

h 0 (6.86)

½ ½ ½

¡ ¡ 000uv1 ¡v u v v v

corr

 ½ ½ Â

× " Ö ×

Every correspondence Öu, v u ,v bringstworowstoamatrix

 

½ ½ ½

¡ ¡

uv1000¡u u u v u

½ ½ ½

¡ ¡

000uv1 ¡v u v v v

  h 0 (6.87) . .

Mh 0 (6.88)

66

T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])  If ξ G H, ξ 0thenbothG, H represent the same homography. We are therefore

looking for one-dimensional subspaces of 3 ¢ 3 matrices of rank 3. Each such subspace determines one homography. Also note that the zero matrix, 0, does not represent an interesting mapping. We need therefore at least 4 correspondences in a general position to obtain rank 8matrixM. By a general position we mean that the matrix M must have rank 8to provide a single one-dimensional subspace of its solutions. This happens when no 3 out of the 4 points are on the same line.

Notice that M canbewrittenintheform

 

½ ½ ½

¡ ¡

u1 v1 10 00¡u1u1 u1v1 u1

½ ½ ½

¡ ¡

u2 v2 10 00¡u2u2 u2v2 u2

.

.

M (6.89)

½ ½ ½

¡ ¡

000u1 v1 1 ¡v1u1 v1v1 v1

½ ½ ½

¡ ¡

000u2 v2 1 ¡v2u2 v2v2 v2   . .

with indices naming different points, which can be rewritten more concisely as

 

 ½ Â

x1 0 ¡u1 x1

 ½ Â

¡

x2 0 u2 x2

.

.

.

M (6.90)

 ½ Â

¡

0x1 v1 x1

 ½ Â

¡  0x2 v2 x2  . .

6.6.2 Calibrated cameras

67 7 Projective plane

F. Casorati. Gli Scholari (1927–1928)

http://www.flickr.com/photos/kraftgenie/4776807815/

68 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

X

Y x x

y π C

σ

Figure 7.1: Geometric model of perspective projection in affine space. Point C has infinite (i.e. not unique) projection, point X has exactly one projection x.PointY has no projection.

7.1 Motivation – perspective projection in affine space

§ 38 Geometric model of perspective projection in affine space The perspective projection of a point X by a camera with projection center C can be obtained geometrically in 3D affine space by taking all lines passing through the points C and X and finding the intersections with the (affine) image plane π.

Three different situations may arise, Figure 7.1.

1. If X C, then there is an infinite number of lines passing through C X, which intersect π in all its points, and therefore the projection of X contains the whole plane π.

2. If point Y  C lies in the plane σ, which is parallel to π and passing through C, then the line passing trough C and Y (which there is exactly one) does not intersect the projection plane π, and therefore, the projection of X is empty.

3. If X does not lie in the plane σ, then there is exactly one line passing through points C and X and this line intersects the projection plane π in exactly one point x. Hence the projection of X contains exactly one point x. Let us compare this affine geometrical model of the perspective projection with the algebraic model of the perspective projection, which we have developed before.

§ 39 Algebraic model of perspective projection in affine space The projection

xβ of X δ by a perspective camera with center projection matrix



¡ P A  A Cδ (7.1)

is



X δ

¡ ηxβ A  A Cδ (7.2) 1

69 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

R for some η È . We shall analyze the three situations, which arise with the geometrical model of affine projection.

1. If X C,then



Cδ

¡ ηxβ A  A Cδ 0 (7.3) 1

i.e. we obtain the zero vector. What does it say about xβ? Clearly, xβ can be

completely arbitrary (even the zero vector) when we set η 0. Alternatively,

we can choose η  0 and thus enforce xβ 0. Both choices are possible. We shall use the latter one since we will see that it better fits the other cases. We R3 will use xβ 0 to (somewhat perversely) represent all non-zero vectors in .

2. If point Y  C but it lies in the plane σ,then



Yδ

Ô ¡ Õ

¡ ηxβ A  A Cδ A Yδ Cδ (7.4) 1

which, taking into account rank A 3, implies

1

¡ ¡ η A xβ Yδ Cδ (7.5)

1

Matrix A ¡ transforms xβ into xδ and therefore its columns  1 ¡ A b1δ b2δ b3δ (7.6)

are the basic vectors of the camera coordinate system in the world basis δ.  Hence

¡ η b1δ b2δ b3δ xβ Yδ Cδ (7.7) which means that vector Yδ ¡ Cδ can be written as a linear combination of the camera coordinate system basic vectors

¡ ηpb1δ ηqb2δ ηrb3δ Yδ Cδ (7.8)

R ¡ with p, q, r È .Now,sinceY lies in a plane parallel to π,vectorYδ Cδ can be written as a linear combination of the first two basic vectors of the camera

coordinate system, and therefore r 0, i.e. 

 p

 

xβ q (7.9) 0

We also see that η  0 since otherwise we would get the zero vector on the

left but that would correspond to Y C, which we have excluded. R 3. If X does not lie in the plane σ, then the coefficient r È in the linear combination

1

¡ ¡ η A xβ Xδ Cδ (7.10)

¡ ηpb1δ ηqb2δ ηrb3δ Xδ Cδ (7.11)

70 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

Table 7.1: Comparison of the geometrical and algebraic projection models in affine space.

Point position Projection

Geometrical model in aff. space Algebraic model in aff. space 

 u 



  X Ê σ one point of π η 0, xβ v ,(xβ 0)

1 

u  



È   C  X σ no point η 0, xβ v , xβ 0 0

 X C all points of π η 0, xβ 0

is non-zero. In that case we can write 

 p 



¡ Õ η q A ÔXδ Cδ (7.12)

r   p

r

q

Ô Õ Ô ¡ Õ  ηr  A Xδ Cδ (7.13) r

1 

 u

½ 



Ô ¡ Õ η v A Xδ Cδ (7.14) 1

As in the case two, η  0 since otherwise we would get the zero vector on the

left and that would correspond to X C, which we have excluded.

The comparison of the two models of perspective projection, Table 7.1 shows that

1. We can always assume η  0.

2. The “projection” of C is represented by the zero vector while the projections of all other points are represented by non-zero vectors.

3. The algebraic model can represent projections of all points in the affine space.

4. The geometrical projection model is less capable than the algebraic projection model since it can’t model the projection of points in σ different from C.

The previous analysis clearly shows that the affine geometrical model of the per- spective projection is somewhat deficient. It can’t model projections of some points in the space. This deficiency lead to inventing a generalized model of the geometry around us in order to model the perspective projection completely by intersections of geometrical entities. This generalization of the affine space is called the projective space. Let us look at the most important projective space, which is the projective plane. We shall first develop a concrete projective plane by improving the affine plane

71 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

A3 A3 A3 A3 A3 A2 A2 A2 A2 A2

O O O O

(a) (b) (c) (d) (e)

Figure 7.2: (a) Two dimensional affine plane A2 can be (b) embedded in the three

A3 3 2 A Ê dimensional affine space .ThereisapointO È , O A .(c)For each point X in A2, there is exactly one line through X and O in A3.(d) There is one pencil of lines through O in A3, which do not correspond to any point in A2. (e) Each line in the pencil corresponds to a set of parallel lines in A2 with the same direction. exactly as much as to achieve what we want, i.e. to be able to distinguish projections of all points in the space. In fact, this will be extremely easy since we already have done all the work, and we only need to “upgrade” the notion of point, line, intersection and join (i.e. making the line from two distinct points). Later, we shall observe that such “upgrade” will also lead to an interesting simplification and generalization of the principles of geometry.

7.2 Real projective plane

7.2.1 Geometrical model of the real projective plane ArealaffineplaneA2 can be imagined as a subset of a real affine space A3, Figure 7.2. There is a point O in A3, which is not in A2.ForeachpointX in A2,thereisexactly one line in A3, which passes through X and O. Now, there is a set of lines in A3, which pass through O but do not pass through any point of A2. This is the set of lines parallel to A2 that pass through O. These lines fill the plane of A3,whichis parallel to A2 and which contains the point O. The set of all lines in A3 passing through O will be called the real projective plane and denoted as P2. The lines of A3 passing through O will be called points of the real projective plane. The previous definition can be given without referring to any affine plane. We can take a point O in A3 and the set of all lines in A3 passing through O and call it a projective plane. In the above example, however, we have obtained the projective plane as an extension of a given affine plane A2.Insuchacase,wecandistinguish two sets of points in the projective plane. The lines in A3 passing through O,whichintersectA2 are in one-to-one cor- respondence with points in the affine plane A2 and hence will be called the affine points of the projective plane1 of the projective plane. The set of lines in A3 passing

1Vlastn´ı body in Czech. Finite points in [16].

72 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

A3 A2 o

b2 b1 b3

σ π O

b2 b1

Figure 7.3: Algebraic model of the real projective plane. through O, which do not intersect A2, are the (“additional”) points of the projective plane and will be called the ideal points of the projective plane2. Notice that this division of the points of a projective plane makes sense only when we start with a given affine plane or when we start with a projective plane and select one plane of lines in A3 as the set of ideal points. Once the set of ideal points is selected, all planes of A3 parallel to the plane in A3 formed by the lines corresponding to ideal points are affine planes corresponding to this set of ideal points. To each ideal point P (i.e. a line l of A3 through O parallel to A2) there cor- responds exactly one set of parallel lines in A2, which are parallel to l.Different sets of parallel lines in A2 are distinguished by their direction. In this sense, ideal points correspond to directions in A2 and can also be understood as points where parallel lines of A2 intersect. Notice that the parallel lines do not intersect in A2, because P is not in A2, but they intersect in the real projective plane obtained as the extension of A2.

7.2.2 Algebraic model of the real projective plane We shall now move from the geometrical model in A3 to an algebraic model in R3 which allows us to do computations.

3

Õ A Let us choose a coordinate system ÔO, b1, b2, b3 in with the origin in O,with

2

Õ A Ô Õ basic vectors b1, b2 from the coordinate system Ôo, b1, b2 in and with b3 ϕ O, o , Figure 7.3. Lines in A3, which pass through O, correspond to one-dimensional subspaces of R3 and therefore, in R3, points of the real projective plane will be represented by one-dimensional subspaces. The real projective plane is the set of all one-dimensional subspaces of R3. The affine plane is a subset of the set of all one-dimensional subspaces of R3, which we obtain by removing all one-dimensional subspaces that lie in one two- dimensional subspace of R3.

2Nevlastn´ı body in Czech. Points at infinity in [16].

73 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

b3 R3

X Y R2 0

b2 b1

Figure 7.4: Points of the real projective plane are represented by one-dimensional subspaces of R3. One selected two-dimensional affine subspace deter- mines the ideal points.

There are (infinitely) many possible choices of sets of one-dimensional subspaces which can model the affine plane within the real projective plane. The choice of a particular subset, which will model the affine plane, can be realized by a choice of a basis in R3.

3

Ô Õ R Let us select a basis β b1, b2, b3 of . Then, all the one-dimensional sub-

spaces generated by vectors 

x 

 

È R xβ y x, y (7.15) 1 will represent affine points, point X in Figure 7.4, and all the one-dimensional sub-

spaces generated by vectors 

 x

 

È R   xβ y x, y ,x 0ory 0 (7.16) 0 will represent ideal points, point Y in Figure 7.4. It is clear that the affine points are in one-to-one correspondence with all points in a two-dimensional affine space (plane) and the ideal points are exactly what we need to add to the affine points to get all one-dimensional subspaces of R3.

7.2.3 Lines of the real projective plane Let us look at lines now. Lines, e.g. l in Figure 7.5 in the affine plane contain points represented by one-dimensional subspaces generated, e.g., by x and y.Thissetof one-dimensional subspaces of points on l fills almost a complete two-dimensional subspace of R3 with the exception of one one-dimensional subspace, generated by z, which represents an ideal point. After adding the subspace generated by z to the set of all one-dimensional subspaces representing points on l, we fill the two-dimensional

74 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

A2 A3

l

l x y

O z

Figure 7.5: Lines of the real projective plane correspond to two-dimensional sub- spaces of R3 but can be also represented by one-dimensional subspaces of R3. subspace of R3 completely, which hence corresponds to the projective completion of theaffinelinel, which we will further call line, too. Hence, in the real projective plane, lines correspond to two-dimensional subspaces of R3. We would like to do calculations with lines as we do calculations with points. Let us develop a convenient representation of lines now. A straightforward way how to represent a two-dimensional subspace in R3 is to select a basis (i.e. two linearly independent vectors) of the subspace, e.g. x and y for the line l. There are many ways how to choose a basis and therefore the representation is far from unique. Moreover, having two bases, it is not apparent whether they represent the same

subspace.

Õ Ô Õ For instance, two pairs of linearly independent vectors Ôx1, y1 and x2, y2 rep- resent the same line if and only if they generate the same two-dimensional subspace.

To verify that, we, for instance, may check whether

 

rank x1β y1β x2β y2β 2 (7.17)

3 where we write all the four vectors x1, y1,x2, y2 w.r.t. a basis β of R . Yet, there is another quite convenient way how to represent a two dimensional

3

subspace in R .Since3 2 1, we can find for each two-dimensional subspace, spec- Õ ified by a basis Ôx, y , exactly one one-dimensional subspace of the three-dimensional dual linear space. Call the basis of this new one-dimensional subspace l. Then there

holds

   lβ¯ xβ yβ 0 (7.18) where β¯ is the dual basis to β. Therefore, we can represent lines in the real projective plane by one-dimensional subspaces in this way. We have developed an interesting representation of points and lines where both points and lines are represented by one-dimensional subspaces of R3.Pointsare R3 represented by one-dimensional subspaces of V , which is connected by ϕ to

75 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

A3 A2

O

l σ

Figure 7.6: The ideal line is the set of all projective points (i.e. all lines of A3 through C, which have no intersection with A2.Itisaplaneσ.Thereisexactly

one, which is perpendicular to sigma, which is generated by vector l . the three-dimensional space A3 of the geometrical model of the real projective plane. The lines are represented by one-dimensional subspaces of the space V¯ ,whichisthe space dual to V .Usingthebasisβ¯ in V¯ , which is dual to basis β in V , the coordinates 3 lβ¯ as well as coordinates of xβ become vectors in R which satisfy Equation 7.18. The line of A3 generated by l in Figure 7.5 is shown as perpendicular3 to the plane generated by x, y. Indeed, in the geometrical model of the real projective plane, we can use the notion of perpendicularity to uniquely construct the (perpendicular) line to the plane corresponding to l in A2.

7.2.4 Ideal line The set of all one-dimensional subspaces of R3, which do not correspond to points in the affine plane, i.e. the set of all ideal points, forms itself a two-dimensional subspace of R3 an hence a line in the projective plane, which is not in the affine plane. It is the ideal line4 of the projective plane associated with the selected affine

plane in that projective plane. It is represented by vector l in Figure 7.6. For each affine plane, there is exactly one ideal line (a two-dimensional sub- space of R3). Conversely, by selecting one line in a projective plane (i.e. one two- dimensional subspace of R3) the associated affine plane is determined as the set of all points (one-dimensional subspaces of R3) which are not contained in the selected ideal line (two-dimensional subspace).

3In A3, line and plane are perpendicular when they contain the right angle. The right angle is one quarter of a circle. 4Nevlastn´ıpˇr´ımka in Czech, line at infinity in [16].

76 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

7.2.5 Homogeneous coordinates Once a coordinate system is fixed in an affine plane, every point of the affine plane has unique coordinates, which are the coordinates of its vector in the basis of the coordinate system. A point in a real projective plane is represented by a one-dimensional subspace of R3. One dimensional subspaces are represented by their bases consisting of a single non-zero vector. There are infinitely many bases representing the same one- dimensional subspace. Two basic vectors of the same one-dimensional subspace are related by a non-zero multiple. Hence, when talking about coordinates of a point in the projective space, we actually talk about coordinates of a particular basic vector of the one-dimensional subspace that represents the point.

For instance, vectors

  

 1 2

    0 and 0 (7.19) 1 2 are basic vectors of the same one-dimensional subspace since they are related by a non-zero multiple. These are two different “coordinates” of the same point in the real projective plane. Hence, the “coordinates” of a point in the real projective plane are not unique. This is so radically departing from the fundamental property of coordinates, their uniqueness, that it deserves a new name. To distinguish the coordinates of a point in the affine plane, which are unique, from the “coordinates” of a point in the projective plane, which are not unique, we shall introduce new name homogeneous coordinates. Homogeneous coordinates of a point in the real projective plane are the coordi- nates of a basic vector of the one-dimensional subspace, which represents the point. Homogeneous coordinates of a line in the real projective plane are the coordinates of a basic vector of the one-dimensional subspace, which represents the line. A point in an affine plane can be represented by affine as well as by homogeneous coordinates. Let us see the relationship between the two. Let us have a point X in a two-dimensional real affine plane, which is represented

by coordinates

x (7.20) y By extending the real affine plane to the real projective plane with the ideal line

identified with the two-dimensional subspace z 0, we can represent point X by a

one-dimensional subspace of R3 generated by its basic vector 

 x   y (7.21)

1

    Â

Thus, X has affine coordinates xy and homogeneous coordinates uvw ,

È R  where u λx, v λy,andw λ 1forsomeλ , λ 0.

Ideal points do not have affine coordinates. Their homogeneous coordinates are

  xy0 Â (7.22)

77 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

A2 A3

l

l x

O

Figure 7.7: A point x is incident with a line l if and only if it can generate the line with another point y. Lines in A3 representing th point and the line are

perpendicular to each other.

R   where x, y È and either x 0ory 0. The zero vector 0 is not a basis of a one-dimensional space and thus represents neither a point nor a line.

7.2.6 Incidence of points and lines We say that a p oint x is incident with line l if and only if it can generate the line with another point y, Figure 7.7. In the representation of subspaces of R3,itmeans that  lβ¯ xβ 0 (7.23) This means that the one-dimensional subspace of R3 representing the line is orthogo- nal to the one-dimensional subspace of R3 representing the point w.r.t. the standard (Euclidean) scalar product. In the geometrical model of th real projective plane it means that the line of A3 representing x is perpendicular to line of A3 representing l. Let us write explicitly the coordinates of the bases generating the one-dimensional

subspaces as

  

 x a

  



xβ y lβ¯ b z c

then we get

ax by cz 0

and for affine points represented with z 1 this formula reduces to

ax by c 0 which is the familiar equation of a line in the two dimensional real affine plane.

78 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

A2 A3

l

y x l

O

Figure 7.8: The join of two distinct points is the unique line passing through them.

7.2.7 Join of points Every two distinct points x and y in a projective plane are incident to exactly one line l.Thejoin of two distinct points is the unique line passing through them. In the real projective plane, two distinct points are represented by two different one-dimensional subspaces with bases x and y. The homogeneous coordinates of this line, i.e. the coordinates of the basic vectors of the one-dimensional subspace representing the line, can be obtained by solving the following system of homogeneous equations for coordinates of the vector l

 lβ¯ xβ 0 (7.24)

 lβ¯ yβ 0 (7.25) (7.26) w.r.t. β and β¯ in R3. The set of solutions forms the one-dimensional subspace, which represents the line l. To find the solution, we shall rewrite the above set of equations in the matrix

form as

 

a

x1 x2 x3

 

b 0 (7.27) y1 y2 y3

c

     

 Â

Â

with lβ¯ abc , xβ x1 x2 x3 ,andyβ y1 y2 y3 . Next, we shall construct two special matrices of rank smaller than three and

develop formulas for their determinants, which equal zero

§ § § §

   

§ § §

§ x1 x2 x3 x1 x2 x3

§ § § §

§ § § §

   

y1 y2 y3 0 y1 y2 y3 0 (7.28)

§ § § §

§ § § § x1 x2 x3 y1 y2 y3

79 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

to get

¡ Õ Ô ¡ Õ Ô ¡ Õ

x1 Ôx2 y3 x3 y2 x2 x3 y1 x1 y3 x3 x1 y2 x2 y1 0 (7.29)

¡ Õ Ô ¡ Õ Ô ¡ Õ y1 Ôx2 y3 x3 y2 y2 x3 y1 x1 y3 y3 x1 y2 x2 y1 0 (7.30)

which can be rewritten as

 

x2 y3 ¡ x3 y2

x1 x2 x3

 

x3 y1 ¡ x1 y3 0 (7.31) y1 y2 y3

x1 y2 ¡ x2 y1 Since x and y represent distinct points, they are independent, and therefore the

solutions form a one-dimensional space. We can therefore conclude that

 

x2 y3 ¡ x3 y2 



¡ È R ÞØ Ù lβ¯ λ x3 y1 x1 y3 ,α 0 (7.32)

x1 y2 ¡ x2 y1 and choose one basic vector of the one-dimensional space of solutions by setting

λ 1

 

x2 y3 ¡ x3 y2 

 ¡ lβ¯ x3 y1 x1 y3 (7.33)

x1 y2 ¡ x2 y1 Interestingly, notice that the we have derived the formula for the vector product in R3

¢ lβ¯ xβ yβ (7.34) Notice, that in the real projective plane as well as in the real affine plane, there is exactly one line incident to two distinct points.

7.2.8 Meet of lines Every two distinct lines k and l in a projective plane are incident exactly to one point x.Themeet of two distinct lines is the unique point incident with them. In the real projective plane, two distinct lines are represented by two different one-dimensional subspaces with bases k and l. The homogeneous coordinates of this point, i.e. the coordinates of the vectors in the one-dimensional subspace representing the point, can be obtained by solving the following system of homogeneous equations for coordinates of the vector x w.r.t. β in R3

 kβ¯ xβ 0

 lβ¯ xβ 0

The set of solutions forms the one-dimensional subspace, which represents point x. To get one basic vector in the subspace, we may again employ the vector product in R3 and compute

¢ xβ kβ¯ lβ¯

80 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

A2 A3

l l

x k

k O

Figure 7.9: The meet of two distinct lines is the unique point incident with them.

Notice, that in the real projective plane there is exactly one point incident to two distinct lines. This is not true in an affine plane because there are (parallel) lines that have no point in common!

7.3 Line coordinates under homography

Let us now investigate the behavior of homogeneous coordinates of lines in projective plane mapped by a homography. Let us have two points represented by vectors xβ, yβ.Wenowmapthepoints,

represented by vectors xβ, yβ, by a homography, represented by matrix H,topoints

½ ½ R 

represented by vectors xβ, yβ such that there are λ1,λ2 È ,λ1λ2 0 ½

λ1 xβ H xβ (7.35) ½

λ2 yβ H yβ (7.36)

Homogeneous coordinates pβ of the line passing through points represented by xβ,

yβ and homogeneous coordinates pβ½ of the line passing through points represented ½

by xβ½ , yβ are obtained by solving the linear systems

Â

 ½ ½

pβ xβ 0andpβ xβ 0 (7.37)

Â

 ½ ½

pβ yβ 0 pβ yβ 0 (7.38)

for a non-trivial solutions. Writing down the incidence of points and lines, we get

 ¡1 ½  ¡1 ½

*

λ1 pβ H xβ 0 pβ H xβ 0

 ¡1 ½  ¡1 ½

*

λ2 pβ H yβ 0 pβ H yβ 0 ¡Â We see that pβ½ and H pβ are solutions of the same set of homogeneous equations. R

When xβ, yβ are independent, then there is λ È such that

½ ¡Â

λpβ H pβ (7.39)

81 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

π

v

C y

l x L Y

k τ

K X

Figure 7.10: Vanishing point v is the point towards projections x an y tend as X and Y move away from π but which they never reach. since the solution space is one-dimensional. Equation 7.39 gives the relationship between homogeneous coordinates of a line and its image under homography H.

7.4 Vanishing points

When modeling perspective projection in the affine space with affine projection planes, we meet somewhat unpleasant situations. For instance, imagine a projection of two parallel lines K, L, which are in a plane τ in the space into the projection plane π through the center C, Figure 7.10. The lines K, L project to image lines k, l. As we go with two points X, Y along the lines k, l away from the projection plane, their images x, y get closer and closer to the point v in the image but they do not reach point v. We shall call this point of convergence of lines K, L the vanishing point5.

7.5 Vanishing line and horizon

If we take all sets of parallel lines in τ, each set with a different direction, then all the points of convergence in the image will fill a complete line h. The line h is called the vanishing line or the horizon6 when τ is the ground plane. Now, imagine that we project all points from τ to π using the affine geometrical projection model. Then, no point from τ will project to h. Similarly, when projecting in the opposite direction, i.e. π to τ, line h has no image, i.e. it does not project anywhere to τ. When using the affine geometrical projection model with the real projective plane to model the perspective projection (which is equivalent to the algebraic model in R3), all points of the projective plane τ (obtained as the projective completion of

5Ubˇ´ eˇzn´ık in Czech. 6Horizont in Czech

82 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

π

h C

τ

Figure 7.11: Vanishing line (horizon) h is the line of vanishing points. the affine plane τ) will have exactly one image in the projective plane π (obtained as the projective completion of the affine plane π) and vice versa. This total symmetry is useful and beautiful.

7.6 Incidence axioms of projective plane

The two properties which we have realized above can be taken as a starting point for developing a more general theory of projective planes. The incidence axioms of the projective plane are a generalization of the incidence axioms of the synthetic

affine plane, Section 3.7.

P L Õ P Let P be a set. We define triple Ô , of points exp P and L exp P . Points and lines are subsets of P. In particular, points are singleton sets, i.e. the P

sets containing exactly one element. We say that point p È is incident [12] with

L ¥  À

(lies in, is on, belongs to, is contained in) line l È ,andwritep l when p l ,

E  À È L

Similarly, p is incident with π È when p π .Linek intersects (meets)

L  À

line l È ,whenk l . P L Õ § 40 Synthetic projective plane We say that Ô , is a synthetic projective plane when the following incidence axioms of projective plane [13] hold for the triple.

PP1 Every two distinct points are incident with exactly one line.

PP2 Every two distinct lines are incident with exactly one point.

PP3 There are at least three distinct lines and there are at least three distinct points. Every line is incident with at least three distinct points and every point is incident with at least three distinct lines.

83 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

P1

l1 l2 l3 l4 l5 l6 l7

¥ ¥

P1 ¥

¥ ¥ P2 ¥

P2 P3

¥ ¥

P3 ¥ P4

¥ ¥

P4 ¥

¥ ¥

P5 ¥

¥ ¥

P6 ¥

¥ ¥ P7 ¥ P5 P7 P6 (a) (b)

P1 P2 P3 P4 P5 P6 P7

            

 0 1 0 1 1 1 0

              0 0 1 1 0 1 1 1 1 1 1 0 0 0

l1 l2 l3 l4 l5 l6 l7

            

 0 1 1 1 1 0 0

              1 0 1 1 0 1 0 0 0 0 1 1 1 1 (c)

Figure 7.12: Fano plane – the seven-point projective plane. (a) Incidence table and (b) the corresponding diagram. (c) Linear representation by Z3.

Now, notice that when we exchange word “point” for word “line” in PP1, we get PP2 and when we exchange word “line” for word “point” in PP2, we get PP1. When we exchange words “line” and “point” in PP3, we get again PP3. This is interesting but also useful since from any true statement about points, lines and their incidence derived from the three axioms, we can get another true axiom just by exchanging words “line” and “point”. Hence it is sufficient to prove only “half” of theorems in such a theory. This phenomenon is called em duality in the synthetic projective plane. This duality is related to the notion of dual linear

space since when the synthetic projective plane can be modeled by subspaces of a Â

linear space, then the incidence is modeled by l x 0 and we can interpret it as Â

Â

lβ¯ xβ 0orasxβ¯ lβ 0. P L Õ Let us now present the smallest incidence structure Ô , satisfying axioms PP1, PP2 and PP3 – the seven point and seven line Fano plane, Figure 7.12. We have extended the incidence table of the four-point affine plane from Fig- ure 3.7 by adding the ideal line l7 and three ideal points P5, P6 and P7 on l7. The new incidence table, Figure 7.12(a), clearly shows that there are always three distinct points incident with each line and that each point is incident with three distinct lines. Axioms PP1 and PP2 clearly hold. The plane can be represented by one-dimensional subspaces of Z3, Figure 7.12(c), which reveals that every non-zero 3 vector in Z represents exactly one point and exactly one line , e.g. P1 and l7.

84 8 Projective space

8.1 Motivation – union of ideal points of all affine planes

Figure 8.1(a) shows a perspective image of three sets of parallel lines generated by sides of a cube in the three-dimensional real affine space. The images of the three of sets of parallel lines converge to vanishing points V1, V2 and V3. The cube has six faces. Each face generates two pairs of parallel lines and hence two vanishing points. Each face generates an affine plane which can be extended into a projective plane

by adding a line of ideal points of that plane. The projection of the three ideal lines

   are vanishing lines l12 V1 V2, l23 V2 V3 and l31 V3 V1. Imagine now all possible affine planes of the three-dimensional affine space and their corresponding ideal points. Let us take the union V of the sets of ideal points of all planes. There is exactly one ideal point for every set of parallel lines in V , i.e. there is in one- to-one correspondence between elements of V (ideal points) and directions in the three-dimensional affine space. Notice also that every plane π generates one ideal

line l of its ideal points and that all other planes parallel with π generate the same

l , Figure 8.1. It suggests itself to extend the three-dimensional affine space by adding the set V to it, analogically to how we have extended the affine plane. In this new space, all parallel lines will intersect. We will call this space the three-dimensional real projective space and denote it P3. Let us develop an algebraic model of P3.Itis practical to require this model to encompass the model of the real projective plane. The real projective plane is modeled algebraically by subspaces of R3.Letusobserve that subspaces of R4 will be a convenient algebraic model of P3. We start with the three-dimensional real affine space A3 and fix a coordinate

3

Õ Ô Õ A

system ÔO, δ with δ d1, d2, d3 .Anaffineplaneπ is a set of points of Õ

represented in ÔO, δ by the set of vectors

 2 2 2

×  È R  Ù

π ØÖx, y, z ax by cz d 0, a,b,c,d ,a b c 0 (8.1)

 ×

We see that the point of π represented by vector Öx, y, z can also be represented

 4

Ö ×  È R Ù R by one-dimensional subspace Øλ x, y, z, 1 λ of and hence π can be seen

as the set

  2 2 2

Ö ×  È R ÙÖ ×Ö × È R  Ù π ØØλ x, y, z, 1 λ a, b, c, d x, y, z, 1 0, a,b,c,d ,a b c 0 (8.2) of one-dimensional subspaces of R4.

Notice that we did not require λ  0 in the above definition. This is because ×

we establish the correspondence between a vector Öx, y, z and the corresponding

 4

Ö × È R Ù R complete one-dimensional subspace Øλ x, y, z, 1 ,λ of and since every linear space contains zero vector, we admit zero λ.

3 3 Â

× È R Ô Õ A

Every Öx, y, z represents in O, δ a point of and hence the subset

3 Â

Ö ×  È R Ù È R Ù A ØØλ x, y, z, 1 λ x, y, z (8.3)

85 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

A3 V3

A2 l

π 

π 

V1 V2 (a) (b)

Figure 8.1: (a) A perspective image of a cube generates three vanishing points V1, V2 and V3 and hence also three vanishing lines l12, l23 and l31.(b)Every plane adds one line of ideal points to the three-dimensional affine space. Every ideal point corresponds to one direction, i.e. to a set of parallel lines. Each ideal line corresponds to a set of parallel planes. of one-dimensional subspaces of R4 represents A3. We observe that we have not used all one-dimensional subspaces of R4 to repre- sent A3. The subset

2 2 2 Â

ØØ Ö ×  È R Ù È R  Ù

π  λ x, y, z, 0 λ x, y, z ,x y z 0 (8.4) of one-dimensional subspaces of R4 is in one-to-one correspondence with all non- zero vectors of R3, i.e. in one-to-one correspondence with the set of directions in A3. This is the set of ideal points which we add to A3 to get the three-dimensional real

projective space

3 Â 2 2 2 2

Ö ×  È R Ù È R  Ù P ØØλ x, y, z, w λ x, y, z, w ,x y z w 0 (8.5)

4 3 3

A + R P

which is the set of all one-dimensional subspaces of .Noticethat π  .

§ 41 Points Every non-zero vector of R4 generates a one-dimensional subspace and

3 Â × thus represents a point of P . The zero vector Ö0, 0, 0, 0 does not represent any point.

§ 42 Planes Affine planes πA3 , Equation 8.2, are in one-to-one correspondence to

the subset

 2 2 2

Ö ×  È R Ù È R  Ù πA3 ØØλ a, b, c, d λ a, b, c, d ,a b c 0 (8.6)

of the set of one-dimensional subspaces of R4. There is only one one-dimensional sub-

4 Â

Ö ×  È R Ù space of R , Øλ 0, 0, 0, 1 λ missing in πA3 . It is exactly the one-dimensional P3

subspace corresponding to the set π  of ideal points of

 2 2 2 Â

ØØ Ö ×  È R Ù È R  Ö ×Ö × Ù

π  λ x, y, z, w λ x, y, z, w ,x y z 0, 0, 0, 0, 1 x, y, z, w 0 (8.7)

86 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

We can take another view upon planes and observe that affine planes are in one- R4

to-one correspondence to the three-dimensional subspaces of .Thesetπ  also R4

corresponds to a three-dimensional subspace of . Hence π  can be considered another plane, the ideal plane of P3. The set of planes of P3 can be hence represented by the set of one-dimensional subspaces of R4

2 2 2 2 Â

Ö ×  È R Ù È R  Ù πP3 ØØλ a, b, c, d λ a, b, c, d ,a b c d 0 (8.8) but can also be viewed as the set of three-dimensional subspaces of R4. We see that there is a duality between points and planes of P3.Theyboth are represented by one-dimensional subspaces of R4 and we see that point X rep-

Â

Ö ×

resented by vector X x, y, x, w is incident to plane π represented by vector

Â

× ¥

π Öa, b, c, d , i.e. X π,when 

 x



 y

Â

π X abcd ax by cz dw 0 (8.9)   z w

§ 43 Lines Lines in P3 are represented by two-dimensional subspaces of R4. Unlike in P2, lines are not dual to points.

8.2 Perspective camera in projective space

Perspective camera model, Equation 5.11, is actually capable of working in P3.First, Õ we used Equation 5.11 with xβ Ô3 1 to describe the perspective projection from

A3 to A2. Then we have extended the real affine plane into the real projective plane Õ and allowed xβ Ô3 0 to project points from the camera principal plane into ideal points in the image. Finally, now, we will use the same model to project all points, 3 2 represented by X Δ,fromP into points, represented by xβ,inP

ηxβ P XΔ (8.10)

4 4

3 3¢

R  È R È R È R  with η È , η 0, xβ , P ,rankP 3andXΔ , XΔ 0. The left side of Equation 8.10 is identical with Equation 5.11. This is because we represented points in images right from the beginning by three-dimensional vectors in the camera coordinate system. The difference is on the right.

Vector X Δ can be understood as



 X δ 

x  w ,w 0

1

y

XΔ (8.11)

 

z Yδ

w t ,t 0 0

3 3 where X δ represents a point of A in basis δ and Yδ represents a direction in A in basis δ.

87 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

X

v

v0

p e3 π f0

C0 e1 f

e2 Õ

C Ôf ,

Figure 8.2: Point X is projected to v0 from the camera center C0,whichisf0 far Õ from the projection plane π. When moving camera center C Ôf away from π along the line perpendicular to π and passing through C0,the projection of X tends to (but never reaches) point v.Pointv is obtained by projecting X along the ray p, which is parallel to the direction of e3. This parallel projection is equal to the perspective projection model

when considering the ideal projection center C Ô,Õ.

This construction partially determines basis Δ, which can be then written as

¡ ¡ © ©

 Δ u d1,ud2,ud3, d4 with δ d1, d2, d3 and u 0 (8.12)

Scalar u allows for a global change of scale on affine points and directions. We can

write vectors of δ in Δ as

    

 u 0 0

0 u 0

d1Δ d2Δ d3Δ (8.13)

      0 0 u 0 0 0

8.2.1 Parallel projection In P3 we can generalize the interpretation of the camera projection matrix to model

parallel projection. Consider a camera projection matrix, Equation 5.52



1 1 ¡ P f KR f KRCδ (8.14)

with elements of K given by Equations 5.44, 5.43, 5.41 and Equation 5.45 

  

Ô ,b Õ f f cos b1 2

 ¡    k13

sin Ô ,b Õ

b1 b1 b1 2

f

K (8.15)

0    k23

sin Ô ,b Õ

b2 b1 2

  001

88 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

To simplify the formulas, we introduce

Ô Õ

1 cos  b1, b2 1

¡ a ,b ,c (8.16)

   Ô Õ   Ô Õ b1 b1 sin b1, b2 b2 sin b1, b2

and write the projection equation 5.46 a follows

 

 fa fb k13

1 X δ 



¡ ηxβ 0 fc k23 R I  Cδ (8.17) f 1 00 1

Figure 8.2 shows that the parallel projection of point X along the direction e3, which is perpendicular to the projection plane π, corresponds to the perspective

projection of X with the camera center C Ô,Õ at the ideal point corresponding to

the direction e3. Let us see an algebraic justification for this claim.

Ö × We use the camera orthonormal coordinate system with basis  e1,e2,e3 , Figure 5.4, and parametrize the projection center Cδ, expressed in the world basis

δ,bythefocallengthf 

 0

¡1 



Õ Ô Õ¡ Cδ Ôf Cδ f0 R 0 (8.18)

f ¡ f0 1 Rotation matrix R ¡ transforms coordinates from  to δ, Equation 5.56, and hence

its third column equals e3δ. Parameter η in Equation 8.17 is the distance of X from Õ C Ôf along e3 in multiplies of f. Therefore, we can write

f g

η (8.19) f

where g is the distance of X from π. Coming back to Equation 8.17, we are getting

§

     

§

fa fb k13 0 §

f g 1 X δ

¡1

§

    



Ô Õ xβ 0 fc k23 R I ¡Cδ f0 R 0 (8.20)

f f § 1 §

00 1 f ¡ f0

§

         

§

abk13 ßf 1 1 0 §

f g X δ

§

        



ß Ô Õ xβ 0 ck23 f 1 R ¡ 1 R Cδ f0 0 § f f 1 1 f ¡ 0 1 00 1 f § f f

We can now compute the limit of the equation for f going to ,

§

         

§

ab0 1 1 0

§ X δ

§

        



¡ Ô Õ xβ 0 c 0 1 R 1 R Cδ f0 0 § 1

001 0 § 0 1

§

   

 Â

§

Ô Õ ab0 r1 ¡r1 Cδ f0

§ X δ

Â

Â

§

   

Ô Õ xβ 0 c 0 r2 ¡r Cδ f0

§ 2 1 Â

001 0 § 1

§

   

 Â

§

Ô Õ ab¡r1 Cδ f0 r1 0

§ X δ

Â

Â

§

   

Ô Õ xβ 0 c ¡r Cδ f0 r2 0 (8.21)

2 § 1 Â 00 1 0 § 1

89 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

d3 X d2 O

d1 x0 e3 v

e2 e1 p π C

Figure 8.3: The image plane of the parallel projection, which is perpendicular to e3, can be placed anywhere, e.g. to pass through the origin O of the world coordinate system. The image coordinate system is placed to the

perpendicular projection of C into π. Â where r1Â , r2 are the first two rows of R.

Let us now investigate the meaning of Equation 8.21. First of all observe that

§

     

 Â

§

Ô Õ 0 ab¡r1 Cδ f0 r1 0

§ r3

Â

Â

§

     

Ô Õ 0 0 c ¡r Cδ f0 r2 0 (8.22)

2 § 0 Â 0 00 1 0 § 1

where r3Â is the third row of R. We see that the camera center is an ideal point. To

¡1 Â

see which point it is, we realize that R is a rotation matrix and therefore R R .

Hence, r1 e1δ, r2 e2δ and r3 e3δ. That leads to

e3δ Cδ Ô,Õ (8.23) 0 which is an ideal point representing the direction r3δ in the world coordinate system. Secondly, notice that f, k13 and k23 disappeared from the projection Equa- tion 8.21. They do not have any geometrical meaning when considering parallel

projection and hence can’t be meaningfully defined in that case. Â

Â

Õ Ô Õ Finally, observe that quantities r1 Cδ Ôf0 and r2 Cδ f0 are the first two coor-

Õ

dinates of C Ôf0 in basis . We can thus define vector

 

Â Õ r1 Cδ Ôf0

x0 (8.24)

Â Õ r2 Cδ Ôf0 which represents the origin of the image coordinate system, which is placed into the perpendicular projection of C into π, Figure 8.3. Parallel projection can be represented by the same formula as perspective pro- jection by allowing the projection center to become an ideal point. We can therefore associate parallel projections of points represented in an orthonormal worked coor-

3

dinate system of A with the of 3 ¢ 4 matrices P with rank P 3, which have the × third row equal to Ö0, 0, 0, 1 .

90 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

§ 44 Pluecker coordinates

91 9 Camera Auto-Calibration and Metrology

Camera auto-calibration is a process when the parameters of images formation are determined from properties of the observed scene or knowledge of camera motions. We will study camera auto-calibration methods and tasks related to metrology in images.

9.1 Internal end external calibration

Considering camera projection matrix, Equation 5.52,



1 1 ¡ P f KR f KRCδ (9.1) it is customary [16] to distinguish internal calibration parameters K and f from external calibration parameters R and Cδ. This is motivated by the frequent situation when a camera with fixed internal calibration parameters is moving in the space. We have seen in Section 5.4 that to measure the angle between projection rays

we needed only matrix K. Actually, it is enough to know matrix1

¡Â ¡1

ω K K

to measure the angle between the rays corresponding to image points x1β, x2β as

¡Â ¡ Â Â 1

x1β K K x2β x1β ωx2β

 

Ô Õ

cos  x1,x2 (9.2) ¡

¡1 1

   Â K x1β K x2β Â x1β ωx1β x2β ωx2β

Knowing ω is however (almost) equivalent to knowing K since we can K be recovered from ω up to two signs.

§ 45 Recovering K from ω Let us give a procedure for recovering K from ω.As-

suming

 

k11 k12 k13

 

K 0 k22 k23 (9.3) 001

we get

 

k k k ¡k k

1 ¡

  

 12 12 23 13 22

¡1

k11 k12 k13 k11 k11k22 k11 k22 k23 m11 m12 m13

1 ¡ k

1 ¡ 23

   

K 0 k22 k23 0 0 m22 m23   k22 k22 001 00 1 001 (9.4)

1In [16], ω is called the the image of the absolute conic.

92 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected])

for some real m11,m12,m13,m22 and m23. Equivalently, we get

 

m m m ¡m m 1 ¡ 12 12 23 13 22

m11 m11m22 m11 m22 m23

m

1 ¡ 23

 K 0 m22 m22  (9.5) 00 1

Introducing the following notation

 

o11 o12 o13

¡Â ¡1

 

ω K K o12 o22 o23 (9.6) o13 o23 o33

yields

    2 o11 o12 o13 m11 m11 m12 m11 m13

2 2

   

o12 o22 o23 m11 m12 m12 m22 m12 m13 m22 m23 (9.7)

2 2

o13 o23 o33 m11 m13 m12 m13 m22 m23 m13 m23 1

1 ¡1

which can be solved for K ¡ up to the sign of the rows of K as follows

    2 o11 o12 o13 m11 m11 m12 m11 m13

2 2

   

o12 o22 o23 m11 m12 m12 m22 m12 m13 m22 m23 (9.8)

2 2

o13 o23 o33 m11 m13 m12 m13 m22 m23 m13 m23 1 provides equations

2 



o11 m11 m11 s1 o11

 

 ßÔ Õ ß

o12 m11 m12 m12 o12 s1 o11 s1 o12 o11

 

 ßÔ Õ ß

o13 m11 m13 m13 o13 s1 o11 s1 o13 o11

 

2 2 2 2

 ¡ ¡ ß

o22 m12 m22 m22 s2 o22 m12 s2 o22 o12 o11 

2

 Ô ¡ ß Õß ¡ ß

o23 m12 m13 m22 m23 m23 s2 o23 o12 o13 o11 o22 o12 o11 

2 2

Ô ¡ Õß ¡

s2 o11 o23 o12 o13 o11o22 o11 o12 ¨

which can be solved for mij with s1 ¨1ands2 1. Hence

  

 

¡1 ß

s1 o11 s1 o12 ß o11 s1 o13 o11

2 2

¡ ß Ô ¡ ß Õß ¡ ß  K  0 s2 o22 o12 o11 s2 o23 o12 o13 o11 o22 o12 o11 00 1

Signs s1, s2 are determined by the choice of image coordinates system. The standard choice is s1 s2 1,

9.2 Single-view camera calibration

9.3 Single-view metrology 9.4 Multiple-view camera calibration

93 Bibliography

[1] Paul R. Halmos. Naive Set Theory. Springer-Verlag, 1974.

[2] Walter Rudin. Principles of Mathematical Analysis. McGraw-Hill, 1976.

[3] Paul R. Halmos. Finite-Dimensional Vector Spaces. Springer-Verlag, 2000.

[4] Carl D. Meyer. Matrix Analysis and Applied Linear Algebra. SIAM: Society for Industrial and Applied Mathematics, Philadelphia, PA, USA, 2001.

[5] Petr Olˇs´ak. Uvod´ do algebry, zejm´ena line´arn´ı. FEL CVUT,ˇ Praha, 2007.

[6] Pavel Pt´ak. Introduction to linear algebra. Vydavatelstv´ı CVUT,ˇ Praha, 2007.

[7] Gilbert Strang. Introduction to Linear Algebra. Wellesley Cambridge, 3rd edi- tion, 2003.

[8] Seymour Lipschutz. 3,000 Solved Problems in Linear Algebra. McGraw-Hill, 1st edition, 1989.

[9] Jim Hefferon. Linear Algebra. Virginia Commonwealth University Mathematics, 2009.

[10] Michael Artin. Algebra. Prentice Hall, Upper Saddle River, NJ 07458, USA, 1991.

[11] Saunders MacLane and Garret Birkhoff. Algebra. Chelsea Pub. Co., 3rd edition, 1988.

[12] R.J. Mihalek. Projective Geometry and Algebraic Structures. Accademic Press, Inc., New York, 1972.

[13] Mary Katherine Bennett. Affine and Projective Geometry. John Willey and Sons, New York, USA, 1995.

[14] David Cox, John Little, and Donald O’Shea. Ideals, Varieties, and Algorithms: An Introduction to Computational Algebraic Geometry and Commutative Alge- bra. Springer, 2nd edition, 1996.

[15] David Cox, John Little, and Donald O’Shea. Using Algebraic Geometry. Springer, 1998.

[16] Richard Hartley and Andrew Zisserman. Multiple View Geometry in Computer Vision. Cambridge, 2nd edition, 2003.

94 Index affine coordinate system, 20 join, 79 affine function, 12 affine plane, 25 Kronecker product, 8 affine space, 18 line at infinity, 76 axioms of linear space, 15 linear model of affine space, 25 axioms of affine space, 19 linear function, 12 basis, 15 linear space, 15 bound vector, 14 marked ruler, 12 camera pose, 36 meet, 80 camera cartesian coordinate system, 36 omnidirectional image, 60 camera calibration matrix, 36 origin of affine coordinate system, 20 camera coordinate system, 32 camera matrix, 33 panoramic image, 61 coordinates, 15 parallel lines, 24 partition, 17 dual basis, 6 perspective camera, 29 dual space, 6 point at infinity, 73 external calibration parameters, 93 principal plane, 30 principal point, 38 focal length, 36 projection center, 30 free vector, 17 projective space, 71 geometric scalars, 13 real projective plane geometric vector, 13 affine point, 72, 74 algebraic model, 73 homogeneous coordinates, 77 geometrical model, 72 homogeneous coordinates of a line, 77 ideal point, 73, 74 homogeneous coordinates of a point, line, 75 77 point, 72 homography, 54 horizon, 82 sphericalimage,60 syntheticaffinespace,24 ideal line, 76 synthetic projective plane, 83 ideal plane, 87 image plane, 30 three-dimensional real projective space, incidence, 24, 83 85 incidence axioms of affine space, 24 trivial affine space, 22 incidence axioms of projective plane, 83 vanishing point, 82 internal calibration parameters, 93 vanishing line, 82 intersection, 24, 83 world coordinate system, 30

95 T. Pajdla. Geometry of Computer Vision 2011-5-2 ([email protected]) world unit length, 43 zero bound vector, 14

96