Computer Vision I - Projective Geometry and a Camera
Carsten Rother
22/12/2014
Computer Vision I: Image Formation Process Roadmap for next five lecture • Appearance based matching (sec. 4.1)
• How do we get an RGB image? (sec 2.2-2.3) • The Human eye • The Camera and Image formation model
• Projective Geometry - Basics (sec. 2.1.1-2.1.4)
• Geometry of a single camera (sec 2.1.5, 2.1.6) • Pinhole camera • Lens effects
• Geometry of two cameras (sec. 7.2)
• Robust Geometry estimation for two cameras (sec. 6.1.4)
• Multi-View 3D reconstruction (sec. 7.3-7.4)
Computer Vision I: Image Formation Process 22/12/2014 2 Some Basics
2 1 • Real coordinate space 푅 example: 2 3 1 • Real coordinate space 푅 example: 2 3 • Operations we need are: scalar product: 풕 푥1 풙 풚 = 풙 풚 = 푥1푦1 + 푥2푦2 + 푥3푦3 where x = 푥2 푥3 cross/vector product: 풙 × 풚 = 풙 × 풚
0 −푥3 푥2 풙 × = 푥3 0 −푥1 −푥2 푥1 0
Computer Vision I: Image Formation Process 22/12/2014 3 Euclidean Space • Euclidean Space 푅2 or 푅3 has angles and distances defined
• Angle defined as:
• Length of the vector 푥: = 푥 푥
푥 • Distance: 푥 − 푦
Θ 푦
표푟푖푔푖푛
Computer Vision I: Image Formation Process 22/12/2014 4 Projective Space
• 2D Point in a real coordinate space: 1 2 2 ∈ 푅 has 2 DoF (degrees of freedom) • 3D Point in a real coordinate space:
1 3 2 ∈ 푅 has 3 DoF 3
• Definition: A point in 2-dimensional projective space 푃2 is defined as 푥 2 푘푥 푝 = 푦 ∈ 푃 , such that all vectors 푘푦 ∀ 푘 ≠ 0 푤 푘푤 define the same point 푝 in 푃2 (equivalent classes)
1 2 • Sometimes written as: 2 ~ 4 1 2 1 2 • We write as: 2 = 4 1 2
Computer Vision I: Image Formation Process 22/12/2014 5 Projective Space - visualization
Definition: A point in 2-dimensional projective space 푃2 is defined as 푥 2 푘푥 푝 = 푦 ∈ 푃 , such that all vectors 푘푦 ∀ 푘 ≠ 0 푤 푘푤 define the same point 푝 in 푃2 (equivalent classes)
A point in 푃2 is a ray in 푅3 that goes through the origin:
푥 w-axis 푦 푤 All rays go through (0,0,0) define a point in 푃2
Plane w=1
Plane w=0 푥 푦 0
Computer Vision I: Image Formation Process 22/12/2014 6 Projective Space
2 3 0 • All points in 푃 are given by: 푅 \ 0 0
푥 2 • A point 푦 ∈ 푃 has 2 DoF (3 elements but norm of vector can be set to 1) 푤
푥 w-axis 푦 푤 All rays go through (0,0,0) define a point in 푃2
Plane w=1
Plane w=0 푥 푦 0
Computer Vision I: Image Formation Process 22/12/2014 7 From 푅2 to 푃2 and back
• From 푅2 to 푃2: 푥 푥 푝~ = ∈ 푅2 푝 = 푦 ∈ 푃2 푦 1 - a point in inhomogeneous coordinates - a point in homogeneous coordinates - we soemtimes write ~푝 for inhomogeneous coordinates • From 푃2 to 푅2:
푥 푥/푤 푦 ∈ 푃2 ∈ 푅2 푤 푦/푤 푓표푟 푤 ≠ 0 what does it mean if w=0?
We can do this transformation with all primitives
Computer Vision I: Image Formation Process 22/12/2014 8 From 푅2 to 푃2 and back: Example • From 푅2 to 푃2: 푥 푥 2 푝~ = ∈ 푅2 푝 = 푦 ∈ 푃 푦 1 - a point in inhomogeneous coordinates - a point in homogeneous coordinates
3 ~ 2 3 4.5 6 2 6/2 푝 = ∈ 푅 푝 = 2 = 3 = 4 ∈ 푃 푝~ = ∈ 푅2 2 1 1.5 2 4/2
w-axis
(6,4,2)
(3,2,1) Plane w=1 (Space 푹ퟐ)
Plane w=0
Computer Vision I: Image Formation Process 22/12/2014 9 Things we would like to have and do
We look at these operations in: 푅2/푅3, 푃2/푃3
Primitives: • Points • Lines • Conics (Quadric in 3D) • (Planes in 3D)
Operations with Primitives: • intersection • tangent
Transformations: • Rotation • Translation • projective • ….
Computer Vision I: Image Formation Process 22/12/2014 10 Why bother about 푃2? • All Primitives, operations and transformations are defined in 푅2 and 푃2
• Advantage of 푃2: • Many transformation and operations are written more compactly (e.g. linear transformations) • We will introduce new special “primitives” that are useful when dealing with “parallelism”
This example will come 2 later in detail. In 푃 they meet in a „point at inifinty“
Two parallel lines in 푅2
Computer Vision I: Image Formation Process 22/12/2014 11 Points at infinity
푥 w-axis 푦 푤 All rays go through (0,0,0) define a point in 푃2
Plane w=1
Plane w=0 푥 푦 0 푥 Points with coordinate 푦 are ideal points or points at infinity 0
푥 푦 ∈ 푃2 Not defined in 푅2 푠푖푛푐푒 푤 = 0 0
Computer Vision I: Image Formation Process 22/12/2014 12 Lines in 푅2 • For Lines in coordinate space 푅2 we can write 푡 푙 = (푛푥, 푛푦, 푑) with 푛 = 푛푥, 푛푦 is normal vector and ||푛|| = 1
• A line has 2 DoF
• A point (푥, 푦) lies on 푙 if:
푛푥 푥 + 푛푦 푦 + 푑 = 0
• Normal can also be encoded with an angle 휃: 푛 = cos 휃 , sin 휃 푡
Computer Vision I: Image Formation Process 22/12/2014 13 Lines in 푃2 • Points in 푃2: 풙 = (푥, 푦, 푤) • Lines in 푃2: 풍 = (푎, 푏, 푐) (again equivalent class: 푎, 푏, 푐 = 푘푎, 푘푏, 푘푐 ∀ 푘 ≠ 0 ) Hence also 2 DoF
• All points 푥, 푦, 푤 on the line (푎, 푏, 푐) satisfy: 푎푥 + 푏푦 + 푐푤 = 0 this is the equation of a plane in 푅3 with normal (a,b,c) going through (0,0,0)
Computer Vision I: Image Formation Process 22/12/2014 14 Converting Lines between 푃2 and 푅2 • Points in 푃2: 풙 = (푥, 푦, 푤) • Lines in 푃2: 풍 = (푎, 푏, 푐)
From 푃2 to 푅2: 풍 = (푘푎, 푘푏, 푘푐) chose 푘 such that ||(푘푎, 푘푏)|| = 1
From 푅2 to 푃2: 2 풍 = 푛푥, 푛푦, 푑 is already a line in 푃
Computer Vision I: Image Formation Process 22/12/2014 15 Line at Infininty • There is a “special” line, called line at infinity: (0,0,1)
• All points at infinity 푥, 푦, 0 lie on the line at infinity (0,0,1): 푥 ∗ 0 + 푦 ∗ 0 + 0 ∗ 1 = 0
0 0 vector for line at infinity w-axis 1
Plane w=1
Plane w=0 푥 푦 A point at infinity (w=0) 0
Computer Vision I: Image Formation Process 22/12/2014 16 A Line defined by two points in 푃2 • The line through two points 풙 and 풙′ is given as: 풍 = 풙 × 풙’ • Proof: It is: 풙 풙 × 풙’ = 풙′ 풙 × 풙′ = ퟎ vectors are orthogonal This is the same as: 풙 풍 = 풙′풍 = ퟎ Hence, the line 풍 goes through points 풙 and 풙′
풙 풙′
Computer Vision I: Image Formation Process 22/12/2014 17 The Intersection of two lines in 푃2 • Intersection of two lines 풍 and 풍’ is the point 풙 = 풍 × 풍’ • Proof: It is: 풍 풍 × 풍’ = 풍′ 풍 × 풍’ = ퟎ vectors are orthogonal This is the same as: 풍풙 = 풍′풙 = ퟎ Hence, the point 풙 lies on the lines 풍 and 풍′
Note the „Theorem“ and Proofs have been very similiar, we only interchanged meaning of points and lines
Computer Vision I: Image Formation Process 22/12/2014 18 Duality of points and lines • Note 풍풙 = 풙풍 = ퟎ (풙 and 풍 are “interchangeable”)
• Duality Theorem: Two any theorem of 2D projective geometry there corresponds a dual theorem, which may be derived by interchanging the roles of points and lines in the original theorem.
The intersection of two lines 풍 and 풍’ is the point 풙 = 풍 × 풍’
The line through two points 풙 and 풙′ is the line 풍 = 풙 × 풙’
Computer Vision I: Image Formation Process 22/12/2014 19 Parallel lines meet in a point at Infinity
풍 1 2 풍 = 0 ; 풍′ = 0 풍’ 1 1
풍 풍’ In 푅2 (Plane 푤 = 1) intersection 푦 0 −1 0 2 0 ′ (-1,1,1) (-1/2,1,1) 풍 × 풍 = 1 0 −1 0 = 1 0 1 0 1 0 (-1,0,1) (-1/2,0,1) 푥 Point at infinty
Computer Vision I: Image Formation Process 22/12/2014 20 2D conic “Kegelschnitt” • Conics are shapes that arise when a plane intersects a cone • In compact form: 풙풕푪 풙 = ퟎ where C has the form: 푎 푏/2 푑/2 C = 푏/2 푐 푒/2 푑/2 푒/2 푓
• This can be written in in-homogenous coordinates: 푎푥2 + 푏푥푦 + 푐푦2 + 푑푥 + 푒푦 + 푓 = 0 where 풙~ = (푥, 푦) • 푪 has 5DoF since unique up to scale: 풙풕푪 풙 = 푘풙풕푪 풙 = 풙풕푘푪 풙 = ퟎ
• Properties: 풍 is tangent to 푪 at a point 풙 if 풍 = 푪풙
Computer Vision I: Image Formation Process 22/12/2014 21 Example: 2D Conic
y A circle: 2 2 2 r 푥 + 푦 − 푟 = 0 x
Parabola: −푥2 + 푦 = 0
Computer Vision I: Image Formation Process 22/12/2014 22 Define a conic with five points
Given 5 points (푥푖, 푦푖, 1) we can write:
as with
That gives:
This is a 5 × 6 matrix. The 1D null-space gives the conic up to scale. Compute null-space with Gaussian elimination or SVD (to come later)
Computer Vision I: Image Formation Process 22/12/2014 23 2D Transformations 2D Transformations in 푅2
Definition: • Euclidean: translation + rotation • Similarity (rigid body transform): Euclidean + scaling • Affine: Similarity + shearing • Projective: arbitrary linear transform in homogenous coordinates
Computer Vision I: Image Formation Process 22/12/2014 24 2D Transformations of points
• 2D Transformations in homogenous coordinates: 푥′ 푎 푏 푑 푥 푦′ = 푒 푓 ℎ 푦 1 푤′ 푖 푗 푙 Transformation matrix
• Example: translation
1 0 푡푥 푥 푥′ 푦 푥′ 푥 푡 푦′ = 0 1 푡푦 = + 푥 1 푦′ 푦 푡푦 1 0 0 1 homogeneous coordinates inhomogeneous coordinates
Advantage of homogeneous coordinates (going into 푃2)
Computer Vision I: Image Formation Process 22/12/2014 25 2D Transformations of points
(of a square)
(two special points on the line at infity )
Here 푟푖푗 is a 2 x 2 rotation matrix with 1 DoF cos Θ −sin Θ which can be written as: sin Θ cos Θ [from Hartley Zisserman Page 44]
Computer Vision I: Image Formation Process 22/12/2014 26 2D transformations of lines and conics All points move: 풙‘ = 푯풙 then:
풙′ 1) Line (defined by points) moves: 풙 ′ −1 t 풍 = (푯 ) 풍 푯
풍 풍′ 2) conic (defined by points) moves: 푪′ = (푯−1)t 푪 푯−1
Proof: 1) Assume 풙1 and 풙2 lie on 풍. Show that 풙’1, 풙’2 lie on 풍’. 푡 ! 푡 −ퟏ 푡 ! 풙’1 풍’ = 0 → 푯풙ퟏ 푯 풍 = 0 → 푡 풕 −ퟏ 풕 ! 푡 −ퟏ 풕 ! 푡 ! 풙1푯 푯 풍 = 0 → 풙1 푯 푯 풍 = 0 → 풙1풍 = 0
2) Homework.
Computer Vision I: Image Formation Process 22/12/2014 27 Example: Projective Transformation
Picture from top Affine transformation Picture from the side (projective transformation) 1. Circles on the floor are 1. Circles on the floor are 1. Lines converge to a circles in the image ellipse in the image vanishing point not at 2. Squares on the floor are 2. Squares on the floor are infinity squares in the image sheared in the image 3. Lines are still parallel
Computer Vision I: Image Formation Process 22/12/2014 28 In 3D: Points • 풙 = 푥, 푦, 푧 ∈ 푅3 has 3 DoF
• With homogeneous coordinates: 푥, 푦, 푧, 1 ∈ 푃3
• 푃3 is defined as the space 푅3 \ (0,0,0,0) such that points 푥, 푦, 푧, 푤 and 푘푥, 푘푦, 푘푧, 푘푤 are the same for all 푘 ≠ 0
• Points: 푥, 푦, 푧, 0 ∈ 푃3 are called points at infinity
Computer Vision I: Image Formation Process 22/12/2014 29 In 3D: Planes • Planes in 푅3 are defined by 4 paramters (3 DoF): • Normal: 푛 = 푛푥, 푛푦, 푛푧 • Offset: d
• All points (푥, 푦, 푧) lie on the plane if:
푥 푛푥 + 푦 푛푦 + 푧 푛푧 + 푑 = 0
• With homogenous coordinates:
풙 휋 = 0, where 푥 = (푥, 푦, 푧, 1) and 휋 = (푛푥, 푛푦, 푛푧, 푑)
• Planes in 푃3 are written as: 풙 휋 = 0
• Points and planes are dual in 푃3 (as points and lines have been in 푃2)
• Plane at infinity is 휋 = (0,0,0,1) since all points at infinity (x,y,z,0) lie on it.
Computer Vision I: Image Formation Process 22/12/2014 30 In 3D: Plane at infinity
Point at infinity Point at infinity
푧 line at infinity
푦
푥
All of these elements at infinity lie on the plane at infinity
Computer Vision I: Image Formation Process 22/12/2014 31 Why is the plane at infinity important (see later)
Plane at infinity is important to visualize 3D reconstructions nicely
Plane at infinity can be used to simplify 3D reconstruction
Computer Vision I: Image Formation Process 22/12/2014 32 What is the horizon?
The ground plane is special (we stand on it)
Horizon is a line at infinity where plane at infinity intersects ground plane
푧
푦 푥
Ground plane: (0,0,1,0) Many lines and planes in our real world meet Plane at infinity: (0,0,0,1) at the horizon (since parallel to ground plane)
Computer Vision I: Image Formation Process 22/12/2014 33 In 3D: Points at Infinity
• Parallel lines in 3D meet at a point at infinty
• Points at infinty can be real points in a camera
0 푏 푎 푏 푐 푑 푓 1 = 푒 푓 푔 ℎ 0 1 푖 푗 푘 푙 0 Real point in 3x4 Camera 3D Point at the image Matrix infinty 3D->2D projection
Computer Vision I: Image Formation Process 22/12/2014 34 In 3D: Lines • Unfortunately not a compact form (as for points)
• A simple representation in 푅3. Define a line via two points 푝, 푞 ∈ 푅3: 풓 = 1 − 휆 풑 + 휆 풒
• A line has 4 DoF (both points 풑, 풒 can move arbitrary on the line)
• A more compact, but more complex, way two define a 3D Line is to use Plücker coordinates:
푳 = 풑풒풕 – 풒풑풕 where det 푳 = 0 here 푳, 풑, 풒 are in homogenous coordinates
Computer Vision I: Image Formation Process 22/12/2014 35 In 3D: Quadrics • Points 푿 on the quadric if: 푿푻 푸 푿 = 0 • A quadric 푸 is a surface in 푃3 • A quadric is a symmetric 4 × 4 matrix with 9 DoF
Computer Vision I: Image Formation Process 22/12/2014 36 In 3D: Transformation
(of a cube)
Computer Vision I: Image Formation Process 22/12/2014 37 In 3D: Rotations Rotation 푹 in 3D has 3 DoF. It is slightly more complex, and several options exist:
1) Euler angles: rotate around, 푥, 푦, 푧-axis in order (depends on order, not smooth in parameter space)
2) Axis/angle formulation: 2 푹 풏, Θ = 푰 + sin Θ 풏 × + 1 − cos Θ 풏 × 풏 is the normal vector (2 DoF) and Θ the angle (1 DoF)
3) Another option is unit quaternions (see book page 40)
Computer Vision I: Image Formation Process 22/12/2014 38 Halfway
3 Minutes Break Question?
Computer Vision I: Image Formation Process 22/12/2014 39 Roadmap for next five lecture • Appearance based matching (sec. 4.1)
• How do we get an RGB image? (sec 2.2-2.3) • The Human eye • The Camera and Image formation model
• Projective Geometry - Basics (sec. 2.1.1-2.1.4)
• Geometry of a single camera (sec 2.1.5, 2.1.6) • Pinhole camera • Lens effects
• Geometry of two cameras (sec. 7.2)
• Robust Geometry estimation for two cameras (sec. 6.1.4)
• Multi-View 3D reconstruction (sec. 7.3-7.4)
Computer Vision I: Image Formation Process 22/12/2014 40 How can we capture the world
• Let’s design a camera • Idea 1: put a piece of film (or a CCD) in front of an object • Do we get a reasonable image?
Computer Vision I: Image Formation Process 22/12/2014 41 Pinhole Camera
• Add a barrier to block off most of the rays • This reduces blurring • The opening is known as the aperture (“Blende”)
Computer Vision I: Image Formation Process 22/12/2014 42 Pinhole camera model
Image plane
Focal point
Pinhole model: • Captures pencil of rays – all rays through a single point • Projected rays are straight lines • The point where all rays meet is called center of projection (focal point) • The image is formed on the image plane
Computer Vision I: Image Formation Process 22/12/2014 43 Pinhole camera – Properties • Many-to-one: any point along the same ray maps to the same point in the image
• Points map to points Focal (But projection of points on focal plane is undefined) plane
• Lines map to lines (collinearity is preserved) (But line through focal point projects to a point)
• Planes map to planes (or half-plane)
(But plane through focal point projects to line) Ground plane
Computer Vision I: Image Formation Process 22/12/2014 44 Pinhole Camera A model for many common sensors: • Photographic cameras • human eye • X-ray machines
Computer Vision I: Image Formation Process 22/12/2014 45 Dimensionality Reduction Machine (3D to 2D)
Computer Vision I: Image Formation Process 22/12/2014 46 Pinhole camera model – in maths
푦 푦 푥
focal length
푦 푌 • Similar trinagles: = 푓 푍 푌 푋 • That gives: 푦 = 푓 and 푥 = 푓 푍 푍
푥 푓 0 0 푋 • That gives: 푦 = 0 푓 0 푌 (remeber “=“ means equal up to scale) 1 0 0 1 푍 2D homogenous 3D inhomogenous coordinate coordinate
Computer Vision I: Image Formation Process 22/12/2014 47 Pinhole camera model – in maths
푦 푦 푥
푥 푓 0 0 푋 That gives: 푦 = 0 푓 0 푌 1 0 0 1 푍 Calibration matrix 푲 In short 풙 = 푲 푿~ (here 푿~ means inhomogeneous coordinates) Intrinsic Camera Calibration means we know 푲
We can also go from image points into the 3D world: 푿~ = 푲−ퟏ 풙
Computer Vision I: Image Formation Process 22/12/2014 48 Pinhole camera - Definitions
• Principal axis: line from the camera center perpendicular to the image plane • Normalized (camera) coordinate system: camera center is at the origin and the principal axis is the z-axis • Principal point (p): point where principal axis intersects the image plane (origin of normalized coordinate system)
Computer Vision I: Image Formation Process 22/12/2014 49 Principal Point
Principal point (푝푥, 푝푦)
Image coordinate system
• Camera coordinate system: origin is at the principal point
• Image coordinate system: origin is in the corner In practice: principal point in center of the image
Computer Vision I: Image Formation Process 22/12/2014 50 Adding principal point into 푲
Principal point (푝푥, 푝푦)
Image coordinate system
푌 푓푌+푍푝 푋 푓푋+푍푝 Projection with principal point : 푦 = 푓 + 푝 = 푦 and 푥 = 푓 + 푝 = 푥 푍 푦 푍 푍 푥 푍
푥 푓 0 푝푥 푋 That gives: 푦 = 0 푓 푝푦 푌 1 0 0 1 푍
Computer Vision I: Image Formation Process 22/12/2014 51 Pixel Size and Shape
• 푓 has a certain unit (m,mm,inch,...) • 푚푥 pixels per unit (m,mm,inch,...) in horizontal direction • 푚푦 pixels per unit (m,mm,inch,...) in vertical direction • 푠′ skew of a pixel
푥 푓 0 푝푥 푚푥 푠′ 0 푋 푓푚푥 푓푠′ 푝푥 푋 That gives: 푦 = 0 푓 푝푦 0 푚푦 0 푌 = 0 푓푚푦 푝푦 푌 1 0 0 1 0 0 1 푍 0 0 1 푍
푥 푓 푠 푝푥 푋 Simplify and choose a pixel as unit: 푦 = 0 푚푓 푝푦 푌 1 푓 now in units of pixels 0 0 1 푍 Final calibration matrix 푲 Computer Vision I: Image Formation Process 22/12/2014 52 Camera intrinsic parameters - Summary
• Intrinsic parameters • Principal point coordinates (푝푥, 푝푦) 푓 푠 푝 • Focal length 푓 푥 푲 = 0 푚푓 푝푦 • Pixel y-direction magnification factors 푚 0 0 1 • Skew (non-rectangular pixels) 푠
For later: We sometimes have to only guess these values and then they are optimized via e.g. bundle adjustment: • 푝 in image center, • 푠 = 0, 푚 = 1 • f= EXIF tag (or guess, e.g. two times image dimension)
Computer Vision I: Image Formation Process 22/12/2014 53