Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices CS 4495 Computer Vision N-Views (2) – Essential and Fundamental Matrices Aaron Bobick School of Interactive Computing Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices Administrivia • Today: Second half of N-Views (n = 2) • PS 3: Will hopefully be out by Thursday • Will be due October 6th. • Will be based upon last week and today’s material • We may revisit the logistics – suggestions? Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices Two views…and two lectures • Projective transforms from image to image • Some more projective geometry • Points and lines and planes • Two arbitrary views of the same scene • Calibrated – “Essential Matrix” • Two uncalibrated cameras “Fundamental Matrix” • Gives epipolar lines Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices Last time • Projective Transforms: Matrices that provide transformations including translations, rotations, similarity, affine and finally general (or perspective) projection. • When 2D matrices are 3x3; for 3D they are 4x4. Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices Last time: Homographies • Provide mapping between images (image planes) taken from same center of projection; also mapping between any images of a planar surface. PP2 PP1 Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices Last time: Projective geometry • A line is a plane of rays through origin – all rays (x,y,z) satisfying: ax + by + cz = 0 x in vector notation : 0 = [a b c]y z l p • A line is also represented as a homogeneous 3-vector l Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices Projective Geometry: lines and points p 2D Lines: ax+ by += c 0 1 p x 2 [abcy] = 0 p1 = [x1 y1 1] = × 1 l p1 p2 Eq of line p2 = [x2 y2 1] T = lx 0 x12 = ⇒ l[ abc] nxy n d (nx, ny) d l1 = [a1 b1 c1] x12 = l1 × l2 l2 = [a2 b2 c2] Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices Motivating the problem: stereo • Given two views of a scene (the two cameras not necessarily having optical axes) what is the relationship between the location of a scene point in one image and its location in the other? Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices Stereo correspondence • Determine Pixel Correspondence • Pairs of points that correspond to same scene point P epipolar line epipolar plane epipolar line CP 1 CP2 Epipolar Constraint • Reduces correspondence problem to 1D search along conjugate epipolar lines Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices Example: converging cameras Figure from Hartley & Zisserman Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices Epipolar geometry: terms • Baseline: line joining the camera centers • Epipole: point of intersection of baseline with image plane • Epipolar plane: plane containing baseline and world point • Epipolar line: intersection of epipolar plane with the image plane • All epipolar lines intersect at the epipole • An epipolar plane intersects the left and right image planes in corresponding epipolar lines Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices From Geometry to Algebra • So far, we have the explanation in terms of geometry. • Now, how to express the epipolar constraints algebraically? Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices Stereo geometry, with calibrated cameras Main idea Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices Stereo geometry, with calibrated cameras If the stereo rig is calibrated, we know : how to rotate and translate camera reference frame 1 to get to camera reference frame 2. Rotation: 3 x 3 matrix R; translation: 3 vector T. Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices Stereo geometry, with calibrated cameras If the stereo rig is calibrated, we know : how to rotate and translate camera reference frame 1 to get to camera reference frame 2. X' = RX + T c c Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices An aside: cross product ab×= c Vector cross product takes two vectors and returns a third vector that’s perpendicular to both inputs. So here, c is perpendicular to both a and b, which means the dot product = 0. ac⋅=0 bc⋅=0 Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices From geometry to algebra X'= RX + T X′⋅(T× X′) = X′⋅(T× RX) T× X′ = T× RX + T×T Normal to the plane 0 =⋅×X′ ( T RX) = T× RX Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices Another aside: Matrix form of cross product 0 − a a b 3 2 1 × = − = a b a3 0 a1 b2 c − a2 a1 0 b3 Can be expressed as a matrix multiplication!!! 0 − a3 a2 Notation: [a ]= a 0 − a x 3 1 ab×=[ ab× ] − a2 a1 0 Has rank 2! Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices From geometry to algebra X'= RX + T X′⋅(T× X′) = X′⋅(T× RX) T× X′ = T× RX + T×T Normal to the plane 0 =⋅×X′ ( T RX) = T× RX Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices Essential matrix X′⋅(T× RX) = 0 X′⋅([Tx ]RX) = 0 Let E = [T x]R X′T EX = 0 E is called the essential matrix, and it relates corresponding image points between both cameras, given the rotation and translation. Note: these points are in each camera coordinate systems. We know if we observe a point in one image, its position in other image is constrained to lie on line defined by above. Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices Essential matrix example: parallel cameras Z P p = [,Zx Zy ,] RI= f Τ Z x p' = [Zx ', Zy ', ] xr T =[ −B ,0,0] l Z 0 0 0 f E= [T x ]R = 0 0 B p p f l r 0 –B 0 B COPL COPR 000 x Τ [x' y '10] 0 By= 0 p′ Ep = 0 Given a known 0−B 01 point (x,y) in the original 0 image, this is a [xy' '1] B= 0 line in the For the parallel cameras, (x’,y’) image. image of any point must lie −By on same horizontal line in each image plane. By' = By ⇒ y' = y Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices Weak calibration • Want to estimate world geometry without requiring calibrated cameras • Archival videos (already have the pictures) • Photos from multiple unrelated users • Dynamic camera system • Main idea: • Estimate epipolar geometry from a (redundant) set of point correspondences between two uncalibrated cameras Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices From before: Projection matrix X w • This can be rewritten as a wxim Yw matrix product using wy = K Φ homogeneous coordinates: im int ext Zw w where: 1 Τ rrr11 12 13 −RT1 Τ Φ = rrr−RT ext 21 22 23 2 − Τ rrr31 32 33 RT3 − fs/0xx o Note: Invertible, scale x Kint =0/ − fsyy o and y, assumes no skew 0 01 Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices From before: Projection matrix X • This can be rewritten as a w matrix product using wxim Yw homogeneous coordinates: = wyimK intΦ ext Zw w 1 pKim= intΦP ext w pc pim= Kp int c Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices Uncalibrated case For a given p= Kp camera: im int c So, for two cameras (left and right): = −1 pc,, left Kp int,left im left −1 pc,, right= Kp int,right im right Internal calibration matrices, one per camera Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices Uncalibrated case −1 −1 pc,, right= Kp int,right im right pc,, left= Kp int,left im left From before, the essential Τ pc,right Epc,left = 0 matrix E. Τ −−11 (Kint,right p im,, right ) EK( int,left p im left ) = 0 Τ−11T − pim,, right(() K int,right EK int,left) p im left = 0 “Fundamental matrix” F Τ T pim,rightFpim,left = 0 or p Fp'0= Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices Properties of the Fundamental Matrix pT Fp '0= T • l = F p ′ is the epipolar line associated with p’ T • l ′ = F p is the epipolar line associated with p T • Epipoles found by Fp’ = 0 and F p = 0 • You’ll see more one these on the problem set to explain • F is singular (mapping from 2-D point to 1-D family so rank 2 – more later) Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices Fundamental matrix • Relates pixel coordinates in the two views • More general form than essential matrix: we remove need to know intrinsic parameters • If we estimate fundamental matrix from correspondences in pixel coordinates, can reconstruct epipolar geometry without intrinsic or extrinsic parameters. Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices Different Example: forward motion e’ e courtesy of Andrew Zisserman Two Views Part 2: Essential CS 4495 Computer Vision – A.