Agenda • Rotations • Camera models • Camera calibration • Homographies 3D Rotations X r11 r12 r13 X R Y = r r r Y 2 3 2 21 22 233 2 3 Z r31 r32 r33 Z 4 5 4 5 4 5 Think of as change of basis where ri = r(i,:) are orthonormal basis vectors r2 r1 rotated coordinate frame r3 7 Shears 1 hxy hxz 0 h 1 h 0 A˜ = 2 yx yz 3 hzx hzy 10 6 00017 6 7 4 Shears y into5 x 7 3D Rotations 8 LotsRotations of parameterizations that try to capture 3 DOFs Helpful ones• 3Dfor Rotationsvision: orthonormal fundamentally matrix, more axis-angle,complex than exponential in 2D! maps • 2D: amount of rotation! Represent a 3D rotation with a unit vector pointed along the axis of • 3D: amount and axis of rotation rotation, and an angle of rotation about that vector -vs- 2D 3D 8 05-3DTransformations.key - February 9, 2015 Background: euler’s rotation theorm Any rotation of a rigid body in a three-dimensional space is equivalent to a pure rotation about a single fixed axis https://en.wikipedia.org/wiki/Euler's_rotation_theorem Review: dot and cross products Dot product: a b = a b cos✓ · || || || || Cross product: a b a b 2 3 − 3 2 a b = b1a3 a1b3 ⇥ 2a b − a b 3 1 2 − 2 1 4 5 Cross product 0 a a b − 3 2 1 a b = ˆab = a 0 a b matrix: 3 1 2 ⇥ 2 a a −0 3 2b 3 − 2 1 3 4 5 4 5 aˆT = aˆ (skew-symmetric matrix) − Approach v R3, v =1 2 || || ✓ x https://en.wikipedia.org/wiki/Axis-angle_representation Rodrigues' rotation formula https://en.wikipedia.org/wiki/Rodrigues'_rotation_formula_rotation_formula v R3, v =1 2 || || x ? ✓ x k x 1. Write as x as sum of parallel and perpindicular component to omega 2. Rotate perpindicular component by 2D rotation of theta in plane orthogonal to omega R = I +ˆv sin ✓ +ˆvvˆ(1 cos ✓) − [Rx can simplify to cross and dot product computations] Agenda • Rotations • Camera models • Camera calibration • Homographies Perspective projection revisited y x (X,Y,Z) (x,y,f) z COP f x = X Z f y = Y Z Perspective projection revisited x f 00 X λ y = 0 f 0 Y 213 20013 2Z 3 4 5 4 5 4 5 Given (X,Y,Z) and f, compute (x,y) and lambda: λx = fX λ = Z λx fX x = = λ Z Special case: f = 1 (normalized image plane) (X,Y,Z) (x,y,1) COP x f 00 X λ y = 0 f 0 Y 213 20013 2Z 3 4 5 4 5 4 5 x X Z y = Y 213 2Z 3 4 5 4 5 3D point is obtained by scaling ray pointed at image coordinate Scale factor = true depth of point [Aside: given an image with a focal length ‘f’, resize by ‘1/f’ to obtain unit-focal-length image] Homogenous notation x X y Y 2z3 ⇠ 2Z 3 4x5 4X5 y Y 2z3 ⌘ 2Z 3 4 5 4 5 For now, think of above as shorthand notation for x X λ s.t. λ y = Y 9 2z3 2Z 3 4 5 4 5 3D rigid-body transformations 3D translations 3D rotations X X + tx X r11 r12 r13 X Y + T = Y + t R Y = r r r Y 2 3 2 y 3 2 3 2 21 22 233 2 3 Z Z + tz Z r31 r32 r33 Z 4 5 4 5 4 5 4 5 4 5 X X r r r t 11 12 13 x Y R Y + T = r r r t 21 22 23 y 2Z 3 2Z 3 2r r r t 3 31 32 33 z 6 1 7 4 5 4 5 6 7 4 5 Aside: 3D affine transformations 3D translations any invertible matrix R X X + tx X r11 r12 r13 X Y + T = Y + t R Y = r r r Y 2 3 2 y 3 2 3 2 21 22 233 2 3 Z Z + tz Z r31 r32 r33 Z 4 5 4 5 4 5 4 5 4 5 X X r r r t 11 12 13 x Y R Y + T = r r r t 21 22 23 y 2Z 3 2Z 3 2r r r t 3 31 32 33 z 6 1 7 4 5 4 5 6 7 4 5 Alternative perspective: change of coordinate system think of camera moving through world coordinate frame X X r r r t 11 12 13 x Y R Y + T = r r r t 21 22 23 y 2Z 3 2Z 3 2r r r t 3 31 32 33 z 6 1 7 4 5 4 5 6 7 4 5 r2 r1 camera r3 T world coordinate frame Camera projection X x f 00 r r r t 11 12 13 x Y λ y = 0 f 0 r r r t 21 22 23 y 2Z 3 213 20013 2r r r t 3 31 32 33 z 6 1 7 6 7 4 5 Camera4 instrinsic matrix5 K 4 Camera extrinsics 53D point in (can include skew & non-square pixel size) (rotation and translation) world4 coordinates5 r2 r1 camera r3 T world coordinate frame λx Aside: homogenous notation is shorthand for x = λ Fancier intrinsics xs = sxx } non-square pixels ys = syy x0 = xs + ox } shifted origin y0 = ys + oy y skewed image axes x”=x0 + s✓y0 ✓ x sx s✓ ox f 00 fsx fs✓ ox K = 0 sy oy 0 f 0 = 0 fsy oy 2 0013 20013 2 0013 4 5 4 5 4 5 Notation [Using Matlab’s rows x columns] X x fs fs o r r r t x ✓ x 11 12 13 x Y λ y = 0 fs o r r r t y y 21 22 23 y 2Z 3 213 2 0013 2r r r t 3 31 32 33 z 6 1 7 4 5 4 5 4 5 6 7 X 4 5 Y = K3 3 R3 3 T3 1 2 3 ⇥ ⇥ ⇥ Z ⇥ ⇤ 6 1 7 6 7 X 4 5 Y = M3 4 2 3 ⇥ Z 6 1 7 6 7 4 5 X Y = A3 3 b3 1 2 3 ⇥ ⇥ Z ⇥ ⇤ 6 1 7 6 7 X 4 5 = A3 3 Y + b3 1 ⇥ 2Z 3 ⇥ 4 5 Notation & claims X x X Y λ y = M3 4 2 3 = A3 3 Y + b3 1 ⇥ Z ⇥ ⇥ 213 2Z 3 6 1 7 4 5 6 7 4 5 4 5 Claims: 1. A 3x4 matrix ‘M’ can be a camera matrix iff det(A) is not zero (geometric intution later) 2. M is determined only up to a scale factor (easy to show) T T m1 a1 b1 T T M = m2 ,A= a2 ,b= b2 2 T 3 2 T 3 2 3 m3 a3 b3 4 5 4 5 4 5 Applying the projection matrix 1 x = ( XY Za + b ) λ 1 1 ⇥ ⇤ 1 y = ( XY Za + b ) λ 2 2 ⇥ ⇤ λ = XY Za3 + b3 ⇥ ⇤ Set of 3D points that project to x = 0: XY Za1 + b1 =0 Set of 3D points that project to y = 0: ⇥XY Z⇤ a2 + b2 =0 ⇥ ⇤ Set of 3D points that project to x = inf or y = inf: XY Za3 + b3 =0 What do these sets look like⇥ in 3D world?⇤ Convention of most of today’s slides (x,y) COP (X,Y,Z) Draw image plane behind pinhole Geometric intuition Rows of the camera matrix describe the 3 planes defined by the image coordinate system a3 a1 y COP a2 x image plane Other geometric properties y x (X,Y,Z) (x,y,f) z COP What’s set of (X,Y,Z) points that project to same (x,y)? x X λ y = A Y + b 213 2Z 3 4 5 4 5 Solve above expression for (X,Y,Z) as a function of (x,y) Other geometric properties y x (X,Y,Z) (x,y,f) z COP What’s set of (X,Y,Z) points that project to same (x,y)? X x 1 1 Y = λw + b0 where w = A− y ,b0 = A− b 2Z 3 213 − X 4 5 x 4 5 1 1 Y =Directionλw + b0 ofwhere ray: w = A− y ,b0 = A− b 2Z 3 213 − X 4 5 x 4 5 1 1 Y = λw + b0 where w = A−COP:y ,b0 = A− b 2Z 3 213 − 4 5 4 5 Affine cameras perspective weak perspective Crucial constraint on camera matrix: T m3 = 0001 ⇥ ⇤ Affine cameras Captures 3D affine transformation + orthographic projection + 2D affine transformation X 1 x ···· Y = ··· 1 2····3 2 3 y 2 3 Z ··· 1 ···· 6 17 6 1 7 4 5 6 7 6 7 4 5 4 5 X a a a b 11 12 13 1 Y = a a a b 21 22 23 2 2Z 3 2 1 3 6 1 7 4 5 6 7 X 4 5 a a a b = 11 12 13 Y + 1 a21 a22 a23 2 3 b2 Z x = AX + b 4 5 • Projection defined by 8 parameters • Parallel lines project to parallel lines • 2D points = linear projection of 3D points (+ 2D translation) Geometric Transformations Euclidean (trans + rot) Affine: preserves Projective: preserves lengths + angles parallel lines preserves lines Point out differences in 2D transformations and 3D transformations Projective Affine Euclidean Agenda • Rotations • Camera models • Camera calibration • Homographies PnP = Perspective n-Point Calibration: Recover M from scene points P1,..,PN and the corresponding projections in the image plane p1,..,pN y Xi x Yi Pi = 2 3 xi Zi z p = y COP 6 1 7 i 2 i3 6 7 1 4 5 4 5 The math for the calibration procedure follows a recipe that is used in many (most?) problems involving camera geometry, so it’s worth remembering: Write relation between image point, projection matrix, and point in space: X x i i Y λ y = M i i 2Z 3 2 1 3 i 6 1 7 4 5 6 7 4 5 mT P mT P Write non-linear relations x = 1 i ,y= 2 i between coordinates: i T i T m3 Pi m3 Pi Estimating a camera matrix y x z Write constraints as linear unknowns M T T m3 Pixi =.