Robert Collins CSE486, Penn State Lecture 12: Camera Projection Reading: T&V Section 2.4 Robert Collins CSE486, Penn State Imaging Geometry W Object of Interest in World Coordinate System (U,V,W) V U Robert Collins CSE486, Penn State Imaging Geometry Camera Coordinate Y System (X,Y,Z). Z X • Z is optic axis f • Image plane located f units out along optic axis • f is called focal length Robert Collins CSE486, Penn State Imaging Geometry W Y y X Z x V U Forward Projection onto image plane. 3D (X,Y,Z) projected to 2D (x,y) Robert Collins CSE486, Penn State Imaging Geometry W Y y X Z x V u U v Our image gets digitized into pixel coordinates (u,v) Robert Collins CSE486, Penn State Imaging Geometry Camera Image (film) World Coordinates Coordinates CoordinatesW Y y X Z x V u U v Pixel Coordinates Robert Collins CSE486, Penn State Forward Projection World Camera Film Pixel Coords Coords Coords Coords U X x u V Y y v W Z We want a mathematical model to describe how 3D World points get projected into 2D Pixel coordinates. Our goal: describe this sequence of transformations by a big matrix equation! Robert Collins CSE486, Penn State Backward Projection World Camera Film Pixel Coords Coords Coords Coords U X x u V Y y v W Z Note, much of vision concerns trying to derive backward projection equations to recover 3D scene structure from images (via stereo or motion) But first, we have to understand forward projection… Robert Collins CSE486, Penn State Forward Projection World Camera Film Pixel Coords Coords Coords Coords U X x u V Y y v W Z 3D-to-2D Projection • perspective projection We will start here in the middle, since we’ve already talked about this when discussing stereo. Robert Collins CSE486, Penn State Basic Perspective Projection Scene Point Perspective Projection Eqns P = (X,Y,Z) X Image Point y x f p = (x,y,f) X Y Z Y x x x y Z Y X y f f Z Z O O.Camps, PSU Robert Collins CSE486, Penn State Basic Perspective Projection Scene Point Perspective Projection Eqns P = (X,Y,Z) X Image Point y x f p = (x,y,f) X Y Z Y x x x y Z Y X y f f Z Z O derived via similar triangles rule x X f Z O.Camps, PSU Robert Collins CSE486, Penn State Basic Perspective Projection Scene Point Perspective Projection Eqns P = (X,Y,Z) X Image Point y x f p = (x,y,f) X Y Z Y x x x y Z Y X y f f Z Z O derived via similar triangles rule x X y Y f f Z Z O.Camps, PSU Robert Collins CSE486, Penn State Basic Perspective Projection Scene Point Perspective Projection Eqns P = (X,Y,Z) X Image Point y x f p = (x,y,f) X Y Z Y x x x y Z Y X y f f Z Z O So how do we represent this as a matrix equation? We need to introduce homogeneous coordinates. O.Camps, PSU Robert Collins CSE486, Penn State Homogeneous Coordinates Represent a 2D point (x,y) by a 3D point (x’,y’,z’) by adding a “fictitious” third coordinate. By convention, we specify that given (x’,y’,z’) we can recover the 2D point (x,y) as x' y' x y z' z' Note: (x,y) = (x,y,1) = (2x, 2y, 2) = (k x, ky, k) for any nonzero k (can be negative as well as positive) Robert Collins CSE486, Penn State Perspective Matrix Equation (in Camera Coordinates) X X x' f 0 0 0 x f Y Z y' 0 f 0 0 Y Z y f z' 0 0 1 0 Z 1 Robert Collins CSE486, Penn State Forward Projection World Camera Film Pixel Coords Coords Coords Coords U X x u V Y y v W Z Rigid Transformation (rotation+translation) between world and camera coordinate systems Robert Collins CSE486, PennWorld State to Camera Transformation PC PW W Y X U Z V Avoid confusion: Pw and Pc are not two different points. They are the same physical point, described in two different coordinate systems. Robert Collins CSE486, PennWorld State to Camera Transformation PC PW W Y X U R Z C V Rotate to Translate by - C align axes (align origins) PC = R ( PW - C ) Robert Collins CSE486, PennMatrix State Form, Homogeneous Coords PC = R ( PW - C ) X r11 r12 r13 1 0 0 cx U Y r21 r22 r23 0 1 0 cy V Z r31 r32 r33 0 0 1 cz W 1 0 0 0 1 0 0 0 1 1 Robert Collins CSE486, Penn State Example: Simple Stereo System Y Z (X,Y,Z) z left y camera located at ( , ) z (0,0,0) y ( , ) x right camera Tx located at x (Tx,0,0) X Left camera located at world origin (0,0,0) and camera axes aligned with world coord axes. Robert Collins CSE486, Penn State Simple Stereo, Left Camera X 1r11 r012 r013 1 0 0 0cx U Y 0r21 r122 r023 0 1 0 0cy V Z 0r31 r032 r133 0 0 1 0cz W 1 0 0 0 1 0 0 0 1 1 camera axes aligned located at world with world axes position (0,0,0) = Robert Collins CSE486, Penn StateSimple Stereo Projection Equations Left camera Robert Collins CSE486, Penn State Example: Simple Stereo System Y Z (X,Y,Z) z left y camera located at ( , ) z (0,0,0) y ( , ) x right camera Tx located at x (Tx,0,0) X Right camera located at world location (Tx,0,0) and camera axes aligned with world coord axes. Robert Collins CSE486, Penn State Simple Stereo, Right Camera X r r r 111 012 013 1 0 0 -Tcxx U Y 0r21 r122 r023 0 1 0 0cy V Z 0r31 r032 r133 0 0 1 0cz W 1 0 0 0 1 0 0 0 1 1 camera axes aligned located at world with world axes position (Tx,0,0) = Robert Collins CSE486, Penn StateSimple Stereo Projection Equations Left camera Right camera Robert Collins CSE486, Penn State Bob’s sure-fire way(s) to figure out the rotation X r11 r12 r13 1 0 0 cx U Y r21 r22 r23 0forget1 about0 thiscy V Z while thinking r31 r32 r33 0about0 rotations1 cz W 1 0 0 0 1 0 0 0 1 1 PC = R PW This equation says how vectors in the world coordinate system (including the coordinate axes) get transformed into the camera coordinate system. Robert Collins CSE486, Penn State Figuring out Rotations X r11 r12 r13 U Y r21 r22 r23 V PC = R PW Z r31 r32 r33 W 1 0 0 0 1 1 what if world x axis (1,0,0) corresponds to camera axis (a,b,c)? X r r a r11 12 13 U1 Xa ra11 r12 r13 U1 Y r r b r21 22 23 V0 Yb rb21 r22 r23 V0 Zc r r 0 r31 32 33 W Zc rc31 r32 r33 W0 1 0 0 0 1 1 1 0 0 0 1 1 we can immediately write down the first column of R! Robert Collins CSE486, Penn State Figuring out Rotations and likewise with world Y axis and world Z axis... same axis in camera coords axis is world coords X r11 r12 r13 U Y r21 r22 r23 V Z r31 r32 r33 W 1 0 0 0 1 1 world X axis (1,0,0) world Z axis (0,0,1) in camera coords world Y axis (0,1,0) in camera coords in camera coords Robert Collins CSE486, Penn State Figuring out Rotations Alternative approach: sometimes it is easier to specify what camera X,Y,or Z axis is in world coordinates. Then do rearrange the equation as follows. -1 T PC = R PW R PC = PW R PC = PW r11 r21 r31 X U r12 r22 r32 Y V r13 r23 r33 Z W 0 0 0 1 1 1 Robert Collins CSE486, Penn State Figuring out Rotations r11 r21 r31 X U r r r Y V T 12 22 32 R PC = PW r13 r23 r33 Z W 0 0 0 1 1 1 what if camera X axis (1,0,0) corresponds to world axis (a,b,c)? r r r X U 11 21 31 1 a ra11 r21 r31 X1 Ua r r r Y V 12 22 32 0 b rb12 r22 r32 Y0 Vb r r r Z0 Wc 13 23 33 rc13 r23 r33 Z0 Wc 0 0 0 1 1 1 0 0 0 1 1 1 we can immediately write down the first column of RT, (which is the first row of R). Robert Collins CSE486, Penn State Figuring out Rotations and likewise with camera Y axis and camera Z axis... same axis in camera coords axis is world coords X r11 r12 r13 U Y r21 r22 r23 V Z r31 r32 r33 W 1 0 0 0 1 1 camera X axis (1,0,0) in world coords camera Z axis (0,0,1) camera Y axis (0,1,0) in world coords in world coords Robert Collins CSE486, Penn State Example y z x 0 0 1 0 -1 0 Rtrain 0 -1 0 Rfly 0 0 1 1 0 0 -1 0 0 Robert Collins CSE486, Penn State Note: External Parameters also often written as R,T X r11 r12 r13 1 0 0 cx U Y r21 r22 r23 0 1 0 cy V Z r31 r32 r33 0 0 1 cz W 1 0 0 0 1 0 0 0 1 1 r r r t R ( PW - C ) 11 12 13 x r21 r22 r23 ty = R PW - R C r31 r32 r33 tz = R PW + T 0 0 0 1 Robert Collins CSE486, Penn State Summary World Camera Film Pixel Coords Coords Coords Coords U X x u V Y y v W Z We now know how to transform 3D world coordinate points into camera coords, and then do perspective project to get 2D points in the film plane.