Notes for Projective Camera Details
Total Page:16
File Type:pdf, Size:1020Kb
Lecture 3 Camera Models 2 & Camera Calibration Professor Silvio Savarese Computational Vision and Geometry Lab Silvio Savarese Lecture 3 - 13-Jan-15 In this lecture, we will discuss the topic of camera calibration. We will start with a recap of camera models. Next, we will formulate the camera calibration problem and investigate how to estimate the unknown parameters. We will see how Lecture 3 radial distortion affects our calibration process, and how we can model it. Finally, Camera Models 2 & we will end with an example calibration session in MATLAB. Camera Calibration • Recap of camera models • Camera calibration problem • Camera calibration with radial distortion • Example Reading: [FP] Chapter 1 “Geometric Camera Calibration” [HZ] Chapter 7 “Computation of Camera Matrix P” Some slides in this lecture are courtesy to Profs. J. Ponce, F-F Li Silvio Savarese Lecture 3 - 14-Jan-15 Let's revisit our projective camera model. Please refer to the lecture 2 notes for Projective camera details. Pinhole perspective projection f f = focal length Projective camera Pinhole perspective projection f y yc f = focal length u , v = offset xc o o (note a different convention w.r.t. lecture 2) C’=[uo, vo] x Projective camera f Units: k,l [pixel/m] f = focal length f [m] uo, vo = offset Non-square pixels α, β → non-square pixels α, β [pixel] Projective camera f y yc f = focal length c x uo, vo = offset , non-square pixels C’ = [u , v ] α β → o o x θ θ = skew angle Projective camera f & )& ) α −α cotθ uo 0 x ( +( + f = focal length ( β +( y + P! = 0 vo 0 u , v = offset ( sinθ +( z + o o ( +( + α, → non-square pixels 0 0 1 0 1 β '( *+' * θ = skew angle K has 5 degrees of freedom! Projective camera jw R,T k f w Ow 'R T $ f = focal length P = P u , v = offset %0 1" w o o & # 4×4 α, β → non-square pixels θ = skew angle R,T = rotation, translation Projective camera jw R,T k f w Ow P! = M Pw f = focal length uo, vo = offset = K[R T]Pw α, β → non-square pixels θ = skew angle Internal parameters External parameters R,T = rotation, translation This slide summarizes the main equation that describes the projective transformation from point P in the world coordinate system (in homogenous j w Projective camera w coordinates) into a point P in the image pixels (in Euclidean coordinate systems). E R,T This equation was also presented in lecture 2. k f w Ow &m1 # " $ M $m ! P'3×1 = M Pw = K3×3 R T Pw4×1 = 2 # % 3×4 $ ! ! $ ! $ $m3 ! m1 m1PW % " # & # & E m P m P m1Pw m2Pw = # 2 & W = # 2 W & → P' = ( , ) # & # & E m m P m3Pw m3Pw "# 3 %& "# 3 W %& In 1993 Olivier Faugeras derived a powerful theorem that relates the properties Theorem (Faugeras, 1993) of a generic projective transformation M with the properties of the matrix A as 3x3 defined in Eq. 13. A is defined as K R and a , a and a are the first, second and &a1 # 1 2 3 M = K[R T ]= [K R K T ]= [A b] A $a ! third row vectors of A, respectively. These relationships are summarized in the 3 = $ 2 ! necessary and sufficient conditions illustrated in the slide. [Eq.13; lect. 2] %$a3 "! Let’s consider now the following exercise: suppose that the two reference systems coincide (R=I and T=0 ) and the camera model has focal length f, Exercise! 3x3 3x1 zero-skew, no offset, square pixels. Can we write a simplified expression for P’ ? E R,T Notice that the result is exactly what we derived for the pinhole camera in j w lecture 2. f P kw Ow P’ iw ! f 0 0 0 $ # & ! # K! I 0 # M = K R T = " $ = # 0 f 0 0 & " $ # & 0 0 1 0 ! x $ " % # w & # y & m1Pw m2Pw xw yw P = w à P' = ( , ) = ( f , f ) w # & E zw m3Pw m3Pw zw zw # & "# 1 %& So far, we have discussed a projective camera model. Given a point in the world, Projective camera its image coordinates depends on its depth. f’ R p’ q’ r’ Q O P We will now describe a simpler model known as the Weak Perspective Projection model. Weak perspective projection In the weak perspective points are: 1) first projected to a reference plane (using When the relative scene depth is small compared to its distance from the camera orthogonal projection) and 2) then projected to the image plane using a projective transformation. f’ zo 1) As the slide shows, given a reference plane π at a distance ζ from the center R 0 p’ R_ of the camera, the 3 points P, Q and R are first projected to the plane π using an orthogonal projection; this generates the points R_, Q_ and P_. q’ (this is equivalent to assigning the z-coordinate of the points P, Q and R to z ) r’ Q o O Q_ This is a reasonable approximation when deviations in depth from the plane are small compared to the distance from the camera. P_ P π 2) Then, the points R_, Q_ and P_ are projected to the image plane using a regular projective transformation producing the points p’, q’, r’. Notice, however, Weak perspective projection that because we have approximated the depth of each point to ζ the projection 0, has been reduced to a simple (constant) magnification. The magnification is equal to the focal length f’ (which is a constant) is divided by ζ (which is also a zo 0 f’ constant). R p’ R_ q’ r’ Q O Q_ P_ P ! f ' π # x' = x ! f ' z # x' = x # # z " → " 0 f ' Magnification m # f ' # y' = y # y' = y # z $ 0 $# z This also simplifies the projection matrix M. Note that the last row of M is now [0 0 0 1] (written as [0 1] using the block matrix notation above). We do not prove Weak perspective projection this result and leave it to students as an exercise. We can observe this simplification algebraically in the next slide. π f’ zo R p’ R_ q’ r’ Q O Q_ P_ P Projective (perspective) Weak perspective &A b# &A b# M = K[R T ]= $ ! à M = $ ! % v 1" % 0 1" In the equations above, m , m , and m denote the rows of the projection 1 2 3 &m1 # &m1 PW # &m1 # matrix. In our full perspective model, m is equal to [v 1], where v is some non- $ ! $ ! &A b# $ ! 3 P! = M P = m2 PW = m2 PW M = = m zero 1x3 vector. On the other hand, m = [0 0 0 1] for the weak perspective w $ ! $ ! $ ! $ 2 ! 3 % v 1" model. This results in the denominator term m P to be 1. As result, the non $m3 ! $m3 PW ! $m3 ! 3 w % " % " % " linearity of the projective transformation disappears and the weak perspective transformation acts as a mere magnifier. E m P m P → ( 1 w , 2 w ) m P m P 3 w 3 w Perspective A b &m1 # &m1 PW # & # M = $ ! = $m ! P = $m P ! % 0 1" P! = M Pw $ 2 ! W $ 2 W ! m 1 %$ 3 "! %$ "! &m1 # & m1 # $m ! $ m ! E = $ 2 ! = $ 2 ! $m3 ! $0 0 0 1! → (m1 Pw , m2 Pw ) % " % " magnification Weak perspective Further simplification leads to the orthographic (or affine) projection model. In Orthographic (affine) projection this case, the optical center is located at infinity. The projection rays are now perpendicular to the retinal plane (parallel to the optical axis). As a result, this Distance from center of projection to image plane is infinite model ignores depth altogether. It’s often used for architecture and industrial design. ! f ' # x' = x # z # #! x' = x " → " # y' = y # f ' $ # y' = y $# z Pros and Cons of These Models • Weak perspective much simpler math. – Accurate when object is small and distant. – Most useful for recognition. • Pinhole perspective much more accurate for scenes. – Used in structure from motion or SLAM. Painted by Wang Hui (1632-1717) and assistants, it was executed before Western perspective was introduced into Chinese art. In the video D. Hockney comments Weak perspective projection the properties of the prospective geometry depicted in the painting. The Kangxi Emperor's Southern Inspection Tour (1691-1698) By Wang Hui Weak perspective projection The Kangxi Emperor's Southern Inspection Tour (1691-1698) By Wang Hui In this section, we will introduce the problem of camera calibration and discuss its importance. Next, we will look at some common methods for solving the Lecture 3 calibration problem. Camera Calibration • Recap of camera models • Camera calibration problem • Camera calibration with radial distortion • Example Reading: [FP] Chapter 1 “Geometric Camera Calibration” [HZ] Chapter 7 “Computation of Camera Matrix P” Some slides in this lecture are courtesy to Profs. J. Ponce, F-F Li Silvio Savarese Lecture 3 - 14-Jan-15 Recall that our camera is modeled using internal, or intrinsic, parameters (K) and external, or extrinsic, parameters ([R T]). Their product gives us the full 3x4 Projective camera projection matrix, shown above in the expanded form. It has 11 degrees of freedom (we discussed that in lecture 2). P! = M Pw = K[R T]Pw Internal parameters External parameters 3× 4 T &α −α cotθ uo # &r1 # &t x # $ ! $ T ! $ ! β R = r T = t K = $0 vo ! $ 2 ! $ y ! sinθ T $0 0 1 ! $r3 ! $t z ! % " % " % " We now pose the following problem: given one or more images taken by a camera, estimate its intrinsic and extrinsic parameters. Notice that from this Goal of calibration point on we change the notation for P which now is denoted as P, and P’ which w Estimate intrinsic and extrinsic parameters is now denoted as p.