CS664 Computer Vision

5. Image Geometry

Dan Huttenlocher First Assignment

ƒ Wells paper handout for question 1 ƒ Question 2 more open ended – Less accurate approximations • Simple box filtering doesn’t work – Anisotropic, spatially dependent

2 Image Warping ƒ Image filtering: change range of image g(x) = T¸f(x)

f g T

ƒ Image warping: change domain of image g(x) = f(T(x))

f g T

3 Feature Detection in Images

ƒ Filtering to provide area of support – Gaussian, bilateral, … ƒ Measures of local image difference – Edges, corners

ƒ More sophisticated features are invariant to certain transformations or warps of the image – E.g., as occur when viewing direction changes

4 Parametric (Global) Warping

ƒ Examples of parametric warps:

translation rotation

projective affine cylindrical

5 Parametric (Global) Warping

T

p = (x,y) p’ = (x’,y’) p’ = T(p) ƒ What does it mean that T is global? – Same function for any point p – Described by a few parameters, often matrix x' x p’ = M*p ⎡ ⎤ ⎡ ⎤ ⎢ ⎥ = M⎢ ⎥ ⎣y'⎦ ⎣y⎦

6 Scaling

ƒ Scaling a coordinate means multiplying each of its components (axes) by a scalar ƒ Uniform scaling means this scalar is the same for all components:

× 2

7 Scaling

ƒ Non-uniform scaling: different scalars per component:

X × 2, Y × 0.5

8 Scaling

ƒ Scaling operation: x'= ax y'= by

ƒ Or, in matrix form: ⎡x'⎤ ⎡a 0⎤⎡x⎤ ⎢ ⎥ = ⎢ ⎥⎢ ⎥ ⎣y'⎦ ⎣0 b⎦⎣y⎦

scaling matrix S What’s inverse of S?

9 2-D Rotation

(x’, y’) About origin

(x, y)

x’ = x cos(θ) - y sin(θ) θ y’ = x sin(θ) + y cos(θ)

10 2-D Rotation

ƒ This is easy to captureθ in matrix form: ⎡x'⎤ ⎡cos(θ ) − sinθ( )⎤⎡x⎤ ⎢ ⎥ = ⎢() ()⎥⎢ ⎥ ⎣y'⎦ ⎣sin cos θ ⎦⎣y⎦

R ƒ Even though sin(θ) and cos(θ) are nonlinear functions of θ, – x’ is a linear combination of x and y – y’ is a linear combination of x and y ƒ Inverse transformation, rotation by –θ – For rotation matrices, det(R) = 1 and R −1 = R T

11 2x2 Transformation Matrices

ƒ What types of transformations can be represented with a 2x2 matrix?

2D Identity? (A rotation) x'= x ⎡x'⎤ = ⎡1 0⎤⎡x⎤ y'= y ⎣⎢y'⎦⎥ ⎣⎢0 1⎦⎥⎣⎢y⎦⎥

2D Scale around (0,0)?

x'= s x * x ⎡x'⎤ ⎡sx 0 ⎤⎡x⎤ ⎢ ⎥ = ⎢ ⎥⎢ ⎥ y'= s y * y ⎣ y'⎦ ⎣ 0 s y ⎦⎣ y⎦

12 2x2 Transformation Matrices

ƒ What types of transformations can be represented with a 2x2 matrix?

2D Rotate around (0,0)? ⎡x'⎤ ⎡cosΘ − sin Θ⎤⎡x⎤ x'= cosΘ* x − sin Θ* y ⎢ ⎥ = ⎢ ⎥⎢ ⎥ y'= sin Θ* x + cosΘ* y ⎣ y'⎦ ⎣sin Θ cosΘ ⎦⎣ y⎦

2D Shear? ⎡x'⎤ ⎡ 1 sh ⎤⎡x⎤ = x x'= x + shx * y ⎢ ⎥ ⎢ ⎥⎢ ⎥ ⎣ y'⎦ ⎣shy 1 ⎦⎣ y⎦ y'= shy * x + y

13 2x2 Transformation Matrices

ƒ What types of transformations can be represented with a 2x2 matrix?

2D Mirror about Y axis? x'= −x ⎡x'⎤ = ⎡−1 0⎤⎡x⎤ y'= y ⎣⎢y'⎦⎥ ⎣⎢ 0 1⎦⎥⎣⎢y⎦⎥

2D Mirror over (0,0)?

x'= −x ⎡x'⎤ = ⎡−1 0 ⎤⎡x⎤ y'= −y ⎣⎢y'⎦⎥ ⎣⎢ 0 −1⎦⎥⎣⎢y⎦⎥

14 All 2D Linear Transformations

ƒ Linear transformations are combinations of … – Scale, ⎡x'⎤ ⎡a b⎤⎡x⎤ – Rotation, = – Shear, and ⎢y'⎥ ⎢c d⎥⎢y⎥ – Mirror ⎣ ⎦ ⎣ ⎦⎣ ⎦ ƒ Properties of linear transformations: – Origin maps to origin – Lines map to lines – Parallel lines remain parallel – Ratios are preserved – Closed under composition e f i j ⎡x'⎤ = ⎡a b⎤⎡ ⎤⎡ ⎤⎡x⎤ ⎣⎢y'⎦⎥ ⎣⎢c d⎦⎥⎣⎢g h⎦⎥⎣⎢k l ⎦⎥⎣⎢y⎦⎥

15 Linear Transformations as Change of Basis j =(0,1) v =(vx,vy)

p’ p

u=(ux,uy) i =(1,0) p ’=4u +3v p=4i+3j = (4,3) p’=4u+3v x x x py’=4uy+3vy ⎡u x v x ⎤⎡4⎤ ⎡u x v x ⎤ p' = ⎢ ⎥⎢ ⎥ = ⎢ ⎥p ⎣u y v y ⎦⎣3⎦ ⎣u y v y ⎦ ƒ Any linear transformation is a basis!

16 2x2 Transformation Matrices

ƒ What types of transformations can be represented with a 2x2 matrix?

2D Translation? x'= x + t x NO! y'= y + t y

Only linear 2D transformations can be represented with a 2x2 matrix

17 Homogeneous Coordinates

ƒ How can we represent translation as matrix? x'= x + t x

y'= y + t y

ƒ Homogeneous coordinates – Represent coordinates in 2 dimensions with a 3- vector ⎡x⎤ x ⎡ ⎤ homogeneous coords ⎢ ⎥ ⎢ ⎥ ⎯⎯→⎯⎯⎯ ⎢y⎥ ⎣y⎦ ⎣⎢1⎦⎥

18 Homogeneous Coordinates

ƒ Add a 3rd coordinate to every 2D point – (x, y, w) represents a point at 2D location (x/w, y/w) – (x, y, 0) represents a point at infinity – (0, 0, 0) is not allowed y 2 (2,1,1) or (4,2,2) or (6,3,3) 1

1 2 x

19 Homogeneous Coordinates ƒ How can we represent translation as matrix? x'= x + t x

y'= y + t y ƒ Last column of homogeneous matrix

⎡1 0 t x ⎤ ⎢ ⎥ Translation = ⎢0 1 t y ⎥ ⎢ ⎥ ⎣0 0 1 ⎦

20 Translation

ƒ Example of translation in homogeneous coordinates

⎡ x'⎤ ⎡1 0 t x ⎤⎡ x ⎤ ⎡ x + t x ⎤ ⎢ ⎥ ⎢ y'⎥ = 0 1 t ⎢ y⎥ = ⎢ y + t ⎥ ⎢ ⎥ ⎢ y ⎥⎢ ⎥ ⎢ y ⎥ ⎢ ⎥ ⎣⎢ 1 ⎦⎥ ⎣0 0 1 ⎦⎣⎢1 ⎦⎥ ⎣⎢ 1 ⎦⎥

tx = 2 ty = 1

21 Homogeneous 2D Transformations

ƒ Basic 2D transformations as 3x3 matrices

⎡x'⎤ ⎡1 0 tx ⎤⎡x⎤ ⎡x'⎤ ⎡sx 0 0⎤⎡x⎤ ⎢ ⎥ ⎢ ⎥⎢ ⎥ ⎢y'⎥ = ⎢0 1 t ⎥⎢y⎥ y' = 0 s 0 y ⎢ ⎥ ⎢ y ⎥⎢ ⎥ ⎢ ⎥ ⎢ y ⎥⎢ ⎥ ⎣⎢ 1 ⎦⎥ ⎣⎢0 0 1 ⎦⎥⎣⎢1⎦⎥ ⎣⎢ 1 ⎦⎥ ⎣⎢ 0 0 1⎦⎥⎣⎢1⎦⎥ Translate Scale

⎡x'⎤ ⎡cosΘ − sin Θ 0⎤⎡x⎤ ⎡x'⎤ ⎡ 1 shx 0⎤⎡x⎤ ⎢ ⎥ ⎢ ⎥⎢ ⎥ y' = sin Θ cosΘ 0 y ⎢ y'⎥ = ⎢sh 1 0⎥⎢ y⎥ ⎢ ⎥ ⎢ ⎥⎢ ⎥ ⎢ ⎥ ⎢ y ⎥⎢ ⎥ ⎣⎢1 ⎦⎥ ⎣⎢ 0 0 1⎦⎥⎣⎢1⎦⎥ ⎣⎢ 1 ⎦⎥ ⎣⎢ 0 0 1⎦⎥⎣⎢1⎦⎥ Rotate Shear

22 Affine Transformations

ƒ Affine transformations are … ⎡x'⎤ ⎡a b c ⎤⎡ x⎤ ⎢y'⎥ = ⎢d e f ⎥⎢ y⎥ – Linear transformations, and ⎢w⎥ ⎢0 0 1 ⎥⎢w⎥ – Translations ⎣ ⎦ ⎣ ⎦⎣ ⎦ ƒ Properties of affine transformations: – Origin does not necessarily map to origin – Lines map to lines – Parallel lines remain parallel – Ratios are preserved – Closed under composition – Models change of basis – Maps any triangle to any triangle (or parallelogram)

23 Projective Transformations

⎡ ⎤ ⎡ ⎤⎡ ⎤ ƒ Projective transformations … x' a b c x ⎢ y'⎥ = ⎢d e f ⎥⎢ y⎥ – Affine transformations, and ⎢ ⎥ ⎢ ⎥⎢ ⎥ ⎣w'⎦ ⎣g h i ⎦⎣w⎦ – Projective warps ƒ Properties of projective transformations: – Origin does not necessarily map to origin – Lines map to lines – Parallel lines do not necessarily remain parallel – Ratios are not preserved – Closed under composition – Models change of basis – Maps any quadrilateral to any quadrilateral

24 Matrix Composition

ƒ Transformations can be combined (composed) by matrix multiplication

⎡ x'⎤ ⎛ ⎡1 0 tx⎤ cosΘ − sin Θ 0 ⎡sx 0 0⎤⎞⎡ x⎤ ⎜ ⎡ ⎤ ⎟ ⎢ y'⎥ = ⎢0 1 ty⎥⎢sin Θ cosΘ 0⎥⎢ 0 sy 0⎥ ⎢ y⎥ ⎢ ⎥ ⎜ ⎢ ⎥ ⎢ ⎥⎟⎢ ⎥ ⎣w'⎦ ⎝ ⎣0 0 1 ⎦⎣⎢ 0 0 1⎦⎥⎣ 0 0 1⎦⎠⎣w⎦

p’ = T(tx,ty) R(Θ) S(sx,sy) p

25 2D image transformations

• Nested set of groups

26 Affine Invariants

ƒ Affine transformations preserve collinearity of points: – If 3 points lie on the same line, their images under affine transformations also line on the same line and, – The middle point remains between the other two points ƒ Concurrent lines remain concurrent (images of intersecting lines intersect), ƒ Ratio of length of line segments of a given line remains constant ƒ Ratio of areas of two triangles remains constant, ƒ Ellipses remain ellipses and the same is true for parabolas and hyperbolas

27 3D Interpretations

ƒ Orthographic projection plus scaling of Euclidean motion in 3D world yields 2D affine transformation – Often called weak perspective imaging model ƒ Set of coplanar points in 3D – Undergo rigid body motion – Scaling – analogous to perspective size with distance but same scaling for all the points – Projection into image plane (drop coordinate)

28 Orthographic Projection

Image World

ƒ Also called “parallel projection”: (x, y, z) → (x, y) ƒ What’s the projection matrix (homogeneous coords)?

29 Weak Perspective Imaging

ƒ Scaling in addition to parallel projection

ƒ Composed with 3D rigid body motion (6 dof)

ƒ Equivalent to affine transformation of plane (6 dof) up to reflection – state without proof

30 Projection – Pinhole Camera Model

ƒ Projection equations – Compute intersection with PP of ray from (x,y,z) to COP – Derived using similar triangles – Parallel projection where d infinite

31 Perspective Projection

ƒ Homogeneous coordinates

Divide by third coordinate

Scaling projection matrix, no effect

32 Perspective Projection

ƒ Composed with 3D rigid body motion (6 dof) ƒ Focal length, d, and distance z two additional parameters ƒ Equivalent to projective transformation of plane (homography) – 3x3 matrix in homogeneous coordinates, 8 dof – Again state without proof ƒ 2D affine and projective transformations correspond to images of plane in space under rigid motion, different imaging models

33 Image Warping

T(x,y) yy’ xx’f(x,y) g(x’,y’)

ƒ Given coordinate transform (x’,y’) = T(x,y) and source image f(x,y) ƒ How do we compute transformed image g(x’,y’) = f(T(x,y))?

34 Forward Warping

T(x,y) yy’ xx’f(x,y) g(x’,y’) ƒ Send each pixel f(x,y) to its corresponding location (x’,y’) = T(x,y) in the second image

Q: What about when pixel lands “between” two pixels?

35 Forward Warping

Points at pixel T(x,y) centers yy’ xx’f(x,y) g(x’,y’) ƒ Send each pixel f(x,y) to its corresponding location (x’,y’) = T(x,y) in the second image Q: What about when pixel lands “between” two pixels? A: Distribute color among neighboring pixels (x’,y’) – Known as “splatting”

36 Inverse Warping

T-1(x,y) y y’ xxx’f(x,y) g(x’,y’) ƒ Get each pixel g(x’,y’) from its corresponding location (x,y)=T-1(x’,y’) in the first image

Q: What about when pixel comes from “between” two pixels?

37 Inverse Warping

T-1(x,y) y y’ xxx’f(x,y) g(x’,y’) ƒ Get each pixel g(x’,y’) from its corresponding location (x,y)=T-1(x’,y’) in the first image Q: What about when pixel comes from “between” two pixels? A: Interpolate color value from neighbors – Nearest neighbor, bilinear, Gaussian, bicubic

38 Bilinear Interpolation (Reminder)

ƒ Sampling at f(x,y):

39 Forward vs. Inverse Warping

ƒ Q: Which is better?

ƒ A: Usually inverse – eliminates holes – However, requires an invertible warp function – not always possible...

projective affine

40 Reference

ƒ R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision, 2nd Ed., Cambridge Univ. Press, 2003.

41