Basic Principles of Imaging and

Lecture #2

Thanks to Shree Nayar, Ravi Ramamoorthi, Pat Hanrahan Computer Vision: Building Machines that See

Lighting Camera

Physical Models



Scene Interpretation

We need to understand the Geometric and Radiometric relations between the scene and its image. A Brief History of Images 1558

Camera Obscura, Gemma Frisius, 1558 A Brief History of Images 1558 1568

Lens Based Camera Obscura, 1568 A Brief History of Images 1558 1568


Still Life, Louis Jaques Mande Daguerre, 1837 A Brief History of Images 1558 1568


Silicon Image Detector, 1970

1970 A Brief History of Images 1558 1568


Digital Cameras 1970 1995 Geometric and Image Formation


1) Pinhole and Perspective Projection

2) Image Formation using Lenses

3) Lens related issues Pinhole and the Perspective Projection

Is an image being formed (x,y) on the screen?

YES! But, not a “clear” one.

screen scene

image plane r =(x, y, z) y optical effective focal length, f’ z axis pinhole x

r'=(x', y', f ')

r' r x' x y' y = = = f ' z f ' z f ' z Magnification

A(x, y, z) B y d B(x +δx, y +δy, z) optical f’ z A axis Pinhole A’ d’ x planar scene image plane B’ A'(x', y', f ') B'(x'+δx', y'+δy', f ')

From perspective projection: Magnification:

x' x y' y d' (δx')2 + (δy')2 f ' = = m = = = f ' z f ' z d (δx)2 + (δy)2 z

x'+δx' x +δx y'+δy' y +δy = = f ' z f ' z image = m2 Areascene Orthographic Projection

Magnification: x' = m x y' = m y When m = 1, we have orthographic projection

r =(x, y, z)

r'=(x', y', f ') y optical z axis x z ∆ image plane z

This is possible only when z >> ∆z In other words, the range of scene depths is assumed to be much smaller than the average scene depth.

But, how do we produce non-inverted images? Better Approximations to Perspective Projection Problems with Pinholes

• Pinhole size (aperture) must be “very small” to obtain a clear image.

• However, as pinhole size is made smaller, less is received by image plane.

• If pinhole is comparable to of incoming light, DIFFRACTION effects blur the image!

• Sharpest image is obtained when:

pinhole diameter d = 2 f ' λ

Example: If f’ = 50mm,

λ = 600nm (red),

d = 0.36mm Image Formation using Lenses

• Lenses are used to avoid problems with pinholes.

• Ideal Lens: Same projection as pinhole but gathers more light!

i o P



1 1 1 Gaussian Lens Formula: + = i o f • f is the focal length of the lens – determines the lens’s ability to bend (refract) light

• f different from the effective focal length f’ discussed before! Focus and Defocus

aperture aperture Blur Circle, b diameter d

i o i' o'

1 1 1 1 1 1 f f Gaussian Law: + = + = (i'−i) = (o − o') i o f i' o' f (o'− f ) (o − f )

d Blur Circle Diameter : b = (i ' − i) i'

Depth of Field: Range of object distances over which image is sufficiently well focused. i.e. Range for which blur circle is less than the resolution of the imaging sensor. Two Lens System


final object image i2 f2 o2 i1 f1

o1 image plane intermediate virtual image lens 2 lens 1

• Rule : Image formed by first lens is the object for the lens.

• Main Rays : Ray passing through focus emerges parallel to optical axis. Ray through optical center passes un-deviated.

i i • Magnification: m = 2 1 o2 o1

Exercises: What is the combined focal length of the system? What is the combined focal length if d = 0? Lens related issues

Compound (Thick) Lens Vignetting

L L L B principal planes 3 2 1

α A α nodal points

thickness more light from A than B !

Chromatic Abberation Radial and Tangential Distortion

ideal actual

F F B G ideal FR actual

image plane Lens has different refractive indices for different . and Image Formation

• To interpret image intensities, we need to understand Radiometric Concepts and Properties.


1) Image Intensities: Overview

2) Radiometric Concepts:

Radiant BRDF

3) Image Formation using a Lens

4) Radiometric Camera Calibration Image Intensities

source sensor

Need to consider normal light propagation in a cone

surface element

Image intensities = f ( normal, surface reflectance, illumination )

Note: Image intensity understanding is an under-constrained problem! Radiometric concepts – boring…but, important!

source dω ( subtended by dA )

dA' R (foreshortened area) dω θr θi dA (surface area) dA

dA' dA cos θ (1) Solid Angle : ω = = i ( ) (4) Surface Radiance (tricky) : d 2 2 R R 2 d Φ 2 What is the solid angle subtended by a hemisphere? L = ( / m steradian ) (dA cos θr ) dω dΦ (2) of Source : J = ( watts / steradian ) dω • emitted per unit foreshortened area per unit solid angle. Light Flux () emitted per unit solid angle

• L depends on direction θr

dΦ (3) Surface Irradiance : E = ( watts / m ) • Surface can radiate into whole hemisphere. dA • L depends on reflectance properties of surface. Light Flux (power) incident per unit surface area.

Does not depend on where the light is coming from! The Fundamental Assumption in Vision


No Change in


Surface Camera Radiance properties

• Radiance is constant as it propagates along ray – Derived from conservation of flux – Fundamental in Light Transport.

dΦ= L dωω dA = L d dA = d Φ 1 1112 2 2 2

22 dωω12= dA r d 21= dA r

dA dA dωω dA =12 = d dA 11 r 2 2 2

∴=LL12 Relationship between Scene and Image

• Before light hits the image plane:

Scene Scene Lens Image Radiance L Irradiance E

Linear Mapping!

• After light hits the image plane:

Image Camera Measured Irradiance E Electronics Pixel Values, I

Non-linear Mapping!

Can we go from measured pixel value, I, to scene radiance, L? Relation between Image Irradiance E and Scene Radiance L

image plane surface patch θ dAs dωs α α dωi image patch dωL dAi

f z

• Solid angles of the double cone (orange and green): 2 dAi cos α dAs cos θ dA cos α  z  dω = dω = s =   i s α 2 α 2   ( f / cos ) (z / cos ) dAi cos θ  f  • Solid angle subtended by lens: (1) π d 2 cosα dω = (2) L 4 (z / cosα)2 Relation between Image Irradiance E and Scene Radiance L image plane surface patch θ dAs dωs α α dωi image patch dωL dAi

f z

• Flux received by lens from dA s = Flux projected onto image dA i

(3) L (dAs cos θ ) dωL = E dAi

2 π  d  4 • From (1), (2), and (3): E = L   cos α 4  f 

• Image irradiance is proportional to Scene Radiance!

• Small field of view  Effects of 4th power of cosine are small. Relation between Pixel Values I and Image Irradiance E

Image Camera Measured Irradiance E Electronics Pixel Values, I

• The camera response function relates image irradiance at the image plane to the measured pixel intensity values. g : E → I

(Grossberg and Nayar) Radiometric Calibration •Important preprocessing step for many vision and graphics algorithms such as photometric stereo, invariants, de-weathering, inverse rendering, image based rendering, etc.

g −1 : I → E

•Use a color chart with precisely known .


g −1 ?

g Pixel Values

0 0 ? 1 90% 59.1% 36.2% 19.8% 9.0% 3.1% Irradiance = const * Reflectance

• Use more camera exposures to fill up the curve. • Method assumes constant lighting on all patches and works best when source is far away (example ).

• Unique inverse exists because g is monotonic and smooth for all cameras. The Problem of Dynamic Range

• Dynamic Range: Range of brightness values measurable with a camera

(Hood 1986)

• Today’s Cameras: Limited Dynamic Range

High Image Low Exposure Image

• We need 5-10 million values to store all around us. • But, typical 8-bit cameras provide only 256 values!! High Dynamic Range Imaging

• Capture a lot of images with different exposure settings.

• Apply radiometric calibration to each camera.

• Combine the calibrated images (for example, using averaging weighted by exposures).



Images taken with a fish-eye lens of the sky show the wide range of brightnesses.