Basic Principles of Imaging and Photometry Lecture #2 Thanks to Shree Nayar, Ravi Ramamoorthi, Pat Hanrahan Computer Vision: Building Machines that See Lighting Camera Physical Models Computer Scene Scene Interpretation We need to understand the Geometric and Radiometric relations between the scene and its image. A Brief History of Images 1558 Camera Obscura, Gemma Frisius, 1558 A Brief History of Images 1558 1568 Lens Based Camera Obscura, 1568 A Brief History of Images 1558 1568 1837 Still Life, Louis Jaques Mande Daguerre, 1837 A Brief History of Images 1558 1568 1837 Silicon Image Detector, 1970 1970 A Brief History of Images 1558 1568 1837 Digital Cameras 1970 1995 Geometric Optics and Image Formation TOPICS TO BE COVERED : 1) Pinhole and Perspective Projection 2) Image Formation using Lenses 3) Lens related issues Pinhole and the Perspective Projection Is an image being formed (x,y) on the screen? YES! But, not a “clear” one. screen scene image plane r =(x, y, z) y optical effective focal length, f’ z axis pinhole x r'=(x', y', f ') r' r x' x y' y = = = f ' z f ' z f ' z Magnification A(x, y, z) B y d B(x +δx, y +δy, z) optical f’ z A axis Pinhole A’ d’ x planar scene image plane B’ A'(x', y', f ') B'(x'+δx', y'+δy', f ') From perspective projection: Magnification: x' x y' y d' (δx')2 + (δy')2 f ' = = m = = = f ' z f ' z d (δx)2 + (δy)2 z x'+δx' x +δx y'+δy' y +δy = = Area f ' z f ' z image = m2 Areascene Orthographic Projection Magnification: x' = m x y' = m y When m = 1, we have orthographic projection r =(x, y, z) r'=(x', y', f ') y optical z axis x z ∆ image plane z This is possible only when z >> ∆z In other words, the range of scene depths is assumed to be much smaller than the average scene depth. But, how do we produce non-inverted images? Better Approximations to Perspective Projection Problems with Pinholes • Pinhole size (aperture) must be “very small” to obtain a clear image. • However, as pinhole size is made smaller, less light is received by image plane. • If pinhole is comparable to wavelength of incoming light, DIFFRACTION effects blur the image! • Sharpest image is obtained when: pinhole diameter d = 2 f ' λ Example: If f’ = 50mm, λ = 600nm (red), d = 0.36mm Image Formation using Lenses • Lenses are used to avoid problems with pinholes. • Ideal Lens: Same projection as pinhole but gathers more light! i o P P’ f 1 1 1 Gaussian Lens Formula: + = i o f • f is the focal length of the lens – determines the lens’s ability to bend (refract) light • f different from the effective focal length f’ discussed before! Focus and Defocus aperture aperture Blur Circle, b diameter d i o i' o' 1 1 1 1 1 1 f f Gaussian Law: + = + = (i'−i) = (o − o') i o f i' o' f (o'− f ) (o − f ) d Blur Circle Diameter : b = (i ' − i) i' Depth of Field: Range of object distances over which image is sufficiently well focused. i.e. Range for which blur circle is less than the resolution of the imaging sensor. Two Lens System d final object image i2 f2 o2 i1 f1 o1 image plane intermediate virtual image lens 2 lens 1 • Rule : Image formed by first lens is the object for the second lens. • Main Rays : Ray passing through focus emerges parallel to optical axis. Ray through optical center passes un-deviated. i i • Magnification: m = 2 1 o2 o1 Exercises: What is the combined focal length of the system? What is the combined focal length if d = 0? Lens related issues Compound (Thick) Lens Vignetting L L L B principal planes 3 2 1 α A α nodal points thickness more light from A than B ! Chromatic Abberation Radial and Tangential Distortion ideal actual F F B G ideal FR actual image plane Lens has different refractive indices for different wavelengths. Radiometry and Image Formation • To interpret image intensities, we need to understand Radiometric Concepts and Reflectance Properties. • TOPICS TO BE COVERED: 1) Image Intensities: Overview 2) Radiometric Concepts: Radiant Intensity Irradiance Radiance BRDF 3) Image Formation using a Lens 4) Radiometric Camera Calibration Image Intensities source sensor Need to consider normal light propagation in a cone surface element Image intensities = f ( normal, surface reflectance, illumination ) Note: Image intensity understanding is an under-constrained problem! Radiometric concepts – boring…but, important! source dω (solid angle subtended by dA ) dA' R (foreshortened area) dω θr θi dA (surface area) dA dA' dA cos θ (1) Solid Angle : ω = = i ( steradian ) (4) Surface Radiance (tricky) : d 2 2 R R 2 d Φ 2 What is the solid angle subtended by a hemisphere? L = (watts / m steradian ) (dA cos θr ) dω dΦ (2) Radiant Intensity of Source : J = ( watts / steradian ) dω • Flux emitted per unit foreshortened area per unit solid angle. Light Flux (power) emitted per unit solid angle • L depends on direction θr dΦ (3) Surface Irradiance : E = ( watts / m ) • Surface can radiate into whole hemisphere. dA • L depends on reflectance properties of surface. Light Flux (power) incident per unit surface area. Does not depend on where the light is coming from! The Fundamental Assumption in Vision Lighting No Change in Radiance Surface Camera Radiance properties • Radiance is constant as it propagates along ray – Derived from conservation of flux – Fundamental in Light Transport. dΦ= L dωω dA = L d dA = d Φ 1 1112 2 2 2 22 dωω12= dA r d 21= dA r dA dA dωω dA =12 = d dA 11 r 2 2 2 ∴=LL12 Relationship between Scene and Image Brightness • Before light hits the image plane: Scene Scene Lens Image Radiance L Irradiance E Linear Mapping! • After light hits the image plane: Image Camera Measured Irradiance E Electronics Pixel Values, I Non-linear Mapping! Can we go from measured pixel value, I, to scene radiance, L? Relation between Image Irradiance E and Scene Radiance L image plane surface patch θ dAs dωs α α dωi image patch dωL dAi f z • Solid angles of the double cone (orange and green): 2 dAi cos α dAs cos θ dA cos α z dω = dω = s = i s α 2 α 2 ( f / cos ) (z / cos ) dAi cos θ f • Solid angle subtended by lens: (1) π d 2 cosα dω = (2) L 4 (z / cosα)2 Relation between Image Irradiance E and Scene Radiance L image plane surface patch θ dAs dωs α α dωi image patch dωL dAi f z • Flux received by lens from dA s = Flux projected onto image dA i (3) L (dAs cos θ ) dωL = E dAi 2 π d 4 • From (1), (2), and (3): E = L cos α 4 f • Image irradiance is proportional to Scene Radiance! • Small field of view Effects of 4th power of cosine are small. Relation between Pixel Values I and Image Irradiance E Image Camera Measured Irradiance E Electronics Pixel Values, I • The camera response function relates image irradiance at the image plane to the measured pixel intensity values. g : E → I (Grossberg and Nayar) Radiometric Calibration •Important preprocessing step for many vision and graphics algorithms such as photometric stereo, invariants, de-weathering, inverse rendering, image based rendering, etc. g −1 : I → E •Use a color chart with precisely known reflectances. 255 g −1 ? g Pixel Values 0 0 ? 1 90% 59.1% 36.2% 19.8% 9.0% 3.1% Irradiance = const * Reflectance • Use more camera exposures to fill up the curve. • Method assumes constant lighting on all patches and works best when source is far away (example sunlight). • Unique inverse exists because g is monotonic and smooth for all cameras. The Problem of Dynamic Range • Dynamic Range: Range of brightness values measurable with a camera (Hood 1986) • Today’s Cameras: Limited Dynamic Range High Exposure Image Low Exposure Image • We need 5-10 million values to store all brightnesses around us. • But, typical 8-bit cameras provide only 256 values!! High Dynamic Range Imaging • Capture a lot of images with different exposure settings. • Apply radiometric calibration to each camera. • Combine the calibrated images (for example, using averaging weighted by exposures). (Mitsunaga) (Debevec) Images taken with a fish-eye lens of the sky show the wide range of brightnesses. .