Introduction to Computer Graphics GAMES101, Lingqi Yan, UC Santa Barbara

Lecture 19: , Lenses and Light Fields

http://www.cs.ucsb.edu/~lingqi/teaching/games101.html Announcements

• Standalone topics this week - No related homework for lectures in this week • Final project ideas will be released soon - Note: submit your (very short) proposal on/before Apr 19, if you want to do something else (encouraged)

GAMES101 2 Lingqi Yan, UC Santa Barbara Imaging = Synthesis + Capture What’s Happening Inside the ?

Cross-section of Nikon D3, 14-24mm F2.8 lens

GAMES101 4 Lingqi Yan, UC Santa Barbara Pinholes & Lenses Form Image on Sensor () London and Upton

GAMES101 5 Lingqi Yan, UC Santa Barbara Shutter Exposes Sensor For Precise Duration ()

The Slow Mo Guys, https://youtu.be/CmjeCchGRQo

GAMES101 6 Lingqi Yan, UC Santa Barbara Sensor Accumulates Irradiance During

GAMES101 7 Lingqi Yan, UC Santa Barbara Why Not Sensors Without Lenses?

Each sensor point would integrate light from all points on the object, so all values would be similar i.e. the sensor records irradiance London and Upton

but there is computational imaging research… GAMES101 8 Lingqi Yan, UC Santa Barbara Pinhole Image Formation

Mo Tzu (c. 470–c. 390 BC)

Aristotle (384–322 BC)

Ibn al-Haytham (965–1040)

Shen Kuo (1031–1095)

Roger Bacon (c. 1214–1294)

Johannes Kepler (1571–1630)

A. H. Zewail, Phil. Trans. R. Soc. A 2010;368:1191-1204

GAMES101 10 Lingqi Yan, UC Santa Barbara Largest Pinhole legacyphotoproject.com

GAMES101 11 Lingqi Yan, UC Santa Barbara Largest Pinhole Photograph legacyphotoproject.com

GAMES101 12 Lingqi Yan, UC Santa Barbara Field of View (FOV) () Effect of on FOV ()

h h FOV = 2 arctan FOV = 2 arctan 2f 2f ✓ ◆ ✓ ◆

Lens Focal length f f Sensor h h For a fixed sensor size, decreasing the focal length increases the field of view. h FOV = 2 arctan 2f ✓ ◆

GAMES101 14 Lingqi Yan, UC Santa Barbara Focal Length v. Field of View

• For historical reasons, it is common to refer to angular field of view by focal length of a lens used on a 35mm-format film (36 x 24mm) • Examples of focal lengths on 35mm format: • 17mm is wide angle 104° • 50mm is a “normal” lens 47° • 200mm is 12° • Careful! When we say current cell phones have approximately 28mm “equivalent” focal length, this uses the above convention.

GAMES101 15 Lingqi Yan, UC Santa Barbara Focal Length v. Field of View

From London and Upton, and Canon EF Lens Work III

GAMES101 16 Lingqi Yan, UC Santa Barbara Effect of Sensor Size on FOV

Object

APS-C

Lens 35mm Full Frame

Sensor(s)

GAMES101 17 Lingqi Yan, UC Santa Barbara Sensor Sizes

Credit: lensvid.com

GAMES101 18 Lingqi Yan, UC Santa Barbara Maintain FOV on Smaller Sensor?

Shorter Lens Lens focal Small Sensor Focal length length

Large Sensor

To maintain FOV, decrease focal length of lens in proportion to width/height of sensor

GAMES101 19 Lingqi Yan, UC Santa Barbara Exposure Exposure

• H = T x E • Exposure = time x irradiance • Exposure time (T) - Controlled by shutter • Irradiance (E) - Power of light falling on a unit area of sensor - Controlled by lens and focal length

GAMES101 21 Lingqi Yan, UC Santa Barbara Exposure Controls in

Aperture size • Change the f-stop by opening / closing the aperture (if camera has iris control)

Shutter speed • Change the duration the sensor integrate light

ISO gain () • Change the amplification (analog and/or digital) between sensor values and digital image values

GAMES101 22 Lingqi Yan, UC Santa Barbara Exposure: Aperture, Shutter, Gain (ISO)

Photoblog Hamburg

GAMES101 23 Lingqi Yan, UC Santa Barbara ISO (Gain)

Third variable for exposure Film: trade sensitivity for grain Digital: trade sensitivity for noise • Multiply signal before analog-to-digital conversion • Linear effect (ISO 200 needs half the light as ISO 100)

GAMES101 24 Lingqi Yan, UC Santa Barbara ISO Gain vs Noise in Canon T2i Credit: bobatkins.com

GAMES101 25 Lingqi Yan, UC Santa Barbara F-Number (F-Stop): Exposure Levels Written as FN or F/N. N is the f-number. Informal understanding: the inverse-diameter of a round aperture

https://www.dpmag.com/how-to/tip-of-the-week/how-and-why-to-use-auto-exposure-bracketing/

GAMES101 26 Lingqi Yan, UC Santa Barbara Physical Shutter (1/25 Sec Exposure)

The Slow Mo Guys, https://youtu.be/CmjeCchGRQo

GAMES101 27 Lingqi Yan, UC Santa Barbara Side Effect of

Motion blur: handshake, subject movement Doubling shutter time doubles motion blur http://www.gavtrain.com/?p=3960 Gavin Hoey

GAMES101 28 Lingqi Yan, UC Santa Barbara Side Effect of Shutter Speed

Note: motion blur is not always bad! Tip: think about anti-aliasing

London

GAMES101 29 Lingqi Yan, UC Santa Barbara Side Effect of Shutter Speed

Rolling shutter: different parts of photo taken at different times

https://www.premiumbeat.com/blog/3-tips-for-dealing-with-rolling-shutter/

GAMES101 30 Lingqi Yan, UC Santa Barbara Constant Exposure: F-Stop vs Shutter Speed

Example: these pairs of aperture and shutter speed give equivalent exposure

F-Stop 1.4 2.0 2.8 4.0 5.6 8.0 11.0 16.0 22.0 32.0

Shutter 1/500 1/250 1/125 1/60 1/30 1/15 1/8 1/4 1/2 1

If the exposure is too bright/dark, may need to adjust f-stop and/or shutter up/down.

• Photographers must trade off (?) and motion blur for moving subjects ()

GAMES101 31 Lingqi Yan, UC Santa Barbara Fast and Slow Photography High-Speed Photography

Normal exposure = extremely fast shutter speed x (large aperture and/or high ISO)

Mark Watson Harold Edgerton Slide courtesy L. Waller

GAMES101 33 Lingqi Yan, UC Santa Barbara High-Speed Photography

GAMES101 34 Harold Edgerton Lingqi Yan, UC Santa Barbara Long-Exposure Photography

https://www.demilked.com/best-long-exposure-photos/ GAMES101 35 Lingqi Yan, UC Santa Barbara Long-Exposure Photography

https://www.demilked.com/best-long-exposure-photos/ GAMES101 36 Lingqi Yan, UC Santa Barbara Long-Exposure Photography

https://www.demilked.com/best-long-exposure-photos/ GAMES101 37 Lingqi Yan, UC Santa Barbara Thin Lens Approximation Real Lens Designs Are Highly Complex [Apple]

GAMES101 39 Lingqi Yan, UC Santa Barbara , ], ￿ ￿￿￿￿ ￿￿￿￿ ]! Today, mm zoom int lens. ￿ ￿￿ ￿￿￿￿ ware approach to ￿ ]. ort that led to the ex- er the photograph is ￿ ￿ ￿￿￿￿ gured with the plenoptic erent glass elements, and ￿ ￿ di ￿￿ [Dickerson and Lepp eld camera records the light traveling ￿ ￿￿ eld camera con ￿ ]. As an example, commodity ￿￿￿￿￿￿￿ ￿￿￿￿￿￿￿￿￿￿ ￿￿ ￿￿￿￿ ￿￿￿￿￿￿￿￿￿￿￿ . ￿￿￿￿ is approach complements the classical optical tech- ￿ int glass lens and a crown glass lens. Finally, in is chapter introduces a new pure-so ￿ ￿￿￿￿￿￿￿ ￿ Real Lens Elements Are Not Ideal – Aberrations ￿ this phenomenon. Zooming a lens requiresof a at non-linear least three shi groups of lens elements relativemaking to it one another, very challenging toaberration maintain correction over a the reasonable zoom range. level However, of thevenience con- of the original zoom systems wasquickly so launched an desirable that intense it research e tremely sophisticated, but complex designtoday forms [Mann that we see lenses contain no fewer than some have as many as all modern lens design work iswhere computer-aided [Smith design forms are iterativelyOne optimized reason by for the a large computer. numbers ofprovide lens greater elements degrees is of freedom that for they the optimizer to achieve the desired optical quality [Kingslake compensating for lens aberrationstaken. a , Chevalier improved the design by splitting the meniscus into a ]. It consisted of a single-element meniscus lens with concave side to ￿￿￿￿ ￿￿￿￿ e most classical example of this might be the historical sequence of improve- nal images by reducing residual aberrations present in any given optical recipe. e central concept is simple: since a light : Spherical ￿ ￿ ￿ . ￿ ￿ [Kingslake Today the process of correcting aberrations by combining glass elements has been car- For , this chapter assumes a light ￿￿￿ system. ments in the original photographic￿￿￿￿ objective, a landscape lens designedan by aperture Wollaston stop. in In cemented doublet composed of a Dallmeyer split the crown lens again, placing one on either side of the central ried to remarkable extremes. Zoom lenses provide perhaps the most dramatic illustration of Figure aberration. niques. along all rays inside the camera, weto can where use they the should computer ideally to have re-sortquality converged. aberrated of rays Digital of correction light of this kind improves the

Real plano-convex lens (spherical surface shape). Lens does not converge rays to a point anywhere.

GAMES101 40 Lingqi Yan, UC Santa Barbara Ideal Thin Lens – Focal Point

Focal () Point

Focal Credit: Karen Watson Length

(1) All parallel rays entering a lens pass through its focal point. (2) All rays through a focal point will be in parallel after passing the lens. (3) Focal length can be arbitrarily changed (in reality, yes!).

GAMES101 41 Lingqi Yan, UC Santa Barbara The Thin Lens Equation

z f f o = f zi f 2 (zo f)(zi ff)=f f 2 2 z z (z +zo z )f + f = f zi o i o i zozi =(zo + zi)f 1 1 1 = + f zi zo

GAMES101 42 Lingqi Yan, UC Santa Barbara Gauss’ Ray Diagrams

Parallel Ray

Chief Ray

Focal Ray

Object Image

GAMES101 43 Lingqi Yan, UC Santa Barbara Gauss’ Ray Tracing Construction

f f

zo zi

What is the relationship between conjugate depths z o ,z i ?

GAMES101 44 Lingqi Yan, UC Santa Barbara Gauss’ Ray Tracing Construction

h h ho ho o = i h h f f zi f o = i zo f f f hi hi

h h h h o = i o = i z f f f zi f o

GAMES101 45 Lingqi Yan, UC Santa Barbara Gauss’ Ray Tracing Construction h h h h o = i o = i z f f f zi f o h z f ho f o = o = h f h z f i i i

zo f f Object / image heights = factor out - applies to all rays f zi f 2 (z f)(z f)=f Newtonian Thin Lens Equation o i z z (z + z )f + f 2 = f 2 o i o i zozi =(zo + zi)f 1 1 1 = + Gaussian Thin Lens Equation f zi zo

GAMES101 46 Lingqi Yan, UC Santa Barbara Thin Lens Demonstration

http://graphics.stanford.edu/courses/cs178-10/applets/gaussian.html

GAMES101 47 Lingqi Yan, UC Santa Barbara Defocus Blur Computing (CoC) Size

zs0 zs zo zi d!

A C

Object Focal Plane Plane

C d0 z z Circle of confusion is proportional = = | s i| to the size of the aperture A zi zi

GAMES101 49 Lingqi Yan, UC Santa Barbara CoC vs. Aperture Size

A side note: hilarious Google translate…

GAMES101 50 Lingqi Yan, UC Santa Barbara Revisiting F-Number (a.k.a. F-Stop)

• Formal definition: The f-number of a lens is defined as the focal length divided by the diameter of the aperture • Common f-stops on real lenses: 1.4, 2, 2.8, 4.0, 5.6, 8, 11, 16, 22, 32 • An f-stop of 2 is sometimes written f/2, reflecting the fact that the absolute aperture diameter (A) can be computed by dividing focal length (f) by the relative aperture (N).

GAMES101 51 Lingqi Yan, UC Santa Barbara Example F-Stop Calculations

D = 50 mm f = 100 mm N = f/D =2

D = 100 mm f = 200 mm N = f/D =2 D = 100 mm f = 400 mm N = f/D =4

GAMES101 52 Lingqi Yan, UC Santa Barbara Size of CoC is Inversely Proportional to F-Stop R. Berdan, canadiannaturephotographer.com

z z f z z C = A| s i| = | s i| zi N zi

GAMES101 53 Lingqi Yan, UC Santa Barbara Ray Tracing Ideal Thin Lenses Examples of Renderings with Lens Focus

Pharr and Humphreys

GAMES101 55 Lingqi Yan, UC Santa Barbara Ray Tracing for Defocus Blur (Thin Lens)

x’’

x’ x’’’

Sensor

zo zi Subject plane

(One possible) Setup: • Choose sensor size, lens focal length and aperture size

• Choose depth of subject of interest zo

• Calculate corresponding depth of sensor zi from thin lens equation

GAMES101 56 Lingqi Yan, UC Santa Barbara Ray Tracing for Defocus Blur (Thin Lens)

x’’

x’ x’’’

Sensor

zo zi Subject plane Rendering: • For each pixel x’ on the sensor (actually, film ()) • Sample random points x’’ on lens plane • You know the ray passing through the lens will hit x’’’ (because x’’’ is in focus, consider virtual ray (x’, center of the lens)) • Estimate radiance on ray x’’ -> x’’’

GAMES101 57 Lingqi Yan, UC Santa Barbara Depth of Field Depth of Field

• Depth of field is the range of object depths that are rendered with acceptable sharpness in an image From London and Upton

Set circle of confusion as the maximum permissible blur spot on the image plane that will appear sharp under final viewing conditions

GAMES101 59 Lingqi Yan, UC Santa Barbara from subjects closer than “infinity” Depth of field circle of confusion by the F number, converge at a point in the focal plane. The area in front of and behind a focused regardless of the lens focal length. With subject in which the photographed image modern autofocus SLR cameras, focusing CircleFigure-15 of Confusion Relationship Between for Depth the Ideal Focal of Field appears sharp. In other words, the depth of is performed by detecting the state of Point and the Permissible Circle of Confusion and Depth of Field sharpness to the front and rear of the focus in the image plane (focal plane) i.e. depth range in a scene wheresubject where image blur in the focal using a sensor which is both optically the corresponding CoC is consideredplane falls within the limits of the equivalent (1:1 magnification) and small enough permissible circle of confusion. Depth of positioned out of the focal plane, and field varies according to the lens’ focal automatically controlling the lens to bring length, aperture value and shooting the subject image within the depth of

distance, so[Canon, EF Lens Work III] if these values are known, a focus area. Lens Ideal focal point rough estimate of the depth of field can be Figure-17 Relationship Between Depth of Fr on o t d calculated using the following formulas: Focus and Aperture f fi ep eld th 2 2 Re Front depth of field = d·F·a /(f + d·F·a) 50mm f/1.8 D ar ep of de 2 2 Aperture th fie pt Permissible of ld h Rear depth of field = d·F·a /(f — d·F·a) foc circle of confusion us f: focal length F: F number d: minimum f/1.8 Permissible circle of confusion circle of confusion diameter

GAMES101 60 Lingqi Yan, UCa: Santa subject Barbara distance (distance from the Circle of confusion first principal point to subject) at maximum aperture Since all lenses contain a certain amount Aperture × Permissible of spherical aberration and astigmatism, Near point limiting shooting distance circle of confusion = they cannot perfectly converge rays from a distance hyperfocal distance + f/5.6 subject point to form a true image point shooting distance Depth of hyperfocal distance × focus at f/5.6 (i.e., an infinitely small dot with zero area). Far point limiting shooting distance distance = In other words, images are formed from a hyperfocal distance - shooting distance composite of dots (not points) having a (Shooting distance: Distance from focal plane to subject) Hyperfocal distance certain area, or size. Since the image Using the depth of field principle, as a becomes less sharp as the size of these If the hyperfocal distance is known, the lens is gradually focused to farther dots increases, the dots are called “circles following formulas can also be used: subject distances, a point will eventually of confusion.” Thus, one way of indicating In general photography, depth of field is be reached where the far limit of the the quality of a lens is by the smallest dot characterised by the following attributes: rear depth of field will be equivalent to it can form, or its “minimum circle of ቢ Depth of field is deep at short focal “infinity.” The shooting distance at this confusion.” The maximum allowable dot lengths, shallow at long focal lengths. point, i,e., the closest shooting distance size in an image is called the “permissible ባ Depth of field is deep at small at which “infinity” falls within the depth circle of confusion.” , shallow at large apertures. of field, is called the hyperfocal distance. ቤ Depth of field is deep at far shooting The hyperfocal distance can be Permissible circle of confusion distances, shallow at close shooting determined as follows: The largest circle of confusion which still distances. appears as a “point” in the image. Image ብ Front depth of field is shallower Hyperfocal f 2 f: focal length F: F number distance = sharpness as sensed by the human eye is than rear depth of field. d•F number d: minimum circle of confusion closely related to the sharpness of the diameter Figure-16 Depth of Field and Depth of Focus actual image and the “resolution” of Minimum circle of confusion Thus, by presetting the lens to the human eyesight. In photography, image hyperfocal distance, the depth of field will sharpness is also dependent on the degree extend from a distance equal to half the Depth of field Depth of focus of image enlargement or projection hyperfocal distance to infinity. This

distance and the distance from which the Far point Near point method is useful for presetting a large image is viewed. In other words, in depth of field and taking snapshots

practical work it is possible to determine Front without having to worry about adjusting Rear Front depth of field depth depth of certain “allowances” for producing images focus the lens focus, especially when using a of field Near point distance which, although actually blurred to a Image Subject distance distance Rear wide-angle Photo-1 Hyperfocal Length Set certain degree, still appear sharp to the depth Condtion Far point distance of focus lens. (For observer. For 35mm single lens reflex Shooting distance example, when cameras, the permissible circle of confusion Focal plane the EF 20mm is about 1/1000~1/1500 the length of the f/2.8 USM is set film diagonal, assuming the image is Depth of focus to f/16 and the enlarged to a 5”×7” (12 cm × 16.5 cm) The area in front of and behind the focal shooting print and viewed from a distance of 25~30 plane in which the image can be distance is set cm/0.8~1 ft. EF lenses are designed to photographed as a sharp image. Depth of to the hyperfocal distance of ap- produce a minimum circle of confusion of focus is the same on both sides of the proximately 0.7m/2.3ft, all subjects within 0.035 mm, a value on which calculations image plane (focal plane) and can be a range of approximately 0.4m/1.3ft from for items such as depth of field are based. determined by multiplying the minimum the camera to infinity will be in focus.)

197 Depth of Field (FYI)

dN dS C A = d A Depth of field Depth of focus N d d C S F = dF A Circle of confusion, C f NN == DA 1 1 1 + = f f DF dF f 1 1 1 DF dF + = DS dS f DS dS 1 1 1 DN dN + = DN dN f DOF = D D F N D f 2 D f 2 D = S D = S F f 2 NC(D f) N f 2 + NC(D f) S S

GAMES101 61 Lingqi Yan, UC Santa Barbara DOF Demonstration (FYI)

http://graphics.stanford.edu/courses/cs178/applets/dof.html

GAMES101 62 Lingqi Yan, UC Santa Barbara Thank you!

(And thank Prof. Ren Ng for many of the slides!)