Sense and Sensitivity Why Do Pixels Shrink? 2009 International Image Sensor Workshop

Pixel race

Not a race for more pixels: it’s a race for smaller pixels! Sense and sensitivity Why do pixels shrink? 2009 International Image Sensor Workshop

• Because we can! Mats Wernersson and Henrik Eliasson • Easy to sell larger pixel counts • Demand for smaller cameras

Rev 1 Rev 2

The Pixel Race - Because we can! How to make sense of the megapixel race

Predicted pixel race

• The maximum useful pixel count is ultimately limited by optical diffraction and photon shot noise. Time line shows when Sony Ericsson Camera Phones were introduced on the market

Slightly longer time between pixel nodes

Rev 3 Rev 4

Resolution Resolution

f/2.8 f/2.0 • MTF calculated as the product of 1 1 diffraction limited optical MTF and • For a 0.8 m pixel pitch, the pixel geometric MTF. The optical 0.9 f-number needs to be lowered 0.9 MTF is a polychromatic MTF under to f/2.0, according to the D65 lighting. 0.8 definition on the previous slide. 0.8

• Dashed lines are the Nyquist 0.7 1.4 m 0.7 1.4 m frequencies for the respective pixel pitch. 0.6 0.6 1.1 m 1.1 m

• This sets an absolute limit for the 0.5 0.5 MTF MTF resolution of a certain sensor/lens f- 0.8 m 0.8 m number combination. 0.4 0.4

• Setting this limit at an MTF of 10% 0.3 0.65 m 0.3 0.65 m means that f/2.8 should not be used for pixel pitches below 1.1 m, in 0.2 0.2 order to take advantage of the increased pixel count. 0.1 0.1

0 0 0 100 200 300 400 500 600 700 800 0 100 200 300 400 500 600 700 800 Spatial frequency [cy/mm] Spatial frequency [cy/mm]

Rev 5 Rev 6

Rev 1 Sharpness Resolution Same optical format, various pixel pitch • So far we have only f/1.4 f/2.0 1 discussed the limiting 1 • Decreasing the pixel pitch resolution of the camera even further, the f-number 0.9 system. The impression of 0.9 must keep on decreasing. sharpness is critically 0.8 dependent on lower spatial 0.8 frequencies. • Lowering the f-number is • Using SQF (subjective quality 0.7 1.4 m 0.7 1.4 m very challenging for the optical factor), we see that there is designer: aberrations increase, 0.6 only a marginal increase in 0.6 SQF range 1.1 m especially off-axis, meaning 1.1 m perceived sharpness when the that it might not be possible to 0.5 pixel size is decreased. 0.5 MTF MTF maintain diffraction limited 0.8 m 0.8 m optics for smaller f-numbers. 0.4 SQF for 24x32 cm image @ 45 0.4 cm viewing distance, 1/2.8” 0.3 0.65 m 0.3 0.65 m optical format

0.2 0.2 SQF values 0.1 1.4 m 92% 0.1 0 0 0 100 200 300 400 500 600 700 800 0.65 m 93% 0 100 200 300 400 500 600 700 800 Spatial frequency [cy/mm] Spatial frequency [cy/mm]

Rev 7 Rev 8

Photon shot noise and pixel size – same sensor Photon shot noise and pixel size – same sensor size size Illuminance: 5000 lux Illuminance: 100 lux

2.8 m 2.0 m 1.4 m 0.8 m 2.8 m 2.0 m 1.4 m 0.8 m t = 1/70 s t = 1/70 s t = 1/50 s t = 1/35 s t = 1/5 s t = 1/5 s t = 1/5 s t = 1/5 s Gain: 0 dB Gain: 0 dB Gain: 0 dB Gain: 0 dB Gain: 10 dB Gain: 10 dB Gain: 14 dB Gain: 18 dB S/N: 90 S/N: 73 S/N: 69 S/N: 56 S/N: 68 S/N: 55 S/N: 39 S/N: 24

Rev 9 Rev 10

How to make sense of the megapixel race The dark side Smaller cameras: Keeping the pixel count means smaller sensor (optical • The maximum useful pixel count is ultimately limited by optical diffraction and format) => image quality for a given megapixel count will decrease photon shot noise. with newer pixel generations. • However, there is a fair chance to maintain a reasonable image quality between pixel generations if the optical format is kept (and disregarding other implications arising from smaller pixels) †.

Aberration and process technology limitations are being replaced by physical limitations (diffraction, photon shot noise). So far, technology has been limiting the image quality.

†G. Agranov et al., 2007 International Image Sensor Workshop, pp 307-310, Ogunquit USA.

Rev 11 Rev 12

Rev 2 u2 Q = K ∫ M (u d) (logu) Sharpness u1

f/2.8 f/2.0 u1 = 3 cycles/deg f/2.8 f/2.0 1 1 Same pixel count – decreasing optical Resolution u2 = 12 cycles/deg 0.9 1.4 m 1.4 m 0.9 format with smaller pixel pitch K = normalizing constant 0.8 0.8 1.1 m 1.1 m Granger & Cupery , Photographic If the pixel count is 0.7 0.7 0.8 m Science and Engineering , Vol. 16, 0.8 m 0.65 m maintained, but the 0.6 0.6 1973, pp. 221-230 0.65 m SQF for 24x32 cm image @ 45 cm f/2.0 0.5 0.5 1 MTF sensor size is MTF viewing distance decreasing, because of a 0.4 0.4 0.9 1.4 m smaller pixel pitch, the 0.3 0.3 resolution will quickly 0.2 0.2 0.8 1.1 m become optics-limited for 0.1 0.1 0.7 0 0 0.8 m 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 a diffraction-limited, Normalized spatial frequency f/f Normalized spatial frequency f/f N N 0.65 m ideal, system. To keep 0.65 m 82% Very good 0.6 f/1.4 the same resolution and f/1.4 1 0.8 m 85% V.g. – Ex 0.5 sharpness, the f-number MTF 0.9 1.1 m 89% Excellent must be decreased. If it 0.4 0.8 1.4 m 91% Excellent is not possible to do this, 0.7 8 MP 1.4 m 0.3

the image quality of 0.6 1.1 m 0.2 cameras with the same 0.5 MTF

0.8 m pixel number will 0.4 0.1

decrease with new 0.3 0.65 m 0 sensor generations. 0.2 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

0.1 Normalized spatial frequency f/f N

0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Normalized spatial frequency f/f N

Rev 13 Rev 14

u2 Q = K ∫ M (u d) (logu) Photon shot noise and pixel size – same pixel Sharpness u1 u = 3 cycles/deg Same pixel count – decreasing optical 1 count, different sensor size u2 = 12 cycles/deg format with smaller pixel pitch K = normalizing constant Illuminance: 5000 lux Granger & Cupery , Photographic Science and Engineering , Vol. 16, 1973, pp. 221-230 SQF for 24x32 cm image @ 45 cm f/2.0 viewing distance 1

0.9 1.4 m

0.8 1.1 m

0.7 0.8 m 0.65 m 0.65 m 63% Fair – Good 0.6

0.8 m 69% Good 0.5 1.1 m 76% Good – V.g. MTF 0.4 1.4 m 80% Very good 2 MP 0.3

0.2 2.8 m 2.0 m 1.4 m 0.8 m 0.1 t = 1/70 s t = 1/70 s t = 1/50 s t = 1/35 s 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Gain: 0 dB Gain: 0 dB Gain: 0 dB Gain: 0 dB Normalized spatial frequency f/f N S/N: 90 S/N: 73 S/N: 69 S/N: 56

Rev 15 Rev 16

Photon shot noise and pixel size – same pixel count, different sensor size How to make sense of the megapixel race Illuminance: 100 lux

• The maximum useful pixel count is ultimately limited by optical diffraction and photon shot noise. • However, there is a fair chance to maintain a reasonable image quality between pixel generations if the optical format is kept (and disregarding other implications arising from smaller pixels). • But if the pixel count is kept constant while pixel size is decreased, image quality will suffer!

2.8 m 2.0 m 1.4 m 0.8 m t = 1/5 s t = 1/5 s t = 1/5 s t = 1/5 s Gain: 10 dB Gain: 10 dB Gain: 14 dB Gain: 18 dB S/N: 68 S/N: 55 S/N: 39 S/N: 24

Rev 17 Rev 18

Rev 3 The best compromise between physical size and image quality The best compromise between physical size and image quality

• How large sensor can we use? • The maximum sensor format that can be used ~1/2.8 inch A mobile phone must be slim

• The camera thickness depends on the sensor format • 1/2.8 = .357 inch = Magnum The sensor format dictates the focal length of the lens

Sensor Lens

Thickness ~ Sensor format

Rev 19 Rev 20

Digital zoom Digital zoom There are some claims that digital zoom will become more usable with an There are some claims that digital zoom will become more usable with an increasing pixel resolution. More pixels should automatically lead to a larger increasing pixel resolution. More pixels should automatically lead to a larger zoom range. zoom range. f/2.8 1.4 m 12MP f/2.8 1.4 m 12MP 1 1

0.9 0.9 1x 0.8 2x 0.8 0.7 0.7

SQF for 24x32 cm 0.6 SQF for 24x32 cm 0.6

image @ 45 cm u2 image @ 45 cm u2 0.5 0.5 viewing distance: MTF Q = K ∫ M (u d) (logu) viewing distance: MTF Q = K ∫ M (u d) (logu) 0.4 u1 0.4 u1 90% ~ Excellent 79% ~ Very Good 0.3 u1 = 3 cycles/deg 0.3 u1 = 3 cycles/deg

u2 = 12 cycles/deg u2 = 12 cycles/deg 0.2 0.2 K = normalizing constant K = normalizing constant 0.1 0.1

0 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Normalized spatial frequency f/f Normalized spatial frequency f/f N N

Rev 21 Rev 22

0.9 0.9 4x 0.8 8x 0.8 0.7 0.7

SQF for 24x32 cm 0.6 SQF for 24x32 cm 0.6

image @ 45 cm u2 image @ 45 cm u2 0.5 0.5 viewing distance: MTF Q = K ∫ M (u d) (logu) viewing distance: MTF Q = K ∫ M (u d) (logu) 0.4 u1 0.4 u1 57% ~ Fair 22% - Off scale - 0.3 u1 = 3 cycles/deg 0.3 u1 = 3 cycles/deg

u2 = 12 cycles/deg u2 = 12 cycles/deg 0.2 0.2 K = normalizing constant K = normalizing constant 0.1 0.1

0 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Normalized spatial frequency f/f Normalized spatial frequency f/f N N

Conclusion: Decent performance with perhaps 2x digital zoom for a good Conclusion: Decent performance with perhaps 2x digital zoom for a good system (not confirmed). system (not confirmed).

Rev 23 Rev 24

Rev 4 Field of view vs CRA Can image processing help?

In pursuit of smaller cameras, especially the height is important. A consequence of decreasing the height is a shorter focal length. Not only does this have a • Distortion correction bad effect on image quality (shading), but also the user experience is negatively affected, leading to perspective distortion (e.g., close distance • Noise reduction portraits). • Sharpening

F = 150 mm @ 2.4 m F = 36 mm @ 0.4 m

Rev 25 Rev 26

Noise reduction – problem solved?

Distortion correction While adaptive noise reduction algorithms Impossible to fix perspective distortion. in general do a good job at preserving Neglectable distortion Perspective distortion Optical distortion sharp edges, the effect on low contrast Straight lines Curved lines detail can be horrible.

Images copyright I3A – the International Imaging Industry Association

Rev 27 Rev 28

Digital sharpening Effect of digital sharpening

Example from existing camera • Introduces artefacts and amplifies noise 1.2 1.1 System MTF Digital sharpening can increase

• EDOF? Same problem. 1 perceived sharpness Diffraction MTF substantially, but at the expense 0.9 of noise amplification and 0.8 introduction of artefacts. 0.7 MTF

0.6

0.5

0.4

0.3 Optical MTF

0.2 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Spatial frequency f/f N

Rev 29 Rev 30

Rev 5 Digital processing CPIQ The impact on image quality due to the ISP is increasing. This leads to improvements but also artefacts that are Since the megapixel race seems out of control, and no-one (except marketing people...) really wants it, an industry-wide effort needs to be made in order to difficult to quantify. move the focus away from pixel count to image quality. The Camera Phone Image Quality (CPIQ) initiative was started by the I3A in 2006. Currently developing image quality measurement methods for: – Sharpness – Color shading – Distortion – Lateral chromatic aberration – Texture sharpness – Subjective image quality assessment

Two different cameras: same sensor, different optics and ISP.

Rev 31 Rev 32

Participating companies in Phase 2: AMD OmnivisionTechnologies, Inc. Hot Topics Aptina Palm, Inc. CDM Optics Philips Lumileds Lighting Co. CPIQ Eastman Kodak Co. Sensata Technologies Wafer Level Cameras Foveon, Inc. Sony Ericsson Mobile Communications Freescale Semiconductor, Inc. Sprint • Important for cost and size reduction FUJIFILM Corporation STMicroelectronics Hewlett-Packard Texas Instruments, Inc. Phase 2 finished April 2009. Image Engineering Tessera • Concerns about optical performance and yield Motorola, Inc. Verizon Wireless Phase 3 started autumn 2009. Open enrollment Nethra Imaging Vista Point Technologies Nokia Corporation Zoran Corporation is taking place at this moment. High Dynamic Range Phase 3 will continue to develop metrics for: • May be the next big step in image quality improvement – Spatial acutance – Color consistency and white balance • True HDR requires proper infra structure (not just sensor) – Luminance shading • Adaptive tone mapping for pleasant representation – Straylight • Challenging for small pixels? – Signal to noise ratio A method for combining all metrics into a single consumer-oriented rating system is the ultimate goal. HD Video • A new “pixelrace” – adding one dimension • High data rates put high demands on infra structure

Rev 33 Rev 34

Original scene radiance image Appendix Rendered into sRGB by calculating the CIEXYZ values from the spectral data at each sample followed by a chromatic adaptation to D65 and conversion to sRGB according to IEC 61966-2-1.

cont. →

Rev 35 Rev 36

Rev 6 The images were simulated under the following conditions: The scene was captured using a Nikon D700 camera with a 28 mm/2.8 Nikkor lens. The relative spectral sensitivity of this camera was previously measured. From the camera spectral data, a radiance scene was calculated such that each pixel contains the radiance spectrum from 400 up to 700 nm. Using only 3 color channels, the spectral accuracy is still reasonably good.

The sensor images were calculated from the same spectral QE in all cases. All noise sources except photon shot noise were set to zero. The raw images were processed by a very simple image pipe: white balance, color correction and gamma correction. The color matrix was calculated using a white-point preserving least squares method.

The sensor parameters that were varied were: - Pixel size - Full well capacity - Conversion gain - ADC voltage swing For the 0.8 um pixel, a full well capacity of 2000 electrons was assumed, together with a conversion gain of 160 uv/e- and an ADC swing of 400 mV. We are thus assuming that the full well capacity is considerably better than estimated from a simple scaling of full well vs pixel area.

The signal to noise ratio was calculated on the rendered images using the Photoshop histogram for the luminosity channel.

Rev 37

Rev 7