Noise, Image Reconstruction with

EE367/CS448I: Computational Imaging and Display stanford.edu/class/ee367 Lecture 10

Gordon Wetzstein Stanford University What’s a Pixel?

to converter à photoelectric effect!

source: Molecular Expressions

wikipedia What’s a Pixel?

• microlens: focus light on photodiode • color filter: select color channel • : ~50% • fill factor: fraction of surface area used for light gathering source: Molecular Expressions • photon-to-charge conversion and ADC includes noise! ISO (“film speed”) sensitivity of to light – digital gain

bobatkins.com Noise

• noise is (usually) bad!

• many sources of noise: heat, , amplifier gain, photon to electron conversion, pixel defects, read, …

• let’s start with something simple: fixed pattern noise Fixed Pattern Noise

• dead or “hot” pixels, dust, …

• remove with dark frame calibration: I − I I = captured dark Iwhite − Idark on RAW image! not JPEG (nonlinear)

Emil Martinec Noise

• other than that, different noise follows different statistical distributions, these two are crucial: • Gaussian • Poisson

• thermal, read, amplifier

• additive, signal-independent, zero-mean

+ = Gaussian Noise x N x,0 ∼ ( ) • with i.i.d. Gaussian noise: b = x +η η ∼ N 0,σ 2 (independent and identically-distributed) ( ) additive Gaussian noise Gaussian Noise x N x,0 ∼ ( ) • with i.i.d. Gaussian noise: b = x +η η ∼ N 0,σ 2 (independent and identically-distributed) ( )

b ∼ N x,σ 2 ( )

2 (b−x) 1 − 2 p(b | x,σ ) = e 2σ 2πσ 2 Gaussian Noise x N x,0 ∼ ( ) • with i.i.d. Gaussian noise: b = x +η η ∼ N 0,σ 2 (independent and identically-distributed) ( )

• Bayes’ rule:

b ∼ N x,σ 2 ( )

2 (b−x) 1 − 2 p(b | x,σ ) = e 2σ 2πσ 2 Gaussian Noise - MAP x N x,0 ∼ ( ) • with i.i.d. Gaussian noise: b = x +η η ∼ N 0,σ 2 (independent and identically-distributed) ( )

• Bayes’ rule: • maximum-a-posteriori estimation: b ∼ N x,σ 2 ( )

2 (b−x) 1 − 2 p(b | x,σ ) = e 2σ 2πσ 2 Gaussian Noise – MAP “Flat” Prior

• trivial solution (maximum likelihood, not useful in practice):

p(x) = 1 Gaussian Noise – MAP Self Similarity Prior

• Gaussian “denoisers” like non-local means and other self- similarity priors actually solve this problem: General Self Similarity Prior

• generic proximal operator for function f(x):

• proximal operator for some image prior General Self Similarity Prior

• can use self-similarity as general image prior (not just for denoising)

• proximal operator for some image prior General Self Similarity Prior

• can use self-similarity as general image prior (not just for denoising)

σ 2 = λ /σ σ = λ /σ (h parameter in most NLM implementations is std. dev.) • proximal operator for some image prior Image Reconstruction with Gaussian Noise x N Ax,0 ∼ ( ) • with i.i.d. Gaussian noise: b = Ax +η η ∼ N 0,σ 2 (independent and identically-distributed) ( ) Image Reconstruction with Gaussian Noise x N Ax,0 ∼ ( ) • with i.i.d. Gaussian noise: b = Ax +η η ∼ N 0,σ 2 (independent and identically-distributed) ( )

b ∼ N Ax,σ 2 ( )

2 (b−Ax) 1 − 2 p(b | x,σ ) = e 2σ 2πσ 2 Image Reconstruction with Gaussian Noise x N Ax,0 ∼ ( ) • with i.i.d. Gaussian noise: b = Ax +η η ∼ N 0,σ 2 (independent and identically-distributed) ( )

• Bayes’ rule: • maximum-a-posteriori estimation: b ∼ N Ax,σ 2 ( )

2 (b−Ax) 1 − 2 p b | x,σ = e 2σ ( ) 2 2πσ • regularized least squares (use ADMM) Scientific

• e.g., Andor iXon Ultra 897: cooled to -100° C • scientific CMOS & CCD • reduce pretty much all noise, except for photon noise What is Photon (or Shot) Noise?

• noise caused by the fluctuation when the incident light is converted to charge (detection)

• also observed in the light emitted by a source, i.e. due to particle nature of light (emission)

• same for re-emission à cascading Poisson processes are also described by a Poisson process [Teich and Saleh 1998] Photon Noise

• noise is signal dependent!

• for N measured photo- • is σ = N , variance is σ 2 = N • mean is N N ke− N • f (k; N) = k! Photon Noise - SNR signal-to-noise ratio N SNR = N

SNR in dB

N : σ = N

2N photons: σ = 2 N 1,000 10,000 N nonlinear! Photon Noise - SNR signal-to-noise ratio N SNR = N

N photons: σ = N

2N photons: σ = 2 N nonlinear! wikipedia Maximum Likelihood Solution for Poisson Noise

• image formation:

• probability of measurement i:

• joint probability of all M measurements (use notation trick ): Maximum Likelihood Solution for Poisson Noise

• log-likelihood function:

• gradient: Maximum Likelihood Solution for Poisson Noise

• Richardson-Lucy algorithm: iterative approach to ML estimation for Poisson noise

• simple idea: 1. at solution, gradient will be zero 2. when converged, further iterations will not change, i.e. Richardson-Lucy Algorithm

• equate gradient to zero = 0

• rearrange so that 1 is on one side of equation Richardson-Lucy Algorithm

• equate gradient to zero = 0

• rearrange so that 1 is on one side of equation

• set equal to Richardson-Lucy Algorithm

• for any multiplicative update rule scheme: • start with positive initial guess (e.g. random values) • apply iteration scheme • future updates are guaranteed to remain positive • always get smaller residual

• RL multiplicative update rules: Richardson-Lucy Algorithm - Deconvolution

Blurry & Noisy Measurement RL Deconvolution Richardson-Lucy Algorithm

• what went wrong?

• Poisson deconvolution is a tough problem, without priors it’s pretty much hopeless

• let’s try to incorporate the one prior we have learned: total variation Richardson-Lucy Algorithm + TV

• log-likelihood function:

• gradient: Richardson-Lucy Algorithm + TV

• gradient of anisotropic TV term

• this is “dirty”: possible division by 0! Richardson-Lucy Algorithm + TV

• follow the same logic as RL, RL+TV multiplicative update rules:

• 2 major problems & solution “hacks”: 1. still possible division by 0 when gradient is zero à set fraction to 0 if gradient is 0

2. multiplicative update may become negative! à only work with (very) small values for lambda (A Dirty but Easy Approach to) Richardson-Lucy with a TV Prior RL RL+TV, lambda=0.005 RL+TV, lambda=0.025

Log Residual Mean Squared Error Measurements Signal-to-Noise Ratio (SNR)

mean pixel value µ signal SNR = = standard deviation of pixel value σ noise PQ t = e 2 PQet + Dt + Nr

P = incident photon flux (photons/pixel/sec)

Qe = quantum efficiency t = eposure time (sec) D = (electroncs/pixel/sec), including hot pixels

Nr = read noise (rms electrons/pixel), including fixed pattern noise Signal-to-Noise Ratio (SNR)

mean pixel value µ signal SNR = = standard deviation of pixel value σ noise PQ t = e signal-dependent 2 PQet + Dt + Nr signal-independent

P = incident photon flux (photons/pixel/sec)

Qe = quantum efficiency t = eposure time (sec) D = dark current (electroncs/pixel/sec), including hot pixels

Nr = read noise (rms electrons/pixel), including fixed pattern noise Next: Compressive Imaging

• single pixel camera • compressive hyperspectral imaging • compressive light field imaging et al., 2013 Marwah

Wakin et al. 2006 References and Further Reading

• https://en.wikipedia.org/wiki/Shot_noise • L. Lucy, 1974 “An iterative technique for the rectification of observed distributions”. Astron. J. 79, 745–754. • W. Richardson, 1972 “Bayesian-based iterative method of image restoration” J. Opt. Soc. Am. 62, 1, 55–59. • N. Dey, L. Blanc-Feraud, C. Zimmer, Z. Kam, J. Olivo-Marin, J. Zerubia, 2004 “A deconvolution method for confocal microscopy with total variation regularization”, In IEEE Symposium on Biomedical Imaging: Nano to Macro, 1223–1226 • M. Teich, E. Saleh, 1998, “Cascaded stochastic processes in ”, Traitement du Signal 15(6)

• please also read the lecture notes, especially for the “clean” ADMM derivation for solving the maximum likelihood estimation of Poisson reconstruction with TV prior!