Noise, Image Reconstruction with Noise EE367/CS448I: Computational Imaging and Display stanford.edu/class/ee367 Lecture 10 Gordon Wetzstein Stanford University What’s a Pixel? photon to electron converter à photoelectric effect! source: Molecular Expressions wikipedia What’s a Pixel? • microlens: focus light on photodiode • color filter: select color channel • quantum efficiency: ~50% • fill factor: fraction of surface area used for light gathering source: Molecular Expressions • photon-to-charge conversion and ADC includes noise! ISO (“film speed”) sensitivity of sensor to light – digital gain bobatkins.com Noise • noise is (usually) bad! • many sources of noise: heat, electronics, amplifier gain, photon to electron conversion, pixel defects, read, … • let’s start with something simple: fixed pattern noise Fixed Pattern Noise • dead or “hot” pixels, dust, … • remove with dark frame calibration: I − I I = captured dark Iwhite − Idark on RAW image! not JPEG (nonlinear) Emil Martinec Noise • other than that, different noise follows different statistical distributions, these two are crucial: • Gaussian • Poisson Gaussian Noise • thermal, read, amplifier • additive, signal-independent, zero-mean + = Gaussian Noise x ∼ N (x,0) • with i.i.d. Gaussian noise: b = x +η η N 0,σ 2 (independent and identically-distributed) ∼ ( ) additive Gaussian noise Gaussian Noise x ∼ N (x,0) • with i.i.d. Gaussian noise: b = x +η η ∼ N 0,σ 2 (independent and identically-distributed) ( ) b ∼ N (x,σ 2 ) 2 (b−x) 1 − 2 p(b | x,σ ) = e 2σ 2πσ 2 Gaussian Noise x ∼ N (x,0) • with i.i.d. Gaussian noise: b = x +η η N 0,σ 2 (independent and identically-distributed) ∼ ( ) • Bayes’ rule: b ∼ N (x,σ 2 ) 2 (b−x) 1 − 2 p(b | x,σ ) = e 2σ 2πσ 2 Gaussian Noise - MAP x ∼ N (x,0) • with i.i.d. Gaussian noise: b = x +η η N 0,σ 2 (independent and identically-distributed) ∼ ( ) • Bayes’ rule: • maximum-a-posteriori estimation: b ∼ N (x,σ 2 ) 2 (b−x) 1 − 2 p(b | x,σ ) = e 2σ 2πσ 2 Gaussian Noise – MAP “Flat” Prior • trivial solution (maximum likelihood, not useful in practice): p(x) = 1 Gaussian Noise – MAP Self Similarity Prior • Gaussian “denoisers” like non-local means and other self- similarity priors actually solve this problem: General Self Similarity Prior • generic proximal operator for function f(x): • proximal operator for some image prior General Self Similarity Prior • can use self-similarity as general image prior (not just for denoising) • proximal operator for some image prior General Self Similarity Prior • can use self-similarity as general image prior (not just for denoising) σ 2 = λ /σ σ = λ /σ (h parameter in most NLM implementations is std. dev.) • proximal operator for some image prior Image Reconstruction with Gaussian Noise x ∼ N (Ax,0) • with i.i.d. Gaussian noise: b = Ax +η η ∼ N 0,σ 2 (independent and identically-distributed) ( ) Image Reconstruction with Gaussian Noise x ∼ N (Ax,0) • with i.i.d. Gaussian noise: b = Ax +η η ∼ N 0,σ 2 (independent and identically-distributed) ( ) b ∼ N (Ax,σ 2 ) 2 (b−Ax) 1 − 2 p(b | x,σ ) = e 2σ 2πσ 2 Image Reconstruction with Gaussian Noise x ∼ N (Ax,0) • with i.i.d. Gaussian noise: b = Ax +η η N 0,σ 2 (independent and identically-distributed) ∼ ( ) • Bayes’ rule: • maximum-a-posteriori estimation: b ∼ N (Ax,σ 2 ) 2 (b−Ax) 1 − 2 p b | x,σ = e 2σ ( ) 2 2πσ • regularized least squares (use ADMM) Scientific Sensors • e.g., Andor iXon Ultra 897: cooled to -100° C • scientific CMOS & CCD • reduce pretty much all noise, except for photon noise What is Photon (or Shot) Noise? • noise caused by the fluctuation when the incident light is converted to charge (detection) • also observed in the light emitted by a source, i.e. due to particle nature of light (emission) • same for re-emission à cascading Poisson processes are also described by a Poisson process [Teich and Saleh 1998] Photon Noise • noise is signal dependent! • for N measured photo-electrons • standard deviation is σ = N , variance is σ 2 = N • mean is N N ke− N • Poisson distribution f (k; N) = k! Photon Noise - SNR signal-to-noise ratio N SNR = N SNR in dB N photons: σ = N 2N photons: σ = 2 N 1,000 10,000 N nonlinear! Photon Noise - SNR signal-to-noise ratio N SNR = N N photons: σ = N 2N photons: σ = 2 N nonlinear! wikipedia Maximum Likelihood Solution for Poisson Noise • image formation: • probability of measurement i: • joint probability of all M measurements (use notation trick ): Maximum Likelihood Solution for Poisson Noise • log-likelihood function: • gradient: Maximum Likelihood Solution for Poisson Noise • Richardson-Lucy algorithm: iterative approach to ML estimation for Poisson noise • simple idea: 1. at solution, gradient will be zero 2. when converged, further iterations will not change, i.e. Richardson-Lucy Algorithm • equate gradient to zero = 0 • rearrange so that 1 is on one side of equation Richardson-Lucy Algorithm • equate gradient to zero = 0 • rearrange so that 1 is on one side of equation • set equal to Richardson-Lucy Algorithm • for any multiplicative update rule scheme: • start with positive initial guess (e.g. random values) • apply iteration scheme • future updates are guaranteed to remain positive • always get smaller residual • RL multiplicative update rules: Richardson-Lucy Algorithm - Deconvolution Blurry & Noisy Measurement RL Deconvolution Richardson-Lucy Algorithm • what went wrong? • Poisson deconvolution is a tough problem, without priors it’s pretty much hopeless • let’s try to incorporate the one prior we have learned: total variation Richardson-Lucy Algorithm + TV • log-likelihood function: • gradient: Richardson-Lucy Algorithm + TV • gradient of anisotropic TV term • this is “dirty”: possible division by 0! Richardson-Lucy Algorithm + TV • follow the same logic as RL, RL+TV multiplicative update rules: • 2 major problems & solution “hacks”: 1. still possible division by 0 when gradient is zero à set fraction to 0 if gradient is 0 2. multiplicative update may become negative! à only work with (very) small values for lambda (A Dirty but Easy Approach to) Richardson-Lucy with a TV Prior RL RL+TV, lambda=0.005 RL+TV, lambda=0.025 Log Residual Mean Squared Error Measurements Signal-to-Noise Ratio (SNR) mean pixel value µ signal SNR = = standard deviation of pixel value σ noise PQ t = e 2 PQet + Dt + Nr P = incident photon flux (photons/pixel/sec) Qe = quantum efficiency t = eposure time (sec) D = dark current (electroncs/pixel/sec), including hot pixels Nr = read noise (rms electrons/pixel), including fixed pattern noise Signal-to-Noise Ratio (SNR) mean pixel value µ signal SNR = = standard deviation of pixel value σ noise PQ t = e signal-dependent 2 PQet + Dt + Nr signal-independent P = incident photon flux (photons/pixel/sec) Qe = quantum efficiency t = eposure time (sec) D = dark current (electroncs/pixel/sec), including hot pixels Nr = read noise (rms electrons/pixel), including fixed pattern noise Next: Compressive Imaging • single pixel camera • compressive hyperspectral imaging • compressive light field imaging et al., 2013 Marwah Wakin et al. 2006 References and Further Reading • https://en.wikipedia.org/wiki/Shot_noise • L. Lucy, 1974 “An iterative technique for the rectification of observed distributions”. Astron. J. 79, 745–754. • W. Richardson, 1972 “Bayesian-based iterative method of image restoration” J. Opt. Soc. Am. 62, 1, 55–59. • N. Dey, L. Blanc-Feraud, C. Zimmer, Z. Kam, J. Olivo-Marin, J. Zerubia, 2004 “A deconvolution method for confocal microscopy with total variation regularization”, In IEEE Symposium on Biomedical Imaging: Nano to Macro, 1223–1226 • M. Teich, E. Saleh, 1998, “Cascaded stochastic processes in optics”, Traitement du Signal 15(6) • please also read the lecture notes, especially for the “clean” ADMM derivation for solving the maximum likelihood estimation of Poisson reconstruction with TV prior!.