I indirect imaging techniques, e.g., MRI (Fourier), CT (Backprojection)

I physical quantities other than intensities are measured I computation leads to 2-D map displayed as intensity

Image acquisition

Digital images are acquired by

I direct digital acquisition (digital still/video cameras), or I scanning material acquired as analog signals (slides, photographs, etc.).

I In both cases, the digital sensing element is one of the following:

Line array Area array Single sensor

Stanley J. Reeves ELEC 7450 - Digital Image Processing Image acquisition

Digital images are acquired by

I direct digital acquisition (digital still/video cameras), or I scanning material acquired as analog signals (slides, photographs, etc.).

I In both cases, the digital sensing element is one of the following:

Line array Area array Single sensor

I indirect imaging techniques, e.g., MRI (Fourier), CT (Backprojection)

I physical quantities other than intensities are measured I computation leads to 2-D map displayed as intensity

Stanley J. Reeves ELEC 7450 - Digital Image Processing Single sensor acquisition

Stanley J. Reeves ELEC 7450 - Digital Image Processing Linear array acquisition

Stanley J. Reeves ELEC 7450 - Digital Image Processing Two types of quantization:

I spatial: limited number of pixels

I gray-level: limited number of bits to represent intensity at a pixel

Array sensor acquisition

I Irradiance incident at each photo-site is integrated over time

I Resulting array of intensities is moved out of sensor array and into a buffer

I Quantized intensities are stored as a grayscale image

Stanley J. Reeves ELEC 7450 - Digital Image Processing Array sensor acquisition

I Irradiance incident at each photo-site is integrated over time

I Resulting array of intensities is moved out of sensor array and into a buffer

I Quantized Two types of quantization: intensities are stored as a I spatial: limited number of pixels grayscale image I gray-level: limited number of bits to represent intensity at a pixel

Stanley J. Reeves ELEC 7450 - Digital Image Processing Spatial resolution

Stanley J. Reeves ELEC 7450 - Digital Image Processing Grayscale resolution

Stanley J. Reeves ELEC 7450 - Digital Image Processing Sensors - CCD & CMOS

CMOS (complementary CCD (charge-coupled device) metal-oxide-semiconductor) Charge-coupled device. I QE of 19-26%. Whole systems Quantum efficiency of 70% I can be integrated on the same (film has 2% QE). device. Camera-on-chip. Mature technology. In I Standard semiconductor device development since 1969. I manufacturing process. Uses photo-diodes in I Each pixel has read-out conjunction with capacitors to I electronics, amplifiers, noise store charge. correction, and ADC. Charge converted to voltage at I Consume far less power than limited nodes. Varied I CCDs. architectures used for read-out. Need more room for electronics. Most of pixel area is light I I Fill-factor generally not as good sensitive. Good fill-factor. as CCDs.

Stanley J. Reeves ELEC 7450 - Digital Image Processing CCD architectures

CCDs function in two stages—exposure and read-out

I Photons are collected and charge is accumulated during exposure

I Area arrays use vertical and horizontal shift registers for read-out

I In some architectures, charge is transferred to an inactive/opaque region before readout

Linear array Full frame transfer

I Pixel intensities are read I The entire pixel area is active sequentially I Time between exposures is significant

I Needs mechanical shutter

Stanley J. Reeves ELEC 7450 - Digital Image Processing CCD architectures

Interline transfer Frame transfer I Charge shifted to adjacent I Need 2x optically active area opaque area and thus are larger and costlier I Subsequently shifted row-wise I Half of the array (for storage) is to a horizontal shift register masked I Complex design (requires I Shutter delay is smaller than full micro-mirrors or microlenses for frame transfer good optical efficiency)

Stanley J. Reeves ELEC 7450 - Digital Image Processing Image formation

I Both CCD and CMOS sensors are monochromatic

I Color images are acquired using color filters overlaid on the sensor

The intensity measured at a pixel is Z ∞ ci = fi(λ)g(λ)x(λ)l(λ)dλ + ηi −∞ I i = 1,..., k are distinct color channels sampled at each location

I fi(λ) - spectral transmittance of color filter I g(λ) - sensitivity of sensor I x(λ) - spectral reflectance of imaged surface I l(λ) - spectral power density of illuminant

I ηi - measurement noise

Stanley J. Reeves ELEC 7450 - Digital Image Processing Spectral response of common illuminants

Source: http://www.ni.com/white-paper/6901/en/

Stanley J. Reeves ELEC 7450 - Digital Image Processing Multiple sensors

I To acquire a 2-D image, multiple CCDs are used to acquire separate color bands

Dichroic prism

I A dichroic prism is used to split incoming irradiance into narrow-band beams

I Red, blue, and green beams directed to separate optical sensors

I Issues: cost, weight, registration Beam splitter in action

Stanley J. Reeves ELEC 7450 - Digital Image Processing Single sensor acquisition

I To avoid the cost and complexity associated with multiple-sensor acquisition, most color digital cameras use a single sensor

I Each pixel is overlaid with a color filter such that only one color channel is acquired at a particular pixel location

I The Bayer array is the most common color filter array

I Green is sampled at twice the density of red and blue since the human visual system (HVS) is more sensitive in the green region of the spectrum I The quincunx sampling arrangement ensures that aliasing in the green channel is least along the horizontal and vertical directions

I The full color image is recovered in a post-processing stage known as demosaicking

Stanley J. Reeves ELEC 7450 - Digital Image Processing Direct color imaging

I The Foveon X3 sensor captures colors at different depths at the same spatial location

I The increased density leads to much better spatial resolution

I The spectral sensitivity functions at the different layers have substantial overlap

I Color separation is a major issue for such sensors

Stanley J. Reeves ELEC 7450 - Digital Image Processing Digital camera pipeline

Lens assembly Focus control Exposure control

I IR blocking (hot I Active auto-focus I Good contrast mirror) systems use IR across image by

I Anti-aliasing: emitters to estimate manipulating blurs to increase distance aperture size and spatial I A passive method exposure time correlation dynamically adjusts I Prevents over- among color the focus setting to and channels to help maximimize under-exposed with high-frequency images demosaicking energy

Stanley J. Reeves ELEC 7450 - Digital Image Processing Digital camera pipeline

I Correct for lens distortion: barrel (fish-eye), pincushion (telephoto), vignetting (reduced brightness at edges)

I Gamma correction to compensate for nonlinearity of sensor response (opto-electronic conversion function)

I Compensation for dark current. Capture appropriate “dark-image”, subtract from acquired image.

I Lens flare (scattered light) compensation (mostly proprietary)

Stanley J. Reeves ELEC 7450 - Digital Image Processing Digital camera pipeline

I HVS remarkably adaptive; e.g., paper appears white under incandescent light or sunlight

I Imaging system will integrate spectral content of irradiance. Without color compensation, images appear unnatural and dissimilar to viewed scenes I White balancing algorithms based on one of two philosophies:

I Gray-world assumption R = krR, B = kbB; kr = Gmean/Rmean, kb = Gmean/Bmean I Perfect reflector method Brightest pixel corresponds to white. R = R/Rmax, G = G/Gmax, B = B/Bmax

Stanley J. Reeves ELEC 7450 - Digital Image Processing Digital camera pipeline

I Reconstruct sparsely sampled signal to form 3-color image

I Multitude of methods based on heuristics, properties of the HVS, and mathematical formulations

I Since the Bayer array is the most common, most algorithms are tailored Bayer demosaicking specifically for it

I Effective algorithms use inter-channel correlation

Stanley J. Reeves ELEC 7450 - Digital Image Processing Digital camera pipeline

I Captured image is in the digital camera color space. Colors are not impulses at specific wavelength. The sensitivity function of the camera color sensors dictates the camera color space.

I The camera-RGB image is transformed to one of many standard color spaces. Most commonly, the transformation is Camera-RGB → CIEXYZ.

I The CIEXYZ space defined by CIE (Commission Internationale de l’Eclairage the International Commission on Illumination) corresponds to the human visual subspace

I Many enhancement algorithms use non-RGB color spaces.

Stanley J. Reeves ELEC 7450 - Digital Image Processing Digital camera pipeline

I Removal of color artifacts due to demosaicking — algorithms based on the constant-hue assumption

I Sharpening — performed on luminance component only

I Denoising — median filters, bilateral filtering, and thresholding

Stanley J. Reeves ELEC 7450 - Digital Image Processing Digital camera pipeline

I Display — Images are converted to a format appropriate for display medium (sRGB for monitors, CMY/CMYK for printers).

I Compression — Most cameras offer flexible compression options. JPEG is standard in current models. Some JPEG2000.

I Storage — Low-end cameras offer only JPEG images as output. Some high-end point-and-shoot cameras and most SLRs will allow for retrieval of RAW images that are unprocessed. RAW images can be processed later on a PC without time and computational constraints.

Stanley J. Reeves ELEC 7450 - Digital Image Processing