Advanced Color Machine Vision and Applications Dr. Romik Chatterjee V.P. Business Development Graftek Imaging 2016 V04 - for tutorial 5/4/2016 Outline • What is color vision? Why is it useful? • Comparing human and machine color vision • Physics of color imaging • A model of color image formation • Human color vision and measurement • Color machine vision systems • Basic color machine vision algorithms • Advanced algorithms & applications • Human vision Machine Vision • Case study Important terms What is Color? • Our perception of wavelengths of light . Photometric (neural computed) measures . About 360 nm to 780 nm (indigo to deep red) . “…the rays are not coloured…” - Isaac Newton • Machine vision (MV) measures energy at different wavelengths (Radiometric) nm : nanometer = 1 billionth of a meter Color Image • Color image values, Pi, are a function of: . Illumination spectrum, E(λ) (lighting) λ = Wavelength . Object’s spectral reflectance, R(λ) (object “color”) . Sensor responses, Si(λ) (observer, viewer, camera) . Processing • Human <~> MV . Eye <~> Camera . Brain <~> Processor Color Vision Estimates Object Color • The formation color image values, Pi, has too many unknowns (E(λ) = lighting spectrum, etc.) to directly estimate object color. • Hard to “factor out” these unknowns to get an estimate of object color . In machine vision, we often don’t bother to! • Use knowledge, constraints, and computation to solve for object color estimates . Not perfect, but usually works… • If color vision is so hard, why use it? Material Property • Color is a material property – it helps tell what an object is made of, its state, etc. Texture is also a material property Wood and Brass? Bananas or What…? Which Cherries are Ripe? With Color it is Easy! Color Afterimage! Color Inspection and Sorting Which peanuts are bad? Is the printing good? Searching and Locating Which Sneakers are Light Blue? Importance of Fast Color Search • Quickly identify hazards! • Grab the food! Milk snake – Batesian (harmless) mimic of poisonous coral snake Color Measuring and Matching Color matching on car interior Medical diagnostics Color Coding Pseudocolor Bar Coding • Microsoft MobiTags™, Railroad color barcodes Computer Identics, 1970s Human Color Vision Stable perceptions of objects despite uncontrolled lighting and imaging geometry Easy to train, flexible, “understands” images Slow, quickly tires, individuals differ, hard to calibrate, drifts. Color (not intensity!) is low spatial resolution Influenced by surroundings in time and space • Optical illusions show problems and give us clues as to how biological vision works Influenced by Surroundings Dramatic example of color contrast effects “Side effect” of our color constancy abilities? Color Machine Vision (CMV) Replaces human vision on tasks that require fast, repeatable color vision. Never tires. Can calibrate to human color vision (sort of!) Can “see” colors we can’t (in IR, UV, etc.) Color Machine Vision Doesn’t “see” and understand the way you do! Requires exact set-up and instructions Difficult to implement when vision task: . Requires extensive task knowledge • What does skin cancer look like? . Has poorly controlled lighting or part presentation • Mobile robots, agricultural automation, railroad barcodes . Is poorly defined • “Find all the defects!” Some Markets for Color MV • Food production and processing • Pharmaceutical inspection • Parts identification • Inspecting or matching colored material • Medical diagnostics • Print and label inspection • Sorting recycled materials • Remote sensing, tracking • Biometrics, traffic monitoring • Measuring paints and pigments Physics of Color Imaging Light • Electromagnetic radiation . Quantized and transmitted as photons . Movement of electrons generates photons, absorption of photons moves electrons • Described by: . Wavelength or frequency . Energy and intensity . Spectrum • Energy per wavelength . Polarization . Geometry • Directions of light sources Wavelength or Frequency c = λf c: Velocity of light in vacuum or ~air (3 x 108 m/sec) λ: Wavelength (780 to 360 nm – red through indigo) f: Frequency (number of cycles / second) • Usually use wavelength, in nanometers (nm) • Frequency in terahertz Green light is about 500 nm 14 14 • ~ 4 x 10 to ~ 8 x 10 Hz or 600 THz Energy and Intensity • Photon energy as a function of wavelength Red, 650 nm => 1.9 electron volts (eV) 1 eV = 1.6 x 10-19 joules (J) • Energy transfer rate increases with: . Photon energy (decreasing wavelength) . Photon flux: photons per second (intensity) • Spectral power: E(λ) * flux . Watts per wavelength = J / sec / λ • Irradiance = Watts / square area (W/m2) • These are RADIOMETRIC measures Radiometric vs. Photometric • Radiometric = Physical measurement of light . Spectrometer • Photometric = Human perceptual response to light . Tristimulus colorimeter (or just colorimeter) . Photometer Spectrometer Colorimeter • Many different photometric measures Spectrum (Radiometric) • A spectrum plots power or irradiance per wavelength • A sensor is specified by spectral response Polarization • Orientation of oscillating electromagnetic wave • Some materials change color with polarization . Humans and Cameras ~ insensitive to polarization . Some animals (bees, fish, etc.) detect polarization Haidinger’s brush Polarizers can reduce “highlights” or glare off of some surfaces and so help color detection. Geometry • Position of lights relative to objects and camera • Can cause color appearance to change . Noticeably with thin films, pigmented materials Ratios of Color Sensor Types • Color vision requires two or more sensors types with different spectral responses Human Sensors’ Spectral Responses NORMALIZED! Why at least two sensor types? Radiometric Metamers • Different spectra give the same color response . Due to limited number (3) of sensors with broad, overlapped spectral responses . Can use to map human (photometric) color vision What if sensor response was narrow (in wavelength) and non-overlapping? A Model of Color Image Formation • Modeled as a function of: . Illumination spectrum . Object transmission or reflection properties • Spectrum and imaging geometry . Sensors’ spectral responses Spectrum at Sensors Illumination 26 87 451 Object “Color” samples A Radiometric Imaging Model • Output of a sensor type, Pi, is a product of: . Illumination spectrum: E(λ) . Object’s reflectance model: Rm(λ,g) . Sensor response spectrum: Si(λ,α) • Sensor Pi output is a function of wavelength (λ), lighting and viewing imaging geometry* (g), and sensors’ acceptance angle (α) *The term imaging geometry is not in general use Reflectance Modeling - 1 • Light reflection off an object or transmission through it is complicated. In general: . Depends on wavelength (λ) . Depends on 3D position of light(s) object surface, and camera (image geometry, g) • In general, can’t reconstruct R(λ) (reflectance) from Pi Lots of work on reflectance models, Rm(λ,g), for computer-generated imagery (CGI). CGI is a “forward” problem and so is easier than color vision’s “inverse” problem Reflectance Modeling - 2 • Function of spectral reflectance as a function of geometry, g : {e,v,b} Simple Diagram of Model Physics Samples Processing Sensor Si(λ,α) n Illumination e v Object Answers R (λ,g) m Computational Factors Object Color Estimation • Given 2 or more sensor output samples, Pi, estimate object’s reflectance spectrum, R(λ) Is the material or the light yellow??! Pi Sensor Readings E(λ) g : {e,v,b} Need to fix or recover, lighting, g, sensor response MV Object Color Estimation • In machine vision we can usually fix lighting, geometry, and sensor response so E(λ) and Si(λ,α) can be measured and known . Not if self-driving cars, robots, drones, etc. • Model reflectance as a multiplicative constant . Assume object is effectively flat and no changes in g • So object spectral reflectivity sampled by the i sensors is ~: In MV, We Usually Don’t! • It is difficult to measure E(λ) and Si(λ) and they drift with time and temperature. • We are usually interested in color changes and colors relative to standard objects’ colors, usually good and bad parts • Take output samples, Pi, as the samples of the object’s spectrum, and forget the math! • Use a white reference patch to compensate for illumination changes • Requires periodic (re)calibration, fixed g Human Object Color Estimation • Human vision recovers the illuminant to get E(λ) and knows Si(λ) • Has to compensate for changes in geometry, g (angle of view, etc.) . A difficult computational (processing) task . Uses knowledge about the world, e.g. light usually comes from above to constrain the problem . Veridical color perception under changing conditions is known as color constancy • MV algorithms for color constancy aren’t as robust Color Constancy • We recover the illumination to make colors appear about the same as lighting changes • Improves reliability of identifying objects and estimating material property by color Color Constancy Extreme Example Human vision uses constraints and computation to give stabile color perception despite illumination changes Illumination More details on the elements in color image Formation, starting with illumination Black Body Radiators • black body radiators . Heat “jiggles” electrons to create broadband radiation . The sun, flames, people, incandescent lights • Visibly glow at ~ 400 C (670 K) (750 F) • Color shifts towards the blue as object heats up: • Used as an illumination standard . Simple to described – function only of temperature Black