Video Spectroscopy meets Artificial Intelligence András Jung, Matthias Locherer, Barbara Darnell Cubert’s latest development ULTRIS S20. The hyperspectral video camera is based on light field technology. With a size of 6x6x6 cm and a weight of 350g the camera provides full frame image cubes for diverse fields of applications, ranging from machine vision up to mobile mapping of biophysical land surface parameters onboard UAVs. The false-color image shows different crops (winter wheat and rapeseed) taken with an ULTRIS form a drone in a height of 80m. With the growing availability of powerful computers, even in the smallest devices, we see an ever-increasing use of imaging techniques in industry and everyday products. Artificial intelligence and machine learning increase the capabilities of our image sensors and our smart devices in every sector rapidly. Nowadays these algorithms rely on the data of BW sensors or RGB sensors at best. Thus, they are missing a profound part of the information the incident light would have to offer. Spectral imaging adds another dimension to this two-dimensional geometric imaging. By definition, a spectral image is the result of capturing the same image in a variety of narrow- band spectral regions, usually in the visible light spectrum and the near-infrared region. This allows for identification of optical features of surfaces that are invisible to conventional cameras or the human eye. These features are directly attributable to the chemical proper- ties of an object. Spectral procedures thus enable important applications such as substance classification, object segmentation, color characterization, quality control, concentration analysis, security validations of products and many more. Furthermore, the typical training of machine learning algorithms based only on RGB images needs hundred thousand of images for the training of typical image recognition algorithms. This training could be sped up dramatically with the addition of spectral imaging data redu- cing the training time to a single image in some applications. Thus far, spectral image data with acceptable resolution have only been obtained by scanning methods (push broom sensors) or with bulky multi-sensor setups. These types of sensors are complex, expensive and difficult to incorporate into machine learning algorithms. cubert-gmbh.com | August 2020 VIDEO SPECTROSCOPY Cubert’s most recent development, the hyperspectral light field camera ULTRIS 20 offers the fastest hyperspectral camera technology worldwide with an ease-of-use of a common digital camera. The ULTRIS, a hyperspectral snapshot camera, provides full capability for real-time video applications. Applying powerful algorithms based on machine learning to the live data stream, allows the retrieval of relevant information within seconds. Video Spectroscopy meets Artificial Intelligence: Machine learning is used to identify the mixture of black and green tea completely automatically. Left: Colored Infrared image of tea samples with different mixing ratio. Right: Result of self-learning regression analysis based on the hyperspectral image information. The image is taken with the ULTRIS 20. Hyperspectral Imaging & Snapshot Systems Hyperspectral imaging, the combination of spectral and spatial information, has enabled advances in fields as diverse as precision farming, machine vision and biomedical imaging. A hyperspectral camera produces a 3D data cube composed of 2D spatial (x, y) and 1D (λ) spectral data. It is typically characterized by the spectral range, the number of spectral chan- nels and the spatial size of the entire image. The data cubes are acquired either by scanning techniques such as push-broom sensors or, in a term proposed by Hagen et al. (2012), snapshot technology for non-scanning spectral cameras, meaning the complete data cube is gathered with one sensor readout. Recently the term hyperspectral snapshot imaging has become an umbrella definition for all non-scanning imaging spectroscopy techniques; however, these different technologies, and their spectral data output, vary widely. cubert-gmbh.com | August 2020 VIDEO SPECTROSCOPY • Prism-based Sensors Using prism-based sensor technology, Cubert designed its first hyperspectral snapshot camera (FireflEYE 185) to offer a 50x50 pixel spatial resolution and 125 spectral chan- nels between 450-950nm. For the first time ever, a hyperspectral camera was able to use all incoming light, while not being reduced to one line (push-broom scanning) or one single wavelength pass (tuneable filters). With up to 70% of the incident light re- aching the sensor, the 185 had a great signal-to-noise ratio and achieved integration times of <1ms. To increase the spatial resolution of the image data, a second 1-MP sensor was integrated into the 185. By merging high resolution panchromatic data of this second sensor with the low-resolution data of the hyperspectral sensor, a data size of 1000x1000x125 was achieved. • Light Field Technology Recently, Cubert has introduced a fundamentally new snapshot technology that enables high spatial and spectral resolution on the same time. The ULTRIS 20 is the first hyper- spectral camera based on light field technology, where both the intensity and direction of incident light rays are used to produce spectral images. The light field approach creates a vast number of images, each equipped with its unique optical filter. The camera com- bines a continuously variable bandpass filter, a multi-lens array and a 20-MP CMOS sensor, to capture the entire hyperspectral data cube in a single sensor readout. The ULTRIS comes with a native resolution of 400x400 pixel, resulting in an impressive 160,000 pixels, each with 125 spectral bands covering 450-950nm, taken in a single snapshot. The 12-bit sensor allows an identification of small differences in spectral inten- sity with remarkably low noise. Data cubes can be readout at up to 8 times per second utilizing a dual gigabit ethernet interface. This corresponds to a value of remarkable 3200 frames per second of a theoretical push-broom sensor or sequential 2D camera. • Other Snapshot Technologies Filter-on-Chip Technology: The filter-on-chip technology provides high spatial re- solution with high image speed, while facing limited spectral resolution and reduced signal-to-noise ratios (S/N). The integrated filters typically are composed of several cells, each of which filters a given wavelength band, and are deposited directly onto CMOS wafers. Until 2019, Cubert used this technology in its ButterflEYE hyperspectral cameras. The development of the light field technology however finally replaced the filter-on-chip cameras, eliminating the drawbacks of limited spectral bands and re- duced signal-to-noise ratios, while keeping the high spatial resolution. Piezo-actuated Fabry-Perot Interferometer: In the UAV market, the Piezo-actuated Fabry-Perot Interferometer camera has gained attention to record 2D spatial images at selected wavelengths. According to Saari et al. (2009) this hyperspectral imager records simultaneously a 2D image at three narrow wavelengths based on the selec- ted three orders of the Fabry-Perot Interferometer depending on the air gap between the mirrors of the Fabry-Perot Cavity. This snapshot imaging technique provides high spatial resolution with selectable spectral bands. The ready-to-use data cube needs image co-registration preprocessing, because the single spectral bands of one image are captured with a small temporal difference. Depending on the speed and local mo- vements there is a spatial shift between the single bands of the HSI data cube, which must be corrected by using an image-matching algorithm (Jacob et al., 2017). cubert-gmbh.com | August 2020 VIDEO SPECTROSCOPY λ λ λ λ x x y x y y x y Cubert FireflEYE prism-based multipoint Cubert ULTRIS light field technology. technology. The data cube is readout The data cube is readout entirely, with entirely, the spatial resolution is low. high spatial and spectral resolution. λ λ λ λ x x x x y y y y Filter-on-chip technology. The data cube is Piezo-actuated Fabry-Perot technology. readout entirely, the spectral resolution is Sequential 2D-Imaging provides high limited and the signal-to-noise ratio is low. spatial and spectral resolution. The sin- gle bands are readout one after another, disabling true video applications. cubert-gmbh.com | August 2020 VIDEO SPECTROSCOPY True Video Spectroscopy While the data cubes from all snapshot and scanning cameras are similar, comparable and in many cases substitutable, the image production itself makes a big difference. One of the biggest differences is the light efficiency. A true snapshot imaging spectrometer like the multipoint or light field approach collects the entire 3D data cube (all spectral pixels) in a single integration period. The available incoming light is completely utilized and not reduced to one line (push-broom scanning) or one single wavelength pass (tunable filters) at one given time unit. The better light efficiency provides excellent signal qualities and higher signal-to-noise ratios in the shortest time. Time is the second advantage because a snapshot imaging spectrometer captures the entire image in a few milliseconds or less, which makes it a game-changer in real-time imaging and video spectroscopy. Cubert’s video imaging technology combines its high resolution and small size with high- speed spectral imaging, forming the basic requirements for true video spectroscopy. In fact, snapshots capture static moments, while real-life