Hyperspectral Imaging - a Technology Update

A White Paper by Dr Nick Barnett, Pro-Lite Technology Ltd May 2021 classification, object segmentation and improved colour characterisation.

Imaging spectroscopy is often divided into two categories: multispectral and hyperspectral. Multispectral cameras measure light in a small number (typically 3 to 15) of spectral bands whereas hyperspectral cameras collect a much larger number (up to several hundred) of distinct, yet contiguous bands across a wide spectral range.

Traditionally, hyperspectral imaging has found Introduction applications in remote sensing and agriculture New developments in imager technology and in using scanning or push-broom cameras installed on computer processing power are rapidly enhancing satellites, aircraft, or UAVs. In recent times, consumer digital camera performance, leading to a developments in spectral imaging technology have wider adoption of imaging in everyday use. been evolving at a pace. Advances in high Additionally, image processing techniques based resolution sensors, electronics and optics are on artificial intelligence and machine learning are providing enhanced push-broom technologies as increasing the capabilities of cameras and smart well as enabling numerous other forms of spectral devices for tasks such as object detection based on imaging. Alternatives to push-broom systems colour and two-dimensional geometric data. include methods using tunable spectral filters as However, these conventional RGB-type cameras well as designs based on Fourier Transform only make use of a limited range of wavelengths in spectroscopy. There are also snapshot spectral the visible part of the spectrum, so are missing out imaging systems employing mosaic arrays of filters on a lot of the available spectral information. and light field camera designs offering video-rate imaging spectroscopy. These technologies offer Spectral imaging captures the same geometric specific strengths that can sometimes be more image but in multiple narrow spectral bands that appropriate than the traditional push-broom can cover the broader visible, near infrared and cameras and have the potential to facilitate many shortwave infrared spectral ranges. This allows for new and exciting applications. identification of optical features of objects that are invisible to conventional cameras or the human The following paper provides a glimpse of some eye. Spectral features are directly attributable to new developments within hyperspectral imaging the chemical properties of an object, such that technology. It is not intended to be a spectral imaging enables tasks such as object comprehensive review but highlights what is a detection and identification, substance rapidly changing imaging landscape.

www.pro-lite.co.uk White Paper: Hyperspectral Imaging – a Technology Update Page 1

Simply the Best: Traditional Push- Broom Cameras for the Highest Quality Data High quality scientific hyperspectral imaging data is required when looking for subtle differences in spectra. Cameras based on push-broom architectures typically provide the best quality data but even within this category there are varying levels of performance. Cameras made by HySpex (Norsk Elektro Optikk, NEO) are a good example of high quality devices that provide optimal spectral fidelity and sharp imaging optics. The care and attention taken to minimise optical distortions such as spatial and spectral misregistration and stray light are fundamental to final image quality Figure 1: The HySpex Baldur S-384 N SWIR hyperspectral and the ability to identify spectral features. camera (930-2500nm) is used to detect asbestos and separate it from other materials in mixed building waste. Five different The latest HySpex SWIR cameras use state-of-the- sample sets containing five different classes of materials are art MCT sensors with cooling down to 150K using a shown: asbestos (green); concrete (blue); terracotta (light blue); ceramics (purple) and unknown mix (red) Sterling cooler providing low background noise, high dynamic range, and exceptional signal to Internal Push-Broom Systems: noise. HySpex’s most recent SWIR cameras also provide exceptional spatial and spectral resolution. from the Laboratory to the Field These systems tend to cost more than some Typically, for laboratory or field measurements, alternatives, but the new Baldur range of industrial push-broom imaging applications require that the cameras from NEO (Figure 1) are more affordable sample object moves within the view of the while guaranteeing a similar level of high-quality camera, either on a conveyor belt or by using an data. Additionally, the combination of the Baldur external translation stage or rotating tripod head. cameras with Breeze run-time software (from However, new cameras have been developed with Prediktera AB) enables classification and the scanning mechanism built into the camera quantification of objects in real-time which is ideal body (the mechanical system moves the for process sorting and segregation applications. spectrometer entrance slit internally). With these cameras there is no need for an external translation stage or moving conveyor belt, instead the camera remains static. This can be useful when space is limited, and it also allows the camera to be easily attached to a microscope for hyperspectral microscopy applications. Examples of cameras using this approach include the VNIR and SWIR SnapScan cameras from Imec (Figure 2) and the VNIR and SWIR SOC-710 cameras from Surface Optics Corporation.

www.pro-lite.co.uk White Paper: Hyperspectral Imaging – a Technology Update Page 2

to achieve with some of the older tunable filter technologies.

Figure 2: The Imec SnapScan SWIR camera used to detect and classify different plastic materials. Tunable Filter Cameras for High Spatial Resolution and Microscopy Applications Hyperspectral cameras using liquid crystal or acousto-optical tunable filters (AOTFs) have been available for many years. In this approach the spectral bands are transitioned sequentially to create a resulting spectral hypercube. More Figure 3: Schematic of Fabry-Perot Filter and the Hinalea recently, tunable filter cameras based on piezo- model 4250 camera. actuated Fabry-Perot interferometry have been Fabry-Perot interferometer cameras also provide developed (for example Hinalea Imaging). great spectral flexibility as they can be used in a A Fabry-Perot interferometer consists of two hyperspectral mode, acquiring hundreds of parallel reflective mirror surfaces with a gap wavebands, or in a multi-spectral mode where the between the mirrors. The passband wavelength of data acquisition is limited to just a few wavebands. the filter is tuned by adjusting the distance In this multi-spectral mode of operation, it has between the mirrors (Figure 3). The filter is placed recently been demonstrated that these cameras in front of an imaging sensor and spectral bands can work at video rates with the application of real- are selected sequentially to generate a hyper-cube. time image classification. The acquisition time depends on the number of Tunable filter cameras are also ideal for spectral bands being acquired and the time hyperspectral microscopy as the cameras can be required to collect light at each band. These easily mounted with no additional scanning imagers are especially attractive for applications mechanism required. Canadian firm, Photon etc, requiring high spatial resolution as they utilise all has developed a high-performance hyperspectral available pixels on the camera sensor. The imaging platform based on tunable volume Bragg technology can be easily integrated into small gratings and global imaging (Figure 4). These camera designs which has previously been difficult essentially provide tunable bandpass optical

www.pro-lite.co.uk White Paper: Hyperspectral Imaging – a Technology Update Page 3

filtering with high efficiency, high out-of-band replicas that share a common optical path. The rejection, and high spectral resolution. The entire resulting interference pattern is measured by a optical field of view is uniformly illuminated, and detector as a function of their relative delay and monochromatic images acquired sequentially. the Fourier Transform of the interferogram yields the intensity spectrum of the light (Figure 5).

Figure 4: The IMA hyperspectral microscope from Photon etc used for photoluminescence imaging. a) PL image extracted at 770 nm, b) false-colour map of the PL central wavelength and c) two PL spectra extracted from the hyperspectral data – see corresponding colours. Images from Prof. David Cooke (McGill University) and Prof Mercouri Kanatzidis (Northwestern Figure 5: The Hera camera from NIREOS used for imaging University). chlorophyll fluorescence in leaves.

These cameras offer high spectral resolution This newly developed camera works in a staring (<2nm in the visible) and high spatial resolution (in mode without the need for a translation stage or the order of 1μm) and have been used for scanning. It has potential advantages for photoluminescence, electroluminescence, hyperspectral imaging in low level light conditions fluorescence and reflectance in material science so is well suited to fluorescence imaging although and life science applications. it has also been shown to produce high quality spectral data for remote sensing and in art and Fourier Transform Hyperspectral cultural heritage applications.

Imaging: Great for Low Light It is relatively straightforward to extend this Conditions technology to longer wavelength range Fourier Transform (FT) hyperspectral imagers applications including in the SWIR which is combine a monochrome imaging camera with a currently in development. Fourier Transform spectrometer. The FT approach results in high optical throughput, due to the Snapshot Hyperspectral: absence of slits and gratings. Signal to noise is good Hyperspectral Imaging at Video as all the wavelengths are measured simultaneously, thus maximising the number of Rates photons reaching the sensor. There are exciting new developments in the field of video rate spectroscopy with advances in the The Hera hyperspectral imager from NIREOS uses a implementation of “light field” technology. The common-path birefringent interferometer design Ultris X20 camera from Cubert GmbH (Figure 6) where the light is split into two co-linear delayed

www.pro-lite.co.uk White Paper: Hyperspectral Imaging – a Technology Update Page 4

incorporates a continuously variable bandpass livestream video. This real-time imaging combined filter, microlens array and an Ultra-HD CMOS with video spectroscopy offers a real game changer sensor with 20 Megapixel resolution. In the light for real-life situations where there are dynamic field design, both the intensity and direction of processes and movement within a scene. There is incident light rays are used to produce spectral also great potential to drive down costs in the images. The result is an image of 400 × 400 pixels, future which should lead to more widespread use. each with 130 spectral bands. That is an impressive Both Surface Optics and Cubert have recently 160,000 pixels, each with 130 spectral bands developed these light field cameras for the visible, covering 350-1000nm. The entire spectral dataset near infrared and SWIR spectral ranges. is obtained during a single detector integration period, so truly snapshot. This spectral data can be Snapshot Mosaic Cameras for captured in real-time at multiple frames per second thus providing a unique video Targeted Applications: Easier Data hyperspectral imaging system. Processing and Cost-Effective in High Volume Mosaic filter-based cameras provide another approach to achieving snapshot spectral imaging although being of multi-spectral rather than hyperspectral type.

Figure 6: The Ultris X20 snapshot camera from Cubert GmbH. Figure 7: The SnapShot SWIR camera from IMEC using a mosaic pattern of 16 SWIR filters. A similar light field camera for multi-spectral imaging has been developed by Surface Optics The team at nanotechnology institute Imec Corp. Their LightShift camera also uses a microlens continue to expand their wafer-level CMOS array and high-resolution imaging sensor but processes to integrate thin-film spectral filters incorporates an array of 16 (4x4) bandpass filters directly onto image sensors. Cameras operating in at the entrance aperture rather than using a the visible and near infrared wavelength range continuously variable bandpass filter. The filter have been available for a few years with mosaic array can contain polarizing as well as bandpass patterns of 3X3, 4X4 or 5X5 filters. In more recent filters and is interchangeable so the camera can be times imec have also launched a SWIR mosaic re-tasked for different applications. cameras (Figure 7). These designs also enable video-rate spectral imaging with real-time The volume of data generated by these light field classification. The spatial resolution of the snapshot systems is a key challenge but both multispectral image will be less than that of the systems can run with real-time classification native image sensor (divided by the number of applied to highlight materials of interest in a spectral bands) but there is some evidence that the

www.pro-lite.co.uk White Paper: Hyperspectral Imaging – a Technology Update Page 5

spatial resolution can be recovered using image integrated into new spectral imaging cameras and processing techniques. The manufacturing process to be disruptive in terms of cost and form factor. used by Imec is easily scalable so there is great promise that this mosaic filter approach can lead to Providing Actionable Data high-volume and low-cost sensors. For the past decade hyperspectral imaging has been an area of extensive research and What Comes Next? development. Application of hyperspectral imaging has involved high costs and complex, time- Extending into the SWIR consuming processing to produce meaningful data. Well established manufacturers of push-broom This has somewhat limited its use to R&D hyperspectral cameras usually have a portfolio of environments in academic research institutes. systems covering the visible, near-infrared and Bridging the gap from being an interesting research shortwave infrared ranges. For the newer spectral tool to providing actionable data for routine imaging technologies most of the first analysis requires implementation of image embodiments are initially realised in the visible and processing algorithms that can automate tasks near infrared spectral ranges using silicon sensors. such as detection, classification, and separation of Real incentives exist to expand the imaging materials. Fortunately, great strides have been wavelength range into the short-wave infrared, but made in this area with the development of InGaAs sensors have historically been expensive machine learning and AI-powered hyperspectral with fewer and larger pixels than the equivalent imaging analytics. Sophisticated analytical silicon sensors. Recent innovations in SWIR sensor software packages from the likes of perClass BV technology are potentially about to change this. and Prediktera AS (Figure 8) can now be used by a Manufacturers such as Sony, Emberion and Imec, range of imaging cameras to implement tasks such are bringing new SWIR sensors to market and as object recognition and classification. fuelling new opportunities for the next generation of spectral cameras.

Sony recently released sensors based on their SenSWIR technology, in which photodiodes are formed on an indium gallium arsenide semiconductor layer and are connected via Cu-Cu connections with a silicon layer which forms the readout circuit. This yields a SWIR image sensor that is sensitive over a broad range of wavelengths covering the visible to SWIR range (from 400nm to Figure 8: Breeze software from Prediktera AS can be used to 1.7μm). Emberion has also developed new imaging classify and quantify objects in real-time. sensors sensitive over a similar VIS to SWIR wavelength range. Their design combines The combination of new camera hardware and nanocrystalline optical absorbers with graphene- smart image processing is significantly increasing based transistors fabricated directly on standard ease of use and general accessibility to a wider CMOS wafers. Similarly, Imec recently presented a community. Some of the new technical approaches new image sensor that uses a thin film of PbS also have the potential to reduce the camera size, quantum dots to capture light in the NIR and SWIR. form factor and cost which should enable this These novel sensors have the potential to be once-complex technology to be further adopted

www.pro-lite.co.uk White Paper: Hyperspectral Imaging – a Technology Update Page 6

into everyday life. So, it seems inevitable that spectral imaging will become more a part of mainstream imaging and will play an increasing role in important topical issues of the day including recycling plastics, precision farming, food analysis and environmental monitoring.

Further Reading An in-depth review of the different technologies used in multi- and hyperspectral imaging systems is provided in this Pro-Lite Technical Note: https://bit.ly/3fHkdaI

The physics of “light field” hyperspectral imaging are explained in this Pro-Lite Technical Note: https://bit.ly/3yDw9CT

The Pro-Lite range of multispectral and hyperspectral imagers is presented here: https://bit.ly/34fmG6V

About Pro-Lite Pro-Lite is a supplier of specialist equipment and services with a technical focus in the following areas of photonics: instruments for measuring light and the optical properties of materials; photometry; lasers and laser safety equipment; opto-mechanics and nano-positioning equipment; optics and optical materials; and spectroscopy and spectral imaging. Pro-Lite Technology Ltd is part of the Pro-Lite Group of Companies, which includes Photometric & Optical Testing Services, SphereOptics Germany, Pro-Lite France and Pro- Lite Iberia.

www.pro-lite.co.uk White Paper: Hyperspectral Imaging – a Technology Update Page 7