HYPERSPECTRAL AND MULTISPECTRAL DATA FUSION BASED ON NONLINEAR UNMIXING Naoto Yokoya,1 Jocelyn Chanussot,2 and Akira Iwasaki3 1Department of Aeronautics and Astronautics, The University of Tokyo, Japan 2GIPSA-lab, Signal and Image Dept., Grenoble Institute of Technology, Grenoble, France 3Research Center for Advanced Science and Technology, The University of Tokyo, Japan ABSTRACT method, namely the ’coupled nonnegative matrix factoriza- Data fusion of low spatial-resolution hyperspectral (HS) and tion’ (CNMF), was recently proposed to enhance spatial high spatial-resolution multispectral (MS) images based on resolution of all HS bands. It is based on a linear mixing a linear mixing model (LMM) enables the production of model (LMM) [2]. This algorithm extracts endmember spec- high spatial-resolution HS data with small spectral distor- tra and high spatial-resolution abundance maps from two tion. This paper extends the LMM based HS-MS data fusion images by alternate nonnegative matrix factorization (NMF) to nonlinear mixing model using a bilinear mixing model [3]. Since CNMF can increase the number of endmembers (BMM), which considers second scattering of photons be- compared with the MAP/SMM, it has a possibility to deal tween two distinct materials. A generalized bilinear model with spectrally more varied scenes. (GBM) is able to deal with the underlying assumptions in the Many researchers have worked on spectral unmixing with BMM. The GBM is applied to HS-MS data fusion to produce the LMM that assumes that an observed spectrum is a linear high-quality fused data regarding multiple scattering effect. combination of several endmember spectra. Although LMM Semi-nonnegative matrix factorization (Semi-NMF), which based unmixing methods can obtain physically meaningful can be easily incorporated with the existing LMM based fu- results, nonlinearity can appear in spectral mixing model [4], sion method, is introduced as a new optimization method for [5]. In recent years, nonlinear unmixing for HS images is the GBM unmixing. Comparing with the LMM based HS- receiving growing attention in remote sensing image inter- MS data fusion, the proposed method showed better results pretation. A bilinear mixing model (BMM), which considers on synthetic datasets. second scattering of photons between two distinct materials, has been studied by several groups [6], [7]. A generalized Index Terms— Data fusion, nonlinear unmixing, bilinear bilinear model (GBM) introduces an effective mean to deal mixing model, semi-nonnegative matrix factorization with the underlying assumptions in the BMM [7], [8]. In this work, the authors apply the GBM unmixing to HS-MS data 1. INTRODUCTION fusion to improve the quality of the fused data introducing semi-nonnegative matrix factorization (Semi-NMF) [9] as a Hyperspectral (HS) imaging sensors generally have larger new optimization for the GBM unmixing. ground sampling distance (GSD) than multispectral (MS) imaging sensors. Data fusion of low spatial-resolution HS and 2. GBM VIA SEMI-NMF high spatial-resolution MS images enables the production of high spatial-resolution HS data with small spectral distortion The BMM considers second-order interactions between dif- [1], [2]. The fused data is useful for accurate classification ferent D endmembers as additional terms in the LMM assum- with fine spatial resolution and thereby enhance applica- ing that third or higher order interactions are negligible. In the tions of HS remote sensing. Several HS-MS fusion methods BMM, the observed L-spectrum of a single pixel z 2 RL×1 is are proposed using pan-sharpening techniques, stochastic given by: method, and unmixing [1], [2]. A maximum a posteriori DX−1 XD (MAP) estimation method was developed to enhance the spa- z = Ea + b e ⊙ e + n; (1) tial resolution of HS data using higher spatial-resolution data i;j i j i=1 j=i+1 such as MS and panchromatic images [1]. This approach used a stochastic mixing model (SMM), which estimates the where E 2 RL×D is the endmember matrix with the i th col- L×1 underlying spectral scene characteristics, in order to develop umn vector, ei 2 R representing the i th endmember spec- D×1 a cost function that optimizes the estimated HS data relative trum, a 2 R is the abundance vector, bi;j is the interac- to the observed HS and MS data. An unmixing based fusion tion abundance between the i th and j th endmembers, ⊙ is the Hadamard (element-wise product) operation, and n 2 RL×1 The GBM unmixing via Semi-NMF is as follows. First, is the additive noise. On the right side, the first term denotes A is initialized by the fully constrained least square (FCLS) ∗ ∗ the linear mixing and the second term represents the bilinear method [10], A is calculated, and B is set as δ × A with mixing. From a physical perspective, the GBM introduces small value of δ. Next, A and B are alternately updated by (5) ∗ the nonlinear mixing coefficient ci;j as bi;j = ci;jaiaj and and (6). If any element of B exceeds that of A , it is replaced ∗ assumes the following constraints: by that of A . To satisfy the abundance sum-to-one constraint, the method from [10] is adopted. XD ai ≥ 0 8i 2 1; :::; D and ai = 1; i=1 (2) 3. HS-MS DATA FUSION BASED ON GBM 0 ≤ ci;j ≤ 1 8i 2 1; :::; D − 1 8j 2 i + 1; :::; D: The aim of HS and MS data fusion is to estimate unobserv- × When the endmembers are known, the GBM unmixing turns able high spatial-resolution HS data (Z 2 RL P ) from ob- × 2 RL Ph to the optimization of ai and ci;j under the constraint of (2). served low spatial-resolution HS data (X ) and high Lm×P Several optimization methods are proposed in [8]. spatial-resolution MS data (Y 2 R ). Lm denotes the The GBM method was applied to small images of syn- number of spectral channels of MS sensor. Ph denotes the thetic and real HS data with three endmembers [7], [8]. number of pixels of HS images. Owing to the trade-off be- When applied to larger images in an unsupervised manner tween spectral and spatial resolutions of two sensors, L > Lm with many endmembers, the optimization process becomes and Ph < P are satisfied. The observed two data are assumed more challenging. The new optimization method based on to be obtained under the same atmospheric and illumination semi-nonnegative matrix factorization (Semi-NMF) [9] is conditions, and geometrically co-registered with radiometric introduced to speed up the optimization process of a whole correction. The HS and MS images can be considered as the image in a matrix form, which can be easily incorporated degraded versions from the high spatial-resolution HS image with CNMF. The observed HS image can be reshaped as a in spatial and spectral domains, respectively. Therefore, X matrix form Z 2 RL×P with P representing the number of and Y are modeled as pixels. The BMM for the whole image is given in a matrix form by X = ZS + Ns; (7) Z = EA + MB + N; (3) Y = RZ + Nr: (8) 2 RD×P 2 RL×D(D−1)=2 where A is the abundance matrix, M × − × 2 RP Ph is the bilinear endmember matrix, B 2 RD(D 1)=2 P is the Here, S is the spatial spread transform matrix × f gPh 2 RP ×1 interaction abundance matrix, and N 2 RL P is the noise with each column vector sl l=1 representing the transform of the point spread function (PSF) from the MS matrix. The GBM unmixing becomes theP minimization of k − − k2 ⪰ D image to the HS l thP pixel value. Each PSF is assumed to Z EA MB F , subject to A 0 , i=1 Ai;k = 1 L × ∗ ∗ be normalized, i.e., m s = 1. R 2 RLm L is the (8k 2 1; :::; P ), 0 ⪯ B ⪯ A , where A = Ai;kAj;k. By k=1 kl (i;j);k spectral response transform matrix with each row vector introducing Z1 = Z − MB and Z2 = Z − EA, (3) is written f gLm 2 R1×L as follows ri i=1 representing the transform of the spectral response function (SRF) from the HS sensor to the MS i th Z1 = EA + N and Z2 = MB + N: (4) band detector. Ns and Nr are the residuals. Owing to physical constraints, all components of E, M, A, k k2 3.1. The CNMF algorithm and B are nonnegative. Therefore, minimization of N F in (4) can be solved by Semi-NMF that factorizes a non- First, we summarize the CNMF method that was proposed restricted matrix X into a non-restricted matrix F and a non- for HS-MS data fusion based on the LMM [2]. The CNMF negative matrix GT as X ≈ FGT [9]. Semi-NMF optimiza- method is composed of alternate NMF unmixing for HS and tion is guaranteed to converge to a local optimum with the MS images to extract high spectral-resolution endmember alternative update rules. With E given and M calculated from spectra and high spatial-resolution abundance maps. In the E, the GBM unmixing is solved by the following update rules LMM, the spectrum at each pixel is assumed to be a linear for A and B: combination of several endmember spectra, which is the sim- p ple version without the bilinear term in (3). Therefore, Z is AT AT :∗ ((ZT E)++AT (ET E)−):=((ZT E)−+AT (ET E)+) 1 1 (5) formulated as Z ≈ EA. The spatially degraded abundance D×P p matrix Ah 2 R h and the spectrally degraded endmem- T T :∗ (( T )++ T ( T )−):=(( T )−+ T ( T )+) B B Z2 M B M M Z2 M B M M (6) Lm×D ber matrix Em 2 R are defined as Ah = AS and where :∗ and := denote elementwise multiplication and divi- Em = RE, respectively.