(Lidar) and Hyperspectral Imagery (HSI) Data

Integrated analysis of Light Detection and Ranging (LiDAR) and Hyperspectral Imagery (HSI) data

Angela M. Kim, Fred A. Kruse, and Richard C. Olsen

Naval Postgraduate School, Remote Sensing Center and Physics Department, 833 Dyer Road, Monterey, CA, USA;

ABSTRACT LiDAR and hyperspectral data provide rich and complementary information about the content of a scene. In this work, we examine methods of data fusion, with the goal of minimizing information loss due to point-cloud rasterization and spatial-spectral resampling. Two approaches are investigated and compared: 1) a point-cloud approach in which spectral indices such as Normalized Difference Vegetation Index (NDVI) and principal components of the hyperspectral image are calculated and appended as attributes to each LiDAR point falling within the spatial extent of the pixel, and a supervised machine learning approach is used to classify the resulting fused point cloud; and 2) a raster-based approach in which LiDAR raster products (DEMs, DSMs, slope, height, aspect, etc) are created and appended to the hyperspectral image cube, and traditional spectral classification techniques are then used to classify the fused image cube. The methods are compared in terms of classification accuracy. LiDAR data and associated orthophotos of the NPS campus collected during 2012 – 2014 and hyperspectral Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) data collected during 2011 are used for this work. Keywords: LiDAR, Hyperspectral, HSI, Raster Fusion, Point Cloud Fusion

1. INTRODUCTION Identification and mapping of materials for applications such as target detection, trafficability, land use, en- vironmental monitoring, and many others requires accurate characterization of both surface composition and morphology. Typical remote sensing approaches to these problems, however, usually concentrate on one specific data type (modality) and thus don’t provide a complete description of the materials of interest. The research described here combines hyperspectral imaging (HSI) and Light Detection and Ranging (LiDAR) in an attempt to enhance remote sensing mapping capabilities. Imaging spectrometers or “Hyperspectral Imaging (HSI)” sensors are passive systems that rely on reflected sunlight to simultaneously collect spectral data in up to hundreds of image bands with a complete spectrum at each pixel.1 This allows identification of materials based on their specific spectral signatures and production of maps showing the surface distribution of these materials. The solar spectral range, 0.4 to 2.5 micrometers, provides abundant information about many important Earth-surface materials such as minerals, man-made objects, and vegetation.2, 3 HSI data provide useful spectral information for detecting targets and/or mapping surface ground cover; however, this is based solely on the surface spectral signature and does not incorporate surface morphology. LiDAR is an active remote sensing method in which pulses of laser energy are emitted, and the time-of-flight is recorded for echoes returning to the sensor. The time-of-flight information, along with the precise location and pointing direction of the sensor in space, is used to determine the exact location from which laser pulses have been reflected. The transmission of many laser pulses enables building up a point cloud dataset representing the 3D arrangement of objects in the scene. Most current LiDAR systems operate at a single wavelength, and the intensity of the reflected pulses is recorded for each of the returned points. This intensity value is typically not calibrated, however, and so provides only a relative measure of reflectivity. Further author information: A.M.K.: E-mail: [email protected], Telephone: 1 401 647 3536

Proc. of SPIE Vol. 9832 98320W-1

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 05/27/2017 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx The strengths of hyperspectral imaging and LiDAR are complementary, and the successful fusion of the two data types has huge potential, but multiple challenges to successfully fusing HSI and LiDAR exist. HSI sensors produce 2D raster-based images that have relatively low spatial resolution as compared to the LiDAR data. LiDAR sensors produce 3D, irregularly gridded point clouds. There isn’t a direct one-to-one correspondence between the two datasets. A successful fusion approach will maintain the spectral detail of the HSI data, and also the spatially detailed 3D information of the LiDAR point cloud. In the work presented in this paper, two fusion approaches are examined. In the first approach to fusion, data are fused into an integrated 3D point cloud, and processed in the 3D environment. To incorporate the 2D HSI information into the 3D LiDAR point cloud, relevant information from the HSI data are extracted and attributed to each LiDAR point as a vector. This is a one to many relationship, in which one HSI raster attribute is mapped to many irregularly spaced LiDAR points. Additional attributes from the LiDAR point cloud based on statistics of local neighborhoods of points are used to quantify the spatial arrangement of LiDAR points. A supervised machine learning decision tree classifier is then used to classify the point cloud attribute vectors. Working with irregularly gridded point cloud data is more challenging than traditional raster-based approaches, but this approach minimizes information loss due to rasterization of the point cloud data. The second fusion approach rasterizes relevant information from the LiDAR point cloud data (including height, intensity, and the statistical measures of local neighborhoods of points) and fuses these with the spectral data by combining them into a raster datacube. This is a many to one relationship, in which many LiDAR attributes are averaged and mapped to one raster pixel corresponding to the spatial resolution of the HSI data. The fused raster datacube is classifed using standard supervised classification approaches. The raster approach is more computationally feasible than the point cloud approach. Individual LiDAR and HSI analyses along with fusion results are presented in Section 5. An evaluation of the success of the fusion processes is accomplished by examining the classification accuracy the fused data products. A discussion of the relative success of working within the point-cloud domain as compared to the raster-domain is given in Section 6. 2. BACKGROUND Multiple studies have demonstrated the utility of combining hyperspectral imaging and LiDAR data, particularly in a raster-based fashion. An overview of previous work, as well as results of an IEEE GRSS data fusion competition for classification of an urban scene, is given in Debes et al., 2013.4 Previous work fusing LiDAR and spectral information in the 3D domain is not as common, but a series of papers by researchers at the Finnish Geodetic Institute introduce prototype terrestrial laser scanning systems for collecting hyperspectral LiDAR point clouds. In these prototype instruments, a supercontinuum “white laser” source is combined with a hyperspectral time-of-flight sensor to actively measure reflectance over wavelengths ranging from 480 – 2200 nm. Results show improved classification of various tree species using the fused data product over either the HSI or the LiDAR data individually. The authors also demonstrate extraction of spectral indices in 3D geometries.5–7 Buckley 2013 concurrently collected terrestrial laser scanner data (Riegl LMS-Z420i) and terrestrial hyperspectral camera (HySpex SWIR-320 m) data with the goal of integrated geological outcrop mapping.8 Products from the hyperspectral camera data are projected into 3D space for improved visual analysis.

3. STUDY AREA AND DATA The area chosen for this study covers a small portion of the city of Monterey, California, which includes the campus of the Naval Postgraduate School (NPS). This area was selected because of the diversity of materials and morphologies at the site and the availability of both hyperspectral and LiDAR data. Figure 1 shows the study area and a small subarea selected for ground validation.

Proc. of SPIE Vol. 9832 98320W-2

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 05/27/2017 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx 2 To facilitate 10 Proc. of SPIE Vol. 9832 98320W-3 Proc. of SPIE Vol. The AVIRIS data used for this investigation were acquired 30 September, 9 4. APPROACH AND METHODS

''`.~ Figure 1: (Left) AVIRISshown true in color greater composite spatial imagethe detail of small at the subarea study used right. area. for (Right) ground The An validation red aerial and box accuracy photograph outlines assessment. with the 0.15 small m subarea spatial resolution showing The height and “intensity” information associated with the LiDAR points was also used in the point cloud 3.2 Hyperspectral Imagery (HSI)HSI data data for thisImaging study Spectrometer area (AVIRIS), were an collectedover airborne the at hyperspectral range 2.4 sensor 0.4 m – with spatial 2.5 224 micrometers. resolution bands using at NASA’s 10 Airborne nm Visible/Infrared spectral resolution across the study area.a A point cloud scan dataset angle with field excellent of canopy view penetration of and 30 some degrees, coverage as of well the as sides multiple of overlapping vertical flightlines, objects. gives processing the large dataset,2 the m data to were avoid firstLAStools the tiled “lasheight” introduction into of utility, 200 tile whichelevation x edge first model 200 artifacts. uses m (DEM), The square the andcase, derivation tiles, points then we of with made calculates classified height an use the as values additional of was height ground buffer the accomplished of of to ground using each classification determine the point as aclassification above delivered process. bare by the earth In the derived addition digital vendor. based DEM to on surface. the the inherent statistical In LiDAR attributes distribution this of of height points and were intensity, additional calculated. attributes 4.1 Point-cloud fusion approach LiDAR data were processed to determine height above ground level using the LAStools utilities. 3.1 Light Detection andLiDAR Ranging data for (LiDAR) this data studyflown onboard were collected a Bell in 206L Octobera helicopter and spot from November size approximately of of 450 2012 approximately m with 50 above an ground cm Optech level, on Orion with C-200 the a system ground. 1541 nm The laser point having density is on average approximately 60 points/m 2011. Ainfrared total (SWIR) of bands from 802 2.0 micrometers visible – were and 2.5 not micrometers used. near were infrared used in (VNIR) these bands analyses. from The 0.4Two SWIR approaches data – are between 1.2 presented 1.2a in micrometers – description this and of study. the The 51 raster-based point shortwave processing cloud approach. processing approach Results is of discussed each first, approach are followed by presented in Section 5. Downloaded From: http://proceedings.spiedigitallibrary.org/ on 05/27/2017 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx Downloaded From: http://proceedings.spiedigitallibrary.org/ on 05/27/2017 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx a acltduigterd(3. m n erifae 815n)seta ad codn otefollowing the to according bands (NDVI) spectral Index nm) Vegetation (831.5 Difference infrared Normalized components near and Principal the nm) Additionally, were dataset. (638.2 components red principal point. raster mapped 7 the LiDAR hyperspectral first was using each resolution the calculated point and to was spatial LiDAR image, attributes m each hyperspectral AVIRIS as 2.4 cloud, 224-band added the original point the in the on pixel into calculated were image data nearest hyperspectral the the cloud to from point LiDAR information and incorporate To data hyperspectral of Fusion 4.1.2 analysis statistics. fusion LiDAR raster-based This the the tiles. of in multiple however, some across approach; of significantly analysis exclusion varied fusion the tile cloud required each point inconsistencies within the values these in of of approach, points. problems range use LiDAR cause the This of as not neighborhoods area did data. of study the values entire intensity the in and across variance z the y, 2012. explain x, as Lague, of to such and analysis features required include Brodu globular be For to by will expanded presented statistic. shows was components was this are PC1” principal of components analysis - values principal three components two Explained high all principal features, Variance have planar cars, will “% For surfaces of or edges. ground image trees described building and the as be roofs (bottom-left), such can building features 2 edges required; The explained linear Figure be building for will In values or variance neighborhood. the (bright) fences, component. of each high most lines, principal points; therefore for power the single and of as recorded coordinate, a component structure such by are principal the features single describing plane a linear by for this completely describe useful almost that is from component points points principal example, each of for by deviation explained variance and the of orientation of percentage 2. analysis the Figure components in are and given principal Statistics plane, include are a Statistics statistics on cloud. the based point neighborhood. of features LiDAR each each and the in of intensities, points in Examples and the point points. heights of LiDAR each of distribution around deviation 3D defined standard the is the describe neighborhood quantitatively cylindrical to calculated radius m 1 statistics A neighborhood point-cloud LiDAR 4.1.1 % Variance Explained - PC1 Surface Normal Angle Intensity viewed when inconsistencies some to led This basis. per-tile a on calculated were statistics cloud point LiDAR “best-fit” a define to used are points of neighborhoods of components z y, x, the of components Principal iue2 ttsiso oa egbrod fLDRpoints. LiDAR of neighborhoods local of Statistics 2: Figure % Variance Explained - PC2 Deviation from best -fit plane ij 4e01011410.Intensity si v Deviation i5 X = i Proc. of SPIE Vol. 983298320W-4 _ r' ,. rt. tte I J , % Variance Explained - PC3 11 % Variance Explained - PC3 (PCA w/ xyz +intensity) -"Jim ' Height Deviation analysis components principal The ,,g /Ìy 't y. . formula:12, 13 NIR − RED NDVI = NIR + RED 4.1.3 Classification of the fused point cloud data Ground truth data were created by manually classifying point cloud data from a small subset of the study area (Figure 1). Seven (7) classes were defined: road, sidewalk, grass, trees, buildings, cars and “other” (Figure 3, top-left). The “other” class includes isolated objects (such as picnic tables and light fixtures) and unknown points. The 3D classification process enabled the creation of complex training classes; e.g. trees contain canopy and trunks, but exclude the ground underneath. The manually classified points were then divided into training, validation, and test sets. The training set (comprised of a random 60% selection of the points) was used to train a supervised machine learning decision tree classifier, while the validation set (20% of the points) was used to prevent overfitting of the classifier. The remaining 20% of the data points were used as a test dataset. 4.2 Raster fusion approach The HSI data for the study site were analyzed using a standardized approach based on n-dimensional spectral attributes and spectral signatures.14 HSI results were stacked with raster layers extracted from the LiDAR data, combined signatures were determined using n-dimensional scatterplots, and supervised classification was used to map the distribution of specific surface characteristics. 4.2.1 Hyperspectral data analysis AVIRIS VNIR and SWIR HSI radiance data were converted to reflectance utilizing the Fast Line-of-sight Atmo- spheric Analysis of Spectral Hypercubes (FLAASH) atmospheric model.15 This is a MODTRAN-based model requiring only knowledge about the HSI acquisition parameters (sensor, date, time, altitude), site characteristics (latitude and longitude, elevation), and some basic assumptions (atmospheric model, visibility). FLAASH adjusts the wavelength calibration as required, calculates the atmospheric water vapor concentration from the radiance spectra, determines reflectance utilizing the measured water vapor and the atmospheric model, masks portions of the spectrum with low signal-to-noise ratios (SNR), and removes residual sensor artifacts – all without any supporting ground or atmospheric measurements. Image-based endmembers and spectral unmixing were then used for independent analysis of the individual spectral ranges. The selected VNIR and SWIR spectral regions were separately linearly transformed using the Minimum Noise Fraction (MNF) transform,14, 16 which conditions n-dimensional data in preparation for determination of endmembers and spectral unmixing. A subset of the MNF data were used to determine the spectrally extreme pixels in each dataset using repeated projections of n-dimensional scatterplots (where “n” is the number of spectral bands) onto a lower dimensional subspace and marking the extreme pixels calculated – the Pixel Purity Index (PPI).14 The PPI results were then analyzed using interactive n-dimensional scatterplots to determine vertices, which typically correspond to unique materials – the “endmembers”. The endmember spectra were visually compared to spectral libraries to determine key spectral features and identify materials. These were then used in a partial unmixing algorithm, Mixture-Tuned-Matched Filtering (MTMF) to map the spatial occurrence and abundance of each endmember for each dataset.14 It should be noted that this spectral attribute analysis approach is significantly different than the principal component analysis approach for HSI data used in the point cloud fusion. The direct analysis of spectral signatures and endmembers results in many diverse classes that may not necessarily correspond to traditional classification of specific objects such as sidewalks, roads, rooftops, etc. This may in part explain some of the differences between the raster versus point cloud results and the large disparities in accuracy compared to the hand-picked validation classes (Section 5). 4.2.2 Raster layer stacking The raster datacube combines spectral information from the HSI data with textural and 3D information extracted from the LiDAR data. Table 1 lists an index value for each of the bands used in the analysis. Spectral endmember abundance images of the AVIRIS data as well as the NDVI raster provide 18 raster images defining surface material characterization. LiDAR height above ground, intensity, and statistical measures (as described in Section 4.1.1) were rasterized to the 2.4 m spatial resolution and identical spatial coverage of the AVIRIS imagery by averaging all of the LiDAR points occurring in each pixel, yielding 10 raster images defining contextual

Proc. of SPIE Vol. 9832 98320W-5

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 05/27/2017 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx and surface texture information. A subset of 22 raster bands (out of the total of 28) were used in the fused HSI/LiDAR raster-based classification process. Five of the LiDAR raster products were not used in the final fused analyses due to inconsistencies introduced during the tiling process (see Section 4.1). The final PCA (PCA3a) LiDAR raster product was also excluded.

Table 1: Raster layers derived from the AVIRIS HSI and LiDAR neighborhood statistics. The index value indicates the arrangement of the layers as ordered in the fused raster layer stack. Layers without an index value were not used in the ﬁnal raster layer analysis.

Index Source Layer

1 AVIRIS VNIR Classification Mean_nD_Classl: Green Artificial Turf`1, Green Tennis Courts (Red)

2 Mean_nD_Class4: Water, Dark Vegetation, Shadows (Blue)

3 Mean_nD_Class5: Asphalt Roads and Asphalt Roofs (Cyan)

4 Mean_nD_Class6: Selected Green Vegetation (Green)

5 Mean_nD_Classl 1: Brown Bare Soil, Asphalt Roads (Yellow2)

6 Mean_nD_Classl7: Bright White Synthetic Roof Material (Red2)

7 Mean_nD_Classl9: Dark Green Artificial Turf#2 (Green2)

8 Mean_nD_Class20: Artificial Turf #3, asphalt roads (Sienna)

9 Mean_nD_Class21: Red Tile Roofs, Red Soil, Red Synthetics (Red3)

10 AVIRIS NDVI Normalized Difference Vegetation Index

1 l AVIRIS SWIR Classification Mean_nD_Classl: Dolomite, Sand Traps, Roads, 2.32 µm (Red)

12 Mean_nD_Class2: Synth Material, Tennis Courts, Rooftops, 2.27µm (Green)

13 Mean_nD_Class3: 2.15, 2.30 pm, Synth Roofs #1 (Blue)

14 Mean_nD_Class6: 2.28, 2.31 pm Synth Roofs #2 (Magenta)

15 Mean_nD_Class8: flat, 2.21 µm, Soils, Sand, Clay (Sea Green)

16 Mean_nD_Class10: 2.06, 2.31, 2.35 p.m, Synth Artificial Turf (Coral)

17 Mean_nD_Classl 1: peak at 2.2 µm, Green (Cyan)

18 Mean_nD_Classl3: 2.01, 2.32 pm, peak at 2.2 µm Dry Veg, Soil (Sienna)

19 LIDAR Rasterized Mean Heights

20 Rasterized Standard Deviation of Heights

21 Rasterized Mean Intensity

Rasterized Standard Deviation of Intensity (not used due to tiling artifacts)

22 Rasterized Angle (angular difference from vertical of surface normal of best -fit plane)

Rasterized Standard Deviation from Best Fit Plane (not used due to tiling artifacts)

Rasterized % Variance explained PCAI (not used due to tiling artifacts)

Rasterized % Variance explained PCA2 (not used due to tiling artifacts)

Rasterized % Variance explained PCA3 (not used due to tiling artifacts)

Rasterized % Variance explained PCA3a (PCA xyz + intensity, not used)

5. RESULTS Results are presented for the point-cloud processing approach ﬁrst, followed by results for the raster-based approach.

Proc. of SPIE Vol. 9832 98320W-6

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 05/27/2017 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx 5.1 Point cloud analysis results Attributes of LiDAR points (including height and intensity) were combined with attributes based on statistics of local neighborhoods of points, and features from the HSI data (including 7 principal components and NDVI). A supervised machine learning decision tree classifier was trained using the training and validation data, and the point cloud data were classified into 7 classes including road, sidewalk, grass, trees, buildings, cars, and an “other” containing small isolated objects and unknown points. To quantify the performance of the point cloud data fusion approach, the trained classifier was applied to the test data set (20% of manually classified points) with the LiDAR data alone, the HSI information alone, and then again using the fused data product. Figure 3 shows the true classes and each of the classification results using the test dataset. A measure of overall accuracy was calculated as the number of correctly classified points divided by the total number of ground truth points. The decision tree classified trained on the fused HSI/LIDAR point cloud yields an overall accuracy of 88.8%. This represents a 3.5% increase in overall accuracy over using the LiDAR data alone, and an increase of over 20% using the HSI data alone. The performance of the HSI data alone appears to be very poor, but this can be explained in part due to the large spatial resolution of the HSI data (2.4 m pixels) as compared to the spatial resolution of the ground truth classes (60 points/m2); this is discussed further in Section 5.3.

Point Cloud Decision Tree Classification Results for Test Dataset (20% of Manually Classified Points)

Road Grass Buildings Other/ Sidewalk Trees Cars unknown

True Classes LiDAR only - 85.3% overall accuracy

HSI only - 64.2% overall accuracy Fused LIDAR & HSI - 88.8% overall accuracy

Figure 3: Results of applying the decision tree classiﬁers to the test dataset, and the corresponding overall accuracy measurements. Fusing the HSI and LiDAR data resulted in a 3.5% improvement in overall accuracy over using LiDAR data alone.

The confusion matrix for the fused HSI and LiDAR point-cloud classification (Table 2) shows classification accuracy on a class-by-class basis. The result of applying the decision tree classifier to a larger portion of the study area is shown in Figure 4.

Proc. of SPIE Vol. 9832 98320W-7

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 05/27/2017 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx Table 2: The confusion matrix for the fused HSI and LiDAR decision tree classiﬁer as applied to the small test set; the overall accuracy (calculated as the number of correctly classiﬁed points divided by the total number of ground truth points) is 88.8%.

Predicted Class Number of Ground SidewalkGrass Road Cars Trees Building Other Truth Points Sidewalk 1581 9813 2387 0 0 0 0 13781 % of Ground Truth 11% 71% 17% 0% 0% 0% 0% 443 71660 7795 0 0 0 0 Grass 79898 1% 90% 10% 0% 0% 0% 0% 171 12070 78303 0 0 0 0 Road 90544 Ñ 0% 13% 86% 0% 0% 0% 0%

0 1 74 10230 178 83 342 ud Cars 10908 0% 0% 1% 94% 2% 1% 3% 2 0 26 171 79 97798 73 108 ~ Trees 98255 0% 0% 0% 0% 100% 0% 0% 0 4 4 79 338 17255 15 Building 17695 0% 0% 0% 0% 2% 98% 0% 0 18 144 184 404 26 1856 Other 2632 0% 1% 5% 7% 15% 1% 71% Number of 2195 93592 88878 10572 98718 17437 2321 313713

,ir.i+''` _-

Figure 4: Results of applying the decision tree classiﬁer to a larger subset of the fused HSI and LiDAR point cloud study area.

5.2 Raster analysis results The AVIRIS data were split into two spectral subsets: VNIR (0.4 – 1.2 micrometers) and SWIR (2.0 – 2.5 micrometers). Results are presented for each of the HSI subsets first, followed by results for the fused HSI/LiDAR datacube. Unfortunately, the HSI data’s large 2.4 m pixel results in the occurrence of multiple materials per pixel in both the VNIR and SWIR. Only the predominant endmember is shown in the materials maps in Figures 5 and 6. Additionally, the spectral diversity of the study area and the fact that there is significant spectral variability within classes makes it extremely difficult to directly relate the spectral classes to traditional ground truth or to those derived using the higher resolution LiDAR data. The HSI data stand alone to determine surface composition, but we later had to consolidate and simplify classes to produce comparable results.

Proc. of SPIE Vol. 9832 98320W-8

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 05/27/2017 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx Selected VNIR n -D Visualizer ROI Means resulting 14 Class 5.lasslass 11 6 and subsequent 3 x 3 sieving and clumping produced the image 3 Proc. of SPIE Vol. 9832 98320W-9 Proc. of SPIE Vol. 0.4 Wavelength (Micrometers) 0.6 0.8 1.0 1.2

Figure 5: (Left) AVIRIS VNIR(Bottom endmember Right) spectra MTMF extracted classification usingto result Table the showing 1 n-dimensional distribution for scatterplotting endmember of descriptions. approach. of selected results, The VNIR with white spectral box the endmembers. shows corresponding the Refer (Top location Center) of MTMF the classification small and subarea (Top used Right) for true-color validation AVIRIS image. 5.2.2 AVIRIS SWIR Forty-eight (48) reflectance corrected spectraleters bands were of used AVIRIS for dataas covering the described the analysis. SWIR in from These Section 2.0 were 4.2. – reduced 2.5 N-dimensional to microm- 35 scatterplotting linearly was combined used bands to using select the endmembers MNF with transform unique SWIR in a total of 9MF/Infeasibility unique ratio VNIR endmembers thresholded (Figure atmap 5, shown 0.008) Table 1). in Figure Pixel-based classification 5. using MTMF (an MTMF 5.2.1 AVIRIS VNIR Eighty-six (86) reflectance corrected spectral bandswere of used AVIRIS data covering for thedescribed VNIR the from analysis. in 0.4 – 1.2 Sectionspectral These micrometers 4.2. characteristics. were Repeated reduced N-dimensional endmembers to with scatterplotting 50 similar was linearly spectral combined used classes bands were to manually using select merged, the endmembers MNF with transform unique as VNIR Downloaded From: http://proceedings.spiedigitallibrary.org/ on 05/27/2017 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx spectral characteristics.14 Repeated endmembers with similar spectral classes were manually merged, resulting in a total of 8 unique SWIR endmembers (Figure 6, Table 1). Pixel-based classification using MTMF (an MTMF MF/Infeasibility ratio thresholded at 0.008)3 and subsequent 3 x 3 sieving and clumping produced the image map shown in Figure 6.

Selected SWIR n-D Visualizer ROI Means iI1rlr r 1íIiií íI Il r1L ii i

*; 0:.zy%. .

-M'

11111 IIIIIIIIIIIII 11111 2 0 2.1 2.2 2.3 2.4 Wavelength (Micrometers) 1

Figure 6: (Left) AVIRIS SWIR endmember spectra extracted using the n-dimensional scatterplotting approach. (Bottom Right) MTMF classiﬁcation result showing distribution of selected SWIR spectral endmembers. Refer to Table 1 for endmember descriptions. The white box shows the location of the small subarea used for validation of results, with the corresponding (Top Center) MTMF classiﬁcation and (Top Right) true-color AVIRIS image.

5.2.3 Combined HSI/LiDAR raster analysis N-dimensional scatterplot analysis of the combined HSI/LiDAR (stacked) raster data cube produced a complex set of “endmembers” with diverse combinations of spectral and spatial characteristics. A total of 31 different classes were defined. Unfortunately, detailed analysis and ground validation of all of these classes is beyond the scope of this investigation. Many of the classes are small and attributable to unique man-made features such as buildings or artificial surfaces. The validity of some classes is unclear. For the purposes of this study, six spatially coherent classes, roughly corresponding to those defined during the LiDAR point cloud analysis were selected for further analysis and validation. The 22-dimensional signatures for these classes are shown in Figure 7. Their spatial distribution for the full study are shown in Figure 8.

Proc. of SPIE Vol. 9832 98320W-10

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 05/27/2017 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx 15 UDAR LIDAR LIDAR - Road Trees Mean Red Roof Mean Mean Grass Height White Roof Height Sidewalk Intensity UDAR Mean _41 Intensity fo LIDAR Mean . 15 Intensity 15

LIDAR Mean

10 10 SWIR Height NDVI 2.2pm 1

VNIR Peak '1 VNIR Red SWIR SWIR 5 Green Tile 2.2pm NDVI VNIR Soils . Vegetation Peak A Asphalt

. I . .. . 1 . . . . 1 . ... 1 5 111 15 20 5 10 15 20 5 10 15 20 lalu Nuwlx:r Layer Nunbef layer Number Figure 7: The six 22-dimensional signatures for selected combined HSI/LiDAR classes similar to those deﬁned during the fused point cloud analysis. (Left) Comparison of signatures for green grass versus green trees; (Center) comparison of signatures for red roofs versus white roofs; (Right) comparison of roads versus sidewalks.

Unclassified Roads/Asphalt. Green Grass Buldings - Red Tile Roofs Green Trees INISiderralks Willi - White Synthetic Roofs Figure 8: (Bottom) Minimum Distance classiﬁcation image map using 22-dimensional “endmember” signatures of the fused HSI LiDAR dataset. The white box shows the location of the small subarea used for validation of results, with the corresponding (Top Left) classiﬁcation results and (Top Right) the true-color AVIRIS image.

Proc. of SPIE Vol. 9832 98320W-11

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 05/27/2017 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx Note in Figure 7 that the combined use of the HSI and rasterized LiDAR data make it possible to separate classes that might otherwise be difficult using only one data modality. For example, the trees and grass, which have similar spectral signatures in the HSI data, are easily mapped as distinct classes when the spectral characteristics are combined with the LiDAR height and intensity information. Also, while rooftops are easily mapped using the LiDAR height data, the addition of HSI information makes it possible to clearly define different roof types. The HSI also contributes additional information towards separation of roads from sidewalks. The two main problems with the rasterized approach with respect to mapping typical ground classes, however, are 1) the HSI spectral mapping results are inherently different than typical ground truth based on human-defined classes, and 2) the reduced resolution precludes full spatial definition of key classes. A confusion matrix (Table 3) was calculated using the small test dataset (Figure 1) and the ground truth data as defined by manually classifying the point cloud data (Figure 9). To calculate a confusion matrix for the 2D raster classification result using 3D ground truth data, each point in the ground truth point cloud was mapped nearest raster pixel. The confusion matrix for the raster result is not directly comparable to the confusion matrix for the point cloud results because not all classes were used in the raster classification approach. The classes used for the raster classification approach include sidewalk, grass, road, trees, and two types of buildings roofs. The “red” and “white” roof classes of the raster classification process were combined into a single “building” class for purposes of calculating overall classification; in the manually classified ground truth data, all buildings were combined into a single class, regardless of roof material. Also, there were pixels that were unclassified in the raster classification results, whereas in the point cloud approach, all points were forced to be classified as one of the ground truth classes. The overall classification accuracy (number of correctly classified points/total number of ground truth points) was calculated using only the classes actually used in the raster-based classification approach (sidewalk, grass, road, trees, and building). The overall classification accuracy for the raster-based approach is poor (51.2%), as expected due to the low 2.4 m spatial resolution of the HSI raster data as compared to the high spatial resolution (60 points/m2) of the manually classified point cloud ground truth. In the 2.4 m HSI pixels, objects such as cars and most sidewalks are represented by mixed spectral signatures. Likewise, within each 2.4 m pixel region, there may be multiple ground truth classes; however, each pixel is assigned an individual class, leading to a high degree of misclassification when measured against the ground truth. It should also be noted that the exclusion due to tiling artifacts of some of the statistical measures used during the point cloud analysis may explain some of the disparities between the raster and point cloud results.

Table 3: Confusion matrix for the fused HSI and LiDAR raster-based classification as applied to the small test set. The overall accuracy of the fused raster classification is 51.2%. The ground truth classes that were not used in the raster classification process (cars and other) were excluded from the confusion matrix. Likewise, there was no unclassified class in the ground truth data.

Predicted Class Number of Ground Sidewalk Grass Road Trees Building Unclassified Truth Points Sidewalk 1428 1657 7874 2573 0 249 13781 %of Ground Truth 10% 12% 57% 19% 0% 2% 3736 16487 36356 22484 98 737 Grass 79898 5% 21% 46% 28% 0% 1% y 1916 355 79180 7495 0 1598 Road 90544 2% 0% 87% 8% 0% 2% al 640 4765 40823 51634 0 393 3 Trees 98255 it 1% 5% 42% 53% 0% 0% 1053 492 8153 1391 5201 1405 Building 17695 6% 3% 46% 8% 29% 8% 0 0 0 0 0 0 Unclassified 0% 0% 0% 0% 0% 0% Number of Classified Points 8773 23756 172386 85577 5299 4382 300173

Proc. of SPIE Vol. 9832 98320W-12

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 05/27/2017 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx 5.3 Comparison of fusion results Ground truth classes were manually defined based on the very high spatial resolution (60 points/m2) point cloud data. In the 2.4 m spatial resolution AVIRIS data, some of these classes represent areas too small to be spectrally distinct (e.g., sidewalks and cars are only represented as spectrally mixed pixels). Additionally, the lower spatial resolution of the raster data results in the loss of edges and sharpness in the combined class definitions and mapping. The diverse nature of the HSI data and spectral mixing at 2.4 m spatial resolution further contributes to spatial coherency problems in the fused raster data analysis. These differences in spatial resolution – 2.4-m AVIRIS pixels versus sub-meter point spacing for the LiDAR data – make a direct comparison of the point-cloud approach versus the raster-based approach challenging. Results of each of the classification schemes as applied to the small test area, with the raster approach classes mapped to the most similar ground truth classes, are shown in Figure 9. The overall classification accuracy of the fused point cloud approach as applied to the test dataset is 88.8% (Table 2), while the accuracy of the fused raster based approach is only 51.2% (Table 3). These numbers are not directly comparable, however, due to the similar (but different) classes used for the two approaches.

Aerial Orthophoto (0.15 m pixels) Fused Point Cloud Classification (60 pts /m2) s

Manually Classified Point Cloud Ground Truth (60 pts /m2) plUll Road Buildings Sidewalk Cars Grass Other/ Trees unknown A. AVIRIS True -color Image Fused Raster Classification (2.4 m pixels) (2.4 m pixels)

Figure 9: (Left) Ground truth point cloud data; (Top Center) true color aerial photo; (Top Right) the fused HSI and LiDAR point-cloud classification result (60 points/m2) with an overall classification accuracy of 88.8%; (Bottom Center) AVIRIS true color image; and (Bottom Right) the fused HSI and LiDAR raster classification result (2.4 m pixels) with an overall classification accuracy of 51.2%.

Proc. of SPIE Vol. 9832 98320W-13

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 05/27/2017 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx 6. DISCUSSION, CONCLUSIONS, AND FUTURE WORK This study illustrates the use of fused HSI and LiDAR data to map the distribution and character of selected Earth-surface materials. Individually, HSI data map surface composition and LiDAR data map surface morphology. Analysis of a point cloud fusion approach and a raster fusion approach illustrates that fusing LiDAR and HSI together provides improved characterization and better classification maps. Each of the fusion approaches has strengths and drawbacks. The main advantage to the point cloud processing approach is the ability to maintain the high-spatial resolution of the LiDAR point cloud data. In the point cloud approach, the information from each AVIRIS pixel is mapped to the much higher resolution of the LiDAR point cloud data. In a fashion similar to “pan-sharpening”, this approach maintains the spatial resolution of the LiDAR point cloud data, while also incorporating information from the HSI analysis. The principal advantage of the raster approach is its simplicity; however, significant losses in classification accuracy resulted from the reduced spatial resolution as compared to the point cloud method. The main challenge in working with the LiDAR point cloud data is the computational complexity of working with irregularly gridded data. The necessity of the tiled processing approach introduced artifacts due to different ranges of values in each tile; while these artifacts did not have a significant effect on the point cloud processing approach, they did cause problems with the raster approach that prohibited use of most of the LiDAR neighborhood statistics. A direct quantitative comparison of the raster fusion approach and the point cloud fusion approach is difficult due to the disparity in spatial resolution of the datasets, but some qualitative observations can be made:

• Adding the LiDAR information into the raster-based HSI process helped distinguish materials which were spectrally similar, such as trees and grass. It also allowed distinguishing between morphologically similar roofs with diﬀerent spectral characteristics (such as red and white roofs).

• For the point cloud approach, adding HSI information into the LiDAR point cloud resulted in an improvement in overall classiﬁcation accuracy of approximately 3.5% as compared to using LiDAR data only, even though the 2.4 m pixel spatial resolution of the HSI data is very low compared to the LiDAR data (60 pts/m2).

• The main challenge in combining HSI and LiDAR data in this study was the large disparity in spatial resolution. It is hypothesized that the improvement in classification accuracies for datasets with more similar spatial resolutions would be more obvious than the results shown here. • While the raster based approach appears to perform very poorly in this work, this is explained by the differences in spatial resolution between the HSI data and the ground truth data that was derived from the LiDAR point cloud data. The raster approach is a useful method for combining HSI and LiDAR data in a way that enables the use of standard spectral processing tools (including commercial software packages), and derives some benefit from fusing the separate modalities.

The ability of the point cloud approach to maintain the spatial resolution of the LiDAR data, while also incorporating the spectral information from lower spatial resolution imagery, has huge potential. More work is needed to make the process more computationally efficient and automated. A large portion of this work was accomplished using code written by the authors. The point cloud and raster based fusion approaches introduced in this work demonstrate the benefit of combining HSI and LiDAR datasets, and also highlight some of the challenges of working with these very different collection modalities. We see huge potential in recently developed spectral LiDAR datasets (such as the Teledyne Optech Titan system17) and the prototype hyperspectral LiDAR systems being developed at the Finnish Geodetic Institute5–7 as these data eliminate the need to perform fusion!

Proc. of SPIE Vol. 9832 98320W-14

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 05/27/2017 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx REFERENCES [1] A. F. H. Goetz, G. Vane, J. E. Solomon, and B. N. Rock, “Imaging spectrometry for Earth remote sensing,” Science 228, pp. 1147–1153, 1985. [2] R. N. Clark, G. A. Swayze, K. E. Livo, T. Hoefen, R. F. Kokaly, S. J. Sutley, J. B. Dalton, R. R. McDougal, and C. A. Gent, “Imaging spectroscopy: Earth and planetary remote sensing with the USGS Tetracorder and expert systems,” Journal of Geophysical Research 108(5), pp. 1–44, 2003. [3] F. A. Kruse, “Integrated visible and near infrared, shortwave infrared, and longwave infrared (VNIR-SWIR- LWIR), full-range hyperspectral data analysis for geologic mapping,” Journal of Applied Remote Sensing 9, 2015. [4] C. Debes, A. Merentitis, R. Heremans, J. Hahn, N. Frangiadakis, T. van Karsteren, W. Liao, R. Bel- lens, A. Pizurica, S. Gautama, W. Philips, S. Prasad, Q. Du, and F. Pacifici, “Hyperspectral and LiDAR data fusion: Outcome of the 2013 data fusion contest,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 7(6), pp. 2405–2418, 2013. [5] T. Hakala, J. Suomalainen, S. Kaasalainen, and Y. Chen, “Full waveform hyperspectral LiDAR for terrestrial laser scanning,” Optics Express 20(7), pp. 7119–7127, 2012. [6] J. Suomalainen, T. Hakala, H. Kaartinen, E. Raikkonen, and S. Kaasalainen, “Demonstration of a virtual active hyperspectral LiDAR in automated point cloud classification,” ISPRS Journal of Photogrammetry and Remote Sensing 66, pp. 637–641, 2011. [7] E. Puttonen, J. Suomalainen, S. Kaasalainen, and Y. Chen, “Tree species classification from fused active hyperspectral reflectance and LiDAR measurements,” Forest Ecology and Management 260, pp. 1843–1852, 2010. [8] S. Buckle, T. H. Kurz, J. A. Howell, and D. Schneider, “Terrestrial LiDAR and hyperspectral data fusion products for geological outcrop analysis,” Computers and Geosciences 2013, pp. 249–258, 54. [9] R. Green, AVIRIS and related 21st century imaging spectrometers for Earth and space science, pp. 335–358. Chapman and Hall/CRC Press, Boca Raton, FL, 2007. [10] M. Isenburg, “LAStools - efficient tools for LiDAR processing.” [11] N. Brodu and D. Lague, “3D terrestrial lidar data classification of complex natural scenes using a multi-scale dimensionality criterion: Applications in geomorphology,” ISPRS Journal of Photogrammetry and Remote Sensing 68, pp. 121–134, 2012. [12] J. W. Rouse, R. H. Haas, J. A. Schell, and D. W. Deering, “Monitoring vegetation systems in the great plains with ERTS,” Third ERTS Symposium, NASA SP-351, pp. 309–317, 1973. [13] C. J. Tucker, “Red and photographic infrared linear combinations for monitoring vegetation,” Remote Sensing of the Environment 8, pp. 127–150, 1979. [14] J. W. Boardman and F. A. Kruse, “Analysis of imaging spectrometer data using n-dimensional geometry and a mixture-tuned matched filtering (MTMF) approach,” Transactions on Geoscience and Remote Sensing (TGARS), Special Issue on Spectral Unmixing of Remotely Sensed Data 49(11), pp. 4138–4152, 2011. [15] M. W. Matthew, S. M. Adler-Golden, A. Berk, G. Felde, G. P. Anderson, D. Gorodetzky, S. Paswaters, and M. Shippert, “Atmospheric correction of spectral imagery: Evaluation of the FLAASH algorithm with AVIRIS data,” Proceedings of SPIE, Algorithms and Technologies for Multispectral, Hyperspectral, and Ultraspectral Imagery IX, Orlando, FL 5093, pp. 474–482, 2003. [16] A. A. Green, M. Berman, P. Switzer, and M. D. Craig, “A transformation for ordering multispectral data in terms of image quality with implications for noise removal,” IEEE Transactions on Geoscience and Remote Sensing 26(1), pp. 65–74, 1988. [17] Teledyne Optech Incorporated, “Optech Titan multispectral lidar system specifications,” 2015.

Proc. of SPIE Vol. 9832 98320W-15

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 05/27/2017 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx