Ultraspectral Data Compression A Dissertation Presented by Rolando Herrero to The Department of Electrical and Computer Engineering in partial fulfillment of the requirements for the degree of Doctor of Philosophy in the field of Electrical Engineering Northeastern University Boston, Massachusetts October, 2012 Abstract Ultraspectral images capture 2D data tuned at different wavelengths across the mid infrared electromagnetic spectrum. Atmospheric Infrared Sounder (AIRS) remote sensors are ultraspectral sensors that generate images with thousands of highly correlated bands and are considered the future of spectroscopy. The major application of AIRS is the acquisition of atmospheric parameters such as gases, temperature and other quantities to perform climate and weather forecast. Because typical AIRS images are well over 40 MB in size and because they are captured in remote locations data compression (before transmission) becomes a very critical issue. Existent compression studies of AIRS data adapt generic multispectral image compression techniques (not necessarily ultraspectral) but do not take into account the particular nature of ultraspectral images. Most of them do not consider correlation beyond one band, use fixed linear prediction (leading to significant distortion) and are not optimized to overcome time and space complexity constraints. Moreover compression studies do not provide analytical models of rate-distortion either. This dissertation focuses on presenting a whole new architecture for the compression of ultraspectral data presenting sound mathematical models that can be used to describe a set of algorithms and their practical implementation. Specifically we will (1) present a new preprocessing reversible stage that will rearrange the data to make it more efficient when the compression stage is performed; (2) present a new linear prediction based compression stage that will improve the compression rate of any given distortion when compared to literature ultraspectral data compression techniques. This involves defining a distortion measure and its effect on real applications; (3) define a mathematical model to approximate the rate- distortion of the compression stage and compare it against the real performance of the proposed algorithm; (4) compare the performance of the overall architecture against that of other state of the art compression schemes. To summarize it can be said that the results of this dissertation will contribute to the understanding of ultraspectral compression as well as the introduction of a novel ultraspectral codec. Contents 1 Introduction ....................................... 18 1.1 MultispectralImaging ............................... 18 1.2 MultispectralData................................. 23 1.2.1 AVIRIS.................................... 23 1.2.2 AIRS..................................... 24 1.2.3 AIRSProcessing............................... 25 1.3 UltraspectralDataCompression . .. 26 1.3.1 LosslessvsLossyCompression . 26 1.3.2 DistortionMeasure ............................. 27 1.3.3 Architecture for Ultraspectral Data Compression . ..... 30 1.4 ObjectivesandContribution . 30 2 ReviewofLiterature .................................. 32 2.1 BasicTechniques.................................. 32 2.1.1 EntropyCoding ............................... 32 2.1.2 VectorQuantization............................. 33 2.1.3 TransformCoding.............................. 34 2.1.3.1PCA................................. 34 2.1.3.2DCT................................. 34 2.1.3.3DWT ................................ 36 2.1.4 Linear Mixture Analysis . 36 2.1.5 LinearPrediction .............................. 36 2.2 PreprocessingTechniques . 44 2.2.1 BandOrdering................................ 44 2.2.2 BiasAdjustedReordering. 45 2.3 CompressionTechniques. .. .. 45 2.3.1 EntropyCoding ............................... 45 2.3.2 VectorQuantization............................. 46 2.3.3 TransformCoding.............................. 47 2.3.3.1PCA................................. 47 2 2.3.3.2DCT................................. 47 2.3.3.3DWT ................................ 47 2.3.4 FractalBasedCoding............................ 48 2.3.5 LinearPrediction .............................. 49 2.3.5.13DADPCM............................. 49 2.3.5.2JPEG-7 ............................... 49 2.3.5.3CALIC................................ 51 2.3.5.4JPEG-LS .............................. 51 2.3.5.5CELP ................................ 52 2.3.5.6SLSQ ................................ 53 2.3.5.7PLT ................................. 53 2.4 ProposedApproach ................................ 53 3 PreprocessingStage................................. .. 55 3.1 AIRSImages .................................... 55 3.2 TheSpeechAnalogy ................................ 58 3.3 PreprocessingSteps ................................ 61 3.3.1 BandPreprocessing ............................. 61 3.3.1.1 Band Normalization . 61 3.3.1.2 SubbandDecomposition . 62 3.3.2 BandOrdering................................ 69 3.3.3 ImageScanning ............................... 71 3.4 Summary ...................................... 86 4 CompressionStage ................................... 91 4.1 PredictionGain................................... 91 4.2 Longtermprediction................................ 94 4.3 InputFrameSize.................................. 104 4.4 PredictionCoefficients............................... 105 4.5 PredictionError .................................. 113 4.6 Summary ...................................... 121 5 RateDistortionAnalysis ................................ 125 5.1 Introduction .................................... 125 5.2 MathematicalModel................................ 126 5.2.1 PredictionError............................... 126 5.2.2 Rate ..................................... 128 5.2.3 Distortion .................................. 130 5.3 TheoreticalRateDistortion . 133 3 5.4 SimplifiedModel .................................. 136 5.5 Summary ...................................... 139 6 PerformanceComparison . .. .. 140 6.1 Introduction .................................... 140 6.2 NewAIRSLPCELPCompression . 141 6.3 OtherCompressionTechniques . 152 6.3.1 3DADPCM ................................. 157 6.3.2 JPEG-7 ................................... 163 6.3.3 CALIC.................................... 175 6.3.4 DWT..................................... 191 6.3.5 3DSPIHT.................................. 205 6.3.6 3DSPECK ................................. 209 6.4 CompressionResults................................ 215 7 ConclusionsandFutureWork . .. .. 216 7.1 Conclusions..................................... 216 7.2 FutureWork .................................... 218 A Rate-DistortionComputation . 219 A.1 Prediction Error Probability . 219 A.2Rate......................................... 220 A.3Distortion...................................... 221 BAlgorithms........................................ 222 B.1LBG......................................... 222 B.1.1LloydAlgorithm............................... 222 B.1.2 Generalized Lloyd Algorithm . 223 B.1.3 Stochastic Relaxation . 224 B.2Durbin-Levinson .................................. 226 B.2.1PredictorofOrderZero. .. .. 227 B.2.2EditorofOrderOne............................. 228 B.2.3PredictorofOrderTwo. .. .. 229 B.2.4PredictorofOrderThree . 230 B.2.5Summary................................... 231 B.2.6 Conversion of Reflection Coefficients to Linear Prediction Coefficients . 233 B.2.7 Conversion of Linear Prediction Coefficients to Reflection Coefficients . 233 Bibliography ........................................ 236 4 List of Figures 1.1 HyperspectralDataStructure. ... 19 1.2 ReflectanceCurve ................................ 19 1.3 HyperspectralPixelVector . 20 1.4 SpectralSolarIrradiance ............................ 21 1.5 AtmosphericAbsorption ............................ 22 1.6 AVIRIS...................................... 24 1.7 AIRS....................................... 25 1.8 Encoder...................................... 29 1.9 Decoder...................................... 29 2.1 PCAaxisrotation: (a)before,(b)after . ... 35 2.2 DWT....................................... 35 2.3 LP ........................................ 36 2.4 SystemIdentification .............................. 37 2.5 Barnwellwindow................................. 40 2.6 ARencoder.................................... 40 2.7 ARdecoder.................................... 41 2.8 MAencoder ................................... 41 2.9 MAdecoder ................................... 41 2.10 ARencoder.................................... 42 2.11 ARdecoder.................................... 42 2.12 MAencoder ................................... 43 2.13 MAdecoder ................................... 43 2.14 FBencoding ................................... 48 2.15 3DADPCM ................................... 49 2.16 JPEG-7...................................... 50 2.17 JPEG-LS..................................... 50 2.18 SLSQContexts.................................. 52 3.1 March3,2004Granule16............................ 56 6 3.2 March3,2004Granule82............................ 56 3.3 March3,2004Granule126 ........................... 57 3.4 March3,2004Granule151 ........................... 57 3.5 March3,2004Granule193 ........................... 58 3.6 Voicedframe................................... 60 3.7 Voicedframe(zoom)............................... 60 3.8 Two-Level Signal Decomposition/Reconstruction . ..... 63 3.9 Row-ColumnDWTComputation. 63 3.10 SecondLevelDecomposition