A Comparison of Compression Codecs for Maritime and Sonar Images in Bandwidth Constrained Applications

computers

Article A Comparison of Compression Codecs for Maritime and Sonar Images in Bandwidth Constrained Applications

Chiman Kwan 1,* , Jude Larkin 1 , Bence Budavari 1, Bryan Chou 1, Eric Shang 2 and Trac D. Tran 3 1 Applied Research, LLC, Rockville, MD 20850, USA; [email protected] (J.L.); [email protected] (B.B.); [email protected] (B.C.) 2 Leidos, Columbia, MD 21046, USA; [email protected] 3 Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD 21218, USA; [email protected] * Correspondence: [email protected]; Tel.: +1-240-207-2311

Received: 14 February 2019; Accepted: 24 April 2019; Published: 28 April 2019

Abstract: Since lossless compression can only achieve two to four times data compression, it may not be eﬃcient to deploy lossless compression in bandwidth constrained applications. Instead, it would be more economical to adopt perceptually lossless compression, which can attain ten times or more compression without loss of important information. Consequently, one can transmit more images over bandwidth limited channels. In this research, we ﬁrst aimed to compare and select the best compression algorithm in the literature to achieve a compression ratio of 0.1 and 40 dBs or more in terms of a performance metric known as human visual system model (HVSm) for maritime and sonar images. Our second objective was to demonstrate error concealment algorithms that can handle corrupted pixels due to transmission errors in interference-prone communication channels. Using four state-of-the-art codecs, we demonstrated that perceptually lossless compression can be achieved for realistic maritime and sonar images. At the same time, we also selected the best codec for this purpose using four performance metrics. Finally, error concealment was demonstrated to be useful in recovering lost pixels due to transmission errors.

Keywords: perceptually lossless compression; error recovery; maritime and sonar images; JPEG2000; X264; X265; Daala

1. Introduction Since the appearance of the Joint Photographic Experts Group (JPEG) standard in the 1990s, image compression has been well-developed [1–6]. Lossless image compression algorithms include JPEG [7] and JPEG2000 [8]. Some recent algorithms such as X264 (software implementation of H.264/AVC standard) [9] and X265 (software implementation of H.265/HEVC standard) [10] also provide lossless compression options. JPEG, X264, and X265 are discrete cosine transform (DCT) based algorithms and JPEG2000 is wavelet based. About 15 years ago, there were some developments in DCT based algorithms, where overlapped blocks known as lapped transforms (LT) were used to further improve the compression [11]. In the past few years, a group of researchers at Xiph have incorporated LT [11] into an open source codec known as Daala [12]. Through several years of rigorous development, Daala has reached a stage where it outperforms X264 and has comparable performance to X265 in terms of a performance metric that mimics human visual system (HVS) [12,13]. In this research, our sponsor had speciﬁed three requirements on the image codec for maritime and sonar images: (1) 10 to 1 compression ratio; (2) the decompressed image should have 40 dBs or

Computers 2019, 8, 32; doi:10.3390/computers8020032 www.mdpi.com/journal/computers Computers 2019, 8, 32 2 of 21 more in HVSm, which may be considered as “near perceptually lossless” performance; and (3) the lost pixels need to be effectively concealed without incurring additional bandwidth. It should be noted that “perceptually lossless” has been defined in digital picture coding since mid-1990s [14]. One conventional way to handle transmission errors is to adopt error correction coding, which will add redundant bits and reduce the data transmission efficiency. We explored error concealment techniques, which do not incur additional bandwidth usage. It is important to emphasize that perceptual performance requires a suitable metric. Some conventional metrics such as peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) may not match well with human’s subjective evaluations. Two metrics in the literature that have better correlation with human perception in the literature were also compared in our studies. In the literature, there are excellent papers discussing perceptually lossless coding [15–18]. The authors of [15] provided a survey of various ideas of embedding human perceptual model into the encoding process of various codecs such as JPEG2000, H264, and H265. The concepts of just noticeable difference (JND) and just not noticeable difference (JNND) [15–18] were introduced and embedded into some of the codecs. The output bit stream is compliant with the standards. Daala also incorporated perceptual vector quantization (PVQ) into its codec [12,13], which resulted in enhanced performance. We would like to emphasize that our paper is not about new developments in the perceptually lossless coding. In our studies, we used the codecs using the default settings and did not change any of the parameters. The only parameters we varied is the quality (or quantization) parameter (qp). The decompressed images were then evaluated using the four metrics, two of which are related to HVS. Similar to some existing papers (e.g., [15]), our sponsor specified 40 dBs or more in terms of HVSm, which may be considered as “near perceptually lossless”. It should be noted that this paper is an extension of our conference paper [19]. There are several key differences. First, we corrected an issue related to the compression ratio computation in our earlier study [19]. Since the input image formats for different codecs are different, we converted all the raw images to different image formats in our earlier study. For instance, X264 and X265 require the format to be Y4M and JPEG2000 and Daala use other formats. However, one issue is that the converted file sizes are slightly different for different formats. In our earlier study [19], each codec was using its own set for computing the compression ratios, causing some slight differences in terms of compression ratios. In this study, we consistently used the original image sizes as references instead of the converted ones. Second, when we generated the results presented in [19], X264 and X265 could only handle YUV420 format. In this study, we used the latest versions of X264 and X265, which can handle YUV444 format. Moreover, Daala has a newer version, which is improved over its previous version. Now, we re-generated all the performance metrics for all the datasets in this study. Third, we added two new sections to this paper containing new maritime and sonar images, and one new section containing images from the Xiph website. This is to demonstrate that the compression algorithms need to be robust to images with different resolutions and modalities. Fourth, we also present new error concealment results for the maritime and sonar images in this paper. Again, our goal was to demonstrate that the error concealment algorithms can handle different image modalities. Although the compression algorithms are well-known, there was still some customization work in this project. For example, the use of YUV444 format instead of the default YUV420 ensured high quality compression for X264 and X265 for still images. Moreover, the error concealment needed to be tailored to the data that we used. One key contribution of our project was to integrate three components (compression, error concealment, and near perceptually lossless evaluation after decompression) in the compression process into a single system to achieve 10 to 1 near perceptually lossless compression for maritime and sonar images. That is, the novelty of our paper is not in new perceptually lossless compression theory, but rather in the integration of existing compression and error concealment technologies to achieve near or even perceptually lossless image compression at 10 to 1 compression for bandwidth constrained and interference-prone applications. Computers 2019, 8, 32 3 of 21

Our paper is organized as follows. Section2 summarizes the technical approach, the various codecs, and the performance metrics. Section3 summarizes all the experiments using actual maritime and sonar images that are of interest to our customer. Visual comparisons of different codecs at 10 to 1 compression are also included. Finally, concluding remarks are given in Section4. Computers 2019, 2, x FOR PEER REVIEW 3 of 20 2. Technical Approach 2. Technical Approach 2.1. Proposed Image Compression and Error Recovery Approach 2.1. Proposed Image Compression and Error Recovery Approach In this study, we focused on objective evaluations using four well-known compression algorithms in theIn literature. this study, This we was focused to ensure on that objective we delivered evaluations the best using algorithm four to well-known our customer. compression Our overall technicalalgorithms approach in the literature. is summarized This was in Figure to ensure1. First, that we we briefly delivered reviewed the best the state-of-the-art algorithm to our compression customer. algorithmsOur overall available technical on approach the market. is summarized At the same in time, Figure we 1. described First, we dibrieflyfferent reviewed performance the state-of-themetrics for algorithmart compression evaluation. algorithms The focus available was on on metrics the mark thatet. can At better the same model time, human we perception.described different We also reviewedperformance the metrics error concealment for algorithm techniques. evaluation. Second, The focus we was obtained on metrics realistic that maritimecan better (lowmodel and human high resolution)perception. andWe sonaralso reviewed images for the algorithm error concealm evaluation.ent Third,techniques. we applied Second, the we various obtained compression realistic algorithmsmaritime (low to the and collected high resolution) images and and generated sonar images various for performance algorithm evaluation. metrics. Finally, Third, wewe alsoapplied applied the advancedvarious compression algorithms algorithms to deal with to the the corrupted collected pixelsimages due and to generated channel errors. various performance metrics. Finally, we also applied advanced algorithms to deal with the corrupted pixels due to channel errors. Error Compression Recovery Performance Performance Metrics Metrics

Images/videos Compression Received Images/videos Decompression (maritime/sonar) Algorithms Communication (maritime/sonar) and (X264, X265, Channel overlapped block Error Recovery transform, JPEG-2000)

Figure 1. ProposedProposed approach approach to evaluating different different compression and error concealment algorithms. 2.2. Short Review of Compression Algorithms 2.2. Short Review of Compression Algorithms We compared image codecs on the market, objectively evaluated each one using diverse maritime We compared image codecs on the market, objectively evaluated each one using diverse and sonar images, and recommended the best codec to our customer. With that in mind, we performed maritime and sonar images, and recommended the best codec to our customer. With that in mind, a brief review of the existing high performance codecs, performance metrics, and error resilient coding. we performed a brief review of the existing high performance codecs, performance metrics, and error 2.2.1.resilient DCT coding. Based Algorithms JPEG [1]: JPEG is the very first image compression standard developed in the 1990s. The video •2.2.1. DCT Based Algorithms counterparts are the MPEG-1 and MPEG-2 standards. It is efficient and is still being used by • JPEG [1]: JPEG is the very first image compression standard developed in the 1990s. The video NASA in space applications [4,5]. counterparts are the MPEG-1 and MPEG-2 standards. It is efficient and is still being used by JPEG-XR [20]: It was developed by Microsoft. The performance is comparable to JPEG-2000. It is • NASA in space applications [4,5]. mainly used for still image compression. • JPEG-XR [20]: It was developed by Microsoft. The performance is comparable to JPEG-2000. It VP8 and VP9 [21,22]: These video compression algorithms are owned by Google. The performance • is mainly used for still image compression. is somewhat close to X-264. However, it is not as popular as X264. • VP8 and VP9 [21,22]: These video compression algorithms are owned by Google. The X-264 [9]: X264 is the current state-of-the-art algorithm in video compression. YouTube uses X264. • performance is somewhat close to X-264. However, it is not as popular as X264. It has good still image compression as well. • X-264 [9]: X264 is the current state-of-the-art algorithm in video compression. YouTube uses X-265 [10]: This is the next-generation video codec and has excellent still image compression and • X264. It has good still image compression as well. • videoX-265 compression.[10]: This is the However, next-generation the computational video codec complexity and has excellent is much still more image than compression that of X264. and In general,video compression. X265 has the However, same basic the structure computational as previous comp standards,lexity is much but contains more than many that incremental of X264. In improvementsgeneral, X265 has over the X264. same Several basic structure studies conclude as previous that standards, X265 yields but the contains same quality many incremental as X264, but withimprovements only half the over bitrate. X264.It Several should studies be noted conclude that X264 that and X265 X265 yields are optimized the same versionsquality as of X264, H264 andbut with H265, only respectively. half the bitrate. It should be noted that X264 and X265 are optimized versions of Daala [12]: Recently, there is a parallel activity at Xiph.org foundation, which implements a • H264 and H265, respectively. • compressionDaala [12]: Recently, codec called there Daala is a [12parallel]. It is basedactivity on at DCT. Xiph.org There arefoundation, pre- and post-filterswhich implements to increase a compression codec called Daala [12]. It is based on DCT. There are pre- and post-filters to increase energy compaction and remove block artifacts. This type of transform is known as lapped transform (LT). Daala borrows ideas from the work in [11], which was written by one of us (T. D. Tran). The block-coding framework in Daala is illustrated in Figure 2. Computers 2019, 8, 32 4 of 21

energy compaction and remove block artifacts. This type of transform is known as lapped Computerstransform 2019, 2, (LT).x FOR DaalaPEER REVIEW borrows ideas from the work in [11], which was written by one of us4 (T. of D.20 Computers 2019, 2, x FOR PEER REVIEW 4 of 20 Tran). The block-coding framework in Daala is illustrated in Figure2.

Figure 2. Daala codec for block-based image coding systems. Figure 2. Daala codec for block-based image coding systems. Figure 2. Daala codec for block-based image coding systems. 2.2.2. Wavelet Based Algorithms 2.2.2. Wavelet Based Algorithms JPEG2000 is a wavelet [8] based compression standard. It has better performance than JPEG. However,JPEG2000 JPEG2000 is a wavelet requires [[8]8 the] basedbased use of compressioncompression the whole im standard.stageandard. for coding It It has has and better better hence performance performance is not suitable than than for JPEG. JPEG. real- However,time applications. JPEG2000JPEG2000 requiresIn requires this study, the the use use ofwe of the themainly whole whole imagecompressed image for for coding codingimages and and henceusing hence isDaala, not is not suitable X264, suitable for X265, real-time for real-and timeapplications.JPEG2000. applications. In this In study, this westudy, mainly we compressedmainly compressed images using images Daala, using X264, Daala, X265, X264, and JPEG2000. X265, and JPEG2000. 2.3. Principle of Error Concealment for Still Images 2.3. Principle of Error Concealment for Still Images 2.3. PrincipleError resilient of Error coding Concealment has some for Still major Images issues. First, it increases overhead and hence lowers the Error resilient coding has some major issues. First, it increases overhead and hence lowers the coding efficiency. Second, error resilient coding can only repair corrupted pixels to a certain extent. In codingError efficiency. resilient Second, coding errorhas some resilient major coding issues. can First, only it repairincreases corrupted overhead pixels and to hence a certain lowers extent. the severe channel conditions, some additional post-processing, such as error concealment, is needed to fix codingIn severe efficiency. channel Second,conditions, error some resi additionallient coding post-processing, can only repair such corrupted as error pixels concealment, to a certain is neededextent. the corrupted pixels and recover lost data. Into severefix the channelcorrupted conditions, pixels and some recover additional lost data. post-processing, such as error concealment, is needed The core idea of the local matrix completion with similar blocks (LMCS) [23] is that, for a missing to fix Thethe corecorrupted idea of pixels the local and matrix recover completion lost data. with similar blocks (LMCS) [23] is that, for a missing pixel, we first extract the corresponding patch and use similar patches in the image to help fill in the pixel,The we core first idea extract of the the local corresponding matrix completion patch and with use similar similar blocks patches (LMCS) in the [23] image is that, to help for a fill missing in the missing value based on matrix completion. If there are not enough similar patches, we fill in the pixel pixel,missing we value first extractbased onthe matrix corresponding completion. patch If there and useare notsimilar enough patches similar in the patches, image we to helpfill in fill the in pixel the with another of our algorithm, known as local patch matrix completion (LPMC) [24]. A flow chart for missingwith another value of based our algorithm,on matrix completion.known as local If there patch are matrix not enough completion similar (LPMC) patches, [24]. we A fill flow in thechart pixel for the LMCS algorithm is also shown in Figure3. More details of our algorithms can be found in [ 23,24]. withthe LMCS another algorithm of our algorithm, is also shown known in Figure as local 3. patch More matrixdetails completionof our algorithms (LPMC) can [24]. be foundA flow in chart [23,24]. for Researchers have used sparse representation in hyperspectral image processing [25–28], denoising [29], theResearchers LMCS algorithm have used is also sparse shown representation in Figure 3. Morein hyperspectral details of our image algorithms processing can be [25–28], found indenoising [23,24]. compressive sensing [30,31], and target recognition ([32], and references therein). Researchers[29], compressive have usedsensing sparse [30,31], representation and target recognition in hyperspectral ([32], and image references processing therein). [25–28], denoising [29], compressive sensing [30,31], and target recognition ([32], and references therein). Input image with missing blocks Input image with missing blocks Get patch p of the pixel Get missing pixels Get patch p of the pixel Get missingat corners pixels Get patches similar to p return if completeat corners Get patches similar to p return if complete Estimate value for more than 5 No each missing pixels Estimate value for similarmore patches?than 5 No each missing pixels similar patches? Yes Do matrix completionYes Update mask Do matrix completion Update mask Do interpolation Do interpolation Figure 3. The flow chart of our LMCS algorithm to deal with missing blocks [23]. Figure 3. The flow chart of our LMCS algorithm to deal with missing blocks [23]. Figure 3. The flow chart of our LMCS algorithm to deal with missing blocks [23]. 2.4. Performance Metrics 2.4. Performance Metrics In many compression systems, researchers use the peak signal-to-noise ratio (PSNR) or structuralIn many similarity compression (SSIM) tosystems, evaluate researchersthe compression use algorithms.the peak signal-to-noise Given a fixed compressionratio (PSNR) ratio, or structural similarity (SSIM) to evaluate the compression algorithms. Given a fixed compression ratio, Computers 2019, 8, 32 5 of 21

2.4. Performance Metrics In many compression systems, researchers use the peak signal-to-noise ratio (PSNR) or structural similarity (SSIM) to evaluate the compression algorithms. Given a fixed compression ratio, algorithms that yield higher PSNR or SSIM are regarded as better algorithms. However, PSNR and SSIM do not correlate well with human perception. Recently, a group of researchers investigated a number of different performance metrics [33]. Extensive experiments were performed to investigate the correlation between human perception with various performance metrics. According to the results found in [33], it was determined that two performance metrics known as human visual system (HVS) and HVS with masking (HVSm) correlate well with human perception. For completeness, we briefly present the above four metrics. PSNR [34] • To generate PNSR, we need to compute the Root Mean Squared Error (RMSE). The RMSE of two vectorized images S (ground truth) and Sˆ (prediction) is defined as

uv tu Z 1 X 2 RMSE(S, Sˆ ) = s sˆ (1) Z j − j j=1 where Z is the number of pixels in each image. The ideal value of RMSE is 0 if the prediction is perfect. The PSNR is related to RMSE deﬁned in Equation (1). If the image pixels are expressed in doubles with values between 0 and 1, then PSNR = 20 log(1/RMSE (S, Sˆ )) (2)

A higher PSNR means better quality. SSIM • This is a metric [35] to reflect the similarity between two images. The SSIM index is computed on various blocks of an image. The measure between two blocks x and y from two images can be defined as (2µxµy + c1)(2σxy + c2) SSIM(x, y) = (3) 2 2 2 2 (µx + µx + c1)(σx + σx + c2) 2 2 where µx and µy are the means of blocks x and y, respectively; σx and σy are the variances of blocks x and y, respectively; σxy is the covariance of blocks x and y; and c1 and c2 are small values (0.01, for instance) to avoid instability. The ideal value of SSIM is 1 for perfect prediction. HVS • The HVS metric is defined as

HVS = 20log(255/MSEH) (4) where I 7 J 7 8 8 X− X− X X e 2 MSEH = K ((X[m, n] X[m, n] )Tc[m, n]) (5) ij − ij i=1 j=1 m=1 n=1 I and J denote image size, K = 1/[(I 7)(J 7)64], X are the discrete cosine transform (DCT) [36] − − ij coeﬃcients of 8 8 image block for which the coordinates of the its upper left corner are equal to i and e × j, Xij are the DCT coeﬃcients of the corresponding block in the original image, and Tc is the matrix of correcting factors [37]. HVSm • Computers 2019, 8, 32 6 of 21

This metric is similar to HVS except that visual masking eﬀects are taken into account. A block diagramComputers is2019 shown, 2, x FOR in PEER Figure REVIEW4. The inclusion of a block containing contrast masking is the only6 of 20 Computers 2019, 2, x FOR PEER REVIEW 6 of 20 diﬀerence between HVS and HVSm. Details can be found in [33].

FigureFigure 4. 4.Flow Flow chart chart for for HVSm. HVSm. Figure 4. Flow chart for HVSm. OneOne example example shown shown in in Figure Figure5 demonstrates 5 demonstrates that HVSmthat HVSm is closer is closer to human to human perception perception than SSIM. than One example shown in Figure 5 demonstrates that HVSm is closer to human perception than SSIM. SSIM.

FigureFigure 5. 5.Comparison Comparison of of SSIM SSIM and and HVSM. HVSM. HVSM HVSM has has better better correlation correlation with with human human perception perception [ 33[33].]. Figure 5. Comparison of SSIM and HVSM. HVSM has better correlation with human perception [33]. OnOn the the website website of of the the authors authors of [ 33of], [33], there there is a table is a containingtable containing the correlation the correlation of diﬀerent of different metrics On the website of the authors of [33], there is a table containing the correlation of different withmetrics human with perception. human perception. For completeness, For completeness, we include we that include table that below table (Table below1). It(Table can be 1). seen It can that be metrics with human perception. For completeness, we include that table below (Table 1). It can be HVSmseen that and HVSm HVS haveand HVS much have higher much correlation higher correla with humantion with perception human perception than PSNR than and PSNR SSIM and in terms SSIM seen that HVSm and HVS have much higher correlation with human perception than PSNR and SSIM ofin Spearman terms of Spearman and Kendall and correlation Kendall correlation coeﬃcients. coefficients. in terms of Spearman and Kendall correlation coefficients.

TableTable 1. 1.Correlation Correlation of of di differentﬀerent metrics metrics to to human’s human’s visual visual perception. perception. Table 1. Correlation of different metrics to human’s visual perception. MeasureMeasure Reference Reference Spearman Spearman Correlation correlation Kendall Kendall Correlation correlation Measure Reference Spearman correlation Kendall correlation HVS-mHVS-m Ponomarenko, Ponomarenko, N.;N.; et et al. al. [33 [33]] 0.984 0.984 0.948 0.948 HVS-m Ponomarenko, N.; et al. [33] 0.984 0.948 HVSHVS Egiazarian, Egiazarian, K.;K.; etet al. al. [37 [37]] 0.895 0.895 0.712 0.712 HVS Egiazarian, K.; et al. [37] 0.895 0.712 NQMNQM Damera-Venkata, Damera-Venkata, N.; N.; et et al. al [.38 [38]] 0.857 0.857 0.673 0.673 NQM Damera-Venkata, N.; et al. [38] 0.857 0.673 DCTune Watson, A. B.; et al. [39] 0.829 0.712 DCTuneDCTune Watson, Watson, A. A. B.; etet al.al. [ 39[39]] 0.829 0.829 0.712 0.712 UQI Wang, Z.; Bovik, A. C. [40] 0.550 0.438 UQIUQI Wang, Wang, Z.; Z.; Bovik, Bovik, A.A. C. C. [40 [40]] 0.550 0.550 0.438 0.438 PSNR Kwan, C.; et al. [34] 0.537 0.359 PSNRPSNR Kwan, Kwan, C.; C.; et al.al. [[34]34] 0.537 0.537 0.359 0.359 SSIM Structural similarity [35] 0.406 0.358 SSIMSSIM Structural Structural similarity similarity [ 35[35]] 0.406 0.406 0.358 0.358 VIF Sheikh, H. R.; Bovik, A. C. [41] 0.377 0.255 VIFVIF Sheikh, Sheikh, H. H. R.; R.; Bovik, Bovik, A.A. C. C. [41 [41]] 0.377 0.377 0.255 0.255 PQS Miyahara, M.; Kotani, K.; Algazi, V. R. [42] 0.302 0.242 PQSPQS Miyahara,Miyahara, M.; M.; Kotani, Kotani, K.; Algazi,Algazi, V. V. R. [R.42 ][42] 0.302 0.302 0.242 0.242

Hence, in addition to PSNR and SSIM, we also used HVS and HVSm for assessing perceptually Hence, in addition to PSNR and SSIM, we also used HVS and HVSm for assessing perceptually lossless compression. lossless compression. 3. Still Image Compression Results 3. Still Image Compression Results 3.1. Data 3.1. Data Computers 2019, 8, 32 7 of 21

Hence, in addition to PSNR and SSIM, we also used HVS and HVSm for assessing perceptually lossless compression.

3. Still Image Compression Results

Computers3.1. Data 2019,, 2,2, xx FORFOR PEERPEER REVIEWREVIEW 7 of 22

We searchedsearched thethe InternetInternet andand foundfound overover 30 low re resolutionsolution color images, over 10 high resolution maritime images, and over 10 sonarsonar images.images. Moreover Moreover,,, we we also included four highhigh qualityquality imagesimages from Xiph’sXiph’s website.website. Although Although th theseese images images from from Xiph Xiph are are not not rela relatedted to to maritime maritime or or sonar sonar images, images, wewe included included them them to to demonstrate demonstrate that that the the proposed proposed frameworkframework framework cancan bebe can usedused be usedforfor diversediverse for diverse images.images. images. TheseThese Theseimagesimages images werewere wereusedused usedtoto demonstratedemonstrate to demonstrate thatthatthat thethe the compcomp compressionression algorithms algorithms need need to to have have consistent performance for imagesimages withwith didifferentﬀerent resolutionsresolutions andand modalities.modalities. We applied fourfour compressioncompression algorithms: Daala, Daala, X264, X264, X265, X265, and and JPEG2000. All All of of them, them, except except JPEG2000, JPEG2000, are are DCT based compression algorithms. algorithms. For For each each image, image, we applie we appliedd these thesefour algorithms four algorithms to compress to compress the images the atimages different at di compressionﬀerent compression ratios. Fo ratios.ur performance Four performance metrics were metrics appliedlied were toto evaluateevaluate applied thethe to compression evaluatecompression the performance.compression performance.

3.1.1. Low Resolution Maritime Images We foundfound overover 3030 images,images, whichwhich areare ofof lowlow resolution.resolution.. If one zooms zooms in to to l looklook at the details, one can notice some artifacts.artifacts. Here, we include a few sample maritime imagesimages inin FigureFigure6 6..

Figure 6. LowLow resolution resolution maritime images in our study.

3.1.2. New High Resolution Maritime Images Those maritime images in Figure 66 areare ofof low low resolution. resolution. ItIt waswas ofof interestinterest toto seesee whetherwhether thethe compression performanceperformance would would change change with with diﬀerent different image image resolutions. resolutions. We found We 12 found high resolution 12 high resolutionmaritime images. maritime Two images. of them Two are of shown themthem are inare Figure shownshown7. inin FigureFigure 7.7.

Figure 7. New high resolution maritime images.

3.1.3. Sonar Images We foundfound moremore thanthan 1010 imagesimages fromfrom thethe Internet.Internet. SomeSome ofof themthem areare shownshown inin FigureFigure8 8.. Computers 2019, 8, 32 8 of 21 Computers 2019, 2, x FOR PEER REVIEW 8 of 22 Computers 2019, 2, x FOR PEER REVIEW 8 of 22

Figure 8. Sonar images in our compression study. FigureFigure 8. 8.Sonar Sonar imagesimages in our compression compression study. study. 3.1.4. High Quality Color Images 3.1.4.3.1.4. High High Quality Quality Color Color Images Images We also used four high quality images from [12]. As shown in Figure9, these images are neither WeWe also also used used four four high high quality quality images images fromfrom [12]. As shownshown in in Figure Figure 9, 9, these these images images are are neither neither maritimemaritime nor nor sonar sonar images. images. However, However, the the imagesimages ha haveve diverse imageimage contents contents and and we we would would like like to to demonstratedemonstrate that that our our framework framework can can also also eeffectivelyeffectivelyﬀectively compresscompress suchsuch images imagesimages at atat 10 1010 to toto 1 11compression. compression.compression.

(a) (b) (a) (b)

(c) (d) (c) (d) FigureFigure 9. 9.High High quality quality images images from from XiphXiph [[12]:12]: (aa)) Crepuscular; Crepuscular; (b (b) )Avenches; Avenches; (c ()c Bike;) Bike; and and (d) ( dWoman.) Woman. Figure 9. High quality images from Xiph [12]: (a) Crepuscular; (b) Avenches; (c) Bike; and (d) Woman. 3.2.3.2. Low Low Resolution Resolution Maritime Maritime Image Image Compression Compression ResultsResults 3.2. LowThereThere Resolution are are cameras cameras Maritime with with Image di differentﬀerent Compression resolutions resolutions Results onboardonboard naval ships. ships. The The objectives objectives of ofthe the study study were:were:There (1) (1) to toare compare compare cameras the the with performance performance different ofof resolutions fourfour compressioncomp ressiononboard algorithms algorithms naval ships. using using The four four objectives performance performance of themetrics metrics study and determine which algorithm has the best performance; and (2) to investigate the performance at were:and determine (1) to compare which the algorithm performance has the of bestfour performance; compression andalgorithms (2) to investigate using four the performance performance metrics at 0.1 0.1 compression ratio and see if we could achieve near or even perceptually lossless quality (> 40 dBs andcompression determine ratio which and algorithm see if we couldhas the achieve best performa near or evennce; and perceptually (2) to investigate lossless the quality performance (>40 dBs inat in HVSm) in the decompressed images. Let us focus on a region near 0.1 compression ratio in Figure 0.1HVSm) compression in the decompressed ratio and see images. if we could Let usachieve focus near on a or region even near perceptually 0.1 compression lossless ratioquality in Figure(> 40 dBs 10. 10. One can immediately observe some big variations in performance for different algorithms and inOne HVSm) can immediately in the decompressed observe some images. big variations Let us focus in performanceon a region near for di 0.1ﬀerent compression algorithms ratio and in metrics. Figure metrics. For low resolution maritime images, JPEG2000 performed the best, followed by Daala, in 10.For One low can resolution immediately maritime observe images, some JPEG2000 big variatio performedns in performance the best, followed for different by Daala, algorithms in terms and of PSNR.terms JPEG2000 of PSNR. JPEG2000 had better had performance better performance because because it uses the it uses whole the image,whole image, whereas whereas the other the other codecs metrics.codecs For use low small resolution blocks. However,maritime forimages, SSIM JPEG2000 and HVSm, performed Daala had the the best, highest followed scores. by ToDaala, our in use small blocks. However, for SSIM and HVSm, Daala had the highest scores. To our knowledge, the termsknowledge, of PSNR. the JPEG2000 high HVSm had bettevaluesr performanceare because the because researchers it uses at the Xiph whole have image, devoted whereas a lot of the their other codecs use small blocks. However, for SSIM and HVSm, Daala had the highest scores. To our knowledge, the high HVSm values are because the researchers at Xiph have devoted a lot of their Computers 2019, 2, x FOR PEER REVIEW 9 of 22 effort in fine-tuning and improving the perceptual quality of the decompressed images by removing small artifacts such as blocky artifacts. For HVS, Daala and JPEG2000 had similar performance (see Figure 10). We also generated a table summarizing the statistical performance of different codecs at 0.1 compression ratio. As shown in Table 2, JPEG2000 and Daala, except SSIM, yielded more than 40 dBs in PSNR, HVS, and HVSm, meaning that Daala and JPEG2000 could achieve near or even perceptually lossless compression. The SSIM metric for Daala was more than 0.9. For this dataset, we think Daala is the best codec because of its high scores in performance metrics and its parallel processing potential (see block processing in Figure 2). JPEG2000, on the other hand, requires the whole image and hence is not suitable for parallel processing. Figure 11 shows a visual comparison of decompressed images using different codecs at 10 to 1 compression ratio. It can be seen that JPEG2000 and Daala had no perceptually loss, whereas X264 hadComputers over smooth2019, 8, 32 reconstruction and X265 had some color distortions. 9 of 21

Table 2. Performance metrics of four codecs at 0.1 compression ratio for low resolution maritime highimages. HVSm Bold values numbers are because indicate the the researchersbest performing at Xiph method. have devoted a lot of their effort in fine-tuning and improving the perceptual quality of the decompressed images by removing small artifacts such as blocky artifacts. For HVS, Daala PSNR and (dB) JPEG2000 SSIM had HVS similar (dB) performance HVSm (dB) (see Figure 10). We also generated a table summarizing the statistical performance of different codecs at 0.1 compression ratio. As shown in TableJPEG20002, JPEG2000 47 and Daala,0.87 except SSIM,41 yielded46 more than 40 dBs in PSNR, HVS, and HVSm, meaningX264 that Daala 39 and JPEG2000 0.725 could 35.5 achieve near 40 or even perceptually lossless compression. The SSIM metric for Daala was more than 0.9. For this dataset, we think Daala is the best codec because of itsX265 high scores in 40 performance 0.85 metrics 38 and its parallel 44 processing potential (see block processing in Figure2). JPEG2000, on the other hand, requires the whole image and hence is not Daala 44 0.92 40.9 52.5 suitable for parallel processing.

(a) (b)

FigureFigure 10. 10. PerformancePerformance metrics metrics for for those those low low resolution resolution maritime maritime images: images: (a ()a )averaged averaged PSNR PSNR in in dB dB ofof all all maritime maritime images images versus versus compression compression ratio; ratio; (b (b) )averaged averaged SSIM SSIM of of all all maritime maritime images images versus versus compression ratio; (c) averaged HVS in dB of all maritime images versus compression ratio; and (d) averaged HVSm in dB of all maritime images versus compression ratio.

Table 2. Performance metrics of four codecs at 0.1 compression ratio for low resolution maritime images. Bold numbers indicate the best performing method.

PSNR (dB) SSIM HVS (dB) HVSm (dB) JPEG2000 47 0.87 41 46 X264 39 0.725 35.5 40 X265 40 0.85 38 44 Daala 44 0.92 40.9 52.5 Computers 2019, 8, 32 10 of 21

Computers 2019, 2, x FOR PEER REVIEW 10 of 22 Figure 11 shows a visual comparison of decompressed images using diﬀerent codecs at 10 to 1 compressioncompression ratio. ratio; It can (c) averaged be seen that HVS JPEG2000 in dB of all and maritime Daala hadimages no versus perceptually compression loss, whereasratio; and X264 (d) had overaveraged smooth reconstructionHVSm in dB of all and maritime X265 had images some versus color compression distortions. ratio.

(a) (b) (c)

(d) (e)

FigureFigure 11. 11. VisualVisual comparison comparison of of decompressed decompressed low low resolution resolution maritime maritime images images using using different diﬀerent codecs codecs atat 10 to 11 compressioncompression ratio: ratio: (a ()a original) original image; image; (b )( decompressedb) decompressed image image using using JPEG2000 JPEG2000 (HVSm: (HVSm: 44.38 44.38dBs); dBs); (c) decompressed (c) decompressed image image using using Daala Daala (HVSm: (HVSm: 52.38 52.38 dBs); dBs); (d) decompressed (d) decompressed image image using using X264 X264(HVSm: (HVSm: 38.03 38.03 dBs); dBs); and (ande) decompressed (e) decompressed image image using using X265 X265 (HVSm: (HVSm: 40.78 40.78 dBs). dBs).

3.3.3.3. High High Resolution Resolution Maritime Maritime Image Image Results Results TheThe purposepurpose ofof this this study study was was to see to whethersee whether the observations the observations for low for resolution low resolution images presented images presentedin Section in3.2 Section were still3.2 were valid still for valid high resolutionfor high resolution images. images. For this For dataset, this dataset, we also we focused also focused on the onregion the region around around 0.1 compression 0.1 compression ratio in ratio Figure in Figure12. Similar 12. Similar to the results to the results of the lower of the resolution lower resolution images, images,JPEG2000 JPEG2000 and Daala and were Daala signiﬁcantly were significantly better than better the others. than the JPEG2000 others. consistentlyJPEG2000 consistently produced a producedhigher PSNR a higher than Daala PSNR at than 0.1 compression Daala at 0.1 ratio. comp Asression explained ratio. above, As expl JPEG2000ained above, performs JPEG2000 wavelet performsdecomposition wavelet on thedecomposition whole image on and the hence whole had image better and energy hence compaction. had better However, energy Daalacompaction. tended However,to have stronger Daala SSIM, tended HVS, to andhave HVSm stronger because SSIM, Daala HVS, incorporated and HVSm overlapped because blocks Daala that incorporated suppressed overlappedblocky artifacts. blocks Daala that also suppressed put more blocky emphasis artifacts. on perceptual Daala also quality put bymore ﬁne-tuning emphasis its on algorithm. perceptual As qualityshown by in Tablefine-tuning3, Daala its reached algorithm. 60 dBs As forshown HVSm in Table at 0.1 3, compression Daala reached ratio, 60 whichdBs for is HVSm perceptually at 0.1 compressionlossless. The ratio, SSIM which of Daala is isperceptually more than 0.9.lossless. The higher The SSIM resolution of Daala images is more produced than 0.9. slightly The higher higher resolutionscores overall images as compared produced to slightly those inhigher Figure scor 10es. overall as compared to those in Figure 10. Figure 13 shows the visual comparison results at 10 to 1 compression. It was difficult to spot any Table 3. Performance metrics of four codecs at 0.1 compression ratio for high resolution maritime differences between the original and the decompressed images. images. Bold numbers indicate the best performing method.

Table 3. Performance metricsPSNR of (dB)four codecs at SSIM0.1 compression HVS ratio (dB) for high resolution HVSm (dB) maritime images. Bold numbers indicate the best performing method. JPEG2000 53.5 0.825 46 50 X264 45 0.7 42HVSm 46 PSNR (dB) SSIM HVS (dB) X265 47 0.875 45(dB) 53 DaalaJPEG2000 50.553.5 0.8250.91 46 49 50 60

X264 45 0.7 42 46

X265 47 0.875 45 53

Daala 50.5 0.91 49 60

Computers 2019, 8, 32 11 of 21 Computers 2019, 2, x FOR PEER REVIEW 11 of 22

Computers 2019, 2, x FOR PEER REVIEW 11 of 22

(a) (b)

FigureFigure 12. 12.Performance Performance metrics metrics forfor thosethose high resolution resolution maritime maritime images: images: (a) averaged (a) averaged PSNR PSNR in dB in dB of allof maritimeall maritime images images versus versuscompression compression ratio; ratio; (b (b) )averaged averaged SSIM SSIM of all of maritime all maritime images images versus versus compression ratio; (c) averaged HVS in dB of all maritime images versus compression ratio; and (d) compression ratio; (c) averaged(c) HVS in dB of all maritime images versus( compressiond) ratio; and (d) averaged HVSm in dB of all maritime images versus compression ratio. averaged HVSm in dB of all maritime images versus compression ratio. Figure 12. Performance metrics for those high resolution maritime images: (a) averaged PSNR in dB of all maritime images versus compression ratio; (b) averaged SSIM of all maritime images versus Figure 13 shows the visual comparison results at 10 to 1 compression. It was diﬃcult to spot any compression ratio; (c) averaged HVS in dB of all maritime images versus compression ratio; and (d) diﬀerencesaveraged between HVSm the in original dB of all maritime and the images decompressed versus compression images. ratio.

(a) (b) (c)

(d) (e)

Figure 13. Visual comparison of decompressed high resolution maritime images using different codecs at 10 to 1 compression ratio: (a) original image; (b) decompressed image using JPEG2000

(HVSm: 50.32 dBs); (c) decompressed image using Daala (HVSm: 59.40 dBs); (d) decompressed image (d) (e) using X264 (HVSm: 46.08 dBs); and (e) decompressed image using X265 (HVSm: 53.13 dBs). FigureFigure 13. Visual13. Visual comparison comparison of decompressedof decompressed high high resolution resolution maritime maritime images images using using di ﬀdifferenterent codecs at 10codecs to 1 compression at 10 to 1 compression ratio: (a) original ratio: ( image;a) original (b) decompressedimage; (b) decompressed image using image JPEG2000 using JPEG2000 (HVSm: 50.32 dBs);(HVSm: (c) decompressed 50.32 dBs); (c) image decompressed using Daala image (HVSm: using Daala 59.40 (HVSm: dBs); 59.40 (d) decompressed dBs); (d) decompressed image usingimage X264 using X264 (HVSm: 46.08 dBs); and (e) decompressed image using X265 (HVSm: 53.13 dBs). (HVSm: 46.08 dBs); and (e) decompressed image using X265 (HVSm: 53.13 dBs). Computers 2019, 2, x FOR PEER REVIEW 12 of 22

3.4. High Quality Color Image Compression Results Here, we present results using the high quality color images in Section 3.1.4. First, the images presented in Section 3.1.4 are neither maritime nor sonar images. The purpose of this study was to see whether we could observe similar trends to those presented above. For this dataset, we also focused on the region around 0.1 compression ratio in Figure 14. JPEG2000 continued to perform well in terms of PSNR and SSIM. However, for HVS and HVSm, Daala and X265 had better performance. In Table 4, one can clearly see that Daala resulted 48.5 dBs in HVSm. This further corroborated that, at 10 to 1 compression, Daala could achieve near perceptually lossless compression. Figure 15 compares the various decompressed images at 10 to 1 compression. Again, it was hard to see any perceptual differences between the original and the decompressed images.

Table 4. Performance metrics of four codecs at 0.1 compression ratio for high quality color images. ComputersBold2019 numbers, 8, 32 indicate the best performing method. 12 of 21

HVSm PSNR (dB) SSIM HVS (dB) 3.4. High Quality Color Image Compression Results (dB) Here, we present results using the high quality color images in Section 3.1.4. First, the images JPEG2000 44 0.84 37.5 42 presented in Section 3.1.4 are neither maritime nor sonar images. The purpose of this study was to see whether we could observeX264 similar trends 38 to those 0.67 presented above. 36 For this 33 dataset, we also focused on the region around 0.1 compression ratio in Figure 14. JPEG2000 continued to perform well in terms of PSNR and SSIM. However,X265 for HVS 38.5 and HVSm, 0.75 Daala and X265 38 had better 45 performance. In Table4, one can clearly see that Daala resulted 48.5 dBs in HVSm. This further corroborated that, at 10 to 1 Daala 41 0.82 39 48.5 compression, Daala could achieve near perceptually lossless compression.

(a) (b)

Figure 14. PerformancePerformance metrics metrics for for those those high high quality quality color color images: images: (a) ( aaveraged) averaged PSNR PSNR in dB in dBof all of allimages images versus versus compression compression ratio; ratio; (b) ( baveraged) averaged SSIM SSIM of of all all images images versus versus compression compression ratio; ratio; ( c) averaged HVSHVS inin dB dB of of all all mages mages versus versus compression compression ratio; ratio; and (andd) averaged (d) averaged HVSm HVSm in dB ofin all dB images of all versusimagescompression versus compression ratio. ratio.

Table 4. Performance metrics of four codecs at 0.1 compression ratio for high quality color images. Bold numbers indicate the best performing method.

PSNR (dB) SSIM HVS (dB) HVSm (dB) JPEG2000 44 0.84 37.5 42 X264 38 0.67 36 33 X265 38.5 0.75 38 45 Daala 41 0.82 39 48.5

Figure 15 compares the various decompressed images at 10 to 1 compression. Again, it was hard to see any perceptual diﬀerences between the original and the decompressed images. Computers 2019, 8, 32 13 of 21 Computers 2019, 2, x FOR PEER REVIEW 13 of 22

(a) (b) (c)

(d) (e)

FigureFigure 15. 15. VisualVisual comparison comparison of of decompressed decompressed high high quality quality color color images images using using different diﬀerent codecs codecs at at 10 10 toto 1 1 compression ratio:ratio: (a(a)) original original image; image; (b ()b decompressed) decompressed image image using using JPEG2000 JPEG2000 (HVSm: (HVSm: 42.59 42.59 dBs); dBs);(c) decompressed (c) decompressed image image using using Daala Daala (HVSm: (HVSm: 52.16 dBs);52.16 ( ddBs);) decompressed (d) decompressed image usingimage X264 using (HVSm: X264 (HVSm:43.34 dBs); 43.34 and dBs); (e) decompressedand (e) decompressed image using image X265 using (HVSm: X265 (HVSm: 47.27 dBs). 47.27 dBs). 3.5. Sonar Image Compression Results 3.5. Sonar Image Compression Results Sonar images are also of interest to our sponsor and hence we included them in our study. Similar Sonar images are also of interest to our sponsor and hence we included them in our study. to in Sections 3.2–3.4, we determined which one of the four algorithms was the best in terms of the four Similar to in Sections 3.2 to 3.4, we determined which one of the four algorithms was the best in terms performance metrics. Moreover, we attmepted to achieve perpetually lossless compression (>40 dBs) of the four performance metrics. Moreover, we attmepted to achieve perpetually lossless compression in terms of HVSm at 0.1 compression ratio. Figure 16 shows the four metrics of diﬀerent codecs for (>40 dBs) in terms of HVSm at 0.1 compression ratio. Figure 16 shows the four metrics of different the sonar images. The results comparing JPEG2000, Daala, X264, and X265 are very similar to the codecs for the sonar images. The results comparing JPEG2000, Daala, X264, and X265 are very similar low and high resolution maritime results. JPEG2000 and Daala consistently outperformed X264 and to the low and high resolution maritime results. JPEG2000 and Daala consistently outperformed X264 X265, which were nearly identical. JPEG2000 yielded the strongest PSNR across all compression rates and X265, which were nearly identical. JPEG2000 yielded the strongest PSNR across all compression because of high energy compaction due to wavelet transform. At compression rates higher than 0.1, rates because of high energy compaction due to wavelet transform. At compression rates higher than Daala produced a stronger SSIM, HVS, and HVSm because of strong emphasis in perceptual quality by 0.1, Daala produced a stronger SSIM, HVS, and HVSm because of strong emphasis in perceptual the developers of Daala. As shown in Table5, Daala and JPEG2000 had more than 40 dBs in HVSm. quality by the developers of Daala. As shown in Table 5, Daala and JPEG2000 had more than 40 dBs This means that near or even perceptually lossless compression could be achieved. in HVSm. This means that near or even perceptually lossless compression could be achieved. Figure 17 shows the comparison between original and decompressed sonar images. One can Table 5. Performance metrics of four codecs at 0.1 compression ratio for sonar images. Bold numbers hardly see any differences because the HVSm scores were high. indicate the best performing method. Table 5. Performance metrics of four codecs at 0.1 compression ratio for sonar images. Bold numbers PSNR (dB) SSIM HVS (dB) HVSm (dB) indicate the best performing method. JPEG2000 42.5 0.815 38 42.5 X264 36 0.66 30.5HVSm 34 PSNR (dB) SSIM HVS (dB) X265 37 0.75 34(dB) 38

DaalaJPEG2000 4042.5 0.8150.875 38 37 42.5 44.5

X264 36 0.66 30.5 34

X265 37 0.75 34 38

Daala 40 0.875 37 44.5

Computers 2019, 8, 32 14 of 21 Computers 2019, 2, x FOR PEER REVIEW 14 of 22

(a) (b)

FigureFigure 16. 16. PerformancePerformance metrics metrics for for those those sonar sonar images. images. (a ()a averaged) averaged PSNR PSNR in in dB dB of of all all sonar sonar images images versusversus compression compression ratio; (b(b)) averaged averaged SSIM SSIM of allof sonarall sonar images imag versuses versus compression compression ratio; ( cratio;) averaged (c) averagedHVS in dBHVS of in all dB sonar of all images sonar versusimages compression versus compression ratio; and ratio; (d) averagedand (d) averaged HVSm inHVSm dB of in all dB sonar of allimages sonar versusimages compressionversus compression ratio. ratio.

Figure 17 shows the comparison between original and decompressed sonar images. One can hardly see any diﬀerences because the HVSm scores were high.

3.6. Error Concealment for Maritime Images As mentioned above, communication channels in maritime environments have strong and random interferences, especially in wireless channels, which create packet errors. Even with good error

correction coding, it is unavoidable to have some missing packets. We recommend that the missing data should be recovered by advanced error concealment algorithms, which do not add any overhead

to the network bandwidth. In this research, we evaluated two error concealment algorithms: our own algorithm [28] and a commercial product called Transformic [43]. We randomly introduced corrupted blocks of sizes 16 16, 8 8, 4 4, and 2 2 in color images and then applied the concealment algorithms to × × × × recover the missing data. Our objective was to see if we could conceal the errors introduced in the communication channel.

Computers 2019, 8, 32 15 of 21 Computers 2019, 2, x FOR PEER REVIEW 15 of 22

(a) (b) (c)

(d) (e)

FigureFigure 17. 17. VisualVisual comparison comparison of of decompressed decompressed sonar sonar images images using using different diﬀerent codecs codecs at at 10 10 to to 1 1 compressioncompression ratio: ratio: ( (aa)) original image;image; ((bb)) decompresseddecompressed image image using using JPEG2000 JPEG2000 (HVSm: (HVSm: 50.85 50.85 dBs); dBs); (c ) (Decompressedc) Decompressed image image using using Daala Daala (HVSm: (HVSm: 58.73 58.73 dBs);dBs); ((dd)) decompressed image image using using X264 X264 (HVSm: (HVSm: 46.5146.51 dBs); dBs); and and ( (ee)) decompressed decompressed image image using using X265 X265 (HVSm: (HVSm: 50.19 50.19 dBs). dBs).

3.6.1. Error Recovery in Maritime Images 3.6. Error Concealment for Maritime Images We used two images to illustrate the performance of error recovery. Mean square error (MSE) was As mentioned above, communication channels in maritime environments have strong and used as the objective metric to compare the two algorithms. In addition, we visually inspected the random interferences, especially in wireless channels, which create packet errors. Even with good recovered images and performed subjective evaluation. error correction coding, it is unavoidable to have some missing packets. We recommend that the missingMaritime data Image should 1 be recovered by advanced error concealment algorithms, which do not add any overhead to the network bandwidth. InFigure this research, 18a shows we theevaluated original two image error where concea alment few red algorithms: blocks indicate our own some algorithm areas of[28] interest. and a commercialFigure 18b showsproduct the called locations Transfor of somemic [43]. corrupted We randomly blocks. Weintroduced then applied corrupted two algorithmsblocks of sizes to repair 16 × 16,those 8 × 8, corrupted 4 × 4, and blocks. 2 × 2 in Wecolor used images RMSE and tothen compare applied the the reconstruction concealment algorithms performance to recover using twothe missingalgorithms. data. For Our this objective image, was the RMSEto see if using we could our methodconceal wasthe errors 6.69 and introduced the RMSE in forthe Transformiccommunication was channel.7.18. When visually inspecting the recovered pixels, one could see that the diﬀerences between our method and Transformic was huge. Comparing Figure 18c–e, it can be seen that Transformic failed to 3.6.1.recover Error the Recovery crane, whereas in Maritime our method Images could successfully recover the crane. In addition, by inspecting Figure 18f–h, one can see that Transformic could not recover the missing block near the building, We used two images to illustrate the performance of error recovery. Mean square error (MSE) whereas our method could reconstruct the missing block. This clearly shows that our method could was used as the objective metric to compare the two algorithms. In addition, we visually inspected eﬀectively conceal the corrupted blocks. the recovered images and performed subjective evaluation. Maritime Image 1 Figure 18a shows the original image where a few red blocks indicate some areas of interest. Figure 18b shows the locations of some corrupted blocks. We then applied two algorithms to repair those corrupted blocks. We used RMSE to compare the reconstruction performance using two Computers 2019, 2, x FOR PEER REVIEW 16 of 22 algorithms. For this image, the RMSE using our method was 6.69 and the RMSE for Transformic was 7.18. When visually inspecting the recovered pixels, one could see that the differences between our method and Transformic was huge. Comparing Figure 18c–e, it can be seen that Transformic failed to recover the crane, whereas our method could successfully recover the crane. In addition, by inspecting Figure 18f–h, one can see that Transformic could not recover the missing block near the Computersbuilding,2019 whereas, 8, 32 our method could reconstruct the missing block. This clearly shows that16 ofour 21 method could effectively conceal the corrupted blocks.

(a) (b)

(f) (g) (h)

Figure 18.18.Comparison Comparison of dataof data recovery recovery methods. methods. Our method Our method performs performs much better much than Transformic:better than (Transformic:a) original image (a) original with highlighted image with boxes; highlighted (b) locations boxes; that (b) have locations missing that blocks; have (missingc) original; blocks; (d) our (c) method;original; (ed)) Transformic;our method; ((fe)) original;Transformic; (g) our (f) method;original; and(g) our (h) Transformic.method; and Please(h) Transformic. note that the Please red rectanglesnote that the do red not necessarilyrectangles do cover not thenecessarily same areas co inver the the three same images. areas in The the purpose three images. of the red The rectangles purpose isof simplythe red torectangles highlight is some simply areas to highlight of interest. some areas of interest.

Maritime Image 2 Figure 1919aa shows a number of corruptedcorrupted blocks in an image.image. Figure 1919bb shows the zoomed in areas of twotwo locations.locations. Figure Figure 19c,d19c,d presents the reconstructed images using our method and the Transformic method,method, respectively. respectively. For For this this image, image, the the RMSE RMSE using using our our matrix matrix completion completion method method was 5.4was and 5.4 theand RMSE the RMSE for Transformic for Transformic wais 5.55.wais Although5.55. Although the di theﬀerence difference between between the MSEs the wasMSEs small, was thesmall, perceptual the perceptual appearance appearance of our of method our method was much was much better better (see Figure(see Figure 19c,d). 19c,d). In particular, In particular, the gapthe betweengap between the two the buildings two buildings was reconstructed was reconstructed correctly correctly by our method. by our Thismethod. example This furtherexample highlights further thehighlights importance the importance of error concealment of error concealment because, even beca if oneuse, uses even error if one correction uses error coding, correction there may coding, still bethere corrupted may still packets be corrupted during packets image transmission. during image transmission. Computers 2019, 8, 32 17 of 21 Computers 2019, 2, x FOR PEER REVIEW 17 of 22

(b) (a)

Figure 19. OurOur reconstructed reconstructed image looks much better than that of Transformic. Our method is better than Transformic: ( a) image with corrupted blocks; ( b) original image; ( c) reconstructed image using our method; andand ((dd)) reconstructedreconstructed image image using using Transformic. Transformic. Please Please note note that that the the red red rectangles rectangles do notdo notnecessarily necessarily cover cover the the same same areas areas in the in the three three images. images. The The purpose purpose of theof the red red rectangles rectangles is simplyis simply to tohighlight highlight some some areas areas of interest.of interest.

3.6.2. Error Recovery in Sonar Images 3.6.2. Error Recovery in Sonar Images Similar to the study presented in Section 3.6.1, we applied two error concealment methods to Similar to the study presented in Section 3.6.1, we applied two error concealment methods to repair damaged blocks in the decompressed images. repair damaged blocks in the decompressed images. Sonar Image 1 Figure 20a20a illustrates the locations of the damage damagedd blocks. Figure 20b20b shows the zoomed in areas of some regions of interest. Two methods were appliedapplied to repair thethe damageddamaged blocks.blocks. Figure 2020c,dc,d shows the reconstructed images. For For this this image, ou ourr method achieved a RMSE of 22.68, whereas the RMSE forfor TransformicTransformic was was 71.32. 71.32. The The di ﬀdifferenceerence was was tremendous. tremendous. By visuallyBy visually inspecting inspecting Figure Figure 20c,d, 20c,d,one can one see can that see our that results our areresults clear are and clear have and more have textures more as textures compared as tocompared the results toof the Transformic. results of Transformic. Sonar Image 2 To further demonstrate the error concealment for sonar images, we include another example. Figure 21a shows the locations of regions of interest with corrupted blocks. Figure 21b,c shows the reconstructed image using the two methods. For this image, the RMSE of our method was 21.87 and the RMSE for Transformic was 65.55. Again, the diﬀerence between RMSEs was huge. By visually inspecting the reconstructed results in Figure 21c,d, one can see that our results look much better than that of Transformic. Computers 2019, 8, 32 18 of 21

Computers 2019, 2, x FOR PEER REVIEW 18 of 22

(a) (b)

FigureFigure 20. 20.Sonar Sonar image image reconstruction reconstruction results results comparison. comparison. Our Our method method is much is much better better than Transformic:than (a) randomlyTransformic: missing (a) randomly blocks; (bmissing) original blocks; image ( withb) original highlighted image boxes; with (chighlighted) reconstructed boxes; image (c) using ourreconstructed method; and image (d) reconstructed using our method; image and using (d) reconstructed Transformic. image Please using note Transformic. that the red Please rectangles note do not that the red rectangles do not necessarily cover the same areas in the three images. The purpose of the necessarily cover the same areas in the three images. The purpose of the red rectangles is simply to red rectangles is simply to highlight some areas of interest. highlightComputers 2019 some, 2, x areas FOR PEER of interest. REVIEW 19 of 22 Sonar Image 2 To further demonstrate the error concealment for sonar images, we include another example. Figure 21a shows the locations of regions of interest with corrupted blocks. Figure 21b,c shows the reconstructed image using the two methods. For this image, the RMSE of our method was 21.87 and the RMSE for Transformic was 65.55. Again, the difference between RMSEs was huge. By visually inspecting the reconstructed results in Figure 21c,d, one can see that our results look much better than that of Transformic. (a)

(b)

(c)

FigureFigure 21. Sonar 21. Sonar image image reconstruction reconstruction results results comparison. compar Ourison. method Our method is much is bettermuch than better Transformic: than (a) originalTransformic: image (a with) original highlighted image with areas; highlighted (b) our results; areas; (b and) our (c )results; Transformic and (c) results.Transformic Please results. note that the redPlease rectangles note that do the not red necessarily rectangles coverdo not the necess samearily areas cover in the the same three areas images. in the The three purpose images. of The the red purpose of the red rectangles is simply to highlight some areas of interest. rectangles is simply to highlight some areas of interest. 3.7. Discussions As mentioned above, our sponsor is interested in achieving 10 to 1 compression or a compression ratio of 0.1 using new or existing compression algorithms and, at the same time, the decompressed image should be near perceptually lossless (40 dBs or more in terms of HVSm). These were the first and second goals. To meet these goals, we collected low and high resolution images (maritime images) and sonar images. The reason for using low and high quality images was because there are cameras with different resolutions onboard the naval ships. It was therefore required that the compression algorithm should work satisfactorily for all images, including sonar images as well. We chose four metrics, two of which (PSNR and SSIM) have been widely used by many people before, but they do not correlate well with human perceptions. The two other metrics, HVS and HVSm, have better correlation with human perceptions. From those metrics for low and high quality optical images and sonar images, we observed that Daala had consistently reached more than 40 dBs at 10 to 1 compression in HVSm in all images that we tested. Other codecs such as X265 and JPEG2000 could also marginally meet the above requirements. It is up to our sponsor to make the final decision on which codec to adopt. Our third objective was to demonstrate the performance of error concealment, which does not incur any additional bandwidth usage. Our experiments showed that error concealment could indeed recover image pixels in those corrupted areas.

4. Conclusions We would like to emphasize that our work was different from those papers on perceptually lossless coding [15–18], which embed HVS model into the coding process. One key objective in our research was to achieve near or even perceptually lossless compression with 0.1 compression ratio for still images (maritime and sonar). The requirement was raised by our customer. We evaluated Computers 2019, 8, 32 19 of 21

3.7. Discussions As mentioned above, our sponsor is interested in achieving 10 to 1 compression or a compression ratio of 0.1 using new or existing compression algorithms and, at the same time, the decompressed image should be near perceptually lossless (40 dBs or more in terms of HVSm). These were the first and second goals. To meet these goals, we collected low and high resolution images (maritime images) and sonar images. The reason for using low and high quality images was because there are cameras with different resolutions onboard the naval ships. It was therefore required that the compression algorithm should work satisfactorily for all images, including sonar images as well. We chose four metrics, two of which (PSNR and SSIM) have been widely used by many people before, but they do not correlate well with human perceptions. The two other metrics, HVS and HVSm, have better correlation with human perceptions. From those metrics for low and high quality optical images and sonar images, we observed that Daala had consistently reached more than 40 dBs at 10 to 1 compression in HVSm in all images that we tested. Other codecs such as X265 and JPEG2000 could also marginally meet the above requirements. It is up to our sponsor to make the final decision on which codec to adopt. Our third objective was to demonstrate the performance of error concealment, which does not incur any additional bandwidth usage. Our experiments showed that error concealment could indeed recover image pixels in those corrupted areas.

4. Conclusions We would like to emphasize that our work was diﬀerent from those papers on perceptually lossless coding [15–18], which embed HVS model into the coding process. One key objective in our research was to achieve near or even perceptually lossless compression with 0.1 compression ratio for still images (maritime and sonar). The requirement was raised by our customer. We evaluated four popular algorithms (JPEG2000, Daala, X264, X265) using four performance metrics (PSNR, SSIM, HVS, and HVSm). In our compression experiments, JPEG2000 performed the best in terms of PSNR at 0.1 compression ratio. However, for all the other metrics, Daala achieved the best scores. It was surprising to ﬁnd that X264 and X265 did not perform as well as Daala. Perhaps because those codecs are video codecs and were not optimally designed for still images. For 0.1 compression ratio, we found that near perceptually lossless compression could be achieved for still images, as the performance metrics were very high ( >40 dBs for PSNR, HVS, and HVSm). In addition, we think Daala is a good choice for practical applications because it is amenable to parallel processing. Our observations can be corroborated by another independent study performed by the Xiph team [44]. We also investigated error concealment algorithms for handling corrupted pixels due to transmission errors. Extensive experiments demonstrated that error concealment is a feasible method to conceal corrupted pixels without incurring additional bandwidth usage.

Author Contributions: C.K. wrote the paper. J.L., B.B., B.C., and E.S. customized the algorithms and generated all the plots and figures. T.T. provided high level guidance. Funding: This research was supported in part by the US Navy under contract N00024-14-P-4560. Acknowledgments: The views, opinions and/or findings expressed are those of the author and should not be interpreted as representing the official views or policies of the Department of Defense or the U.S. Government. We would also like to express our sincere gratitude towards the reviewers for their detailed comments and suggestions, which tremendously improve the quality of our paper. Conflicts of Interest: The authors declare no conflict of interest.

References

1. Strang, G.; Nguyen, T. Wavelets and Filter Banks; Wellesley-Cambridge Press: Wellesley, MA, USA, 1997. 2. Kwan, C.; Li, B.; Xu, R.; Li, X.; Tran, T.; Nguyen, T. A Complete Image Compression Codec Based on Overlapped Block Transform. Eurosip J. Appl. Signal Process. 2006, 2006, 010968. [CrossRef] Computers 2019, 8, 32 20 of 21

3. Pennebaker, W.B.; Mitchell, J.L. JPEG: Still Image Data Compression Standard; Springer Science & Business Media: Berlin/Heidelberg, Germany, 1993. 4. Kwan, C.; Larkin, J. Perceptually Lossless Compression for Mastcam Images. In Proceedings of the 9th IEEE Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (IEEE UEMCON 2018), New York, NY, USA, 8–10 November 2018. 5. Kwan, C.; Chou, B.; Larkin, J.; Budavari, B. Perceptually Lossless Compression of Mastcam Images with Error Recovery. In Proceedings of the Signal Processing, Sensor/Information Fusion, and Target Recognition XXVIII (Conference SI219), Baltimore, MD, USA, 15–17 April 2019. 6. Kwan, C.; Li, B.; Xu, R.; Tran, T.; Nguyen, T. SAR Image Compression Using Wavelets. Proc. SPIE 2001, 4391, 349–357. [CrossRef] 7. JPEG. Available online: https://en.wikipedia.org/wiki/JPEG (accessed on 26 April 2019). 8. JPEG-2000. Available online: https://en.wikipedia.org/wiki/JPEGhttp://en.wikipedia.org/wiki/JPEG_2000 (accessed on 26 April 2019). 9. X264. Available online: http://www.videolan.org/developers/x264.html (accessed on 26 April 2019). 10. X265. Available online: https://www.videolan.org/developers/x265.html (accessed on 26 April 2019). 11. Tran, T.D.; Liang, J.; Tu, C. Lapped transform via time-domain pre-and post-filtering. Ieee Trans. Signal Process. 2003, 51, 1557–1571. [CrossRef] 12. Daala. Available online: https://xiph.org/daala/ (accessed on 26 April 2019). 13. Valin, J.-M.; Terriberry, T.B. Perceptual Vector Quantization for Video Coding. Vis. Inf. Process. Commun. VI 2015, 9410, 941009. 14. Watson, A.B. Digital Images and Human Vision; The MIT Press: Cambridge, MA, USA, 1993. 15. Wu, H.R.; Reibman, A.; Lin, W.; Pereira, F.; Hemami, S. Perceptual visual signal compression and transmission. Proc. Ieee 2013, 101, 2025–2043. [CrossRef] 16. Wu, D.; Tan, D.M.; Baird, M.; DeCampo, J.; White, C.; Wu, H.R. Perpetually lossless coding of medical images. Ieee Trans. Med. Imaging 2006, 25, 335–344. [CrossRef][PubMed] 17. Oh, H.; Bilgin, A.; Marcellin, M.W. Visually lossless encoding for JPEG2000. Ieee Trans. Image Process. 2013, 22, 189–201. [PubMed] 18. Tan, D.M.; Wu, D. Perceptually lossless and perceptually enhanced image compression system & method. U.S. Patent 9,516,315, 6 December 2016. 19. Kwan, C.; Shang, E.; Tran, T. Perceptually lossless image compression with error recovery. In Proceedings of the 2nd International Conference on Vision, Image and Signal Processing, Las Vegas, NV, USA, 27–29 July 2018. 20. JPEG-XR. Available online: https://en.wikipedia.org/wiki/JPEG_XR (accessed on 26 April 2019). 21. VP8. Available online: http://en.wikipedia.org/wiki/VP8 (accessed on 26 April 2019). 22. VP9. Available online: http://en.wikipedia.org/wiki/VP9 (accessed on 26 April 2019). 23. Zhou, J.; Kwan, C. High Performance Image Completion using Sparsity based Algorithms. In Proceedings of the Computational Imaging III, Orlando, FL, USA, 15–19 April 2018. 24. Zhou, J.; Ayhan, B.; Kwan, C.; Tran, T. ATR Performance Improvement Using Images with Corrupted or Missing Pixels. In Proceedings of the Pattern Recognition and Tracking XXIX, Orlando, FL, USA, 18–19 April 2018. 25. Chang, C. Hyperspectral Imaging: Techniques for Spectral Detection and Classification; Springer Science & Business Media: Berlin/Heidelberg, Germany, 2003. 26. Kwan, C.; Ayhan, B.; Chen, G.; Chang, C.; Wang, J.; Ji, B. A Novel Approach for Spectral Unmixing, Classification, and Concentration Estimation of Chemical and Biological Agents. Ieee Trans. Geosci. Remote Sens. 2005, 44, 409–419. [CrossRef] 27. Qu, Y.; Qi, H.; Ayhan, B.; Kwan, C.; Kidd, R. Does Multispectral/Hyperspectral Pansharpening Improve the Performance of Anomaly Detection? In Proceedings of the 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), Fort Worth, TX, USA, 23–28 July 2017; pp. 6130–6133. 28. Zhou, J.; Kwan, C.; Ayhan, B. A High Performance Missing Pixel Reconstruction Algorithm for Hyperspectral Images. In Proceedings of the 2nd. International Conference on Applied and Theoretical Information Systems Research, Taipei, Taiwan, 27–29 December 2012. 29. Kwan, C.; Zhou, J. Method for Image Denoising. U.S. Patent 9,159,121, 13 October 2015. Computers 2019, 8, 32 21 of 21

30. Elad, M. Sparse and Redundant Representations: From Theory to Applications in Signal and Image Processing; Springer Science & Business Media: Berlin/Heidelberg, Germany, 2010. 31. Dao, M.; Kwan, C.; Koperski, K.; Marchisio, G. A Joint Sparsity Approach to Tunnel Activity Monitoring Using High Resolution Satellite Images. In Proceedings of the 8th IEEE Annual Ubiquitous Computing, Electronics & Mobile Communication Conference, New York, NY, USA, 19–21 October 2017; pp. 322–328. 32. Ayhan, B.; Kwan, C. On the use of Radiance Domain for Burn Scar Detection under Varying Atmospheric Illumination Conditions and Viewing Geometry. J. Signal Image Video Process. 2016, 11, 605–612. [CrossRef] 33. Ponomarenko, N.; Silvestri, F.; Egiazarian, K.; Carli, M.; Astola, J.; Lukin, V. On between-coefficient contrast masking of DCT basis functions. In Proceedings of the Third International Workshop on Video Processing and Quality Metrics for Consumer Electronics VPQM-07, Scottsdale, AZ, USA, 25–26 January 2007. 34. Kwan, C.; Zhu, X.; Gao, F.; Chou, B.; Perez, D.; Li, J.; Shen, Y.; Koperski, K. Assessment of Spatiotemporal Fusion Algorithms for Planet and Worldview Images. Sensors 2018, 18, 1051. [CrossRef][PubMed] 35. SSIM. Available online: https://en.wikipedia.org/wiki/Structural_similarity (accessed on 26 April 2019). 36. Ochoa, H.D.; Rao, K.R. Discrete Cosine Transform, 2nd ed.; CRC Press: Boca Raton, FL, USA, 2019. 37. Egiazarian, K.; Astola, J.; Ponomarenko, N.; Lukin, V.; Battisti, F.; Carli, M. New full-reference quality metrics based on HVS. In Proceedings of the Second International Workshop on Video Processing and Quality Metrics, Scottsdale, AZ, USA, 22–24 January 2006. 38. Damera-Venkata, N.; Kite, T.D.; Geisler, W.S.; Evans, B.L.; Bovik, A.C. Image quality assessment based on a degradation model. Ieee Trans. Image Process. 2000, 9, 636–650. [CrossRef][PubMed] 39. Watson, A.B.; Solomon, J.A.; Ahumada, A.J.; Gale, A. Discrete cosine transform (DCT) basis function visibility: Effects of viewing distance and contrast masking. Proc. SPIE 1994, 2179. [CrossRef] 40. Wang, Z.; Bovik, A.C. A universal image quality index. Ieee Signal Process. Lett. 2002, 9, 81–84. [CrossRef] 41. Sheikh, H.R.; Bovik, A.C. Image information and visual quality. Ieee Trans. Image Process. 2006, 15, 430–444. [CrossRef][PubMed] 42. Miyahara, M.; Kotani, K.; Algazi, V.R. Objective picture quality scale (PQS) for image coding. Ieee Trans. Commun. 1998, 46, 1215–1226. [CrossRef] 43. Transformic. Available online: http://www.vision.ee.ethz.ch/~{}mansfiea/transformic/ (accessed on 26 April 2019). 44. New Daala. Available online: https://people.xiph.org/~{}jm/daala/revisiting/ (accessed on 26 April 2019).

© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).